The BugBash Podcast

You hit 100% test coverage, deploy to production, and the system still falls over. An old story, maybe aside from the 100% coverage percent part. In this episode, David sits down with Marco Primi and Sean Allen from the Antithesis Red Team to talk about a likely missing piece in your testing strategy: workloads. They break down exactly what workloads are, why they are so helpful for testing distri...

Why do LLMs struggle to build complex architecture?  According to Unmesh Joshi, Distinguished Engineer at ThoughtWorks, it often comes down to a lack of shared abstractions, of a shared vocabulary. Today he joins Will Wilson and David Wynn to unpack how we learn about, build, and test distributed systems. We cover the value of building miniature versions of systems like Kafka to grasp the basics,...

Can you stress-test a React UI completely hands-free? In this episode, Oskar Wickström and I are joined by Jonathan Chan to discuss the origins of Fuzzmap. From a janitor’s closet in Figma—where end-to-end tests once lived on a "spicy pillow" MacBook—Jonathan shares how he applied coverage-guided fuzzing (inspired by AFL) to the world of web front-ends. We dive into the gnarly hacks requ...

With the "Agentic Era" coming online, teams are onboarding new AI "minions" into our codebases every thirty minutes. So a critical question emerges: Are we actually becoming more productive, or just more busy? Jessica Kerr (Jessitron) joins me to explore symmathesis—the theory of learning systems made of learning parts. We dive into viewing software as a teammate, how the defin...

Today we’re talking with Rain from Oxide Computing, tracing their journey from the massive, data-driven scale of Meta to the high-stakes, air-gapped world of shipping a 'cloud in a box'. We talk about moving from an environment of over 10,000 engineers—where a 10% tooling improvement is worth a thousand people—to a culture where you’re shipping hardware that simply cannot be patched once i...

Everyone wants reliable software. But nobody wants to test messy legacy code. Today, Lewis Campbell from Outdata joins the show to share a practical approach for bringing deterministic simulation testing to existing systems.  We get into why React components are terrible for business logic, and why front end static typing stops non-determinism at the door. We also take on non-tech aspects, like th...

Building the rr replay debugger wasn't about academic purity. It was about survival. Today, creator Rob O'Callahan reveals the "necessary” complexity that makes time-travel debugging possible on Firefox. We dig into the engineering nightmares behind the tool: from patching 2-byte system calls to the constant fear of silent hardware failures. Rob explains how rr became a "protecte...

It’s been nine years since Designing Data-Intensive Applications became the standard text for distributed systems. Today, Martin Kleppmann and Chris Riccomini join the show to pull back the curtain on the upcoming Second Edition. After all, the era of local disks are giving way to cloud-native object storage. So we discuss why modern databases are being rebuilt entirely on top of S3. From there, w...

Large Language Models can generate code in a flash, but that code is notoriously unreliable. Traditional unit tests often can’t put enough guardrails in place to ensure correctness… even if they’re written by the LLM itself. This is where property-based testing (PBT) becomes essential. Today, we're joined by David R. MacIver, creator of the PBT library Hypothesis, and now an Antithesis employe...

Software testing research is exploding, but in practice, most companies' testing approaches seem stuck in the past. Where does that gap come from? It often boils down to the distance between academic promises and the practical needs of developers who need usable tools and fast results. In this episode, David talks with Rohan Padhye, head of the PASTA research group at Carnegie Mellon Universit...

Integrating non-deterministic, non-durable elements like AI agents into our workflows tends to lead to a lot of do-overs. But restarting AI processes can be costly, burning through tokens and losing valuable progress. Wouldn’t it be easier if there was always a clear checkpoint to restart a task from? Today I talk with Qian Li, co-founder of the DBOS durable execution engine, about reliability, er...

How do you test for bugs that only appear when a user clicks frantically, or when asynchronous data loads in an unexpected order? Standard UI tests often miss the subtle stuff that happens all the time in the stateful, dynamic applications. In this episode, Paul Ryan and I sit down with Oskar Wickström, creator of the QuickStrom framework, among other things, to explore how to apply generative tes...

AI coding assistants promise incredible speed, but what happens when you run straight into a wall of buggy code and technical debt? In this episode, Clare Sudbery, a software engineer with over 25 years of experience, discusses a crucial paradox for modern developers.  The secret to harnessing AI's power isn't to move faster, but to slow down. Clare explains why deliberate, rigorous practi...

Some bugs are so rare, they can take years to track down and fix. What if you could find and fix five of them per day ?  For Joran Dirk Greef, the creator of the TigerBeetle database, that's not a wild dream — it's how his team works every day. While most people think building a new database takes a decade, Joran's team built TigerBeetle in just three and a half years. The key is a uni...

When is a bug not really a bug? In this episode, host David Wynn talks with SRE veteran Dan Slimmon about a radical idea: chasing perfect code might not be the best way to make your service reliable. Dan argues that once your code is "good enough," most outages aren't caused by code defects. They're caused by weird interactions between different parts of a system or by users doin...

Every map of a complex territory is inherently wrong, but without one, we're completely lost. So what happens when the territory is the vast, ever-changing landscape of distributed systems? In this episode, David Wynn sits down with Kyle Kingsbury, the renowned researcher behind Jepsen, to discuss a monumental effort to chart this landscape: the new Distributed Systems Reliability Glossary. Ky...

Is it just a fact of life that software is broken? Our industry often operates as if the answer is "yes." We write tests, we fix bugs, but we seem to accept a certain level of failure as the cost of doing business. Our guest today is tired of it. Isaac Van Doren is a software engineer at Paytient, a healthcare payment solutions provider,   and he’s "sick of software being broken all...

How do you prove the correctness of services that underpin a huge portion of the internet?  At the scale of Amazon Web Services, traditional testing falls short. In this episode, Antithesis’ own Will Wilson and Ben Collins talk with Marc Brooker, a Distinguished Engineer who has spent nearly 17 years building core AWS infrastructure like S3, Lambda, and Aurora Serverless. Marc gives us the inside...

The BugBash Podcast kicks off with a bit of pre-history -- the story of FoundationDB, one of the first companies to successfully use deterministic simulation testing to accelerate development, assure reliability, and build something legendary. Today, FoundationDB is the hidden layer in Snowflake, DeepSeek, and many core systems at places like Apple and Goldman Sachs. But it started with some guys...