Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Chaos Engineering Strengthens Your Test Suite
Transcript
- Lucas: So you've got a thousand passing unit tests, your integration suite is green, your coverage numbers look great — and then your service goes down at 3 PM on a Tuesday because a downstream database decided to take a nap. The tests didn't catch it because they never ran against a database that was actually failing. Luna: And that's where chaos engineering comes in. It's the practice of deliberately breaking things in a controlled way to see what happens. Lucas: Exactly. Netflix essentially invented the modern version of this around 2011 with something called Chaos Monkey — a tool that randomly terminates instances in their production environment. The idea was: if you're constantly killing servers during business hours, your engineers are forced to build systems that survive instance failure without anyone noticing. Luna: Right, because if you only test against happy paths — the database is up, the network is fast, the CPU is cool — you're training your team to assume those conditions hold. And they don't. Lucas: And that's the core insight: your test suite, no matter how thorough, is ultimately running against a synthetic environment. Chaos engineering runs experiments directly in production — but with guardrails. You don't just pull the plug on everything. You define a hypothesis — 'the system should still serve traffic if one of three database replicas goes down' — and then you run a small experiment to validate that. Luna: And if the experiment fails, you've found a gap that no amount of unit testing would ever surface. But you've also learned it in a way that's safe — because you controlled the blast radius. Lucas: You know, it's interesting — we deliberately keep this show ad-free because we believe conversations about software quality should be driven by what's actually useful, not by sponsorship pressure. If you find value in episodes like this one, and you want to support that choice, the link is buy me a coffee dot com slash fexingo. But back to chaos engineering — the key word there is 'controlled'. Luna: Yeah. The goal isn't to cause an outage. It's to build confidence that your system can handle the unexpected. Lucas: So let's talk about how you actually run one of these experiments. Because a lot of teams hear 'chaos engineering' and imagine somebody randomly unplugging servers with zero warning. That's not the practice anymore. Luna: No, it's evolved. There's a maturity model. You start small. Lucas: Right. The first step is to identify your steady state — a set of metrics that define normal behavior. For a web service, that might be p99 latency under 200 milliseconds, error rate below 0.1 percent, and throughput within a certain range. You pick one hypothesis — say, 'if we kill one instance in our load-balanced pool, p99 latency stays under 200 ms'. Luna: Then you inject the failure. You terminate that one instance — preferably during low-traffic hours first — and watch the metrics. If they stay within bounds, the experiment passes. If not, you've got a bug on your hands. Lucas: And you do this in a way that minimizes risk. You set a blast radius — limiting the experiment to one availability zone, or to a subset of users, or to a staging environment that mirrors production. Some teams start with 'game days' where they simulate a scenario in a non-production environment before touching production at all. Luna: But there's also value in running experiments in production — because that's where the real network conditions, the real traffic patterns, the real dependencies live. Staging environments are never a perfect copy. Lucas: That's exactly why Netflix does it in production. And they have a whole suite of tools now — Chaos Monkey is just one. There's also Chaos Kong, which simulates an entire AWS Region going down. They run these experiments during business hours, so the engineers are at their desks to respond. Luna: That's a cultural shift too. It requires a team that's not afraid of failures — because failures are treated as learning opportunities, not blame events. Lucas: And that's probably the hardest part. You can't just install a tool and start killing instances. You need executive buy-in, you need a blameless postmortem culture, and you need monitoring that's good enough to catch regressions quickly. Luna: So what does a practical first experiment look like for a team that's never done chaos engineering? Lucas: I'd recommend starting with latency injection. Because latency is one of the most common failure modes in distributed systems — a service doesn't crash, it just slows down. You can use a tool like AWS Fault Injection Simulator or the open-source Chaos Toolkit to add, say, 200 milliseconds of delay to requests going to a specific dependency. Luna: And then you measure whether your timeouts, your retry logic, your circuit breakers actually fire as expected. Lucas: Exactly. A classic finding from these experiments is that retry storms actually cause more harm than the original failure. Your service calls a downstream that's slow, your retry logic kicks in, now you've got three times the requests piling onto an already struggling dependency. Chaos engineering catches that because you see the emergent behavior. Luna: Right. Unit tests would never catch a retry storm because each individual unit is tested in isolation. The storm only happens when the system runs as a whole. Lucas: And that's the argument for adding chaos engineering to your QA strategy. Not replacing your existing tests — they catch logic bugs and regressions just fine. But chaos engineering catches what I call 'system-level fragility' — the things that only show up when real traffic meets real failures. Luna: So when should a team consider starting? At what scale? Lucas: I'd say once you have more than a handful of microservices and your uptime matters — you're serving real users, you have SLAs — you're ready. You don't need Netflix scale. You need a hypothesis, a monitoring setup, and a small blast radius. Luna: And a willingness to learn. Because the first experiment might fail. That's the point. Lucas: If every experiment passes, you're not being aggressive enough. The goal is to find weaknesses before your customers do. Luna: One thing I've seen in practice: teams that run chaos experiments regularly start to design their systems differently. They instinctively build in redundancy, timeouts, graceful degradation — because they know next month's experiment will test that specific failure mode. Lucas: That's the long-term payoff. It shifts the culture from 'we hope it works' to 'we know it works because we broke it on purpose and it survived'. Luna: So if someone listening wants to try this tomorrow, what's the one concrete step? Lucas: Pick one dependency that your system cannot live without — a database, a cache, an auth service. Write down the steady-state metric you expect to hold. Then — in a staging environment first — inject a 300-millisecond delay into calls to that dependency and watch what happens. If your p99 latency stays under your threshold, you're in good shape. If it blows up, you just found your first chaos-driven improvement. Luna: And that's a much more powerful test than any assertion you could write in a unit test. Lucas: Because it's the system that's doing the testing. You're just creating the conditions. The system tells you whether it's resilient or not. Luna: Alright, so next time someone says their test suite is comprehensive, maybe ask: 'Have you tried killing a server yet?' Lucas: That's exactly the question. Thanks for listening — we'll be back next time with another angle on building software that actually works.