Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Chaos Engineering Finds Weaknesses Tests Miss
Transcript
- Lucas: So earlier this year, a major UK bank ran a chaos engineering experiment on their payment processing system. They deliberately killed one of their primary database nodes during a low-traffic window. The system was supposed to fail over to a replica within seconds. Instead, it took over twelve minutes — and during that window, thousands of transactions were queued with no visibility. Luna: Twelve minutes for a database failover that should take seconds? That sounds like a disaster waiting to happen. Did they find out why? Lucas: Yeah, it turned out the replica had a different configuration for connection pooling. It wasn't configured to accept the same number of simultaneous connections as the primary. So when the failover happened, clients couldn't connect fast enough, and the system essentially choked. No unit test, integration test, or staging environment would have caught that — because those environments used identically configured replicas. Luna: That's the whole point of chaos engineering, right? You test in production with controlled experiments to find these configuration drifts and unexpected behaviors. Lucas: Exactly. Chaos engineering isn't about breaking things randomly. It's a disciplined practice — you start with a hypothesis about your system's steady state. You define a metric like '99th percentile payment latency under 500 milliseconds'. Then you inject a controlled fault — kill a node, spike CPU, saturate network — and measure whether the steady state holds. Luna: And if it doesn't hold, you've found a genuine weakness. Not a theoretical one from a test case, but something that would have caused real problems in production. Lucas: Right. The key difference from traditional testing is the environment. Traditional QA runs in staging or pre-production, which is often a sanitized replica. Chaos engineering happens in production — but with safeguards. You define a 'blast radius' so the experiment can't cascade. You have a rollback plan. And you always run during low traffic first. Luna: That brings up a question: how do you start with chaos engineering without putting your users at risk? Most teams are terrified of touching production. Lucas: Fair point. The industry standard is to start small — really small. Netflix, which pioneered chaos engineering with their Chaos Monkey tool, began by randomly terminating instances in non-critical services during business hours. They learned that their auto-scaling groups handled it fine. Then they graduated to more impactful experiments. Luna: So they built confidence gradually. I've also heard of companies using 'game days' — scheduled events where teams practice responding to failures. That's a good low-risk first step. Lucas: Exactly. Game days are essentially chaos engineering with human involvement. You simulate a failure, like turning off a database or dropping network packets, and then the on-call team practices their incident response. The goal isn't just to find system bugs — it's also to train people. Because ultimately, resilience is a combination of robust code and competent humans. Luna: I want to go back to the configuration drift problem. The bank found that their staging environment had perfectly matched replicas — so no test caught the mismatch. Chaos engineering in production revealed that. How common is that kind of drift? Lucas: Incredibly common. A 2024 survey by Gremlin found that over 60% of companies that adopted chaos engineering discovered at least one configuration gap within the first three experiments. Common examples: load balancers pointing to the wrong backend, caching layers with different eviction policies, or firewall rules that block failover traffic. Luna: And those are exactly the kind of things that automated tests rarely cover. Tests are great for logic bugs, but they're terrible at catching infrastructure-level discrepancies. Lucas: Right. That's why chaos engineering complements automated testing rather than replacing it. Automated tests give you confidence that your code works as intended. Chaos engineering gives you confidence that your system works under real-world conditions — including when things go wrong. Luna: What about the organizational challenges? I imagine it's hard to get buy-in from management to intentionally break things. Lucas: It is. The most common objection is 'we can't risk an outage'. The counter-argument is: you already have outages — you just don't know about them yet. Chaos engineering helps you find them in a controlled way rather than waiting for a real disaster. The bank example: if they hadn't found that failover bug in a controlled experiment, it would have hit them during a real failure, causing a two-hour outage and possibly millions in losses. Luna: So it's a risk management trade-off. A small, controlled disruption today prevents a large, uncontrolled one tomorrow. Lucas: That's exactly the pitch. And it's not just about prevention — it's also about learning. Every chaos experiment generates documentation about how the system behaves under stress. That knowledge is invaluable for incident response teams. Luna: What tools are used for chaos engineering? I know Netflix's Chaos Monkey is open-source, but there are also commercial platforms. Lucas: Yeah, the ecosystem has matured. Open-source tools like Chaos Monkey, Litmus, and Chaos Toolkit are popular. On the commercial side, Gremlin and ChaosSearch offer managed platforms with built-in experiments and safety controls. Most of them integrate with Kubernetes, AWS, Azure, and GCP. The key is to pick something that lets you define experiments as code — so they're repeatable and version-controlled. Luna: And you mentioned safety controls — what does that look like in practice? Lucas: Common safety controls include: automatic experiment halt if certain metrics exceed thresholds, time limits on how long a fault can run, and 'blast radius' limits that restrict the experiment to a subset of instances or a specific region. For example, you might only kill one pod in a ten-pod deployment, or only drop traffic for 1% of users. Luna: That sounds like a solid approach. Start with a small blast radius, monitor closely, and expand gradually. I can see how this becomes part of a regular release cycle — maybe even automated. Lucas: Some teams are doing exactly that. They run a set of chaos experiments as part of their CI/CD pipeline on staging environments — not production — but still testing resilience. The real gold is production, but that requires more maturity. Luna: Before we wrap up, I want to go back to something you said earlier about training people. That's a big aspect often overlooked. Even if the system handles a failure gracefully, if the on-call engineer doesn't know how to diagnose and respond, you're still in trouble. Lucas: Absolutely. That's why game days and chaos engineering go hand in hand. In fact, one of the most valuable outcomes of chaos engineering is the improvement in incident response documentation. Teams often realize their runbooks are outdated or incomplete. Lucas: And on that note — if you're getting value from these episodes, consider supporting the show. A couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Luna: Yeah, it really does make a difference. Helps us keep the content ad-free and focused on what matters. Lucas: So to circle back: chaos engineering isn't about being reckless. It's a disciplined practice to uncover hidden weaknesses. Start with a hypothesis, inject a controlled fault, measure the steady state, and learn from the outcome. The bank that found the failover bug? They fixed it in a week and now run chaos experiments every month. Luna: And that's the kind of proactive reliability that sets great teams apart. Thanks, Lucas. Lucas: Thanks, Luna. See you next time.