Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Chaos Engineering Strengthens Your Software Testing
Transcript
- Lucas: So you've got a test suite that passes, your coverage numbers look good, your CI pipeline is green — and then your service goes down under a sudden traffic spike because a database connection pool exhausted and nobody tested for that failure mode. Luna: That's the classic gap between 'tests pass' and 'system works under real conditions.' And it's exactly where chaos engineering comes in. Lucas: Exactly. Chaos engineering is the practice of intentionally injecting failures into a system — in a controlled way — to see how it behaves and where it breaks. It's not about causing random havoc; it's about building confidence that your system can withstand unexpected disruptions. Luna: And it started pretty famously at Netflix with Chaos Monkey, which randomly kills instances in production. Lucas: Right. Back in 2011, Netflix built Chaos Monkey to randomly terminate EC2 instances in their AWS environment. The idea was simple: if your system can survive a server disappearing without notice during business hours, you've got real resilience. And that philosophy evolved into a whole discipline. Luna: But most teams aren't Netflix. How does a smaller team approach this without bringing down their entire application? Lucas: That's the key — you start small and controlled. The core practice is to run experiments in a staging or production-like environment, or even in production but with careful safeguards. You define a 'steady state' of normal behavior — say, your API responds within 200 milliseconds with a 99.5 percent success rate. Then you introduce a failure, like killing one server instance, and measure whether the system still meets that steady state. Luna: So it's really hypothesis-driven testing. 'I believe that if one database node goes down, the system will still serve requests within two seconds.' Then you test it. Lucas: Exactly. And that hypothesis is your steady-state baseline. The experiment either validates it — meaning your resilience mechanisms worked — or it invalidates it, and you've found a real weakness to fix. Luna: One thing I've seen teams struggle with is the blast radius. How do you make sure the failure doesn't cascade into a real outage? Lucas: Blast radius control is critical. You always start with the smallest possible failure. Instead of killing all database replicas, you kill one. Instead of injecting latency into every request, you inject it into one percent of traffic. And you run experiments during low-traffic periods initially. Tools like Gremlin let you set a blast radius explicitly — you can target specific hosts, regions, or even individual users. Luna: And you always have a rollback plan — a way to stop the experiment and restore normal operation immediately. Lucas: Right. Most chaos engineering platforms have a 'Halt' button that kills all active experiments. You should also have manual runbooks. But the goal isn't to avoid all failures — it's to discover them safely. Luna: Let's talk about a concrete example. Say you're running an e-commerce site, and you want to test what happens if the payment gateway becomes slow. Lucas: Great scenario. You'd set your steady state: checkout completes in under three seconds, with no errors. Then you inject a one-second delay into calls to the payment API, for, say, five percent of requests. You monitor checkout times, error rates, and also whether the system starts queuing orders or timing out. If checkout times stay under three seconds for the affected requests, great. If not, you've uncovered a dependency that isn't properly timeout-handled. Luna: And that kind of test is exactly what catches hidden coupling between services — like when a downstream latency spike ties up connection pools upstream and causes a cascading failure. Lucas: Precisely. That cascading failure is one of the most dangerous patterns in distributed systems. Chaos experiments are designed to reveal it before it happens in production. Luna: Now, if today's conversation gave you something useful — maybe a new way to think about testing resilience — I want to mention something briefly. This show stays ad-free because of listener support. There's no sponsor, no ads, just people who find value in it. So if you'd like to help keep it that way, you can drop a few bucks at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a simple way to support the kind of deep-dive conversations we're having here. And it genuinely makes a difference — keeps us independent and focused on what's useful. So if you've gotten value, consider it. Luna: Alright, back to chaos. So what's a common mistake teams make when they first start chaos engineering? Lucas: The biggest one is jumping straight to complex scenarios without first building basic observability. If you don't have good monitoring — metrics, logs, traces — you can't measure the steady state, and you certainly can't tell what broke during the experiment. So step one is making sure you can actually see what's happening. Luna: Right, because the whole point is to learn from the experiment. Without observability, it's just chaos. Lucas: Another mistake is running experiments only in staging. Staging environments are often smaller, have different traffic patterns, and don't have the same resilience mechanisms as production. The result is false confidence — your staging test passes, but production still breaks. Luna: So you need to run chaos experiments in production eventually, but with safety nets. Lucas: Yes. And that's where the discipline gets more rigorous. You use techniques like canary analysis — start with a tiny percentage of users or traffic, monitor the effect, and only expand if the system remains healthy. You also schedule experiments during off-peak hours and always have human operators watching the dashboards. Luna: What about tooling? Is it all custom, or are there off-the-shelf solutions? Lucas: There's a healthy ecosystem now. Gremlin is probably the most well-known commercial platform — it supports CPU, memory, disk, network, and even stateful failure injection like killing processes. Chaos Toolkit is an open-source alternative that lets you define experiments as code. And for Kubernetes specifically, there's LitmusChaos and Chaos Mesh, which integrate directly with your cluster. Luna: And Netflix open-sourced their Chaos Monkey as part of the Simian Army, right? Lucas: They did. The Simian Army includes Chaos Monkey, but also Latency Monkey, Conformity Monkey, and others. Those are more for testing cloud infrastructure resilience. They're still widely used as a reference implementation. Luna: So let's say a team wants to start. What's the first experiment they should run? Lucas: I'd recommend starting with a simple network latency experiment on a non-critical service. For example, add 100 milliseconds of latency to calls from your user-service to your notification-service, in your staging environment. Measure the impact on user-service response times. If it stays within acceptable bounds, great. If not, you've found a timeout or retry issue you can fix before it hits production. Luna: That's a low-risk way to build the muscle. Then you can graduate to killing a pod or instance. Lucas: Exactly. And you document every experiment: what was the hypothesis, what was the result, what did you learn. Over time, you build a library of known failure modes and how your system handles them. That becomes a powerful knowledge base for on-call engineers. Luna: It also helps with incident response training. If you've already seen what happens when a database goes down during a GameDay, you're calmer when it happens for real. Lucas: That's a great point. Chaos engineering doubles as a training tool. Teams that run regular experiments develop muscle memory for triage and recovery. They know which dashboards to look at first, which runbooks to follow, and they're less likely to panic. Luna: So chaos engineering isn't really about breaking things — it's about learning how your system behaves under stress, in a safe way. Lucas: Exactly. It's a proactive testing discipline that shifts finding failures from 'when an incident happens' to 'when you choose to experiment.' And every experiment that reveals a weakness is a chance to make your system more robust before users ever notice. Luna: Thanks, Lucas. That's a solid framework for anyone thinking about getting started. Lucas: My pleasure. And remember — start small, observe everything, and always have a halt button.