Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Chaos Engineering Experiments Without Breaking Production
Transcript
- Lucas: So Luna, I was reading about a payments company that ran a chaos experiment on their checkout service, and the result surprised even their senior engineers. They simulated a thirty-second database outage during a low-traffic window, and what they found was that their retry logic — which they thought was solid — actually amplified the load once the database came back. Luna: Wait, so the retries made things worse after the outage? That sounds backwards. Lucas: Exactly. The retry queue had no backoff, so when the database recovered, all those queued requests hit it at once. It took the database another twenty minutes to stabilize. And the team only found this because they ran a controlled experiment, not because a real outage hit them. Luna: That's a perfect example of why chaos engineering isn't just about breaking things for fun. Lucas: Right. And that's the angle I want to dig into today. Chaos engineering gets this reputation as this wild, Netflix-style practice where you randomly kill servers in production. But the actual discipline is a lot more structured. It's about forming a hypothesis, running a small experiment, and learning something specific about your system's behavior. Luna: So it's like a scientific method for your infrastructure. But how do you start without, you know, breaking everything? Lucas: Well, the first principle is to define a steady state. That's your baseline — what does normal look like? For the payments company, normal meant that 99.5 percent of checkout requests completed within two seconds. That's their steady state hypothesis. The experiment is then designed to see if the system can return to that state within a certain time after a failure is injected. Luna: Okay, so you're measuring against a clear metric. But what about the blast radius? I've heard that term thrown around a lot. Lucas: Blast radius is basically how many users or services are affected if the experiment goes wrong. The key is to start small. Instead of taking down a whole database cluster, you target a single instance or a single user segment. You can even use feature flags to limit the experiment to internal testers or a specific region. Luna: That makes sense. And you'd want to have an automatic rollback mechanism in place before you even start, right? Lucas: Absolutely. The rollback is your safety net. If metrics deviate from the steady state beyond a threshold, the experiment should self-terminate and restore normal operations. For example, you might set a rule that if error rates exceed one percent for more than thirty seconds, the chaos tool automatically stops the injection and re-enables all traffic. Luna: So it's not just about the experiment itself — it's about having the safety infrastructure around it. Lucas: Precisely. And that's what separates a gameday from a real incident. A gameday is a planned exercise where you simulate a failure, but chaos engineering is more continuous. You're constantly probing your system with small, automated experiments. Luna: I've seen some teams use tools like Chaos Monkey or Gremlin. What's your take on those? Lucas: Those are great for getting started. Chaos Monkey was Netflix's original tool — it randomly terminates instances in production. But modern platforms like Gremlin let you inject specific faults like CPU spikes, network latency, or even a full region outage. The important thing is not the tool itself, but how you design your experiments. You still need a hypothesis and a way to measure the result. Luna: And for smaller teams that don't have Netflix's resources, what's a realistic starting point? Lucas: Start with your most critical service — the one that, if it goes down, you lose money or reputation. For a lot of companies, that's authentication or payments. Then pick a failure mode that's common, like a database connection timeout or a slow DNS response. Run the experiment during off-peak hours, and make sure you have monitoring in place to see what happens. Luna: That sounds manageable. But how do you convince your manager that it's worth the risk? Lucas: That's the perception problem. Chaos engineering sounds scary, but the reality is that you're trading a small, controlled risk for a much larger risk of an unpredictable outage. The way to pitch it is to ask: what's the cost of a thirty-minute outage during Black Friday? If that's millions of dollars, then spending a few hours a month on controlled experiments is a no-brainer. Luna: And you can always start in staging, right? You don't have to go straight to production. Lucas: Sure, staging is a great place to practice. But the real value comes from testing in production — not because you want to cause harm, but because that's where the real traffic patterns, the real data, and the real network conditions live. Staging often has sanitized data and predictable traffic, so you won't catch those subtle issues like the retry storm we saw earlier. Luna: Fair point. And I guess that's why the steady state metric is so important — you need to know what 'normal' looks like in production before you start poking it. Lucas: Exactly. And that's actually where a lot of teams struggle. They don't have good baseline metrics. So before you run any chaos experiment, make sure you have solid monitoring and alerting. If you can't see your error rate or latency in real time, you're flying blind. Luna: So what's a good first experiment for a team that's never done this? Something that's low-risk but still teaches you something. Lucas: A classic is to kill one instance of a load-balanced service. That's a failure that happens all the time in cloud environments — a VM gets recycled, a container restarts. So you pick one instance out of ten, you terminate it, and you watch to see if your traffic seamlessly shifts to the remaining instances. The steady state metric would be something like '99 percent of requests still succeed within the same latency.' Luna: That sounds like a great starting point because the blast radius is tiny — you still have nine other instances handling traffic. Lucas: Right. And you learn whether your load balancer health checks are actually working, whether your connection pools are sized correctly, and whether your retry logic is sane. It's a low-stakes way to expose a whole class of issues. Luna: One thing that comes to mind is the cultural aspect. How do you get developers on board with chaos engineering? It's easy to feel like you're being blamed for code that isn't 'robust enough.' Lucas: That's a real concern. The key is to frame it as a learning exercise, not a blame game. When an experiment surfaces a weakness, you celebrate the fact that you found it in a controlled environment instead of in an incident. Netflix actually has a culture of 'chaos engineering' where they encourage teams to share their findings openly, without fear of repercussions. Luna: So it's about building a blameless culture. And I guess you also want to make sure you have an on-call person during the experiment, right? Lucas: Absolutely. You should always have someone who is ready to respond if the experiment goes sideways, and they should have the authority to abort it at any moment. That's part of the safety net. You also want to run experiments during business hours sometimes, because that's when you have the full team available. But you balance that with off-peak windows to minimize user impact. Luna: And you mentioned feature flags earlier. How do they fit into chaos experiments? Lucas: Feature flags are a great way to scope an experiment to a subset of users. You can turn on a flag that causes, say, a two-second artificial delay for five percent of your traffic. That way you're testing resilience without affecting everyone. And if things go wrong, you just flip the flag off — that's your instant rollback. Luna: That's a nice pattern. So it's like a canary deployment, but for failures instead of new features. Lucas: Exactly. And the more you practice, the more you build a muscle memory for how your system behaves under stress. When a real incident happens, you're not panicking — you've already seen this movie before, and you know what to do. Luna: It's like a fire drill for your infrastructure. Lucas: That's a perfect analogy. And actually, speaking of drills, we should mention that chaos engineering isn't just about technical failures. You can also simulate things like a malicious attack or a sudden spike in traffic. The same principles apply — steady state, small blast radius, and a rollback plan. Luna: So it's a general-purpose resilience tool. And I love that it's so data-driven. You're not guessing — you're measuring. Lucas: Right. And that's what makes it so valuable. You're turning assumptions into facts. You might think your system is resilient, but until you test it, you don't really know. Luna: Hey, Lucas, you know what? This conversation has been really useful. I mean, I've always been curious about chaos engineering, but I never knew where to start. Now I feel like I could actually pitch a small experiment to my team. Lucas: That's great to hear. And if this episode gave you something you can use, that's exactly what we hope for. You know, we keep this show ad-free, and it's because of listeners like you. If you ever feel like supporting us, the smallest gesture goes a long way. It's just buy me a coffee dot com slash fexingo. No pressure, seriously — only if today was worth it to you. Luna: Yeah, even a coffee helps us keep the servers running and the mics on. Lucas: And we're back to the nuts and bolts. So where were we? We were talking about the importance of having good observability before you run your first chaos experiment. Let's get into a concrete example of how to set up that baseline. Luna: Please. I want to know what metrics I should be tracking in the weeks before I even touch a production server. Lucas: For a typical web service, you'd want to track request latency, error rate, and throughput. But you also want to look at system-level metrics like CPU, memory, and disk I/O. The trick is to capture a few weeks of data during different times of day and week, so you know what 'normal' looks like even during peak traffic. Luna: And you'd want to see that your alerting actually fires on those metrics, right? Otherwise you won't know if something's wrong during the experiment. Lucas: Exactly. Before you run any chaos experiment, make sure your alerts are working. You can test that by injecting a minor fault and checking that your monitoring picks it up. It's like a test for your tests. Luna: That's a good point. And after the experiment, what do you do with the findings? Lucas: You write a postmortem, even if it's a short one. What was the hypothesis? What did you observe? What did you learn? And most importantly, what's the action item? Maybe you need to add a retry backoff, or increase your connection pool size, or add a circuit breaker. The experiment should lead to a concrete change. Luna: So it's a continuous improvement loop. Run experiment, learn, fix, repeat. Lucas: Exactly. And over time, you build a suite of experiments that cover your most critical failure modes, and you run them regularly — maybe weekly or monthly. The goal is to get to a point where your system is so resilient that a single instance failure is a non-event. Luna: That's a great goal to aim for. For anyone listening who's convinced, what's one small step they can take this week? Lucas: Start with a single service, define one steady state metric, and run a simple experiment in staging. Then once you're comfortable, move to production with a feature flag and a tiny blast radius. That's it. You don't need to build a whole chaos engineering platform on day one. Luna: And if it goes well, you'll be surprised at how much confidence it gives you in your system. I know I will be. Lucas: And that confidence is the real payoff. Chaos engineering isn't about being reckless — it's about being prepared. So next time you look at that monitoring dashboard, ask yourself: what would happen if that database just went away? And then go find out, in a safe way.