Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Chaos Engineering Tests Your System Resilience
Transcript
- Lucas: So, Luna, you've probably heard the saying: 'It's not a question of if your system will fail, but when.' Most teams try to postpone that 'when' with monitoring, redundancy, and load testing. But there's a whole discipline built around the opposite approach — actively breaking your own system to see how it behaves. Luna: You mean like deliberately causing outages? That sounds terrifying — and kind of counterintuitive. Lucas: Exactly. It's called chaos engineering. And the idea is simple: instead of waiting for a real incident to expose a weakness, you inject failures on purpose, in a controlled way. That way, you learn about your system's blind spots before your customers do. Luna: I've heard of Netflix's Chaos Monkey — that's the tool that randomly kills instances in production, right? Lucas: That's the most famous example. Netflix started working on it around 2011 after a major AWS outage took down their streaming service. They realised that even though they had redundant systems, they didn't actually know if those redundancies worked in practice. So they built Chaos Monkey — a piece of software that randomly terminates virtual machines in their production environment. Luna: During normal business hours? In production, not staging? Lucas: Yes. That's the key. The whole point is to test the system under real traffic, with real data. But they didn't start with full-blown production chaos. They built in safeguards — a blast radius. You only let the monkey kill one instance per availability zone, for example. And if an experiment causes a real customer-facing issue, there's an automatic rollback mechanism. Chaos engineering is not about being reckless; it's about scientific hypothesis testing. Luna: So you formulate a hypothesis like: 'If one server goes down, the load balancer will route traffic to the remaining servers without any latency increase'? Lucas: Exactly. You define a steady state — say, response time under 200 milliseconds — then you introduce a failure and measure whether the system stays within that steady state. If it doesn't, you found a weakness. And you fix it before it becomes a real incident. Netflix calls this 'exercising your resilience muscles.' Luna: I imagine not every company can afford to run chaos experiments in production like Netflix does. What about smaller teams? Lucas: Great point. Netflix has a dedicated resilience engineering team. But there are tools that make chaos engineering accessible to smaller teams. The company Gremlin, for example, offers a platform that lets you run scheduled failure experiments with predefined blast radius controls. You can start in a staging environment, then graduate to production after you're confident. And cloud providers like AWS have built-in services like Fault Injection Simulator that let you test without writing custom tooling. Luna: So the barrier to entry is lower than people think. What kinds of failures do teams typically inject first? Lucas: The most common are: shutting down an instance, killing a process, adding latency to network calls, or making an external API return a 500 error. There's also 'black hole' testing where you block traffic to a specific dependency. The idea is to start small and simple. You don't go straight to knocking out a data centre. Luna: And how do you know if the experiment was successful? I mean, beyond 'the system didn't crash'. Lucas: You measure against your steady-state hypothesis. Did response times stay within your defined threshold? Did the error rate stay below, say, 0.1 percent? Did the automated failover actually happen within the expected time? If yes, you just validated that your resilience strategy works. If not, you've found a gap. Luna: I can see how this is different from traditional load testing. Load testing usually pushes the system to its limit to find performance bottlenecks. But chaos engineering is about uncovering unknown unknowns — like a service that silently depends on another service that nobody remembered to include in the dependency graph. Lucas: Right. Load testing is about capacity. Chaos engineering is about correctness under failure. And the two complement each other. A lot of teams now do both. The really mature organisations run chaos experiments continuously, integrated into their CI/CD pipeline. Every time they deploy a new microservice, they run a small chaos test against it to make sure the circuit breakers and retries work. Luna: That sounds like a lot of upfront investment in tooling and culture. How do you convince management to sign off on something that might break things intentionally? Lucas: You frame it as insurance. The cost of a major outage — in lost revenue, customer trust, and engineering time — is almost always higher than the cost of running controlled experiments. And you don't have to start with production. Start with a staging environment that mirrors production. Show that you can find and fix issues that would have caused real downtime. Once you have a few wins, it's easier to get buy-in for production experiments. Luna: So it's a bit like the concept of 'shifting left' but for reliability — catching failure scenarios early, before they hit users. Lucas: Exactly. And it also fosters a culture of blameless postmortems. When an experiment reveals a weakness, the goal is to fix the system, not to blame a person. That mindset is crucial for any team that wants to be truly resilient. Luna: You know, the whole idea of investing a little effort now to prevent a much bigger problem later — that's something we believe in around here too. Lucas: Yeah, it's funny you mention that. Because this podcast is ad-free and listener-supported, and the same principle applies. A small contribution from listeners who get value out of shows like this helps us keep producing content that digs into topics like chaos engineering. If you've found today's episode useful, you can support us at buy me a coffee dot com slash fexingo. Luna: And it really does make a difference. Every bit helps us stay independent and focused on the topics that matter to you. Lucas: Alright, back to chaos. One thing I want to emphasize is that chaos engineering isn't a one-and-done activity. Systems evolve. New dependencies get added. Configurations change. Even if you pass a chaos experiment today, the same weakness could reappear three months later after a deployment. So the experiments need to be part of a continuous practice, not a quarterly exercise. Luna: So you treat resilience as a recurring test suite, like regression tests. Lucas: Exactly. And you can automate the whole process. Tools like Chaos Toolkit let you define experiments as code and run them on a schedule. You can even set them up to run automatically after every deployment, as part of your CI/CD pipeline. The experiment either passes — meaning your system handled the failure gracefully — or it fails, and you get an alert. Luna: What's the most surprising thing you've seen come out of a chaos experiment? Lucas: There's a classic story from Netflix. They ran an experiment where they killed an entire AWS availability zone — multiple servers at once. The system was supposed to automatically shift traffic to the remaining zones. But what actually happened was that the load balancers routed traffic to a different service that then crashed because it hadn't been tested under that load pattern. They found a cascading failure that no one had predicted. That one experiment probably saved them from a real multi-hour outage down the line. Luna: And that's the whole point — discovering the failure modes you didn't know existed. Lucas: Right. So if you're a software team and you haven't tried chaos engineering yet, I'd suggest picking one simple failure scenario — like killing one instance in a staging environment — and just observing what happens. You might be surprised at what you learn. Luna: Good advice. I know what I'm bringing up in our next sprint planning. Lucas: And if you do run that experiment, let us know what you find. We might do a follow-up episode on your results.