Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Uses Chaos Engineering to Test Production Resilience
Transcript
- Lucas: So there is this idea in software engineering that if you aren't testing your production system under stress, you don't actually know if it will hold up when it matters. And one approach that's been gaining real traction, especially in the last couple of years, is chaos engineering. Luna: And just to ground us — chaos engineering isn't about randomly breaking things, right? It's about running controlled experiments to uncover weaknesses before they cause an outage. Lucas: Exactly. And what's interesting is that a lot of engineering leaders still think of it as something only Netflix or Amazon do. But I recently spoke with a CTO at a mid-sized fintech company — about 300 engineers — who has been running chaos engineering in production for the last 18 months. And their story is pretty instructive. Luna: What's the specific thing they found? I'm guessing it wasn't something a normal integration test would catch. Lucas: Right. So in their case, they decided to inject a small amount of latency into their payment processing service — just an extra 200 milliseconds on about 5 percent of requests. And within an hour, they saw a cascading failure start to form. The payment service had a timeout of two seconds, and downstream services that depended on it started queuing requests. One of their queue services hit its limit, which caused a different service to start dropping transactions. It was exactly the kind of failure path that no unit test or staging environment would ever expose. Luna: That's the kind of thing that keeps me up at night. And it's exactly why you want to find it during a Tuesday afternoon experiment rather than on Black Friday. Lucas: Yeah. And I think a lot of teams hesitate because they're scared of causing real harm. But the CTO I spoke with had a very structured approach. They started with what they called 'game days' — scheduled, pre-approved experiments with a very limited blast radius. They'd pick a single service, define a hypothesis like 'if we add 100 milliseconds of latency to the auth service, the login flow should still complete within the SLO.' And then they'd run the experiment for 15 minutes with automatic rollback if any error rate exceeded a threshold. Luna: That makes sense. And I imagine they had to get buy-in from the rest of the leadership team before they could even start. Lucas: Absolutely. That was one of the biggest lessons — this is as much a cultural change as a technical one. The CTO spent three months just socializing the idea, running tabletop exercises, showing how the experiments would have a kill switch and wouldn't affect customer-facing metrics. And then they started with one team that was already pretty mature in their observability practices. Luna: And that first team — were they the ones who found the payment latency cascade? Lucas: Actually no, that came later. The first team ran experiments on a caching layer that had no customer impact. They basically proved the process was safe. Then other teams started volunteering. The payment service experiment was about six months in, after they had built some internal tooling to automate the injection and monitoring. Luna: That pacing is smart. You don't want to go from zero to 'let's break the payment system' on day one. Lucas: Right. And this is actually where I think a lot of the conversation around chaos engineering misses the mark. People hear the word 'chaos' and think it's about random destruction. But the whole point is to be more scientific about reliability. You're forming a hypothesis about how your system should behave under specific conditions, and then you're testing that hypothesis in a controlled way. Luna: So it's really a form of experimental science applied to distributed systems. And if you do it well, you end up with a much deeper understanding of your actual dependencies and failure modes. Lucas: Exactly. And that understanding translates directly into better incident response. The CTO I spoke with said that after a year of running these experiments, their team's mean time to resolution for production incidents dropped by about 40 percent. Because they had already seen the failure patterns in a controlled setting, so when something similar happened in the wild, they knew where to look. Luna: That's a huge improvement. And it makes intuitive sense — you're effectively building a mental model of your system's weak points through repeated exposure. Lucas: Yeah. And I think that's the core insight for any engineering leader listening today: you don't need to be Netflix. You can start very small. Pick one service that has good monitoring, define a single hypothesis, run a 15-minute experiment with a rollback plan. And if you do that once a month, within a quarter you'll have a much better sense of where your system actually breaks. Luna: And if today's conversation gave you something you want to try with your team, that's exactly the kind of thing that keeps these shows going. A couple of dollars a month at buy me a coffee dot com slash fexingo genuinely makes a difference — keeps them ad-free and focused on practical content like this. Lucas: Yeah, it really does. So back to the fintech CTO — one of the other things they did that I thought was clever was they built a 'chaos dashboard' that showed the blast radius of every experiment in real time. Any engineer in the company could see exactly which services were affected and whether the experiment was within the safety limits. Luna: That transparency probably helped with buy-in too. People could see that it wasn't just some secretive SRE project. Lucas: Exactly. And it also served as a forcing function for good observability. You can't run these experiments if you don't have solid metrics and logging in place. So it actually pushed teams to improve their monitoring across the board. Luna: So chaos engineering becomes a catalyst for better operational maturity. That's a double win. Lucas: Yeah. And the CTO told me that after about a year, the culture shifted. Engineers started proactively suggesting experiments for their own services. They'd come to the weekly reliability review and say 'I think our payment reconciliation service is brittle under high load — can we design an experiment to test that?' Luna: That's the sign of a healthy engineering organization. When reliability becomes everyone's job, not just the platform team's. Lucas: Right. So the takeaway for me is that chaos engineering is accessible if you treat it as a gradual practice, not an all-or-nothing initiative. Start with a small, safe experiment, learn from it, and scale from there. Luna: And the fintech CTO's example shows that even a modest program can uncover truly critical failure modes. That payment latency issue would have eventually caused a major outage. Lucas: Exactly. And they fixed it before any customer ever noticed. That's the whole point.