Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Uses Chaos Engineering to Build Resilient Systems
Transcript
- Lucas: There's a moment that every CTO I know dreads. It's 2 AM, the pager goes off, and you realise your database cluster just silently failed over — except the application didn't notice for ninety seconds. By the time it did, your checkout queue was backed up by ten thousand orders. Luna: That's the kind of failure that makes you want to test everything before it happens. Which is exactly what chaos engineering promises. Lucas: Right. But most teams think chaos engineering means randomly breaking production servers — Netflix-style — and they panic. Today I want to talk about a much more targeted approach. I've been following how the CTO of a mid-sized e-commerce platform — let's call it ShopGrid — uses controlled fault injection to find single points of failure. Luna: ShopGrid. I remember them. They do about two hundred million in revenue, maybe three hundred engineers. Not a Netflix-scale org by any stretch. Lucas: Exactly. And that's the point. Chaos engineering isn't just for streaming giants. This CTO — her name is Priya — started with one experiment. She picked their checkout service, which depends on a three-node Cassandra cluster. And she scheduled a fifteen-minute window at 3 AM on a Tuesday. Luna: Low traffic, controlled blast radius. Smart. What did she do? Lucas: She shut down one Cassandra node. That's it. The cluster should handle that — it's a three-node ring with replication factor three. Any single node can go down and the ring rebalances. But what she found was that the application's connection pool was configured with a single retry and a ten-second timeout. When the first node went down, the pool tried that node, waited ten seconds, then retried once — another ten seconds. Total: twenty seconds of failed checkout attempts before the pool realised it needed to talk to the remaining nodes. Luna: Oof. Twenty seconds of checkout failures at 3 AM is a blip. Twenty seconds during Black Friday is a disaster. Lucas: Precisely. So the first experiment uncovered a configuration bug that would have cost them millions. And here's the key — Priya didn't just fix the timeout. She changed the connection pool to use a circuit breaker pattern, and she added a health-check endpoint that the pool pings every five seconds. Luna: So one experiment, one concrete fix. That's the model, right? You form a hypothesis — 'our system can tolerate a single node failure' — then you test it. Lucas: That's the core of chaos engineering. You define a steady-state hypothesis, you inject a fault, you measure whether the system deviates from that steady state. If it does, you've found a bug in your resilience. If it doesn't, you've validated your assumption. Luna: But I've heard engineers push back hard on this. 'Our system is too fragile for chaos. We'll break everything.' How do you get past that? Lucas: That's the number one objection. And Priya's approach is instructive. She didn't start in production. She ran the experiment in staging first — same architecture, simulated traffic. She documented exactly what would happen, got sign-off from the VP of engineering, and then scheduled the production experiment with a rollback plan. Luna: So the staging run proved the test itself wasn't dangerous. And the rollback plan was basically 're-enable the node.' Lucas: Exactly. And once the team saw the value — they fixed that connection pool bug in two days — the resistance dropped. People started proposing their own experiments. The database team wanted to test what happens if the primary read replica goes down during a flash sale. The payments team wanted to simulate a third-party gateway timeout. Luna: It becomes a culture shift. Instead of fear of failure, you get curiosity about failure modes. Lucas: That's exactly the phrase Priya used: 'curiosity about failure modes.' And she built a small internal tool that lets any team schedule a fault injection experiment with a form. You pick the service, the fault type — node failure, latency injection, resource exhaustion — the blast radius, and the time window. It runs automatically, checks the steady-state metrics, and sends a report. Luna: So she automated the boring parts. That's smart because the manual overhead was probably the second biggest objection after 'our system is too fragile.' Lucas: Right. Manual experiments take time. You have to coordinate, document, monitor. With the tool, a team can run an experiment in staging every sprint. And they started running one production experiment per quarter, always during the lowest-traffic hour, always with a rollback button. Luna: What were some of the other things they found? I'm guessing the connection pool wasn't the only hidden bug. Lucas: Oh, they found plenty. One experiment injected five hundred milliseconds of latency into their Redis cache layer. They expected the application to degrade gracefully — serve stale data, maybe fall back to the database. Instead, a misconfigured timeout in the caching library caused requests to queue up, leading to a cascading failure across five microservices. Luna: A five-hundred-millisecond delay shouldn't cause a cascade like that. That suggests a really brittle timeout configuration. Lucas: Exactly. The library had a default connection timeout of one second, but the thread pool was fixed at fifty threads. When each thread waited a full second, the pool saturated in about fifty concurrent requests. And their checkout service routinely handles two hundred requests per second during peak. So the queue backed up in seconds. Luna: And that fix — increasing the thread pool or reducing the timeout — is the kind of change that's trivial once you know the problem exists. But you never would have found it without the experiment. Lucas: Right. And this is why I think chaos engineering is one of the highest roi investments a CTO can make, even for a team of fifty engineers. You don't need a dedicated SRE team. You need one person with a hypothesis, a staging environment, and a Friday afternoon. Luna: I want to zoom out for a second. Are there failure modes that chaos engineering can't catch? Like, does it miss anything obvious? Lucas: Good question. Chaos engineering is great for finding issues in distributed systems — network partitions, node failures, latency spikes — things that are hard to predict. But it won't catch logic bugs, security vulnerabilities, or data corruption issues that don't cause observable deviations in your steady-state metrics. You still need code review, penetration testing, and backup validation. Luna: So it's one tool in a larger resilience toolbox. But it's a tool that's historically been underused outside of big tech. Lucas: Underused and over-mystified. I think part of it is the name. 'Chaos engineering' sounds like you're causing chaos, when really you're being extremely methodical. Some teams call it 'resilience testing' or 'game days' to get buy-in. Luna: Game days. That's a good term. It frames it as a practice session, not a destruction derby. Lucas: Exactly. And the payoff is real. Priya told me that after six months of quarterly production experiments, their mean time to recover from outages dropped by forty percent. Because every experiment was a drill. The on-call engineers had already seen that failure mode, already run the playbook. Luna: Forty percent is huge. That's the kind of metric that justifies the investment to a CFO. Lucas: And it doesn't cost much. Open-source tools like Chaos Monkey, Litmus, or Gremlin's free tier can get you started. The real investment is time and culture. You have to be willing to say, 'We expect things to break, and that's okay because we'll learn from it.' Luna: I think that's a really healthy mindset for any engineering organization. And honestly, if today's conversation gave you something usable — maybe a concrete idea for your next sprint — that's the kind of thing that keeps this show ad-free. If it was worth a coffee to you, you know where to find it: buy me a coffee dot com slash fexingo. Lucas: Yeah, it really does help us keep doing deep dives like this one. And speaking of deep dives — one thing I want to come back to is how Priya prioritises which experiments to run. She doesn't just pick random services. Luna: Right, there has to be a framework. Otherwise you're just breaking things at random. Lucas: She uses a simple matrix. On one axis: how critical is the service to revenue? On the other: how complex is its dependency graph? The services that score high on both — like checkout and payments — get tested first. Then you move down to medium-criticality services like recommendations or search. Luna: So it's risk-prioritised. Start with the biggest potential blast radius. Lucas: Exactly. And within each service, you start with the failure mode that's most likely to occur. For most teams, that's a single node failure, not a whole-region outage. You don't need to simulate an asteroid hitting your data centre on day one. Luna: Start small, learn fast, iterate. That's the engineering way. Lucas: It is. And it works. So if you're a CTO or a tech lead listening and you've been hesitant to try chaos engineering, pick one service, one failure mode, and one hour next week. Run the experiment in staging. See what breaks. I think you'll be surprised at what you find. Luna: And if nothing breaks, that's also valuable — you've validated your architecture. Lucas: Right. Either way, you learn something. And that's the whole point.