Latest / Cybersecurity Business with Fexingo: Security Companies, Breaches, and Enterprise Defense / Why CISOs Are Using Security Chaos Engineering to Test Resilience
Transcript
- Lucas: Luna, I want to talk about something that sounds counterintuitive on its face — deliberately breaking your own security systems in production to see if they hold up. Luna: You mean chaos engineering for security. I have read about Netflix's Chaos Monkey, but that was more about infrastructure resilience, right? Lucas: Exactly. The core idea is the same — inject failures into a live environment to uncover weaknesses before they cause an outage. But security chaos engineering goes a layer deeper. Instead of just killing servers, you're simulating things like a TLS certificate expiring, an API rate limit being hit, or even a simulated ransomware encryption event on a replica database. Luna: That sounds terrifying for most security teams. The typical mindset is 'protect the production environment at all costs.' How do you get buy-in for something that intentionally breaks things? Lucas: It starts small. You don't start in production. You start in staging or a non-critical replica. And you frame it as a learning exercise — this isn't about blaming teams, it's about gathering data. Netflix famously built a culture where failure is treated as information. Security teams are starting to adopt that same ethos. Luna: I remember a case study from a large European bank. They ran a chaos experiment where they simulated a ransomware encryption event on a replica of their payment system. The result? They discovered their backup restore process had a bottleneck that would have taken 72 hours to fully recover. Lucas: Seventy-two hours is a lifetime in a ransomware incident. By finding that in a controlled setting, they could fix it before an actual attacker exploited it. The bank estimated that experiment saved them roughly three million dollars in potential downtime costs. Luna: So what tools are teams actually using for this? Is it all custom scripts, or are there off-the-shelf solutions now? Lucas: There is a growing ecosystem. AWS has Fault Injection Simulator, which integrates with EC2, ECS, and RDS. There's also an open-source project called Litmus that's designed for Kubernetes environments. And then you have dedicated security chaos platforms like ChaosSearch and Gremlin, which offer pre-built attack scenarios like DNS spoofing or privilege escalation. Luna: Gremlin I know from reliability engineering. So they've added security-specific failure modes? Lucas: They have. You can schedule a 'blast radius' — which is their term for the scope of the experiment — and it will only affect a small percentage of traffic. The goal is to validate that your monitoring alerts fire correctly, that your incident response team actually gets paged, and that your automated defenses kick in as expected. Luna: That raises an interesting point. Traditional penetration testing is point-in-time. You hire a firm, they test for a week, you fix the findings, and then six months later things have changed. Chaos engineering offers continuous validation. Lucas: That's exactly the value proposition. Pen testing is like a health checkup once a year. Chaos engineering is like a treadmill stress test that runs every sprint. And because it's automated, it can catch regressions — things that break after a code deploy or a configuration change. Luna: But I imagine the cultural hurdle is still huge. Most security teams are measured on uptime and incident counts. If you start breaking things on purpose, your metrics might look worse. Lucas: That's a real tension. What leading teams do is create a separate 'resilience score' that tracks how many experiments were run and how many blind spots were found — not how many incidents occurred. You have to decouple the chaos metrics from the operational metrics. Some organizations even have a dedicated 'chaos engineer' role that reports to the CISO, not to the IT ops director. Luna: So who is doing this well right now? I know Netflix is the poster child, but they're a unique engineering culture. Lucas: Beyond Netflix, I've seen adoption at financial services firms like Capital One and JPMorgan Chase. Also at e-commerce companies like Etsy and Shopify. They all have a strong DevOps culture to begin with. For more traditional enterprises, the entry point is often using chaos engineering to test their disaster recovery and backup processes — because that's a lower-risk, high-value scenario. Luna: That makes sense. Every CISO I talk to is worried about ransomware. Testing your recovery process under simulated pressure seems like a no-brainer. Lucas: And it catches things you wouldn't expect. I spoke with a CISO at a mid-sized manufacturing company who ran a chaos experiment that simulated a DNS outage — and discovered that their primary authentication server couldn't fall back to the secondary because of a misconfigured firewall rule. That was a five-minute fix, but under a real attack, it would have locked everyone out. Luna: That's the kind of blind spot you only find by running the experiment. So how do you get started if you're a team of, say, five to ten people? Lucas: Start with one experiment. Pick a low-criticality system — maybe your internal wiki or a dev environment. Use a tool like Litmus or Gremlin's free tier. Define a hypothesis: 'If we kill this container, the load balancer should route traffic to the remaining instances within 30 seconds.' Run it, observe, document what breaks, and then fix it. That's your first cycle. Luna: And then incrementally expand the blast radius. I like that it's a gradual approach. Lucas: Exactly. Over time, you build a library of experiments that cover different attack vectors — network partition, resource exhaustion, certificate expiry, even simulated insider threats like a privileged user exfiltrating data. Each experiment strengthens your defensive posture. Luna: You know, speaking of strengthening the show's posture — this episode is possible because of a small group of listeners who chip in monthly through Buy Me a Coffee. It's called buy me a coffee dot com slash fexingo. We're ad‑free, and that support literally funds the research and production for episodes like this. Lucas: Yeah, it's a handful of people who find value in these conversations and want to keep them going. If that's you, we appreciate it more than we can say. It keeps us independent and focused on what matters. Luna: Alright, back to chaos. One question I had — does security chaos engineering overlap with purple teaming? Because purple teaming is also about simulating attacks. Lucas: They're complementary but distinct. Purple teaming is a manual, scenario-based exercise where a red team attacks and a blue team defends, often with a specific objective. Chaos engineering is more automated and continuous, and it's not adversary-led — it's injecting specific failures and measuring system response. Purple teaming is great for testing detection and response, while chaos engineering tests resilience and recovery. Luna: So you could use chaos engineering to find the gaps, and then purple teaming to validate that the fixes work against a real adversary. Lucas: That's a smart workflow. Some organizations do exactly that — they run weekly chaos experiments, and then quarterly purple team exercises to stress-test the improvements. The combination gives you both breadth and depth. Luna: What about compliance? Are regulators starting to look at this? Lucas: Not yet in a formal way, but I think it's coming. The financial industry already has mandated stress testing for capital adequacy. It's not a huge leap to imagine that regulators like the SEC or the European Central Bank might eventually require resilience testing for critical IT infrastructure — especially after high-profile outages and ransomware attacks. Some insurance companies are already asking about chaos engineering programs when underwriting cyber insurance policies. Luna: That would be a powerful incentive. 'Show us your chaos experiment results or your premium goes up.' Lucas: Exactly. And I think that's where the industry is heading — from reactive defense to proactive resilience validation. Security chaos engineering is one of the most pragmatic ways to get there. Luna: So if someone wants to start today, what's the single best resource? Lucas: The O'Reilly book 'Chaos Engineering' by Casey Rosenthal and Nora Jones is still the foundational text. For security‑specific content, look up the 'Security Chaos Engineering' papers from the Chaos Community Day conference. And the Litmus project has excellent getting‑started guides for Kubernetes environments. Luna: Great. I think this will be one of those practices that looks obvious in hindsight — like why wouldn't you test your defenses under controlled failure conditions? Lucas: Right. The question is whether your team is willing to learn from failure in a safe space, or wait for a real attacker to teach you the same lesson at a much higher cost.