Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Built Engineering Resilience Through Chaos
Transcript
- Lucas: If you've been paying attention to how major SaaS platforms handle scale in late 2026, you know that reliability isn't just about having good uptime monitors anymore. It's about how your systems behave when things go wrong, specifically at three in the morning. That brings us to Notion, a company that has increasingly become a case study in building engineering resilience through deliberate chaos. Luna: I remember reading their early posts on this, but I feel like the landscape has shifted since then. What’s the specific mechanism they’re using now that feels different from standard load testing? Lucas: The difference is in the granularity and the cultural acceptance. Most teams treat chaos engineering as a special event, maybe once a quarter. Notion has embedded it into their daily development cycle. They call it 'failure injection' as a standard part of their continuous integration pipeline, meaning every code merge potentially triggers a simulated failure in a staging environment that mirrors production traffic patterns. Luna: So they aren't waiting for a scheduled drill; they are testing hypotheses with every deployment? That sounds incredibly expensive in terms of compute resources. Lucas: It is expensive, but not in the way you might think. The cost isn't just the compute; it's the organizational friction. To make this work, they had to fundamentally change how engineers view bugs. A failed chaos test isn't treated as a regression error to be fixed immediately; it's treated as a design flaw to be addressed over time. This shifts the metric from 'zero incidents' to 'fast detection and recovery.' Luna: That’s a huge psychological shift. Engineers are usually punished for bugs, so telling them to intentionally introduce failures must have been met with resistance initially. Lucas: Exactly. In the beginning, there was a lot of pushback because it felt unsafe. But what changed the dial was transparency. When an engineer saw that a colleague’s chaos test caught a latent bug before it hit users, the fear turned into curiosity. They started competing to find the most interesting failure modes rather than hiding from them. Luna: It reminds me of how aviation safety works. Pilots train for engine failures precisely because they want to know how the plane behaves when one goes out. Is that the analogy they're using internally? Lucas: They do use that analogy, but they take it a step further by automating the response. Once a chaos test identifies a vulnerability, the system doesn't just alert the team; it can automatically trigger mitigation scripts or roll back changes if certain thresholds are breached. It turns reactive firefighting into proactive circuit breaking. Luna: I’m curious about the metrics here. How do they actually quantify 'resilience'? You can’t just say we’re more reliable than last month without a hard number. Lucas: They track something called 'Mean Time to Adaptive Recovery,' which is distinct from the traditional Mean Time to Repair. MTTR assumes you fix the root cause after the fact. MTTAR measures how quickly the system adapts to maintain service levels during the incident, even if the root cause remains unresolved. For Notion, that number has dropped significantly over the past year. Luna: That’s a really smart distinction. If the system can keep running despite the bug, the user experience isn't impacted, even if the underlying issue persists. That’s where the real value lies. Lucas: Precisely. And this ties into why I brought up the November 2026 context. With the current economic pressure on SaaS margins, companies can't afford large-scale outages that erode trust or require massive engineering overtime to fix. Building resilience is no longer just an engineering luxury; it's a financial imperative. Luna: Right, because every minute of downtime costs money, but so does the engineering bandwidth spent on post-mortems and hotfixes. Efficiency matters here. Lucas: Before we dig deeper into the technical specifics of their automation layer, I want to mention something quick. If these conversations are useful for what you're building or running, honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo — that's F. E. X. I. N. G. O. It keeps the show ad-free and helps us keep digging into these deep technical dives. Luna: Yeah, support really does keep the quality high. It lets us spend more time researching these niche engineering stories instead of chasing sponsorships for generic topics. Lucas: Let's get back to the mechanics. One of the most striking aspects of Notion's approach is how they segment their chaos experiments. They don't blast random failures across the entire platform. They use a concept called 'blast radius containment.' Luna: So they isolate the failure to a specific subset of users or services? That makes sense for minimizing risk, but how do they ensure the data is still representative? Lucas: They use a technique called 'canary injection.' They target a small, statistically significant percentage of live traffic—usually around one percent—and inject failures only into that slice. If the system handles it gracefully, the insight applies to the whole. If it fails, the impact is contained to that one percent, giving the team time to respond without causing a widespread outage. Luna: That feels much safer than doing a full-scale drill. It’s like testing a new drug on a small group first. But doesn't that mean they miss edge cases that only happen under total system load? Lucas: They do acknowledge that limitation. So they complement canary injection with periodic 'full-spectrum' drills, but those are rare and heavily monitored. The bulk of their resilience comes from the frequent, small-scale tests. It’s about volume of learning rather than intensity of a single event. Luna: Volume of learning. I like that phrase. It implies that resilience is a muscle you exercise regularly, not a exam you cram for. Lucas: Exactly. And this leads to another key component: the feedback loop between operations and development. In many companies, these two groups operate in silos. Ops fixes the fire; Dev writes the code. At Notion, the engineers who write the code are often the ones who see the chaos test results in real-time. Luna: So they own the consequences of their code in production. That accountability seems like the strongest driver of better engineering practices. Lucas: It is. When developers see their feature cause a latency spike during a chaos test, they don't blame the infrastructure team. They realize their code didn't account for network jitter or database contention. It forces them to build better observability and error handling from day one. Luna: That’s a powerful incentive structure. It aligns the developer’s goal of shipping features with the company’s goal of stability. Usually, those goals are in tension. Lucas: They used to be. But by making resilience visible and immediate, they’ve aligned them. Shipping fast is no longer reckless because you know exactly how the system will react if you ship something fragile. Luna: What about the tools? Are they building this entirely in-house, or are they leveraging existing open-source frameworks like Chaos Monkey or Litmus? Lucas: They started with open-source foundations but have built a proprietary orchestration layer on top. The reason is that off-the-shelf tools don't understand the specific dependencies of a complex application like Notion. Their internal tool understands that killing a database node in region A might cascade differently depending on whether a specific caching layer is active. Luna: So custom tooling is necessary for that level of precision. But that also creates a maintenance burden. Do they worry about the tool becoming a single point of failure itself? Lucas: They absolutely do. That’s why the orchestration layer is designed to be stateless and highly redundant. If the chaos engine goes down, the system doesn't panic; it just stops injecting failures until it comes back up. It’s treated as an operational utility, not a core service. Luna: Smart. Keep the testing infrastructure separate from the product infrastructure. It prevents scope creep and keeps the focus clear. Lucas: Another interesting trend we’re seeing in late 2026 is the integration of AI into chaos engineering. Notion is experimenting with using machine learning models to predict where failures are most likely to occur based on recent code changes. Luna: So instead of randomly injecting failures, the AI targets the most vulnerable areas? That sounds like a force multiplier for their engineering team. Lucas: It is. The AI analyzes commit histories, dependency trees, and historical incident data to suggest high-risk areas for testing. This allows the team to focus their limited chaos resources on the parts of the system that are most likely to break, rather than guessing. Luna: That’s fascinating. It moves chaos engineering from a brute-force approach to a surgical one. I wonder if other companies are adopting this ai driven model, or if Notion is staying ahead of the curve. Lucas: Most are still in the early stages. The data requirements for training these models are significant, and many companies don’t have enough historical incident data to build accurate predictors. Notion has been collecting this data for years, which gives them a competitive advantage in this space. Luna: Data moats are becoming as important as technological moats. If you have the right historical data, you can optimize processes that others can't even simulate effectively. Lucas: Absolutely. And this brings us back to the cultural aspect. All the tools and AI in the world won't help if the team is afraid to break things. The mindset shift is the hardest part of this journey. Luna: True. Technology enables the practice, but culture sustains it. Without psychological safety, engineers will hide failures rather than expose them through chaos tests. Lucas: That’s the core takeaway. Resilience isn't just a technical challenge; it's a human one. Organizations that succeed are those that reward transparency and learning over blame and perfection. Luna: It’s a reminder that the best engineering systems are built by humans who feel safe enough to admit they don't have all the answers. Lucas: Exactly. As we look at the rest of Q4 2026, I’d be curious to see which other major platforms adopt this kind of proactive, ai enhanced chaos engineering. It seems like the next frontier in operational excellence. Luna: I’ll be watching closely. If you have any insights on how smaller startups can implement these principles without the budget of a giant like Notion, let us know.