Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Datadog Slashed Incident Response Time by 40 Percent
Transcript
- Lucas: Luna, you know how every major platform has that one outage that changes everything. For Datadog, that was July 2024, when a cascading failure in their own monitoring pipeline took down their dashboard for 47 minutes. They call it the Dogfood Incident. Luna: Right — I remember reading their postmortem. They ended up redesigning their entire incident command structure, not just the tech. Lucas: Exactly. And these are exactly the kind of real-world engineering stories that make this podcast useful for what you're building or running. If you find these deep dives valuable, listener support is what keeps us ad-free and independent. You can support the show at buy me a coffee dot com slash fexingo. Luna: Yeah, it really helps us bring you these episodes. So what actually happened during those 47 minutes? Lucas: Well, Datadog's monitoring pipeline itself had a dependency chain collapse — an upstream metric store went down, which triggered backpressure, which caused cascading timeouts. And the alarming thing was, their own incident response process didn't have a clear commander. Multiple teams jumped in, but nobody had the authority to make the tough call to restart the service. Luna: So the problem was as much organizational as technical. Lucas: Bingo. So the CTO, after the postmortem, introduced three major changes. First, they created a dedicated Incident Commander role. That one person has full authority during an incident — they can shut down deploys, redirect teams, even roll back without waiting for a director's sign-off. Luna: That's a big culture shift. Was it hard to get buy-in? Lucas: They started with a pilot in the SRE team, and after a few incidents where having a clear commander cut resolution time by nearly half, other teams adopted it. The key was training — they ran mandatory game-day drills where every senior engineer had to play Incident Commander at least once. Luna: Game-day drills — like fire drills for outages? Lucas: Exactly. They simulated realistic failures — a database failover, a DNS outage, a DDoS attack — with a dedicated game master who threw in curveballs. The goal wasn't just to test runbooks, but to practice decision-making under pressure. They found that the biggest delays weren't technical — they were people waiting for someone else to act. Luna: So the drills trained confidence and delegation. Lucas: Right. And the third change was their post-incident review process. They moved from a blame-focused timeline to a systems-thinking approach. Every review asks: what in our architecture or process allowed this to happen? And they track action items with owners and deadlines — no more 'we should fix this someday.' Luna: And the results? I've seen claims of 40 percent MTTR reduction. Lucas: That's real. They measured their mean time to resolve across all P1 incidents before and after — dropped from 45 minutes to 27 minutes on average. That's a 40 percent improvement. And their time to acknowledge — the gap between alert and first responder picking it up — went from 8 minutes to under 2. Luna: So the Incident Commander role and the drills probably drove that acknowledgment time down. Lucas: Exactly. Because now when an alert fires, the on-call knows exactly who to escalate to. There's no 'who's in charge?' deliberation. The other interesting metric: they saw a 60 percent reduction in incidents that required a rollback, because commanders were more confident making surgical fixes instead of wholesale reverts. Luna: That's a really compelling case study. What did the CTO say was the hardest part? Lucas: He said it was convincing senior engineers to give up control during incidents. Many of the best engineers wanted to jump in and fix the problem themselves, not coordinate. The drills helped — they showed that a well-run incident command actually gets the best engineers working on the right things faster. Luna: So the cultural shift was about trusting the process rather than the heroics. Lucas: Exactly. And that's something any CTO can apply, regardless of stack. The specifics — the role, the drills, the review format — are all adaptable. Datadog open-sourced their incident command handbook, by the way. It's worth a look. Luna: I'm going to check that out. It's a reminder that the biggest reliability gains often come from how you organize your people, not just how you design your systems. Lucas: Absolutely. And that's the takeaway for this episode. Whether you're running a hundred-microservice platform or a monolithic app, the principles of clear command, practice under pressure, and blameless learning can radically change your incident outcomes.