Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Uses Blameless Postmortems to Improve Reliability
Transcript
- Lucas: So there's this CTO at a mid-sized fintech company, about 150 engineers, and last year they had a payment outage that lasted 47 minutes and lost them an estimated $2.3 million in transaction revenue. Luna: That's the kind of incident that gets someone fired at a lot of places. Lucas: Right, and that's exactly the instinct she wanted to fight. Instead of asking who caused it, she asked what system allowed it to happen. That shift is the core of blameless postmortems, and it's a practice that's been getting a lot of attention in the last few years, but very few teams actually do it well. Luna: What did she do differently? Lucas: She introduced a five-part postmortem template that reframes every incident as a system failure, not a human one. The template has five sections: timeline, impact, root cause, contributing factors, and action items. But the key is that in the root cause section, you're not allowed to name a person. You can only name a process, a tool, or a missing check. Luna: That sounds simple, but I imagine the cultural pushback was real. Senior engineers especially might feel like you're soft-pedaling mistakes. Lucas: Exactly. The CTO told me that the first few postmortems were tense. One staff engineer kept saying, 'Well, if Alice hadn't deployed that config change...' and she had to gently redirect him. But she stuck with it, and after about three months, people stopped looking for culprits and started looking for patterns. Luna: Did she see actual reliability improvements? Lucas: Yes, and the numbers are pretty striking. Over the next six months, the number of incidents with similar root causes dropped by 60 percent. And not just payment incidents—deployment failures, database timeouts, all of it. Because the postmortems were surfacing systemic issues that had been there for years. Luna: So the postmortems themselves became a kind of monitoring tool. Lucas: Exactly. And there's broader data to back this up. A 2024 survey by the DevOps Research and Assessment group found that teams practicing blameless postmortems resolve incidents 40 percent faster on average and have 30 percent lower change failure rates. Luna: But I want to push on something. Doesn't blamelessness sometimes let people off the hook? If someone repeatedly makes the same error, at what point does it become an accountability issue? Lucas: That's the most common objection, and it's a fair one. The key distinction is between individual error and systemic vulnerability. If the same person makes the same mistake twice, that's a pattern the system should catch. If it doesn't, that's a system failure too. But if someone is willfully negligent—ignoring a known process, skipping a required review—that's not a blameless postmortem scenario. That's a performance management problem. Luna: So blameless doesn't mean consequence-free. Lucas: Right. Blameless means you assume good intent and look for the conditions that made the error possible. If those conditions are present, anyone could have made the same mistake. And that's a much more productive framing for improving the system. Luna: I've heard that some teams struggle to implement postmortems because they take too much time. What did this CTO do about that? Lucas: She set a strict one-hour timebox for the meeting, and the template was designed to be filled out collaboratively in real time. She also made it a rule that the postmortem had to happen within 48 hours of the incident, while the details were still fresh. That speed actually saved time because people didn't have to reconstruct what happened from memory weeks later. Luna: And having a template keeps the conversation focused. Lucas: Exactly. The template also requires action items to have an owner and a due date. No vague 'improve monitoring' items. Every action is specific, like 'Add alert for database connection pool exhaustion when usage exceeds 80 percent.' And those items are tracked in a shared backlog, not just filed away. Luna: That seems like a great way to actually close the loop. Otherwise postmortems just become documentation theater. Lucas: Right. And one of the side effects the CTO noticed was that the postmortems started improving cross-team collaboration. Because the incidents often involved handoffs between teams—say, the payments team and the infrastructure team—the postmortem forced them to talk about those handoffs and find gaps. Luna: So it became a communication tool as much as a reliability tool. Lucas: She said it was the single most effective change she made in her first year as CTO. And it cost nothing except an hour per incident. Luna: If today's conversation gave you something usable, it's worth mentioning that this show is ad-free and listener-supported. A small group of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's what funds episodes like this one. Lucas: Yeah, it's a simple model. We'd rather keep the content focused on what actually works in engineering teams, not on selling ads. If you find value in the show, that link is there. Luna: Okay, back to postmortems. One thing I'm curious about is how you handle incident severity. Do you do a postmortem for every single incident, or just the major ones? Lucas: The CTO I talked to does postmortems for any incident that caused customer-facing impact or took more than two hours to resolve. For smaller incidents, they do a five-minute 'mini-mortem' in a shared doc—just timeline, root cause, one action item. That keeps the bar low so people don't dread the process. Luna: I like that. It strikes a balance between rigor and overhead. Lucas: And she also rotates the facilitator role. Every postmortem is led by someone from a different team, not the engineering manager. That way, the facilitator isn't biased by team dynamics, and more people get trained in systems thinking. Luna: That's smart. It builds the skill across the organization. Lucas: Right. And it prevents postmortems from becoming a blame game in disguise. When the facilitator is from a neutral team, it's easier to keep the conversation on process and systems. Luna: What about the action items? How do they make sure those actually get done? Lucas: She created a dedicated column in their project management tool called 'reliability backlog.' Each postmortem action item goes there automatically, and it's reviewed in the weekly engineering leadership meeting. If an item hasn't been started after two sprints, the CTO herself asks why. That sends a signal that reliability work is not optional. Luna: So the postmortem becomes a forcing function for continuous improvement. Lucas: Exactly. And over time, the backlog shrinks because the same types of issues stop recurring. The CTO said that after about nine months, they were doing fewer postmortems overall, and the ones they did do were about novel problems, not repeat failures. Luna: That's the goal, right? To make the system more resilient so you can focus on building new features. Lucas: Right. And I think the lesson here is that blameless postmortems aren't just about being nice. They're a practical tool for finding the real root causes of incidents, which are almost never 'someone made a mistake.' They're about missing tests, unclear ownership, or inadequate monitoring. Luna: And the template gives everyone a shared language for talking about failure without defensiveness. Lucas: That's the part I think is underrated. When you normalize talking about what went wrong in a structured way, you build psychological safety, and that pays dividends far beyond incident response. Luna: So if someone listening wants to start this tomorrow, where do they begin? Lucas: Start with the next incident, no matter how small. Use a simple template—timeline, impact, root cause, contributing factors, action items. Mandate that no person's name appears in the root cause section. Facilitate it with someone from a different team. And track the action items in a visible backlog. Do that consistently for three months, and you'll see the shift. Luna: That's a concrete starting point. I think a lot of teams overcomplicate this. Lucas: The best practices are often the simplest ones. The hard part is the discipline to keep doing them.