Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / Why Your Team Should Run Blameless Postmortems
Transcript
- Lucas: You know, I was reading through Stripe's engineering blog from way back — 2015, I think — and there's this one postmortem that I keep coming back to. Luna: Which one? They've had a few famous ones. Lucas: The one about the 'critical' database migration that brought down payments for about 27 minutes. The engineer who triggered it was their most senior DBA — the person everyone trusted most. Luna: Right, and they could have easily said 'that person made a mistake, let's retrain them'. But instead they dug into why the system allowed one person to take down production. Lucas: Exactly. And that's the core of what we're talking about today: blameless postmortems. It's one of those practices that sounds soft — like, 'oh, we're just being nice to each other' — but it's actually one of the most brutally technical disciplines an engineering team can adopt. Luna: Before we go deep — and I know we will — I want to quickly mention that this is exactly the kind of conversation we get to have because we don't run ads on these episodes. If you find value in unpacking real engineering practices like this, you can support the show at buy me a coffee dot com slash fexingo. It's how we keep this ad-free and focused on what matters. Lucas: Yeah, I love that we can just talk about the work without selling anything. So back to Stripe — that postmortem didn't just blame the DBA. They asked: why did we not have a peer review process for database migrations? Why was there no canary deployment stage? Why was the rollback procedure undocumented? Luna: So they turned a human error into a list of system improvements. That's the blameless philosophy in a nutshell. Lucas: Right. And the term 'blameless' gets misinterpreted a lot. It doesn't mean no accountability. It means you don't stop at 'Bob fat-fingered the query'. You ask: what conditions made Bob's single action possible? Luna: What's the actual psychology behind blame? Why do teams default to it? Lucas: There's a concept from cognitive science called the 'fundamental attribution error' — when something goes wrong, we attribute it to the person's character or ability. When it goes right, we attribute it to the situation. So when a deployment breaks, it's easy to say 'that engineer was careless'. But when it works, we say 'the system was robust'. Luna: So blame is a cognitive shortcut. It feels efficient. Lucas: It does, but it's actually counterproductive. Because if you blame Bob, Bob learns to hide mistakes, not to fix systems. And Bob is probably the person with the most context about that failure — the last person you want to silence. Luna: What does a good blameless postmortem actually look like in practice? Give me a template. Lucas: Sure. Start with a timeline — what happened, in chronological order, with timestamps. No analysis yet, just facts. Then a 'contributing factors' section — not root causes, but all the things that lined up. This might include a missing test, a timeout that was too short, a monitoring alert that fired but no one saw it. Luna: And then the action items? Lucas: Yes, but each action item must address at least one contributing factor. And here's the key: every action item needs an owner and a deadline. If it doesn't, it's a wish. Also, the postmortem should be shared broadly. At Stripe, they published them internally for everyone to read. Luna: Didn't Google's SRE book say something similar? That postmortems should be 'blameless' and 'written for an audience of engineers who weren't involved'? Lucas: Exactly. Google's SRE culture is basically the gold standard. They say that if a postmortem doesn't result in at least three concrete action items, it probably wasn't thorough enough. And they also recommend a 'time to mitigate' metric — how long between first alert and when the system was stable again. Luna: What about smaller teams? If you're a startup with five engineers, do you still need formal postmortems? Lucas: I'd argue especially then. Because when you're small, every outage is a huge percentage of your credibility. And you don't have the safety net of multiple teams watching each other. A simple shared doc with a standard template works. The important thing is the ritual — the team sits down, reviews the timeline, and writes down what they'll change. Luna: I've seen teams where the postmortem devolves into a blame session despite good intentions. How do you prevent that? Lucas: It starts with the language in the document. Never use the word 'he', 'she', or the engineer's name in the narrative. Use 'the deployer', 'the on-call engineer', 'the reviewer'. That small linguistic shift makes a huge difference. Also, the person who was most involved should be the last person to speak in the meeting — let others share their perspective first. Luna: That's a really practical tip. I've also seen teams appoint a 'postmortem facilitator' who isn't the team lead — someone whose job is to keep the conversation constructive. Lucas: Yeah, a neutral facilitator is huge. Especially if the incident was high-severity. The facilitator's only job is to make sure the team doesn't skip from 'what happened' to 'whose fault' without going through 'why did the system allow it'. Luna: Let me push back a little. Some people argue that blameless culture can lead to 'no consequences' for repeated mistakes. How do you handle someone who keeps causing the same type of outage? Lucas: That's a fair concern. But if the same person causes the same failure multiple times, a blameless postmortem will actually surface that pattern — and the system is clearly not preventing it. The question becomes: is this a training issue, a process gap, or a misalignment of skills? The postmortem gives you data to have that conversation. You don't need blame to identify that someone needs a different role or more support. Luna: So blameless doesn't mean low standards. It means high standards for the system, not just for the people. Lucas: Exactly. And that's the shift that's so hard for traditional managers to make. Especially in industries that came from manufacturing or operations where 'human error' is the end of the investigation. Luna: Speaking of other industries, I was reading about how aviation handles this. Every incident goes to a neutral board — the pilot isn't even in the room for the initial analysis. That's essentially a blameless postmortem at scale. Lucas: Aviation is the classic parallel. They have the 'just culture' model — they distinguish between honest mistakes, reckless behavior, and malicious intent. Only reckless and malicious get disciplinary action. Honest mistakes get systemic fixes. That's exactly the framework we should borrow in engineering. Luna: We actually have time for one more example — what's a company that did this really well recently? Lucas: I like what Etsy did a few years back. They have a 'Blameless Postmortem of the Month' award. It's a bit tongue-in-cheek, but it signals that the company values learning over punishment. They also integrated postmortems into their onboarding — every new engineer reads three past postmortems before they ship any code. Luna: That's smart. It sets the expectation from day one: we break things, and we talk about it openly. Lucas: And it normalizes failure. The best engineering teams I've seen don't have fewer incidents — they have more incidents, but they're smaller and they're resolved faster. Because everyone is constantly learning what the weak spots are. Luna: So if a team wants to start doing blameless postmortems tomorrow, what's the first step? Lucas: Pick the last incident that bothered everyone — the one people still grumble about in stand-up. Write a timeline. List contributing factors. Propose three changes. And then schedule a 30-minute meeting where the only rule is: no names, no blame. Do that once, and you'll see the difference. Luna: I think a lot of teams resist because they think it takes too much time. But a 30-minute postmortem that prevents a 2-hour outage later is a huge net positive. Lucas: Absolutely. And the more you do it, the faster it gets. After a while, it's just part of the cycle — incident happens, postmortem happens, system gets better. It becomes a habit, not a chore. Luna: I feel like we've barely scratched the surface. But I think the core message is clear: blame is a waste of engineering time. Systems thinking is where the leverage is. Lucas: Yeah, and if you take one thing from this episode, I hope it's the idea that your postmortems should be your best source of product feedback — not just for reliability, but for how your team works together.