Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Postmortems at Google Cut Downtime 50 Percent
Transcript
- Lucas: Google's internal analysis of its incident postmortems from 2023 showed something surprising. Teams that consistently did blameless postmortems saw a 50 percent reduction in repeat incidents over two years. Luna: Fifty percent? That's huge. But what exactly makes a postmortem 'blameless' in practice? It sounds nice in theory, but I imagine it's hard to enforce. Lucas: It's not just about avoiding blame — it's about systematically looking for systemic causes. Google's framework focuses on five whys, action items with owners, and a timeline of events. The key is that every incident is treated as a learning opportunity, not a performance review. Luna: That kind of systematic learning is exactly why we keep these conversations ad-free. Listener support on buy me a coffee dot com slash fexingo helps us keep doing that. Lucas: Yeah, it's a small gesture that makes a big difference. So back to Google — their data also showed that the biggest drop in incidents came from teams that did postmortems within a week of the event. Luna: Interesting. So speed matters too. I remember a DevOps survey from a couple years ago that found most organizations take over two weeks to do a postmortem, if they do one at all. Lucas: Right. And that delay dilutes the learning. People forget details, the context fades, and the incident loses urgency. Google mandates a draft within five business days, then a review within ten. Luna: So what does a good postmortem actually look like? Walk me through the anatomy. Lucas: Sure. Every postmortem starts with a timeline: exactly what happened, when, and who was involved. Then a summary of impact — users affected, revenue loss, duration. Then the root cause analysis, and finally action items: specific, measurable steps to prevent recurrence. Each action item gets an owner and a deadline. Luna: And they track those action items? That seems like a lot of overhead. Lucas: They have a dedicated tool — internally they call it the Postmortem Dashboard. It tracks completion rates, and org leaders review them quarterly. If an action item is overdue, it escalates. The culture is that reliability is everyone's responsibility, not just SRE. Luna: That reminds me of a talk from a Google SRE director who said, 'If you haven't had a major incident in the last quarter, you're probably not pushing hard enough.' Lucas: Exactly. The goal isn't zero incidents — it's learning from every one. In fact, Google's research found that teams with the lowest incident rates also had the highest postmortem participation. Correlation, not causation, but it's compelling. Luna: I imagine the cultural shift is the hardest part. How do you get engineers to admit mistakes without fear? Lucas: It starts at the top. Managers need to model vulnerability — share their own mistakes, participate in postmortems as equals, and never punish someone for surfacing a problem. Google has a principle called 'security and reliability are everyone's responsibility' that reinforces this. Luna: So what's the one thing a small startup could do tomorrow to start building this culture? Lucas: Pick your last big incident — maybe the one that woke you up at 3 AM — and run a blameless postmortem with just the people involved. Write a simple doc: what happened, why, what we'll do differently. Then actually do the action items. That one practice compounds over time. Luna: I like that. It's actionable. But what about the friction — some engineers see postmortems as bureaucratic overhead. Lucas: That's a real risk. The key is to keep them short and focused. Google recommends no more than one page unless it's a P0. And the action items should be small — something you could do in a sprint, not a multi-quarter project. Luna: One page for a major outage? That's discipline. Lucas: Absolutely. The discipline forces clarity. If you can't describe the root cause in two sentences, you probably haven't found it yet. And that's another lesson from Google — depth of analysis matters more than length. Luna: Are there tools that help enforce this? I know some teams use Jira templates or Confluence, but it's still manual. Lucas: Google uses a custom tool, but there are off-the-shelf options like PagerDuty's Postmortems or even a simple GitHub issue template. The tool is less important than the commitment to actually write and track them. Luna: Let's talk about a real example. What's a famous Google postmortem that changed how they operate? Lucas: The 2020 Gmail outage — lasted over four hours, affected 1.5 billion users. The root cause was a configuration change that overloaded the storage system. The postmortem led to a complete rethink of how they rollout config changes — canary testing, gradual rollouts, automated rollback. Luna: And I bet that postmortem was studied across the entire org. Lucas: It was required reading for all SREs and many product teams. That's another piece of Google's culture: postmortems are shared broadly, with lessons anonymized if needed, but the goal is org-wide learning. Luna: I've heard some companies worry that sharing postmortems externally could hurt their reputation. Google shares some publicly. Does that help? Lucas: It builds trust. When customers see that you're transparent about failures and have a plan to fix them, they're more likely to stay. Google's public postmortem for the 2020 outage actually received positive feedback from enterprise clients. Luna: So what's the current state of postmortems in the industry? Are most companies following Google's lead? Lucas: It's uneven. The State of DevOps report from 2025 showed that high-performing teams are three times more likely to do blameless postmortems than low performers. But overall, only about 30 percent of organizations do them consistently. Luna: That's a huge opportunity. So if a CTO is listening and wants to start, what's the first conversation they should have with their team? Lucas: Set the expectation: from now on, every incident will have a postmortem within a week. Lead by example — write the first one yourself, even if it's a minor issue. Show that it's safe to be honest. Then track action items and celebrate the learnings. The culture will shift within a quarter. Luna: It sounds simple, but I imagine the hardest part is sustaining the discipline when things are calm. Lucas: That's exactly right. The teams that see the 50 percent reduction are the ones that do postmortems for every P2 and above, not just the big ones. Consistency is everything. Luna: So what is one thing Google still struggles with? No organization is perfect. Lucas: One challenge is that postmortems can become too inward-looking. Sometimes teams focus on the technical details and miss organizational or cultural root causes — like burnout or communication gaps. Google has been working on adding 'systemic causes' as a required field in their templates. Luna: That's a good reminder — software runs on people, not just code. Lucas: Exactly. And that's the deeper value of postmortems: they reveal the human side of reliability. When done well, they build trust, improve communication, and ultimately make the team stronger. Luna: I've seen that happen at smaller companies too. It's not just a Google thing. Lucas: Not at all. The principles scale down — start small, stay consistent, and always ask 'what can we learn?'