Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Uses the Five Whys on Incidents to Find Root Causes
Transcript
- Lucas: If you've ever sat in a postmortem where the root cause was written as 'human error', you know that's not a root cause—it's a conversation-ender. Luna: Right. It basically says 'someone messed up' and stops there. But the real question is why the system allowed that error to happen. Lucas: Exactly. And that's where the Five Whys method comes in. It's a simple technique from Toyota's manufacturing system—when something goes wrong, you ask 'why' five times to peel back the layers from symptom to systemic cause. Luna: Five times feels almost arbitrary. Is it always exactly five? Lucas: It's a rule of thumb. Sometimes you get to the real cause in three questions, sometimes it takes seven. The point is to keep digging until you hit a process or system failure that you can actually fix. Luna: Can we walk through an example? I think that's where the method really comes alive. Lucas: Sure. Sarah Chen is the CTO at a mid-size fintech company called PayBridge. She shared a case from last year. One of their core databases went down for eleven minutes during peak transaction hours. Luna: Eleven minutes in fintech is an eternity. Lucas: Exactly. So the team did a Five Whys session. First why: why did the database go down? Answer: a schema migration consumed all available I/O. Second why: why did that migration run during peak hours? Answer: it was deployed automatically as part of a routine release pipeline. Luna: Okay, so automation didn't check the timing. Lucas: Third why: why didn't the pipeline have a guardrail to block deployments during peak? Answer: the team had never defined peak hours in the pipeline configuration. Fourth why: why wasn't that defined? Answer: the team assumed migrations were always safe and didn't need gating. Luna: That assumption seems fragile. Lucas: Fifth why: why did the team assume migrations were always safe? Answer: because previous migrations had never caused issues, and there was no process to review migration scripts for performance impact. That's the systemic root—a missing peer review step for any database change. Luna: So the real fix isn't about the migration tool or the pipeline—it's about adding a mandatory review for database changes, regardless of urgency. Lucas: Right. Sarah's team added a lightweight review: any schema change must be approved by another engineer before merging. They also added a check in the pipeline that blocks deployments during defined peak windows unless explicitly overridden. That override requires a second approval. Luna: I like that the Five Whys forced them to address both the technical and the cultural side. Lucas: It does that naturally because the method keeps asking 'why' until you can't go further. And it works best when you do it in writing, not just in a meeting. Writing forces clarity. Luna: A lot of teams I talk to skip the writing step—they just talk through it verbally. Lucas: That's a mistake. When you write it down, you can see if your causal chain actually holds together. And you can share it with others who might spot a gap. Sarah's team uses a shared document template with five rows, each with a 'why' and the answer. Then at the bottom, they list the countermeasures—specific actions that prevent recurrence. Luna: One critique I've heard is that the Five Whys can oversimplify complex systems. If there are multiple contributing factors, a single chain might miss them. Lucas: That's a fair point. The classic Five Whys assumes one linear path. But in reality, incidents often have multiple causes. A good facilitator recognizes when the chain branches and runs multiple Five Whys in parallel. Luna: So you'd have multiple 'fifth whys'? Lucas: Exactly. For the database outage, a secondary chain might start from 'why did the migration script have a performance issue?' That could lead to a different root: the test environment didn't have representative data volume, so the migration seemed fast in testing. Luna: That adds another countermeasure: improve test data fidelity. Lucas: Right. The method is flexible. What matters is that each 'why' leads to a cause that is actionable. If you ever answer with 'human error,' you haven't gone deep enough. Human error is a symptom, not a cause. Luna: This connects to something Sarah said in her write-up that stuck with me: 'The goal isn't to find who to blame, but to find what in our system allowed the mistake to happen.' Lucas: That's the blameless postmortem philosophy in a nutshell. And the Five Whys is a simple tool to operationalize that. You don't need expensive software—just a document and a willingness to be honest. Luna: Before we move on, a quick honest thing. A handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these episodes. So if today's tech conversation gave you something usable, that's where the show stays ad-free and independent. Lucas: Yeah, it's a small group that keeps this going. No pressure, just if it's useful to you and you want to keep it coming. Anyway, back to the Five Whys—one trap I see teams fall into is stopping at a cause they can't change. Luna: Like if the root cause is 'we need more engineers'—that's not actionable in the moment. Lucas: Exactly. Sarah trained her team to ask one more question after each Five Whys: 'What would prevent this from happening again?' If the answer is something you can't do this quarter, you haven't found the real root. Keep digging. Luna: That question seems like a good litmus test. Lucas: It is. And it keeps the exercise grounded. You want countermeasures you can implement this week or at least this sprint. For the database outage, the review step was set up in two days. The pipeline guardrail took a bit longer because it required changes to their CI/CD system. Luna: So the Five Whys works best when paired with a culture that prioritizes small, fast improvements. Lucas: Right. And it scales up and down. I've seen it used for a one-line bug fix and for a multi-team outage. The structure is the same. The discipline is the same. Luna: What advice would you give to a CTO who wants to introduce this method to their team? Lucas: Start with one incident. Do a written Five Whys with the team involved. Share the result broadly. Make it visible that the goal is learning, not blaming. And model it yourself—if you make a mistake, offer to do a Five Whys on your own decision. Luna: Lead by example. Lucas: Exactly. Sarah did that. She once triggered a config change that caused a brief outage, and she volunteered to be the subject of a Five Whys session. That sent a powerful signal. Luna: That's the kind of vulnerability that builds trust. Lucas: And it makes the method stick. Because if the CTO is willing to be analyzed, everyone else will feel safe to participate. Luna: One last thing—what's the biggest misconception about the Five Whys? Lucas: That it's a root cause analysis technique. It's actually a questioning technique to help you discover potential root causes. The real analysis comes after, when you evaluate which countermeasures are most effective. The Five Whys gets you to the candidate causes, but you still need to validate them with data. Luna: So it's a starting point, not the end. Lucas: Exactly. Pair it with data from your monitoring, your logs, your timeline. The Five Whys gives you a hypothesis; the data confirms or refutes it. That combination is powerful. Luna: And it doesn't require any new tools—just curiosity and a shared document. Lucas: That's why I think it's one of the highest-leverage practices an engineering leader can adopt. Low cost, high impact, and it builds a learning culture.