Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Uses an Incident Cost Calculator to Prioritize Reliability Work
Transcript
- Lucas: So there's this CTO I've been following — runs engineering at a mid-size fintech, about 150 developers. And he told me something that stopped me cold. Luna: I'm listening. Lucas: He said his team logs every incident — but until last year, no one knew what those incidents actually cost the company. They tracked severity labels, time to resolve, all that. But nobody could answer the question: 'How much money did this outage lose us?' Luna: That's a huge blind spot. Especially when you're trying to convince the board to let you spend engineering time on reliability instead of features. Lucas: Exactly. So he built what he calls an 'incident cost calculator.' It's basically a spreadsheet with five inputs. And once he started running every incident through it, the numbers changed how the company prioritizes. Luna: Five inputs — okay, what are they? Lucas: First: engineering hours spent on the incident, from detection to post-mortem. Second: the blended hourly rate for those engineers — including on-call pay if it's after hours. Third: revenue lost from churned accounts that the team can reasonably link to the outage. Luna: That's the hard one — attribution. Lucas: He doesn't try to be perfect. He uses a trailing 30-day average of daily active users who stop using the service after an incident, times average revenue per user. It's an estimate, but it's directional. Luna: And the last two inputs? Lucas: Fourth: a reputation multiplier — a flat percentage based on severity. P0 incidents get a 1.5x multiplier on the subtotal. P1 gets 1.2x. The idea is that major outages have a brand cost beyond immediate revenue. Fifth: any direct costs like AWS overage charges or credits issued to customers. Luna: So the formula is basically — engineer cost plus churn cost, times a reputation factor, plus direct costs. Lucas: You got it. And the first time he ran it on a five-minute P0 outage — just five minutes — the calculator spit out forty-three thousand dollars. Luna: Forty-three grand for five minutes? How? Lucas: Because it happened at peak trading hours. They had twelve engineers on the bridge for two hours. A dozen high-paid engineers, plus on-call double-time. The churn estimate was about eight thousand in lost subscription revenue that month. And the reputation multiplier kicked it up. The direct costs were negligible — just some extra database capacity. But the number still shocked the VP of Product. Luna: I bet. That's a compelling argument for spending fifty grand on a proactive fix. Lucas: That's exactly what happened. The CTO went to the product leadership meeting and said, 'We had three P0 incidents last quarter. Average cost per incident: thirty-one thousand dollars. Total: ninety-three thousand. I'd like to spend seventy-five thousand on a circuit breaker implementation that would have prevented two of them.' Luna: And they approved it? Lucas: They approved it in one meeting. Because the cost of the incident was right there. Before that, the conversation was always — 'Reliability is important, but we need to ship this feature.' Now they had a common language. Luna: I can see why this works. But I also wonder — does this create a perverse incentive? Like, if you're tracking incident costs, do teams start hiding incidents to keep the numbers low? Lucas: That's a real risk. The CTO told me he mitigates it by making the calculator a tool for the incident commander, not for performance reviews. It's not used to penalize teams. It's used to decide where to invest. If a team hides an incident, they lose the chance to get funding for the fix. Luna: And the transparency means someone else might catch it. Right — the on-call logs, the monitoring alerts. Lucas: Exactly. The other objection I hear is about overhead. If you have to calculate this for every incident, doesn't that take time? Luna: That was my next question. Lucas: So the CTO keeps it simple. He has a template spreadsheet. After the post-mortem, the incident commander fills in five cells. It takes maybe ten minutes. And he only runs it for P0 and P1 incidents, not for minor blips. Luna: Ten minutes per post-mortem — that's nothing. Especially if it unlocks a seventy-five-thousand-dollar budget. Lucas: Right. And the beauty is, once you have a few quarters of data, you can start spotting patterns. Like, maybe your database layer accounts for sixty percent of incident costs. That tells you where to focus. Luna: So it's not just a persuasion tool — it's a prioritization tool. Lucas: Exactly. The CTO told me that after six months, they had a ranked list of the most costly incident types. And they tackled them in order. The first fix — a query optimization — cost about twenty thousand to implement and eliminated a recurring incident that was costing them ten thousand per month. Payback period: two months. Luna: That's the kind of ROI the board loves. Lucas: It's also the kind of thing that changes how the company thinks about reliability. Because suddenly it's not just 'the site went down, that's bad.' It's 'this specific change pattern costs us thirty grand every time it happens.' Luna: So what's the catch? What does the CTO say about when this approach doesn't work? Lucas: He says the hardest part is getting the churn estimate right. If you overestimate it, you look like you're inflating numbers. If you underestimate it, you underinvest in reliability. He spent three iterations calibrating the churn model before he felt confident showing it to the CFO. Luna: That makes sense. You need a model that holds up to scrutiny. Lucas: The other thing he mentioned: this only works if you have decent incident tracking already. If you don't know how many incidents you're having, or you don't have post-mortems, you can't even start. Luna: Right — you need the raw data before you can layer cost on top. Lucas: But if you do have that foundation, the calculator is surprisingly easy to build. Five inputs. A spreadsheet. A little bit of calibration. And suddenly you have a shared language for reliability that goes beyond 'feelings' and 'best effort.' Luna: Yeah, that's powerful. I think a lot of engineering leaders struggle to make the case for reliability work because it feels like asking for a blank check. This gives them a dollar figure. Lucas: Exactly. And speaking of dollar figures — if today's episode gave you something you can actually use, you know, the thing that keeps this show ad-free and independent is listener support. We've got a buy me a coffee page, it's buy me a coffee dot com slash fexingo. No pressure, just if you found this helpful. Luna: Yeah, it's a small way to keep these conversations coming. And we really appreciate it. Lucas: So back to the calculator — one more thing the CTO said that stuck with me: he showed the calculator to his team and said, 'This isn't about blaming anyone. It's about making sure we spend our time on the things that actually matter.' And that shifted the culture. Luna: That's the real win. Not just the budget, but the mindset. Lucas: Exactly. Because once engineers see that a five-minute outage costs forty-three thousand dollars, they start taking reliability personally. In a good way. They start proposing proactive fixes themselves. Luna: And that's when you know you've built a reliability culture, not just a reliability process. Lucas: Yeah. So if you're an engineering leader trying to get buy-in for reliability work, try putting a price on your next incident. You might be surprised what the calculator says.