Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How a CTO Uses Error Budgets to Balance Speed and Stability
Transcript
- Lucas: There's a tension that every engineering leader feels — how fast can we ship without breaking things? Most teams eventually realize they can't maximize both. So what do you do? Luna: You pick a number. That's the idea behind error budgets. Lucas: Exactly. Error budgets come from Google's Site Reliability Engineering playbook. The core insight is simple: if you define a target for reliability — say 99.9 percent uptime — then the remaining 0.1 percent is your error budget. You can spend it on risky deployments, or conserve it by being cautious. Luna: And once the budget is gone, you stop shipping until it resets. That forces a real trade-off. Lucas: Right. And it takes the emotion out of it. Instead of a CTO arguing with a VP of Product about whether we're moving too fast, you have a clear signal: we've burned through our error budget this month. Time to stabilize. Luna: So how does a CTO actually set that number? Is it just picking 99.9 percent out of thin air? Lucas: Good question. I talked to a CTO at a mid-sized fintech company — about 150 engineers — who implemented error budgets about a year ago. He told me they started by looking at their actual historical uptime. They were already at 99.95 percent for most services. So they set their SLO — service level objective — at 99.9 percent, giving themselves a 0.05 percent buffer. Luna: That's interesting — they set it below what they were already hitting, so they had room to take risks. Lucas: Exactly. He said the goal wasn't to force them to be more reliable — they were already reliable enough. It was to give them permission to be less reliable on purpose when it made sense. The error budget for each service per month was about 0.1 percent downtime. For a service running 24/7, that's roughly 43 minutes of allowed downtime per month. Luna: So if they have a bad deployment that causes ten minutes of downtime, they've used up a quarter of their budget for that service. Lucas: Right. And the team can see that in real time. They have a dashboard that shows the remaining budget for each service. When it drops below 50 percent, the team gets a warning. When it hits zero, there's a hard stop: no more deployments to that service until the next month. Luna: What about incidents that aren't caused by deployments? Like a cloud provider outage? Lucas: That's a key nuance. Error budgets count all downtime against the budget, regardless of cause. Because from the user's perspective, the service was down. The CTO I spoke with said they initially considered excluding external dependencies, but decided against it. If AWS goes down and your service goes down, your users don't care who's at fault. Luna: That seems harsh but fair. It forces teams to build resilience even against things they can't control. Lucas: Exactly. And it changes the conversation. Instead of blaming the cloud provider, the team asks: what can we do to make our service survive that kind of failure? Multi-region deployments, graceful degradation, fallbacks. Luna: But not every service needs 99.9 percent. A payment processing service is different from an internal reporting dashboard. Lucas: That's where the CTO made a smart move. They defined three tiers. Tier one — customer-facing transactional services — got 99.9 percent. Tier two — important but not critical — got 99.5 percent. Tier three — internal tools — got 99 percent. Each tier had a different error budget, so teams could calibrate their risk tolerance. Luna: And if a tier three service burns through its budget, nobody panics. They just wait until next month. Lucas: Right. The CTO told me the real win wasn't the budget itself — it was the clarity. Before error budgets, every decision about whether to deploy was a negotiation. Now it's a data point. The team knows they have 20 minutes of downtime left this month, and they can decide if this feature is worth spending that time. Luna: But what happens when the budget runs out and there's a critical bug fix that needs to go out? Do they really wait? Lucas: That's the exception. They have a process for that. If there's a security vulnerability or a data loss bug, the team can request an emergency exemption. But the exemption goes through a review board — the CTO, the VP of Product, and the head of security. And the exemption itself burns even more budget from the next month. So there's a cost. Luna: It sounds like that keeps the exemption from being abused. Lucas: Exactly. In the first year, they only had three exemptions. Each one was justified, but the team felt the pain of losing next month's budget. It made them think twice about whether something really was an emergency. Luna: So how do you communicate this to the rest of the company? Especially product managers who want features shipped? Lucas: The CTO made it transparent. There's a shared dashboard that shows the error budget for every service. Product managers can see it. CEOs can see it. When a team says 'we can't deploy because our budget is zero,' it's not an opinion — it's a fact. And the product team can see when the budget resets. They can plan their releases around it. Luna: That's a big cultural shift. From 'the engineering team is blocking us' to 'the budget is telling us no.' Lucas: Exactly. And that's the whole point. The error budget becomes a neutral arbiter. It takes the heat off the CTO and puts it on the system. The CTO told me that after six months, product managers started asking 'what's the error budget for this service?' before they even proposed a feature. That's when you know it's working. Luna: I can see how that would reduce friction. But what about teams that have a really conservative budget they never use? Are they just being too cautious? Lucas: That's a great point. The CTO said they actually encourage teams to use their budget. If you're not spending any error budget, you're probably shipping too slowly. They want teams to deploy frequently and accept some failures. The budget is there to be used, not hoarded. Luna: So it's a tool for finding the right balance, not just a policing mechanism. Lucas: Exactly. And if today's conversation gave you something usable, you might enjoy knowing these episodes stay ad-free entirely because of listener support. If you found value, you can help keep it going at buy me a coffee dot com slash fexingo. Luna: It's a small gesture that makes a big difference for us. And it keeps the content focused on what matters — no sponsors, no interruptions. Lucas: Back to error budgets — one thing I found surprising was how they handled the budget at the end of the month. If a service has unused budget, does it roll over? Luna: I would guess not, because that would encourage hoarding. Lucas: No rollover. Each month resets. But they do something interesting: they publish a 'burn report' at the end of each month, showing how each service spent its budget. Teams can see which deployments consumed the most budget, and whether the risk was worth it. Luna: That's a great learning tool. You can start to see patterns — maybe a certain type of feature always causes incidents. Lucas: Right. The CTO said they found that deployments involving database schema changes were responsible for 60 percent of their budget consumption. So they invested in better migration tooling and automated rollback testing. That cut the burn rate by half. Luna: So error budgets don't just limit risk — they help you identify where to invest to reduce risk. Lucas: Exactly. And that's the virtuous cycle. You start with a budget, you learn from how you spend it, and you get better over time. The CTO told me that after a year, their overall uptime actually improved, even though they were shipping more features. Because they were spending their budget on the right things. Luna: It sounds like error budgets have become part of their engineering culture. Lucas: They have. New engineers are trained on it in their first week. The dashboard is one of the most viewed pages in the company. And the CTO says it's the single best decision he's made as a leader. Not because it's a silver bullet, but because it forces honesty about trade-offs. Luna: And that honesty is what makes a team resilient. You can't fix what you don't measure. Lucas: Right. And you can't have a productive conversation about risk without a shared language. Error budgets give you that language. It's a small change that has a huge impact on how decisions get made. Luna: So for any CTO listening who wants to try this, where do they start? Lucas: Start with one service. Pick your most critical one. Look at its historical uptime. Set an SLO that's slightly below that. Then track the budget for a month. Just observe. Don't enforce any gates yet. See how the team reacts. You'll learn a lot just from seeing the data. Luna: And then introduce the hard stop once people understand the concept. Lucas: Exactly. The CTO I spoke with did a three-month trial on one service before rolling it out company-wide. By the time they turned on the hard stop, the team already understood why it mattered. That made the transition smooth. Luna: It's a great example of using a simple mechanism to solve a complex human problem. Lucas: And that's the thing about good engineering leadership — it's not about the fanciest tools. It's about creating structures that make the right decisions obvious.