Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How a CTO Uses Observability to Reduce On-Call Fatigue
Transcript
- Lucas: Let's start with a number that stopped me: one CTO at a fintech startup told me their on-call team used to get pinged two hundred and forty times a week. That's about thirty-four alerts per night. After they redesigned their observability stack, that number dropped to seventy-two alerts per week — a seventy percent reduction. Luna: Seventy percent is huge. But did they just tune down the thresholds, or was it something more structural? Lucas: Much more structural. The CTO's name is Mira Patel — she runs engineering at a payments company called ClearRoute — and she said the problem wasn't the infrastructure. It was the philosophy. They had alerts for everything: CPU above eighty percent, memory above seventy-five, error rate above one percent. But none of those told you whether the customer could actually complete a payment. Luna: So they were measuring symptoms, not the actual user experience. Lucas: Exactly. Mira's team shifted to what she calls outcome-based observability. They defined three service level indicators: payment success rate, checkout latency at the ninety-fifth percentile, and the rate of duplicate payment attempts caused by client retries. That's it. Three SLIs. Everything else became diagnostic context — not alert triggers. Luna: That's a much smaller surface area. But how do you convince the team that it's okay to let the CPU hit ninety percent without paging someone? Lucas: That's where the service level objectives come in. Mira set an SLO of ninety-nine point nine percent uptime on payment success rate. As long as that was green, they would not alert on any infra metric. They had to trust that if the CPU was high but the SLI was fine, the system was absorbing the load. And if the SLI started to degrade, then — and only then — they'd look at the CPU as part of the root cause analysis. Luna: Right, so the alert becomes an investigation trigger, not a panic button. Lucas: Yes. And this is where it gets interesting. Mira said the seventy-two alerts that remained after the redesign were not all equally actionable. They further categorised them into pages and noise. Pages required a human to take action within five minutes. Noise was everything else — things you should look at during business hours. They set up automated runbooks for the noise category, so a lot of those alerts just created a ticket and maybe a dashboard annotation. Luna: How did the team respond to that? I imagine some engineers were sceptical about not being alerted right away. Lucas: The team was split initially. The senior engineers loved it because they'd been waking up for nothing. But junior engineers worried they'd miss something important. Mira handled that by pairing two-phase alerting with what she calls a learning rotation. The first week on-call, you're paired with a senior who explains the runbooks and the reasoning behind each alert. The second week, you're solo but with a senior on standby — no questions asked. Luna: That's a really nice structure. It turns on-call from a high-stakes solo event into a supervised learning experience. Lucas: Exactly. And the results went beyond just fewer pages. Mira measured incident response time before and after the change. The median time to acknowledge an alert dropped from ninety seconds to twelve seconds. That's because when an alert did fire, it was genuinely important — so engineers didn't ignore it. And the mean time to resolve dropped by about forty percent, partly because the runbooks were customised per alert type. Luna: So we have fewer alerts, faster responses, and better learning. That's a pretty compelling case. What about the engineers' well-being? Did on-call burnout actually decrease? Lucas: Mira tracks a metric she calls 'on-call satisfaction' — a quarterly survey where engineers rate their experience from one to five. Before the redesign, the average was two point one. After six months, it hit four point three. She also saw a forty-five percent reduction in unplanned PTO days taken by engineers who had been on-call the previous week. That's a pretty strong proxy for reduced burnout. Luna: That's data I haven't seen many CTOs collect. It's smart — it treats on-call as a product that should be designed for the user, which is the engineer. Lucas: Right. And the other side is that the company saved money. Mira estimated that each non-critical page cost about thirty dollars in wasted engineer time — context switching, waking up, trying to go back to sleep. Multiply that by the hundred and sixty-eight alerts they eliminated per week, and you're looking at roughly five thousand dollars a week in reclaimed productivity. That's over a quarter million a year. Luna: That's real. And it's not just the cost of the alerts — it's the cognitive load of being on edge all night. Lucas: Yeah. And look, this is a conversation we can have because we don't have to sell you anything. We deliberately keep these episodes ad-free. If you find value in that approach and want to support it, the link is buy me a coffee dot com slash fexingo. No pressure. Luna: It's a small way to keep this kind of specific, practical conversation going. Lucas: Back to Mira's approach — she also introduced something she calls a 'post-incident review that doesn't blame'. Every on-call incident, regardless of severity, gets a twenty-minute blameless postmortem within twenty-four hours. The goal isn't to find who pushed the bad code. It's to find why the system allowed the bad code to cause a failure. Luna: That's the core of good incident analysis. But twenty minutes seems tight. How do they keep it from devolving into a finger-pointing session? Lucas: They have a strict format. The first five minutes are a timeline — what happened, when. The next ten minutes focus on what they call the 'defense layer' — the monitoring, the canary, the circuit breaker — and which layer failed. The last five minutes are action items. No discussion of who did what. Mira says the key is a facilitator who can kill any comment that starts with 'he said' or 'she didn't'. Luna: That's a discipline. And I'm guessing the action items are rarely 'add more alerts'. Lucas: Exactly. The most common outcome is a runbook update or a test improvement. She showed me the data: over six months, seventy percent of postmortem actions were either automated tests or runbook changes. Only ten percent were new alerts. The rest were things like dependency version upgrades or config tweaks. Luna: So the observability redesign didn't just quiet the noise — it changed the culture around incidents. Lucas: That's the real win. Mira put it this way: 'On-call used to be something engineers dreaded. Now it's a rotation where you learn the most about the system.' And that shift is directly tied to having the right signals and the right process around them. Luna: I think a lot of engineering leaders hear 'observability' and think it's about tools — Datadog, Grafana, whatever. But this story shows it's really about agreement on what matters. Lucas: Absolutely. The tools are table stakes. The real work is defining those three SLIs, setting the SLO, and then having the discipline to not alert on everything else. Mira's team spent six weeks just on the SLI definition phase — arguing about what really counted as a customer-facing failure. Luna: Six weeks sounds long. But if it saved them a quarter million a year and cut burnout, it's probably the best investment they made. Lucas: Mira would agree. She told me the biggest resistance came from the infrastructure team, who had built dashboards for every metric they could scrape. Letting go of those alerts felt like losing control. But once they saw the new alerts — the ones that actually correlated with user-facing problems — they became converts. Luna: So the next time an engineer in my network complains about alert fatigue, I know exactly what to suggest. Lucas: Start with three SLIs. And don't be afraid to throw away the rest. Thanks for listening, and we'll see you next time.