Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How We Used Observability to Prevent Outages Before They Happened
Transcript
- Lucas: There's a question I hear from engineering leaders all the time — and I used to ask it myself. 'How do we get better at responding to incidents?' And the honest answer is: you don't want to get better at responding. You want to have fewer incidents. Luna: Right. The whole industry is obsessed with postmortems and runbooks, but the real win is catching something before it reaches production. Lucas: Exactly. And that's where observability — real observability — shifts from a debugging tool to a prevention tool. I've got a case in mind. A mid-size fintech company, about 200 engineers. Their SRE team built something I think is worth unpacking. Luna: Let's hear it. What'd they do? Lucas: They set up a canary-based anomaly detection system. The idea is simple: before any production deploy, they route a small percentage of live traffic — maybe five percent — to the new version running in a separate cluster. But instead of just checking HTTP status codes, they instrumented the canary with high-cardinality metrics. Every request gets tagged with user ID, endpoint, latency percentile, memory usage per pod. Luna: So they're not just asking 'is it up?' They're asking 'does it feel weird?' Lucas: Exactly. And that caught a memory leak two hours before a scheduled production deploy. The canary's memory usage per pod started climbing linearly — nothing alarming at first, but the SRE team's dashboard flagged it because the trend deviated from the baseline they'd built from the previous week's data. Luna: Two hours is huge. Most teams would have only noticed after the full rollout, when pager duty starts lighting up. Lucas: That's the key. They didn't need a better runbook for the memory leak — they needed a way to see it coming. And they did. They rolled back the canary, fixed the issue, and redeployed the next day. The incident never happened. Luna: So how do you make observability proactive instead of reactive? Because most teams I talk to are drowning in dashboards they never look at. Lucas: That's the trap. Monitoring gives you alerts for things you already know to look for — CPU above 90 percent, error rate above one percent. Observability means you can ask questions you didn't know you'd need to ask. And to do that, you need three things: high-cardinality data, the ability to slice it arbitrarily, and a culture that rewards exploration. Luna: The third one is the hardest. Engineers are taught to fix things, not to sit and stare at graphs. Lucas: Right. So the fintech company did something smart. They gave each SRE four hours every week for what they called 'observability office hours.' No on-call, no tickets — just time to build custom dashboards for services they were curious about. One engineer built a dashboard that correlated database connection pool exhaustion with a specific query pattern from a mobile client. That caught a regression that would have hit all users within a week. Luna: So the investment wasn't in more tooling — it was in time. That's a leadership decision. Lucas: Exactly. And I think that's the underrated part. You can buy Datadog or Honeycomb or Grafana, but if your team doesn't have the time to explore, you're just paying for blinking lights. Luna: You mentioned a framework earlier — the three-tier observability stack. What does that look like in practice? Lucas: Sure. Tier one is 'is the service up?' — basic health checks, pings, status codes. Most teams stop here. Tier two is 'is the service behaving?' — latency distributions, error budgets, slow queries. That's where you catch the memory leak before it kills the pod. Tier three is 'is the system changing?' — diffing traffic patterns, dependency versions, config drifts. That's where you catch the cascading failure before it starts. Luna: And most observability tools claim to do tier three, but you actually have to build the baselines and the diffs yourself. Lucas: Right. The tool is just the canvas. The painting is the team's curiosity. And leadership has to protect that time. Luna: I think this ties into something we've touched on before — the idea that the best engineering cultures celebrate prevention, not heroics. If your team's biggest wins are all-nighters putting out fires, you've got a culture problem. Lucas: One hundred percent. And that's a hard cultural shift because firefighting is visible. Prevention is invisible. Your CEO doesn't see the outage that didn't happen. But your team feels it — they go home on time, they sleep through the night, they trust the system. Luna: So what's the first thing a CTO or VP of Engineering should do this week to move toward proactive observability? Lucas: Pick one service that's critical but not yet catastrophic. Give one engineer half a day to build a tier-two dashboard for it. No other agenda. Just explore. I've seen that single investment uncover issues that would have become pager-able incidents within a month. Luna: And the cost is basically zero — just a few hours of salary. That's the kind of thing that makes me think a lot of teams are sitting on a goldmine of prevention potential they haven't tapped. Lucas: If today's conversation gave you something usable — a framework, a story, a nudge — and you want to see more conversations like this stay ad-free and independent, listener support is what makes that possible. You can find us at buy me a coffee dot com slash fexingo. No pressure, no perks — just a way to keep the show going. Luna: Yeah, it's a small thing that adds up. And it lets us keep doing deep dives like this one. Lucas: So back to the canary approach — one thing I love about it is that it also changes the conversation around deployments. Instead of 'did we break anything?' it becomes 'what did we learn about this new version?' Luna: That's a reframe I hadn't thought of. It turns a deployment from a risk event into a learning event. Lucas: Exactly. And when you treat it as a learning event, you naturally invest in observability because you want to see more. You want to understand the system's behavior under the new code. Luna: So the tooling follows the mindset. You don't buy your way into prevention — you think your way into it. Lucas: That's the whole episode in one sentence, honestly. Luna: Alright, I'm going to go tell my team to block off Thursday afternoon for dashboard building. Lucas: That's the spirit. And if you do, let us know what you find. We might do a follow-up episode on what teams actually discover when they start looking. Luna: I'd love that. I think the listeners would too.