Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One Team Cut Their Incident Response Time by 70 Percent
Transcript
- Lucas: There's this stat I keep turning over in my head. A mid-stage SaaS company — about forty engineers — cut their mean time to acknowledge an incident from twelve minutes down to under four. That's a 70 percent improvement. And they didn't hire anyone or buy a new tool. Luna: Twelve minutes to even acknowledge something's on fire? That feels long. My old team used to average around eight and we thought that was fine. Lucas: Right, and that's exactly the baseline most teams accept. But this team realised that every extra minute of unacknowledged alert meant the issue was compounding. A single customer-facing outage that lasts five extra minutes can cost thousands in support tickets and churn risk. Luna: So what did they actually change? I'm guessing it wasn't just 'tell people to be faster.' Lucas: No, and that's the key. They ran a pre-mortem first. Before changing anything, they gathered the whole engineering team and asked: 'If our incident response fails next quarter, what went wrong?' The number one hypothesis was alert fatigue — too many noisy alerts drowning out the critical ones. Luna: Classic. You can't respond faster if you're sifting through twenty alerts an hour. Lucas: Exactly. So they made three structural changes. First, they redesigned the on-call rotation from a single global rotation to a follow-the-sun model. Three shifts: Americas, EMEA, Asia-Pacific. Each shift had a dedicated primary and secondary. That alone cut the average time to first touch because the primary wasn't getting paged at 3 a.m. and fumbling to wake up. Luna: Follow-the-sun works great if you have engineers in those time zones. Not every team does. Lucas: True. This company already had a distributed workforce, so it was feasible. But the second change applies anywhere: they implemented a tiered escalation protocol. Every alert gets classified as P1, P2, or P3. P1s go directly to the primary's phone with a five-minute acknowledgement SLA. P2s go to the secondary first. P3s go to a Slack channel and only escalate if not acknowledged in thirty minutes. Luna: That's smart — you're effectively routing noise away from the people who handle the real fires. Lucas: And the third change was the most controversial internally. They introduced a practice called 'swarming.' Instead of the on-call engineer trying to fix everything alone, the first responder's job is to acknowledge and then immediately pull in whoever is needed — even if that means a senior engineer drops their feature work. The first responder stays on the incident until resolution, not just until they hand it off. Luna: That sounds expensive. Doesn't that kill productivity on feature work? Lucas: It did at first. But they tracked it. Before swarming, the average incident had three handoffs and took forty-five minutes to resolve. After swarming, it dropped to twenty-two minutes. The feature work interruption was real but smaller than the cost of prolonged outages. And they found that the same engineers kept getting pulled in, so they started rotating who was the designated 'swarm target' each week. Luna: So the follow-the-sun addressed time zone gaps, tiering cut noise, and swarming reduced handoffs. That's a three-pillar approach. I like it. But I wonder — does this scale beyond forty-person teams? Lucas: Good question. At larger orgs, you often have dedicated SRE teams. But the principles still apply. I've seen teams of two hundred use a variant where the tiered escalation goes to a channel with a rotating incident commander who triages and assigns. Luna: And the pre-mortem — that seems like the unsung hero here. They diagnosed their own bottleneck before tinkering with the rotation. Lucas: Absolutely. Most teams jump straight to changing the schedule or buying a better alerting tool. This team spent two hours in a room with a whiteboard and came out with a clear theory. That upfront investment probably saved them months of trial and error. Luna: If today's tech conversation gave you something usable — maybe a new way to think about on-call or a pre-mortem for your own team — we'd love it if you considered supporting the show. We deliberately keep Tech Leadership ad-free because we think that makes for a better listen. If you value that, the link is buy me a coffee dot com slash fexingo. Lucas: Yeah, and it genuinely helps us keep doing deep dives like this one. No pressure, just a simple way to say 'this is useful.' Now, back to the tiering — one detail I want to highlight is how they defined severity. They didn't use uptime alone. They used customer impact: P1 meant a revenue-affecting feature was down for paying customers. P2 meant degraded performance for a subset. P3 was everything else. Luna: That's a much better heuristic than 'server CPU above 90 percent.' Lucas: Right. And they enforced it by tying the classification to the monitoring setup. Any alert that didn't have a clear customer-impact label was automatically demoted to P3 until the team could reclassify it. That forced engineers to write better alert definitions. Luna: So the alert quality improved over time as a side effect. That's the kind of flywheel you want. Lucas: Exactly. And the result was that after three months, the number of P1 alerts actually went up — because people stopped ignoring real issues. But the MTTA stayed under four minutes because the team trusted that if it was a P1, it mattered. Luna: That's counterintuitive. Most teams would see more P1s and think something's broken. Lucas: Yeah, but it's actually a sign of health. You're surfacing problems that were previously buried. And because the response was faster, the time to resolve dropped, so the total time spent on incidents didn't increase. In fact, the team reported a 30 percent reduction in after-hours pages because the follow-the-sun model meant incidents were handled during someone's working hours. Luna: I want to zoom in on the handoff problem. You mentioned three handoffs per incident before swarming. Where were those handoffs happening? Lucas: Typically: on-call engineer acknowledges, investigates for a few minutes, realizes it's a database issue, pages the DBA. DBA looks at it, fixes part of it, but then needs a code change, so pages the backend engineer. Backend engineer ships a fix, but then the original on-call has to verify. That's three handoffs. Each one adds context loss and delay. Luna: Swarming cuts that to one handoff — from the alert to the first responder, who then stays on it. Lucas: Right. And the first responder doesn't have to know how to fix everything. They just have to be the glue: acknowledge, pull in the right people, coordinate the fix, and verify. It's a coordination role, not a deep technical role. Luna: That changes the skill set you need for on-call. You can put junior engineers on primary rotation if they're good at communication and escalation. Lucas: Exactly. And that's actually what this company did. They made the primary role more about incident management and less about deep debugging. Senior engineers were the swarm targets. That helped with burnout because juniors felt empowered to escalate without needing to know everything. Luna: I can see that being a huge morale boost. Instead of dreading on-call because you might get a 3 a.m. page you can't handle, you know you have a clear process. Lucas: Yeah, and they also introduced a 'no-blame' policy for escalations. If you escalate, you're doing the right thing. No one ever gets dinged for pulling in a senior engineer unnecessarily. That was a cultural shift that took a few weeks to stick. Luna: What about the tools? You said they didn't buy anything new, but did they change how they used existing tools? Lucas: Good catch. They used PagerDuty and Slack, which they already had. But they reconfigured PagerDuty to enforce the tiered escalation — P1 alerts went straight to phone with a high-urgency notification, P2 went to phone but with a different sound, P3 only posted to Slack. They also set up an automated 'incident channel' in Slack that included the primary, secondary, and any swarm targets. Luna: So the main investment was time and process design, not tooling. Lucas: Exactly. And that's the lesson I keep coming back to: most incident response improvements are process improvements dressed up as technical changes. The team that makes the biggest leap isn't the one with the best monitoring — it's the one that explicitly designs its escalation and handoff protocols. Luna: I want to ask about the pre-mortem itself. How did they structure that two-hour session? Did they just freeform brainstorm? Lucas: They had a facilitator — the engineering manager — who started by stating the premise: 'It's six months from now, and our incident response has failed. We've had a major outage. Write down what went wrong.' Everyone wrote silently for five minutes, then shared. They grouped the notes into themes: alert fatigue, poor handoffs, lack of clear ownership, and after-hours burnout. That gave them four focus areas. Luna: And then they prioritized. Alert fatigue and handoffs were the top two. That's how they arrived at the three changes. Lucas: Right. And they also used the pre-mortem to identify what was working — the team already had good monitoring and a blameless culture. So they didn't try to fix those. That's key: don't fix what isn't broken. Luna: If another team wanted to replicate this, what's the single most important step? The one thing they should do first? Lucas: Run the pre-mortem. Seriously. Spend two hours with your team and a whiteboard. You'll likely find that your biggest bottleneck isn't what you think it is. And you'll get buy-in because the team owns the diagnosis. Then implement the tiered escalation — it's the highest-leverage change because it directly reduces noise and ensures the right people see the right alerts. Luna: I think the follow-the-sun piece is harder to copy, but the tiering and swarming are universally applicable. Even a five-person startup could do tiered escalation. Lucas: Absolutely. You could set up P1, P2, P3 in any alerting tool in an afternoon. And swarming is just a process change — you don't need a tool for that. So the barrier to entry is really low. Luna: One thing that struck me is that this team measured MTTA, not just MTTR. Most teams focus on time to resolve, but acknowledging faster creates accountability. Lucas: Right. MTTA is a leading indicator. If your MTTA is high, your MTTR will never be low because the clock starts ticking when the alert fires. By cutting MTTA, they gave themselves a head start on resolution. And they tracked it per shift, which helped them see that the Americas shift had the fastest MTTA because that's where most of the senior engineers were based. Luna: So they used that data to balance expertise across shifts — maybe move one senior engineer to the APAC rotation. Lucas: Exactly. And that's the kind of iterative improvement that compounds. They didn't get it perfect on day one. They made the three changes, measured for a month, then adjusted. The follow-the-sun shifts originally had equal numbers of engineers, but after a month they rebalanced based on incident volume per time zone. Luna: So the process was: pre-mortem, implement, measure, adjust. That's a classic cycle, but it's rare to see teams actually execute it. Lucas: It is. And I think the reason it worked here was that the engineering manager treated incident response as a product. They set clear metrics, iterated based on data, and involved the whole team in the design. That's the leadership takeaway: treat your operational processes with the same rigor you'd treat a customer-facing feature. Luna: That's a good place to wrap up. I'm going to suggest my team run a pre-mortem next sprint. Thanks, Lucas. Lucas: Do it. And let me know what you find. For everyone else, if you try any of these ideas, we'd love to hear how it goes. Until next time.