Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One CTO Turned On-Call Pain Into Developer Leverage
Transcript
- Lucas: So there's this assumption that on-call is just something you have to suffer through. That if you're a developer, part of the job is getting woken up at two in the morning for a minor alert that could have waited until morning. And most engineering teams accept that as the cost of doing business. Luna: Right. And it's a huge morale killer. I've seen teams where the best engineers try to game the rotation to avoid peak hours. Lucas: Exactly. But one CTO I talked to recently, Sarah Chen at a company called Logiform — they're a mid-sized SaaS platform for form automation, about 80 engineers — she decided to flip the whole model. She didn't just tweak the rotation schedule. She redefined what on-call actually means. Luna: Okay, I'm interested. What did she change? Lucas: She introduced what she calls a 'triage sprint.' Instead of a weekly rotation where someone is on call 24/7 and still expected to do their normal work, she assigns one developer to a dedicated two-week triage role. That person's only job for those two weeks is to handle incoming incidents — but with a strict severity filter. Luna: Wait — severity filter? So not every alert reaches them? Lucas: Right. She defined a clear threshold: only incidents that affect paying customers in production, or that cause data loss or a complete service outage, get escalated to the triage engineer. Everything else — minor latency spikes, non-critical warnings, automated test failures — gets routed to a low-priority queue that's reviewed during regular hours by the team that owns that system. Luna: But doesn't that risk missing something? A minor alert could be the canary for a bigger issue. Lucas: That's the pushback Sarah got too. But her argument is that if a minor alert is truly a harbinger, it'll recur or escalate within a few hours. And the owned team sees it the next morning anyway. What she found was that the noise was drowning out the signal. Before the change, the on-call engineer was getting paged for things like a single 99th-percentile latency spike that auto-resolved in 30 seconds. Luna: Yeah, I've been that engineer. It's exhausting and it trains you to ignore pages. Lucas: Exactly. So with the triage sprint model, the engineer only gets paged for genuine severity-one or severity-two incidents. And here's the key: they are expected to fully own the incident from detection through post-mortem. They write the incident report, they propose the fix, and they follow up to make sure the fix is deployed. It's not just a handoff. Luna: So it's a learning rotation, not a punishment. That's a completely different mindset. Lucas: It is. And the results are pretty striking. After six months, Logiform saw a 40 percent drop in after-hours pages, because the triage engineer was proactively fixing root causes during their sprint. They had the time and focus to actually improve monitoring, add better alerts, or patch flaky services. Before, nobody had the bandwidth. Luna: Forty percent is huge. But what about the engineers who aren't on triage? Do they just ignore system health entirely? Lucas: No, and this is the other smart piece. Every engineer is still responsible for the health of the services they build. But they're not expected to drop everything the moment a minor alert fires. Instead, they get a daily summary of low-severity issues in their domain, and they triage them during normal work. The triage engineer handles the fire drills. Luna: So it's a clear separation of concerns. The triage engineer is the shield. Lucas: Exactly. And Sarah also tracked a second metric: code contributions from engineers who had completed a triage sprint. They increased by 22 percent in the three months after their rotation. The theory is that those engineers now have a much deeper understanding of system behavior in production, so they write more robust code and fix issues faster. Luna: That makes sense. You can't really understand a system until you've seen it fail under real load. Lucas: There's also a confidence effect. Logiform runs quarterly incident response drills, and after the triage program, the average time to declare a severity level dropped by 35 percent. Engineers felt more comfortable making the call to escalate or not. Luna: I can see how that would happen. When you've practiced the full incident lifecycle for two weeks straight, you internalize the decision tree. Lucas: Right. And a big part of that is the 'no escalation' rule I mentioned. The triage engineer is empowered to resolve incidents without pulling in senior engineers, unless the incident meets the severity threshold. That forces them to actually make decisions. Luna: But what if they make the wrong call? That's a real risk. Lucas: Sarah's view is that making a wrong call is fine as long as it's reversible. And most incident responses are reversible — you can roll back a deploy, you can restart a service. The only non-reversible things are data loss or security breaches, and those are automatically severity one anyway. So the triage engineer is never alone on those. Luna: That's a good heuristic. 'Is this decision reversible?' If yes, let the triage engineer own it. Lucas: Exactly. And the data shows that mistakes were rare. Over six months, there were two incidents where the triage engineer should have escalated earlier. Both were caught within 15 minutes and had no customer impact. The learning from those was incorporated into the training for the next rotation. Luna: So the program also creates a continuous improvement loop. The post-mortems feed back into the triage playbook. Lucas: Yes. And Sarah told me one more thing that really stuck with me. She said the best side effect was that engineers stopped seeing the system as someone else's problem. When you've been the triage engineer, you have personal scars from that service. You become its champion. Luna: I love that. Ownership through pain, but structured pain. Lucas: Right. And it's not like this is a silver bullet. Logiform is a specific company with a certain culture and scale. But the principles — dedicated triage sprints, severity filtering, no escalation for non-critical issues — those are portable. Any team could try a two-week pilot. Luna: If today was actually useful to you, the way these stay ad-free is listener support. Buy me a coffee dot com slash fexingo. Lucas: Yeah, we hear from listeners that the practical focus is what keeps them coming back. That coffee link is how we keep this ad-free and focused on what actually works. Luna: So back to the triage sprint — one thing I'm curious about is how they handle the transition between sprints. Is there a handoff document? Lucas: Great question. Yes, they have a handoff template. The outgoing triage engineer writes a one-page summary of incidents handled, unresolved issues, and any changes they made to runbooks. Then they do a 30-minute sync with the incoming engineer. Sarah said the first couple of handoffs were rough, but after three cycles they had a solid rhythm. Luna: That's smart. It also means knowledge doesn't get lost when the rotation ends. Lucas: Exactly. And the handoff itself is a forcing function: you can't leave the role until you've documented what you learned. That alone improves the quality of the post-mortems. Luna: I want to try this with my team. I think our biggest hurdle would be convincing the senior engineers that it's okay not to be on the hook for every alert. Lucas: That's a cultural shift. Sarah said she started by having a senior engineer volunteer for the first triage sprint. After two weeks, that engineer became the biggest advocate. They saw how much deeper their understanding of the system got. Luna: So the proof is in the doing. I'm going to pitch a two-week pilot to my team and see if we can get a volunteer. Lucas: If you do, let us know how it goes. We love hearing from listeners who try things we cover.