Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How a CTO Uses an Incident Commander Role to Handle Outages
Transcript
- Lucas: So you're a CTO, it's two in the morning, your pager goes off — the checkout flow is down. By the time you get to Slack, there are already twelve people in the channel, each one asking 'should I roll back?' 'is it the database?' 'who's checking the CDN?' Total chaos. Luna: I've been in that room. It's the worst feeling — everyone wants to help, but nobody's steering. Lucas: Exactly. And that's why a growing number of engineering orgs are borrowing a concept from emergency services: the incident commander. One person whose only job during an outage is to coordinate — not to fix anything. Luna: So they're like the fire chief who never picks up a hose. They're directing traffic. Lucas: Right. I talked to a CTO at a mid-sized SaaS company — about 150 engineers — who rolled this out after a particularly bad outage last year. He told me their pre-commander approach was essentially 'whoever shouts loudest takes charge.' Mean time to resolution was averaging about four hours for critical incidents. Luna: Ouch. Four hours of a payment system down can cost serious money. Lucas: Exactly. So they modeled their incident commander role on the Incident Command System used by fire departments. Five core responsibilities: first, establish command — that's literally announcing 'I'm the commander' at the start of the incident. Second, set the incident objectives — for example, 'restore checkout within thirty minutes.' Third, assign roles: a comms lead for stakeholders, a tech lead for debugging, a scribe for the timeline. Luna: The scribe is underrated. Every postmortem I've been in that lacked a timeline ended up with finger-pointing. Lucas: Absolutely. Fourth, the commander maintains the big picture — they're not looking at logs, they're tracking which teams are working on what. And fifth, they declare the incident over and initiate the postmortem. Luna: What happens if the commander is the most senior person in the room? Do they still resist debugging? Lucas: That's the hardest part. The CTO I spoke with said his biggest mistake in the first drill was jumping into a database query. He had to train himself to literally sit on his hands. He now has a rule: if the commander types a command on any production system, they automatically hand off command to someone else. Luna: That's a strong rule. But does it scale? What about when you have a multi-team incident — say, frontend, backend, and infrastructure all involved? Lucas: That's where the model really shines. They assign sub-commanders for each domain. The overall commander talks to the sub-commanders, not to individual engineers. So the frontend sub-commander says 'we have a CDN cache issue, estimated fix twenty minutes,' and the overall commander decides whether to redirect traffic or wait. Luna: So it's a tree structure. Commander at the top, sub-commanders below, and engineers doing the actual work. That prevents the information overload. Lucas: Exactly. The company ran a drill once a month for three months before they used it in a real outage. When the real one hit — a database connection pool exhaustion — the commander declared at minute one, assigned roles by minute two, and they had a fix deployed in twenty-two minutes. Their previous average was two hours for database incidents. Luna: That's a huge improvement. But I wonder — does this work for smaller teams? If you only have five engineers on call, can you spare a commander who doesn't touch code? Lucas: Fair question. The CTO I talked to said for teams under ten engineers, they combine the commander and scribe roles — one person does both — because the incident complexity is lower. But they never combine commander and tech lead. Those two must always be separate. Luna: That makes sense. So the principle is: one person always has the helicopter view, never gets sucked into the weeds. Lucas: Exactly. And they also stress that the commander role is not tied to seniority. A junior engineer who's calm and organized can be a better commander than a senior engineer who wants to fix everything. They've had level-three engineers command incidents with level-seven engineers debugging. Luna: That's a culture shift. How do you get senior people to take orders from someone more junior during a crisis? Lucas: Drills. They practice with scenarios where the commander is deliberately junior. After a few drills, people see that the role is about process, not authority. The CTO told me the only pushback came from one senior engineer who refused to be commanded — until he was the one debugging and the commander kept the noise away. He became a convert after that. Luna: I want to talk about tools for a second. Do they use a specific platform for this? Lucas: They use Slack, but with a very structured channel naming convention. Each incident gets a channel like '#inc-2026-06-27-checkout'. Within that, only the commander can post in a dedicated 'commander-thread'. Everyone else posts in a 'debug-thread'. The scribe maintains a pinned post with the timeline. They also have a Slack workflow that auto-generates a postmortem document when the incident is declared over. Luna: So the structure is enforced by process, not just by personality. That's smart. Lucas: Right. And the CTO told me that the biggest win wasn't even the faster resolution times — it was the reduced burnout. Before the commander model, on-call engineers would get paged, jump into a chaotic channel, and feel responsible for everything. After, they knew they only had to focus on their assigned task. One engineer told him it felt like going from being a one-person fire department to being part of a well-trained crew. Luna: That's a powerful shift. And I think it's a great example of how a simple organizational change — not a new tool, not more automation — can have an outsized impact. Lucas: If today's tech conversation gave you something usable, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Luna: Yeah, it's a small thing that makes a real difference. Keeps the whole thing ad-free and focused on the content. Lucas: So back to the commander model — one more thing that came up in the conversation. They also use a 'shadow commander' program. Every incident has a primary commander and a shadow who observes and debriefs afterward. That way, they're building a bench of trained commanders. Luna: That's a great way to scale the practice without relying on one or two people. How long does someone shadow before they take command? Lucas: They observe at least three real incidents and pass a simulation. The simulation is a tabletop exercise where they walk through a scenario like 'the database replica is lagging and the primary fails over.' They're evaluated on whether they establish command, assign roles, and avoid debugging. It takes most people about three months to become certified. Luna: So it's a serious investment. But the payoff is clear — faster recovery, less stress, and a culture where everyone knows their role. Lucas: Exactly. And that's the thing — the CTO said the certification process itself actually improved their incident response. Because people had to think through scenarios in advance, they found gaps in their runbooks. So the training became a forcing function for preparedness. Luna: That's a nice side effect. Any advice for a CTO who wants to start implementing this tomorrow? Lucas: Start small. Pick one upcoming drill or a low-severity incident and explicitly designate a commander. Don't change anything else. Just have that person say 'I'm the commander, my job is to coordinate.' See how it feels. Then iterate. The CTO I spoke with said the first time they tried it, the commander accidentally started debugging after five minutes. They laughed about it in the postmortem and tried again next week. Luna: Progress over perfection. I like that. Lucas: Yeah. And that's really the takeaway — you don't need a perfect system on day one. You just need one person willing to say 'I'll steer, you fix.'