Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / Why Your Engineering Team Needs an Incident Commander
Transcript
- Lucas: There's a moment in every major outage where chaos threatens to take over. The alerts are firing, Slack is blowing up, the VP of Sales is standing at your desk asking when the site will be back. And the worst thing you can do in that moment is let everyone jump in and try to help. Luna: Because more cooks in the kitchen, more fingers on keyboards — that often makes things worse, right? Lucas: Exactly. That's why more engineering teams are adopting the role of an Incident Commander. It's a concept borrowed from emergency response — fire departments, military — and it's surprisingly effective for handling production outages. Luna: So instead of five senior engineers each pulling a different log, there's one person directing traffic. Lucas: Right. The Incident Commander, or IC, doesn't fix anything. Their job is to triage, communicate, and make rapid decisions about who works on what. They keep the rest of the team focused on the actual fix. Luna: I've seen this work well. A friend at a mid-size e-commerce company told me about a black Friday incident where their site went down for 47 minutes. They estimated it cost them about $300,000 an hour in lost revenue. Lucas: That's a painful number. And without an IC, that outage could have easily stretched longer. The engineering director told me they had fifteen people in a war room chat, each proposing a different theory. One engineer started restarting database replicas, another was pushing a config change. No coordination. Luna: Classic recipe for extending the outage — or even making it worse. Lucas: They eventually designated a senior engineer as IC. That person immediately stopped everyone from making changes and asked each person to report what they'd observed. Within ten minutes, they had a clear picture: a database connection pool was exhausted because a new deploy had a connection leak. Luna: And the IC was able to say 'you, roll back the deploy; you, increase the pool limit temporarily; you, start drafting the postmortem.' Luna: So what does an IC actually do during an incident? Walk me through the key responsibilities. Lucas: First, they establish the severity level — is this a Sev-1 where the site is down, or a Sev-2 where a feature is degraded? Then they assemble the right responders. They assign a scribe to document the timeline. They declare the incident publicly in a channel so everyone knows it's being handled. Luna: And they keep stakeholders updated without pulling engineers away from fixing. Lucas: Right. The IC is the single point of contact for status updates. They say 'we are investigating,' 'we have identified the root cause,' 'we are testing a fix.' That keeps the VP of Sales from texting the lead engineer directly. Luna: That alone is worth the role. How do you train someone to be a good IC? It's not something you learn from a book. Lucas: Most teams start by having senior engineers shadow experienced ICs. They go through tabletop exercises — simulated incidents where they practice the role. And they run post-incident reviews that evaluate the IC's decisions, not just the technical fix. Luna: One thing I wonder: does every team — even a small startup of five engineers — need an official IC? Or is that overkill? Lucas: Honestly, for a team of five, you can probably just have one person say 'I'm IC' at the start of an incident. The title matters less than the mindset. But once you have more than ten engineers, you want a formal rotation. Otherwise, the same two people end up doing it every time and burn out. Luna: And you want to rotate the role so that everyone gets experience leading under pressure. It builds resilience across the team. Lucas: Yeah. A lot of teams also pair a technical IC with a communications IC — one focuses on the fix, the other handles updates. That works especially well for long-running incidents. Luna: Quick honest thing — we're able to make episodes like this because a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo. That's literally what funds making this many episodes ad-free. We don't run ads, we don't have sponsors, it's listener supported. Lucas: Yeah, and we really appreciate it. It keeps us independent and lets us dig into topics like this without worrying about pleasing an advertiser. So if you find value in the show, that's where you can help. Luna: And if you already do — thank you. Now, back to incident command. Lucas, you mentioned tabletop exercises. Can you give an example of what that looks like? Lucas: Sure. You gather the team in a room or a video call and you say 'Okay, it's 2 PM on a Tuesday. Your database master has just crashed. Writes are failing. What do you do?' Then the designated IC starts calling out actions, and the team responds with what they'd check. Luna: It's like a fire drill for production incidents. Lucas: Exactly. And you throw in curveballs — 'the primary on-call engineer just got a phone call and is now unavailable,' or 'the monitoring dashboard is showing incorrect data.' It trains people to adapt. Luna: One thing I've seen teams struggle with is the IC overstepping and trying to fix things themselves. When the pressure is on, it's hard to resist. Lucas: That's the biggest trap. The IC who starts debugging loses the big picture. They miss new alerts, they stop communicating, and they become a bottleneck. The rule should be: the IC does not touch any keyboard except to type in the incident channel or make a call. Luna: So it's a discipline. You need to trust the engineers you've assigned to actually fix it. Lucas: And that trust is built through practice. Which is why the best teams run incident drills regularly, not just after a crisis. They make it part of their engineering culture. Luna: I've also seen companies tie the IC role into their career ladder. Being a skilled IC becomes a path to senior or staff engineer. Lucas: That's smart. It recognizes that incident response is a distinct skill — it's not the same as writing great code. Some of the best engineers I know are terrible ICs because they can't resist diving into the code. And some engineers who are average coders are fantastic ICs because they stay calm and coordinated. Luna: So you're saying you should pick your ICs based on temperament, not seniority alone. Lucas: Absolutely. The calmest person in the room might be a mid-level engineer. They should be the IC, not the VP of Engineering who wants to prove they can still fix things. Luna: Good point. I want to circle back to something you mentioned earlier — the scribe role. How important is that? Lucas: Hugely important. The scribe documents everything: when each alert fired, who was paged, what actions were taken, when the fix was deployed. That timeline is gold for the postmortem. Without it, you're reconstructing from memory, and memory is terrible. Luna: Especially after a stressful incident. Everyone's recollection differs. Lucas: Exactly. So the scribe is a non-technical role that anyone on the team can fill — even a product manager or a technical writer. It frees the technical folks to focus. Luna: I like that. It also gets non-engineering stakeholders involved in the incident process, which builds understanding across the company. Lucas: For sure. One more thing: after the incident is resolved, the IC leads the postmortem — but they are not the target of blame. The postmortem is about the system, not the people. The IC's decisions are reviewed as part of that, but the tone is always blameless. Luna: And that blameless culture is what makes people willing to step up as IC in the first place. Lucas: Right. If you punish the IC for a decision that seemed right at the time, no one will want the role. So celebrate the ICs, learn from the incidents, and keep improving the process. Luna: I think the key takeaway is that incident command is a skill you can teach, and it's one of the highest-leverage investments an engineering team can make. Lucas: Totally agree. The cost of an outage isn't just lost revenue — it's lost trust, team morale, and sleep. A good IC reduces all of those.