Latest / The Tech Career Podcast with Fexingo: Engineering Jobs, Interviews, and FAANG Career Strategy / How FAANG Engineers Use On-Call Rotations to Build Better Systems
Transcript
- Lucas: So, full disclosure: I used to dread on-call rotations when I was an engineer. It felt like the week where I got zero feature work done and just put out fires. Luna: Same. And I think that's still the default take for a lot of engineers — on-call is a tax you pay. Lucas: Right. But I want to argue today that on-call might actually be one of the fastest ways to build real production engineering skills, and — if you document it right — a serious career lever. Let's ground this in a specific case. Luna: Please. I need to hear this. Lucas: I was talking to an engineer at Amazon — let's call her Priya — who was a mid-level SDE 2. She'd been on a team that owned a critical microservice handling payment confirmations. Every other week she'd get paged at 2 AM for a cascade of database timeouts. Luna: Something about payment systems always breaks at 2 AM. It's like a law of physics. Lucas: It really is. So for about six months, Priya kept getting paged for the same class of issue. A downstream service would have a transient blip, her service would retry aggressively, and that retry storm would overload the database. Then everything would recover on its own after maybe 15 minutes. Luna: So the classic 'it fixes itself' pattern. Super dangerous because nobody bothers to fix it. Lucas: Exactly. But instead of just squashing each alert and going back to bed, Priya started logging every incident in a shared doc. She categorized them: timestamp, duration, root cause pattern, whether a runbook existed. After about three months, she had 15 incidents — and 12 of them were essentially the same bug: an aggressive retry with no exponential backoff. Luna: So she found a pattern in the noise. That's the key move. Lucas: It is. She then wrote a one-page design doc proposing a fix: implement exponential backoff with jitter, and add a circuit breaker so that if the downstream service is down for more than five seconds, her service stops calling it altogether. She socialized it with her team lead, got it prioritized, shipped it — and the pager alerts dropped by 80 percent. Luna: That's a real impact metric. I can already see where this is going for her promo packet. Lucas: Exactly. When her performance review came around, she didn't just say 'I responded to X incidents.' She said 'I identified a systemic retry storm pattern affecting 80 percent of our payment confirmation alerts, designed the fix, and reduced on-call burden by 80 percent.' That's a narrative. Luna: And it's a narrative that ties directly to two of the four promo criteria: impact and scope. But also — I want to push back a little. Not every on-call rotation yields that kind of clear pattern. Sometimes it's just random, unrelated fires. What then? Lucas: Fair. If the incidents are truly random, that's a signal too — it might mean the system is so fragile that everything breaks in different ways. In that case, the high-leverage move might be to push for an architectural stability sprint. Or to invest in observability so you can understand what's actually happening. Luna: So you're saying the on-call itself tells you what to work on. You just have to listen. Lucas: Exactly. The pager is a sensor. And the engineers who treat it as such — instead of as a nuisance — are the ones who end up building the deepest system knowledge. They know which queries are slow, which dependencies are flaky, which deployments tend to roll back. Luna: That knowledge is hard to get any other way. You can't learn the production failure modes from a design doc. Lucas: No. And that's why, at the best FAANG teams, senior engineers still rotate through on-call. They don't outsource it to junior engineers. Because being on-call gives you the authority to say 'this fix is worth doing now' — you've felt the pain yourself. Luna: Okay, so what's the concrete advice for someone who's on-call right now and wants to turn it into promo material? Walk me through the playbook. Lucas: Three steps. Step one: keep an incident log. Not just the alert timestamp — track the symptom, the suspected root cause, the time to resolution, and whether a runbook existed. After a month, you'll have data. Luna: And the data is your ammunition for the next step. Lucas: Right. Step two: find the top one or two patterns that cause the most pages or the longest downtime. Propose a fix with a concrete metric target — like 'reduce pages by 50 percent.' Step three: write a short postmortem or design doc and socialize it. You don't even have to implement the fix yourself. Just identifying the pattern and socializing a solution is already leadership behavior. Luna: That last point is important. I think a lot of engineers assume they have to be the ones to code the fix to get credit. But at FAANG, identifying the right problem and getting buy-in is often more valued than writing the code. Lucas: Exactly. Code is cheap. Judgment is rare. And on-call is one of the best training grounds for developing production judgment. Luna: So, Lucas — speaking of training grounds and judgment — this conversation about being deliberate with your work reminds me of something. We keep this podcast going without ads, and that's only possible because listeners like the folks who stick with us. Lucas: Yeah, it's true. And if today's conversation gave you one actionable idea — like starting an incident log — and you want to support more conversations like this, listeners can buy us a coffee at buy me a coffee dot com slash fexingo. Luna: It's a small gesture that makes a big difference for us. And we genuinely appreciate it. Lucas: For sure. So — back to the playbook. I want to talk about one more thing: what do you do when the on-call is genuinely overwhelming? Like, when you're getting paged every night and there's no clear pattern? Luna: That's a real problem. And I think the instinct is to just power through, but that leads to burnout fast. Lucas: Absolutely. In that scenario, the first thing is not to fix the incidents — it's to fix the on-call process itself. Add a pre-on-call handoff where the outgoing person walks you through the current issues. Make sure runbooks exist for the top five alert types. If they don't, write them as part of your on-call shift. Luna: Yes. Because every time you respond to an alert without a runbook, you're burning cognitive load that could be spent on the actual fix. Lucas: Exactly. And once you have runbooks, you can start looking for automation opportunities. If a runbook says 'restart the service', write a script that does it and ties it to the alert. Now you've eliminated an entire class of pages. Luna: So it's a flywheel. Better runbooks lead to better automation, which leads to fewer pages, which gives you time to fix deeper issues. Lucas: Exactly. And that flywheel is exactly what gets noticed in performance reviews. It's not 'I handled 50 pages.' It's 'I reduced pages by 40 percent and freed up the team to work on feature X.' Luna: Alright, I'm convinced. I'm actually going to start an incident log in my current rotation. Thanks for the push. Lucas: Glad to hear it. And for everyone listening — next time you get paged at 3 AM, try to see it not as an interruption, but as a clue. The system is telling you exactly where it's weak.