Latest / The Tech Career Podcast with Fexingo: Engineering Jobs, Interviews, and FAANG Career Strategy / How FAANG Engineers Debug Production Issues Under Pressure
Transcript
- Lucas: You've been at a company for six months, you're on call for the very first time, and suddenly your phone buzzes — it's a PagerDuty alert. Something is broken in production. A feature is returning errors for a significant percentage of users. Your heart rate goes up. What do you do first? Luna: Honestly, my first instinct would be to panic and start clicking random dashboards. Lucas: Exactly, and that's exactly the wrong move. Panic leads to tunnel vision. The best engineers I've seen — at Google, at Meta — they take a deep breath and follow a playbook. Literally, they open the runbook for that service. Luna: Runbooks — those step-by-step guides that everyone says they'll write but never do. Lucas: Right. But at FAANG companies, runbooks are mandatory for any service that goes into production. They're tested in drills. So the first thing you do is open the runbook. It tells you the most common failure modes and how to check them. Luna: OK, but what if the runbook doesn't cover this specific issue? You're in uncharted territory. Lucas: Then you go to the next step: isolate the blast radius. You need to figure out what's affected. Is it one user? One region? One API endpoint? You look at the monitoring dashboard — error rates, latency, traffic volume. If you have distributed tracing, you follow a single failing request through the system. Luna: Distributed tracing is huge. At my last company, we used something like Jaeger. It saved us hours once when a microservice was silently dropping database connections. Lucas: Exactly. Tracing shows you where the time is spent and where the error originates. So you identify the service that's failing. Then you check its recent deploy history — sometimes the issue is a new code change that was just rolled out. If there's a recent deploy, you can often roll it back quickly. Luna: Rollback is the safest move if you're unsure. You can always investigate later. Lucas: Absolutely. The priority is to restore service, not to find the root cause immediately. That's the first rule of incident response: stop the bleeding. Once you've rolled back or mitigated, then you do the postmortem. Luna: Postmortems at FAANG are famously blameless. You're supposed to focus on the system, not the person. Lucas: Right. A good postmortem uses the 'five whys' technique. You keep asking 'why' until you get to a systemic cause. For example: Why did the service fail? Because the database connection pool exhausted. Why did it exhaust? Because the query timeout was set too high. Why was the timeout set too high? Because the default configuration was copied from a different service. Why was it copied? Because there was no review for configuration changes. So the root cause isn't 'an engineer set the wrong timeout' — it's 'the configuration change process didn't require review.' Luna: That's a much more actionable takeaway. You can then add a configuration review step. Lucas: Exactly. And that's what separates great engineering orgs from average ones. They treat every incident as an opportunity to improve the system. They write action items and track them to closure. Luna: I want to go back to the moment of panic. What's a concrete technique to calm down and focus? Lucas: A senior engineer once told me: 'Read the error message. Actually read it.' Most junior engineers see an error and immediately start guessing. But the error message often tells you exactly what's wrong — a null pointer, a timeout, a missing file. So force yourself to read it out loud. Luna: That's surprisingly effective. I've caught myself skipping over error messages because I assumed they'd be cryptic, but half the time they're not. Lucas: Another trick: look at the logs from the time the incident started. If you have structured logging with correlation IDs, you can trace a single user's request across services. That's gold. Luna: But not everyone has that level of observability, especially at smaller companies. Lucas: True. So let's talk about what you can do without fancy tools. The most universal technique is the binary search method: you look at the most recent changes, you isolate variables, you test one thing at a time. You don't change five things and hope. Luna: Change one thing, measure, then change another. It's basic scientific method, but under pressure we forget it. Lucas: Exactly. And here's a pro tip: when you're stuck, explain the problem to a rubber duck. Or to a colleague. Verbalizing the issue often makes the answer pop out. Luna: Rubber duck debugging — classic. I've had countless times where describing the bug to someone else triggered the 'aha' moment. Lucas: And that colleague doesn't even need to be familiar with your system. Just the act of structuring the problem helps. Luna: Speaking of help, a lot of listeners might be wondering: when do you escalate? Is there a rule? Lucas: There's a general guideline: if you haven't made progress in 15 minutes, escalate. Ping your team's on-call lead or a senior engineer. Don't wait an hour. The cost of delay is huge. Luna: Yeah, the pressure to solve it yourself can be intense. But escalating early is actually a sign of maturity. Lucas: Totally. Now, I want to talk about one more thing: how to debug when you can't reproduce the issue locally. This happens all the time in distributed systems. Luna: That's the worst. It works on my machine. Lucas: Right. So you need to gather as much information from production as possible. Enable debug logging on a single instance, or use feature flags to selectively enable tracing for a small percentage of users. Canary deployments help here. Luna: Canaries are great — you route a tiny fraction of traffic to a new version and monitor for errors before rolling out to everyone. Lucas: Exactly. And if you suspect a race condition or a timing issue, you can use chaos engineering — deliberately inject failures to see how the system behaves. But that's more of a proactive measure. Luna: Let's talk about proactive measures, because the best debugging is the one you never have to do. Lucas: Absolutely. Good monitoring, alerting, and runbooks prevent most incidents from becoming crises. Also, conduct regular incident drills — game days where you simulate a production outage. They train your muscle memory. Luna: And doing blameless postmortems after every incident, even minor ones, creates a culture of learning. Lucas: Right. So to summarize: when the pager goes off, take a breath, read the error, check the runbook, isolate the blast radius, look at recent changes, roll back if possible, and escalate early. Then after it's resolved, write a thorough postmortem. Luna: That's a solid framework. I know I'll be using it next time my phone buzzes. Lucas: And if today's tech conversation gave you something usable, you know what keeps these shows ad-free and independent — it's the listeners who chip in a couple of dollars a month. Buy me a coffee dot com slash fexingo. Seriously, that small support makes a real difference. Luna: Yeah, it's literally how we keep the episodes coming without any sponsor interruptions. So if you're enjoying the show, consider it. Lucas: Alright, back to debugging — one last tip: always update the runbook after the incident. Future you will thank you. Luna: Future me is always grateful. Great episode, Lucas.