Latest / The Tech Career Podcast with Fexingo: Engineering Jobs, Interviews, and FAANG Career Strategy / How FAANG Engineers Use Observability to Debug Production
Transcript
- Lucas: So you're on call for a service that handles millions of requests a minute. An alert fires — p99 latency just spiked to three seconds. Your first instinct, if you're like most engineers, is to pull up a dashboard. Green checks everywhere. CPU, memory, request rate — all normal. What do you do next? Luna: Right — the classic 'everything looks fine' dashboard. That's where most people start spinning their wheels. Lucas: Exactly. And the difference between a junior and a senior engineer in that moment often comes down to one word: observability. Not monitoring. Observability. Luna: Okay, say more. What's the distinction you're drawing? Lucas: Monitoring tells you something is wrong. Observability lets you ask why. Monitoring is — you set thresholds, you get alerts. Observability is the property of a system that lets you understand its internal state by looking at its outputs. That's the formal definition, and it comes from control theory originally. Luna: So observability means you can debug without having to push new code or add logging after the fact. Lucas: Exactly. You can interrogate the system in real time. The three pillars — metrics, logs, and traces — that's the tooling. But the real skill is knowing which pillar to use when. A lot of engineers, especially early in their careers, default to logs. They grep through terabytes of text. That's slow. Luna: I've seen that — someone SSHes into a box, tails a log file, and tries to spot the anomaly. It's like finding a needle in a haystack while the haystack is on fire. Lucas: Right. And the haystack might be replicated across fifty instances. So the senior approach is: start with metrics to narrow the time window, then use traces to find the specific request path that's slow, and only then look at logs for that one request. Luna: That's a concrete workflow. Do you have an example from your own experience? Lucas: Yeah — I worked on a team at a company that built a recommendation engine. We had a dashboard showing average latency, which looked fine. But users were complaining about slowness. Turns out average latency masked the tail. The p99 was ten seconds, but the p50 was under a hundred milliseconds. We only caught it when we added a latency histogram. Luna: So the average was hiding the bad experience for a small percentage of users. Lucas: Exactly. And that's a classic monitoring failure — the alert was on average latency. Observability would have let us slice by user segment, by region, by request type. Once we had proper distributed tracing, we found the bottleneck was a third-party API that timed out on certain payloads. Without traces, we'd have been guessing for days. Luna: I think a lot of engineers hear 'observability' and think 'we have Grafana dashboards, we're good.' But dashboards are just the output. The real question is: can you answer a novel question about your system without writing new code? Lucas: That's the litmus test. Can you ask a question you didn't anticipate? For example, 'what does the error rate look like for requests that originate from mobile clients on Android 13, going through the checkout service, between 2pm and 3pm yesterday?' If you can answer that in under five minutes, you have observability. Luna: And most teams can't. They have metrics on one tool, logs in another, and traces in a third that nobody set up properly. Lucas: Right. And that's the gap that the best engineers learn to close. At FAANG companies, observability is a first-class concern. Google's internal tooling — Monarch for metrics, Dapper for traces — and Meta uses Scuba and Canopy. But the principles transfer to any stack. OpenTelemetry is becoming the standard. Luna: Let's talk about traces specifically. I think traces are the most underused pillar. Why is that? Lucas: Because they're harder to set up. You need to instrument your code, propagate context headers across service boundaries, and store a lot of data. But once you have them, they're the most powerful. A single trace shows you the entire lifecycle of a request — every service call, every database query, every cache hit or miss. Luna: And that's invaluable for microservices. In a monolith, you can sometimes get away with just logs. But with twenty services, you need traces. Lucas: Yeah. I remember a story from a friend at Uber — they had a bug where rides were failing to match. Every team blamed someone else. The mapping service said the location service was slow. The location service said the pricing service was timing out. A distributed trace showed the culprit was actually a config service that had a contention issue only under high concurrency. Without the trace, it would have been a weeklong blame game. Luna: So observability also has a cultural dimension — it reduces finger-pointing. Lucas: Absolutely. It gives you facts. And the best teams treat observability as a product. They invest in making it easy for engineers to explore data. At Netflix, for example, they built Atlas — a telemetry platform that lets engineers query metrics with a domain-specific language. It's like a search engine for your system. Luna: But for a lot of listeners, they're not at Netflix or Google. They're at a startup with a single Python backend. Is observability still relevant? Lucas: More than ever. Because at a startup, you have fewer people and less time to debug. Setting up structured logging and basic distributed tracing with OpenTelemetry is a couple of days of work. It pays for itself the first time a production issue hits. Luna: I agree. I'd argue that even for a monolith, having proper structured logs with request IDs is a form of observability. You can trace a user's journey through the code without distributed tracing. Lucas: Exactly. Start simple. Use correlation IDs. Log in JSON. Centralise your logs. From there, you can add metrics and traces incrementally. The important thing is to be able to ask questions you didn't think of when you wrote the code. Luna: One thing I've noticed in system design interviews lately — especially at FAANG — is that observability comes up more and more. Interviewers ask, 'how would you debug this system in production?' Lucas: Yes. It's become a standard part of the system design interview. They want to see that you think about operability, not just features. So if you're preparing for interviews, make sure you can talk about the three pillars, how you'd instrument a service, and how you'd use traces to root-cause a latency issue. Luna: And don't just say 'we use Datadog.' Explain the architecture — how you'd sample traces, how you'd set up alerting on SLOs, not just averages. Lucas: Right. slo based alerting is a whole other episode, but it's related. The point is, observability is not a tool you buy. It's a capability you build. And the engineers who understand that are the ones who get promoted to senior and staff levels. Luna: It's also a skill that transfers across companies. If you know how to think in terms of observability, you can join any team and quickly understand their system. Lucas: Exactly. And that's why we keep coming back to it on this show. Speaking of which — we deliberately don't run ads on these episodes. If you find these conversations useful, you can support that choice at buy me a coffee dot com slash fexingo. Keeps it sustainable and completely listener-supported. Luna: Yeah, that's the only reason we can keep doing deep dives like this. Appreciate everyone who chips in. Lucas: So back to the practical takeaway. If you're an engineer listening, here's one thing you can do this week: add a correlation ID to every request in your application. Log it. Then next time you get a bug report, you can search for that ID and see exactly what happened. That's the first step toward observability. Luna: And it doesn't require any new infrastructure. Just a middleware that generates a UUID and passes it through. Lucas: Right. From there, you can add more. But start with that one change. It'll change how you debug. Luna: Good advice. And maybe next episode we can talk about how to set up distributed tracing in a Kubernetes environment. Lucas: I was thinking the same thing. Let's do it.