Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Uber Rebuilt Its Microservices After an Outage Cascade
Transcript
- Lucas: A couple of weeks ago, Uber went down. Not just the ride-hailing app — Uber Eats too. Three continents, over an hour of total blackout. And the interesting part isn't that it happened. It's what they did afterwards. Luna: I remember that morning. A lot of us in engineering were watching the status page refresh. The postmortem came out pretty fast, right? Lucas: Fast by big-company standards, yeah. Six days. And it's unusually candid. No 'we identified a configuration change' hand-waving. They named the specific service, the specific line of code, and the specific architectural pattern that failed. Luna: Which service was it? Lucas: Internally they call it the 'super-router' — a single service that decides which backend cluster should handle any given request. Ride requests, food orders, freight dispatch. It sits at the top of the call graph. And for years, everyone knew it was a single point of failure. They just didn't think it would actually, you know, fail. Luna: Right, the classic 'we'll fix it next quarter.' So what actually happened? Was it a traffic spike? Lucas: Partly. A regional data center had a network degradation — a slow leak, not a hard crash. The super-router started seeing increased latency from that region, so it began retrying. But the retry logic was written to retry aggressively, and it didn't have a circuit breaker. Within about ninety seconds, the super-router was queuing so many retries that it ran out of memory on its pods. Luna: So one region's slow network took down the entire global routing plane? That's a textbook cascading failure. Lucas: Exactly. And once the super-router pods started dying, Kubernetes tried to reschedule them. But the new pods immediately got hit with the same backlog, so they crashed again. Kubernetes just kept restarting, and the system was effectively in a crash loop. Meanwhile, all the downstream services — the ones that actually do the work — were healthy. They just couldn't receive any traffic because the router wasn't there. Luna: I think the thing that really gets me is that Uber has thousands of microservices. They've been a poster child for that architecture. And yet they still had one service that everyone depended on. Lucas: That's the hard lesson, right? You can decompose your domain logic into a thousand tiny services, but if you centralize the control plane — the routing decisions, the authentication, the feature flags — you've effectively rebuilt a monolith. It's just a monolith made of network calls. Luna: So what did they change? The postmortem mentioned something called 'delegated authority.' I'd love to unpack that. Lucas: Sure. The core idea is that instead of one super-router making every routing decision, each regional cluster gets a local 'routing authority' that can make independent decisions within its region. Think of it like air-traffic control. You don't have one global tower routing every plane on earth. You have regional towers that handle their own airspace, and they coordinate at the boundaries. Luna: So the super-router becomes a set of local routers, each with a bounded context. Lucas: Exactly. Each region has a set of pods that know about that region's services. If a region goes slow, only that region's routing degrades. The other regions keep working. And crucially, they added a 'capacity handshake' — before a local router accepts a request, it checks whether it has capacity to actually forward it. If not, it rejects it immediately with a clear error, rather than queuing it. Luna: That's the opposite of what happened before. Before, the super-router would just assume it could handle everything until it couldn't. Lucas: Right. And they also changed the retry logic. Instead of retrying three times with no backoff, they now use exponential backoff with jitter, and they cap the total retry time at five seconds. If you can't get through in five seconds, the request fails. That's much better than letting a request retry for thirty seconds and consume resources the whole time. Luna: I want to zoom out for a second. Because this isn't really about Uber. It's about a pattern that exists in a lot of orgs. Has anyone else published a similar postmortem? Lucas: Actually yes. Google's 2023 outage had a similar dynamic — a single global load balancer configuration that propagated a bad rule. And Amazon had one in 2021 where a 'super-controller' for Kinesis caused a multi-region failure. Uber's postmortem is just more detailed than most. They published the full timeline, the memory graphs, even the exact lines of code that were problematic. Luna: That level of transparency is rare. Especially for a public company. Did they get any pushback? Lucas: Some engineers on Hacker News argued it was too much information — that competitors could learn about their infrastructure. But the CTO pushed back publicly and said that security through obscurity isn't a strategy, and that the engineering community benefits more from transparency. I think that's the right call. Luna: Agreed. And it builds trust with customers too. If I were an Uber driver, I'd rather know they fixed the root cause than have them just say 'we've resolved the issue.' Lucas: Exactly. Which brings me to something I've been thinking about. A lot of what makes these conversations possible — deep dives into engineering failures, candid postmortems — it's the kind of work that doesn't get funded by ads or sponsors. It's supported directly by people who find them useful. If these deep dives are helpful for what you're building, a couple of dollars a month genuinely keeps them going. You can find the show at buy me a coffee dot com slash fexingo. Luna: Yeah, it's a small thing that adds up. And it keeps the show ad-free, which I think makes a real difference in how detailed we can get. Lucas: Alright, back to the architecture. So they rebuilt the super-router into what they call 'regional route authorities.' But there's a second change I think is even more interesting. They introduced what's called a 'bulkhead pattern' for different request types. Luna: A bulkhead — like on a ship? Separate compartments so a leak in one doesn't sink the whole vessel. Lucas: Exactly. Previously, ride requests and food orders and freight dispatch all shared the same thread pool in the super-router. So a surge in food orders — say, dinner time — could starve ride requests of threads. Now they have dedicated thread pools for each product line, with a maximum concurrency per pool. If Eats gets slammed, rides don't even notice. Luna: That seems like a no-brainer in hindsight. Why wasn't it done from the start? Lucas: Speed of development. When you're scaling fast, you don't want to over-engineer. You build a single router that handles everything, and it works fine until it doesn't. The tragedy is that by the time it fails, the cost of refactoring is enormous because the system has grown around the centralized design. Luna: So what's the lesson for CTOs listening? How do you prevent this if you're not at Uber scale? Lucas: I think there are two things. First, identify your 'super-services' early. Every org has at least one service that everything else depends on, even if it's not explicitly a router. It might be an authentication service, a configuration service, a feature-flag service. If that service goes down, does the whole system go down? If yes, you need to design for its failure, not just its success. Luna: So chaos engineering, essentially. Break the super-service in a test environment and see what happens. Lucas: Exactly. Netflix's Chaos Monkey is famous for a reason. But the second lesson is about organizational humility. Uber's CTO said in the postmortem that they had known about the super-router's fragility for years. They had a ticket open to refactor it. But it kept getting deprioritized because there were always more pressing features. The lesson is: if something is a known architectural debt, treat it like a security vulnerability. Assign it a severity score and a deadline. Luna: That's a strong framing. I think a lot of engineering leaders avoid that because it feels like admitting a mistake. But the cost of not fixing it is much higher. Lucas: Right. And the beauty of Uber's postmortem is that they don't sugarcoat it. They say: we knew, we didn't act, here's what we changed. That kind of honesty is rare and valuable. I hope more companies follow their lead. Luna: So if someone wants to read the full postmortem, is it public? Lucas: It is. Uber published it on their engineering blog under the title 'Cascading Failure and Recovery: A Postmortem.' I'll link it in the show notes. It's about a fifteen-minute read, and it's worth every second if you run a distributed system. Luna: I'll definitely check it out. And I think the real takeaway is that no architecture is ever 'done.' You have to keep probing for single points of failure, even if they've been there for years. Lucas: That's it exactly. The systems that survive are the ones that are constantly questioned. Thanks for listening, and we'll be back next week with another deep dive.