Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / Uber Rebuilt Its Dispatch Engine for 30 Million Daily Rides
Transcript
- Lucas: If these conversations are useful for what you're building or running, you might already know the show runs ad-free — more on that in a minute. Lucas: Let's talk about Uber's dispatch engine. Not the last-mile routing or the surge pricing — the core problem of matching a rider with a driver in under 200 milliseconds, 30 million times a day. Luna: That's essentially a real-time auction happening on every street corner, right? But the constraints are brutal — you can't batch and wait, you have to decide almost instantly. Lucas: Exactly. And the old architecture did that well enough for maybe 5 million rides. But as Uber grew into Asia and Latin America, the system started hitting hard limits. The matching service was effectively a monolithic state machine — it held all active ride requests and driver positions in memory on a single cluster of servers. Luna: So if that cluster had a hiccup, you lose a city's worth of ride matching. I remember that caused a few high-profile outages. Lucas: Right. So the team knew they had to go fully distributed. But here's the interesting design choice: they didn't go with a standard consistent hash ring for partitioning drivers and riders. They used geohashes — dividing the globe into a grid of rectangles, then assigning each rectangle to a set of dispatch workers. Luna: That's elegant in theory, because a rider and driver in the same grid cell are naturally colocated. But what happens at the boundaries? If you're standing on the edge of a geohash cell, the driver one block over might be in a different cell. Lucas: That's exactly the problem they had to solve. Their initial design had a small overlap zone — they'd replicate driver state to neighboring cells. But that introduced a subtle bug during the rollout. The geohash precision level they chose was about 150 meters per cell. In dense cities like Mumbai or São Paulo, you'd have dozens of drivers in a single cell. The overlap zone created duplicate dispatch proposals — two workers would simultaneously try to match the same rider, leading to what they called 'phantom driver surges.' Luna: Meaning the rider sees a driver assigned, but the driver never gets the request? That's a terrible user experience. Lucas: Worse — the driver might get two requests for the same rider and accept both. So the team had to introduce a two-phase commit between the dispatch workers. Not a full Paxos or Raft — they built a lightweight consensus layer that used a lease-based lock on each ride request. The worker that obtained the lease got to propose a match; other workers would reject duplicates. Luna: So they traded a bit of latency for consistency. How much of a latency hit did they take? Lucas: About 20 milliseconds on average. But that was acceptable because the probabilistic matching algorithm they used actually reduced the number of candidate comparisons. Instead of evaluating every driver in the cell, they sampled a subset based on driver availability and historical acceptance rates. The net effect: matching latency stayed under 200 milliseconds, but the system became horizontally scalable and much more resilient. Luna: It's interesting that they kept the matching algorithm in-house rather than using something like a geospatial database or a prebuilt solution. Lucas: Right. They evaluated a few — including PostGIS and some commercial route-optimization engines. But they found that the customization around driver incentives and expected trip value made a generic solution too inflexible. Plus, the latency requirements were tighter than most off-the-shelf geospatial indexes could guarantee. Luna: That's a recurring pattern in these rebuild stories: when your core competitive advantage is a real-time decision, you probably have to own the algorithm. Lucas: And you have to own the observability of that algorithm. One thing the Uber team emphasized in their postmortem was that they caught the phantom driver surge bug not through load testing, but through a custom dashboard that tracked 'match proposals per ride request' — a metric they added specifically for the rollout. They saw the spike in duplicates before any customer complaints came in. Luna: That's a good reminder that the right observability signal is often a business metric, not just CPU or memory. They were monitoring system behavior that mapped directly to user experience. Lucas: Exactly. If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. It's how we keep the conversations going without sponsors. Luna: Yeah, and it really does make a difference. We hear from listeners who say these deep dives change how they think about their own systems. Lucas: Alright — back to Uber. The other big decision was around state management. In the old monolithic dispatch, the in-memory state was implicitly consistent because it was all in one process. In the distributed version, each worker maintained local state for its assigned geohash cells. But drivers move between cells constantly. Luna: So you need a handoff protocol for when a driver crosses a geohash boundary. Lucas: Right. They built what they called a 'state transfer channel' — essentially a lightweight gRPC stream between workers. When a driver's GPS update places them in a new cell, the old worker sends a snapshot of the driver's current state to the new worker. The handoff had to complete within 50 milliseconds to avoid stale data. Luna: That sounds like a microcosm of the whole system design: strict latency budgets on every operation, and a lot of attention to the failure modes of those operations. Lucas: Exactly. And they tested those failure modes extensively. One of the more creative techniques they used was what they called 'chaos dispatch' — a testing framework that randomly dropped state transfer messages and injected latency on the gRPC streams. They found that under high driver density, a single dropped handoff could cascade into a 'black hole' cell where no drivers appeared available, because the system thought they were still in the previous cell. Luna: So they had to implement a heartbeat mechanism — if a worker didn't receive a state transfer confirmation, it would fall back to a global directory lookup. Slower, but correct. Lucas: Correct. That fallback was the safety net. And it's a great example of a principle that comes up a lot in these rebuilds: optimize for the common path, but have a correct slow path that prevents data loss or inconsistency. Luna: The system is now handling over 30 million daily rides. What's the most surprising thing about how it's performed in production? Lucas: Probably the fact that the geohash grid itself became a bottleneck — not computationally, but operationally. The team had to dynamically adjust the precision level based on city density. In Manhattan during rush hour, they use a finer grid to spread the load across more workers. In rural areas, they coarsen it. That required a control plane that could rebalance cells without disrupting active rides. Luna: So the architecture is self-tuning to some extent. That's a level of sophistication that took years to develop. Lucas: Exactly. And it's a reminder that even a well-designed distributed system needs ongoing investment in the tooling around it. The dispatch engine itself is maybe 30 percent of the code; the monitoring, testing, and operational tooling is the rest. Luna: That's a good note to end on: the real work is in the systems that keep the system running. Lucas: Absolutely. Next episode, we'll look at how a major bank rebuilt its core transaction engine — a very different set of constraints, but some surprising parallels.