Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Uber Rebuilt Its Dispatch Engine for 30 Million Trips
Transcript
- Lucas: So, Uber's dispatch engine — the system that decides which driver gets which rider — was originally built for about 10 million trips a day. By 2023, they were doing triple that, and the architecture was showing real strain. Luna: I remember reading that matching times were creeping up. Was it a hardware bottleneck or a software one? Lucas: Almost entirely software. The original system was a monolithic matching service running on a single cluster of servers per city. Each city had its own instance, but within a city, all the logic was in one process — stateful, with in-memory queues for available drivers and pending requests. Lucas: As trip volume grew, the per-city monolith started hitting concurrency limits. The matching loop — essentially a continuous scan of available drivers against incoming requests — would take longer and longer. Average match time went from around 2 seconds to over 4 seconds during peak hours. Luna: Four seconds doesn't sound terrible on its own, but in a real-time marketplace, that's an eternity. Riders start cancelling, drivers get frustrated. Lucas: Exactly. And the problem wasn't just latency — it was also throughput. The monolith could handle maybe 20,000 matches per second per city. In a city like Jakarta during rush hour, they were pushing 35,000. So requests were queuing up on the inbound side. Luna: So they decided to rebuild. What was the new architecture? Lucas: They moved to a fully event-driven, distributed system. The core insight was to decouple the matching logic into separate services: a request intake service, a driver availability service, a geospatial index, and a matching orchestrator. All communication happens through Apache Kafka topics. Lucas: Each service is stateless and horizontally scalable. The geospatial index, for example, uses a geohash-based partitioning scheme. So the city is divided into grid cells, and each cell's driver and request data lives on a separate Kafka partition. That way, matching logic only needs to look at a small subset of data. Luna: I remember reading about that geohash approach. It also helps with load balancing — hot spots like downtown get more partitions. Lucas: Right. They dynamically adjust the geohash precision based on demand. In dense areas, the grid cells are smaller, so each partition handles fewer requests. In suburban areas, cells are larger. That kept partition sizes fairly uniform, which is crucial for Kafka throughput. Luna: What about the matching algorithm itself? Did they change the logic or just the plumbing? Lucas: They changed it significantly. The old system used a first-in-first-out queue per city — the longest-waiting rider got matched first, period. That's fair, but it's not optimal for marketplace efficiency. The new system uses a scoring function that weighs wait time, proximity, driver rating, surge pricing, and even destination predictions. Lucas: They also added a feature called 'batch matching' — instead of matching one rider at a time, the orchestrator collects a batch of pending requests and available drivers over a 500-millisecond window, then solves a local optimization problem to maximize overall match quality. Kind of like a mini auction every half second. Luna: That's a trade-off though. You're adding a fixed 500-millisecond delay for every match. But I'm guessing the overall latency still went down because the optimization is faster than the old sequential loop? Lucas: Exactly. The end to end matching time dropped from 4 seconds to under 500 milliseconds on average. The batch window is less than the variability in GPS updates anyway, so riders don't perceive the delay. Lucas: But there was a cost — they had to give up on strict FIFO. Some riders wait a few seconds longer than others if a better match is available. They track this as a fairness metric now, and it's usually within acceptable bounds. Luna: Let's talk about the migration. How do you swap out the heart of a live marketplace without breaking anything? Lucas: They used a shadow mode approach. For 18 months, every incoming request was sent to both the old system and the new system in parallel. The new system's matches were logged but not acted upon — they compared the results offline. That let them validate correctness and performance without risk. Lucas: Once they had confidence, they started rolling out city by city. They'd flip a configuration flag for a small city, run it for a week, monitor every metric, then expand. Luna: That's slow. 18 months of parallel runs plus rollout. How do you keep the org motivated through that? Lucas: That's the unglamorous part. The engineering team — about 80 people across 5 squads — had to maintain the old system and build the new one simultaneously. There were definitely moments of burnout. But they structured the work so each squad owned a clear vertical slice of functionality in the new system, and they celebrated small wins like 'first successful match in shadow' or 'latency below 1 second in a test city'. Luna: It sounds like the kind of project that tests a CTO's ability to keep focus. I imagine there was pressure to just patch the old system instead. Lucas: There was. Every quarter, someone would propose adding more memory or scaling vertically. But the CTO at the time, Thuan Pham, was adamant that the monolith would hit a hard ceiling within two years. He made the case that the rebuild was not just a performance upgrade but a platform for future features — like pooled rides, scheduled rides, and multi-modal trips. Lucas: And he was right. The new system's modular architecture made it much easier to add those features. Pooled rides, for instance, require a completely different matching logic — the system has to find two riders going in similar directions. Under the old monolith, that would have been a massive refactor. In the new system, they just added a new service that plugs into the orchestrator. Luna: That's the real ROI of a rebuild — not just speed, but extensibility. It's hard to measure that in advance though. Lucas: Exactly. And that's the kind of judgment call that defines technical leadership. You have to bet on the future shape of the product. Luna: So for a listener who might be facing a similar decision — patch or rebuild — what's the heuristic? Lucas: I'd say: look at the rate of new feature requests. If your product roadmap requires changes that fundamentally don't fit your current architecture — like adding batch matching to a FIFO system — then a rebuild is probably worth the pain. If you're just hitting capacity limits, try scaling vertically or optimizing first. Lucas: In Uber's case, they had both problems — capacity and feature fit. The old system couldn't scale cost-effectively, and it couldn't support pooled or scheduled rides. That made the rebuild a strategic necessity. Luna: And the results speak for themselves. Sub-500 millisecond matches, support for new product lines, and a platform that's handling 30 million trips a day. Lucas: Right. It's a classic example of a well-executed architecture rebuild. And it's also a reminder that the hardest part isn't the technology — it's the organizational stamina to see it through. Luna: Yeah. That's something we try to get at in these conversations — the human side of technical decisions. Lucas: Absolutely. And speaking of the human side, if these conversations are useful for what you're building or running, we want to keep them coming. We deliberately don't run ads on this show — no sponsors, no interruptions. Luna: It's a choice we made from day one. If you find value in episodes like this, the simplest way to support that ad-free approach is to buy us a coffee. The link is buy me a coffee dot com slash fexingo. Lucas: No pressure, ever. Just a quiet way to help keep this going. Now back to Uber's dispatch — one thing I find fascinating is how they handled the state machine for each trip. Luna: Oh, the state machine? Tell me more. Lucas: Every trip goes through a defined set of states: request sent, driver assigned, driver en route, driver arrived, on trip, completed. The old system implicitly encoded this in the monolithic code — lots of if-else branches. The new system makes it explicit with a lightweight state machine service that stores each trip's current state in a key-value store. Lucas: This seems like a small detail, but it has huge implications for reliability. If a service crashes mid-match, the state machine can recover the trip's progress. It also makes auditing and debugging much easier — you can replay a trip's state transitions. Luna: That's a pattern I've seen in many rebuilds — moving from implicit state to explicit state machines. It's almost always worth the upfront design effort. Lucas: Definitely. And it's one of those things that looks like overhead until you need it. Uber's dispatch now processes over 30 million trips a day, with a system that's both faster and more maintainable than what they had before. That's the kind of outcome every rebuild aims for. Luna: And it's a reminder that the best architecture decisions are often the ones that make future changes easier, not just solve today's problem.