Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How DoorDash Rebuilt Its Logistics Engine for 25 Million Daily Orders
Transcript
- Lucas: So DoorDash is now processing about 25 million orders daily across more than 7,000 cities — up from maybe 8 million a day just two years ago. And that kind of growth put their original dispatch architecture under serious strain. Luna: Yeah, I remember hearing about outages during peak lunch hours in 2024. The dispatch engine was essentially a monolith, right? Lucas: Exactly. They had a centralized dispatch service that handled everything — matching orders to dashers, routing, eta calculations. It worked fine at 8 million orders, but as they scaled, latency started creeping up, and any regional spike could affect the whole system. Lucas: The straw that broke the camel's back was a particularly bad Super Bowl Sunday in 2025. They had a record 35 million orders, and the dispatch service hit a wall. Some markets saw order acceptance delays of over 20 seconds. That's an eternity in food delivery. Luna: Twenty seconds is basically cold fries. So what did they do? Lucas: If these conversations are useful for what you're building or running, honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Keeps the show ad-free and independent. Luna: Yeah, we love making this, and listener support is what keeps it going. Back to the rebuild. Lucas: So they embarked on what they internally called 'Project Mercury' — a complete rewrite of the dispatch engine into an event-driven, sharded architecture. The core idea was to split dispatch logic per geographic market, each with its own processing pipeline. Luna: That makes sense — if a market in Chicago has a snowstorm, it shouldn't affect dispatch in Miami. Lucas: Right. They used S2 cells from Google's geomapping library to partition the world into hierarchical grids. Each market is a set of S2 cells at a certain level. Then they assigned a dedicated Kafka topic and consumer group per market for order lifecycle events. Lucas: The key was that each market's dispatch logic runs in its own Kubernetes pod, with autoscaling based on order volume in that specific region. So a market like New York City might have 20 pods during lunch, while a smaller market has one. Luna: So they essentially turned the dispatch monolith into a fleet of tiny dispatch services. What about consistency? If a dasher crosses market boundaries mid-order, does that cause issues? Lucas: Great question. They introduced a 'market handoff' protocol. When a dasher crosses a boundary, the origin market completes its final state and sends a handoff event to the destination market's Kafka topic. The destination market then picks up the order from that state. They guarantee at most once delivery for handoffs, because losing a handoff is better than duplicate dispatching. Luna: Interesting — so they prioritized availability and latency over strict consistency in that edge case. That's a classic tradeoff. Lucas: Exactly. They also added speculative dispatching. Instead of waiting for one dasher to accept, the system sends an order to the top three candidate dashers simultaneously. The first to accept gets the order, and the others get a cancellation signal. Luna: That must reduce wait times, but doesn't it risk overloading dashers with too many pings? Lucas: They were careful about that. They set a maximum of two speculative dispatches per order, and they track dasher acceptance rates. If a dasher declines too often, their candidate score drops. The result? Average delivery wait time dropped 12 percent in initial rollouts, and dasher satisfaction actually improved because they get more relevant offers. Luna: Twelve percent is significant. Did they have to change anything about their data layer to support this? Lucas: Yes — they migrated from Postgres to CockroachDB for the dispatch database. They needed a distributed SQL database that could handle cross-market reads and writes with low latency. CockroachDB's automatic sharding and multi-region support fit well with their market per pod model. Lucas: They also implemented a 'last writer wins' conflict resolution for dasher state updates, because in a speculative dispatch scenario, you might get concurrent updates to a dasher's availability. LWW is simple and fast. Luna: So they moved from strong consistency to eventual consistency in some parts. That's a big shift for a logistics system. Lucas: It was a deliberate tradeoff. They found that for most operations, the probability of conflict was low, and the cost of a conflict was just a slightly delayed dispatch — not a lost order. They monitored conflict rates and they stayed below 0.1 percent. Luna: That's pretty low. How did they test this at scale before going live? Lucas: They built a simulation framework called 'Mercury Sim' that replays historical order data against the new architecture. They could simulate entire regions, inject failures, and measure latencies. The simulation ran on a separate Kubernetes cluster and could process a week's worth of orders in a few hours. Lucas: They also did a series of 'chaos engineering' days where they randomly killed pods in a live shadow environment to see how the system recovered. That caught a few subtle bugs in the handoff protocol. Luna: So they were able to validate the architecture before touching production. How long did the whole project take? Lucas: From start to full rollout — about 14 months. They started in March 2025, had a shadow mode running by September, and fully cut over by May 2026. That's pretty fast for a rewrite of this scale. Luna: What was the trickiest part in your view? Lucas: I think it was the geo-partitioning. Deciding the right S2 cell size for each market was surprisingly complex. Too fine-grained and you get too many partitions, too coarse and you lose the benefits of isolation. They ended up using level 12 S2 cells, which cover about 100 square kilometers each, but they allow dynamic merging of cells during low-traffic periods to save resources. Luna: So the system adapts not just by scaling pods, but by reshaping the partition boundaries based on demand. Lucas: Exactly. They call it 'adaptive geo-sharding.' The partition manager — a separate service — monitors order volume per cell and merges or splits cells accordingly. This happens every 5 minutes, so the system is constantly reconfiguring itself. Luna: That's elegant. What's next for the team? Lucas: They're looking at adding predictive dispatch — using machine learning to pre-position dashers in areas where orders are likely to come in, based on historical patterns and real-time events like concerts or sports games. They've been testing it in a few markets and seeing another 8 percent reduction in wait times. Luna: So the architecture they just built is already enabling new features. That's the sign of a good rebuild — it unlocks the next layer. Lucas: Totally. And it's a lesson in knowing when to break up a monolith, even if it introduces some complexity. For DoorDash, the cost of complexity was worth the gain in reliability and speed. Luna: Yeah, cold fries are a powerful motivator.