Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How DoorDash Cut Its Machine Learning Model Latency by 70 Percent
Transcript
- Lucas: So, Luna, you know that moment when you order food on an app, and it just feels snappy? The map updates, the ETA recalculates, the restaurant suggestion pops up instantly? Luna: Yeah, and I never really think about what's happening under the hood. But I imagine there's a lot of machine learning going on there. Lucas: Exactly. For a company like DoorDash, machine learning is baked into almost every step — predicting delivery times, estimating demand, matching drivers with orders. But there's a hidden cost: every model inference takes time. And in a world where users expect instant responses, latency is the enemy. Luna: So what did they actually do about it? Because I read somewhere that they managed to cut their ML latency by something like 70 percent. Lucas: That's right. And it wasn't a single silver bullet. It was a combination of three techniques: model distillation, quantization, and moving inference to the edge. Let's unpack each of those. Lucas: First, model distillation. The idea is simple: you train a big, complex model — a 'teacher' — that's very accurate but slow. Then you train a smaller, simpler model — a 'student' — to mimic the teacher's outputs. The student is faster because it has fewer parameters. DoorDash used this to shrink some of their models significantly without losing much accuracy. Luna: So you're trading a bit of accuracy for a lot of speed. But how much accuracy are we talking about? Because if it's too much, that's a problem. Lucas: Good question. In their case, the accuracy drop was minimal — often less than one percent. For use cases like ETA prediction, that's acceptable. And when you're running millions of predictions a day, the latency savings add up quickly. Luna: Okay, so distillation shrinks the model. What's quantization? Lucas: Quantization is about reducing the precision of the numbers the model uses. Instead of 32-bit floating point numbers, you go to 8-bit integers. That cuts the model size and speeds up computation because the hardware can handle integers faster. And on some hardware, like mobile chips, it's even more efficient. Lucas: But quantization isn't free. You can lose accuracy if you're not careful. DoorDash used something called quantization-aware training, which simulates the lower precision during training so the model learns to compensate. That way, they got the speed boost with minimal accuracy loss. Luna: So distillation and quantization are both about making the model smaller and faster. But you also mentioned edge deployment. What's that about? Lucas: Edge deployment means running the model on the device itself — the driver's phone, the customer's phone — instead of sending data to a cloud server and waiting for a response. That eliminates network round trips, which can be tens of milliseconds or more, especially on a weak cellular connection. Lucas: For DoorDash, that meant taking some of their models, like the ones that predict delivery times based on real-time traffic, and running them locally on the driver's phone. They even had a clever way to update those models in the background without user intervention. Luna: That's fascinating. But that must have been a huge engineering effort. How did they get the models onto millions of devices? Lucas: It required a whole platform team to build the infrastructure. They had to create a model distribution system that could push updates to devices efficiently, and they had to ensure that the models ran consistently across a huge variety of Android and iOS devices. It wasn't trivial. Luna: I bet. And what about accuracy? Running on the edge means the model doesn't have access to all the cloud data, right? Lucas: Right. So they had to be selective about which models went to the edge. They kept the most time-sensitive ones there, like the ETA recalculations, and left the ones that need lots of data, like demand forecasting, in the cloud. It was a hybrid approach. Lucas: And they measured everything end to end. They didn't just look at the model inference time in isolation. They tracked the full latency from when a user tapped a button to when the response appeared on the screen. That's what allowed them to see the real impact of these changes. Luna: So what was the actual number? I know you said a 70 percent cut, but what did that mean in real terms? Lucas: For some of their services, they went from around 200 milliseconds of model inference time down to around 60 milliseconds. And because some of those models were being called on every app open, the cumulative savings were massive. It directly improved app responsiveness and user satisfaction. Luna: That's impressive. But there must have been a lot of resistance internally. I mean, ML engineers love their big, accurate models. Taking a 10 percent accuracy hit might have been a tough sell. Lucas: That's the crux of it. The technical solutions were only part of the story. The other part was changing the culture. They had to convince ML engineers that latency was a feature, not just a nice to have. They started by setting latency budgets for each service, just like you'd set a memory or CPU budget. Lucas: And they created a cross-functional team that brought together ML engineers, platform engineers, and mobile developers. That broke down silos. Suddenly, ML engineers understood the constraints of mobile devices, and mobile developers understood what the models needed. Luna: That's a classic org design problem. You need the right incentives and the right teams to make that kind of optimization happen. Lucas: Absolutely. And it's a lesson that applies beyond DoorDash. Look, if you're running any kind of ml driven product, latency should be a first-class consideration from day one. It's easy to ignore it when you're prototyping in a notebook, but in production, it can make or break your user experience. Luna: And it's not just about speed. Lower latency can also reduce costs, because you're using less compute. Did DoorDash see any cost savings? Lucas: They did. By moving inference to the edge and using smaller models, they cut their cloud compute costs significantly. They didn't give exact numbers, but it was a meaningful reduction. So it was a win-win: better performance and lower costs. Luna: That's the kind of story that any CTO would love to hear. And it ties into something we've talked about on this show before: the importance of measuring and optimizing your entire system, not just individual parts. Lucas: Exactly. And if you're in a role where you're making decisions about how to allocate engineering resources, this is a great example of where to look for wins. It's not always about adding more servers or more GPUs. Sometimes it's about making your existing models leaner and smarter. Luna: Yeah. And that's the kind of thinking that separates good engineering organizations from great ones. Lucas: Before we wrap up, I want to mention something that's close to our hearts here at the podcast. Listen, you've been listening to us talk about optimizing systems and making trade-offs. And we try to bring you these stories every week, completely ad-free. That's only possible because of a few listeners who choose to support us on buy me a coffee dot com slash fexingo. Luna: Yeah, it's a simple way to keep this show going without any corporate sponsors. If you've gotten value from these episodes, a small contribution goes a long way. Lucas: But no pressure. We're just happy to have you here. Now, back to DoorDash — there was one more interesting thing they did that we haven't touched on yet. Luna: Oh, what's that? Lucas: They invested in custom hardware for some of their cloud-based models. They used something called an inference accelerator, which is a specialized chip designed specifically to run ML models faster and more efficiently. That gave them another boost for the models that had to stay in the cloud. Luna: Interesting. So they combined software and hardware optimization. That's a comprehensive approach. Lucas: Right. And the key takeaway for me is that there's no one-size-fits-all solution. You have to look at your specific use cases, your infrastructure, and your user expectations. DoorDash's approach worked because they were methodical about measuring and iterating. Luna: So what would be the first step for a CTO who wants to start optimizing ML latency in their own org? Lucas: I'd say start by measuring. Instrument your models in production and track end to end latency. Understand where the time is going — is it the model itself, the network, or the post-processing? Then set a latency budget and work backwards from there. And don't be afraid to question your assumptions about accuracy. Sometimes a slightly less accurate model that responds in 50 milliseconds is better than a perfect one that takes 300. Luna: That's a great note to end on. Thanks for joining us on The CTO Podcast. Until next time, keep building. Lucas: Yeah, thanks for listening. We'll see you in the next episode.