Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / Inside the Migration That Cut Uber's Data Costs in Half
Transcript
- Lucas: So Uber runs one of the largest real-time data platforms on the planet. Every day they ingest something like a trillion events — driver locations, trip statuses, surge pricing signals. And for years, that system was built on a homegrown streaming layer they called the Uber Data Platform, which served analytics for dispatch, ETA prediction, and fraud detection. Luna: A trillion events a day. That's the kind of scale that makes most CTOs just say no, right? Lucas: Exactly. And the problem is, scale has a price. By 2024, the infrastructure behind that platform was consuming a massive chunk of Uber's cloud bill. We're talking hundreds of millions of dollars annually. So they launched a two-year initiative to re-architect the whole thing, and the headline result is that they cut data storage and compute costs by fifty percent. Luna: Fifty percent is huge. How do you even start a migration like that without breaking the ride-hailing app? Lucas: Well, the first thing they did was admit that their old architecture, which was built around a lambda-style model, had become too monolithic. Every team that needed real-time data got a copy of the stream, and storage was duplicated across multiple systems. So you had the same event sitting in Kafka, in a key-value store, and in a search index. That duplication was one of the biggest cost drivers. Luna: So they had to stop copying everything everywhere. But real-time systems need low latency. How do you square that with centralizing storage? Lucas: That's where the clever part kicks in. They moved to a tiered storage model. Hot data — which is anything needed within the last few minutes for live dispatch — stays in memory. Warm data, say up to a few hours old, sits on SSDs. And cold data, which is only accessed for offline analytics, gets pushed to cheaper object storage, like S3. The key is that they didn't force every query to hit the hot tier. Each workload gets routed to the appropriate tier. Luna: So it's a bit like a cache hierarchy in a CPU, but for data pipelines. Lucas: Precisely. And to make that work, they adopted Apache Pinot as their core real-time analytics engine, replacing several in-house systems. Pinot is built for exactly this kind of scale — it's designed to serve sub-second queries on trillions of rows, and it natively supports that tiered storage. So they didn't have to bolt on some custom cache layer. Luna: I've heard of Pinot, but I thought Uber had its own engine, called Speedy. What happened to that? Lucas: Speedy is still there, but it got a new role. Instead of being the general-purpose query engine, it's now a lightweight, specialized engine for serving pre-computed features — like a driver's current location or a rider's preferred surge factor. That's a much narrower workload, so Speedy can be optimized for high throughput and low latency on those reads, while Pinot handles the ad-hoc analytical queries. Luna: So they essentially split the compute from the storage. Disaggregation, right? Lucas: Exactly. They decoupled the query engines from the underlying storage layer. In the old world, each engine had its own copy of the data. Now, they share a common storage tier, and each engine just reads what it needs. That alone cut storage costs by around forty percent, because they eliminated all that duplication. Luna: Forty percent just from deduplication. What about the other ten percent? Lucas: The rest came from compression and lifecycle management. They introduced columnar compression formats, and they set up automated policies to move data to colder tiers as it aged. For example, raw trip events are only kept hot for a few minutes; after that, they're rolled up into metrics. That reduced the total data volume they had to store. Luna: But this is a massive migration. How did they roll it out without causing a fleet-wide outage? Lucas: They ran both systems in parallel for almost a year. Every new query engine was tested against the old system, comparing results and latency. They only switched over traffic when the new system matched or beat the old one. That's a huge investment in time and engineering, but it's the only way to de-risk a change of this magnitude. Luna: A year of parallel running sounds expensive. Did they have to hire a bunch of extra engineers to maintain both? Lucas: They actually did the opposite. The team shrank slightly during the migration, because they weren't maintaining the old system as actively. The trick was to treat the old system as frozen — no new features, only critical bug fixes. That way, the team could focus on building the new platform. Luna: Freezing the old system is a bold move. What if something critical broke and they needed to add a feature quickly? Lucas: There was some risk, but they had a clear migration plan with milestones. Each team was given a deadline to move their queries over, and the old system was fully decommissioned after the last team cut over. That pressure forced the org to prioritize the migration over incremental improvements to the legacy stack. Luna: I imagine there was a lot of internal resistance. Engineers love their own systems. Lucas: Oh, definitely. That's probably the hardest part. You have teams who've built their own analytics pipelines over ten years, and now you're telling them to throw that away for an open-source engine. To convince them, the platform team did a series of bake-offs — running both systems side by side on real workloads and measuring p99 latency, throughput, and cost per query. Once teams saw the data, most came around. Luna: Data beats opinion, as they say. But what about the cultural shift? Did they have to change how engineers worked? Lucas: Yeah, they moved to a more centralized ownership model. In the past, each business unit owned its own data pipeline. Now, the platform team provides standardized schemas and APIs. Teams don't build custom ingestion anymore; they just plug into the platform. That was a hard sell because it reduced autonomy, but it also reduced duplication and operational burden. Luna: So you trade some autonomy for efficiency. That's a classic CTO dilemma. Lucas: Right, and it's one that requires strong leadership. The CTO at the time had to make a call and stick with it. But they also gave teams a voice in the design of the new platform. Each team had a representative in the architecture reviews, so they felt some ownership over the result. Luna: And the result is half the cost. But what about performance? Did they sacrifice any query speed? Lucas: The surprising thing is that latency actually improved. Because Pinot is so well-tuned for real-time queries, the p99 latency for dispatch decisions dropped by around twenty percent. And for some categorical queries, like 'show me all active drivers in this geofence,' it's even faster. The tiered storage means that hot queries are landing in memory, so they're snappier than before. Luna: So you get lower costs and better performance. That's the dream, but it rarely happens in real systems. Lucas: It doesn't, which is why this case is so instructive. The key is that they didn't just swap one engine for another. They rethought the entire data flow, eliminating redundant systems and optimizing for the actual access patterns. That holistic approach is what made the win possible. Luna: It reminds me of that old saying: 'You can't optimize what you don't measure.' They clearly had deep visibility into where the costs were. Lucas: Absolutely. They had a cost per query metric for every single workload. That's what allowed them to identify the biggest offenders and target them first. In fact, they found that a small number of hot queries were consuming most of the resources, so they optimized those with caching and pre-aggregation. Luna: So the fifty percent savings came from a mix of architectural changes and workload-specific tuning. It wasn't one silver bullet. Lucas: Exactly. And that's a lesson that applies to any company running a data platform, not just at Uber's scale. You have to look at the whole pipeline, from ingestion to query, and find where the waste is. Sometimes it's as simple as changing your data format or adding a cache layer. Luna: Speaking of lessons, I've been thinking about how this relates to something our listeners often bring up. You know, the whole question of whether to build or buy your data stack. This migration shows that even a company with massive engineering resources chose to lean on an open-source project like Pinot, rather than build everything in-house. That's a signal for smaller teams too. Lucas: That's a great point. And it ties into something we always emphasize on this show: knowing where your core competency lies. For Uber, the core is the logistics and the real-time decisioning. They don't need to be the world's best query engine developers. So using Pinot lets them focus on what they do best, while still having a tailorable system. Luna: Right. And this kind of high-level engineering insight is exactly what keeps our listeners coming back. We get a lot of support from people who value these deep dives, and that support is what keeps us ad-free and independent. If you find these conversations useful for what you're building or running, you can help us keep going by buying us a coffee. It's buy me a coffee dot com slash fexingo. Every little bit helps, and we genuinely appreciate it. Lucas: Yeah, it's a small way to keep the lights on without selling out to advertisers. We'd rather keep doing this purely for the love of it, and listener support makes that possible. So if you're in a position to chip in, that's fantastic. If not, no worries — just keep learning with us. Luna: Absolutely. Now, back to the technical nitty-gritty. One thing I want to dig into is the operational side. Running a parallel system for a year must have had some interesting failure modes. Did they hit any nasty surprises? Lucas: Oh, plenty. The biggest one was data consistency. When you run two systems in parallel, you have to ensure that both are seeing the same data at the same time. They had a few incidents where the new system lagged behind the old one due to backpressure in the pipeline. They had to build a reconciliation job that compared the two systems' outputs and flagged any discrepancies. Luna: That sounds like a nightmare to debug. Lucas: It was. They ended up writing a tool they called 'the diff' that would run every hour, comparing counts and checksums for critical tables. If there was a mismatch, it would trigger an alert, and engineers would have to trace back through the pipeline to find where the divergence occurred. It was a huge time sink, but it built confidence in the new system. Luna: So that's the hidden cost of a safe migration — you spend a lot of engineering effort on validation and reconciliation. Lucas: Exactly. And that's why many companies rush migrations and then regret it. You can't skip the boring work of verifying that the new system is correct at every step. Uber's willingness to invest in that validation is what ultimately made the switch successful. Luna: It's also a reminder that cost optimization projects are not just about the financial numbers. They're about managing risk and maintaining trust in your systems. Lucas: Well said. So, for a CTO looking at their own data stack, what's the one actionable takeaway from this case? I'd say it's to measure the true cost of every query and every data copy. You might be surprised at how much you're spending on duplicate storage or inefficient queries. And if you see those numbers, you can start to make the case for a migration like this. Luna: And don't be afraid to bet on open source. Pinot is a great example of a community-driven tool that can handle billion-scale events without blowing up your budget. Lucas: Definitely. The days of building everything in-house are over, even for companies with Uber's resources. The smartest engineers are the ones who know when to leverage the ecosystem and when to build. That's the real lesson here. Luna: And where do you see this going next? Will Uber push further into cost optimization, or is there a new frontier? Lucas: I think the next frontier is going to be about making these platforms self-optimizing. Instead of engineers manually tuning tier storage and compression, you'll see machine learning models that predict access patterns and automatically move data to the right tier. That's where the industry is heading, and Uber is already experimenting with that. Luna: So the future is not just cheaper, but smarter. I like that. Thanks for breaking this down, Lucas. Lucas: Always a pleasure. And if this episode sparked an idea for your own infrastructure, we'd love to hear about it. Drop us a note or just keep experimenting. That's how progress happens.