Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Walmart Rebuilt Its Supply Chain with Real-Time Data
Transcript
- Lucas: Walmart moves about five billion units through its supply chain every year. That's roughly thirteen million items every single day across four thousand seven hundred stores. Luna: And each one of those items has to be in the right place at the right time, or the whole system breaks. Lucas: Exactly. And for decades, Walmart ran that system on batch processing. Every night, mainframes would crunch the day's sales data and decide what to reorder. But a few years ago, they realized that batch is blind. You can't see a shelf go empty in real time. Luna: So they tore it down and rebuilt it on streaming data. That's what we're talking about today. Lucas: Right. The project started around 2019, and the core insight was simple: inventory is a time-series problem. The question isn't just how many units you have, but how fast they're moving right now. So they went all-in on Apache Kafka as the backbone. Luna: Kafka is the standard for event streaming, but Walmart's scale is different. What kind of throughput are we talking? Lucas: They process over a trillion events per day across their data platform. That includes point of sale scans, truck GPS pings, warehouse robot movements, even weather data. Every event gets published to Kafka topics and consumed in near real time. Luna: So instead of waiting for the nightly batch, they can see a pallet leave a distribution center and start routing a replacement within minutes. Lucas: Exactly. And the key architectural decision was to use Apache Beam for the processing layer. Beam lets them write one pipeline that runs both batch and streaming, so they can reprocess historical data with the same code that handles live events. Luna: That's smart. But moving from mainframes to a cloud-native stack—what did that migration actually look like? Lucas: They chose Google Cloud as the primary provider. The migration was phased by region. They would replicate the mainframe data into a Kafka topic, run the new pipeline in parallel, and compare outputs. If the new system matched the old one for a few weeks, they'd cut over. Luna: So it was a slow, careful swap. Not a big bang. Lucas: That's the only way to do it when your system is the backbone of a half trillion dollar retailer. One of the most interesting pieces is the demand forecasting model they call 'Eddie.' It's a machine learning system that predicts store-level demand by the hour. Luna: Eddie. Sounds like a person you'd trust with your grocery list. Lucas: Exactly. Eddie combines historical sales, local events, weather forecasts, and even social media trends. If a snowstorm is predicted for Bentonville, Eddie increases the order for shovels and bread three days in advance. Luna: And that reduces stockouts. Do we know by how much? Lucas: Walmart reported a sixteen percent reduction in out-of-stocks in pilot stores. That's huge. For a retailer where every one percent improvement in inventory accuracy can mean hundreds of millions in revenue. Luna: But there's a trade-off. Streaming data is expensive. Storing and processing trillions of events adds up. Lucas: It does. So they built a tiered storage system. Hot data—events from the last 24 hours—lives in Kafka's internal storage for low-latency access. Warm data goes to a streaming data lake, and cold data goes to object storage. They also use data locality: each store processes its own events on edge devices before sending summaries to the cloud. Luna: Edge processing. That reduces the data volume going upstream. Lucas: Right. A store's local system runs lightweight ML models that flag anomalies—like a sudden spike in milk sales—and only sends those alerts to the central platform. Everything else is batched hourly. Luna: That's a hybrid architecture. Real-time when it matters, batch when it doesn't. Lucas: Exactly. And that's the lesson most companies miss. You don't need real-time for everything. You need it for the decisions that decay fast. For Walmart, that's shelf replenishment during peak hours, or rerouting trucks around a hurricane. Luna: Speaking of hurricanes—when a big storm hits, how does the system adapt? Lucas: Walmart has a dedicated event response pipeline. When a hurricane warning is issued, the system automatically rebalances inventory across nearby distribution centers. It also adjusts delivery routes in real time, and it pre-positions high-demand items like water and generators. Luna: So the same streaming infrastructure that handles daily replenishment also handles disaster response. That's elegant. Lucas: It is. And it shows how thinking of supply chain as a data platform, not just a logistics operation, unlocks new capabilities. If you're listening and this kind of deep technical thinking is useful for what you're building or running, I want to mention something quickly. Luna: Yeah, it's a good moment for it. Lucas: The reason we can have these conversations without ads or sponsors is listener support. If today helped you think through a problem, the best way to keep it going is a small contribution. Buy me a coffee dot com slash fexingo. Luna: Even a few bucks helps. And it keeps the show focused on the engineering, not the commercials. Lucas: Anyway, back to Walmart. One thing I find fascinating is that they open-sourced some of their internal tools. They released a project called 'OneStore' which is a reference architecture for retail edge computing. Luna: So they're not keeping all this secret. They want the ecosystem to adopt similar patterns. Lucas: That's the Walmart way. They know that if suppliers and logistics partners run on compatible systems, the whole network becomes more efficient. OneStore includes Kubernetes at the edge, local databases, and a message bus. Luna: So a smaller retailer could theoretically use that blueprint. Not at Walmart's scale, but the patterns transfer. Lucas: That's the key takeaway. You don't need a trillion events a day. But if you treat inventory as a streaming data problem, you can reduce waste and improve availability. Start by identifying the fastest-decaying decision in your operation and stream that one signal. Luna: For most businesses, that's probably the point of sale data. If you know what just left the shelf, you can decide what to reorder in minutes instead of days. Lucas: Exactly. And you don't need Kafka at Walmart scale. A simple queue with a few thousand events per hour is enough to start. The principle is the same: move from batch to streaming for the decisions that matter most. Luna: Walmart's real-time rebuild is a case study in how to modernize without breaking what works. Batch processing isn't dead, it's just not the only tool anymore. Lucas: And the next frontier they're exploring is using the same streaming data to power in-store robots and automated fulfillment. Once you have real-time visibility, you can start automating the physical moves. Luna: That's a whole other episode. But for today, this was a great look under the hood. Lucas: Thanks for listening. We'll be back next week with another deep dive.