Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Walmart Rebuilt Its Supply Chain with Real-Time Data
Transcript
- Lucas: If these conversations are useful for what you're building or running, you're in the right place. I'm Lucas. Luna: And I'm Luna. Today we're talking about Walmart's supply chain — and not the generic 'how they moved boxes' story, but the actual technical rebuild underneath. Lucas: Right. Walmart operates over 10,500 stores globally, plus a massive e-commerce operation. For decades, their supply chain ran on batch updates — inventory numbers refreshed every 24 hours, sometimes longer. Luna: Which is wild when you think about it. A store could sell out of something at 10 AM, but the system wouldn't know until the next morning. Lucas: Exactly. And in a world where Amazon can deliver same-day, that latency becomes a competitive liability. So around 2020, Walmart started a major initiative called 'Real-Time Supply Chain' — moving inventory tracking from batch to near real time, across all stores and warehouses. Luna: And that's not just an IT project — that touches every part of the business. Lucas: It does. At the core, they built a unified data platform that ingests point of sale transactions, warehouse management system updates, trucking telemetry, and supplier feeds — all streaming into a central event bus. They use Apache Kafka for that, processing over 50 million events per day at peak. Luna: Fifty million — and that's just the inventory events, not including customer transactions or logistics. Lucas: Right. The architecture is interesting because they didn't try to replace everything at once. They kept existing ERP systems for financial reporting, but layered a real-time layer on top. So stores still have their legacy systems, but now there's a streaming pipeline that picks up changes and propagates them through. Luna: So it's a strangler fig pattern — gradually wrapping the monolith without ripping it out. Lucas: Exactly. One of the key technical challenges was handling data consistency across such a diverse set of sources. A store in rural Arkansas might have a different inventory system than one in Dallas. So they built a canonical data model — a standard schema for what an 'inventory event' looks like, with fields like SKU, store ID, timestamp, quantity change, and event type. Luna: And then each source system has an adapter that translates its native format into that canonical model? Lucas: Yes. They call them 'connectors' — lightweight microservices that poll or listen for changes, transform, and push to Kafka. Walmart has over 12,000 suppliers, each with different APIs, file formats, update frequencies. So the connector layer had to be extensible and easy to onboard. Luna: That's a huge operational burden. How did they manage it without a dedicated team per supplier? Lucas: They built a self-service portal for suppliers. Instead of Walmart engineers writing custom connectors, suppliers can use a standard template — a REST API or a flat-file upload — and the system validates the data against the schema. If it passes, it's automatically ingested. If not, the supplier gets an error report. Luna: So they shifted the integration work to the suppliers, but gave them the tools to do it themselves. Lucas: Right. And that scaled. Today, over 80 percent of supplier data flows through that self-service pipeline. The remaining 20 percent are legacy systems that require manual handling — and Walmart is actively migrating those off as contracts renew. Luna: Let's talk about the edge computing piece. Stores aren't data centers — they have limited connectivity, especially in rural areas. Lucas: That's a huge part of the story. Walmart deployed what they call 'store edge nodes' — small servers in each store, running a local instance of the data pipeline. When a cashier scans a product, the point of sale system sends the event to the edge node, which updates the local inventory count and then syncs with the central Kafka cluster when bandwidth is available. Luna: So even if the internet goes down, the store can still track inventory locally and reconcile later. Lucas: Exactly. They designed for eventual consistency — the central system might be a few minutes behind, but the store always has an accurate view of its own stock. That's critical for things like 'buy online, pick up in store' — if the system thinks there's one unit but the shelf is empty, that's a bad customer experience. Luna: And that's a problem that batch systems just can't solve. By the time the batch runs, the customer is already in the parking lot. Lucas: Exactly. The real-time pipeline also feeds the e-commerce platform directly. When a customer adds an item to their cart on Walmart.com, the system checks the nearest store's inventory via the edge node — not a central database. That lookup happens in under 100 milliseconds. Luna: So the edge nodes are essentially acting as distributed caches that are always warm. Lucas: Yes. And they're not just for inventory. Walmart also uses them for in-store analytics — staffing optimization, checkout wait times, even shelf-scanning robots. The edge node aggregates all that data and sends summary metrics to the cloud, not raw streams. Luna: So they're reducing bandwidth costs by processing locally and only transmitting what's needed. Lucas: Right. Over a year, that saves millions in data transfer and cloud compute costs. It's a practical example of frugal engineering — not just throwing hardware at the problem, but designing for the constraints of the physical world. Luna: I want to ask about the organizational change. This isn't just a technology shift — it changes who owns data, how decisions are made. Lucas: Absolutely. Previously, inventory data was owned by the supply chain team, and IT just ran the systems. In the new model, engineering teams own the data pipelines end to end. They work directly with store operations and suppliers to define SLAs for data freshness — things like 'inventory must be accurate within 30 seconds for 99.9 percent of items.' Luna: So engineering is now on the hook for business outcomes, not just uptime. Lucas: Exactly. That required a cultural shift. Walmart created cross-functional 'domain teams' — each focused on a part of the supply chain, like 'store replenishment' or 'e-commerce fulfillment.' Each team has engineers, product managers, and operations specialists who sit together and own the entire flow from data generation to decision. Luna: And that's the kind of organizational alignment that's harder to achieve than any technical change. Lucas: It is. Walmart's CIO at the time, David Glick, said the hardest part wasn't Kafka or edge computing — it was convincing the supply chain leaders to trust real-time data over their legacy reports. They had to run parallel runs for months, showing that the real-time system's numbers matched the batch numbers, to build trust. Luna: So they had to prove the new system was as accurate as the old one, even though it's fundamentally different. Lucas: Right. And that's a lesson for any CTO: the technical solution might be sound, but if the business stakeholders don't trust it, it doesn't matter. You have to invest in validation and change management. Luna: Speaking of investing — and this is a bit off the main thread, but it's related to sustaining work like this — a couple of dollars a month is genuinely what keeps these conversations going. If you've gotten something out of our shows, buy me a coffee dot com slash fexingo. It really does make a difference for a small operation like ours. Lucas: Yeah, it's true. We've stayed ad-free because of listener support, and every contribution, no matter how small, helps us keep doing deep dives like this one. So thanks to anyone who's already chipped in. Luna: Alright, back to Walmart. One thing I find interesting is how they handle peak demand — holiday season, Black Friday, that kind of spike. Lucas: Great question. During peak times, their event volume can jump 10x. They designed the system to auto-scale: the Kafka clusters can add partitions, the edge nodes have a queue that stores events locally if the central system is overwhelmed, and the processing layer uses Kubernetes to spin up more consumers. Luna: So the edge nodes act as buffers, not just caches. Lucas: Exactly. And they do capacity planning based on historical patterns — they know that Cyber Monday sees a massive spike in online orders, so they pre-scale the pipeline a week in advance. They also run 'chaos engineering' drills where they simulate a regional outage to test the system's resilience. Luna: That's smart. Most companies only think about scaling up, not about what happens when something breaks. Lucas: Right. Walmart's team publishes some of their learnings — they've talked about how they handle data deduplication when events arrive out of order, and how they use idempotent writes to prevent double-counting. For example, if a store sends the same inventory event twice, the system detects it via a unique event ID and ignores the duplicate. Luna: That's a classic distributed systems problem. And in a supply chain context, double-counting could mean ordering twice as much stock or, worse, not ordering enough. Lucas: Exactly. The cost of a bug is real — it's not just a bad user experience, it's millions of dollars in wasted inventory or lost sales. So they invested heavily in testing and monitoring. They have dashboards that track data freshness per store, per supplier, and even per SKU, with alerts if any metric deviates. Luna: So they can see a problem in one store within minutes, not days. Lucas: Yes. And that speed has changed how Walmart operates. For example, if a product is selling faster than expected in one region, the system can automatically reroute trucks from a nearby warehouse — something that used to take a team of analysts three days to figure out. Luna: So the real-time system enables operational agility at a scale that batch processes never could. Lucas: Exactly. And that's the takeaway for any CTO looking at legacy systems: the goal isn't to be real-time for its own sake. It's about enabling decisions that were previously impossible. Walmart's system now handles over 100 million inventory updates per day, and the average latency from a store shelf scan to a central dashboard update is under five seconds. Luna: That's an impressive number. And it shows what's possible when you combine good architecture with organizational alignment. Lucas: It does. And the real test will be whether they can maintain that as they continue to grow — opening new stores, adding more suppliers, expanding internationally. But the foundation seems solid. Luna: Alright, that's all we have time for today. Thanks for listening.