Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Zalando Cut Checkout Failures by 45 Percent
Transcript
- Lucas: So last week, Zalando quietly published a case study about how they cut checkout failures by 45 percent during their peak traffic events. And the interesting part isn't the number itself — it's the sequence of architectural decisions that got them there. Luna: I remember Zalando being one of those classic monolith turned microservices stories. Did they finally break up the checkout monolith? Lucas: Exactly. For years, their checkout was a single monolithic service that handled everything from cart to payment to order creation. It was reliable at normal scale, but during Black Friday or their end of season sales, the whole thing would start to buckle. They'd see timeouts spiking, and every extra request would just add more pressure. Luna: So they decided to split it into separate services. But what did they actually split it into? Lucas: They broke it into four core services: cart management, pricing, payment, and order orchestration. Each service owns its own data and communicates via asynchronous events. The key shift was moving from a synchronous request-response flow to an event-driven one. Luna: That's a big change. How did they handle consistency across those services? That's the hard part. Lucas: They used a distributed saga pattern for order processing. So when a customer hits 'place order', the order orchestration service sends a 'create order' event, then the payment service processes the payment, then the inventory service reserves stock, and so on. Each step is an event, and if any step fails, the saga runs compensating actions — like releasing a payment hold or restocking the cart. Luna: So it's like a state machine where every step has a fallback. That's how you avoid the classic 'partial order' problem. Lucas: Right. And they built this on top of a combination of Redis and Postgres. Redis handles the fast, ephemeral state — like the current step in the saga and temporary cart data — while Postgres is the system of record for the final order. That split means they can scale the read-heavy parts horizontally without locking the database. Luna: So Redis for speed, Postgres for durability. That's a pattern I see more often in e-commerce now. Lucas: Yeah. And they didn't just flip a switch. They ran this in parallel with the old monolith for a few months, comparing outcomes on a small percentage of traffic. They could measure the failure rate in real time and only cut over when the new system was actually better. Luna: That's the careful approach. But what about the peak traffic itself — did they do anything else to handle the spikes? Lucas: They introduced a technique they call 'traffic shaping' — basically, they throttle the rate at which checkout requests are accepted, allowing a controlled queue to build up instead of letting a flood of requests hit the services all at once. Users might see a brief spinner, but the system stays responsive and the order actually goes through. Luna: So you trade a little latency for a lot of reliability. That's a fair trade during a flash sale. Lucas: Exactly. And they also invested heavily in chaos engineering. Their team regularly injects failures into the checkout pipeline — like killing a payment service instance or adding artificial latency to Redis — to see how the system behaves. They call these 'game days', and they practice them monthly. Luna: That's great in theory, but in practice, most teams don't have the time or budget for that. How do you get buy-in from leadership? Lucas: That's a real question. One thing that helped Zalando was tying the chaos drills to actual incidents. They started by recreating past outages in a staging environment, showing how the new system would have handled them. That builds a much stronger case than just saying 'we should test for failures'. Luna: So it's like a post-mortem turned proactive test. That's smart. Lucas: Yeah. And they also made sure the chaos experiments were automated, so they didn't require a lot of manual effort every month. They built a tool that randomly picks a service and injects a failure, then runs a series of checks to see if the system recovers within a target time. That keeps the cost low. Luna: Automated chaos is the way. If you have to schedule it manually, it's not going to happen regularly. Lucas: And it's worth mentioning that this kind of work — the monitoring, the event-driven architecture, the game days — it's the stuff that doesn't show up in a feature roadmap, but it's what keeps the business running when it really matters. Luna: Speaking of things that keep things running but don't get on a roadmap — this podcast is exactly that for a lot of listeners. It's ad-free, and that's only because of the people who chip in a few dollars a month. If you've gotten something out of these episodes, you can support us at buy me a coffee dot com slash fexingo. Even a couple of bucks genuinely makes a difference in keeping this going. Lucas: Yeah, honestly, every little bit helps. It means we can keep digging into case studies like this without any sponsors pulling the strings. Luna: And back to Zalando — one of the more interesting side effects of the new architecture was that their error rates dropped not just during peaks, but at all times. Because the services are decoupled, they could deploy changes to pricing without affecting payment, and they could scale individual services independently based on load. Lucas: That's a key benefit of microservices that often gets overlooked: deployability. They went from a monthly release cycle on the monolith to deploying services multiple times a day. That's a huge change in velocity. Luna: Did they hit any snags with that? I can't imagine moving to microservices is all smooth sailing. Lucas: Oh, absolutely. They talked about debugging getting harder because a single order can now span multiple services. They had to invest in distributed tracing to stitch together the full request path. They built a custom tracing solution using OpenTelemetry, which gives them a timeline of every event in a saga. Luna: So you trade monolithic simplicity for observability complexity. But if you're disciplined, the payoff is worth it. Lucas: Right. And they also had to rethink their data model. In the monolith, they could just do a join across tables. Now, each service owns its data, so they can't just query across services. They had to introduce a read-model pattern — a denormalized view that's updated via events, which powers the order history and status pages. Luna: So they're essentially trading consistency for availability, which is the classic trade-off in distributed systems. Lucas: Exactly. And they were very clear that this isn't the right approach for every company. If your checkout is simple and you don't have peak traffic problems, a monolith is probably fine. But if you're seeing multi-second timeouts during sales, the event-driven saga pattern is a proven way to get out of that hole. Luna: What would you say is the one takeaway for someone listening who's facing similar issues? Lucas: Don't start by rewriting everything. Zalando didn't do a big-bang rewrite. They took the checkout flow, identified the hot spots — the parts that failed under load — and split those out into services first. Then they added the event-driven layer and the compensating actions. Incremental steps, each with clear metrics. Luna: So the lesson is: focus on the pain point, not on the architecture for its own sake. Lucas: Exactly. And measure everything before and after. They tracked checkout failure rate, average order completion time, and infrastructure cost per order. Those numbers are what convinced their leadership to keep going. Luna: That's a good reminder that the business case has to be as solid as the technical one. Lucas: For sure. And it's also worth noting that they didn't just focus on the software — they also invested in capacity planning. They modeled their expected peak traffic based on historical data and marketing spend, then did load testing against those numbers. That's proactive, not reactive. Luna: So they're not just hoping they'll survive the next sale; they're actively preparing for it. Lucas: Right. And the result is that during their last major sale event, checkout failure rates stayed below one percent, even when traffic hit double the previous peak. That's a massive improvement from the five to seven percent failure rates they were seeing before. Luna: That's a huge impact on revenue, too. If you're losing five percent of checkouts, that's millions of dollars in lost sales. Lucas: Exactly. And that's the story we should all keep in mind: architecture decisions have real business consequences. It's not just about tech debt — it's about whether you can actually serve your customers when they need you most. Luna: Well said. If you're thinking about making a similar change, the tools and patterns are out there — you just have to apply them methodically. Lucas: And that's what we're here for. We'll be back next week with another deep dive.