Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Shopify Handles Black Friday Without Breaking
Transcript
- Lucas: We are deep in the run-up to Black Friday 2026, and for the last few years, Shopify has been quietly running one of the most interesting infrastructure experiments in ecommerce — moving away from a shared-nothing approach to something they call 'cell-based architecture.' Luna: I've heard the term thrown around in infrastructure circles, but I'm not sure I could explain it to someone in five seconds. Walk me through it. Lucas: Sure. So imagine the old way — a single monolithic database, or even a cluster, where every merchant's data lives together. When a store like Gymshark or Kylie Cosmetics goes viral during Black Friday, that store's traffic spike can saturate shared resources — database connections, CPU, memory — and slow down every other store on the platform. Luna: Right, the 'noisy neighbor' problem. I've seen that kill latency for entire SaaS tenants. Lucas: Exactly. Shopify's answer was to split the platform into cells — essentially isolated shards, each containing a subset of merchants, with their own database, their own application instances, their own load balancers. One cell's traffic spike cannot bleed into another cell. Luna: So if a store on Cell 42 gets hit with a million visitors in an hour, Cell 43 doesn't even notice? That sounds like the dream. Lucas: That's the theory, and it largely works. But here's the nuance — the isolation isn't free. Each cell needs to maintain its own infrastructure, and routing traffic to the correct cell requires a top-level routing layer that maps every incoming request to the right shard. That routing layer itself becomes a single point of failure if not designed carefully. Luna: So you're essentially trading a monolithic database bottleneck for a monolithic router bottleneck. How do they solve that? Lucas: They don't use a single router. They use a fleet of stateless proxies, each running a consistent hashing algorithm that maps the store subdomain to a cell. The proxy fleet can scale horizontally, and if one proxy goes down, another picks up the same hash ring. Luna: Okay, that makes sense. But I've read that during peak Black Friday, some Shopify stores handle over 10,000 requests per second. What's the surprising bottleneck in that kind of load? Lucas: You'd think it's the database queries, right? Turns out, the first bottleneck they hit was TLS handshake overhead at the proxy layer. Every new HTTPS connection requires a cryptographic handshake, and when you have thousands of connections per second, that becomes a CPU sink. Luna: Interesting. So they're not just optimizing SQL queries — they're optimizing crypto. What's the fix? Lucas: Two things. First, they implemented TLS session resumption aggressively — so repeat visitors don't do a full handshake. Second, they moved the TLS termination point closer to the edge, inside the cell's own load balancers, so the top-level proxy only does plain HTTP routing, which is much cheaper. Luna: That's clever. But doesn't that mean each cell needs to handle its own TLS termination? That adds operational complexity. Lucas: It does, but they decided the tradeoff was worth it. And to manage that complexity, they've invested heavily in automation — provisioning a new cell is a fully scripted process that takes about 15 minutes. They also do something unusual: 'Failure Injection Fridays.' Luna: Failure Injection Fridays sounds like something I'd want to attend with popcorn. Lucas: Honestly, it's probably terrifying. Every Friday, the platform reliability team randomly kills a component — a database node, a load balancer, a cache cluster — and sees if the cell survives. If it doesn't, they fix the gap before the next Friday. They've been doing this for over two years now. Luna: That's chaos engineering, but on a weekly, almost ritualized schedule. I imagine the first few Fridays were brutal. Lucas: They've talked about how early on, they'd kill a cache node and discover that a critical service had no fallback — it would just return 500 errors. Now, after dozens of iterations, most services have graceful degradation paths. They even open-sourced part of their sharding toolkit, called 'Cellar,' so other platforms can adopt the pattern. Luna: I want to push back a little — cell-based architecture sounds great for Shopify, which has over a million merchants. But is this overkill for a smaller ecommerce platform? Like, if I'm running a store on a smaller provider, should I be thinking about this? Lucas: I think the honest answer is no — not until you hit a certain scale. The operational overhead of managing multiple cells, the routing layer, the automation tooling — it's significant. For most platforms, a well-tuned monolithic database with read replicas and caching will handle Black Friday just fine. Luna: So where's the tipping point? When do you know you need cells? Lucas: Shopify engineers have said roughly when you have more than 10,000 merchants, and you start seeing 'blast radius' incidents — where one tenant's traffic spike causes a platform-wide outage. That's when the isolation benefits start to outweigh the complexity costs. Luna: That's a useful heuristic. And I appreciate that they open-sourced the toolkit — it's rare to see a company give away the very thing that gives them a competitive advantage on reliability. Lucas: It's a smart move, actually. By open-sourcing Cellar, they get free contributions from the community, they build goodwill, and they also make it easier for their own engineers to move between companies without losing familiarity with the tooling. It's a talent play as much as a tech play. Luna: Speaking of talent and community, this feels like a good moment to mention something that keeps shows like this one going — listener support. We don't run ads, and that's a deliberate choice. A couple of dollars a month is genuinely what keeps these episodes coming — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: Yeah, it's a small thing that makes a big difference for us. We're able to stay independent and cover topics like this without any sponsor influence. Luna: Exactly. And we're back to the architecture — one last thing I wanted to ask. During Black Friday 2025, did they have any major incidents despite the cell isolation? Lucas: They did, actually. There was a routing-layer bug where a software update caused the hash ring to rebalance incorrectly, and about 2 percent of stores were temporarily routed to the wrong cell. That meant those stores' customers saw stale inventory or checkout errors for about 4 minutes before the routing was fixed. Luna: So even the best isolation doesn't protect you from control-plane failures. That's humbling. Lucas: It is. And it's why they now run the routing layer as a canary deployment — a new version rolls out to 1 percent of traffic first, then 5 percent, then 20, then full. That 4-minute incident taught them a lesson they won't forget. Luna: So going into Black Friday 2026, what's the biggest risk they're worried about? Lucas: From what I've read, they're most concerned about 'slow drip' failures — not a sudden spike, but a gradual degradation of a critical dependency, like a DNS provider or a CDN edge, that slowly increases latency across all cells without triggering any single alert threshold. They've been investing in more sophisticated anomaly detection that looks at latency percentiles across cells over time. Luna: So the next frontier isn't just isolation — it's observability across isolated systems. That's a fascinating challenge. Lucas: It really is. And it's a reminder that as your infrastructure becomes more distributed, the hardest problems are often the ones you don't see coming.