Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Cloudflare Migrated 30 Million Requests Per Second Without Downtime
Transcript
- Lucas: If these conversations are useful for what you're building or running, you're in the right place. I'm Lucas. Luna: And I'm Luna. Today: Cloudflare's migration of thirty million requests per second to a new proxy stack. Zero downtime. Lucas: Right. So Cloudflare sits in front of about twenty percent of the web. Every request — whether it's a page load, an API call, a video stream — passes through their edge. And for years, that edge ran on a proxy called NGINX, modified heavily, but still NGINX at the core. Luna: And they decided to replace it? That's like swapping the engine of a plane mid-flight. Lucas: Exactly. They built a new proxy from scratch in Rust — they call it Pingora. The motivation was that NGINX's architecture, which is event-driven and single-threaded per worker, wasn't scaling cleanly for their specific use case. They wanted better control over connection handling, TLS termination, and request routing. Luna: So they didn't just upgrade NGINX. They rewrote the whole thing. Lucas: Correct. And the migration meant every single request that hit Cloudflare's network needed to go through the new stack. That's thirty million requests per second at peak. And they couldn't afford even a blip — because their customers' sites go down if Cloudflare goes down. Luna: So how do you even begin to test that? You can't simulate thirty million requests per second in a lab. Lucas: You can't. So they relied on a technique called canary deployment — but at massive scale. They started routing a tiny fraction of live traffic — like one percent — to the new Pingora stack, and they monitored everything: latency, error rates, CPU usage, memory. Then they slowly ramped that percentage up over weeks. Luna: But what about state? Connections, TLS sessions, keep-alives. If a request lands on the new stack and the next request from the same user lands on the old stack, that could break. Lucas: Great question. They solved that with connection draining. When they wanted to move a particular customer's traffic, they'd gracefully close existing connections on the old stack and let new connections be established on the new one. That way, no single user saw a mix of stacks during a session. Luna: So it was per-customer, not per-request. That's smart. Lucas: Yeah. They had a control plane that could tag a customer as 'migrated' and then the edge nodes would handle the transition. The key was that the migration was reversible. If a customer saw increased error rates, they could flip them back to NGINX in seconds. Luna: And did they have to roll back any? Lucas: They did. In the early stages, they found a few bugs — memory leaks in certain request patterns, edge cases with HTTP/2 prioritization. But because they could roll back individual customers, the blast radius was tiny. Luna: Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Keeps us ad-free and focused on stories like this. Lucas: Yeah, it's a small way to say these deep dives matter. And we appreciate it. Luna: So back to the migration — how long did the whole thing take? Lucas: About six months from first canary to full cutover. But they didn't just flip a switch. They gradually increased the percentage of customers on Pingora until NGINX was handling almost zero traffic. At that point, they could decommission the old stack. Luna: Six months feels fast for something that touches every request on the network. Lucas: It is fast. But it's also a testament to their engineering culture. They had a dedicated team, clear metrics, and a culture of incrementalism. No big bang releases. Luna: And the result? Performance gains? Lucas: Significant. They saw a ten percent reduction in latency on average, and for some customers, up to thirty percent. That's because Pingora is more efficient at connection reuse and TLS handshakes. Plus, they can now iterate faster — adding features in Rust instead of patching NGINX configs. Luna: So it wasn't just a migration. It was an architectural upgrade. Lucas: Exactly. And the lesson for CTOs is: you can replace your core infrastructure if you have the discipline to canary, monitor, and roll back. It's not about bravery. It's about process. Luna: And about having a rollback plan for every customer. Lucas: Right. Cloudflare proved that even the most critical piece of infrastructure can be swapped out — as long as you treat the migration as a series of small, reversible steps rather than one giant leap. Luna: Which is a good way to think about any big engineering change. Lucas: Exactly. Next time you're staring down a migration of your own, remember: canary, measure, roll back, repeat.