Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Cloudflare Built Its Global Edge Network for 100 Million Requests per Second
Transcript
- Luna: So Lucas, we've talked a lot about companies rebuilding their core systems — but I want to zoom out a bit. If you're serving the whole internet, how do you even think about building the network itself? Lucas: That's exactly what Cloudflare had to figure out. Today, they handle over 100 million HTTP requests per second — that's about 20 percent of all internet traffic. And they do it from over 300 cities worldwide. But the architecture that makes that possible is surprisingly elegant. Luna: I've always heard about their 'anycast' network. Can you explain what that actually means in practice? Lucas: Sure. Anycast is the trick where multiple servers around the world advertise the same IP address. When you type cloudflare dot com, your request goes to the nearest data center — not a central one. That's how they get sub-10 millisecond response times for most users. Luna: Right, but anycast alone doesn't solve everything. How do they handle things like SSL termination at that scale? Lucas: Great question. Every edge server terminates SSL for millions of domains. They use a custom TLS library that's optimized for their hardware — things like session resumption and OCSP stapling are done locally. They claim they can handle 10 million new TLS connections per second per server. Luna: That's insane. But the real magic is probably in the caching layer, right? How do they keep data consistent across 300 locations? Lucas: That's where Quicksilver comes in — their distributed key-value store. It's built on top of a consistent hash ring, but with a twist: they prioritize eventual consistency for performance. So if you update a cache rule in one city, it propagates globally in under a second, but there's no blocking. Luna: So if a bank updates its firewall rule, the change might not be instant everywhere? Lucas: For security rules, they have a separate path — a globally replicated database that uses Paxos for strong consistency. But for most caching, eventual consistency is fine. The key insight is they never block a request waiting for a database write. Luna: And they also run a huge DDoS mitigation system. How does that integrate with the edge? Lucas: All traffic goes through the edge, so DDoS detection happens at every server. They built a real-time flow analyzer that samples packets and looks for attack signatures. It can block a 2.5 terabit-per-second attack in seconds — without dropping legitimate traffic. Luna: That's got to be incredibly hard to tune. False positives could take down a site. Lucas: Absolutely. They use machine learning models trained on years of attack data. But they also have a human-in-the-loop for large attacks. Their network operations center can take over if needed. Luna: And then there's Workers — their serverless platform. How do they run arbitrary code at the edge? Lucas: Workers use a custom runtime based on the V8 JavaScript engine. Each Worker runs in a sandboxed isolate — not a container — so cold starts are about 50 microseconds. They can spin up a new Worker for every single request if needed. Luna: Fifty microseconds. That's basically nothing. How do they manage state across those Workers? Lucas: They have a key-value store called Workers KV, which is replicated globally. It's optimized for reads — p99 read latency under 10 milliseconds. But writes are eventually consistent, so you can't rely on it for transactional data. Luna: Speaking of hardware, I read they build their own servers. What's the thinking there? Lucas: They designed a custom server called the 'Gen X' — it's an Intel Xeon system with 128 gigabytes of RAM and a bunch of NVMe SSDs. By building their own, they control the cooling, the power efficiency, and most importantly, the network interface cards. They've even written their own firmware for the NICs. Luna: That's the kind of vertical integration you only see at the biggest scale. But what about the OS? I've heard they have something called Kernel by User? Lucas: Yeah, it's a minimal Linux kernel they've stripped down to just what they need — no unnecessary drivers, no desktop subsystems. It's compiled with their own optimizations for low latency and high throughput. They run it on every edge server. Luna: If these kinds of architectural deep dives are useful for what you're building or running, I wanted to mention that listener support is what keeps this show ad-free and focused on the technical details. People who value that can buy us a coffee at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small way to help us keep digging into how systems like Cloudflare actually work. So back to the architecture — one of the most innovative parts is how they handle dynamic content. They don't just cache static files; they can also accelerate APIs. Luna: How do they do that without breaking the API's semantics? Lucas: They use a technique called 'TLS early data' or 0-RTT — so the first request from a client doesn't need a full handshake. They also compress headers and use HTTP/2 multiplexing. For API responses, they can cache based on custom headers set by the developer. Luna: So the developer controls the cache key. That's powerful. But what about the network backbone itself? How do they move traffic between data centers? Lucas: They have a private backbone connecting all their data centers — it's called the Cloudflare Global Network. They use BGP anycast normally, but for inter-datacenter traffic they use MPLS tunnels with traffic engineering. They also peer with major IXPs to keep latency low. Luna: And I assume they have a lot of redundancy. What happens if a whole data center goes offline? Lucas: Anycast handles that automatically. BGP withdraws the routes for that data center, and traffic shifts to the next nearest one. They simulate this regularly with 'failover tests' — intentionally taking a data center offline to see if routing works. Luna: That's the kind of chaos engineering you'd expect. One more thing — how do they handle DNS? Cloudflare runs one of the largest DNS resolvers, 1.1.1.1. Lucas: Their DNS resolver is built on the same edge infrastructure. They use a custom DNS server called 'RRDNS' that's written in Go and heavily optimized for concurrency. They handle over 10 million DNS queries per second per server, with sub-10 millisecond response times. Luna: And it's all anycast as well, right? So you get the nearest resolver. Lucas: Exactly. The same IP — 1.1.1.1 — is announced from every data center. That's the beauty of anycast: simplicity for the client, complexity for the network. Luna: It's remarkable how much thought goes into what seems like a simple CDN. The combination of custom hardware, OS, and software is really what sets them apart. Lucas: And it's that integration that lets them handle 100 million requests per second while keeping costs low. Their average cost per request is something like one one-hundredth of a cent. That's the kind of efficiency you only get when you control the full stack. Luna: So what's next for edge architecture? Do you see edge compute replacing traditional cloud servers? Lucas: Not entirely — there are still workloads that need strong consistency or long-running processes. But for latency-sensitive, stateless tasks, edge is already winning. I think we'll see more hybrid models where edge handles the hot path and the cloud does the heavy lifting. Luna: And that's where the interesting engineering challenges lie — making those two worlds work together seamlessly. Lucas: Exactly. Cloudflare's architecture shows that when you treat the network as a computer, you can do things that weren't possible before. And the best part is, they're open about most of it — their blog posts are a goldmine for anyone building distributed systems. Luna: Great place to start if you want to learn more. Thanks, Lucas. Lucas: Thanks, Luna. Until next time.