Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Cloudflare Rebuilt Its DNS for 100 Million Queries Per Second
Transcript
- Lucas: In late 2023, Cloudflare's DNS network handled a sustained attack of over 100 million queries per second. Not for a few seconds — for several minutes. And it held. Luna: That is insane. Most DNS infrastructure would buckle at a tenth of that. Lucas: Right. So the obvious question is how. And the answer isn't just 'more servers.' It's a deliberate architectural evolution that started years earlier. Cloudflare runs what they call an authoritative DNS — meaning when you type a domain like 'example.com,' Cloudflare is the system that tells the world which IP address to go to. They handle about 28% of all web domains. Luna: That's a staggering market share. So what was the bottleneck before they rebuilt it? Lucas: The old system used a software resolver called BIND — the Berkeley Internet Name Domain. BIND is battle-tested, it's been around since the 80s. But it's single-threaded for each query, meaning each request blocks until a response is ready. At Cloudflare's scale, BIND was creating serious latency spikes. They measured p99 latency hitting nearly 100 milliseconds during traffic peaks. Not acceptable for DNS, where the expectation is single-digit milliseconds. Luna: So they replaced BIND entirely. That's a huge bet. Where did they start? Lucas: They built a custom DNS resolver in Rust. Rust gave them memory safety without a garbage collector, and they could control every CPU cycle. They called the project 'RR' — Rust Resolver. The key insight was to use an event-driven, asynchronous architecture. Instead of one thread per query, they use a small pool of threads that handle thousands of concurrent connections using epoll and async I/O. That alone cut p99 latency from 100 milliseconds to under 2 milliseconds. Luna: Two milliseconds? That's a 50x improvement. But raw speed isn't the only challenge at 100 million queries per second. How do you even get that many packets into the box? Lucas: Exactly. The network interface itself becomes a bottleneck. Cloudflare uses a technique called kernel bypass — they wrote their own network driver that grabs packets directly from the NIC to userspace, skipping the Linux kernel's networking stack entirely. They use a library called DPDK — Data Plane Development Kit. That lets them process millions of packets per second per core without context switching. Each server can handle about 12 million queries per second. Luna: So they're not just throwing hardware at it — they're optimizing every layer of the stack. Lucas: Right. And the hardware layer matters too. They don't use standard servers. They use custom-built machines with dual 64-core AMD Epyc processors, and they've tuned the NUMA topology so that each CPU socket handles its own set of NICs and memory. No cross-socket traffic. That alone reduces latency by about 30 percent. Luna: I want to talk about the caching strategy. DNS has multiple layers of cache — the browser cache, the OS cache, the resolver cache, and the authoritative cache. How does Cloudflare structure theirs? Lucas: They use a two-level cache. Level one is an in-memory hash table in the RR process itself — it stores the most frequently queried domains, basically the top 1 percent of traffic, which accounts for about 80 percent of all queries. That cache is lock-free, using RCU — read copy update — so readers never block. Level two is a distributed cache using Redis clusters spread across data centers. If a query misses level one, it hits level two, and that reduces the number of queries that actually need to go to the authoritative backend — the database that holds the real DNS records. Overall, the cache hit rate is over 90 percent. Luna: That means only 10 percent of queries touch the backend database. But when that 10 percent is still 10 million queries per second, the backend itself has to be designed carefully. Lucas: Right. The backend is a distributed key-value store they built internally, called 'Quicksilver.' It's based on FoundationDB — a transactional key-value store designed for strong consistency. DNS records need to be consistent — if you update a record, every query should see the new value within seconds. Quicksilver replicates writes to all data centers using a consensus protocol, and it handles about 200,000 write operations per second across the fleet. They also have a mechanism called 'zone-based sharding' — each domain's records live on a specific shard, so queries for that domain always hit the same backend server, which improves cache locality. Luna: One thing I find fascinating is how they handle the 'many-eyeballs' problem. When a major website changes its IP address, suddenly millions of clients around the world all query for the same record at once. That's called a thundering herd. Lucas: Exactly. Cloudflare solved that with a technique they call 'request coalescing.' If multiple queries for the same domain arrive at the same server simultaneously, only one query goes to the backend. The others wait on a local mutex and get the result once it's back. That prevents the backend from being overwhelmed. And they apply a similar technique at the network level using anycast — the same IP address is advertised from all 330 cities. Traffic naturally spreads based on BGP routing, so no single data center gets all the load. Luna: Let's talk about how they test this. You can't just flip a switch and hope it works at 100 million queries per second. Lucas: They have a dedicated testing infrastructure called 'Flamethrower' — a cluster of servers that generate synthetic DNS traffic at scale. They can simulate up to 50 million queries per second from multiple locations. During the rebuild, they ran continuous load tests, gradually increasing the traffic until they found breakpoints. One early breakpoint was the memory allocator — they were using the default glibc malloc, which caused fragmentation under high concurrency. They switched to a custom allocator called mimalloc from Microsoft, which reduced memory usage by 40 percent and improved throughput by 15 percent. Luna: I love those details. It's not just the big architecture — it's the allocator choice, the NUMA tuning, the cache structure. All those small optimizations compound. Lucas: And a lot of those optimizations came from analyzing production traffic patterns. They instrumented every query with a trace ID that logs the full path through the system. They can replay any real-world traffic pattern in the test environment. That level of observability is what let them make surgical changes rather than guessing. Luna: Speaking of production — what's the current state? Have they hit any new limits? Lucas: They've peaked at about 120 million queries per second during the largest DDoS attacks. And the system still keeps p99 latency under 5 milliseconds. But they're already planning the next iteration. The bottleneck now is the network itself — the optical transceivers on the servers max out at 100 gigabits per second. They're working with hardware vendors on 400-gigabit NICs. And they're experimenting with RDMA — remote direct memory access — to allow servers to share memory pools across the data center without network overhead. Luna: It's fascinating how pushing one boundary reveals the next. And this kind of deep engineering is exactly why we love building in this space. If these conversations are useful for what you're building or running, listener support is what keeps this show ad-free. Lucas: Yeah, it's a listener-supported show — no sponsors, no ads. If you find value in episodes like this, you can help at buy me a coffee dot com slash fexingo. Every bit goes straight into production time. Luna: And we're back to the architecture. Lucas, you mentioned they're looking at RDMA. Could that eliminate the need for the two-level cache? Lucas: Potentially. If every server can access a shared, ultra-low-latency memory pool, the local cache becomes less important. But RDMA introduces its own complexity — you need reliable, low-jitter networks, and the memory coherence model is tricky. Cloudflare's team is prototyping a system where the hot cache is actually a distributed hash table spread across all servers using RDMA reads. Early benchmarks show p99 latency under 1 millisecond for a cache miss, which is competitive with the current two-level approach. Luna: That's ambitious. Let's zoom out — what's the biggest lesson for other engineering teams building at scale? Lucas: I think it's the principle of 'measure everything, then optimize the bottleneck.' Cloudflare didn't start by rewriting the resolver. They started by collecting detailed latency histograms, packet traces, and resource utilization data. They found that BIND was the bottleneck. Then they built RR. Then they found the NIC was the bottleneck. Then they used DPDK. Each step was driven by data, not dogma. That's the playbook. Luna: And they didn't over-abstract. They built exactly what they needed, when they needed it. No premature generalization. Lucas: Exactly. The Rust resolver is purpose-built for DNS. It doesn't try to be a general-purpose async framework. That focus is why it's so fast. And it's also why the system has been stable for over two years without a major redesign. Luna: I think that's a good note to end on. Thanks for walking through this, Lucas.