Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Cloudflare Rebuilt Its DNS for 100 Million Queries per Second
Transcript
- Lucas: Cloudflare handles roughly 100 million DNS queries per second during peak DDoS attacks. That's not a theoretical ceiling — it's a real number they hit in late 2024 during the largest DDoS event ever recorded. Luna: And they rebuilt the entire DNS stack to get there. That's what we're digging into today — how Cloudflare replaced a system that already worked well enough for most people. Lucas: Exactly. If you operate an engineering org that's scaling fast, this kind of rebuild is a familiar tension. The old system works, but it won't work tomorrow. Lucas: And speaking of what works — we keep this show ad-free because we think conversations like this should be accessible without interruption. If you find value in these deep dives, you can support the show at buy me a coffee dot com slash fexingo. It's a simple way to keep us going, and we appreciate every bit of it. Luna: Yeah, it really helps us keep the focus on the engineering, not the sponsor reads. So back to Cloudflare — where did they start before the rebuild? Lucas: They started with a fairly standard setup: BIND, the Berkeley Internet Name Domain software that's been around since the 80s. It's battle-tested, but it's single-threaded and not designed for the throughput they needed. Lucas: By 2020, they were already seeing bottlenecks. The BIND process would spike CPU to 100 percent under attack, and they'd have to spin up more instances, which meant more servers, more cost, more complexity. Luna: So they decided to build their own resolver. What did they call it? Lucas: RRDNS. Stands for 'Rusty Roadrunner DNS' — a nod to both the language and the speed. They wrote it in Rust for memory safety and performance. Luna: Why Rust specifically? There are other fast languages like C or Go. Lucas: Good question. Rust gives them zero-cost abstractions and memory safety without a garbage collector. For a DNS resolver that has to handle millions of concurrent connections, you can't afford GC pauses. And they wanted to eliminate entire classes of bugs like buffer overflows that could be exploited in a security-critical service. Lucas: The team also wanted to push performance boundaries. In benchmarks, RRDNS handled 40 percent more queries per second than the previous bind based setup on the same hardware. Luna: That's a significant improvement. But the tricky part must have been the migration — how do you swap out DNS infrastructure without downtime? Lucas: They used a technique called 'shadow traffic'. They deployed RRDNS alongside the existing BIND servers and gradually mirrored a percentage of live queries to the new system. They compared responses byte for byte before serving them to clients. If there was a mismatch, they'd fall back to BIND's answer. Lucas: Over months, they increased the shadow percentage. Once they were confident, they flipped the primary to RRDNS and kept BIND as a hot standby. Luna: So they never actually cut over in one big event. That's smart — especially for DNS where even a few seconds of downtime means millions of websites unreachable. Lucas: Right. And they had to handle edge cases like DNSSEC validation. BIND had its own validation logic; RRDNS had to replicate that exactly. They found subtle differences in how some edge-case domains were handled and fixed them iteratively. Luna: What about the anycast routing? Cloudflare uses anycast to spread traffic across their global network. Did they have to change that? Lucas: No, the anycast layer stayed the same. But they did optimise how RRDNS handles the per-connection state. In BIND, each query spawns a new process or thread. RRDNS uses an event-driven, single-threaded per-core model with shared-nothing architecture. That reduces context switching and cache misses. Lucas: They also implemented a custom caching layer that stores recent responses in a lock-free hash table — again, in Rust, using atomic operations. Luna: Lock-free data structures are hard to get right. Did they have any incidents during the rollout? Lucas: A few. One notable issue was with the TTL — time to live — handling. BIND has a quirk where it can serve slightly stale records under load to reduce upstream queries. RRDNS initially didn't replicate that behaviour, and some customers saw increased latency because it was fetching fresh records too aggressively. Lucas: They fixed it by adding a 'stale cache' mode that serves expired records if the upstream is unreachable. That's become a standard feature in many modern resolvers now. Luna: So the rebuild wasn't just about throughput — it also improved resilience. Lucas: Absolutely. And they proved it during the 2024 attack. The attack peaked at 100 million queries per second, targeting a single customer's domain. Cloudflare's infrastructure absorbed it without any customer-facing impact. RRDNS handled the load with CPU usage at about 60 percent across the edge nodes. Luna: That's impressive. But maintaining that level of performance requires constant tuning. What's their monitoring stack look like? Lucas: They built custom metrics exporters that feed into Prometheus and Grafana. They track queries per second per server, response time percentiles, cache hit ratio, and error rates. They also have a real-time dashboard showing the health of every DNS node across 330 cities. Lucas: One interesting metric: 'query latency at the 99.9th percentile' — they target under 10 milliseconds globally. During the attack, it spiked to 15 milliseconds but stayed within their internal threshold. Luna: That's a tight SLA. So what's the biggest lesson for engineering leaders from this story? Lucas: I think it's the shadow-traffic pattern. You can rebuild a critical system without taking risks if you invest in the testing infrastructure. Cloudflare spent months building the comparison framework before they even considered switching traffic. Lucas: Also, choosing Rust wasn't just about speed — it was about reducing cognitive load on the team. Memory safety guarantees mean fewer late-night incidents due to segfaults. Luna: And they open-sourced RRDNS, right? What's the benefit there? Lucas: Yes, the core resolver is on GitHub under an open-source license. It helps with recruiting — developers want to work on high-profile projects — and it also means the community can contribute optimisations. Cloudflare gets free R&D from the broader Rust ecosystem. Luna: Any downsides? Open-sourcing a security-critical component seems risky. Lucas: There's some risk, but Cloudflare argues that transparency improves security. More eyes on the code means vulnerabilities get found faster. And they keep their proprietary DDoS mitigation logic separate, so attackers can't directly see the secret sauce. Lucas: Overall, this is a case study in how to modernise infrastructure that the internet depends on. Incremental migration, a language that prioritises correctness, and a culture of rigorous testing. Luna: And they did it without any major outages. That's the mark of a mature engineering org. Lucas: It really is. I think the question every CTO should ask is: what's the BIND in your stack? The thing that's reliable but outdated, and how do you start shadowing a replacement before it breaks?