Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How MongoDB Rebuilt Its Replica Set Protocol for Planet-Scale Writes
Transcript
- Lucas: If these conversations are useful for what you're building or running, we want you to know why they exist. We keep this show ad-free and independent — no sponsors, no sponsored segments, no corporate gatekeeping. The only reason we can do that is listener support. So if today's episode gave you something concrete, buy me a coffee dot com slash fexingo — that's the entire business model. And we mean it, no pressure, just a quiet way to keep the lights on. Luna: Yeah, it's a small gesture that adds up. And it keeps the show exactly the way it is — no one telling us what to cover. Lucas: Exactly. So — back to MongoDB's protocol. Luna, you remember the infamous 'election storm' incidents on Hacker News a few years back? Luna: Vaguely. Refresh me — the one where a single node failure cascaded into a cluster-wide outage because re-elections took too long? Lucas: That's exactly it. So MongoDB's replica sets have used a Raft-inspired consensus protocol since version 3.2, roughly 2015. It worked fine for moderate write loads — say, 5,000 writes per second per primary. But as users scaled up to 50,000 or even 100,000 writes per second, the election protocol became a real bottleneck. Luna: Why? Isn't election something that only happens during failures? Why would write throughput affect it? Lucas: Great question. The issue is that under high write load, the primary node's heartbeat interval — the signal it sends to secondaries to say 'I'm alive' — can get delayed just enough to trigger unnecessary elections. In Raft, if a follower doesn't hear from the leader within a certain timeout, it calls an election. So if your primary is busy processing writes, those heartbeats can jitter. And when an election starts, all writes stop until a new primary is chosen. Luna: So the system becomes its own worst enemy. More writes lead to more jitter, more jitter leads to more elections, more elections lead to less throughput. Lucas: Exactly. And the default election timeout in MongoDB was around 10 seconds. Under normal conditions, that's fine. But under heavy load, a 200-millisecond heartbeat delay could cascade. There's a well-known incident from a large US retailer — we won't name them — where a three-node replica set experienced a network partition. One node got isolated, the other two could still talk to each other but couldn't reach a majority. The election process took 47 seconds because the nodes kept voting for themselves. Luna: Forty-seven seconds of write downtime in a retail environment — that's millions of dollars in lost transactions. Lucas: Exactly. So MongoDB's engineering team, led by their core server group in New York, decided to rewrite the election protocol. They didn't just tweak the timeout values. They changed the fundamental mechanism. The key innovation was what they call 'speculative leader leases'. Luna: Speculative leases — that sounds like something from distributed systems literature. Is it similar to what Google's Chubby does with lease-based locks? Lucas: It's in the same family, but simpler. The idea is this: instead of waiting for a full election round — which requires votes from a majority of nodes — a candidate can issue itself a short-term lease immediately after receiving just one additional vote beyond its own. So if you have a three-node set, the candidate votes for itself, gets one other vote, and instantly becomes the primary for a very short period — like 500 milliseconds. During that lease, it can accept writes. If no other node challenges within that window, the lease becomes permanent. Luna: So it's an optimistic election. You assume you'll win, start serving, and back off if a conflict arises. Lucas: Exactly. And the risk is low because the lease is so short. If there's a true split-brain scenario, the other node will realize the lease holder is alive and step down. But in practice, most elections are uncontested — one node is clearly ahead in log term. So speculative leases cut election latency from 10 seconds to under 2 seconds in MongoDB's benchmarks. Luna: And what about the heartbeat jitter problem? Does the new protocol address that? Lucas: It does, indirectly. Because elections are faster, the system can tolerate a few missed heartbeats without triggering a full-blown election storm. They also increased the heartbeat frequency from once per second to twice per second, and made the heartbeat timeout adaptive based on recent network latency. So the system becomes more resilient to transient delays. Luna: So they didn't just change the protocol — they also changed the operational parameters. That's the kind of holistic fix that a lot of teams skip. Lucas: Right. And they shipped it in MongoDB 7.0, which came out in mid-2023. According to MongoDB's own published benchmarks, write throughput under failure scenarios improved by 5x in some cases. One of the early adopters was Adobe, which runs a massive MongoDB cluster for its Experience Cloud platform. They saw replica set failover times drop from 12 seconds to under 3 seconds during controlled failover tests. Luna: That's a huge improvement. But it also raises a question: why not just switch to a different consensus algorithm entirely? Like Multi-Paxos or EPaxos? Lucas: That's the classic engineering trade-off. MongoDB's replica set protocol is deeply embedded in their storage engine, oplog, and driver ecosystem. Rewriting the consensus layer from scratch would have taken years and risked introducing new bugs. Instead, they optimized within the existing framework. It's a pragmatic choice — ship improvements now rather than a perfect solution five years from now. Luna: And it clearly worked. But I'm curious — did they consider any alternative approaches before settling on speculative leases? Lucas: They did. According to some of the technical blogs from MongoDB engineers, they evaluated three options. Option one: increase the heartbeat timeout so elections are less frequent — but that would delay legitimate failovers. Option two: use a leader lease scheme similar to the Google Chubby approach, where a leader periodically renews a lease with a quorum — that's more robust but adds complexity. Option three: the speculative lease approach they ultimately chose. It was the best balance of simplicity and performance. Luna: And it seems like they also benefited from the fact that most MongoDB deployments are three-node replica sets. In a three-node set, speculative leases are particularly effective because you only need two votes to win. Lucas: Exactly. The protocol is optimized for the common case. But they also tested it on larger sets — five-node, seven-node — and it still performed well. The lease time scales linearly with the number of nodes, but in practice, most clusters are small. Luna: So what's the lesson for CTOs and engineering leaders who are considering similar protocol changes? When should you optimize and when should you replace? Lucas: I think the key takeaway is to measure first. MongoDB only undertook this redesign because they had hard data showing that election latency was a top driver of customer incidents. They quantified the problem — 47-second outages, 12-second failovers — before designing the solution. Too many teams try to optimize a system without understanding the actual bottleneck. The other lesson is to ship incrementally. They didn't rewrite the entire consensus layer. They added one new mechanism — speculative leases — and kept everything else the same. That reduced risk and accelerated delivery. Luna: And they also maintained backward compatibility. Old clients and drivers didn't need to change, right? Lucas: Correct. The protocol change was transparent to applications. That's another hallmark of good system evolution — you improve the infrastructure without forcing your users to adapt. So if you're running MongoDB 7.0 or later, you're already benefiting from this work, even if you never knew the election protocol was a problem. Luna: Which is exactly how a well-designed system should behave — the complexity is hidden from the user. Lucas: Exactly. So next time you're staring at a distributed systems problem, remember MongoDB's approach: measure the pain, identify the common case, make one targeted change, and validate with real-world data. It's a blueprint that works for more than just databases.