Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How MongoDB Rebuilt Its Document Store for Multi-Terabyte Workloads
Transcript
- Lucas: If you run a database in production long enough, you will hit a wall. Not a performance wall you can tune around — a fundamental architectural wall where the thing just stops behaving the way it's supposed to. Luna: I've been there. A 12-terabyte MongoDB shard that took 47 hours to rebalance. That's not a bug report — that's an existential crisis for the ops team. Lucas: 47 hours — that's actually the exact number that kicked off the two-year rewrite we're talking about today. A single customer complaint about a shard that size, and MongoDB's core storage engine team realized they had to rethink WiredTiger from the ground up for multi-terabyte workloads. Luna: And before we go deep — if you're building something that needs to scale, or just love understanding how these systems actually work under the hood, this is the kind of conversation that keeps this show ad-free. Listener support at buy me a coffee dot com slash fexingo is what makes that possible. So if today's episode is useful to you, that's the way to keep it going. Lucas: Yeah, exactly. And speaking of useful — let's get into what the MongoDB team actually changed, because the fix wasn't obvious. Lucas: So the old architecture — the WiredTiger storage engine that MongoDB acquired in 2014 — was designed for a world where most databases fit in memory or on a single SSD. Per-document compression was clever: it kept individual documents small, which made queries fast. But it had a hidden cost. Luna: The cost being that rebalancing meant decompressing and recompressing every single document on the move, right? Lucas: Exactly. When you rebalance a shard, you're copying data from one node to another. With per-document compression, the source node has to decompress each document, serialize it over the wire, and the target node has to recompress it. On a 12-terabyte shard, that's trillions of individual compression operations. Luna: And compression is cpu bound. So you're tying up compute that could be serving queries, all just to move data around. Lucas: Right. The MongoDB team benchmarked it: at 8 terabytes per node, rebalance time started growing super-linearly. At 12 terabytes, you're looking at two full days of near-total resource contention. The customer who reported that was running a financial analytics platform — they couldn't add new shards without a maintenance window that lasted a weekend. Luna: And that's the kind of pain that forces a rewrite. What was their alternative? Just throw more nodes at it? Lucas: You could, but that just moves the problem. More nodes means more rebalancing events. The team realized they needed to change the compression boundary: instead of compressing individual documents, compress whole pages of data at once. That way, when you move a page, it stays compressed — no decompress-recompress cycle. Luna: So the trade-off is that you lose some compression ratio — a page contains more internal fragmentation than a single document. Lucas: Exactly. They measured about a 15 percent increase in storage cost per gigabyte. But in exchange, rebalance times dropped from 47 hours to about 70 minutes for that same 12-terabyte shard. That's a 40x improvement. Luna: And 70 minutes is still not great, but it's within a single maintenance window. That changes the operational model completely. Lucas: It does. The team also rewrote the write-ahead log. The old WAL logged every document modification individually. On write-heavy workloads, the WAL could grow to hundreds of gigabytes and replay time became unbounded. They switched to a block-based WAL that logs page-level changes. Luna: That's a much more natural boundary — page size is fixed, so replay time becomes predictable, regardless of how many documents you touch. Lucas: Predictability was actually the north star for the whole rewrite. The lead engineer on the project told me they wanted to eliminate 'unknown unknowns' from the operational model. If you have a 10-terabyte shard, you should be able to say, 'this rebalance will take about an hour,' not 'somewhere between 2 and 48 hours.' Luna: I remember when I worked at a fintech startup — we had a 6-terabyte MongoDB cluster, and every time we added a node, the ops team would literally schedule a war room. Because you never knew if it would take 3 hours or 30. Lucas: That uncertainty is worse than the downtime itself. You can plan for a two-hour window. You can't plan for 'maybe the whole weekend.' So the MongoDB team delivered this in two phases. Phase one was the compression change — that shipped in MongoDB 6.0 in 2024. Phase two was the WAL rewrite — that's rolling out in 7.0 later this year. Luna: Two years of work, and most users will never notice it, except that their databases stop breaking in weird ways. Lucas: That's the paradox of storage engine work — the better it is, the less anyone notices. But for the teams running these clusters at scale, it's the difference between a system you trust and one you dread touching. Luna: So what's the lesson for other engineering leaders? When do you decide to do a ground-up rewrite versus patching around the problem? Lucas: The MongoDB team had a clear signal: a single metric — rebalance time — that was breaking their SLA with customers. And they had data showing it was a fundamental compression boundary issue, not something they could optimize away. If you can point to one number that's gone nonlinear and you understand the root cause, a rewrite is often cheaper than a series of band-aids. Luna: That's a good heuristic. If the fix involves changing the core contract between storage and compute, you're probably looking at a rewrite. Lucas: And the MongoDB team was smart about it — they didn't try to rewrite everything at once. They shipped the compression change first, got feedback, then tackled the WAL. Two separate releases, each with its own rollback plan. Luna: Incremental rewrites. The most counterintuitive thing in engineering — you're rebuilding the foundation while the house is still standing. Lucas: And that's the real skill. The team had to maintain backward compatibility with existing WiredTiger files throughout the transition. They built a migration path where old and new page formats could coexist on disk. That's not trivial when you're changing the fundamental unit of compression. Luna: Did they have to support cross-version replication? Like, a node running the old format talking to a node running the new one? Lucas: Yes, and that was the hardest part. They added a protocol version negotiation at the replica-set level. When you upgrade, nodes can detect which version their peers support and fall back to a common format. It adds some complexity to the wire protocol, but it means zero-downtime upgrades. Luna: That's exactly the kind of detail that separates a production-grade rewrite from a research project. Lucas: It is. And the result is that MongoDB can now credibly support shards up to 50 terabytes per node — which was their internal target. The old architecture topped out at around 15 before things got painful. Luna: Fifty terabytes per node — so a three-node replica set can hold 150 terabytes, and rebalancing is still predictable. Lucas: Exactly. And the storage cost increase of 15 percent is a small price to pay for that kind of operational predictability. Especially when you consider that most of those customers are already paying for SSDs and compute — the 15 percent is just a bit more storage spend, not a new line item. Luna: And if you're running at that scale, you're probably also running compression on the file system level anyway, so the actual overhead might be even less. Lucas: That's a good point. Many of their hyperscale customers use ZFS or btrfs with compression, so the page-level compression from WiredTiger is just one layer. The total overhead might be 5 to 10 percent in practice. Luna: So the rewrite wasn't just about fixing a pain point — it opened up a new scale tier that wasn't accessible before. Lucas: Right. And that's the kind of engineering that doesn't make headlines, but it makes the difference between MongoDB being a good choice for a startup and a viable choice for an enterprise with petabyte-scale data. Luna: It's almost like the opposite of 'move fast and break things.' They moved deliberately, broke things in a controlled way, and ended up with a more robust foundation. Lucas: The irony is that the old per-document compression was elegant — it was a beautiful piece of engineering for its time. But elegance doesn't scale. The new page-level approach is uglier — there's more internal fragmentation, more metadata overhead — but it scales predictably. That's the trade-off. Luna: And the lesson is: know when your elegance becomes a liability. Lucas: That's a great way to put it. For MongoDB, that moment came with a 47-hour rebalance. For your team, it might be a different number. But the key is to watch for those nonlinearities — they tell you when it's time to go back to the drawing board. Luna: And maybe, if you listen closely, your own infrastructure is already sending you that signal. Lucas: Alright — next time we'll look at how one team actually ran that migration in production without waking up on-call. Thanks for listening.