Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How MongoDB Rebuilt Its Storage Engine for Planet-Scale Writes
Transcript
- Lucas: So MongoDB has been running on WiredTiger as its default storage engine since 2014 — a mature, battle-tested B-tree engine. But a couple of years ago they quietly started a project to replace it from the ground up with something they're calling the LSM engine. Luna: LSM meaning log-structured merge-tree, right? That's the same family of data structures that underpins systems like Bigtable, Cassandra, RocksDB. Lucas: Exactly. And the reason they gave is interesting. WiredTiger was designed for a world where SSD and NVMe drives were already fast and random I/O was cheap. But MongoDB's growth — especially in IoT, time-series, and event-logging workloads — exposed a fundamental mismatch. WiredTiger's B-tree does fine on reads, but write-heavy workloads cause cascading page splits, fragmentation, and a ton of write amplification. Luna: So they're swapping out an engine that's been in production for over a decade. How do you even begin that conversation as a CTO? 'Hey team, let's rewrite one of the most performance-critical pieces of our stack'? Lucas: That's exactly the leadership question. I spoke with a former MongoDB engineer who worked on this migration. He said the key was that they didn't frame it as a rewrite. They framed it as a new storage engine option — not replacing WiredTiger, but offering a second engine for workloads WiredTiger wasn't purpose-built for. That changed the risk calculus internally. Luna: Smart framing. It also means they can run both engines in the same cluster, migrate gradually per replica set or per collection. That's a huge operational hedge. Lucas: Right. And technically, the new engine is closer to LevelDB or RocksDB in design. Data is written to an in-memory memtable first, flushed to immutable SSTable files on disk, and then periodically merged in the background. That sequential write pattern dramatically reduces write amplification compared to B-tree's random page updates. Luna: But the trade-off is read performance, isn't it? LSM trees can have higher read latency because you might have to check multiple levels of SSTables to find a key. Lucas: That's the classic trade-off, and MongoDB addressed it with a couple of tricks. One is a bloom filter per SSTable — a probabilistic data structure that quickly tells you 'this key is not in this file.' So the reader can skip entire files without opening them. The other is a row-based approach they call 'delta encoding' — they store multiple versions of a document within the same SSTable and use a base version plus diffs, which reduces the number of files you need to merge. Luna: So they're not just swapping an engine, they're redesigning the compaction strategy to minimise the read penalty. That's the kind of deep systems thinking that separates a migration from a rebuild. Lucas: Exactly. And the migration itself was phased over two major releases. In the first release, the LSM engine shipped as an optional pluggable storage engine — you could enable it on a per-collection basis. That let early adopters run it on non-critical workloads, shake out bugs. In the second release, they made it the default for new clusters, but WiredTiger remained fully supported and users could opt out. Luna: That's a textbook gradual rollout. But what about the operational burden? Operations teams now have to understand two engines, two sets of tuning knobs, two monitoring dashboards. Lucas: It's a real cost. MongoDB did publish a compatibility guide and a set of recommended defaults — basically hiding the complexity behind sensible defaults for 90 percent of use cases. But the engineer I spoke to admitted that for the first year, support tickets about write latency doubled because people were migrating collections without understanding the compaction interval. Luna: That's the kind of thing every engineering leader dreads — a performance regression that's actually a configuration mismatch, not a bug. Lucas: And it led to a really interesting outcome. MongoDB realised they needed to instrument the engine better — expose compaction metrics, write amplification ratio, number of SSTable levels — inside their cloud monitoring service, Atlas. So the migration actually pushed them to improve observability across the board. Luna: I love that. A necessary technical migration becomes an opportunity to fix the blind spots you didn't know you had. Lucas: Yeah. And this is where I think the story is most useful for CTOs listening. When you're planning a foundational infrastructure change, think about what secondary improvements you can bundle into the same initiative. MongoDB didn't set out to rebuild their monitoring, but the LSM migration forced them to. Luna: So let's talk numbers. What kind of write throughput improvement did they see in practice? Lucas: The internal benchmarks they published show about 3x write throughput on workloads with 80 percent writes, 20 percent reads — the typical IoT ingestion pattern. Write amplification dropped from about 12x on WiredTiger to about 4x on the LSM engine. That means your SSDs last longer and your iops cost goes down. Luna: Three x is dramatic. But I wonder — are there workloads where WiredTiger still wins? High-read, low-write, point lookup heavy? Lucas: Absolutely. MongoDB's guidance is that if your workload is 90 percent point reads and 10 percent writes, stick with WiredTiger. The B-tree gives you a single seek per read. The LSM engine might need to check two or three files even with bloom filters. Luna: So the right answer is 'it depends,' and the CTO's job is to make sure the team has the tools to measure which one is better for their use case. Lucas: Exactly. And that brings us to a broader pattern I've noticed. When a database like MongoDB invests in a second storage engine, it's acknowledging that one size doesn't fit all. Compare that to the relational world, where Postgres and MySQL have largely stuck with one engine. The NoSQL world is more pragmatic about offering choice — but choice comes with cognitive overhead. Luna: And the choice isn't just technical — it's organisational. Do you have the expertise to tune an LSM tree? Do you have the operational maturity to handle two engines? For a startup, probably not. For a mature engineering org with a dedicated database reliability team, absolutely. Lucas: That's the framing I'd want every CTO to walk away with. Not 'should I migrate to LSM?' but 'what is my team ready to operate?' MongoDB's migration succeeded because they made the new engine optional, instrumented it deeply, and gave the community time to adapt. Luna: Speaking of community — I know this show is ad-free and listener-supported. We get a lot of questions about how we keep it that way. And honestly, a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that literally funds the production of this many episodes. Lucas: It's true. No ads, no sponsors, no tracking. Just people who find these conversations useful and want to keep them coming. It's a small group already, but it makes a real difference. Luna: Yeah. Anyway — back to storage engines. One thing I wanted to ask about the LSM engine is how they handle consistent backups. If you're writing sequentially and compacting in the background, how do you ensure a consistent snapshot? Lucas: Great question. MongoDB uses a concept called 'checkpointing.' The engine periodically writes all in-memory memtables to immutable SSTables and then fsyncs a checkpoint record. Backups can read from that consistent checkpoint without locking. It's actually simpler than WiredTiger's checkpoint mechanism because LSM trees are append-only — no in-place updates to roll back. Luna: So the architectural simplicity of append-only pays dividends not just on write throughput but on operational tasks like backup and replication. Lucas: Exactly. And that's a lesson that extends beyond databases. When you're designing any system that needs to be highly available, think about whether you can make your core data structure append-only. It often simplifies crash recovery, replication, and observability. Luna: I think that's the kind of insight our listeners can apply next week, not just something to file away. Lucas: Absolutely. And if you want to dive deeper into MongoDB's LSM engine, their engineering blog has a series of posts with detailed benchmarks and trade-off discussions. We'll link it in the show notes.