Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Scaled Its Real-Time Sync Engine
Transcript
- Lucas: You open Notion, start typing a line in a database view, and within a second your teammate in another time zone sees the same character appear. That seems almost boring now — but getting there took a total rebuild of how their sync engine worked. Luna: And what were they doing before the rebuild? A lot of 'save and refresh', I imagine. Lucas: Pretty much. Early Notion used a simple polling model — every few seconds the client would ask the server 'any changes?' and pull down a full snapshot. It worked fine for solo use, but as soon as teams started editing the same page, you'd get merge conflicts, lost edits, people overwriting each other's work. The support tickets piled up. Luna: So they decided to invest in a proper real-time sync layer. That's when they started looking at CRDTs — conflict-free replicated data types. Lucas: Exactly. CRDTs are a family of data structures that allow multiple users to edit the same content independently and then merge those edits without a central coordinator. No locking, no conflict resolution server. Each client applies operations locally and propagates them asynchronously. As long as the data type is designed right, all replicas converge to the same state. Luna: But Notion isn't just plain text — it's blocks, databases, rich text, inline code, embeds. How do you define a CRDT for a block-based document? Lucas: That's the hard part. Notion's content model is basically a tree of blocks — each block has a type, properties, and children. They built their own CRDT called 'BlockSet' where each block is identified by a globally unique ID, and the operations are insert, update, and delete. The trick is that deletions are tombstones — you never truly delete data, you mark it as removed, otherwise concurrent insertions could re-create a block that someone else deleted. Luna: So tombstones pile up over time. That must affect storage and memory. Lucas: It does. Notion uses a log-structured merge tree for storage — every edit appends to a log, and a background compaction process merges tombstones and old versions. But they also have a garbage collection mechanism that, after a certain number of days, actually prunes tombstones if all replicas have acknowledged the deletion. That's a carefully orchestrated handshake. Luna: And what about ordering? If two people edit the same block at the same time, how do they decide which edit wins? Lucas: CRDTs avoid 'last writer wins' because that can silently drop data. Instead, Notion uses a hybrid logical clock — a combination of a timestamp and a node ID. Each edit carries a timestamp, but the clock is monotonic even across restarts. If two edits have the same timestamp, the node ID breaks the tie. But the real mechanism is that edits are commutative: for a rich text block, two concurrent insertions at the same position are both preserved by using fractional indices — they assign each character a position like 'a', 'b', between them you get 'ab', and so on. No data loss. Luna: That's elegant. But it also means the sync protocol has to be efficient — you can't send the full CRDT state on every change. Lucas: Right, they use a delta-based approach. Each client sends only the operations it hasn't yet broadcast. The server acts as a relay — it receives deltas from one client, merges them into its own state, and then sends the new deltas to all other subscribed clients. But here's the critical detail: the server does not resolve conflicts. It just forwards. All conflict resolution happens on the client side using the CRDT merge rules. Luna: That must make the server architecture simpler and more scalable. Lucas: Much simpler. The server is essentially a stateful router. It holds a copy of the document state per workspace, but it doesn't need to understand the semantics of the blocks. They use WebSockets for the real-time channel, with a fallback to long-polling on networks that block WebSockets. And they have a heartbeat mechanism to detect disconnections — if a client drops off, the server buffers deltas for up to 30 seconds. When the client reconnects, it sends its latest clock timestamp, and the server replies with any deltas it missed. Luna: But what happens if a client is offline for longer — say, an airplane mode? They'd have a huge backlog of operations. Lucas: That's where the 'offline-first' design kicks in. The client stores all local operations in a local database — IndexedDB in the browser. When it comes back online, it replays them in order. If the CRDT merge produces a conflict — for instance, you edited a block offline that someone else deleted — the offline edits are not lost. They become orphan blocks that appear at the bottom of the page, and the user can decide what to do with them. Notion chose to never silently discard data. Luna: That sounds like a deliberate tradeoff for user trust over pristine data structures. Lucas: Exactly. And it's paid off. But even with all that design, there was a major incident in 2021 — a 48-hour data loss bug that affected some users. The root cause was a race condition in the tombstone compaction logic. The compaction process ran on a schedule and purged tombstones that it thought were fully replicated, but due to a network partition, some replicas hadn't actually acknowledged the deletion. So a block that was deleted on one client re-appeared on another, and vice versa. Users saw pages with random blocks appearing and disappearing. Luna: How did they fix it? That sounds like a fundamental flaw in their consensus model. Lucas: They added a two-phase acknowledgment protocol for tombstone pruning. Now, before any tombstone is removed, the server must receive a confirmed acknowledgment from every replica that has been active within the last 48 hours. If a replica hasn't checked in, the tombstone stays. They also introduced a 'safety period' — no tombstone younger than seven days is ever pruned. And they built a formal verification model of the compaction logic using TLA+ to catch similar race conditions. Luna: TLA+ again. It's become the go-to tool for proving distributed systems correctness. Lucas: It's an investment, but when your entire product promise is 'your data is safe and instant', you can't afford to guess. Notion's engineering team now runs every new sync feature through a TLA+ model before writing a line of code. They also have chaos engineering tests that simulate network partitions, clock skews, and delayed messages. Luna: Let's talk numbers. What's the scale they're handling now? Millions of documents, millions of blocks per document? Lucas: As of early 2026, Notion has over 100 million users and processes billions of block operations per day. The largest single workspace has over 10 million blocks — that's a document tree with hundreds of thousands of pages. The sync engine keeps the median end to end latency under 100 milliseconds for the 95th percentile. That's from when a keystroke is made on one client to when it appears on another. Luna: And they're doing all this without a dedicated sync team. I've read that the sync engine is maintained by the core platform group, which also handles auth, storage, and search. Lucas: That's right. They keep the team small — about 12 engineers — but they've built a culture of deep ownership. Each engineer owns a component end to end: the CRDT algorithm, the WebSocket server, the offline storage, the compaction logic. They run their own load tests and on-call rotations. It's a sharp contrast to the 'move fast, break things' approach some other collaboration tools took in their early days. Luna: It's worth noting that Notion's sync engine is closed source. But the CRDT concepts they use are well documented in research papers — specifically the 'LogootSplit' algorithm for text and the 'RGA' algorithm for ordered lists. Lucas: Exactly. Any team building a collaborative app today can learn from their approach. The key lessons: start with a clear data model, invest in formal verification for conflict resolution, design for offline-first from day one, and never delete data without a consensus protocol. Luna: Speaking of learning from good engineering — and I know this is a bit meta — but the fact that we can have these deep dives every week comes down to a small group of listeners who support the show directly. No ads, no sponsors. Just people who find value in these conversations and chip in a few bucks a month on buy me a coffee dot com slash fexingo. Lucas: Yeah, and it genuinely makes a difference. It covers hosting, research time, and lets us keep the show ad-free and focused on the technical decisions that actually matter. If these episodes help you think about your own engineering challenges, that's exactly the kind of listener support that keeps it sustainable. Luna: Right. And we're not asking everyone — just those who can and want to. Every contribution goes right back into making the show better. Lucas: So back to Notion — one more thing that impressed me. When they rebuilt the sync engine, they didn't do a big bang migration. They gradually rolled out the new protocol to a small percentage of workspaces, monitored latency and error rates, and only increased the rollout when they had high confidence. That took almost a year from the first prototype to full production. Luna: Patience really is a competitive advantage in infrastructure. It's easy to underestimate the complexity of 'just syncing' until you're the one debugging a merge conflict at 2 AM. Lucas: Yeah, and that's the thing — the magic of Notion's sync is that you never think about it. And that invisibility is the result of years of careful engineering tradeoffs.