Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Rebuilt Its Sync Engine for 100 Million Blocks
Transcript
- Lucas: So Notion has become the default productivity tool for a lot of engineering teams — it's basically a wiki, a project tracker, and a database all in one. But what most users don't think about is what happens when you drag a block from one page to another, and your teammate on the other side of the world sees that move happen in real time. Luna: Right — the sync engine under the hood. And Notion recently rebuilt that engine from scratch. I saw a talk from one of their infrastructure engineers, and the numbers are wild. They're now handling over 100 million blocks per workspace. Lucas: Yeah, a hundred million blocks. That's a lot of nested to-do lists and embedded databases. The old system was a simpler json based sync — every time you made a change, the whole page's JSON would be serialized and sent to the server, which then broadcast it to all connected clients. Luna: Which works fine when you're the only one editing. But the moment two people edit the same page — say one person adds a heading, another reorders a bullet — you get conflicts. Lucas: Exactly. And Notion's old approach was basically last write wins. So if you and I both edit the same block, the last person to save would overwrite the other's work. That's fine for a personal wiki, but Notion was being used by teams of fifty, a hundred people editing the same project page. Luna: So they needed something like Google Docs' operational transformation or Figma's CRDT approach. They went with CRDTs — conflict-free replicated data types. Lucas: That's right. CRDTs allow each client to make changes independently and then merge them later without a central coordinator. Figma uses a custom CRDT for its multiplayer design engine. Notion built something similar but tuned for a block-based document model. Luna: What was the key design decision? I mean, there are off-the-shelf CRDT libraries — why build custom? Lucas: Great question. The standard libraries — like Automerge or Yjs — are generic. They work well for plain text or simple JSON structures. But Notion's data model is deeply hierarchical: you have pages, blocks within pages, properties within blocks, comments, relations between blocks. A generic CRDT would either be too slow or too memory-intensive at scale. Luna: So they designed a block-specific CRDT. Each block becomes an independent CRDT instance, and blocks are connected via parent-child references. That way, a change to one block doesn't require merging the whole page. Lucas: Precisely. They also introduced something called 'block-level version vectors' — essentially each block tracks its own version history. When you edit a bullet point, only that bullet's version vector is sent to the server, not the entire page. That cuts network payload significantly. Luna: And the server? It doesn't need to resolve conflicts anymore — it's just a relay and a persistent store. The clients do all the reconciliation. Lucas: That's the beauty of CRDTs. The server becomes almost stateless in terms of conflict resolution. But it does introduce a new challenge: cache invalidation. If every client has its own local copy of the block tree, how do you ensure that stale data gets evicted? Luna: Right — because if I'm offline for a bit and come back, my local CRDT might have a diverged state. Notion's solution, as I understand it, is a technique called 'operation-based merge with causal history.' They embed a hash of the entire block's history in every operation. If a client's hash doesn't match the server's, the client knows it needs to fetch the latest state. Lucas: That's smart. They also implemented a 'recently used' cache on the client that invalidates blocks not accessed in the last 15 minutes. For actively edited pages, the blocks stay in memory. For archived pages, they get evicted and fetched from the server on next access. Luna: The engineering cost here is non-trivial. They had to rewrite the entire sync layer, which meant migrating every existing workspace from the old json based model to the new CRDT model. That's a migration that affected millions of users. Lucas: Yeah, and they did it incrementally. They ran both engines in parallel for months, verifying that the new engine produced the same final state as the old one. Only once they had high confidence did they flip the switch. Luna: And the performance improvement? The latency for a typical sync operation dropped from about 200 milliseconds to under 50. For 99th percentile, it went from over a second to about 150 milliseconds. Lucas: Those numbers matter, especially for real-time collaboration. When you're in a meeting and someone adds a comment, you want to see it within a heartbeat, not after a coffee break. The new engine also reduced server bandwidth by about 40 percent because they're sending smaller, targeted updates. Luna: But here's the question I keep circling back to: Did Notion over-engineer this? I mean, how many users actually collaborate on the same page simultaneously? For most individuals, the old sync was fine. Lucas: That's a fair challenge. I think the answer lies in Notion's product trajectory. They're moving upmarket — targeting enterprise teams at places like Airbnb and Slack. Those teams have dozens of people editing the same project tracker. Without real-time conflict resolution, Notion would hit a ceiling. Luna: So it's a bet on future usage patterns. They're essentially buying insurance against a class of problems that would become showstoppers if they waited. Lucas: Exactly. And the rebuild also unlocked new features. For example, offline editing — you can now edit blocks on a plane, and when you reconnect, the CRDT merges your changes. That wasn't possible with the old json based sync. Luna: I also think it changed the team structure. Notion's engineering team grew from about 3 people when they started to over 60 during this rebuild. That kind of infrastructure work attracts top talent. Lucas: It does. And speaking of sustaining that kind of deep engineering work — it's one of the reasons shows like this exist. We get to peel back the hood on systems that most people just use. And if these conversations are useful for what you're building or running, it's worth mentioning that listener support is what keeps the podcast ad-free and independent. Luna: Yeah, it genuinely makes a difference. A couple of dollars a month, if you can spare it, goes toward covering hosting, research, and the time we spend digging into these topics. Buy me a coffee dot com slash fexingo is the easiest way. Lucas: Right — and we're back to Notion's architecture. I want to talk about one more aspect: how they handle block-level permissions within the CRDT model. Luna: That's a good one. Because not every user should be able to edit every block — you might want read-only access to certain pages. Lucas: Exactly. Notion's permission model is based on block ancestry. Each block has a parent page, and permissions are inherited from the page. But in a CRDT system, the client might have a local copy of a block it shouldn't have access to. So they added a server-side filter: the server will not relay operations for blocks the user doesn't have permission to edit. Luna: Which means the client can still have stale data, but it can't mutate it. And if the client tries to push an edit to a restricted block, the server silently drops it and sends back the correct state. Lucas: That's a clean design. It also means that the client never needs to fetch permissions for every block — it just tries an operation and the server enforces the policy. That reduces latency for legitimate edits. Luna: One thing I still wonder about: the cost of storing version vectors for 100 million blocks. That's a lot of metadata. Lucas: They actually compress the version vectors using a run-length encoding scheme. Since most blocks are edited infrequently, the vector is mostly zeros. They store only the delta — the changes since the block was last synced — and reconstruct the full vector on the server when needed. Luna: So the storage overhead per block is roughly a few hundred bytes, even with version history. That's impressive. Lucas: Yeah. And they also prune old versions after 90 days for blocks that haven't been accessed. That keeps the database from ballooning. They use a custom garbage collector that runs on a weekly schedule, identifying blocks with no recent operations and compacting their history. Luna: This is the kind of infrastructure work that doesn't make the product roadmap but makes everything else possible. It's like the plumbing in a skyscraper — you don't see it, but if it fails, the building is unusable. Lucas: That's a great analogy. And I think what Notion did is a blueprint for any product that needs to scale from personal tool to team platform. You have to invest in the sync layer early, even if only a small percentage of users need real-time collaboration today. Luna: So the takeaway: if you're building a collaborative product, consider CRDTs from day one if you can. Migrating later is painful but doable — and the payoff in latency and offline support is real. Lucas: Absolutely. And if you want to dive deeper, the talk by Notion's infrastructure team at their developer conference has all the gritty details — including the exact trade-offs they made on consistency vs. availability. We'll link to it in the show notes.