Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Rebuilt Its Sync Engine for 100 Million Blocks
Transcript
- Lucas: If these conversations are useful for what you're building or running, we just might have the right episode today. Notion recently rebuilt its sync engine from the ground up to handle over 100 million blocks of content. That is a lot of nested bullets and databases. Luna: Hundred million blocks. And if you've ever used Notion, you know that block is their atomic unit—a paragraph, an image, a to-do. That's a massive graph to keep in sync. Lucas: Right. The old sync engine was basically a monolithic blob—it would send the entire page as one JSON payload on every change. That worked when pages were small, but as Notion grew, pages got bigger. Some power users were hitting pages with ten thousand blocks, and the sync latency was brutal. Luna: So they moved to what—delta sync with CRDTs? Conflict-free replicated data types, for anyone listening who hasn't geeked out on that yet. Lucas: Exactly. They switched to a crdt based delta sync, where only the changed block gets transmitted, and the client merges it locally. The key insight was that most edits are small—a typo fix, a drag and drop reorder. You don't need to resend the entire page. Luna: And CRDTs give you eventual consistency without a central conflict resolver. That's a big deal for a product that needs to work offline. Lucas: Yeah, offline was actually one of the main drivers. Notion's offline mode was notoriously flaky because the old sync couldn't merge conflicts well. With the new engine, they designed for offline-first from the start. The CRDT structure means each block has a unique ID and a version vector, so concurrent edits from different devices just merge. Luna: I remember Simon Last, Notion's co-founder and now interim CTO, gave a talk about this at some internal offsite. He said the old sync engine was 'held together by duct tape and prayer.' Lucas: He did. And it's interesting because Simon stepped into the CTO role specifically to oversee this rebuild. Before that, Notion didn't really have a dedicated CTO—the engineering team was small enough that the founders could just make decisions. But once you're syncing a hundred million blocks daily, you need a formal architecture. Luna: So what was the actual performance improvement? I've heard numbers like 40% reduction in sync latency. Lucas: That's right. Internal benchmarks showed that for pages with over five thousand blocks, the time to sync a single edit dropped from about 800 milliseconds to under 200. That's the difference between feeling snappy and feeling laggy. Luna: And that matters because Notion's whole value prop is that it's a fast, fluid canvas. If it lags, people go back to Google Docs or Roam. Lucas: Precisely. The other big change was in bandwidth. The old full-page sync for a ten thousand block page could be a 2 megabyte payload. The delta sync is typically under 10 kilobytes per edit. That's huge for mobile users on limited data. Luna: I want to dig into the trade-offs. CRDTs are known for being memory-intensive—each node stores a lot of metadata. How did Notion handle that? Lucas: They did two things. First, they used a specific CRDT variant called RGA—replicated growable array—which is optimized for list-like structures like blocks in a page. Second, they pruned version vectors aggressively. If a block hasn't been edited in 30 days, they compress its vector into a single timestamp. That cut memory overhead by about 60%. Luna: So they traded some consistency metadata for storage, but kept the conflict resolution guarantees. Smart. Lucas: Yeah. And the engineering team had to rewrite the entire client-side sync layer—that's the iOS, Android, and web apps. That took about 18 months from design to full rollout. They did a gradual migration: first 1% of users, then 10%, then everyone. They had a kill switch in case things went wrong. Luna: Did anything go wrong? I recall some early reports of duplicate blocks appearing for a small subset of users. Lucas: There was a bug in the merge logic for nested blocks under heavy concurrent edits. A user on a plane making changes offline while another user on the same page online—when the offline user came back, some blocks would duplicate. They fixed it within 48 hours by adding a deduplication pass on the server side. Luna: That's actually a great example of how CRDTs aren't magic. They handle basic conflicts, but edge cases in complex data models still bite you. Lucas: Totally. Notion's data model is particularly tricky because blocks can contain other blocks—a to-do inside a toggle inside a column. That nesting creates a tree structure that CRDTs don't handle natively. They had to layer a custom reconciliation on top. Luna: So what's the takeaway for other engineering leaders thinking about adopting CRDTs? Is this something every collaborative app should do? Lucas: I'd say CRDTs are great if you need offline-first and low latency, but they add complexity. For Notion, the payoff was clear because their core product is built on real-time collaboration. But if you're building something like a CRM where edits are mostly sequential, a simpler operational transform approach might be fine. Luna: Operational transform—like Google Docs uses. That's the classic approach, but it requires a central server to order operations. Notion wanted to avoid that single point of failure. Lucas: Right. And actually, Notion's new engine also supports peer to peer sync for local networks—like if you're in an office without internet, devices on the same LAN can sync directly. That's a feature they couldn't have done with the old architecture. Luna: Interesting. I didn't know that. So this rebuild wasn't just about performance—it enabled new product capabilities. Lucas: Exactly. And that's the mark of a good architecture change. If the only benefit is 'faster,' you might not justify the engineering cost. But if it unblocks new features—offline, LAN sync, better conflict resolution—then it's a strategic investment. Luna: We should note that Simon Last has since stepped back from the CTO role. Notion hired a permanent CTO from outside—a Microsoft veteran. But the sync engine rebuild was his last big architectural stamp. Lucas: Yeah. And it's a good case study in how a founder cto can come in, fix a core technical debt, and then hand off to a professional engineering leader. Not every company needs a permanent CTO from day one. Luna: If today's deep dive into Notion's sync engine was useful for how you think about scaling your own product, that's exactly the kind of value we try to bring each week. And the way these episodes stay ad-free is listener support—buy me a coffee dot com slash fexingo. Lucas: Yeah, it really helps keep the conversations focused on the substance, not the sponsors. So if you got something out of this, consider throwing a coffee our way. Luna: Alright, back to the architecture. One last thing I want to ask—how did Notion manage the developer experience during the migration? Eighteen months is a long time to maintain two sync engines. Lucas: They ran both engines in parallel for about six months. Feature flags toggled which engine each user used. The engineering team had to maintain the old engine's bug fixes while building the new one. They said it was painful but necessary. Luna: And did they lose any engineers over it? Major rewrites sometimes cause turnover. Lucas: A few people left, but mostly because they wanted to work on product features, not infrastructure. Simon was transparent about the timeline and the reasons. The team that stayed was highly motivated. Luna: That's the cultural side of technical leadership—keeping people aligned during a long rebuild. Lucas: Yeah. And ultimately, the rebuild was a success. Notion's sync latency is down, offline works reliably, and they've laid the groundwork for real-time multiplayer features that they're rolling out now. It's a blueprint for any company that outgrows its first sync architecture. Luna: And that's a good note to end on. Next episode, we'll look at how another company tackled a similar scale problem—but in a completely different domain.