Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Rebuilt Its Sync Engine for Offline-First
Transcript
- Lucas: If you are building or running a product where people expect their data to just be there — on the phone, on the laptop, on a train with no signal — then you know the sync problem is one of those deceptively hard ones. Notion just finished a major rebuild of how they handle that, and I think there is a lot to unpack. Luna: This is the block-based sync engine they wrote about in their engineering blog a few weeks ago, right? The one that moves from optimistic local writes to a hybrid model? Lucas: Exactly. So Notion stores everything as blocks — a paragraph, an image, a database row — each block has a unique ID, and the whole document is a tree of blocks. The old architecture was straightforward: you made a change, it wrote to the local store immediately, then sent the delta to the server, and the server broadcast it to other clients. That is optimistic local-first in a nutshell. Luna: And the problem? Conflicts, presumably. Two people editing the same block offline. Lucas: Yes, but also something more fundamental — the local store and the server could get out of sync in subtle ways. Notion used a last write wins strategy for conflicts, which worked fine for simple text edits, but when you have database views, formulas, linked databases — the block tree has interdependencies. A change in one block could cascade, and last write wins would sometimes produce inconsistent state. Luna: I remember reading about one specific bug: if you deleted a block while offline, and someone else on the server edited a child of that block, the reconciliation would sometimes resurrect the deleted block but with the edit applied. Very spooky. Lucas: That is exactly the kind of edge case that drove the rebuild. So the team decided to redesign the sync engine around what they call a 'hybrid local-sync' model. The core insight: instead of treating the local store as a cache of the server state, treat it as an authoritative replica that can diverge temporarily, but with a much richer conflict resolution layer. Luna: Meaning they moved from simple last write wins to something more like operational transform, or CRDTs? Lucas: They went with operational transform, but with a twist. Standard OT works well for linear text, but Notion's block tree is not linear. So they built a custom OT layer that understands the block hierarchy. Each operation is tagged with its position in the tree, and the transform function knows how to merge operations that affect parent and child blocks. They also added a version vector per document, so each client knows exactly which operations it has seen. Luna: That sounds like it adds a lot of complexity on the client side. How big was the engineering effort? Lucas: The blog post says it took about 18 months from initial design to full rollout, with a team of around 12 engineers at peak. They rewrote the client-side sync module in Rust — that part is interesting — and compiled it to WebAssembly for the web client, and used native bindings for mobile. Rust gave them the performance and memory safety they needed for the OT logic, which runs on every keystroke. Luna: And the migration? They have millions of daily active users. You cannot just flip a switch. Lucas: Right. They used a feature flag system and gradually enabled the new sync engine for a small percentage of users, watching for sync errors and latency regressions. One key metric: the old engine had about a 0.5 percent sync error rate — meaning an operation that failed to sync and required manual resolution. The new engine brought that down to under 0.3 percent. That is a huge improvement at Notion's scale. Luna: And latency? Users are very sensitive to how fast their edits appear on other devices. Lucas: They actually saw a slight improvement — median sync latency went from about 200 milliseconds to 180 milliseconds. But the bigger win was in consistency: the new engine eliminated the 'ghost block' bugs where deleted blocks would reappear, and the 'phantom edits' where an edit would show up on one device but not another. Those were the most common user complaints. Luna: That is a good moment to mention something — this show is listener-supported, no ads. A handful of people chip in each month through buy me a coffee dot com slash fexingo, and that literally is what funds making this many episodes. If you find the technical deep dives useful, that is a simple way to keep them coming. Lucas: Yeah, and it genuinely matters. We don't have to chase ad revenue, so we can pick the angles we think are most interesting. Thanks to everyone who already does that. Anyway, back to Notion — one other design decision I found clever: they introduced a 'reconciliation log' on the server that records every operation applied to a document. Luna: Similar to a write-ahead log in databases? Lucas: Exactly. When a client comes back online after being offline for a while, it sends its version vector to the server, and the server replies with all the operations the client missed, compressed and batched. The client then replays those operations through the OT engine to bring its local state up to date. That avoids the need for a full document sync every time. Luna: What about peer to peer sync? Two clients on the same local network, both offline from the internet — can they sync directly? Lucas: Notion considered that, but it is not part of the current architecture. The design assumes that all sync goes through the server, even if the server is just a local relay. The team argued that server-mediated sync simplifies conflict resolution because there is a single source of truth for ordering. True peer to peer would require a consensus protocol or a CRDT approach, which adds complexity. So for now, it is server-authoritative but with robust offline support. Luna: That makes sense. But what about privacy? If everything goes through the server, the server sees all operations. Notion has enterprise customers who care about that. Lucas: They addressed that with end to end encryption for the block content — though the block structure and metadata are not encrypted, because the server needs to understand the tree to do OT. So it is a trade-off. They also offer on-premise deployment for large enterprises, where the server runs in the customer's VPC. In that case, the sync engine is the same, but the data never leaves the customer's infrastructure. Luna: And how does the new engine handle large documents? Some Notion pages have thousands of blocks. Lucas: They optimized the OT engine to handle tree operations in logarithmic time relative to the depth of the tree, not the total number of blocks. So editing a deeply nested block is just as fast as editing a top-level one. They also added lazy loading for parts of the document that are not in view, so the client only loads the blocks it needs. That was already present in the old engine, but the new one makes it more aggressive. Luna: I want to ask about testing. How do you test something like this? You cannot simulate every possible offline scenario. Lucas: They built a sync simulator that generates random sequences of operations across multiple virtual clients, with random network partitions and reconnections. The simulator runs millions of scenarios and checks that the final state on all clients is consistent. They also have fuzz testing that feeds the OT engine with malformed operations to see if it crashes. And they dogfooded it heavily — Notion employees used the new engine for months before the public rollout. Luna: So what is the takeaway for other builders? If you are thinking about offline-first for your app, what should you consider? Lucas: First, understand your data model. Notion's block tree is hierarchical, so OT made sense. If your data is more like a graph or a simple key-value store, CRDTs might be simpler. Second, invest in the conflict resolution logic early — do not rely on last write wins unless you are okay with occasional data loss. Third, test with real network conditions. Notion's simulator was key. And fourth, consider the client-side language carefully. Rust or a similar systems language might be worth it for performance-critical sync code. Luna: Any known limitations of the new engine? Every system has trade-offs. Lucas: One limitation is that the OT engine does not handle concurrent moves of the same block well. If two people drag the same block to different parents while offline, the reconciliation picks one arbitrarily. Also, the system does not support offline creation of linked databases — that requires a server round-trip to establish the link. And the reconciliation log grows unboundedly for active documents, so they had to implement log compaction that snapshots the state periodically. Luna: Compaction intervals? Lucas: They do it every 1000 operations per document, or every 24 hours, whichever comes first. The snapshot is stored on the server, and clients can download it instead of replaying the entire log. That keeps the sync fast for new clients joining a document. Luna: It is impressive they pulled this off without a major outage. I remember when Evernote tried something similar years ago and had weeks of instability. Lucas: Notion learned from those examples. They had a very conservative rollout: started with 1 percent of users, then 5, then 20, then 50, then 100 over three months. They also had a kill switch that could instantly revert all users to the old engine if something went wrong. They only had to use it once, during the first week, when they discovered a race condition that caused duplicate blocks. They fixed it in six hours and resumed. Luna: And now the new engine is the default for all users? Lucas: As of last month, yes. The old engine is fully deprecated. The engineering team is now working on the next challenge: real-time collaboration with multiple cursors, similar to what Google Docs has. That requires a different set of trade-offs, and they are building on top of the new sync engine. Luna: That will be fun to watch. Multi-cursor with offline support — that is the holy grail. Lucas: Exactly. And I think Notion's approach shows that with a solid sync foundation, you can add those features incrementally. The hard part is getting the foundation right.