Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Built a Secure AI Sandbox
Transcript
- Lucas: If you have been watching the enterprise software landscape this quarter, you have noticed a shift in how companies are approaching artificial intelligence. It used to be about speed, now it is about safety. Specifically, how do you let your engineering team use large language models without accidentally leaking proprietary source code or customer databases into a public API? That is the exact problem Notion solved last year. Luna: Right, because the old way of just wrapping an external API call was never going to work for a document collaboration platform where trust is the entire product. So they built their own containment layer? Lucas: Exactly. They created what they call an internal inference sandbox. Instead of sending prompts to a third-party provider, they route them through a secured, isolated cluster of GPUs that only has access to non-sensitive model weights. The key detail here is that the sandbox is air-gapped from their production user data. If a developer wants to test a new feature using real customer documents, those documents stay on the secure side, while the inference happens on the safe side, with no direct line between them. Luna: That sounds like a massive operational overhead. Did they have to rewrite their entire backend to support this separation, or was it more of an infrastructure wrapper? Lucas: It was definitely an infrastructure wrapper first, but it forced some interesting changes in their coding standards. Essentially, every AI request gets tagged with a sensitivity level at the code level before it even hits the network. If the tag is high, the request is blocked entirely from the sandbox. It forces developers to think about data classification before they write a single line of logic. It slows down initial prototyping, but it prevents catastrophic leaks later. Luna: So it acts as a guardrail against developer error rather than just a technical firewall. I guess that makes sense when you consider that most breaches happen from human mistakes, not just system vulnerabilities. Lucas: Precisely. And the beauty of this approach is that it scales. As Notion added more AI features, like summarization and search, they didn't need to build new security protocols for each one. The sandbox handles the compliance part automatically. You just plug your model into the gateway, and the gateway enforces the rules. It turns security from a manual checklist into an automated property of the architecture. Luna: That is a smart way to handle complexity. But does this mean smaller companies can’t replicate this? It seems like you need serious capital to maintain that kind of GPU cluster just for testing purposes. Lucas: It is expensive, yes. But the principle is replicable even if the scale isn't. Smaller teams can use managed services that offer similar isolation, or they can implement stricter code-review policies that mimic the tagging system. The core lesson here is about shifting security left. You don't wait until launch to worry about data leakage; you build the constraints into the development environment itself. It changes the culture of engineering from move fast and break things to move carefully and verify everything. Luna: I see. So the cultural shift is actually the harder part than the technical implementation. Getting developers to accept friction in their workflow requires a lot of buy-in from leadership. Lucas: Absolutely. Leadership has to explain why the extra step matters. Notion’s CTO team made sure to frame it as enabling innovation rather than restricting it. They showed their engineers that this sandbox actually lets them experiment with riskier ideas because they know the baseline is safe. It turns security into an enabler, which is a much better message to give a creative team. Luna: That framing is crucial. If you just say no, people find a workaround. If you say here is a safe place to play, they actually use the tool. It reminds me of how game developers use staging servers to test patches without affecting live users. Lucas: Good analogy. It is all about controlled environments. Whether you are testing a game update or an AI model, you want to see what breaks before it breaks for your customers. Notion’s sandbox is essentially a staging server for intelligence. It allows them to iterate rapidly on the cognitive aspects of their product while keeping the core data vault completely sealed off. Luna: Do you think this will become a standard requirement for all SaaS platforms within the next two years? Or is it too niche for general-purpose apps? Lucas: I think it will become standard for any company handling sensitive intellectual property. For a note-taking app, it is essential. For a social media feed, maybe less so, but even there, data privacy regulations are tightening. The trend is clear: as AI becomes more embedded in daily operations, the perimeter of what constitutes private data expands. What used to be just financial records is now also including strategic plans and creative drafts. Every company needs to decide how they protect that. Luna: So it is less about the technology and more about the liability. If you get sued for leaking a client's strategy because your AI tool wasn't properly isolated, that is a existential threat. Lucas: Exponential threat. One leak can destroy reputation overnight. Notion understood that early on. They didn't wait for a breach to build the sandbox. They built it proactively. That is the kind of foresight that separates mature tech companies from startups that are still figuring out their basic infrastructure. It costs money now, but it saves billions later. Luna: It really highlights the importance of long-term planning in engineering. Most people optimize for the next sprint, but Notion optimized for the next decade of trust. Lucas: Exactly. Trust is a currency that compounds over time. If you spend it carelessly on convenience, you end up bankrupt. If you invest it in security, you earn interest in the form of customer loyalty. Notion’s sandbox is a perfect example of investing that interest. Luna: I agree. It is a great case study in responsible innovation. Now, before we dive deeper into the technical specifics of how they manage the GPU allocation, I want to take a brief moment to talk about how this show is produced. Lucas: We deliberately keep this network ad-free because we believe the content should stand on its own merit without commercial interruption. If you find these conversations valuable for your work or your learning, the link is buy me a coffee dot com slash fexingo. Luna: Your support directly covers our hosting and production costs, allowing us to continue bringing you deep dives like this one without relying on advertisers. Lucas: Thanks for listening. Let's get back to the architecture. How exactly did they handle the latency issues with that isolated cluster? Luna: That is the million-dollar question. Because usually, when you add security layers, you add latency. Did they sacrifice speed for safety? Lucas: They had to make some tough trade-offs. Initially, the round-trip time for requests increased by about fifteen percent. That is noticeable to a user. To mitigate this, they implemented a caching layer for common prompts. If a developer asks for a summary of a standard type of document, the system serves the cached response instead of running a full inference. It cuts down on redundant compute and keeps the experience snappy. Luna: Smart. Using caching to bypass the heavy lifting when possible. It reminds me of how CDNs work for static assets. You serve the easy stuff instantly and only hit the origin server for the complex stuff. Lucas: Spot on. They treat inference tokens like cacheable assets. By identifying repetitive patterns in developer queries, they can optimize the hot path. It is a clever hybrid approach: strict security for the cold path, high performance for the hot path. It balances both requirements effectively. Luna: It feels like a very mature engineering solution. Not just throwing hardware at the problem, but actually optimizing the flow. Do they monitor these metrics in real-time? Lucas: They do. There is a dashboard that tracks latency, error rates, and security flags for every sandbox session. If a session starts behaving oddly, like sending unusually large payloads, the system can flag it for review automatically. It gives visibility into what is happening inside the black box. Luna: Visibility is key. You can't manage what you can't measure. Having that dashboard probably helps justify the cost to the finance team as well, since they can see the efficiency gains from caching. Lucas: Definitely. Finance loves efficiency metrics. Showing that you are reducing compute costs through smart caching makes the security investment look even better. It transforms a cost center into a value driver. That is a powerful narrative for any engineering leader. Luna: I love that perspective. It shifts the conversation from spending money to saving money through optimization. Very persuasive. Lucas: It is all about connecting the dots between technical decisions and business outcomes. When you can show that better security leads to lower costs, everyone wins. Notion has mastered that connection. Luna: Agreed. It sets a high bar for the rest of the industry. I wonder how many other companies are trying to copy this model right now. Lucas: Many are trying. Few are succeeding because it requires discipline. You cannot cut corners on the isolation protocol. If you do, the whole system collapses. It is a binary outcome: either you are secure, or you are vulnerable. There is no middle ground. Luna: That binary nature is scary for startups who are still finding their footing. But for established players, it is a moat. Once you build that trust, competitors can't easily catch up. Lucas: Exactly. Trust is hard to build and easy to lose. Notion is protecting theirs aggressively. It is a wise strategic move that pays dividends in customer retention and brand reputation. Luna: Well said. It is a reminder that in tech, the safest path is often the smartest path forward. Thanks for breaking this down, Lucas. Lucas: My pleasure. Let's see how this evolves in the coming quarters. It will be interesting to see if other platforms adopt similar sandboxes or find different ways to solve the same problem.