Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Notion Cut Notification Noise by 70 Percent
Transcript
- Lucas: So about a year ago, Notion's engineering team noticed something odd in their support tickets. Users weren't complaining about features or bugs; they were complaining about notifications. Too many of them, too noisy, and a growing number of people were just muting everything. Luna: Which is a dangerous path for a product like Notion, where notifications are how people get pulled back into their workspace. Lucas: Exactly. And the fix wasn't just to send fewer notifications. They went deeper and built a relevance model that decides not just when to notify, but whether a notification is worth the user's attention at all. And the result was a seventy percent reduction in notification noise, while engagement with the notifications that did go out actually went up. Luna: Seventy percent is a big number. How did they get there? Lucas: It started with a question that sounds simple but is actually brutal in practice: what makes a notification useful? They looked at their data and found that a huge chunk of notifications were about edits to pages that users hadn't visited in weeks. The system was pinging them about activity in stale workspaces, and users were just tuning out. Luna: So the first move was to stop notifying about things users didn't care about. Lucas: Right. But it's not just about cutting the obvious junk. They had to build a model that could predict, for each user and each event, the likelihood that the user would actually want to know about it. That meant pulling together a bunch of signals: how recently they'd interacted with the page, how often they collaborated with the person who made the edit, even time of day. Luna: And that's the kind of thing that sounds like a data science project, but it's really an infrastructure project, right? Because you need to capture all those signals in real time. Lucas: Bingo. Their original notification pipeline was a mess of different event streams, each with its own format and destination. Some events went to an email service, some to a push provider, some to an in-app feed. To build the model, they had to unify all of that into a single event bus, with a common schema, so the relevance engine could see everything at once. Luna: That's a classic architectural refactor. What did that take? Lucas: It took about six months of work by a team of four engineers. They moved incrementally, starting with the most noisy channels, like email digests, and then expanding. The key was to keep the old pipeline running in parallel while they built the new one, then flip over channel by channel. That way, they could measure the impact at each step. Luna: And measuring impact isn't just counting clicks. What did they actually track? Lucas: They looked at something they call 'notification effectiveness.' That's the percentage of notifications that lead to a meaningful action within a certain window, like opening the page and making an edit, or leaving a comment. They also tracked the mute rate, because if fewer people are muting, that's a good sign. And they tracked overall session length after a notification, to see if people were actually coming back and sticking around. Luna: So it's not just about fewer notifications, but better ones. And I guess the seventy percent reduction in noise is a consequence of that. Lucas: Exactly. The model would suppress notifications it predicted as low-value, and the ones that got through had a higher chance of being useful. And here's the counterintuitive part: they actually increased the volume of notifications in some cases, for the most engaged users, because the model learned that those users wanted to know about every little change in their active projects. Luna: That's interesting. So it's not a blanket cut, it's a segmentation play. Lucas: Right. And that's a big deal for an engineering org because it means building for the long tail of user behavior. But I think the biggest technical challenge was making the relevance model fast enough. Notifications have to be sent in near real time, so they had to score each event in under fifty milliseconds. They ended up using a lightweight machine learning model that runs in-process, with the features pre-computed in a stream processor. Luna: What about the cost? Running a model on every event, plus the stream processing, that can't be cheap. Lucas: That's the thing. They actually saw their notification infrastructure costs drop by about thirty percent, because they were sending seventy percent fewer notifications. The compute for the model was more than offset by the savings on email and push delivery. So it was a win-win. Luna: That's the kind of story that makes a CFO happy. Lucas: It does. And it's a good reminder that thoughtful engineering can reduce cost and improve the user experience at the same time. But there's a human angle too. The team had to fight the instinct to just add more settings. They could have built a dozen new toggles for users to control notifications, but that just pushes the burden onto the user. Instead, they made the default smart. Luna: So the AI is doing the work that users used to have to do manually. And that's the kind of thing that makes a product feel magical, when it just gets out of the way. Lucas: Exactly. And it's a pattern that we're seeing across the industry. More products are moving from batch notifications to intelligent, event-driven ones. It's a shift from 'notify everyone about everything' to 'notify each person about what matters to them.' Luna: And I imagine this has implications beyond notifications. The same relevance model could be used to tailor the entire in-app experience, maybe the home screen or the weekly digest. Lucas: That's exactly where they're taking it. They're already using the same infrastructure to power a new 'for you' tab in the app, which surfaces the most relevant pages and updates. So it went from fixing a pain point to becoming a core part of the product strategy. Luna: That's a nice arc. Before we keep going, and I know we've got a bit more to cover, I just want to say something about this show itself. We keep these episodes ad-free, and that's only possible because of listeners who chip in a couple of dollars a month. If you've gotten something out of these conversations, buying us a coffee at buy me a coffee dot com slash fexingo genuinely makes a difference. It keeps the lights on and the episodes coming. Lucas: Yeah, Luna, that's right. And it's a small thing that goes a long way. But back to Notion — I want to talk about how they got the engineering organization on board with such a big change. Because it wasn't just a technical project, it was a cultural one. Luna: How so? Lucas: For one, they had to convince teams that weren't directly affected that this was worth the investment. The notification system touches almost every feature area, from comments to tasks to permissions. So they set up regular syncs with product managers and engineers from each group, and they made the progress transparent, sharing dashboards that showed the impact on key metrics like mute rate and activation. Luna: So it was about building trust and showing momentum. Lucas: Exactly. And they also had to be careful about regression. They set up a robust A/B testing framework, so every change to the model was rolled out to a small percentage of users first. They had guardrails in place, like a maximum notification frequency per user, so even if the model went haywire, no one would get spammed. Luna: That's smart. It's like a circuit breaker for user experience. Lucas: And they learned a few lessons that I think are broadly applicable. First, you can't just rely on user settings. Defaults matter more, because most people never change them. Second, you have to be willing to let go of some notifications that are technically important but not user-relevant. And third, the work doesn't end once you ship the model. You have to keep refining it, because user behavior changes over time. Luna: On that note, what's next for the team? Lucas: They're now looking at bringing the same approach to in-product notifications, the little toasts and badges that appear when you're using the app. They want to reduce the number of times a notification interrupts your flow when you're in a focused work session. And they're also exploring using the relevance score to decide when to send a notification versus when to just leave it in the in-app feed. Luna: That's a good nuance. Not every notification needs to be a push. Sometimes it's enough to just have it waiting there. Lucas: Right. And that's the kind of thinking that separates a good notification system from a great one. It's not just about delivering information, it's about respecting the user's attention. And that's a principle that can apply far beyond notifications. Luna: So if someone out there is listening and thinking about doing something similar, what's the first step you'd recommend? Lucas: Start by auditing your current notification inventory. List every type of notification you send, and for each one, ask: does this need to be sent in real time? Does it need to be a push, or could it be in-app? And most importantly, is there a chance the user doesn't care about this? Then, focus on the top offenders. You don't need a fancy model from day one. A simple rule-based system can cut a lot of noise. The model can come later, once you've got the data to train it. Luna: That's practical. And it's a nice reminder that you don't need to boil the ocean. Sometimes a simple fix is all you need. Lucas: And if you do decide to build a relevance model, make sure you're measuring the right things. Vanity metrics like open rates can be misleading. Focus on actions that actually move the needle for your product, whether that's retention, engagement, or revenue. Luna: Well, Lucas, any final thoughts on this one? Lucas: I think the bigger takeaway is that good engineering is often invisible. Users don't notice when notifications are smart, they just notice when they're annoying. Notion's story shows that investing in that kind of invisible quality pays off, both in user happiness and in reduced costs. And that's a trade-off any CTO can get behind.