Latest / Notion in Practice / Notion's Head of AI on Building Agents Your Team Can Trust: Sarah Sachs E10

Transcript

Notion's Head of AI on Building Agents Your Team Can Trust: Sarah Sachs E10 · Notion in Practice

• Tim Jeffries: Hello, everyone. Very excited to share with you today a special episode of the Notion in Practice podcast. We were very fortunate to be able to get access to Sarah Sachs, who leads AI engineering at Notion. So this is a bit of a different take on what we normally do. ⁓ We're actually interviewing someone who for Notion, who's in the weeds of how AI works. She's literally leading the team that's building it. And so we're going to spend next Jerwin Parker: Mm-hmm. Tim Jeffries: Half an hour with her. And I just wanted to kind of give you a little bit of a sense of what's coming. So yeah, Sarah gets into the technical detail. And as much as possible, we tried to explain what she was talking about as she was talking about it. And at the end, we'll wrap a little bit and try and kind of delve into some of those things. But I think some of the stuff to listen out for that was really interesting was Sarah highlighting these tensions between Jerwin Parker: Mm-hmm. Tim Jeffries: ⁓ the capability of the tools and then making them accessible to all of us, us, and how that's a key feature for Notion. ⁓ how there's this challenge with permissioning and autonomy. ⁓ something for us who are leading the kind of these businesses and trying to run organizations that want to use this stuff. Real tensions there for us. And you know how we operate, we're always trying to think what does this actually mean for someone who's trying to use it in a business, not just how do the features work. There's a bunch of stuff in there about Jerwin Parker: Mm-hmm. Tim Jeffries: when you should use AI and when you should use deterministic reasoning, like code instead. And we will talk with Sarah about the new features that Notion have released, Notion Workers and a few other things to help us all with that. And then obviously we talk about cost and versus kind of business outcome. So really excited to share it with you. I hope you enjoy it and we'll see you on the other side. Welcome everyone to this episode of the Notion and Practice Podcast. This is first episode live in studio. Tim, Sarah, ⁓ thanks for joining us. Thank you having me. Thank you for joining. okay, so We've had many ⁓ questions come through, but before we get into what you do and who you are, because you are the head of engineering here at Notion. For the AI team. And for the AI team. you just promoted me. I did. Wink I really want to go through to your story. Like I've looked had a little bit of a sneak peek on your LinkedIn. It's quite extensive. You started your career at Google. And ⁓ you've bounced around to different places going up into the AI space. And I think you've been in AI for quite some time now. So was this something that you always wanted to do growing up? Yeah. no. No. up, I my parents were both biology professors at UC Berkeley. So growing up, the only thing I knew is that I didn't want to be a biology professor. At UC Berkeley, I wanted to do something new and interesting. All right. And then I went to like a liberal arts college, knowing that I liked calculus, ⁓ which a career. And I didn't know what to do with that. and then I studied applied math. And then in applied math, I was like, but how does the computer do a progression? Like literally how? And I had an advisor who was like, You should take it into a CS class. So I kind of stumbled there. ⁓ when I was a sophomore in college, Netflix came out with the Netflix problem, which is where they were giving like million dollar prizes to people that could solve recommendation algorithms on Netflix. ⁓ wow. And big data had just become a big thing. right when I was a sophomore in college. So ⁓ and like ⁓ the twenty twelve presidential election was the first one where they were using massive targeting on Facebook. And just there's all this concept of ⁓ finding like signal from noise and ⁓ decis decisions around it. And so that's always stumbled into it. in my internships at Google, you know, I did some research in computer vision. My internships in my career at Google was all about population insights. So actually our Google Maps team is in Sydney, so was collaborating with the Google Maps team. ⁓ to figure out like, is this restaurant busier than usual? does this place have a gluten-free menu offering? Can we guess that? Because people that also go to gluten-free bakeries go to Cafe Sydney, right? ⁓ figuring out how to determine based on population patterns. ⁓ And that's kind of where I started my career. That was classical machine learning. Yeah. Right. that wasn't generative. It wasn't until I was at Robin Hood leading the AI teams there where we were doing classical machine learning. for customer support at Robinhood in 2022 when generative agents started coming out that I kind of helped at Robinhood, particularly on the customer support space, super early, thinking about do we need all these customer support agents when people forget their two FA codes? Right. And and like very kind of simple customer support tasks. And how do we have machine learning understand when it needs to get escalated to a qualified agent? and that kind of led me here. I was in the financial services space during that transition. Yeah. Right. And financial services, I guess, is is a lot of regulated. Yeah. So that's what's interesting. And I actually think Robin had done a great job navigating that. But in highly regulated spaces, especially in twenty twenty one, twenty twenty two, it was hard, at least in the United States, for regulators to understand that you couldn't promise a hundred percent of the time this output would be accurate. and you it was even harder to measure if the output was accurate ninety percent of the time because it wasn't A yes no question. It wasn't a binary classification. It was generative. and so that was really challenging. But ⁓ actually think those constraints were good for me in terms of my learning. Yes. Because it forced me to dig into how the systems worked much greater ⁓ kind of be an evangelist for ⁓ Yes. ⁓ and like the regulators that collaborated with Robin Hood. ⁓ so it forced me to learn how things work in a way that I've carried with me a lot. And then speaking of carrying that a lot, how did you carry that to one of the world's biggest systems? ⁓ Brain for empowering a lot of organizational knowledge at Notion. What does the head of AI engineering got that right? Am I getting my getting that right? Yeah, yes. does a day-to-day actually look like? Yeah. So I think for us at Notion, we have a really fun problem. on the one hand, we really serve frontier AI companies. You know, OpenAI is one of our largest, ⁓ most active AI customers. And Our job is to bring the frontier of capabilities for machine learning to frontier companies. At the same time, our job is to bring the frontier to companies that don't know how to get started. Right. And balancing between the two use cases means that you're constantly thinking about the tension of accessibility with capability. And they're naturally in conflict with each other. Same with permissioning. Autonomy and permissioning are opposite and they're in conflict. And so most of my day is spent Working with our legal team and our security team and our product teams, thinking about how to present our users to those conflicts or handle them for our users. Some things are user choices, but at least naming those conflicts is a huge part of what it means to do machine learning today. Yes. it's much more of an infrastructure problem than it is a statistics problem like it used to be. I see, I see. Moving from Robin Hood to Notion, what was what drew you to Notion specifically? you know, Robinhood was a fantastic place when it came to creativity with financial instruments. ⁓ myself am not a big trader. And of towards the end of my tenure there, realized it was really hard to be creative about something that I didn't get that much joy personally and abusing. I didn't it was hard to build user empathy if I myself wasn't like the target customer. And so my number one criterion was I want to do AI in a company of a product that I love. so much of AI and working in AI is being creative about solving user problems, even if they don't ask for that solution. Right. It's finding the problem and creating a solution, not just what they asked for. And so I wanted a job where I had extreme user empathy and I loved Notion personally. and I used it in a couple of work to place settings. ⁓ and so that was easy. It was actually like I found someone that worked at Notion and I asked for a referral and it was kind of white cook line and sinker. That was it. I didn't really work the market that much because I no one wanted to be here. Yeah. Yeah. Yeah. How long have you been there now? Almost two years. ⁓ okay. Yeah. Wow. And in that job the whole time? At Notion, yeah. Yeah. You the scope is gnarly, honestly. the AI team when I joined, it was like five people. And it was kind of like what does it mean to kind of build this product out? ⁓ now obviously our team is ⁓ you know, we're some magnitude larger than that. But also what's different is that ⁓ everyone in the company is building an AI product, whereas two years ago it was kind of an add-on that you could pay extra for. and so it's been really interesting to see how AI has become critical to the product, not just kind learning these hard lessons and building something that was ultimately like an add-on to the tool. I think today the reason that it's included in the business enterprise plans is. welcome back everyone. My head was spinning after that chat and Joe and we had a few technical hiccups that made our day a whole lot more interesting. Jerwin Parker: We did, we did, but what you see there is is a manifestation of making sure that everything flows together and being quite ⁓ resourceful in the moment. So I'm just glad that you guys were able to listen to that. I'll say that's all I'll say. Thank you. Tim Jeffries: Big thanks for your editing prowess, mate. We really appreciate it. but yes, so much stuff. I mean, Sarah was amazing to chat to. ⁓ I was honestly working very hard to keep pace. just, you know, when you get to sit with someone who's super clever and is obviously managing a lot and thinking about a lot, you yeah, there wasn't there was never going to be enough time. But we've prepped five takeaways today, not just three, because there's so much good content in there. You can't use Notion effectively as a business enterprise if we're not utilizing AI. And it also freed us to make AI critical to every user journey that we thought was necessary in Notion when we included it in the same plan. There was no more, okay, how do we make sure that this is top class experience that Avice not enabled? Things like search. Eventually it was like, I manage the search team. You should just have vector indices if you're if you're a large workspace, right? And it's just like, let's just make that included. And that's really freed us to build much more beautiful product as well. Jerwin Parker: Mm. Mm-hmm. Tim Jeffries: I think the first one for me is is about how AI has become infrastructure now. At first it it wasn't really. It was just sort of this add-on chatbot, you know, that ran next to it. But now we're at this point where we're thinking about agents and skills and workers and the permissioning. And it's actually a whole sort of fundamental layer to the infrastructure of our digital tools now. And So much in two years. I was just thinking about like what the last ⁓ my god. I mean we launched research mode a year ago. Yeah, right. Research we launched our agent, single player agent in September. Yeah. it's crazy. It's humbling. and can tiring, ⁓ but it's so invigorating to get to feel like you're always surfing at the top of the wave. Yeah. Jerwin Parker: Mm-hmm. Mm-hmm. Mm-hmm. Tim Jeffries: is positioning themselves as ⁓ of a key player, not the people making the best models, but the people who are building that infrastructure that we can reliably run on top of. You know, and the wave never stops. Sometimes you crash and then you evolve back on. Yeah. But it it never stops. ⁓ and so yeah. I'm I'm here for the ride. That's what I signed up for. Yeah. I I don't think it's gonna slow down though now, is it? Like for a long time, like I'm I've been using Notion for most of the last ten years. And the early days it there wasn't a lot of development. Like new features would come out and you would be like really excited because there were new features. Now I can't keep up with the new features. So yeah. Jerwin Parker: Yeah, exactly. And I think traditionally you think notion was that, you know, just docs and databases and and then you just chat to us where all your business knowledge lives. But now it's like now you can build these very I mean, we saw like which was talking about custom agents, permissioning workers, SDKs, what you just mentioned, Tim, it's now they're part of that very serious technical conversation. And yeah, it's it's it's it's gonna be very exciting to see how this goes in the next twelve months, I'd say. Tim Jeffries: The first thing I'll say is, you know, our job is to bring the frontier to the Fortune 5 million and to close that gap as much as possible. If we're gonna keep the gap this big and the frontier labs are moving at this speed, yeah, I have to make it so that my team can move at that speed. And that in and of itself is utilizing AI naturally. And so when you ask what does like it means to lead the AI teams in ocean, yeah, it's also making sure that we're iterating and working as quickly as the products with that we want to bring to our customers. The other thing I tell candidates and my team is. Yeah. Yeah. And I I don't think most people watch this stuff, but interesting listening to conversations with, say, people at Anthropic who rely on Notion as their system of record. Like if it's good enough for the people who are making these frontier models, then it's a good sign it's going to be good enough for the rest of us. And I love Sarah's kind of angle about saying, well, our job is to make this crazy technology that they're building accessible to what did she call it? The Fortune 5 million. Took me a little while to work out what she was saying, but Jerwin Parker: Mm-hmm. Mm-hmm. Tim Jeffries: I once took a hot yoga class, once being the primary word, the only hot yoga class I ever took. The instructor started off with this big speech about it's really important that you take water breaks, you enter child's pose if you ever get dizzy. Water is super important. Yeah. And about halfway through the class, I realized that she never announced like a group water break, right? Everyone was taking water breaks on their own time. And if I didn't decide to take a water break, I wouldn't pass out. And the class was designed that way. Yeah. That's what it's like working in AI today. There's no Jerwin Parker: Mm-hmm. Fortune that was the first time I heard that term too, Fortune five million. Tim Jeffries: The rest of us. It was cool. Yeah. Yeah. Yeah. No, I thought that was good. The other one liner that she had, I think second takeaway is ⁓ tokens don't grow on trees. I thought that was really good. and it's a sign that we're we're moving on from ⁓ think it's the other one line. I mean, I listened to her for a couple of days up in Sydney. we're not we're not into token maxing, we're into outcome maxing. ⁓ I thought that was a great line as well. And Jerwin Parker: Mm-hmm, mm-hmm. Tim Jeffries: the exception of December twenty fifth to January first, which hopefully tends will continue to be a quiet time. Yes. There is no group water break in AI. You have to do it on your own pace, right? And so a lot of it has to do with like self-energy management. And to work in this field, you'll either burn out or you have to learn it. Yeah. Wow. one of the things we always try and do is like keep our finger on the pulse about what's going on in the market here and the challenges that people are facing and the questions they're having. Jerwin Parker: Mm-hmm. Mm-hmm. Tim Jeffries: Yeah, it's sort of shows the maturity of where o of this market, of what's happening with AI. No longer is it just about how many tokens do you need to get something done? but but yeah, like how can we really focus on getting the outcome that's needed. And so maybe we were all lulled into it a little bit with like custom agents being free for so long. We didn't really have to stop and think about it. But now that we do, a whole bunch of other really important conversations are rearing their head. Which is hard to do because it's so much noise, but lots of noise at the moment about agents coming online and UCPH pricing coming online. ⁓ I got my first bill this month. ⁓ and I've been thinking about it a lot, writing about it, ⁓ talking about it. And so our bill was $500. and so presuming that's too high. Well, ⁓ I'm undecided. think it depends. Yeah. I don't think that we're done yet making it affordable. Jerwin Parker: Mm-hmm. Mm-hmm. Mm-hmm. Exactly. Exactly. And it's like it moves to the next point as well, Tim, is sort of like you use AI for reasoning and then what was that? She said, like something for you wrote an article about it, right? Code for plumbing. Yes. Yes. Tim Jeffries: Code code for the plumbing. Yeah, yeah, yeah. So that working out is this a deterministic process that actually doesn't need AI to stop and think about it. We don't need to burn tokens, which are expensive, not just because we're getting charged a lot for them, but because they are actually expensive. I think that's like a an important point to note is that it has felt to a lot of us like this stuff is free, but it's not. And so if we're going to use it, we should use it well, we should use it wisely. Yeah. ⁓ I think the industry isn't done, and I think that we, with our developer launch, are in a very good place. we have a project internally called Project Tokens Don't Grow on Trees. And true, tokens don't grow on trees. Yeah, right. That's good. And there's different ways to address that. One is that not all tokens cost the same. ⁓ And it so that our users understand model selection, we can help them with model selection and we partner in the right way to make models affordable. Jerwin Parker: Mm-hmm. Mm-hmm. Tim Jeffries: And so yeah, I I've been thinking a lot about this. Actually, the kind of resource that we're going to give away today is about like a model for people to use as they think about their own processes and systems to try and work out is this a deterministic process that like we could use code for? And we'll talk about Notions Workers platform in a second, or is this something that genuinely needs AI? It needs some reasoning, some thought to happen and we need to apply intelligence to it. looks like notion doing reinforcement learning on open weight models. That looks like partnering with the open weight model community. ⁓ means using our bargaining power with frontier models to maintain low rates and smaller models. Right. Yeah. And the other part is our developer platform, not everything needs to be solved by a token. ⁓ So being able to ship code that runs ⁓ deterministically, sometimes at higher quality. Yes. And having it be a one time cost of building ⁓ that worker that's hosted is an incredible Jerwin Parker: Mm-hmm. Mm-hmm. Yes. Tim Jeffries: And that little model also helps you think about, well, should this be technology that does this reasoning or technology that d does this deterministic work or or should it be a human? ⁓ yeah. No important stuff in there. Incredible asset. Even ⁓ we're shipping right now, we've silently shipped it. computer where agent has access to a computer where it can store files and access them via CLI. It costs, you know, maybe one to two cents to pull up the computer. Yes. But at a certain token amount, it's actually saving our customers tons of money. So when have lots of context, keeping in the transcript is effing expensive, right? And why are you doing that? Because we are creating systems built on token maxing. Jerwin Parker: Mm-hmm. Yeah. Space. Yeah. Speaking of which, when you when you're talking about does it need to be human or does it need to be the the AI having like this sort of judgment and orchestration, that goes to the next point is like when met Sarah mentions that permission permission context is the real moat. Like, you know, when you when you multiply your agents ⁓ and problem ⁓ sort less becomes like which agent is the best, but Tim Jeffries: Yeah. And Frontier Labs are creating systems that encourage you to do that because that's how their system works. Yeah, exactly. You know, they make money, they would tell you that they don't have enough capacity and they don't want you to be unnecessarily using too many tokens. And I think there's an element of truth to that, but I'm sure they don't hate to token abuse, right? Yeah. ⁓ we are outcome maxing. We're not token maxing, right? A majority of our revenue is seat-based pricing. A majority of how we sell notion is by making it so that your processes make you more efficient. ⁓ Jerwin Parker: Where where's like the ⁓ where does the most context live within certain, ⁓ know, databases or or places within Notion or not even just Notion, but whatever Notion has access to as well, right? ⁓ I thought that was pretty a big thing to think about. Tim Jeffries: Yeah. Yeah. Yeah, yeah, absolutely. But so I'm seeing that heaps in the market at the moment, people who have their context spread across different sources. That's the real challenge now. Not which tool you're using, but how do you make sure AI has access to it? And when it does have access to it, like is it the right access? You know, is it can it just run wild in your Notion workspace and also like we cover a lot of that spend. We also want it to be cheaper because it helps our merchants, right? It's bad business. Yeah. And the cheaper that we make our custom agents, the more people can use it. Yeah. You know, when were developing custom agents, I made a email agent that completely changed my life. It was an email software factory. I never checked emails, was auto rejecting recruiters, recruiters that were like higher quality companies would Jerwin Parker: Yeah. ⁓ Yeah. Yes. Tim Jeffries: Do whatever it wants or SharePoint or in you know in these other spots. So how do you gather the context so AI can see it? And then when it when you can see it, how do you ensure that it has the right permissions and people who are using it, you know, aren't getting access to things they're not meant to? And it it's become a complicated thing to think through. And certainly in our work we're we're spending a lot of time helping clients think through the the governance of these tools because the more automated everything becomes, the more essential I would it would help me reference something in the news about them. It says my server rejection, just in case. It was all my packages, tracking and returns, everything. It was $45 a month. Yeah. Insane. and I was like, I would not pay $45 a month for this. Something is broken. ⁓ it ended up being how we how we trigger agents from each other, right? There's like a lot of low-hanging fruit that we can do to help customers in product make it cheaper. And that is like an ongoing project. And that is the product. Yeah. Right. Yeah. ⁓ I Jerwin Parker: Mm-hmm. Mm-hmm. Yeah. Tim Jeffries: I paused about whether five hundred dollars is too much because the outcomes for the business have been something that I couldn't have done any other way. Right. I would have had to pay a person full time to do the stuff that these agents are doing. So we're not using custom agents for everything, we're gonna use it for the things that need to be automated. So things like anytime one of the team sends an email, custom agent has a look and says, Is that email in our people database? Is it connected to a project? And if it is, grabs the email and puts it on the project. Now Jerwin Parker: out. ⁓ Tim Jeffries: Even in a situation where previously would have had permission to ⁓ something. So maybe they could access like a whole database of of of jobs and each job had the financial information about what was going on for it. In the past, it would have been quite tricky for someone to assess all of that financial data across the whole database all at once. But with AI, it's made it so easy that even when you have Notion knows about all the context to do with that project, right? Beforehand, that never ever happened. There was no way to do that. And we're just doing that on every front, every Slack channel that is about it, every calendar entry, you know, all all all of our kind of world. So actually Notion AI knows more about our clients and their work than than we do now. Yeah. So that's that's 500 bucks a month. That's cheap. Exactly. And I think when people think about it as software, it gets confusing. But when you think about it as processes and people, Jerwin Parker: Mm-hmm. Tim Jeffries: permissions to the same things as you had before. Once you add the kind of AI superpowers to your toolkit, you can potentially that information in a way that isn't appropriate. ⁓ and so yeah, there are there are a bunch of curly questions that come out there. And ⁓ very clear that Notion is positioning themselves as the platform where the system of record and its permissions ⁓ the agents ⁓ the way they interact with it all should be, all in that all in that one spot. ⁓ it makes a ton of sense. Yeah. A ton of sense. I mean, at Notion, we haven't really had to limit any of our usage because and this is true on coding agents, by the way. Like ⁓ have the right enterprise controls with Notion AI so that firstly, like we can select the model for certain tasks. Like I think our task triesher is on haiku, right? ⁓ no one can modify that. But if I were just like spinning off lots of clawed processes in a company without Notion, I would never have that enterprise control. Jerwin Parker: Yeah. Yeah, 'cause you think about it, it's from a governance perspective, it's a lot easier to manage because it's all within right? Like context, permissions, collaborations. Like when you have your governance teams or your your data platform teams or like even your security teams come in and and they're like auditing, like, hey, what as has access to what? Notion's the best place because all that data and context lives in a very simple spot in Notion. Like you just look at the UI, so like Tim Jeffries: Yeah. And we can set price limits on agents, not just on users. Yeah. That itself is like a huge distinction. but I do like I do think that as a society, we're over indexing on tokens solving all problems. Yeah. Really, they need to orchestrate and reason. They don't need to do all tasks. Jerwin Parker: Done done and done. I think that's why you mentioned before with anthropic using Notion, I'm not surprised because like you got the most complex technical builds. Like what they're doing is so technical that they probably need a system that's so simple to continue executing. And you know, I think that's that's really cool. I think you mentioned that, Tim. Tim Jeffries: as a society, we're over indexing on tokens solving all problems. Yeah. Really, they need to orchestrate and reason. They don't need to do all tasks. Yeah. And so that's why our CLI launch and our like NTN and our developer tools, they're not just something that developers like because they integrate with Notion. It actually frees our users from Yeah. And so that's why our CLI launch and our like NTN and our developer tools, they're not just something that developers like because they integrate with Notion. It actually frees our users from Yeah. Yeah. ⁓ and the final one I think is it it sort of speaks to that our question about so what? Like what difference does this make to someone who's actually trying to run a business? And I I think for me that the outcome, the thing that's changed is before we were talking about which tools you're using. Are you using Claude or are you using Chat GPT? think it's less of the question now and more about Their business is depending on token pricing. Yeah. Their business is depending on token pricing. Yeah. I think that's good. I mean, we wanted to get into that because I think like a lot of our audience are not technical. So even most of those words that you just said, people are like, I don't know what she's talking about. But I think we can break them down, yeah. Because this idea that we talked about a little bit that people are using agents to do things that don't need intelligence, right? So they're spending tokens on things that it's plumbing. It's like moving data around. Imagine if every time Jerwin Parker: Mm-hmm. Mm-hmm. Tim Jeffries: your actual workflows. Like are you stepping back and considering the workflows that you're building and are you applying the right tool at the right time? Let's imagine I have a custom agent that takes a CSV spreadsheet. Yes. does the right queries and then ⁓ you know updates a cost reporting graph on Notion. Yeah. That does not we do not need to reinvent how to do that every time with a prompt. You use a prompt one time to write a a Python script and then use the Notion API to update the chart with the data from the Python script. Right. Jerwin Parker: Mm-hmm. Yes. Tim Jeffries: It's not just like, ⁓ this model is the best. but actually there's a real opportunity now to redesign the workflows of the business, to take advantage of AI, not just speed up something that a human did before, but do it in a completely different way. ⁓ and all of this infrastructure that Notion of Building kind of allows for that. So I know and that's a huge question for a lot of a lot of businesses who've who've done things the same way with human actors for a long time. And when they brought in technology, technology was just Jerwin Parker: Mm-hmm. Tim Jeffries: Why can't we host that on Notion? There's no reason not to. Yeah. And I think that more and more developer focused companies that have these resources have been asking for it. Right. But with the Notion agent making it really accessible, you don't it's kind of like cloud code making software available. Yes. If we do our jobs well, you don't need to know that it's doing that. Right. We can see that it's a repeated workflow. We can know what code we might want to be executing and that costs. Jerwin Parker: Mm-hmm. Tim Jeffries: automating something that a human did. It wasn't fundamentally changing how it was done, but that's the opportunity now, which is pretty huge. Jerwin Parker: Yeah. Mm-hmm. Yeah, it's a wildly it's like the wild, wild west right now. with all the changes and every time ⁓ an AI model changes. which I wanna give a shout out to Notion. Every time a new ⁓ model up or update, Notion is so quick to update the models. Like I remember when Fable Five just came out and then I had access we had access like what, 30 minutes later or something? It crazy. ⁓ Tim Jeffries: much less to host code than to be running it ⁓ an AI like language model. And we'll do that for our customers and replace it for them. That's the goal. Right. Today it's pretty developer focused you have to write the worker. Yeah, I wanted to ask about the kind of accessibility piece because at the moment you still have to write the code, although not many humans. That's actually the sprint my team is doing right now. Right. Yeah, because you're using Cloud Code to write the code, right? wanted to get it out as fast as possible. In general, we like to actually have a sticker on my laptop that says Yeah. Yeah, I I feel like I'm finding out about the models launching because they appear in the drop down list in Notion rather than before I read anything about it. ⁓ I do feel like the probably the biggest like the single takeaway, you know, if people listen to all of that and were like, man, there's so much in there, what should I really be thinking about? Is Yeah, I think the question is that that people can ask is well, on the Notion platform now, should AI be doing the judgment work of, you know. Jerwin Parker: Yeah. Tim Jeffries: ⁓ observe and prod or live in a hallucination, right? Which is like at the end of the day, like you are living in a hallucination if you don't know how it works. Yeah. And the best way to see like the jagged edges of our product is to ship it. And we have a lot of customers that are pretty technical and also not technical, but their heads of IT are technical, right? You just need one person in your company that understands how to code and they're setting up a process for the company. And so that kind of like head of IT technical person has been our entry point. Jerwin Parker: Mm. What should I do now? Yeah. Tim Jeffries: of of this piece of work? Or should should code be doing it? Like is this something that the code can do? Because the workers launch really brings that automation, which once upon a time you had to do off platform. You needed to go to Zapier Automatic or to N8N or any of those kind of tools to be able to do it. But now you don't. Now you can create a Notion worker. And it's still relatively nerdy to spin it up, but was interesting to hear Sarah talking about Jerwin Parker: Mm. Mm. Tim Jeffries: And that also freed us. Remember how I mentioned like autonomy and explainability ⁓ are constantly at odds with each other, autonomy and permissioning complexity and accessibility. Those are at odds. Yeah. So something like our developer platform launch, we didn't necessarily prioritize accessibility. We prioritized capability. Yeah. because for processes, not every user of the custom agent needs to know how to build the worker for the custom agent. that mindset switch. Jerwin Parker: Yeah. Yeah, yeah. Mm-hmm. Yeah. Tim Jeffries: actually know the plan is to make that accessible to everyone else. Version one is very just developer focused, but before too long, people are going to be able to use AI and Notion to get the code written and get it deployed. And so be able to kind of roll out their own workers. But I I I think that's the big question. And and to be honest, I mean, is it two weeks or three weeks since we were in the room with Sarah? I have spent most of that time building workers. Jerwin Parker: Yeah. Tim Jeffries: was critical for us to milestone how we build. Okay. Yeah. Yeah. So I mean, workers is kind of what we were just talking about. This be able to run code to replace some of what agents are doing for us at the moment. People in the past have been using ⁓ and Make and whatever. And this like this trying to draw like a connection to what people know now. So this bit of notion is gonna potentially replace some of that. Of course. And also imagine if you could Jerwin Parker: It's so cool, yeah. Yeah. Uh-huh. Tim Jeffries: I came home, I got my bill, I was like, yeah, this is a lot of money to be spending. And really when I think about it, a bunch of what we're doing is deterministic. It's plumbing. And so I've written a worker to do our calendar syncing, a worker to do our email syncing, a worker to do half the process of updating projects, statuses, like a bunch of that. it's very accessible like conceptually. And then I think once notion build the tools to mean that you don't have to Jerwin Parker: Very nice. Tim Jeffries: Just use prompting to make your own Zapier actions. Yeah. Right. I think if you look at the type of actions that Zapier can take, basically experts who know about every API. Like I think the landscape of what you can do with a Google presentation lives on Zapier, right? That's kind of or Microsoft Teams, right? Imagine all those capabilities, but your agent could figure it out for you and you could build it for yourself. Yeah. I think Yeah, mean, I think those marketplaces probably will always exist, but it's certainly a similar experience. Jerwin Parker: Mm-hmm. Tim Jeffries: open up a terminal anymore, you can just talk to AI to do it. It'll revolutionize the cost side of things for people. And and also more than just cost. Code will g deliver them a very precise result over and over and over again. Whereas, you know, AI will have to stop and think about it every time and might make slightly different choices. So ⁓ yeah, I think that's the big takeaways. Should AI be doing this or should code be doing this? I can do it all on the one platform. So the difference being that it also you get to customize it for yourself with an agent. ⁓ and it also has the permissioning flexibility of Notion. Right. ⁓ which those kind of built out solutions might not. What's the advantages to having it in house too? Like I always think as soon as you have to go off one platform onto another platform, you introduce security and exactly a range of other issues. So that's why our managed agents launch has been probably like the underdog success story. We we Jerwin Parker: Mm. Mm-hmm. Tim Jeffries: Need to make those decisions. Jerwin Parker: Exactly. Exactly. So I know we always sometimes well mostly give our audience a gift. Do we have a gift for the people mate who've have made it funny enough to the end of this episode? Tim Jeffries: We collaborated with Claude and Anthropic and we're launch partners Claude Manage Agents. And it's a very similar idea, ⁓ is if you have a task that gets filed in Slack and you want Notion to figure out all the context of that task and then give that context to a coding agent to take a first stab at something before handing it off to an engineer, can all of that process live in Notion and have all the context and permissioning of the work that you're doing? Yeah, well, I think I was referring to it before, but I I think that model, that framework for how should I think about my workflows, how should I think about my kind of AI processes, I think that is to be really useful to people because they will be able to take, you know, let's say it's their onboarding flow, right? ⁓ they're they're using custom agents at certain points in it. They're trying to they're trying to automate the process and they're they're finding it hard to work out. Do I need to use AI here? So imagine if your coding agent only had access to the what the engineer had access to, right? for some engineers, that would be fantastic. And for people that you don't want coding specific systems, it's also fantastic. Right. Yeah. ⁓ this is especially true with data. like I think ⁓ data querying, data access, kind of like the data analytics realm is like a famously permissioned space. Yes. Yeah. Right. and it should stay that way. And so ⁓ allows administrators basically control Or do I need to use code here? ⁓ what what kind of work is this? And, you know, that other question about should it be technology at all or should it be a human? So yeah, with this episode, we'll give you this model and some instructions for how to use it ⁓ to help you run that process through. Jerwin Parker: Mm-hmm. Tim Jeffries: And it allows model interoperability now ⁓ can have a codex agent review a clot code PR using of the data that lives in Notion and all the prompting from my custom agent on what the goal of the software is, the context of the team ⁓ and over Slack. And it can all be administered by a systems administrator. Right. Yeah. Okay. Yeah, cool. So it sounds like there's some fast following going on with workers. Like some of my questions were about. Jerwin Parker: Perfect. Well guys, to check it out in the description below. ⁓ feel free to reach out to us if you have any questions. Our DMs are always open. and also to reach out to Sarah and and follow Sarah on all her socials will be at the bottom below. Thank you guys so much for watching and we'll see you at the next episode. Tim Jeffries: Thanks. See you soon. Jerwin Parker: Ha ha ha. Tim Jeffries: Okay, I I can see what it can do in practice. ⁓ I'm I'm not coding, so I'm like it's a step beyond where I would like naturally be comfortable with, but in the last twelve to eighteen months I feel like AI is coded for me. So I think that step is mental, by the way. I think that if you download a Claude Code, workers ⁓ has all of the prompting for Cloud Code to figure out what to do. ⁓ that is not something that we need to ask all of our customers to figure out how to do. Especially when we sell API connections to quad and side notion. yes. I think in twelve to eighteen months, if you feel like you need to be a developer to utilize our developer platform, I should get fired. and and we're not doing our job to our customers. because the whole point is we want to bring the frontier to companies that don't feel like they need to be experts in what's happening. Same with model selection. ⁓ You should have to know GLM five versus Minimax two point five versus haiku. What a waste of your time. Yeah. Especially like how does one work with emails? One is better with charts. One is, you know, and and their cost. And like some of them they do more tool calls. So is it cheaper at the end of the day? You know, that's not that's not your job, that's our job. And the the more that we can package it in a way that's accessible to non technical people, the better we're doing our job. Yeah. Yeah. Yeah, nice. So how does this relate to the other things from Dev Day? Like I was getting my head around Agent S DK. Agent API. Yes. Some of those I felt like as I understood them or they were further out. Yeah. Is that fair? Yeah. ⁓ yeah, I think that like I said, we launch early. Yeah. And for developers, it makes a little bit more sense when you think about how you want to wrap agents. wrap is in ⁓ what if I wanted my custom agent to search over Decagon and understand all the customer feedback before kicking off a cloud code task? Yeah. And then that making that cloud to a task and then that I can pass off in a task management system to an employee and flock for them. Right. Yeah. Yeah. ⁓ I think that the agent sprawl is real. And I think that Notion is in a bad position if we think that we necessarily make better customer call analysis agents than Dekago. right? Like and work coding agents. Notion's not trying to make a better coding agent than Kodaks. Right. ⁓ what we're trying to do is make it so that we're the best place that agents collaborate. Yes. Yeah. And so that means external agents need to be able to collaborate. And our bet is that the system of record has to be durable. if you're having ⁓ company that's running on Decagon agents and ramp agents and codecs, then some people are using Cloud. And by the way, there's your Notion Enterprise Search Agent. Yeah. ⁓ where's the source of truth? Like where's the durability? and see that in Notion today for search. Right. I think it's ⁓ in the double digits, close to thirty percent of all of our search traffic is coming from MCPs and agents. Really? Our customers are agents. Yeah, yeah, yeah. You know, we are the best pace for enterprise collaboration, whether it be agents or humans. And that means it can't just be our agents. Because when a human is doing their work and they're collaborating with customer support on a Notion doc, then Codecs should be able to collaborate with Dekagom via Notion. Yeah. Right. Yeah. So if we break down those two things, the agent SDK is The piece that allows external parties to talk to AI inside of Notion. Yeah, right. What is that kind of like who are they talking to? There's a few, you know, I was thinking, are they talking to Al? Like are they talking to my customer agent? Are they talking to agent? What is it? Yeah. So we work really hard on exactly that portioning to make it quite clear. so when utilizing ⁓ Notion having an SDK that can like call in our agents, for instance. We see a lot of people that want to use Notion Enterprise Search something that's inside of like their c company's ⁓ agent. And we want to make that as clean as possible. Same with like our expertise ⁓ actually having it that Notion can write to Notion and use Notion and think about all the different tools available. but I believe if you're using it via your off token, it has access to what you have access to, but We should check that emotionally, I'm not sure. Yeah, yeah. That's ⁓ technically that's our ⁓ C P A P I team, which is but what I can say is that our kind of frontier selling point is promotioning. So Yeah. ⁓ Yeah, yeah, absolutely. See how that I mean it it's exciting. I I've probably slept a little bit on using AI tools outside and then trying to bring them in. ⁓ but I'm seeing a lot of people in the kind of market who have especially Claude, it's having a bit of a hot moment here at least. And so people are using Claude and they're building kind of skills and yeah, you know, processes in Claude and then they're using Notion and sometimes they're talk you know, they're using MCP, sometimes they're not, and they've got these kind of two separate worlds. So I can see even in the maybe parts of the market that are not super technical, people are, you know, not in not in the tech space at all, but they're working in both these worlds and they want them to connect. I mean, that's what I mean when I say we want to be the best system of record for e jumps. Notion. Is the system of record of choice for Claude. So if you were to ask Claude and you have all these connections, we want to maintain that. Yes. by making it so that you don't have to think about it. Yeah. ⁓ and I think that's fine for personal use. I think when it's collaborative use, that it gets a little bit more tricky. and that's where I think custom agents and using our native setup, and you can still have it execute Claude code via managed agent. So that's why we launched that. I think it's where it gets it gets quite tricky, but I have heard, especially Australia, is like very claw-cutted. Yeah. ⁓ and not intended. And I think that's also interesting because I think it makes sense. It's super capable, but I get scared for companies that have vendor lock-in too soon, particularly how they is implement ⁓ their workflows. Because for instance, there was a long time where GPT-55 was better than FS46. Yeah. And there was a huge like mass of people that felt like they didn't have access to frontier capabilities because they were locked into their setup. So ⁓ a lot of our top customers have found ways to utilize Notion Skills and Notion as the markdown source of truth for their agents so that they can have all of the coding agents available. And if we do our job well and we're negotiating for the Fortune five million, we give them massive discounts because your company, I'm presuming you don't have a fifteen percent enterprise contract discount with a four of us. No. Yeah. You you don't have the bargaining power available. And if you have vendor lock in, Your entire business process, you don't have any bargaining capability. You're you're stuck. Yeah. And you can't negotiate at scale. And so, in some sense, we're doing our customers a giant service. If we were to just be reselling tokens, right? I mean, we could make a business out of that. I think that's what open router's business model is, right? We make it so that you can change between agents, even coding agents now. All of your skills live in a collaborative permission space and we negotiate rates for you. And if open AI goes down, that's fine because we have a collaboration with AWS and we have backup models. And if all of Open AI models are bad, that's fine because we always have vendor optionality on every product that we sell. Mm-hmm. Yeah. So Sarah, let's let's ⁓ tie it all together now. cause my brain is now buzzed with all this information. ⁓ I look at the viewers as well. so I wanted to tie it back and and and go to the point that you mentioned ⁓ earlier you guys have been Building fast and building in the open. I guess from a viewer's perspective, and a as a Notion fanboy as well, is what's the initial initial response been like? You know, it's been surprisingly positive, right? Which is to say that I think that AI has a lot of jagged edges. And I'm constantly impressed with Notion Power User's ingenuity to use a system of primitives to build what they want. And that's kind of the Notion Dream, right? Which is that it's a series of building. And sometimes they meet customers that have these phenomenal ways of using Notion. And like they'll have different agents that talk to each other with different permission models and different access points and different triggers in ways that they use Notion better than we do. Right. Honestly. And I think that's one of our biggest lessons with Notion CLI, by the way, which is being able to edit Notion from a coding agent. Yeah. if we build the right primitives and trust our users, the rest kind of falls naturally, especially in the H of AI where primitives. Are kind of the cornerstone of being able to do things. Right. and so I think my biggest surprise, like when we were building developer platform, ⁓ first reaction was like, okay, I need to sit down with Notion AI and wrap my head around this. Right. MCP, CLI, remind me why our customers care. Like, you know, six months ago, that's where I was. Yeah. I lead engineering for AI at Notion, right? So I've been very surprised at how thirsty people are for making it work for them. And how much they just trust the primitives. And I think that if we keep investing in making our primitives accessible for agents and we keep investing in that system of record as a collaboration point, we can kind of get over the hump of like, how do we make an AI transition as a company? Right. ⁓ if we never sold agents, if we just made our system of record work as well as it does today, which I'm lucky with the base of what agents need to do, I think we're in a great place. I'm really proud of how the team has navigated. And it's so good to hear because I think Well, another notion, fanboy, but like the primitive spit, I don't know if most people like use that language, but the building blocks, like the Lego blocks that is that is, you know, that is notion ⁓ a for a lot of us. And so to to have that with AI, like I didn't expect that. I kinda thought, ⁓ AI will come along and run alongside. But now it feels like we've got AI Lego blocks. Exactly. Right. Like you have ⁓ model selection is one, triggers is another one. Permissions is one. work is one, memories are another. Exactly. Skills are a perfect example. ⁓ and it used to be that was the most inaccessible recognition. I that's where assistance came from, right? It's like ⁓ see a blank page and then you see these people on TikTok and they're like, How do I get from there to there with my business? Yeah. And it used to be that you, you know, had to hire someone to do that for you. And exactly. Now it's about ⁓ people can operate the top of their license and focus on their goals. Yeah. And we can help them with the how. And ⁓ that's our that's our job. Yeah. Nice. I wanna ask how are you gonna make sure you don't pass out in the hot yogurt of Yeah. Yeah. It's like are you taking a break when you're seeing me? Honestly, I think speaking to customers is one of the most invigorating things. Yes. Especially speaking to customers that don't that aren't YouTube, that aren't fanboys. I mean this is invigorating 'cause it's like ⁓ cool. But it's the people that are like, and like it forces to go back to like what are we selling and why. Yes. ⁓ then to see their faces like kind of light up when I just open up my laptop and show them how my team uses Notion. I don't have like pre planned demos. ⁓ sitting with our customers and hopefully there's nothing too secretive. I just open my laptop and I'm like, ⁓ team is collaborating on this task. Here's the stand up dog. Triggered the task, the trigger of the codecs, that you know. And think it's that in of itself is invigorating to I'm also psychotic. Like I do this for fun and this is my trip. And then hopefully I'll I don't know, touch some ocean. ⁓ Free course. ⁓ Yeah. Well thanks so much. Thank you for having me. Thank you so much for joining us. Yeah. Awesome. Thank you. Thanks.