Latest / Technically Speaking / What actually is AI research? (w/ Jonathan Frankle, Chief AI Scientist at Databricks)
Transcript
- Megan Reynolds: spent your career doing influential AI research. Jonathan Frankle: God, you're calling it career, I feel so old. Megan Reynolds: You've built MosaicML and. You sold it to Databricks for $1.3 billion, but let's start really simply. What actually is AI research? Jonathan Frankle: God, I just, the word career here is just such a like - Megan Reynolds: I've sent you in a spin. Jonathan Frankle: Oh, I'm just like, I'm thinking about my father who's getting ready to retire, and like, Dad, what did you think of your career? I was like, oh man, I'm so old now. So just to set expectations, my diploma arrived from MIT for my PhD three years and one month ago. So there were some delays in that process due to startup life, and also there was apparently supply chain shortage of parchment. So the diploma arrived very late, according to MIT. But technically, if we're counting the diploma arriving as a graduation date, only three years and a month out of my PhD so let's go easy on the whole career thing. Megan Reynolds: That'll take them never. Jonathan Frankle: But yeah AI research it you know I think people look at it a lot of different ways I have my own strong opinions on kind of what it should look like but the way I would describe it is kind of in the same way that a physicist who or cosmologist let's say looks at the universe and you know we've just been given this giant artifact we can observe light we can observe particles, we can observe planets, we can observe how they interact with each other, but nobody has handed us any fundamental laws. Jonathan Frankle: Every law of physics that we have is our attempt to simplify and describe the universe in terms we understand that have predictive power about future phenomena in the universe. The same is true about biology in some sense. We've been handed these wildly complex creatures, but nobody sat down and said, ah, biologists, why don't you design life from the ground up? We've just been handed life and life in this particular form and we've, you know, our goal is understand it. How does it work? Let's take it apart. And then, you know, does that have enough predictive power that we can design medicine? We can design interventions that change how life works in ways that are beneficial to us in some way. Jonathan Frankle: And I look at AI research the same way. And it feels counterintuitive because we've built this. This has been human created, designed from scratch. But really, the way I look at it is that in some sense, imagine you're God creating the universe and you create this whole soup of particles, you design some laws of how they interact with each other, you pick some fundamental constants, and then you set the whole process in motion and go hang out on the beach or whatever you do if you're able to create the universe for, oh, 15-odd billion years, and you come back and you see life on Earth and you ask, how did that get there? Jonathan Frankle: that wasn't, you know, you just created this soup of particles and you know some laws of interaction and extraordinary emergent behaviors happened that would be very difficult to predict from the first principles of the ingredients and I look at modern generative AI the same way. We create this soup of trillions of parameters at this point, ways they interact with each other in terms of the network architecture. We take all this data and toss it in and then we you know we have energy, we have compute and we just mix it together with a pretty simple set of rules, all things considered, human understandable set of rules for a really long time. Jonathan Frankle: And then you get ChatGPT out the other end. Obviously there's more to it than that. Megan Reynolds: Maybe connect those dots a little more. Jonathan Frankle: Well, no, there is no more connection than that in some sense. It is that, you know, you could imagine if you're creating the universe and you set the gravitational constant off by just a little bit, everything clumps together and nothing interesting happens. Or you set it off a little bit in the wrong direction, everything just spreads apart And nothing interesting happens, but if you set that gravitational constant and like you know, whatever other constants again I last took physics in 11th grade. So forgive me for this but you know although I read a cool article about it last night because I that's all I get but like you set it just right and you set the conditions for a lot of interesting interactions to happen and Those interactions cascade in ways that somehow you do wake up 15 billion years later And in some tiny part of the universe, if you get everything just right, you get life on Earth. Jonathan Frankle: This, like, phenomenally interesting thing that you would never predict from the initial laws of physics. Megan Reynolds: But we are, in a sense, create, you know, we are creating this technology. Or is your point that actually this technology exists and we are shaping it? Like, what's our... Jonathan Frankle: It's that we're creating this technology, but in the same sense that you create the initial conditions for the universe and let the universe run for 15 billion years and see what happens. And we get enough shots at doing this. And we've done this at enough small scale universes and bigger universes and even bigger universes that we have some reasonably good principles for how to set all the laws of physics and how to kind of like let things roll out and how to inject energy into the system metaphorically, you know, data and what order to show it that it does tend to consistently lead to some really interesting things at the end but our understanding of how you get from point A to point B is, and what process is that play there? or how these interactions happen, what you would call kind of like, what you would describe as biology or describe as like chemistry or kind of all these intermediate sciences that come between basic laws of physics and us sitting here having a society. Jonathan Frankle: Those are all still yet to be understood. But we do know that if you set the initial conditions just right and kind of tweak a few things along the way and nudge it here and there, actually you get some pretty interesting things out the other end. Megan Reynolds: Okay, and so the AI research is is just is the discovery is the understanding is the you know figuring out What the outcomes are going to be like how would you define Jonathan Frankle: what the outcomes are what the mechanisms are that drive this process in the same Way that you create the laws of physics you get life on earth a bunch of things in between like molecules and you know subatomic particles and atoms and chemistry and Then eventually you get more complex things like proteins and then how proteins interact And, you know, then DNA, something, something, something, again, I last took biology in ninth grade. Megan Reynolds: Yeah, I think the analogy is reaching its limit. Jonathan Frankle: But there are all these intermediate abstractions that we don't quite understand yet. And there's probably some method to the madness somewhere in there. But it really is that we have started this highly complex process. We have coarse-grained control over aspects of it that we seem to think might influence it in good directions. We have things like RL or things like fine-tuning. But at the end of the day, we really do have the basic conditions at the beginning of our universes, the end conditions that are these really interesting sophisticated models, and the research is really understanding all these abstractions and what happens on those 15 billion years in a way that's human comprehensible. Jonathan Frankle: And that's hard. And it may... We're kind of fortunate in some sense that we live in a universe where we can describe a lot of the behaviors in ways our little brains can understand. Like force equals mass times acceleration. I'm really glad it's that and not force equals like some highly complex combination of a jillion things And maybe it is underneath, but at least that simplification is good enough to there is Megan Reynolds: a way there is a way to communicate it I mean you mentioned you know fine-tuning you mentioned RL Maybe like zooming right in like you get into the work you get into work in the morning Sit down at your desk at the office You do AI research. What do you what do you actually do? What do you do in your day? Jonathan Frankle: A lot of meetings? No, the like the question I usually ask are the basic, you know, it's gonna fit into this framework very well. I ask, okay, what experiments did we run? Like basically what were the initial conditions we set up and how did they do? Did we create a bunch of molecular soup or did we create life? And was that life interesting or was that life not interesting? Did we make for whatever metrics we care about that measure the intelligence that we care about in context and I can get more specific than that. Jonathan Frankle: Did those numbers improve and what did we change and how did those numbers change and can we make any predictions about what we should keep changing in order to make those numbers improve in the direction we want and then ask ourselves are those numbers truly meaningful in the things that matter for our customers. That is kind of in some sense, if I could describe the life of a researcher in industry is that there's a lot of like fixing broken GPUs and making things run fast and understanding understanding why something stupid happened, and why was the data not good? Jonathan Frankle: Oops, we had a bug here, a bug there, but at the end of the day, it really is that question of change initial conditions, see how the outcome is influenced, and then keep changing initial conditions and try to build a model in your mind, possibly even somewhat mathematically of how initial conditions affect outcome in a way where you can make more concrete kind of specific changes to the initial conditions to change the outcome. In our case, it's about yeah, I was Megan Reynolds: gonna say give it maybe an example just to make that really tangible So I think it it makes sense as a flow, but I think the more Yeah, you can give something concrete. I think that'd be helpful Jonathan Frankle: So we're currently training models that use Databricks Databricks is a really complex platform It is it allows you to process massive amounts of information very efficiently at insane scales So when people ask me what Databricks does and I really need to explain it like to my mom I say, well, we do everything that involves really big data. And then people go to me and go, well, okay, is that interesting? And I'm like, no, really, really, really big. The biggest data sets in the world, petabytes and exabytes and things that cause entire regions of cloud providers to kind of, you know, wilt under the pressure of how much data is going through it. Jonathan Frankle: And we have this extraordinary platform. It's a very sophisticated machine that allows you to do this. And that means that you know for people like me it's an extremely accessible machine And I love to drive it, but we would love it to be accessible to everybody in the world We have a set of products called Genie that are all about allowing all sorts of users who wouldn't normally be able to hop into this you know very complex machine and Drive it to be able Megan Reynolds: to drive a machine. Jonathan Frankle: It means being able to quite simply Let's say I have hundreds of petabytes of data involving everything about my enterprise going back you know decades and I just want to ask a really simple question like my favorite question to ask on our own Databricks data is actually what was the lifetime revenue of MosaicML my old startup including all of its iterations as a separate company through the transition period integration with Databricks and the legacy products that continued after that but not new products that we called under the Mosaic AI brand but kind of the continuation of that what was the lifetime revenue across hundreds of petabytes of data at Databricks that was represented in all sorts of different forms over the years. Jonathan Frankle: It's not necessarily all consistent because the company evolved in a lot of ways. It's an extremely simple natural language question. But what is the answer? That involves sending an agent out to go and explore tens of thousands of SQL tables, notebooks, dashboards, PDF documents, emails, Slack messages to try to uncover how we did billing, how we logged that data over time, what the products were, and come back with a simple answer that's just a number. That is a really hard thing to do. That's an extremely expensive and slow thing to do if you're using an off-the-shelf agent. But you can train a custom model that knows how to explore Databricks, how to find data, what signals tend to mean things, how people naturally create data when you're in a large enterprise. Jonathan Frankle: And, you know, there are distractors. You have a SQL table, but you also have random views of that SQL table people have created and abandoned over the years. You have dashboards that people use day to day and dashboards that someone created as a one-off that are actually wrong in the same way that Google Docs tend to accumulate over time. And then nobody marks a Google Doc as obsolete. He will just stop using it. So how do you know? Megan Reynolds: How do you know? I mean, and you mentioned your custom training model. What does that mean? Jonathan Frankle: You let the model figure it out through reinforcement learning. You give the model some very hard questions. You let it try to answer them. You provide some rubrics of what success looks like. And you do this on a huge number of GPUs on a ton of data over a very long period of time. again, initial conditions, and the models tend to get better at this if you kind of set everything right, and you get really good at predicting how to change the data, change the environment, change the reward signal, you get the model to learn effectively. Jonathan Frankle: You then end up with a model that actually gets really good and really efficient at this, and we call that Genie. Megan Reynolds: And so is that AI research as a flow, or is that a form of AI research? Jonathan Frankle: That is, I think it's an example of the same high-level flow that I think anyone in AI research is doing by and large, which is some amount of trying to get a certain outcome, like get a model to be good at something, and to understand how you tweak all the knobs at the beginning of time so that 15 billion years later, it is good at that. And various hypotheses for what happens in between potentially, or we say screw it, let's not understand anything that happens in between, let's just try to fit functions that tell us how if you vary the initial conditions in a very regimented way. That varies the resulting conditions in a very regimented way. That scaling laws is one prominent example of this where you can predict based on how much data and how big the model is, you can predict for future models that are even bigger and have even more data, what performance they're likely to get by fitting a bunch of small models to this, which is now key for how pre-training happens at all the big labs. But at the end of the day, whatever your discipline or whatever your kind of philosophy on how to approach this problem, whether you're trying to understand the middle whether you believe the middle is even understandable, your goal is to be able to predict how you can change the initial conditions and improve things. For me, that involves changing the data, changing what kinds of objects I give, this interact with, what kinds of fake toy Databricks workspaces I give it and how big they are and how complicated they are. And it changes how I give feedback to the model and how I describe what a right and wrong answer look like on questions that in many cases don't have one right or one wrong answer, which is hard. Megan Reynolds: How how do you how do you do that because I guess everything that you're saying is like there's a flow in AI research you're setting their initial conditions which is also where I think these initial conditions all should be and then you're also driving towards a result that you have decided on. How do you balance those two things like to me it feels very like circular reference you're kind of building these things and figuring out where you want to go with it, like, how do you tangibly do that, like, in a regular day or in a regular year? Jonathan Frankle: How are you making decisions? I think that's what makes it really hard, is you have to ask yourself of the million different things you could change at the beginning, Megan Reynolds: which ones matter. Jonathan Frankle: So, you know, let's say I did reinforcement learning on, pick your favorite open source model, Qwen 2017. Megan Reynolds: And just a quick reinforcement learning, break that down simply before we dive into that. Jonathan Frankle: Very, you know, it's really, really simple, you give a model a problem to solve, you let it try to solve that problem, you give it feedback on how well it tried to solve that problem, it could just be a number like you got it right, you get one point, you got it wrong, you get zero points, you got it really, really right, you get five points, you've got it really, really wrong, you get negative two points, you can kind of make up your own point scale, it just has to kind of be somewhat consistent and Megan Reynolds: not too crazy. Okay, so like a child, give it stickers and... Jonathan Frankle: Exactly, give it stickers for both when it's doing well and when it's doing less well, and you know, there are algorithms to take that information, propagate it back into updating the model so it's more likely to have done well in the future on this problem, and then you proceed. Megan Reynolds: Okay, but I feel like, and so continue with the example, because I think that the frame of the question was, you know, how do you know what outcome you're driving at? And so it feels like that's also a question for reinforcement learning, like if you also need to set. So RL is a technique, you Jonathan Frankle: need to have a measure of what success looks like. And so, you know, for example, on this problem of us training our own custom models for Genie. we have a lot of different measures. There's an initial measure which just checks, for basic tiny behaviors, is it using Databricks correctly in this way, and in that way, and in that way? Because we need to check that. If it's not using this part of Databricks properly, maybe we really messed something up. Megan Reynolds: What does properly mean? Properly as a human would use it? Jonathan Frankle: Or I would say even properly in, is it calling the API correctly? Or the API request malformed? Okay, so which Megan Reynolds: is tangibly testable. Jonathan Frankle: Exactly. So, you know, basic sanity checks, like, did we break that? Or, you know, did we at least fix that from a model that wasn't doing this properly before or required a lot of instructions to be able to do this before? Did we kind of teach the model just to do this naturally? Okay, good. You know, but if that's not going right, let's say that fails. Was it because the model we started with, you know, some open source model wasn't good at this and we just didn't make it better? Was it because we didn't prompt it well? Was it because our prompt is actually a little bit out of date because someone just pushed an update to the API? Jonathan Frankle: Was it because we picked the wrong learning rate? You know, just one little parameter that governs how the weights change? Was our data bad? Was there a mistake in our data? Was our data not formatted in the chat format properly? Like, this is where the initial conditions really bite you. And where, you know, where it sounds really simple to say, yeah, just change the initial conditions and try to predict how that will affect things. okay for this one very simple right or wrong benchmark you know you can already get into a lot of trouble if you're not careful or at least have a lot of mystery to what you did and why it affected things and there's so many things you may have inadvertently changed in the initial conditions that you don't know about but that's not you know obviously if it's using one API call properly that does not mean it can use Databricks properly and solve all the problems so round one Megan Reynolds: you know Jonathan Frankle: let's say we survive that round okay next is let's give it some real tasks on some very simple fake Databricks environments and Megan Reynolds: what's what do you mean by fake Databricks environment Jonathan Frankle: so we've like you know gone and you know obviously we're very careful with data in general like data is our lifeblood so we've gone and we've made up a fake Databricks environment we've created some fake you know we've just made up a fake bank or a fake you know insurance company or something like that Megan Reynolds: okay like a fake customer of Jonathan Frankle: yours yeah completely fake customer we've created a fake Databricks workspace we've added fake SQL tables and fake fake notebooks and fake documents and fake emails and everything, because again, we want to have our own toy environments that are fully, we can fully control them, we can fully control the complexity, the sophistication, how messy they are, and it gives us a lot of control over kind of how realistic they are. Megan Reynolds: Okay, so you have a view on the outcome, so you can control the outcome somewhat. Jonathan Frankle: Well, right now this is just the playground, but now you need to set up a problem, like go and ask what was our revenue for our consumer goods products in quarter three of 2024 for this fake company, where hopefully consumer goods is not just spelled out in one obvious place. We just go look it up and bring it back, but it's actually going to figure out what products count as consumer goods and where are they? Oh, they're actually stored in three or four different places because they're in different lines of business. Make it interesting. Jonathan Frankle: And then we have some kind of rubric that says what good and bad look like. It's basically an end to end eval or an integration test kind of in software form. If the first thing I described as a unit test, this is now an integration test of, you know, end to end, did it complete a task? Like a real task that might look reminiscent of what a real user of Databricks would ask in natural language. Megan Reynolds: And where you have the answer already. So there's another method you could use on your own to go back and dig and find that. Exactly. Okay. Jonathan Frankle: Or we've kind of designed it so, you know, we knew the answer in advance and then designed the problem. Okay. So great. You know, you can verify that answer. If the answer is a number, you can verify that. But most questions people ask aren't actually one clean number. It's usually a report or a paragraph describing something. You know, asking like, you know, who, you know, let's say you're, you know, you're in retail and you're trying to sell something to, you know, to a customer. You can ask like, you know, what are the biggest headwinds for a certain demographic group for our business? Jonathan Frankle: There's not a right or wrong answer. That's sophisticated and complicated and non-obvious and so you need to have you know it's much more nuanced on how you grade that and you do want some kind of automatic grading you don't have to have a human look at that answer every time because it's you know very tedious you need to run this a lot as you're doing a lot of training runs so you have to design LLM judges and verifiers and more complex rubrics that admit a wide variety of answers and score them in some way you need to kind of grade them the way you would grade a student's essay in school Megan Reynolds: and those are all techniques for grading yes okay so you have to Jonathan Frankle: do all this just to get one number for one example and I need hundreds of these examples, that gets really painstaking and sometimes you and you want to keep updating the data in this environment like adding more stuff to it to make it more interesting sometimes you break your previous examples and don't realize it. So this is very complex but this still isn't good enough to put into production even if it passes this because this is a fake workspace that we've created it's not gonna necessarily look real and it's going to you know, we've made up all these questions. Who knows like what our users are actually asking. So then there's another step of we had to actually go and start A-B testing this in production. Megan Reynolds: I mean, Jonathan Frankle: you know, you know, sometime we have certain customers who are really excited about trying out our new features. And so we partner with them and we'll say, hey, you know, like we've got this really cool new model that we're testing out. We think it's going to be faster and better, but we're not sure and we want to, can we plug it in and give it to like, you know, well, some percentage of your queries will go to this instead of something else, instead of our old model. and like we just want your feedback. Like, did you like it? Did you notice? Can you give us a thumbs up, thumbs down occasionally? Like, is it okay if we like run some telemetry on your queries and understand what's happening? And you know, it's great to have customers who are excited about this because, you know, it's one thing to just put it out there and look at some telemetry. It's another thing to have a partner. You can just sit down and ask them like, did you like this? Did it feel good? Was it like, were the answers better? Megan Reynolds: I mean, do they have their own metrics? Cause I mean, you know, obviously there's a lot of, Oh, the vibes, it feels good. Do they have their own, I mean, are they making you off against other models? They must have, I guess, the more sophisticated ones are the companies that you're ideally testing this with, because they will be able to give you richer feedback. And so. Jonathan Frankle: So it's less other models because this is a Databricks-specific feature. There's only kind of one model out there that has been taught to use Databricks with this level of sophistication and can do this. So there's just kind of Genie. But there are many ways to power Genie. We could use, you know, a closed model. We could use an open model. We could use this custom model or that custom model. We can make various trade-offs of how slow it is versus, you know, the quality of it. And figuring out what feels good and what people actually like and what they're, you know, what is useful to them, what actually makes them productive. Jonathan Frankle: I wish there were just a number that told me that. But it's really, I mean, one of the things I love about my job is that I'm a researcher. My title is Chief AI Scientist, not Chief Customer Best Friend. But I'm allowed and encouraged to go spend a lot of time with our customers. As I told my team, the ultimate eval is, did you make the customer happy and successful? Not, did you make the benchmark go up on the academic benchmark? Did you win the leaderboard? Did you like make our internal benchmark go up? It's just make the customer happy. And if that's our eval, that means I need to go talk to our customers about whether they liked this new thing that we shipped. And if we thought it was great on all our numbers, but they didn't like it, But that's the most important signal, and the numbers weren't a good proxy. Megan Reynolds: Do you have an example of that? Like a tangible example of where internally, your team were all like, yes, we've built an amazing model for this customer. They're gonna love it. We're hitting all the things that we wanted to hit. And then you've gone and spent time with them and the customer's like, yeah, I think we might need a different approach. Do you have an example of that? Jonathan Frankle: I don't wanna give specific examples because I prefer to talk about the finished products and our beautiful work. But I will say, this is every single day. Like we're, you know, our favorite customer is our internal users. The biggest user of Databricks is Databricks. We all use Databricks all the time. And so my first guinea pig is my team. We use our own stuff. Megan Reynolds: And so what's an example where I guess a percentage of the team were like, this is a good one. We've done really well here. Like I felt really good about this. And then the, I guess the people in the team and who were using it or evaluating it, but like, oof, no, it doesn't work. Do you have an example? So I mean, it must happen, it's research, it must happen. Jonathan Frankle: So there's one, you know, there's one common pattern that happens where the models do get very good at exploiting the benchmarks or exploiting the reward. And it's this behavior, you know, it's, like people have different names for it, but scattergunning tends to be one behavior for it where when you're hoping, you know, you're looking for the model to mention certain important things as part of your evaluation, the model will start to mention everything, or like kind of give you, well, Well, if your question meant this, the answer is that. And if your question meant this, the answer is that. And kind of will basically have find a way to like give you three or four answers at once. And if you're just looking for like, did it mention all the right things, which is one very reasonable way of trying to build one of these rubrics, the answer is yeah, because the model gave you four different answers. And so it, yeah, it covered everything. Jonathan Frankle: And so it makes the numbers look really good. And then you actually have the team start interacting with the model and you're like, this sucks. Megan Reynolds: Yeah, it's just because it's just it doesn't you didn't actually prove that it was doing the thing you wanted it to do It just cheated on the test Jonathan Frankle: exactly and it may not have like, you know And that you know and then you have to go back and ask well Am I just measuring the wrong thing in my benchmark because clearly my benchmark was not a good proxy for how the model felt in Production do I need to add this is another thing I measure. How do you even know when that's the case? Sometimes users actually do ask like, you know Ambiguous questions and this is the best approach for the model to try to do this Or was it that what you were actually telling the model to optimize for was wrong? Or is it that actually the underlying model has just a fundamental tendency to do this baked in from pre-training? And actually, it's hopeless to get the model not to do this or it takes some effort to do this. You need post-processing, like again, initial condition to final result and understanding the proxy between those two and what you change, extremely hard. But this is a pretty common failure mode where you thought you measured everything And the model, like, found one other sneaky way to look really good on all the metrics and actually, and learn to do this because that's basically what you inadvertently told it to do. Or at least you gave it the option to do that and that was a great way of, you know, solving the problem. Jonathan Frankle: But it ended up being a terrible experience for the user. Megan Reynolds: And so, I mean, it's, as you say, it sounds very challenging. It sounds like there's moving pieces, the market's obviously accelerating very quickly, you have different models. To be an AI researcher, like, what does it mean to be a good AI researcher? because it sounds like what you're saying is you could do all of the steps exactly right, and still something would kind of, you know, go a little bit wrong. Yeah, what does it mean to just be an amazing AI researcher? What does that look like? Jonathan Frankle: I can only speak for what I hire, and like the kinds of people who I've hired over the years, and I've been, you know, we have our own particular culture on the team and our own particular approach, and you know, I'm sure if you asked someone at OpenAI and someone at Anthropic, they give very different answers. Megan Reynolds: Can you describe your culture? Jonathan Frankle: For me, it's about empirical rigor. Like, I don't want you to tell me that you think this is the case. I want to see numbers. If I don't have numbers, I don't believe you. Like, why should I believe you? That's the only truth we have if we are tweaking initial conditions and looking at results is numbers. And then I want you to think critically about does this number make sense and did we measure this phenomenon properly? You know, and that measure might be exactly what I just described of kind of you know okay cool you saw the benchmark go up but was the benchmark measuring the right thing like did you go and look at some of the data and see like does this number actually make sense like do we why should we believe this number it's kind of constantly questioning until you get down to the ground truth of like did we make it good and never being satisfied until you are really really confident you've made it good in the eyes of yourself and the customer and I think it's kind of that level of empirical and, you know, you can call it critical thinking, skepticism, kind of, you know, whatever you want to call it, but just being absolutely ruthless about trying to get to empirical truth and being comfortable handling the ambiguity that comes with that. Megan Reynolds: I was going to say, that feels like quite a tension. It's empirical truth. It's comfort and ambiguity because you say you want a number, but none of you on the team may necessarily know what you'd want that number to actually be. There's a number, and it should be backed up, But how do you handle that over a year of research? What does a good year look like? Jonathan Frankle: I mean, part of it is we should constantly be dissatisfied by our numbers. We should always be striving to make them better. We still need to make progress, and we should be willing to ship things that are not perfect, because the only way that you learn is by shipping things that aren't perfect. There's no substitute for making contact with reality. There's no amount of theory and planning that can help you build a model that's actually good in production. The most effective thing you can do is get it in front of users, ideally users who you have a close relationship with and who will give you honest feedback, and then iterate. Jonathan Frankle: So I'm big on like very fast ship cycles, working end to end, even when things are messy. You know, I often with my team, there's always this tendency like, but our benchmarks are so bad. They're not good yet. We know all these problems with it. We'll let you know when the benchmark suite is ready. And my nudge is always like, you have until the end of this week and whatever you have at that point is what you were calling our first set of benchmarks. And don't worry in two more weeks like every two weeks will release an updated version of the benchmark like and we'll call it v1 and v2 and v3 But you need to ship the best of what you have at any given time because we need to actually You know touch reality. We need to at least try to do end-to-end training runs You can't wait until the conditions are perfect for this because if you've done that you've missed the market And you missed your opportunity So you need to be kind of very fast at iteration and I think we've seen this across the industry now where people are realizing making contact with reality and fast iteration times with real users is really important. Jonathan Frankle: Like we see OpenAI and Anthropic now releasing updates very quickly. It used to be there was a GPT-3 and then GPT-4. Now there's 5.1 and 5.2 and 5.3 really, and each one is getting much better. Like I think across the board for both Anthropic and OpenAI it's been really impressive to watch these intermediate versions keep up with what users are doing. There's also this non-stationarity to the whole thing where as you give someone a better model or a better tool, their behavior changes in ways that mean that your tool was no longer better and they want something else. So to give you the easiest example of this, when Google Search added those AI summaries at the top that feel like you're talking to a chatbot on a single turn of conversation, it certainly changed how I use Google. Jonathan Frankle: I was always trained to put in keywords and use quotation marks and don't ask Google a question. This isn't, you know, Ask Jeeves was like, you know, was kind of in some sense a bad way of describing what you're supposed to do when you use a search engine. But now I ask natural language questions to Google because it responds like a chatbot and handles that natural language. But can you imagine the poor people who created that AI summary feature based on how people were using Google and immediately by virtue of the existence of that feature, all the user's behavior changed and then suddenly the feature probably broke immediately because it wasn't built for a behavior that didn't exist prior to the feature existing and so that constantly happens and so the faster iteration cycle you also have to keep up with your users. Megan Reynolds: And so I think that's a really interesting point and something I think a lot of people are curious about is you know we see these big model launches and suddenly you know there's a lot of chatter about well you feel the jump in the performance what is what is actually happening in between those model jumps like what's the what's AI research that is happening to improve things, like what's going on behind the scenes? Jonathan Frankle: I can give you my best guesses, speaking of someone who I don't have direct visibility into what exactly people are doing at these labs, but it's everything from they pre-trained a new model and they want to now take versions of that pre-trained model and start to adapt them into products. Megan Reynolds: And what does it mean pausing that? What does it mean to pre-train a model? Jonathan Frankle: Great question. It means starting from the literal beginning of the universe and just teaching a model on a huge amount of data what natural language looks like. And typically, when you pre-train a new model, it's because you have either new techniques that you believe will make this process more efficient or lead to a higher quality result in some way or you simply have more GPUs or more compute now and so you can do an even bigger, longer run than you could ever have done before. You have more data, you have higher quality data. You have some belief that by doing this, you're gonna give yourself a much better starting point. You're going to kind of are much better kind of you know You're it's like priming the walls before you paint them if you will and you're going to do a better job of doing that So you can paint something more interesting Megan Reynolds: and you mentioned so you mentioned techniques you mentioned kind of compute power Like length of running What are how big are those levers are they equal are they like how do you balance those things? Jonathan Frankle: So we have a lot of science to actually describe this not all of it is public anymore But at least you know in the you know a couple years ago So we had a lot of science to say the relationship between more data, more compute, and bigger models that gave you actually pretty good predictions as to what quality you would expect if you change these resources. And you can imagine if you have a very tiny model and you give it a lot of data for a long time, that may be less effective than if you had a bigger model and gave it less data. There's kind of like this trade off between, if I give you a fixed amount of resources, do you spend it having a really giant model and a little bit of data that you train It only sees a few examples a really tiny model and a lot of data because tiny models You know you can spend your budget every iteration is cheaper So it can see a lot more data or is there somewhere in between and we actually have very good science on Kind of what the right ratios are quote-unquote right or at least the ratios that will use your budget to take you the furthest in terms Of intelligence that doesn't always mean You know that model that you get is packaged best to actually serve to people that model could be very big and and big models are really inefficient to actually put on chips for inference. Jonathan Frankle: So you may actually make a choice that you're going to be less efficient. You're going to train a smaller model and more data in a way that is sub-optimal and does not use your resources as well as you could at training time, so that it's in a better package for inference. Megan Reynolds: What does it mean to be a better package for inference? Why is that important? Jonathan Frankle: Smaller and cheaper so that each token costs less. When you actually use the model and you query the model, we all look at the dollars per token or dollars per million tokens at this point. You get that number down. your underlying costs are lower because the model is smaller so it uses less compute at inference time or the model has some you know work to its architecture that makes it very efficient to serve. It may make it more challenging to train. It may actually make it less efficient at training time like it absorbs data less readily but that's worth it because it's costing you a little bit on the front end but you're going to use that model so many times when you actually serve it that it's worth spending a little bit more than you would have or using your budget less efficiently at training time so that you can use your resources more efficiently or deliver a lower latency or a faster experience to your user at inference time. Jonathan Frankle: And so you have to weigh this trade-off, you have to predict how much people are going to use this model, predict how good it's going to be, and predict if the model has a new capability, predict how badly people are going to want that. Sometimes you miss because you're trying to advance the frontier, and if it's the frontier, that means nobody's ever seen anything like it before. How do you know how badly people are going to want it? Megan Reynolds: And how well the behavior will be. Jonathan Frankle: So it's hard. These are really complicated problems. Megan Reynolds: And is there a general trend you're seeing with the labs and with where AI research is moving in an agreed upon, OK, actually, we do want smaller models because we want cheaper inference? Is there a general trend? Or are people, how differentiated are the research labs at this point? Jonathan Frankle: So from a trend perspective, I think we've seen forever that I think the most valuable thing is pushing the frontier of intelligence, I've just become convinced by that and you know And Megan Reynolds: what does that mean practically? Jonathan Frankle: That means smarter models that are more capable in fundamental ways and can do things you wouldn't have predicted before so if you're a really simple example of this when GPT-3 came out the remarkable thing about it was that you didn't have to train it to do a lot of things it just seemed to naturally pick it up from language like translation used to be something that you had to like get a lot of fine-tuning data and train a model to do translation and with GPT-3 you could do things like you know, where is the bathroom or as the French would say and then just let the model fill in the rest And it would just naturally do translation. You no longer had to teach it do that zero shot learning is what we called it Or you know what became prompting? Jonathan Frankle: That was something that was kind of new and interesting that was by pushing the frontier of intelligence We got a capability. We didn't expect necessarily or at least as a community I certainly at the time was not expecting that I was very surprised that a model could do that It sounds obvious today, but that was pretty remarkable at the time and that was only four years ago less than that So well, I guess no ChatGPT was four years ago. GPT. Anyway The the years all blend together in my old age at the end of my career Megan Reynolds: end of your long career, Jonathan Frankle: but you know the you know, so we see this happen all the time where kind of a new model comes out and With you know where we've just trained it more put in more data We've just, you know, kind of done all the things that, you know, push us further along this intelligence frontier for some vague notion of it, or we just made the model bigger and we've hoped that that's made it more intelligent. And we can ask it to do some things that it really didn't do well before and can do now. Like a more recent example is everybody was very excited by coding models that could do tab completion. You start writing a line of code, it predicts what the rest of the line is going to be and you hit tab. And, you know, everybody was like tab, tab, tab, tab, tab, and, you know, life was good. Jonathan Frankle: and then the models got good enough that agentic coding just worked. You just describe in natural language what you want, like what software you want, what you want it to look like. You give it some feedback as it writes large blocks of code. That is a new capability. That has advanced the frontier fundamentally. The models got good enough at coding that they could do something that they really struggled with before and you go from something that's reasonably popular to now agentic coding, which is responsible for a huge amount of the revenue. Jonathan Frankle: So that was something that I don't think was obvious in foresight, but was something that people had wanted to do for a long time and started actually working. And you never quite know what you're going to get when you push the frontier, because it's kind of hard to imagine something until you have it. Megan Reynolds: So I guess what you're saying is that the labs are still quite differentiated, like that they are taking things. Jonathan Frankle: Differentiated from each other or from the rest of? Megan Reynolds: From each other and you know they they are still everyone is deciding the research that they want to do I mean I guess it sounds like you're saying there isn't this agreed upon pattern of AI research that everyone is like there's not like a trend of like oh we've realized that we just need smaller models or we just need XYZ everyone's kind of taking different. Jonathan Frankle: I'd say there are macro trends in both kind of overall research approaches like we do want to push the frontier of intelligence and we believe that just bigger models and more data and harder problems to train the models on is the way to do that and there's kind of you know and then within that there are probably lots of nuances at the labs about you know what science they've done what they've seen working in their settings and you know the little details of like fundamental decisions you've made about how you operate these systems you actually change what works and what doesn't work if you look at some of the papers you know from you know the Chinese labs that have been quite open about this or Microsoft or you know Meta more recently they're like you know you can kind of tell that like they're starting from slightly different places and so slightly different things seem to work for them but you know the general trends of how you make progress here are relatively well understood it's just a matter of pushing the envelope and doing so in a really rigorous way and the industry trends tend to be reasonably well understood as well like there is demand for new capabilities like cyber security with kind of the current generational models that's come out in the past few months that was something interesting that wasn't capable before but it's not something there's serious demand for But there's also always demand for kind of more speed and you know more responsiveness like just and in computing in general like big leaps and product quality often come from making something much more responsive so we've seen fast mode for some of these agents where you know you are able to get more tokens per second and if you're coding it kind of like keeps your attention and keeps you focused on one problem you're not just jumping from agent to agent to agent and waiting half an hour for things to happen. Megan Reynolds: What's the trade-off for that speed? Jonathan Frankle: Sometimes it's nothing. Sometimes it is that you just throw a lot of GPUs at the problem. Megan Reynolds: That sounds expensive. Jonathan Frankle: It's expensive, but you parallelize. If you have a model of a fixed size and you're running it on one GPU, or you're running it on 50 GPUs, let's say the cost per token is the same. If you have enough traffic that it will keep 50 GPUs busy, you can have that model on one GPU and 50 copies of it, you can have that model spread across 50 GPUs and just be able to handle a lot more traffic in the same way that for training a model, doubling the number of GPUs doesn't necessarily double the cost. If you have perfect linear scaling and efficiency, it just means you get it done twice as fast. Jonathan Frankle: You can actually, if you have enough demand to saturate your infrastructure, you can just build out more infrastructure. But sometimes it requires fundamentally different chips that have different properties that are really good at this high throughput. And so sometimes that does make it more expensive because the chips are not produced in volume or just more expensive to use or have fancier components or things like that. And sometimes it does mean you need smaller models. And so you have to find a way to cram more intelligence into a smaller space as much as you can. And sometimes that means there is a trade-off between quality and speed. Jonathan Frankle: And that's often the case, that you wanna go and use, you know, let's say Claude Haiku or GPT Luna, those are very fast models that are known to be cheaper and, you know, not as intelligent as if you were to use, you know, Claude Opus or if you were to use, you know, GPT Sol. Those are very smart models that are a little slower. And sometimes you need that intelligence and you're willing to wait and you're willing to pay the cost. And sometimes you just want speed and the quality is good enough or speed kind of has its own kind of quality or it's a real time application. And so you need you need the speed to be iterate on that. Megan Reynolds: And how do you make those decisions internally about which models to use? You mentioned sometimes it's open source or open weights. Sometimes it's the closed models. Yeah, how are you making those decisions? Is it per use case? Is it per type of research? Jonathan Frankle: Oh, yeah, per use case, for sure. And it really revolves around the product experience. What is going to make our users successful? And is this a case where they need the ultimate quality and they're willing to wait a few minutes for this for a very sophisticated model to take its time, versus a case where they just want an answer in two seconds. And they're gonna ask a lot of questions or it's a chatbot and they need to interact in real time. Those are very different use cases and what model is best for this changes by the day seemingly right now. Jonathan Frankle: Because the closed models are continuing to get better. Gemini Flash 3.7 just came out. It's an excellent, very fast model. That was interesting and that kind of changed the calculus for products that needed to be fast in real time. Can Megan Reynolds: you give a couple of examples of like a customer use case that you would be like for sure we need to use a fast model or an open model, does that have any bearing on it? Open versus closed? Jonathan Frankle: Yeah. Anything that's real time, anything where you know it's more like a chatbot scenario or where somebody expects a response right away. Megan Reynolds: Customer service. Jonathan Frankle: Yeah. You need to use a you know a fast model. People want responsiveness especially if there's a voice component involved and it's conversational. it has to be a really small, really fast model. And I love that all the labs plus all the open source models have released amazing things in this area recently. And oftentimes that's actually ripe ground for fine tuning because you have some much more contextual intelligence. Your enterprise has some very specific data that you don't want the model to take a bunch of time making a bunch of tool calls to relearn that data every time it answers a query you want to bake that into the model. So it's ripe ground for fine tuning and customization especially in these real-time scenarios. And that's a place where I love to play personally. Megan Reynolds: Do you always need fine-tuning or what's an example? What's a situation where you maybe wouldn't need fine-tuning or wouldn't want fine-tuning? Jonathan Frankle: Oh, you often don't. Sometimes it's just that there's nothing contextual. You're okay, it's a bot for coding and the coding is very general. Yeah, these models are really well-tuned for coding. That's kind of what they're built for at this point. But the flip side is you're doing coding, you have an agent that is coding in Databricks notebooks. That's a very specific thing. There are very specific features. There's very specific APIs. There's a lot of context there on how do you Spark efficiently. There's a lot of kind of domain knowledge that we have as an enterprise. Jonathan Frankle: And my job is to take an open source model, fine tune it and bake that in because the alternative is that I need to create a bunch of prompts and a bunch of skill files and a bunch of other things to every time that a base model, whether open source or closed, is coming to this problem. It's starting fresh and I need to get all this into the prompt burn, you know Several thousand tokens before we've even gotten off the ground and then hope that that's enough for the model to learn How to do this despite the fact that it hasn't had any reinforcement learning it hasn't gotten any feedback I'm doing this right or wrong that intelligence isn't baked into the model And so it's very natural in those scenarios if we want to have a great experience for our customers Just from a pure quality of experience perspective do a lot of fine-tuning. Jonathan Frankle: That's a really natural place to be not just from a cost, but from a quality. But it usually means I can also bake a great level of intelligence into a model that would be faster than if I just used something off the shelf. And so I can get both quality and speed if I'm really careful and thoughtful about fine-tuning in the right way. It's hard and very time-consuming to fine-tune, so. Megan Reynolds: Yeah, what does it actually mean to fine-tune practically? Jonathan Frankle: Oh, it goes from just writing a prompt and looking at the output to developing an entire data set, developing these evals that we discussed, or at least better versions of it. Megan Reynolds: Like the playground. Jonathan Frankle: The playground, the environment, the reward signal. It's arduous. Megan Reynolds: Okay, so that practice in total is fine-tuning. Jonathan Frankle: Yeah, oh yeah. And the like, it's not that you fine-tune once. That's like saying you go to the gym once and then you're strong. You go to the gym every day to stay strong and stay strong in the way you want to stay strong and you get stronger over time and it compounds over time. But you know, you take a week of vacation and you're away from the gym, you know, You're you already you know atrophy a little bit and you got to pick back up Megan Reynolds: Wow, okay So that really is a constant process and does it matter if you change the model underneath like if you fine tune around a single model Can you then swap that into something newer? Jonathan Frankle: You can but there's often a lot of details that change sometimes models just have little quirks or idiosyncrasies or things You kind of need to teach them that seem to have been missed in kind of the previous process Sometimes they just have a very different architecture and sort of use them efficiently You have to do a lot of work on kind of making sure they behave well on your GPUs and your training setup up. So there's often a lot of work, but you know, at the end of the day, a lot of machine learning is just getting the data right, getting the reward right, and getting the measurement right. And so a lot of it does transfer. But if you commit to doing fine-tuning and building custom models, it's a long-term commitment. Jonathan Frankle: It's an expensive and demanding thing. And so I usually, you know, my joke with customers who want to do fine-tuning on Databricks is that I'm going to say no multiple times before I'm going to say yes that you can fine-tune because I want to make sure you've exhausted every other reasonable possibility because once you do this it's a commitment. You're kind of getting married to fine -tuning for a while. It brings with it huge efficiencies, huge cost savings and the ability to deliver oftentimes unique experiences that you can't do with just using an off-the-shelf model but it's a commitment. Megan Reynolds: What's something a customer would say to you that would convince you? Megan Reynolds: Would they know that if we can achieve XYZ performance, it will make us XYZ revenue. Jonathan Frankle: I mean, I usually look at it as, are you committed to this use case? Like is this use case popular enough that you're likely, you already have a lot of customer traffic for it. Even if the cost is too high, or like the numbers don't pencil out today, just convince me even with something that's more expensive than it should be, or slower than it should be, that your customers want this. This is a use case that's gonna be around for a while. If you do all this work and prepare all this data, and then suddenly it's actually like eh, this wasn't that important and we completely changed our product, what a shame. Like that was a lot of effort you put in. So kind of commitment and consistency and seeing that it's either at scale or scaling in a relatively predictable way the same way you would with any product. Like you're not gonna put a lot of work into making a product perfect until you see that there's actually some traction for it. Jonathan Frankle: And then it is asking like do you have the data? Like do you, can you create one of these playground environments or do you have real-world data you can work off of or you know So whatever it is, do we actually have the ingredients to be successful here? Do we have reasonable belief we can assemble the ingredients? Do you know how to measure success? I don't want you coming to me after a long fine-tuning campaign and saying, like, is my model good? We need to have a measure beforehand that we agree. Jonathan Frankle: If it does well at this measure, we're reasonably sure that we built a good model. Megan Reynolds: Which I would imagine filters out a large percentage of the people who maybe ask for that upfront. Jonathan Frankle: Exactly. But the people who do ask for it are very committed and have really strong motivation and are willing to see this through over a long period of time. But I apply the same process internally at Databricks. Lots of people want a fine tune. They've heard about reinforcement learning. They're like, oh, my product would be so much cheaper and faster if only I had a fine tune model. But are you willing to commit? And is it worth it? And I think those are fair questions to ask. I think it's kind of everybody wants their own custom model right now because who doesn't? But there are lots of ways to infuse who's your specific enterprise knowledge and your intelligence into your model, that kind of contextual or domain intelligence that lives implicitly in all of the data that you have on Databricks and in the rest of the world, there are lots of ways to infuse that into an AI agent and fine-tuning is merely one option. Jonathan Frankle: And so I try to help people understand that the goal is usually completely correct. You don't wanna just use an off-the-shelf model that's generic, you want something that is specific and differentiated for your product. Otherwise, what's the point? And differentiated for the fact that you do have decades or for some of our customers centuries of accumulated knowledge and brand and intelligence and expertise latent in all the data that you have, but how you infuse that into an agent. There are lots of choices and lots of options and fine-tuning, although it sounds really popular if you look at Twitter or if you look at LinkedIn in for those who partake, it sounds like a much better deal than it is, and you should consider the full spectrum of trade-offs. Megan Reynolds: What would you say is the biggest blocker generally for enterprises from adopting and fine-tuning a model? What's the biggest thing? Is it the data? Is it the understanding of what they're looking to get out of it? What would you say is the most consistent thing that you see? Jonathan Frankle: I think it's two things. One is that they haven't found the killer use case for AI yet in their domain They're still kind of discovering that and that's not surprising This is a brand new technology and we're still all discovering how to work with it, you know from you know From the most sophisticated users down I'm still discovering how to use AI in my life and that's a moving target So why should any business expect that they have solved this problem immediately? It's a long process. It takes a lot of experimentation. It's a product and human question not just a technical question But if you don't have a use case you're really sold on yet fine-tuning often doesn't make sense because of that commitment And the other is just in general and this is across enterprise adoption of agents. How are you going to know if your agent is good? Jonathan Frankle: Let's say that a new version of you know cloud or GPT came out tomorrow. Should you upgrade? How do you know? Just because the model looks better on the benchmarks doesn't mean it actually got better on your very specific use case You are not a general-purpose benchmark every enterprise in every enterprise use case is very distinct There's no such thing as finance or knowledge work. There's millions of individual use cases of, you know, all sorts of complexity and nuance and context. And this model is not, and a new model is not gonna get better at every single use case. There's always gonna be trade-offs that'll get better at some things and worse at others. Jonathan Frankle: Hopefully it gets better at most things. But, you know, Anthropic and OpenAI and Google don't have, you know, a benchmark for your specific thing. So how do you know? Claude, 5-point whatever, GPT, 4-point whatever, 5-point whatever, 6-point whatever comes out or Gemini 3 or 4 or 5 or whatever it is, comes out. Should you upgrade to the next version? Megan Reynolds: Do the most sophisticated companies, the more sophisticated the Fortune 500, do they have these benchmarks? Is that a practice that they've developed that they have gotten good at figuring out what they do want to drive? What's that? Jonathan Frankle: I think we're all making progress on that. I think if you ask anyone, do you have confidence in your benchmark? The answer is going to be no. Because I think even the mere practice of building benchmarks and knowing how to measure qualitative phenomena in the world is still new. I think this is a fascinating, ripe area of science. But if I ask you, like, you know, measure how good a legal brief or a blog post is, I'm going to have AI write a blog post using this basic set of information about the world and this day and time, give me a rubric that will tell you how good this blog post is. Jonathan Frankle: That's hard. It's just a really hard thing to do. It would be, and even if I had you and all your time, it would still be painstaking for you to give me ratings, but I don't. I don't even have an agent that is close to you in terms of your sophistication and your opinionation here, and I'm also giving it a rubric that's meant to be generic across the whole range of natural language answers that could come out. But even if I wanted to, even if I had a way to replicate you, how do I get the knowledge out of your brain such that I can codify it in a way where I could have somebody or an AI system or just someone less sophisticated? Jonathan Frankle: Grade it. That's really hard. And it has to be repeatable and very fast if we're going to try a lot of models and a lot of approaches and a lot of prompts and what have you. So I'd say that discipline is still new. We don't know how to do that Megan Reynolds: so no one is fully confident, but there are certain companies with certain use cases where they are Confident enough that they are willing to spend the money and spend the time to work on it Jonathan Frankle: I would say increasingly confident Increasing and there is enough of a discipline that if you want a really good benchmark for something qualitative And you're willing to put in the work you can get to something reasonably good But it is a lot of work right now, and I think for many of these things the question isn't you know Or it's not everybody should be doing this or everybody should be fine-tuning. From a scientific perspective, it's how do we make it easier? How do we use all of our knowledge of machine learning and of humans and psychology to make it a little bit easier if you'd either describe what you want or know that you got what you wanted. Megan Reynolds: And how do we do that? Jonathan Frankle: I have no clue. Megan Reynolds: What's your best, what's your, or like what's the biggest thing that you wanna work on this year that you think will make a change in that? Jonathan Frankle: I want to spend a lot of time with humans trying to get them to do these things and just qualitatively for me seeing what works and what doesn't work and what people like and don't like. It's product research as much as it is scientific research. I just want to spend a lot of time with my customers trying to help them do this. Megan Reynolds: But what does spending time with your customers mean tangibly? What are the questions you're asking them? Jonathan Frankle: I'm sitting down with them asking them, like, here's some inputs to your agent and here's some outputs from your agent. Are they good? Why or why not? What do you like and what don't you like? Like, let's see if we can write this into a more structured rubric. Let's see if we can get your buddy on the team to also look at that same rubric and rate some things and see if you two remotely agree with each other when you actually rate these things. If not, that's not a very good rubric and not a very good process. Jonathan Frankle: But it's, you know, there are numbers involved and there's measurement involved and you can start to build structure out of nothing. But in some sense, the skill of any good scientist, the skill of having a PhD, is the skill of being able to create structure on a high level question that has no structure, and at least finding a starting point and being able to work from there. That's if you learn one thing from a PhD, it's that. And here, I feel like all of us as a field are kind of starting from scratch. Jonathan Frankle: I don't think anyone looks at any benchmark in the world and says, that's a really good benchmark that fully measures this capability in the perfect way. I don't think any of us are satisfied. And coming back to what I look for for people on my team, it's people who are never quite satisfied, you know, until they've hit truth, and there's no such thing as hitting truth, so you're always just driving closer. People who are rigorous and critical thinkers, and people who are empirical. So in some sense, this is the ultimate problem, because you have to be rigorous and empirical, not just about the machine learning, but about the humans. Jonathan Frankle: And that's very hard. Megan Reynolds: And what, maybe closing on this, what do you think is the biggest unsolved problem in AI today that you want to work on? Jonathan Frankle: For me, it's all these questions around how we close the gap between very impressive capabilities in these models and very impressive intelligence and harnessing it and putting it to work in a way that we trust and that is reliable, that incorporates your context, incorporates all of the information you have and does what you want it to do. And there are, I think, two aspects of that that are nearest and dearest to my heart. One is this measurement question. How do we help people measure and know if their model is doing what they want to do, or their agent is doing what they want? Jonathan Frankle: It's an extremely hard question. It's the question of how you do performance reviews, except we don't do performance reviews on models once every six months or once a year. We want to do it constantly because we don't have good prediction about the reasonability of models, and they can do very silly things at scale. in a way that one human has a much harder time doing very silly things at scale. They can make one localized mistake usually, but not a mistake in 10,000 places at once. The other question I think about a lot is governance. Jonathan Frankle: How do you kind of give the agent access to the right things at the right times in a way that sufficiently restricts it and sufficiently enables it? And there are extreme examples of this where you say, say, oh yes, I'll give you, here's my API key to Databricks, go to town agent. And the other which is, please confirm for every Databricks table and every Databricks volume and every command that you run on Databricks, please say yes and yes and yes if you want to do this, neither of which I think are workable because agents, even if they work really well 99 % of the time, again, they can make mistakes at scale. like a human does not have enough hours in the day to make some of the kind of edge case mistakes that an agent will make just naturally because you're giving it a lot of work to do and letting it run for a long time in parallel. Jonathan Frankle: What do Megan Reynolds: you think is the sweet spot there for governance? How do you approach that? Or tell your customers to approach that? Jonathan Frankle: It's a work in progress. I'm really happy. We have an industry-leading governance solution in Unity Catalog and so it's really nice to kind of be at the tip of the spear as a member of the research team at the tip of the spear of the best product in the industry and be able to kind of start exploring that question again in collaboration with customers and figuring out what works for them because again it's a human question not just a scientific question. The right answer from a scientific perspective is highly fine-grained permissions for literally everything and make the humans spend a lot of time approving and imagining all the ways that you could combine any two permissions in in creative ways that might go wrong. Megan Reynolds: But that's not practical. Jonathan Frankle: It's not practical, and human imagination is limited on the time scale of the number of interesting things an agent can do left to its own devices. And so we need a different approach, but is it a model that specifically manages governance and flags things, which I think we're starting to see show up in the coding agents a little bit, or classifiers, or things like that? Is it just rethinking what governance looks like entirely? And especially as we get into worlds where it's not about governing a particular, you know, artifact, like governing a SQL table or a document, but it's governing information. Jonathan Frankle: When these agents have memory, that memory amalgamates different governance contexts together potentially, you know, what you know and what someone else in your team knows may get amalgamated, but you should not know everything that someone else in your team knows if you're the manager. And so managing, like, what should someone know? managing information, not managing access control in specific documents or SQL tables or notebooks or what have you is a really hard problem. But it's a very human problem. It's something that we as people are quite good at understanding like, should this person know this? Or is it appropriate for me to say this in this context? When has it become appropriate? Or is this an urgent scenario where I actually do need to share more information than I would otherwise? Or when is it time to bring someone into the fold on this? Megan Reynolds: Yeah, you have your own judgment, you have your own reasoning, but then sounds like that is something that would be possible to bake into a model at some point? Jonathan Frankle: Maybe. But the cost of a mistake is very high. You share the wrong little bit of information about some important strategy change at the company. And that can be very harmful if it's shared too soon or shared at the wrong time or shared in the wrong place. And so it's a very tricky problem that humans are quite good at. And I think as of right now, doing this, like, designing a process that does this at scale in a reliable, predictable way that kind of does what you want is extremely difficult. Jonathan Frankle: And especially with agents, this is now happening at an extremely large scale. Like, one, my favorite simple example of this, everybody has seen this, at an organization you have lots of Google Docs. And how many times have you just given up on adding each person you wanted to add to the document and made it view only for the entire organization? Nobody's going to find that unless they're looking for it. And nobody's going to go looking for it, so it's usually fine. An agent has the ability to look at everything. Jonathan Frankle: Agents can scale in ways that human governance wasn't built for, or where our current instantiations of these governance solutions assumed kind of human fallibility and, you know, that humans are constrained by time and space in ways that agents are not. And so even merely using what we have today is probably already going to be obsolete in the age of agents in the sense of at least the way that we've implemented this. So we're thinking a lot just even, you know, in using Unity Catalog, like, do we need to tighten things up a little bit to make sure that our, the governance structure that we have today is actually implemented fully in a way that is a little more watertight than it used to be. Jonathan Frankle: Let alone thinking about what the future of governance looks like in a world where, you know, we do need, you know, agents to handle kind of these things that we would say require qualitative and, you know, kind of empathetic human judgment. That's a hard thing to do. So I'm excited about it. These are incredible problems. These are problems, quite frankly, you can only work on one company in the world. Just happens to be Databricks, lucky me. But to me, those are the problems where if we don't make good progress in solving them, they're going to become major frictions on enterprise AI adoption and just AI adoption in the world because if you don't know whether an agent is doing what you want And you don't know whether an agent is handling information in a careful way That's a pretty scary thing to deploy at scale in any context in the world So I kind of treat this as like, you know an urgent like, you know You know alarm bells are ringing like we need to work on these problems not just on making agents smarter But also the rails that we're gonna put them on as we put them into, you know, our world Megan Reynolds: Scary, but exciting. Jonathan Frankle: No, not scary. I think it's incredibly exciting. I just think, you know, we've built this, you know, again, I've got all these bad metaphors. We built this incredible, very fast train and we just need to build the rest of the rails. That's not so bad. The fact that we have this miraculously good artificial intelligence technology which was unthinkable five years ago is extraordinary. We're living in a future that I don't think anyone really imagined. It's a problem of success us, that we have to think about what to do with this technology and how to make it as reliable as possible. So I don't think it's very scary to me. I'm excited. These are solvable scientific problems. Jonathan Frankle: These are things we can make a lot of progress on in short order and at least our team at Databricks can make a lot of progress on them in short order because we have amazing customers we can partner with and we have world-class implementations of the existing solutions and now we just need to bring them into the future.