Latest / The Information Bottleneck / Sara Hooker on the End of Static AI

Transcript

Sara Hooker on the End of Static AI · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the Information Battleneck Podcast. And today we have Sara Hocker and she's the co-founder and the CEO of Adaptation Lab, which came out of a satellite earlier this year with 50 million dollars to build AI systems that land continuously instead of being retrained from scratch. before she was a researcher at Cohir, she led The researcher the research at Cohir and did a lot of very cool projects that we will talk about them today. Hi Sarah, thanks for joining us. Sara Hooker: Yeah, it's lovely to be here. It's super nice. I'm delighted to have the chance to finally meet you both. So I've been following the podcast for a while. Allen Roush: Yeah, yeah. Ravid Shwartz-Ziv: Thanks. Allen Roush: It's a pleasure, Sarah, and given that you led research at Cohear, I think you you know a person by the name of Jay Alamer, I believe. Yeah, yeah. Jay Yeah, Jay Jay's Sara Hooker: Yeah, of course. Yeah, Jay's fantastic. Allen Roush: the illustrated guide to transformers was extremely Sara Hooker: Yes. Allen Roush: important to me back in r when he wrote it. Sara Hooker: Yeah. Jay is one of those unique people who thinks a lot about both like the edge of technology as well as how to communicate any one time. He's probably one of the most exceptional educators I've met, so that's really great. Ravid Shwartz-Ziv: Yeah, it it's so important these days, you know, like for example like social media, right? Like to to to be able, you know, to communicate your your research. like I found it like really important, you know, like to to to explain the research like in simple terms that like other people will understand, which are these like I I don't want to mention companies that are doing, you know, like very like yeah, PR like that we are like very like We have the best models and we will scare all the the people from one hand but still communicate what you're doing. yeah. So anyway, so maybe let's let's talk about your your your new startup, like not so new startup. what is like why we need more startups in this in this field or like what is your bet actually about the future of the of the field? Sara Hooker: Yeah, I mean I think w why we need more startups in this field probably I'm not sure if we do, but maybe I'll share why this is exists and maybe why a lot of smart people or b a lot of people are questioning like where they spend their time. Frankly, like a I'm not one of those people who like, I've always wanted to do a startup. It's just not my style. I wasn't here three years ago being like watching YC videos. It never I never thought I would do something like this. And so I think that there's probably a reckoning of like where the next channels of progress will come from. Like in some ways, I think a lot of what we see now is so much of the last 10 years can be explained by just scaling certain dynamics of like how we build intelligence, like pre-training size or amount of data. And that leads to a very exploitable structure in science. Typically you set up a lab, it's very structured around how do you scale most efficiently, it's very engineering first. I think the next era and why you see maybe what we like cheekily call neo labs or like these bats on like different directions is because those dynamics are not as clearly exploitable for the next ten years. So if you can't just double the size of a model I think the question becomes like, what are the dynamics of how you build faster intelligence? And so so much of that is in a different part of the model life cycle, arguably. It's about how do you leverage test dynamics, how do you explore the world. And that I think the recipe ends up mattering a lot again. The ideas end up mattering a lot again, which is why I I I guess we call it like the return of research, but I think that's quite a fun part of this, which is that it means that. you can have different bets and different like perspectives on where you go next. And that's why I think adaptation was important to me. I think I was the the when you start something new, you firstly have to say like, how is my idea different? Because I think in research a core thing is if you try and imitate someone else's rubric, you spend a lot of time not going very far, so you have to justify why is the idea different. And then the second is like, why does this benefit doing it from the beginning again? And for both those questions I felt like my work over the years had been really around this question of how do you do efficient adaptation. I worked a lot on multilingual, I worked a lot on efficiency. I was at DeepMind before I led the research on Metcrohe. And those multilingual is a very fun setting to to work on this because It both has real world impact, which drives me a lot in my work. I candidly like problems where if you make progress, it translates to real world impact. But it has some of the most fundamental problems in machine learning that exist. So multilingual is typically small sample, so we can't just brute force it. And also it it is a very constantly shifting environment. So language changes and evolves a lot. So that setting was very important. And that with adaptation, it's really just expanding that to any Non-verifiable tasks. Like I think there's a massive open problem there of how do we make AI work for all these different tasks and how do we allow models to adapt efficiently. So that's what brought me to adaptation. And I think that's what's made this chapter super special. I'm a very problem-driven person, so it's always been about a problem each chapter of my career. Ravid Shwartz-Ziv: So what is the solution? Okay, so now we know the problem, but what is the solution? Sara Hooker: fun. that's so fun. I love how you like but what's the solution? So, like our mission is how do we how do we teach models how to adapt as efficiently as possible? And frankly, like my North Star is real time. So, how do you teach models to identify an incoming task? And then every part of the pipeline from your data to your model to even the interface should change dynamically real time. What's the solution? So That question almost amounts to like why now? Like why do you care about this particular formulation? One is we have we have way more flexible expression in models. And I think so much of what's interesting is that when you're hosting a model, you can infer a lot about incoming tasks, right? So you can tell this task has these properties and you can do that diagnosis very quickly. So the idea that you just build a single model, you don't change any aspect of it, feels in some ways outdated. What matters more now is given you in first certain properties about the task, you should be treating it as like a meta-learning problem. And every aspect of your stack should change and dynamically condition based upon the task. So that's our focus. Like a lot of our focus is we should be changing things ranging from the weights to how you do even memory stores to how you generate data on the fly. And the goal is that that also conditions how. you explore. So if you have long horizon tasks, so I think a l a lot of the reason why people care now about continuous learning is largely because our tasks have grown much longer. But if you condition properly, you can better explore that space in a way more efficient way. W was that Allen Roush: So Sara Hooker: enough of a solution? Allen Roush: So so I I I'm intrigued by how, you know, it intuitively for most people, they think that language models and most AI systems should operate in like continuous learning, right? They think that the model is like literally training on their outputs as they they're typing in. And I guess what I'm wondering is, I mean, since you've r you're doing a whole startup on continuous learning, so clearly, you know, you've you find it very valuable. but What what do you specifically like see as behavior from models that are continuous learning? Like any differences from kind of existing paradigm of AI? Sara Hooker: Yeah, this is good. Let's back it up a bit. Like yeah, so firstly, like what do we even mean by continuous learning? And yeah, I think what you're pointing to is, you know, when people engage with a model, oftentimes their assumption is like whatever I say the model's updating and especially with context s stuffing, like is learning over time. the the truth is like most models are basically frozen in time. We ship a model, this is particularly true when When you're using an open weights model, it's basically static. It's like that's the version of the model that you'll always use. And the main goal of continuous learning is how do you actually allow these models to learn based on new incoming information? What people often miss is like the goal is actually to do that while also preserving capabilities. So you can get a model to learn new info. Like you can hammer the weights, you can fine-tune, you can basically trash all the existing information. And get a model to do what you want, that's not as fun and objective, candidly. Like the real the real pursuit is can I add a capability while retaining my general capabilities? And that's the goal of continuous learning. So why does it, you know, I think to to your point of like what's the state of the world now? I think there's a range of continuous approaches out there. Like I think that maybe I'll share it in terms of what I find interesting and what I find pretty. boring. I think boring approaches are you basically do ongoing training like in a production setting like every week you basically have some amount of data that you're incorporating. I find that boring because it's super expensive, right? And so not many my my main mandate is that there should be more companies and communities out there who can build their own intelligence and that approach is never going to get us there, right? So if you that's typically used in a few in a Industry labs where there's an ongoing productionization of models. Not that interesting, just because of the sheer inefficiency of having to curate data, balance whatever. Another approach on that side is just context stuffing. So you see a lot of this in like harnesses, and it's just you basically max out your context. Again, from my perspective, go for it, but pretty boring. Ravid Shwartz-Ziv: Mm-hmm. Sara Hooker: I think that you that people typically do that when they're using a proprietary model that they don't have control over. And when you set out to do something, my interest is mainly giving Ravid Shwartz-Ziv: Yes. Sara Hooker: people a lot more control. And so you should be leveraging all aspects of the model from the weights to to your non-parametric spaces to give maximum flexibility Ravid Shwartz-Ziv: Mm, mm. Sara Hooker: and change that. So what I find more interesting in this spectrum is like you know, you should be really treating it as a co optimization problem of like how do you alter your weights and also alter these like memory sampling spaces, exploration spaces and all that should be really tailored to the task because Frankly, for our constraint, we care a lot about efficiency. For people to to to really for this to matter to people in terms of they feel like they're getting intelligence that adapts, but also that they would use instead of a proprietary model, it has to feel fast that it's being adapted, right? So I think for that, like for us, the constraint is often that we want to incorporate new information as quickly as possible. so that's the fun part. I think that's the way more interesting sense because Frankly, like looking at the wider ecosystem, where we are now, the most intelligent model will interact. And so who does that the fastest and incorporates information from the environment the fastest? That will be one of the that that defines what the smartest model is. Ravid Shwartz-Ziv: But how do you see like the future? Do you see it like yeah, every company now I have like my own base model and I have like kind of like I don't know the infrastructure to to collect the data and like to adapt my model to the to the specific problem? Or like you think that like it will be I don't know, kind of like what thinking machines are doing, right? That like you just send the data to to to other companies that fine tune your models and and like Sara Hooker: Yeah, I'll be a bit cheeky Ravid Shwartz-Ziv: What what? Sara Hooker: Yeah, I'll be a bit cheeky. I think that it has to be that so much of the burden is taken off the end user about this this begun automatically. I think that every company should have their own model and it should be continuously adapting. The Thinking Machines approach is super developer-heavy, right? It basically assumes you already know a ton about hyperparameters. Even what we've done with autoscientists is basically the first step towards that. We don't want we want people to feel like they don't have to understand all the hyperparameters. Parameters that we automatically configure a lot of the dynamics of training just so people have much more predictable outcomes. Because the truth is, if it feels as difficult as knowing all of those con the those concepts about like how LoRa rank combines with your data and your end performance, people are just going to go back to prompt engineering. And one of the reasons, like I said at the beginning, I care about problems which translate to world real world impact. Like one of the reasons I care about both of Efficiency and also continuous learning is that I think that it's really important we give people more options to own their intelligence than just prompt engineering. I think prompt engineering is basically an itch for control, but like you basically have no actual levers to have control, and so people end up writing these massive scripts. So, yes, I think that our measure of progress is that it's way easier to own your own intelligence, and it feels as fast as like prompt engineering. Allen Roush: Do you think that fine tuning will you know, I we we see these fine tuning platforms such as like T Kinter, you know, are are is that how it's pronounced, Tinker? Sara Hooker: I think it's called Tinker. But that was but that sounded lovely as well. Yeah. Allen Roush: Tinker, Tinkinter. I I'm mixing it up I'm mixing it up with the name of a of a a a software library, sorry. but yeah, yeah, platforms like this for fine tuning as a service. what what do you think of them so far? Because it seems like for most open weights models, fine tuning, I mean there there's certain customers obviously that that live and die by it, but it seems like most customers in most domains You know, they tried fine-tuning maybe years ago, for example. there was like a financial version of Chat GPT early that Bloomberg was using. But by the time But by Yeah, yeah, but but it it always seems like by the Sara Hooker: Yeah. Sebastian German worked on that. He's fantastic. Allen Roush: time it finishes getting ready to ship the next version of the pre-trained, you know, checkpoint has come out and and often leapfrogged it. Sara Hooker: Yeah. I mean, I'd say a few comments. I think so. One is you're completely correct. I think people have tried fine-tuning of being scarred. It's worth asking why they've been so scarred. So typically, if you're outside of Frontier Lab, when you do fine tuning, you show up only with the data for the task you care about, and probably variable quality data. and you don't you you might actually care about an entirely different task tomorrow, but because this is a task a day, that's what you care about. So I would say most fine-tuning Was plagued by two things traditionally. One, a lack of ML knowledge about how do you steer these massive models. The other is like a bias towards only having the data that you have and not general purpose data. So you can't really prevent catastrophic forgetting. Here's my read. I think that unless you make it way more predictable outcomes for fine-tuning, so unless you pair like a gentic, a lot of our focusing. Even with auto scientists for training as data agents that figure out the exact right mix, all the things that we do massive long PhDs for to figure out the weighting, the distribution, and also the augmentation of data. Unless you automate also that component, and in fact, like all the science behind this the how do you build frontier models, I think it's a toll order to expect many different companies to take the tinker approach. The tinker approach is very good for if you want an excellent platform. platform for very specialized alignment techniques as well. So they do much more sophisticated sampling and online alignment. Again, I think that you may want to do that if you are a certain subset of companies. A lot of companies want some alternative and their motivators are mainly financial. They want to offload some amount of traffic onto their own intelligence to control essentially token maxing and a massive amount of spend. Where I think fine-tuning fits is it's one piece of the puzzle. Like, frankly, we should be doing, and a lot of our mandate is like every single part of the stack should be adaptable and it should be automatically tailored to your task. But fine-tuning is a p important part of that because then you can sub out certain pieces of your your ecosystem for much cheaper alternatives. That's where it resonates, and that's where it also maybe having much more predictable outcomes there and also just having Really self improving agents which configure exactly the right settings, I think that's gonna empower a lot more people to understand the benefits of that. Do I think that is the sum total of what you should do? No, because I think there's so many pieces of your owning intelligence means that you should be having a way more flexible stack and like actually meta optimizing the whole series of choices, including your harness. I actually think one of the big fascinating divides right now is that people either focus only on fine tuning or they focus solely on harness design and actually kind of like many different innovations your your best bet is you treat it as a long horizon task and you're optimizing many different pieces at the same time. Ravid Shwartz-Ziv: So you do you think like auto scientist is the the solution here? That like do you think like we should have like you know one like machine or like one system that will try to to optimize everything end to end? Sara Hooker: A hundred percent. And I'll say why. So so what do our version of autoscientist, you can think of it. I'll talk about really this mandate of like auto research and then what autoscientist is. Auto auto research is this topic where you have some improving ability to iterate on a research question from like generating a hypothesis to implementing tooling to figuring out an outcome. That I'm less like candidly immediately. bullish on, I think it's a very wide scope and very wide area. Do you have an innovative question? How does that translate? How do you avoid hacking? Auto scientists, which is what we've proposed, but is incredibly centered on a certain space. How do you optimize your training of like an algorithm, right? And like actually, if you think about it, we have like these fairly predictable building blocks. We still have transformers, so like a lot of what's built on is that. We have like a large search space when it comes to what data to include, but it's not infinite. And we have a lot of components that are like how do you do search time that I think are very exploitable. And there I see like massive, I mean, even with what we've what we've seen, you just get massive lift from just optimizing these known search bases and giving people way more predictable outcomes. And so I think empowering that self-improvement loop across AI tools, it gives massive, it enfranchises users who traditionally would have had to go get a PhD in in machine learning or computer scientists to feel like they can iterate on like what intelligent. intelligence they build, but but nice, go for it. I can tell so you nice, go for it. Ravid Shwartz-Ziv: No, so I like I I don't know. I like it can be great if you can be more specific, like because like there is like the the like this self improvement that like people at Anthropic are are like are thinking about, right? That like the the next apples will train like the the current apples like we train the the next model, right? And like we choose everything and we'll find the right hyperparameters for money. And like there are like other types of like self improvement or like auto research that yeah, this kind of like co a co-improvement, right? It like the like the the human will talk with the with the with the agent and we'll we'll discuss about like ideas and then it will go and implement it. So how do you see it? Like or like what are the like the main like components that you're actually try to optimize? It's like better hyperparameters, Sara Hooker: Yeah. Ravid Shwartz-Ziv: like different like metrics, like data maybe like generate data, filter Sara Hooker: Yeah. Ravid Shwartz-Ziv: data The pipeline Sara Hooker: Yeah. Ravid Shwartz-Ziv: to optimize the pipeline. Sara Hooker: And I think that's why I made the distinction. Like, actually, we're solely focused on all those building blocks that are go into building intelligence. So anthropic, for example you gave about anthropic self-improving their next model based upon their current that fits in this paradigm. The difference is anthropic will never release that because they think it's part of their bread and butter. Whereas on mandate is like solely driven by allowing more builders to like own frontier intelligence. And so it's everything. You described like data agents that figure out the exact most optimal data mix, which can get quite complex, right? But so when you're doing frontier AI trainings, you have many different capabilities you want to optimize for. That's a very optimizable space. I'm super bullish on like those results and massive leaps we've seen already, to things like hyperparameters, but beyond that. So, like, even your question of like memory right now, we treat kind of universally the same for old tasks, but there's no reason to do that. Like memory and how you Store memory should differ by tasks, like, there should be way more fluid representations that are tailored. Sampling budget should be adaptive, like you should use far less for some problems, far more for others. Like these are all things that can be auto-learned and self-improved based upon your knowledge of the task. And so for me, this is the most promising. That wider problem of you know, how do you how does a AI agent help with hypothesis-driven candidally as a As a co-agent formulation, I like it more, but I I still think there's a very unbounded search space there. And I'm I I think I'm less bullish that like that immediately matters. I care more about like, hey, someone already knows the question they want to answer. Let's allow them to build Frontier AI in a powerful way that gets them to the answer to their question. That for me feels like a super meaningful mission. Yeah, but what do you think? 'Cause I can tell you've thought about this. Ravid Shwartz-Ziv: And so so I think like yeah, I think like it's I I don't know. I I feel that like there there are a lot of hype, you know, especially in the Silicon Valley and like San Francisco echo chamber, you know, that like about about it and and I think like it's really hard it actually like to measure it, right? That like now you are doing like self improvement, you have an agent, right, that's like making this, like How you compare it to humans, right? Like how you compare it to to to people that actually doing it, or like people that using like coding agents, right? Like it's not clear like there is a lot of buzz about, yeah, swarms of agents that are doing like fully autonomous work and optimize my my pipeline or my my my problem. And I'm not sure that like we actually know how to how to measure it to to very d simple baseline. so yeah. I I like Sara Hooker: One name. I'll just share. I totally agree. I think it's important Different minute. One way we did it for autoscientists was that we had basically a set of researcher set hyperparameters for different models and we compared it to autoscientists. and we found autoscientists far outperformed. The main reason though I think is an interesting one, which is that all our researchers are very used to working with certain model families. So probably in your work you repeatedly use the same model families. Most frontier labs you get accustomed to using. Certain modern families and the having the ability to do a search base across all available models, it tends to favor that type of like self-improvement and self-optimization. But it is interesting. I agree. What you're pointing at is there's a very important variable, which is so unbounded swarms of agents tends to have massive implications for cross dynamics as well. So one of our sharp constraints is that has to be a factor in the equation. You want to be as efficient as possible while self-improving some of these spaces. Yeah. Ravid Shwartz-Ziv: And from your experience, like what so I I know like there are like several like recent papers that you know like look on on self improvement in in some like benchmarks and at least some of them found that like the ideas, like the depth of the ideas of the agent themselves like were quite shallow, right? Like it was like very like Sara Hooker: Yeah. Ravid Shwartz-Ziv: tiny improvement from previous idea. So what is your take? Like do you all also see it in in your work? Sara Hooker: Yeah, so I think this again gets the difference between like our we have a well-defined search base, right? So we we're optimizing the training and the optimization as well as the test time towards a defined objective. I think what you're speaking about is this more undefined problem of like the agent proposes the research or the agent proposes some solution, whereas here it's pretty clear our optimization space and what can be configured. I think you're right, for that wider problem of like you're trying to be an almost AI scientist companion. It has had super murky results. And candidly, I think that there was a great paper by led by Sayash Kapoor out of Princeton, which just came out, which again showed this on a held out test set of like actual Europe submissions, that it wasn't successful. And so, again, going back to my original point, I'm less bullish on that direction because I do think it's just too unbounded a search base, and it's really unclear if there If these are comparable across different situations, but where they have been clear test sets, which this paper was one of them, it shows very clear minimal lift and kind of more of a grumpy take, which I tend to agree with. I think when your components are known and it's an optimizable search space. This is one of the big benefits of this type of model, right? Like you can process massive amounts of information. So applying self-improvement to chart interpretation, to data stratification, to optimization hyperparameters, to test time budgets, for me that feels super clearly compelling as like a research direction, but also this plays to the forte of these models, these are known recipes. So and it has massive implications for everyday people being able to leverage it. And that's I'm much more bullish about. Allen Roush: Do do you think that models are capable of choosing how they're they they prompt or sample th themselves. So what I mean by this is just as an example, you know, you can ask models to generate the sampling settings that they will then subsequently apply to themselves before responding, and even set it up like a like a sampling router. a and you could do this, I claim, with any setting. You know, you could have the model decide for this answer, do I apply this Laura or that Laura? Like, do you think approaches like this are are part of the future? Sara Hooker: Yes. Allen Roush: Okay, good. Sara Hooker: Yes. Yeah. And I think that's again like a very clear we have known components. Like that's a very optimizable space in terms of self-improvement. Cause you have to think what is s self-improvement. You're leveraging a lot of your and your a lot of your historical data, you're leveraging what you understand the task to be and you're just dividing the search space in a much more effective way. So a hundred percent. Things like even your speculative decoding strategy, I think, are up for grabs and should change based upon tasks. And so I think that's massive lift that you can see. Allen Roush: Yeah, yeah, there are types of speculative decoding where it would run faster based on if it's like coding or not. Yep. Yep. Sara Hooker: A hundred percent. A hundred percent. And you can think of this this should even vary by verifiable versus non-verifiable task. You should change that type of approach. And so we should be exploiting that in a much more systematic way. Yeah. Ravid Shwartz-Ziv: W what what do you think about like non verifiable task? Do you think like we because again like verifiable tasks, yeah, we all know math like is a solved problem, right? but what about like other types of tasks? Do you think like we will see also like progress as we saw in in these tasks? Sara Hooker: I think that for that to happen there has to be way more care about the interface. So actually I think This is pretty important and I don't think it's talked about enough. In verifiable, non-verifiable tasks. So, first of all, let's acknowledge the importance of the most economic tasks are non-verifiable. We've just so happened to exploit the surface of code, which gets us a surprising long amount of the way there because it's connected to the internet. But a huge amount of everyday tasks are non-verifiable. This is the heart of the challenge. And what's interesting is that for those tasks, I think it's gonna be. Absolutely paramount that people care about the interface and that the interface has to dynamically change based on the task. Why do I say this? You need way more feedback along the way for non-verifiable tasks. So, where have we seen the most acceleration of progress for non-verifiable tasks? It's for design, and typically that's because, like, I think that design apps, in particular, that Leverage AI have really focused on interface. So you get a ton of micro signaling through what people choose as The image through how they apply filters afterwards, by the co-design essentially. The other major task that we've seen a lot of progress on has been code. And again, we've had very clear interfaces for engineers. That feedback loop has been super tight and it's accelerated a lot. So, code it's intertwined, it was verifiable, but it got a ton of human feedback and a lot of gold standard signal from like what people chose to check in. So I think that what's interesting is that computer scientists will are going to have to care about the interface. Interface again in a serious way, because the thumbs up, thumbs down, or the chat interface is not sufficient to give enough signal, and checklists are not coherent enough to suffice in the absence of human signal about the actual like where these components like actually used. This poses a dilemma for researchers. It actually is a bad thing because interface typically is much more linked with distribution, right? Who has control of Of the interface? I think that's a really interesting question. Like people are gonna have to do, I think there's a lot of also just fascinating innovation that will have to happen with interface. Like, you know, now we have cheap enough software, the interface should change instantly based on the task. but I also think it really means we should be co-designing some of these algorithms with interface feedback, which we don't currently do. We largely treat it as an offline process. Ravid Shwartz-Ziv: But it also means something about the speed of the progress, right? Like till now like we had this like exponential progress because like everything is prifiable and like we don't need to wait for humans, right, to give us like feedback. But now when we actually need them like for example, like we we hosted like Maxwell and go like Daphne Collar that say that look for for for science, right? Like we actually need to wait for the feedback in the real world. Like we can't have this speed, right, and like this amount of of progress. do you agree with that? Sara Hooker: So I think what's interesting is that both kind of miss the point. So I'll give you an example. So I worked on Aya for a long time, and Aya was a series of models that was expressly focused on doubling the coverage of languages that models at the time covered. So when we first started Aya, I think the most comprehensive model was covered about 50 languages, and we doubled it to 101. And then successive families really improved that coverage. It was what was interesting is that that first Aya project was probably the most painful I've ever done because we We worked with researchers all over the world, and we basically partner with them to get some coverage, sufficient coverage, for Aya, the first model. And there's two things you realize when you do that. One is that asking people for data without providing something of value back is an absolutely miserable experience. Because it's no one likes no one likes to prepare data. The second is if people know that they're preparing data that will be seen by other humans. It's a weird type of data. So it means that it's like very contrived sometimes. Sometimes it's too short because it takes too long to write long. Why do I bring up this story? So, yes, I think the lack of like human feedback for verifiable, for non-verifiable problems is an issue. Somehow we have to get more signal. I don't think the way to get that signal is gonna be through how we've acquired data in previous cycles, right? I don't think it's a simple annotation problem. I think that it's again. Getting feedback about how people actually use these workflows in their everyday. To do that, you need to create something reasonably useful and useful enough where people feel like they want it to be better and are grumpy about it. And I think we might be there with some non-verifiable things. There's certain things, I see more and more slideshows that are generated by AI and It's pretty bad, but it's Ravid Shwartz-Ziv: It's getting better. Sara Hooker: it's easy enough. People try it. Yeah, it's getting better. And so I do think it's gonna be that question of how do you leverage Ravid Shwartz-Ziv: Ha ha. Sara Hooker: frustration and like that type of informal signal. But I think that's why interface design matters. Because too much is being lost now because it's all just prompt-based and it's very high touch when someone's grumpy to give Feedback in a way that can be standardized. And I think that's a crucial piece. So I would say that more than anything. Whereas when Maxwell talks about it, he's thinking about traditional, you know, data collection and going out. And I don't think that's gonna get us further either. We have to care about the interface. We have to care about how people give this grumpy feedback in between. Ravid Shwartz-Ziv: So how how do you see it? How do you see like the future of this interface? Like is like custom interface for each person, for each task, but like how exactly it looks it will look like? Sara Hooker: Yeah. So people don't like that, by the way. They don't like too much customization interfaces. Ravid Shwartz-Ziv: Ready. Sara Hooker: There has to be some sense of stability. I think this is one of the core questions of how do you do it. I do think you have to have adaptive interfaces. So the interface should change based on what you're doing. Right now, typically what that looks like is a lot of AI slop, right? So if you go to like an interface generator, like you end up with one website, another website, you never return to them. So there has to also be some no Of data persistence. And so I think both are important. Like there has to be some reason why it becomes valuable over time. Rather than like a throwaway app for this, a throwaway thing for that. So that's where I think some shared stability of like features is pretty important. And that's actually quite interesting as a research problem. Because it's it's it you have to have some degree of diversity or tailoring, and the but enough stability where people feel like they want to return to it and it provides value. Allen Roush: So I love that you're talking about the interface in general because I've I've been highly critical of the oversimplification of everything. I get that we have to go for the lowest common denominator to chat GPT, but when we start looking at all the other kinds of interfaces that have been coming out, for example, with creative writing, they they had stuff coming out in 2023 and 2024, like Uba Booga and Silly Tavern, which is chock full of settings for configuring mostly samples. Settings. Now with open weight harnesses, we've got stuff like, you know, of course, open claw, but which took all the Ravid Shwartz-Ziv: Are you there? Allen Roush: mindshare, but stuff that's much more configurable, such as Pi and OmaiPie, or like news Hermes or Airmaz, I don't know how it's pronounced. But the Sara Hooker: Yeah. Allen Roush: the the point being, open source and open, you know, harnesses are embracing a lot of what you're talking about, even up to the idea of like kind of changing it up. based on the task that you're performing. but closed source has gone perennially the other direction and argued that they have to for safety and alignment reasons. Indeed, even things as simple as temperature, which had already been gated to between zero and point to Ravid Shwartz-Ziv: Is the Allen Roush: zero to two, were fully ripped out of being supported I think when Opus like four point six came out or Fable, Sara Hooker: Mm. Allen Roush: somet something like that. So W w w what do we do given the the the safety and alignment implications of supporting customizability? Sara Hooker: I feel like those are two different conversations. So I guess I guess the implication you're drawing is that customization or access to configuration maybe like impacts some degree of impacts a safety question. Allen Roush: Yes, correct. Sara Hooker: And maybe there are Ravid Shwartz-Ziv: Yeah. Sara Hooker: versions of this customization that do, like you spoke to temperature, Ravid Shwartz-Ziv: Yeah. Sara Hooker: but you know, arguably that's more Anthropic's general stance that the model should not be exposed. You basically have exp you know access to the prompt, not else. I'm not sure if that conflicts with an idea that the interface or how you are able to give feedback sh can be more interesting, that it doesn't just have to be prompt-based, that there can be like some UI levers that you give better feedback through that are not as granular. So it's it's possible. So that might be a separate conversation. your general point though that like most frontier labs haven't deviated from from the giving just a monolithic chat feels somewhat related to the it indeed the desire for like control over the degrees of freedom a user has, right? So maybe that's the general point. and I agree. I think that they they so their bread and butter is building proprietary models. To some degree with the slowing down of pre-training gains, so most pre-training gains now are about data, not about size. Data is a way it's a way lower barrier to overcome. So, you know, data quality is not as much of a, it's not as much, it's a much more exploitable advantage than do you have enough GPUs for like tripling the size of your model each training? So I think that they feel in general much more cornered by like what they're what is their barriers to entry, which has made them double down on like the only thing we're gonna give access to is the prompt. They're worried about any type of in general leaking of their competitive. advantage. Is that a sensible strategy? Probably not. Like we the the the the kind of The innovation and space has shifted. What's gonna dominate a lot is who is able to leverage information from an environment, incorporate it quickly, get really rich feedback from tools and long horizon settings. All of this is helpful if you have more feedback from users about what's working and what's not. OpenFor gets a ton of that, right? So they they know what plugins someone installs, they know what how the really the degree to which users want. to use different types of profiling of traces. Like this is, yeah, if you can, if you can understand and track that in some way, it gives you a lot of insight into how actual intelligence is being built in the real world. So I don't under I don't think it's a successful long-term strategy, but I think for the immediate setting of a large company that feels like they have to either from an authentic position want to control for safety reasons or from maybe a more hard to justify position, one to control for just preserving that barrier to entry. Like I think it's their their medium-term position. But I don't think it's sustainable. Most innovation doesn't progress that way by trying to control things. Allen Roush: So s so so you're w it it sounds like you're also alluding to this idea that I I've been telling investors that distillation is far more effective when you control the model that you're distilling from, which is how it was invented. It wasn't an attack when it was first coined, right? So it appears that distillation actually runs one way, right? That the Americans can distill the Chinese models, but that the Chinese have to deal with all sorts of struggles, namely no access to the model, circum anti-circumvention, you know, stuff built into the harness. do you think that China will stop open sourcing or or open accessing models at least to Americans, as a result of them maybe more quietly but but notably being, you know, vulnerable Ravid Shwartz-Ziv: Yeah. Allen Roush: to distillation? Sara Hooker: no. I think Chinese might stop open sourcing models, but I think for them it's an advantage to be the leaders in the field if they decide to open source models. It would also be hard to just do a nation-based prevention of like dissemination of open weights. Like once you release it, it's kind of hard. I would say so Ravid Shwartz-Ziv: Yeah. Sara Hooker: I do think Chinese labs probably openly, I I mean I've had conversations with a few different labs, they're pretty open, at least on a researcher. researcher level that distillation has been a large part of progress to date. So I don't think We it I'm sure it goes both ways, right? Like I I think distillation in general is is kind of a very powerful technique. Certainly within labs, people distill from their latest model and they distill back. you are correct that like proprietary models have certain perturbations, like don't reveal the real reasoning traces, often include false reasoning traces that can erode performance if you just train on it unilaterally. But there's plenty of active research about how do you How do you retroactively compose a reasoning trace that's sensible? So you can do that as well synthetically. Probably the more important thing is that it's much harder to mimic or or distill larger horizon synthetic tasks, right? So this is the big hurdle Ravid Shwartz-Ziv: Yeah. Sara Hooker: going forward. It's much easier in the realm that we came from when everything is basically a one-turn. That's that's an easier distillation. If you have a compounding task and you have erroneous reasoning traces, trying to mimic That or create your own, like it can compound error over time. also say, I I think that there has been plenty of distillation all around. Probably, I do probably agree with the assessment of some proprietary labs. Chinese labs probably benefited more in the past, but long horizon tasks probably mean it's not as clear-cut how you can just distill, and it will matter a lot more the ecosystem and how you leverage data quality. And you see that in some of the conversations with Chinese labs. They're talking now about setting up their own data. functions in a more rigorous way. They kind of know they can't do it in the same same manner now. But yeah. I think this is widely known though. I think researchers do from China do talk about this. If if you are with them one-on-one. Ravid Shwartz-Ziv: So so how why do you think like suddenly we saw like this like big jump in the Chinese model like sizes, right? Like suddenly like in like a few weeks we we saw like right like almost like three trillion parameters models. do you think this is just coincidence or like this is kind of like shared knowledge that flow around like how to train such models? Or maybe distillation? Sara Hooker: Yeah, it's actually kinda fascinating. Yeah, I mean distillation would be an argument that you wouldn't need to go big. So if they were successfully distilling, I don't know if it would be combined with so much scaling of model size. I think it's a fascinating question. I I'm curious what you both think. Certainly Chinese labs talk a lot more openly to each other than American labs. So that was another thing I found really fascinating. Was like Chinese researchers they don't see the they don't see the methodology as proprietary. They see it as very matter of fact. Like what they do find the the the the advantage for them is like it with integration of products and like also within speed of like execution so I think that that hypothesis that they shared at priori and like kind of aligned that their size was more productive I I could believe. yeah why do you both think there was this super kind of switch in scale? Ravid Shwartz-Ziv: So yeah, I I I agree with you. I think like it looks that like at some point they they they understood that like distillation will not give them enough, you know, that like we we actually need to build Sara Hooker: Yeah. Ravid Shwartz-Ziv: and like they did really like I don't know like I read like the technical reports of some of them and like they did very nice, you know, like engineering tricks that like it's amazing. Like they really understand how Sara Hooker: Yeah. Ravid Shwartz-Ziv: to how to extract so much more compute then like American loves and it's Sara Hooker: Mm-hmm. Ravid Shwartz-Ziv: unbelievable. Allen Roush: Yeah. Sara Hooker: I completely agree with that. I think that's another thing is that their competitive advantage, what they see it as, is how they make the system work, not the methodology of an individual component, which is another reason why I think they're so open about methodology. Allen Roush: Yeah. Yeah, and I mean, I've I've always attributed most of the the Chinese you know, c upsurgents to hardware innovation. In particular, going back to the Deep Seek paper, they wrote, you know, two whole pages of what I called like a love letter to NVIDIA about things they wanted Sara Hooker: No. Allen Roush: changed in Blackwell or or Vera Rubin, for to make things easier for them. It's specifically about not losing accumulated precision. on low precision like math operations. and and so I I've also heard people in the VLLM and SG Link community mention that, you know, when Chinese New Year happens, everybody goes on vacation for PRs to all the major inference engines. And so I've always thought that they were really good at, you know, accessing using hardware that might seem not as powerful and even maybe laundering it as being the main naughty thing that China does. But I've always wanted to kind of argue that distillation is is overemphasized and by Western labs and arguably used as FUD, right? Fear uncertainty and doubt. specifically on the grounds of, you know, China gives us the models, distillation is an order of magnitude more effective when you have the log probabilities. The Chinese don't get Western log probabilities, we get theirs. And so that's when my argument is that this seems like an issue that runs almost the opposite direction of what Western three letter agencies want to keep saying, you know, for example. But but hearing what you say, Sara Hooker: Yeah, man. Allen Roush: I'm reevaluating this too. I take your opinion very seriously, sir. Sara Hooker: Yeah, I I think there's probably a difference going forward to today, but to date I suspect a lot of the yeah, I I think that a lot a lot of the progress admittedly they would say is that they got to skip all the data setup Ravid Shwartz-Ziv: Yeah. Sara Hooker: of functions, all the passive human annotation because they could rely on distillation. going forward I don't think it's as clear cut. I think long horizon tasks are way harder to do placebo synthetic reasoning traces for. I just think the error Compounds in uncertain ways. Also, arguably, this latest edition of the model, it just can't be explained by distillation alone. So there's there's more going on there. Certainly, though, once you've distilled and have a fairly good model, it also becomes easier to propagate forward those benefits. So the answer is more complicated, but I don't think that should. I think the focus on distillation also misses the fact that, like, basically, Frontier Labs gave up their fight for distillation when they when they argued that you couldn't train on. The internet, although that you could train on the internet. So we also forget Ravid Shwartz-Ziv: Yeah. Allen Roush: That too. Sara Hooker: the legal landscape hasn't been decided, but basically I think it's very the same legal arguments apply. If your if your main argument is that there's some transformative aspect of how the data is being used, it also applies that by the time it goes through a model, it's being transformed and differs. So so I think it's I I don't know. I think that some of this it will resolve itself candidly. and I think that I, while it has certainly accelerated the progress of certain labs, everything here on out is much harder and it requires a lot more investment and how do you avoid compounding error on long horizon tasks. Allen Roush: I actually I wanna talk with you about licensing. and be partly because I think Kohir also did some interesting licensing stuff at the time, but I'm influenced by Kimi K three's license, where it's free to use even internally for like agentic traces at a company. But if you're a serverless inference platform, for example, Together AI, fireworks, AI, any of them from the the bigger neo clouds or hyperscalers, if you're over twenty million dollars in revenue, you have to negotiate a deal with the the Kimi. Folks. So what do you think of like like do you think that licenses will get more interesting or do you think that most will still use like MIT? Sara Hooker: Yeah, I think they've already gone more interesting. I mean, Llama was the first series to use a kind of odd tier license. They did a lot of work there early on. I think most of the Aya family was also towards the end some hedge license. So it was like free and then I can't remember where the cutoff was, but there was some Some cutoff about whether it was commercial. Candidly, I think it depends whether you think licenses are gonna be arbitrated. You know, so far we haven't had a clear case in court of like any of these pr Frontier model providers arbitrating. I think it's more of It's more of defining and trying to see what this new landscape looks like, if if you th if you if you can think of it like that. And so we'll only really know when someone tries to put that to the test. What's the logic for Kimmy? Almost certainly the logic is that this data is super valuable, right? And they don't get to see any of the data if it's not hosted by them. So their logic is very pure that they don't want an inference provider who then decides to decide it's also a Model. provider of its own, th they they want to preserve some of the work that they put in, which is very fair. So innovation typically needs some guardrails to like preserve the rights of the people who invested time and effort to innovate. Will it work? I think it's it we'll only know when the first case is brought. And then given the how murky each of these cases so far has been, it's not clear. So it's it's clear why they're doing it. It's not clear if it will work in the same way it's not clear that Meta's particular licenses will be effective in the long run. I don't know. I don't know if they've tried to exercise those rights against anyone. I'm not sure we will know unless it was brought to court, so we'll have to see. Yeah, what do you both think? Ravid Shwartz-Ziv: So I don't know I think like the general question is about like open source, right? Like what what is the the the use case or like how can you make money from from open source? So like I think this will define like all the thing that y you need to do after it, right? Sara Hooker: Yeah. And I think that Yeah. And that's why I think it's a valid the I understand both the reasoning of Kimmy and also I think it's a very valid reason, right? So they're basically saying for for for for companies with large amounts of capital who may exercise this in a way in which they're accruing all the data in a they they should. Like we should have dynamics where we preserve rewards for people who innovate. I I think the the probably the question is despite of being super valid, will we protect those? rights when it comes down to it. And I don't know the answer to that question. Ravid Shwartz-Ziv: And w what do you think about open source versus closed source models for the long term? Do you think like we will see the the gap will will continue to decrease or do you think like it will be, you know yeah. Sara Hooker: Yes. Yeah, a hundred percent. And the reason is simple is that pre-training just size alone is not enough. And so everything's moving to post training dynamics, test time compute. And there it the advantages or the barriers to entry are much lower. So innovation but having the best idea matters more. So until we change our architecture, so we've been stuck with Transformer, it's been a very sticky architecture for the last decade, it's plateauing now, it's saturating. That's why we don't see as much in terms of just Like tripling, quadrupling in size, like I think we're ripe for a new architecture. If the architecture changes, those scaling dynamics completely change in terms of the representation, then we'll see again a conversation about, you know, who knows the how to optimize that architecture. That's a totally different conversation. But as long as we are we're in this plateau, like this almost interlude between the last breakthrough and this this new. New era where it's much more about interaction and moving AI into the real world, then I think there's no clear competitive advantage and it will start to narrow pretty significantly. And in fact, we may even see, as it matters much more integrations and feedback, and efficiency, we may see the competitive advantage shift completely towards people who do open source, but also. one thing we haven't talked about is OpenCore has a massive advantage because it's almost defining the standards for the harness. And that means that it will define the way a lot of people think about well, what is the right harness? And as it has more and more plugins, or people use it in certain ways, that could end up disadvantaging severely models which don't choose to take part, right? So an anthropic has basically decided it wants nothing to do with open core. So it's pretty fascinating. These dynamics are totally shifting. I think it's wide open. For the idea to win again in a powerful way. Ravid Shwartz-Ziv: And this is like a cute shift from like several months ago, right? Because like I don't know on like February when Mythos has announced, right? Like we all thought that like, now like we are like in a different area, right? It's like huge model with like totally different level of of of performance and now like entropic will just accelerate, right? Like will use their own model to to accelerate and like and and increase the gap. and somehow like in in several months like we see totally different dynamics, right? Like we see like the open source models are much better, like other companies, it looks that like they reached kind of like the same level. Sara Hooker: Yeah, I don't yeah, I don't disagree. I think well the way mythos was portrayed probably added to that misunderstanding because so you can have a you can expand the size of a model and get much better at a narrow distribution. Typically when you add size, you'll gain better at the long tail. And I think the way that mythos was portrayed, it made it it I think it was portrayed as like a universal step forward for old tasks, but probably for a lot of tasks you still wouldn't use that size of model. You would use a different set. And I think again, what we're seeing is yes, you could probably quadruple the size of your model, but you're kind of paying more and more to learn this narrow part of the distribution. And the question is: can you get there more easily? Can you get there with like test time or something like that? And I think that's why we see this convergence as people are figuring out other ways to get to that long tail component. Ravid Shwartz-Ziv: Did you th do you think that Anthropic did like a mistake in when they bet on metals, you know, and also like all the narrative that like to yeah, we we have such a a a huge capable model and like everyone needs to to be afraid of it? Sara Hooker: A hundred percent. I mean, I think they annoyed pretty much every single person and stakeholder. They annoyed researchers because, like, how you know what is this that I'll use a model that has been purposefully crippled or I don't have access to that model, or you're take you're storing my data. They annoyed everyday citizens because of like just sheer lack of communication about pricing and access. They annoyed the government to the degree That they then took hold of the model and refused access. So, and I think what it showed most was maybe a a degree of arrogance that maybe is it showed how far apart the conversations are within anthropic from the reality of the world. And I think that that was actually maybe. I think that we hold huge power as computer scientists doing what we do now. And we should also just remember that we have to be humble and realize that these you mentioned it, but like these pockets of San Francisco, they are small parts of the world. And you we may I think oftentimes those conversations tend to reverberate to the degree that things that seem straightforward in those rooms are then projected onto the world. And it's expected that people just just accept them but the the truth is As a small handful of researchers at a as a private company don't br you know, I think it it came across as presumptuous to determine how other people should access technology. And I I suspect like part of what's happened is they've learned that that is the reaction. I have a lot of respect for Anthropic. A lot of people I greatly admire are there, but I also strongly believe that we're building technology for the world, and I think that we have to be We have to not presume that we know the moral code that every part of the world lives by. Well, who should get access to life-changing technology. Ravid Shwartz-Ziv: and it Allen Roush: I Sara Hooker: Yeah, maybe that was too grumpy, but I do feel pretty strongly about it because I think that it Ravid Shwartz-Ziv: I agree. I actually agree, yeah. Allen Roush: No. Ravid Shwartz-Ziv: I agree. You know, from someone like outside San Francisco echo chamber I I totally agree. So like maybe we can talk about AI safety, you know. Sara Hooker: I love how you said that. Like on that segregate. Yeah. No, I'm happy to Ravid Shwartz-Ziv: Like yeah, ma maybe like it's kind of like related but like not exactly like like you you wrote like the paper a while ago about compute threshold, right? That you argue that flow thresholds are like bad proxy. Sara Hooker: Mm. Ravid Shwartz-Ziv: and the question is like actually what will you regulate instead? Right? Or like do you think like or maybe like more broadly, like what types of regulation If you think there is like a good type of regulation that we actually need to apply, right? Like open source versus closed source models, like all these things are related, but like what do you think is the path when we look forward? Sara Hooker: Yeah, so the paper I wrote, it was on it was called On the Futility of Compute Thresholds. That was quite specific. Like a few years ago, a bunch of different regulations came out, including the EU AI Act, including in the US, basically saying that there was this flopped threshold that would guarantee extra scrutiny. And my main point was that This was kind of absurd because first of all, those flop thresholds would change rapidly. The second is that with improvements to optimization, more and more small models could be just as powerful. And hard-coding a threshold for fast-moving technology is completely the wrong way to do things. So it I f I think at the time it was very clear to me that setting a threshold pretty much just advantaged really like providers who would probably make much more powerful models that are much smaller. That was the trend. Like, you know, distillation really allows this to a large degree. And it was already out of date, the flop threshold by the next year. Like there'd been models that were just both much bigger and powerful and smaller models that were more powerful. What should you do instead? I think there's a few things. so on the topic of like what you should use to basically identify models, I I think percentile based, if I were to argue, is super important. And percentile based on a on a private national evaluation of some kind. So another issue that we've seen which is being epidemic is like basically labs training on test sets. So I think the whole question of like public evaluations, I I candidly don't think it matters unless it's a private evaluation that only a nation Or that governance structure has access to that they've composed to measure the risk of their citizens. And then I think that you use percentile on that test set to determine what you scrutinize. Like there's a valid counterpoint right, which is that, that's too much work for these agencies. But to be fair, you kind of already know what the big models are that are being released in a year. And you kind of already know to some degree which ones you should benchmark. And so I think that's probably what I would. Would suggest. Most of our understanding of risk is relative. So our metrics for understanding what we should apply more scrutiny to should also be relative. On the topic of open versus closed, I think so. I also think there it's been a conversation that's lacked nuance. So firstly, one is the following open weights does increase risk. I think we can all acknowledge this. Like the fact that you have Weights which can then be used in a way that's hard to trace, like this increases risk. The question which always has to be posed is is the benefits from open weights is it is the increased work risk worth it? I would say undoubtedly, because it accelerates innovation in so many places. And it also democratizes how we make progress on science. And science is super important because of how it builds on these things. So I I think that on both sides there's been a Lack of nuance on that topic. One is I think people who are very pro-open source don't like to acknowledge that there is additional risk, and you have to because there are some things we can start to do. I've been a big fan of inserting Easter eggs into model weights, so like you can think of some things or or nuances, even how it responds to certain questions that tell you whether it's that model that's being used. I think and Easter eggs can be far more complex. This is quite fun as a topic. But yeah, very Ravid Shwartz-Ziv: No, I I yeah, I have like kind of like different like perspective about like safety. Like I I think it like yeah, like and and we saw it like Sara Hooker: good, go for it. This is great. Ravid Shwartz-Ziv: in the open AI hugging face accident, right? It like hugging f like open AI was like Sara Hooker: What what did you think of that, by the way? Why did you think of that announcement? Ravid Shwartz-Ziv: So so first of all it's crazy. Like I I had like their their talk, right? Like a black head talk and and like it's crazy. It's crazy that like they found like they they didn't know this that like the the there are like I don't know how who knows like how many agents that like read and write and and and and do some something about on the on like this like writing messages till like the their server basically collapsed, right? Like and and then they just tr delete these these messages and train it f again like without like roll back from like all the all the the these models. but I think like it shows that yeah it may be that like they they they can say that really care about safety, right? But their like their core use case and their c their core business is not about that. And it doesn't really like We can't we can't regulate them and we can't force them to to care about safety as like you know as as as we want to. And I think like we must have I think like open source models and and and and and other companies that we will focus on specifically on safety and like you know like training models to to prevent and like to to to defend. from these models like we must see it and like I think like governments and and of course private sections like need to to try to develop models you know like it will be a race but it's it would be a race like similar to like we saw it before right we saw it like with the internet and we saw it like with meta with with many other like in many other like cases that like yeah they're they are like I don't know Softwares and and and and and some tools that become much much better in trying to attack, but other companies also try to develop tools and and and methods to to defend. And I think like we can't have both. Like we can't have that open AI will will develop both models, like try to to to say, you actually need to force your models not to do it because they don't have this like incentive to do it. It's not their business, like they are in a conflict of interest here. Sara Hooker: Yeah. Yeah, but I guess the counterpoint would be it should be in their business, so to some degree. Which I guess you are saying I don't trust them to Ravid Shwartz-Ziv: Yeah Sara Hooker: make that part of their business. Yeah, which maybe is fair, but I think other people would say, well, if you have such a platform and you have so many users, you have some degree of responsibility to make that part of your business. Candidly I think this will be resolved when the first individual citizens get hurt by some of these agentic workflows. I I I think it's only a matter of time. hug hugging face in some ways was a v victimless issue, right? The people who got hurt were developers, re you know, who temporarily got, you know, had a issue. But I I frankly think this will be resolved when everyday citizens have some type of, you know, bank transaction which is not being endorsed by them or there is some change to their health insurance that they did not accept. And then I think it will happen quite swiftly. I suspect governments will step in because the truth is, then it will be quite clear that the constraints of safety and commercial are very intertwined. Until then, I do hope. I mean, I think that you know, companies which have large purview should have responsibility in a mandate in these areas. I thought the most interesting thing, Candidly, about the the release. Of this disclosure was that it felt like almost everyone was a bit proud that it happened, which which I I again I think it might help that it was kind of a victimless issue, right? It's just a bunch of you know developers on Huggy Face. But that will change quite quickly when it's an everyday citizen or someone who who had something about their form modified without their permission. And so I do hope that seriousness is brought to it over time. Because now we even saw a follow-up announcements of critical issues and it almost feels like it's celebratory it's kind of celebratory in some ways. Like my my model is just as dangerous as yours, which is slightly absurd. Yeah. Yeah. Allen Roush: Yeah. Yeah, yeah. Yeah, anthropic, right? Ravid Shwartz-Ziv: Yeah, I think there was someone like in Australia, if I'm not get wrong, like like they they there was an agent, I think open claw claude or something like that, that like changed the appointment of someone, like delete some appointment and change like some medical appointment, like in order to to make it faster, or something like that. Sara Hooker: Yeah. But those immediately stand out, right? Because that's an everyday citizen and that's so I think it's that will be what changes the tenor of the conversation. And I actually suspect finance will lead the way as more and more agents are integrated into finance. There's already discussion of protocols between banks and they typically have Have much more historical mandate for for figuring out w what these transactions look like because they're very costly for them as well. So I always try and think about that. Where are incentives aligned at multiple levels to create some of the pioneering frameworks? And you're correct. Unfortunately, there is not quite the incentive structure that's aligned right now. I think I'm not as keen as you to let the c companies off the hook. I think that should be, but I do think that will happen as as we see much more Just users who are trying to use this in different ways. Open core right now, part of the issue is you still have to be pretty avant-garde to use OpenCore, right? Like you have to be like I know there's examples of grandmars in China that are already brought up, but you kind of have to be already pretty you're you're willing to buy a Mac Mini, you're like willing to invest the time and downloading. It's when some of these interfaces make it way easier for the average citizen to use it that I think we're gonna see just a rush of momentum around protecting people. Allen Roush: Yeah, yeah. The you know, Ravid Ravide is maybe a lot more rational and less reckless than I am, you know, on on these opinions, like I've I've found myself getting much more describing myself in terms like cyber libertarian as I watch how the frontier labs operate. I think a lot of the critique they have of somebody might try to make a bioweapon at home, I've you know, we've spoken to some people in the healthcare and biofield who mention that even if you have super intelligence, the weapon wet lab side of things is heavily gated by, you know, the physical Sara Hooker: yeah, I I wrote a paper on this by the way, because I was so tired of having this discussion. So I wrote a paper That explicitly was about why biorisk is the whole all the papers to date have basically only focused on the ability to retrieve information, not on the ability to execute on information quickly. I mean, not even quickly, but ability to execute. And most of the so most traditional control over biohazard is governed by supply side constrictions. And Allen Roush: Mm. Sara Hooker: candidly, that's where most of the risk has been historically, and that doesn't change because of access to information. Information because those protocols are still really strict. Who gets to own sort of materials? So I totally agree. I think the discourse around Allen Roush: Yeah. Sara Hooker: that is very inflammatory, candidly. And like, no, Allen Roush: Yeah. Sara Hooker: I think there's other risks which are super, super important and don't get enough discussion, like cyber threat, like some of the hacks that we just saw. And so, Allen Roush: Yeah. Sara Hooker: but for some reason, people love talking about bio risks. And I think I Allen Roush: Okay Sara Hooker: figured out why. It's because traditionally within governments, this is an estate. Established risk, which Ravid Shwartz-Ziv: Yeah. Sara Hooker: has brought in funding. And I realized like that's probably why it was centered, because it basically has a whole institution around it built out already. So if you want to quickly get into the discourse about national security, if you were strategically thinking about how, biorisk is very easy because it's almost automatically prioritized. It's considered one of the most egregious Allen Roush: Yeah. Sara Hooker: risks for national security. So it's super fascinating once you understand those dynamics. and why certain problems were chosen. Allen Roush: Well, in in my time the debate community, they were deba arguing everything would cause extinction every single debate round. So I've I've been listening to people claim the sky's falling forever. I tend to think that even if there are these kind of scenarios of a lone wolf terrorist with a super intelligent uncensored model, I tend to think it's a compute problem, meaning that the people who will clean it up, the rest of society, have orders of magnitude more compute and thus better in principle models to kind of overrun. your, you know, naughty, unaligned model. I mean that's what happened with Hugging Face here too. It wasn't quite as extreme, but I claim it was a Sara Hooker: There's one counterpoint. So cybersecurity, typically you can have very cheap techniques that work extremely well for cybersecurity. Because again, cybersecurity is kind of fascinating. It's about a weakness spot w point, which is which can actually be very d it's not characterized by compute cost always. most often it's it's it's characterized by ease of access. And so so many very famous cybersecurity attacks were actually Extremely cheap to deploy because you only need to find one vulnerability to get in. So cybersecurity is actually fascinating because it throws into chaos the idea that You your adversary has to be well enfranchised. They just have to scan the surface area in you know in an easier way. And I do think AI enables that. So that is one. I always tend to think about risk. I I like to have nuanced these conversations, and I think all of risk should be what's the likelihood that something happens, but also the impact if it does. I'm I don't subscribe to the idea that we should allocate a lot of resources to low probability events with massive impact. I know that's a very that's a certain framework people ascribe but I think you know if we if we lived like that we'd all be in bomb shelters in New Zealand which I think some people choose to do. Candidly I'm much more interested about you know high probability events that have you know middle to high impact I think that's a sweet spot where we're gonna see a lot of it and some of that is cyber I I do think that is fair to characterize it as pretty high risk. Allen Roush: Yeah. Yeah. I I think some of these other medium to high risk things that that we really need to be terrified about are mass persuasion, super persuasion, and then and Sara Hooker: A hundred percent, absolutely. Allen Roush: then linguistic colonialism. so langu so Sara Hooker: Yeah. Allen Roush: humans already speak like language models now, and so I'm worried Sara Hooker: Yeah. Allen Roush: that like authentic language development is effectively dead now. And v these are things that I don't hear many people talk about, you know, comparatively speaking. So Sara Hooker: Yeah, I definitely don't I think misinformation and persuasion I frankly think is people talk about it but they don't have very precise conversations about it. They basically acknowledge it as an issue and then say w Where this is our destiny, like we're basically stuck with this. And I actually think there's people forget how much interfaces impact how people perceive information. And to be honest, most people use a handful of interfaces to engage with AI. So I think there's a lot more control than we acknowledge we have over how AI information is presented. With persuasion, I think also people don't talk enough about like how AI has been used for companionship, but it's actually a massive issue. And like no Allen Roush: yeah. Well Sara Hooker: one in academia talks about it, but it's huge. It's like, you know, if you look at what Allen Roush: Yeah. Sara Hooker: are the apps that are most downloaded, AI Companionship has like some of the most extreme numbers of downloads and we don't talk about it. Allen Roush: Yeah. So so Sarah, just related to that, the LLM sampling community, all the people that are trying to push like high temperature, removing repetitive Sara Hooker: Mm-hmm. Allen Roush: phraseology like slop, our anti slop paper, the very first group of people who realized there were serious slop was the AI companionship community being like, Why is my boyfriend, AI boyfriend, always named Kale and my AI girlfriend always named Alara? Right. So I'll I wanna Sara Hooker: Ha ha ha Allen Roush: give them a huge amount of credit in whenever creativity, like creative writing type use cases come Sara Hooker: Yeah. Allen Roush: up, they are always at the forefront in some kind of like open Sara Hooker: That's so Allen Roush: source repo. Even today, model uncensoring, right? Heretic from Philip Emmanuel Sara Hooker: Yeah. Allen Roush: Wideman, who is one of my co-authors now. That's also the number one use case was AI companionship, even though I claim uncensored models are now a big deal for enterprises in the form of defenders, like cybersecurity defenders, similar Sara Hooker: Yeah, yeah. Allen Roush: to GLM five point two. So it's really weird that the the companionship community is now important for national security. Sara Hooker: I know it's treated And if you think about the things that impact human our quality of life the most, like one is how we engage with like things like human connection and loneliness. And to be honest, the fact that the academic community has completely skewed this, only Dee Yang, who's at Stanford, her group did a fantastic paper on this. But otherwise, I mean I'm sure after this episode goes out, a million Ravid Shwartz-Ziv: Mm-hmm. Sara Hooker: people are gonna come in being like these are the papers, but I just think in terms of the relative impact on humans versus how little we talk about it in like these types of settings, it's kind of absurd the discrepancy. It you know, if you think about it, I also think about that historically in machine learning. I don't know if you guys remember when the OKCupid was one of the early dating sites, they had a machine learning blog. It was it Allen Roush: Yes. Yeah, in two thousand nine. Yeah. Sara Hooker: was fantastic. It was one of the best machine learning like online digital. Digital blogs. And why it was fantastic was that you realized that dating apps in general have way more impact over people's lives than other types of technology in terms of like mixing of socioeconomic classes, exposure to different geographies. And like what's interesting is again, we kind of under-treat it in these academic circles, but it's a very visceral reminder of like this is the way that AI will have a massive impact on people. Allen Roush: I I want to point out the okay Cupid data was extremely and remains extremely controversial today because it's often Sara Hooker: Yeah. Allen Roush: used within you know black pill and incel Ravid Shwartz-Ziv: Mm. Allen Roush: communities to give explains yeah. Sara Hooker: was it? Okay. Okay. Allen Roush: Well because it it shows that men are Ravid Shwartz-Ziv: Yeah. Allen Roush: basically screwed on and not not the way they want to be. Right. And and I also Sara Hooker: Yeah. Allen Roush: want to bring up another two two interesting things related to what you said. One is the entire existence of Hugging Face's equivalent website for image models, civet.ai, no matter how much they didn't want it to be true, they effectively turned into de facto like a not safe for work website. And that's where that's where you go to Sara Hooker: interesting. Allen Roush: download all the the diffusion models for image and video generation, right? Not hugging face. Sara Hooker: Mm. Allen Roush: You go there, but you know it's so icky I had to I I had to like warn people when I was at an image generation startup about that website and that's why they don't get very much money is because they can't c get out of that use case. And then I'll Sara Hooker: Mm. Allen Roush: find I'll finalize by saying I forget if it was NURIPS 2024 or 25. But like right in the morning, I I showed up to I think it was a creative AI track, and there was a paper there called Penimatron. And basically it was like a feminist art project being like, Hey, a lot of these AI models being used for companionship are really focused on one gender in their bodies. Let's create Sara Hooker: Yeah. Allen Roush: a data set for the other genders bodies. And literally, it was like kind of kind of like performance art, but they were unironically asking the people that came up to them. It's like, hey, you want to help us on our data labeling with the Sara Hooker: Yeah. Allen Roush: implication of exactly what you're thinking? And I'm like, no, but this is hilarious. So I've I've seen at least one Nurrips paper like confront this, but only one. So Sara Hooker: Yeah. Yeah, but if but it's interesting 'cause yeah, it determines so much. And if you if you just treat it as a simple unaltered representation of like the additional features severely skew also how you find your soulmate or your partner, right? Like I mean, I think what you're referring to and I didn't realize this was a big topic, but there are features, Where you can list your preferences or who you want to show in search that basically can silo your world or super expand it. And with AI companionship as well, these apps are super popular because to some degree they're kind of sycophantic. They just repeat back to you, which I think people it feels appealing in the short term, but the medium-term implications are super interesting, right? Like what happens when you substitute for human connection something that is a bit more sycophantic? So totally deserves more. treatment. I don't know how we got on to this topic, but I'm glad we Allen Roush: Yeah. I I I Ravid Shwartz-Ziv: But Sara Hooker: did. Allen Roush: Yeah. I I I I hate syncopancy by the way. I really w wish models would stop with that. Ravid Shwartz-Ziv: But wha what Sara Hooker: Yeah, Ravid Shwartz-Ziv: do you think like are the most let's say Sara Hooker: yeah, yeah. Ravid Shwartz-Ziv: impactful topics that like AI can be, you know, can have effect to the world? You know, like everyone or like when when when asking this question they can like at least like the big companies, right, the big labs say, Yeah, we will treat cancer or like things like that, but what do you think are the the most impactful use case of AI? Sara Hooker: Yeah, I mean for me I I I really care about how do you accelerate innovation, but not just like the we often think about scientific discovery, but I think about it as like where is AI in in companies now? It's basically at Frontier Labs, a handful of Fortune five hundreds, and then it's everyday developers left to fight for themselves. There's this massive middle of like if we can empower intelligence to adapt to different companies and the questions we're exploring. I think that really accelerates like expansion of knowledge and what problems people are working on. I think in terms of tactical domains, I I I think that in general anything where there is Massive amounts of information that can now be reduced or extracted in powerful ways, I see as really powerful, right? So, or where it was it's very expensive to create customized versions of that w I think are also ripe. So in the first category, scientific discovery is interesting for a reason, right? These are small micro patterns that we're not very good at extracting. As long as there's sufficient data, the question now is: will there be sufficient data collection? I think that's super promising for healthcare in general. On the latter, where customization is traditionally being quite expensive, I think of something like education, where now even the idea of adaptable interfaces is quite powerful, right? So non-sequential processes, people coming in at different times. Typically the difficulty there is that humans don't like stability. And a lot of learning is accountability. It's people doing the same thing at the same time and me feeling like I should show up. because you showed up, so it's hard to introduce too much customization. But that is interesting, right? Like where should we have for the first time it's software is relatively cheap. And so the idea of how you should change software based on different people is pretty cool. I I think that's kind of exciting. Yeah. What do you both Allen Roush: Making it ephemeral. Sara Hooker: think? Ravid Shwartz-Ziv: No, I think like yeah, I I I kind of like agree that like th like there are a lot of topics and like it looked that like everywhere that like you try you can have benefits from from using AI, right? and it really depends on like what you actually want to do. But I I feel that like AI is like very like we need to make it, you know, just a tool, right? Like if someone is an expert, like we need to give this tool to the experts, you know, like so Like I I I did my Sara Hooker: Yeah. Ravid Shwartz-Ziv: my my PhD in computation on neuroscience and then it was like it was really amazing because like there are people that like spend their like entire life on like I don't know on understanding some like part of the of the brain or something like that. And and then like at some point people Sara Hooker: Mm-hmm. Ravid Shwartz-Ziv: from AI like came and say, so like we can use like AI to to understand the the brain, for example, right? And and and and they all failed. and I think like this combination I think w again like we need to be kind of like humble and and to understand that like we can give tools to to people Sara Hooker: Yeah. Ravid Shwartz-Ziv: but like we still there are so many domains and fields and problems that we need these expertise or like the like to in order to solve the problem we need expertise and and very specific knowledge about about this problem and this is very hard. Sara Hooker: And yeah, I totally agree. What I think about a lot is why I got into computer science. I was really, I wanted to understand how to represent the world. Like that was like my big driver. I was like, I felt like that would give you so much end-to-end control over like the question that you asked. And in some ways, we've been remarkably successful, right? Like now it feels within to some degree, like the next few decades, like you have a powerful representation, and now the question matters a lot. Like, why did we, you know, wh why did we build this representation? representation to get and so I feel like how do you empower domain experts who have the questions to feel like their work or their their iteration cycle is accelerated for me feels pretty special. because it's kind of interesting. Like our notion of like our even field of computer scientist science is almost that's the first thing we decided to cannibalize. Which is kind of interesting, but like it's really because like for us that's part of like getting to the question, you know? Super interesting Ravid Shwartz-Ziv: And this is what we know, right? Like we know how to run experiments, so like we can we try to optimize it, right? This is the first thing that we try. but the like do you Sara Hooker: Yeah. Yeah, yeah. Yeah. Ravid Shwartz-Ziv: feel that we have a better understanding of the representation these days? Sara Hooker: I think we have local optima. I think I I think so like where where is the current representation? What do we know we need to change? Like one is gradient updates are super expensive and damaging. So like continual learning is a clear outstanding question. The second is Training in general by gradient optimization is pretty unstable, right? We have to go really big, we have to train for a long time to get to a stable convergence. But probably the most important is we don't have Meaningful curriculum learning. Like we don't have the a way most attempts at curriculum learning, I know our rel has been different, but that's largely been unverifiable, which I think is a very specific case where we have a clear sense of difficulty. But outside of that, our attempts at curriculum learning have been kind of noisy. And n the other thing is we don't really discard information that easily from training. And so all these mean that I think it's gonna be very hard. The main test for d for for AI now is interaction with the world. It's understanding our physical world, it's providing incorporating information very quickly, all the aspects of transformers and these limitations are gonna make that hard. It doesn't mean that we don't have a very useful expensive way to do that. We do. It's just that it's expensive. And so I think the next question, the next breakthrough will be can you do that in a much faster way? Much more like our own intelligence, which is pretty cheap. We incorporate, we discard a lot of information. We're purposefully limited in a bunch of ways where we can't see changes that are like not log scale. We discard most unless you do something dramatic, I probably discard a lot of what you do. So it's kind of interesting. Like we we have all these biological limitations to preserve efficiency and we need to extend some of the same to odd deep neural networks. But yeah, this is kind of the crux of what lies ahead. That's what makes the next 10 years fun. Ravid Shwartz-Ziv: So do you think it's possible? Sara Hooker: In an expensive way, Ravid Shwartz-Ziv: But in a cheap way. Sara Hooker: yes. Yeah. But most problem No, no, I don't think so. And I think so one thing that people forget is that the how i how expensive something is also also dictates how people are willing to interact with something. And so I think that until things are until our intelligence is a lot cheaper for learning, it will also dictate how people are willing to give feedback, right? And and I'm talking about this much more complex problem of like AI engaging in the real world, like being able to dictate where a robot goes or like where someone navigates, I think those will require much more cheap form of intelligence and like much better incorporation of real-world knowledge. Yeah. Well also too bound to language. Language was a very big leap forward mainly because Because it's how we've stored valuable information historically. So representing it well unlocked all the historical data that humans have. But there's no reason why language should be the foundation of intelligence. Actually, there's many reasons it shouldn't be. It's like it's it's dictated by our intelligence and how quickly we can absorb information. So I think it's been a very useful leap forward because we've just unlocked a lot of information. But the key question is like, how do we represent what comes next? Ravid Shwartz-Ziv: So we we have like we're almost out of time but like maybe like just like two sentences about like there is the now like again a debate about neurosymbolics and like if like current system are like neuro symbolics and and and versus I don't know normal models. What do you think about it? Do you have a strong opinion? Sara Hooker: what's the debate? No, I don't. What's your opinion? I I tend to look unless something's been specifically optimized to fulfill a certain constraint, I tend to think I tend to try and avoid reading tea leaves about whether it could accidentally fulfill that constraint. So neurosymbolics I think falls into that category. None of these models were optimized with that objective in mind. So I would be surprised if they fulfill some of those properties. But yeah, what do you think? Ravid Shwartz-Ziv: Yeah, I think like it's kind of like mix and it really d depends on like what the question that you want actually like to ask. because yeah, like we can claim Sara Hooker: Yes. Ravid Shwartz-Ziv: the harnesses these days are like neuro symbolic, right? And like maybe the tokens are like and like they're the wrapper and now we Sara Hooker: I see. Ravid Shwartz-Ziv: need we can argue about or like debate about like what is the importance of the base model and tokenization and the harnesses and like this mix. so it really depends on the on the question that you that you actually want to ask. Right? so probably again, I don't have a strong opinion here. I think like sa the the answer is somewhere in the in the middle. yeah. Sara Hooker: Yeah, do you think Allen Roush: Yeah. Sara Hooker: it's it's more metaphorical though, or do you think it's yeah Ravid Shwartz-Ziv: Yeah. Yeah, I I yeah, I think like so so yeah, I don't know, like it really I don't think it really matters, you know, like if like a neural network that that the trend Sara Hooker: Yeah. Ravid Shwartz-Ziv: on tokens and like on like some high level concepts of of human is like neurosymbolic or not, like yeah. we have one question, like time for one question from from the from the audience. so someone asked about tokenization in IA so one hundred and nineteen countries means many language compressing information at different rates and and and current tokenizers that don't account for that. what is your view about tokenizations in different language, like many multi language settings? Sara Hooker: So I mean firstly I guess tokenization as a whole is kind of an example of of why being limited to language i is a bit tough. So tokenization in general, your your s inserting a strong prior about how information should be represented. So unlike our original CNNs where basically the filters could learn, you would set some things like you would set your your stride, you would set like the how it traversed your filters, you would have a lot more freedom about the saliency with which those were extracted. I think with with language models you learn a tokenization representation And it's almost fixed. So even for post-training, you would have to post-train for considerable time to add a new language. So that's the first thing is that tokenization is imperfect because it's a very heavy prior on what can be learned. It basically means that you have to introduce that type of language or distribution very early on in training for it to have a reasonable chance of being integrated. We we did a bunch of work on this, there's several different tokenization papers. One of which we did was like trying to figure out how could we cheaply introduce it? So at what stage of training could you start to introduce something? Because if you don't have access to pre-training, it turns out you can and like you can be clever about it. But tokenization budget also matters. Basically your tokenization is a capacity aspect of the model and it means that how you distribute that and sample for different languages matters a lot. So languages have overlap and so you can think of that as another optimization problem. So Diana Who was a research scholar and is now at Cohere did a fantastic paper on that. Like, how do you optimize the sampling for different language families? What does this mean in general? I guess there's two takeaways as we end. One is that heavy priors like that probably should they condition a lot more about what's possible for our modeling space than we think. The second is that some of that should be auto optimized. So given that you have Have a distribution that you care about, you should probably self-improve and automatically optimize that sampling. And so I wouldn't use a default off-the-shelf sampling kind of tokenization approach. And so that's kind of interesting too. So it kind of relates back to this is an optimizable space and you should exploit it fully through using something like AutoScientist. Ravid Shwartz-Ziv: And what what do you think about byte latent transformers and like entropy based tokenizers? Sara Hooker: I think they're fun. I think in general we need more especially when we know something is imperfect in a hack. So Jeffrey Hinton had a fantastic grumpy thought about I don't know if you remember Max Pooling, which was with convolutional neural networks. He always said, Allen Roush: yeah. Sara Hooker: It's terrible this works so well, because it basically would discard, it took the maximum of in-between layers. and he said, It's unfortunate it works so well, because that means we're stuck with it. And the same as tokenization. It's very important we have exploration of alternatives. Unfortunately, so far, none have had most papers on alternatives have basically been stating they match performance. And sometimes if you look at the charts closely, it's clear you have to put a bit more effort in, things like that. But we have to have those deviations because it's unfortunate tokenization works so well. Like it's clearly not the future. And so we have to be investing in different approaches. Ravid Shwartz-Ziv: Okay. Sara, we out of time. Do you have anything else that you want to add, to promote? Sara Hooker: No, this was fantastic. I enjoyed it. So Alan, send me the Ravid Shwartz-Ziv: Yeah. Sara Hooker: name of your chair. I'm impressed. It's got the grab a task component. Send it Allen Roush: yeah, I'll do that so yeah. Ravid Shwartz-Ziv: Ha ha Sara Hooker: send the link. Yeah. Amazing. Really good seeing you both and so much fun. I enjoy this a lot. It's super cool. Ravid Shwartz-Ziv: Thank you so much. It was a real pleasure. Thank you so much. Sara Hooker: Yeah. Amazing. That's cool. Hopefully you'll remember me next week, Rabid, Ravid Shwartz-Ziv: For sure. Don't worry. Sara Hooker: 'cause like but what a show. Amazing. Okay, chat to you bust soon. Ravid Shwartz-Ziv: Thank you, sir. Sara Hooker: Thank you.