Latest / The Information Bottleneck / Physical AI for Science: Why It’s Harder Than Coding | Rishabh Agarwal (Periodic Labs)
Transcript
- Ravid Shwartz-Ziv: Hi everyone and welcome back to the Information Button podcast. And today we have the pleasure to host Risha Agarwell, right? A founding member at Periodic Lab, agent professor at McGill University. Before he was at Meta, also he was a research scientist at Google DeepMind. He finished his PhD at Mila. So hi! Thanks for joining us. Rishabh Agarwal: Yeah yeah, it's my pleasure to be here. Ravid Shwartz-Ziv: So today we we mostly want to talk about physical AI and the the assumptions or how you see the the the field in the current in the current time. So let's start about maybe the the differences. how does physical AI and like the real world is different from coding agent, for example? What are the assumptions at least that You think are relevant here. Rishabh Agarwal: Yeah, so I guess there's a lot of differences and a lot of similarities. Maybe the one obvious one is well, you have to go to the physical world or the real world, while for a coding agent you can mostly stay within the computer, right? So there's a lot of things Where you can maybe let's say, I don't know, if you want to figure out how to navigate websites or something, you can just figure out how to do this within the computer. You never have to go out of it. So you can create maybe simulations of this easily where you are just trying to do this sort of thing. But let's say the difference is let's say if you're in the physical world, you're saying, I'm trying to make this material. Well You have to get out of the computer at some point to say, I want to make this thing. So someone has to mix a bunch of powders, maybe shake it, shake them, then heat it up, and then hopefully something happens. So this is maybe the key difference. And this already basically tells you where this is going, right? maybe for things like the coding agents, because it's mostly about compute, it's more or less a game of compute and how well of these environments we can create. Within the digital world. So it's like digital clones of, I don't know, maybe DoorDash, Uber, Shopify, or all these kind of things. And then you can have also doing you have GitHub where people are doing a lot of this coding work, which you can use to do something. You usually don't have this scale of data. Or record for things in the physical world. someone has to start something. Like I don't know, the the coolest example I remember seeing on I think X was there was someone who was giving you free cleaning services, except they will be recording this. So that was a very interesting way to collect data, right? That you are getting something out of it. But what the provider was getting was all the data of all like what the houses look like and where the things are and whatnot. So I think this is maybe the key difference. The other is I I guess digital stuff is usually much cheaper than physical stuff, if you think about it. There's a lot more barrier to get to the physical world, right? you're maybe saying, you ask your agent to order a coffee for you. Okay, that's nice. Sure, they can do this. I I can do that. But if you say, hey, agent, please make this new material, it's not as straightforward, right? you have to go through all you have to, for example, worry about, well, Is it going to be safe? Is this agent going to mess up something there? Is it going to produce, let's say, some sort of poisonous gas, for example? That could be a consideration which could happen. Or does it have actually the right materials to start with? Like, do I need to order all these things? So there's like more sort of barriers if I'm again going to this example. and then I guess from an AI perspective, maybe we are back to the regime of data starts becoming a bottleneck again, if you think about it. Cause I I guess maybe the other way to think about this is for coding environments and whatnot, current AI is very strong to make these environments on on their own, or we can actually ask higher experts to figure out r how to work with AI to make these digital environments per se. what do you do that for like physical environments? Like simulating a physical environment is actually a very hard problem. if you think about it. 'Cause i it's like you can't just simulate reality most of the times. if we can simulate reality, then of course we will make a l really fast progress. But it's actually much harder to simulate reality in a lot of domains. So there's no For example, maybe example is let's say this is a straw man, but let's say you want to think about that room temperature superconductor thing, which we say we want to pursue. can you just simulate that? Like, can you just think, really think, keep thinking, keep thinking forever, and and be like, yeah, I think if I do this, this will happen. Well, it's actually much easier to sort of try things out. And it's no different in maybe any sort of scientific field that you can think a lot and that can actually move somewhere, but unless you experience some new information, which is you try something out, you observe. this is what happened. That actually gives you a lot more information than what you could have done by just thinking and thinking and thinking in some sense. So it's like the clone of well, you can simulate all sorts of things in your head, but Maybe just experiencing reality is a much faster way often to get information, even if it's slower. It might be the information that you never get if you just have your own thoughts per se. So I think this changes the name of the game of let's say, even if you're trying to learn from this, you need to have some sorts of data which you can learn from. Which is also somewhat scalable, but it's never gonna be as abundant as, I don't know, the current coding environments because of just the how expensive this is and how much of a barrier it is to set up like someone needs to set up a lab to collect data. And this already sets up, like not everyone is doing this. Like I I think you maybe everyone saw Anthropics example of they were doing biology, and I think they're saying, Well, regardless of the results they're saying, they're saying, Well, we would have an agent that needs a lab to do something. If they could have an agent just I think They use this word, right? geniuses in a data center. So they could have those geniuses running forever, versus maybe some geniuses have access to the lab. And I'm guessing the geniuses with the lab probably would go further because they can experience these new bits of information that is not available to the first one. I don't know. Maybe maybe you don't. Ravid Shwartz-Ziv: Mm maybe l let's talk a bit about entropic like recent like biological announcements, Rishabh Agarwal: Yeah, yeah, sure. Ravid Shwartz-Ziv: right? Like a and because I think like it's it's a good example of how do you say it like coding like company, coding models company that that from their own reasons wants to make like some progress on biological experiments, right? And basically again they have this huge announcement, right? we found these these and these Rishabh Agarwal: Mm-hmm. Ravid Shwartz-Ziv: sites that are CRISPR or whatever. And at the end it's not like it's not even right a paper or something like that. It's not something like that, right? if you Rishabh Agarwal: Mm-hmm. Ravid Shwartz-Ziv: take a regular lab, right? like biological lab or like computational bioinformatics lab, it's not something that like w So excited, right? Like, yeah, we found something. Now we need to do the hard warp. Like it may be that like we are using some tools, right? to find it, but this is not something that like they they they will not frame it. we actually we we are using like the most advanced tools in with I don't know, thousands of agents in order to find to find something with PR and like press release. But But Enthropic d ha have done it and and and my question is like is it something like in inherent in in in every like it can be like physical labs or like like in the industry, right? It's like you want to do like you are doing a lot of things but PR and and and these like I don't know announcements are are Very important part because you need you have the investors, you you need to to to track good scientists, you have a lot of things that you don't have when you're talking about like wet lab in you the university. Rishabh Agarwal: Yeah, I think it's well I guess maybe there are two f thoughts I have here. You're right. I guess the the cadence of communication from a frontier AI lab or AI companies is much higher than let's say, and maybe much more frequent than let's say like an academic lab. And they'll they'll have something really big, they'll announce otherwise they won't, right? They'll just write some paper and move on. They won't do a big PR campaign. And part of it is Because I guess the amount of monies involved in in all this thing requires maybe some sort of Gradual or continual progress and some indication of that. And then I think there's all the side effects of how much there is hype versus how much this is real, whatnot. That's like I I think I'll leave that to the experts. Cause basically I I think I saw this thing from Dario himself where he was saying, well, back like few few years ago, AI was really terrible at doing any sort of math, right? Like p and now it's actually solving some of the hardest problems. Now. He's saying, well, maybe something like this can also happen in this field. Now it's hard to say because math and biology are very different things, or math of any other empirical field is very different. Because for math, you maybe never have to get out of the computer. Everything is there within the computer, while for these things you do. So maybe that's one huge difference. But it's also you you just can't either ignore that, either, right? In some sense. there is I guess if you go five years ago, like it's it's probably very hard to even predict that this is what would have happened. I I remember there used to be these forecasts of would AI solve these high school math pro LMPR problems in the next couple of years or not? And people are like, nope, it's not gonna do it. it's only twenty five percent, Ravid Shwartz-Ziv: Yeah. Rishabh Agarwal: fifty percent, whatever. And then it we never realized how much Ways yeah, pr progress happens in all sorts of ways. And I think when people were making these guesses or predictions, they were mostly thinking about pre-trained models and how pre-training scaling would cause this. But then we had chain of thought, then we RL, then we had more inference time compute stuff. So I think people wouldn't have guessed those things, and that's why they are more pessimistic. Similarly, I would say it's hard to ignore this for sure. It's I think maybe only this will only unfold in the next few years that we'll see how this 'cause it is Well, every day one thing I I feel is doing science is much harder than possibly creating these coding agents in some sense. And I I I guess it's also getting into the difficulty ladder, right? Now let's say you see a new version of a frontier model release or whatnot, you don't feel the same gap as you would feel from, I don't know, a couple of years ago where there was a huge boost. it went from something really useless to like, now you can use it. N versus now it is like, well, it was already good, but now it's slightly even better now. You really feel that difference or not? So I think maybe their coding is also getting into the harder regime of okay, what does the user want? Where are the actual use cases? Are what are the really problems? now we have this problem of slop. we could we everyone likes to say, everyone's like wipe coding or doing a lot of this stuff. But who is actually making sure that this is not slop or this is not gonna basically just blow in, blow and blow? so I so I would say We science definitely is already starting in a hard mode sort of thing. it's it's not because maybe one thing I would say is being here at Periodic, I think just AI as an AI researcher, I understand maybe like ten percent of the science we do. And rest is actually beyond my understanding. But then it's kinda hard to create any AI task or evals or anything, right? If you don't even understand the science part. So I need to work with the scientists and vice versa. to make sure that we understand the domain, we do something about it, and then we move on. And I think maybe this is the the difference, one other difference, you mentioned what is the difference between a coding agent versus doing the science. I think ML researchers or computer scientists, they're already experts in like they understand coding very well, right? And they can actually make things, they can see whether something is doing well or not. and generally this is true for any use case where we are also the users and experts and we understand this. Versus if you go to a domain where you don't understand, at least the current paradigm, the way it works is we need someone who understands this. Like it's not like we unleash AI on something and it can just do everything on its own. Maybe that's what people are going towards, right? But it It hasn't been true historically. the simple answer being, well, there are still researchers or LLM researchers, and a lot of them, which are in the labs, but if AI can just do everything on its own, maybe we will see very few researchers. Maybe only 1% of those researchers will be there. If the people say that, AI can just do AI research really well, and even if AI can do AI research well, can it do physics research really well? can it actually run Ravid Shwartz-Ziv: So Rishabh Agarwal: experiments? So that's a different gap. Allen Roush: I'm I'm also curious, so if AI is this good today and if open source models are, you know, can be trivially uncensored on basically on the day of their release, what kind of regulations do you think are going to happen, should happen, and you know, do you think a pause is good? Do you think extinction risk is a real thing, etc.? Rishabh Agarwal: Yeah, I think so the okay, maybe the my thought process here is there are some real risks. Like you must have seen the cybersecurity incidents which are happening, and maybe this will happen more and more because now we have access to these advanced systems that people with ill intention or maybe not even ill intention, people who just don't specify things fully can end up like causing this sort of harm. And then I guess the the extension risk extrapolation is a bit too further down for me, beyond the real risks, because At the end, the way it would happen is if we give all sorts of okay, maybe just to give you a practical example. Let's say we have a physical lab and we are letting AI run experiments all the time in the lab. There needs to be some oversight, right? Because let's say if the AI proposes some unsafe experiment, produces, as I was saying, produces a poisonous gas. And this has happened. I think there was some recent news article. There was this un unfortunate incident where I think a student ended up I don't know exactly what happened, but they ended up producing this poisonous gas, and then there were some like real bad things. because of that? but in principle you can imagine if we don't have oversight on this thing, AIs can also end up making some mistakes. And here these mistakes are much more costly. they're not just I ordered the wrong flavor of coffee or wrong size. It's it's Allen Roush: But but s sorry to interrupt, that poisonous gas thing, it did that just happen really recently at a school? Yeah, that happened at Rishabh Agarwal: Yeah yeah yeah that that happened later. Allen Roush: right next to me here at a high school twenty minutes away. I live in Oregon. I know all about this incident by the way. Rishabh Agarwal: Yeah, so you can imagine, right? if now imagine this thing, except we have these AIs which are basically controlling labs without any oversight, or maybe only AI oversight. How much do you trust? So you can't really if if someone does that, I think, that then we are in this big risky zone of, okay, well. These kind of and maybe it's will be still restricted to the lab, but then still these kind of things can happen in principle. So so I think the scenario where I see these kind of things happen is where we really just give away all sorts of controls to AI in some sense. it's not being overseen. And there are already regulations here, right? Which is like, I think even when human scientists are doing this, they have to make sure they are aligned with the safety regulations and they're not doing some of these things which will lead to this. Some kind, some kind of mistakes can still happen. But there is a lot and lot of regulations. And I will trust there is no reason that the same regulations wouldn't apply to AI experimentation, right? Like you would have to meet the same bar. We wouldn't like, we want to accelerate, Ravid Shwartz-Ziv: Yeah. Rishabh Agarwal: so let's just change the bar for AI. Is that so that's where I stand on all of this? That at least in a lot of these fields, all these things exist, and we should just piggyback on those. Like for example, if you hack someone or you do something bad. We will come after you. the there should be rules that basically will say, you did a bad thing. actually this is this is not good, right? And now let's say you do the same thing for AI. that, AI did a bad thing. Do you just call AI a third party and say, AI did a bad thing, not me? I don't know. Like worry, like think about control the AI or figure that out. So I I think that that's maybe the the part where the current regulations have to figure out how to deal with that ambiguity per se. But my feeling is there are already these regulations which exist and apply to humans. And maybe we should think about, well, AI is no different if it is actually given the same affordances as humans. If it can do the same sort of things as humans are allowed to do, so maybe it should also follow similar sort of laws in some sense. And this makes things harder, right? Like it's like it's more, it doesn't let you accelerate as fast as possible, if you think about it. But it is the right thing to do. Like otherwise you're risking a lot in some sense. so s for example in our own lab we would have a lot of these safety focus and human scientists I guess looking at it and then there is a tension, right? Like you can't scale this fully, so there has to be some way so that we can have AI overseeing AI, but then also humans also involved in the loop somewhere. And if AI becomes smart enough that maybe we are very confident to not have human oversight, that's to be seen. It's unclear. But it depends on the stakes, I would say. So so I'm not Yeah, maybe this is my non answer, but this is my sort of opinion on all this other thing. Ravid Shwartz-Ziv: I I want to to to go back to physical physical Rishabh Agarwal: Mm-hmm. Ravid Shwartz-Ziv: science and and AI. Can can you describe us like the the process, right? Like what parts like you actually you're doing, which one you actually need you need experts, which one like you what are the bottlenecks Rishabh Agarwal: Yeah. So I guess Ravid Shwartz-Ziv: are? Rishabh Agarwal: Yeah, so I guess if you think about any sort of scientific process, it's generally like basically you have a loop of sorts, right? You want to figure out some you have some sort of hypothesis. So you start there always. so this is like let's say the starting point of a loop. Then you want to figure out a way to test this hypothesis, which is like how do I try this out? And then let's say you do end up doing this, you end up seeing some sort of Ravid Shwartz-Ziv: L let's can you give us a specific example? Rishabh Agarwal: Yeah, yeah, yeah. So let's say if you have a material science lab, which we do have, it looks like, well, you want to start with something like, what material do I want to make? this is the thing which will be really amazing. Let me try to make this thing. Or this is the thing I want to make. Okay, cool. So we decided, agreed on this. So let's say just for sake of argument, we decide for some really amazing material. Now the question becomes, okay, how do I make this thing? Because no idea. It's like it's a new thing. Assume it's a new thing, right? It's not something that people have already done. So you you're starting with, okay, well, let me try out and then you I guess Let's say you also figure this one out. Let's say you figure out how to make this thing. Maybe you don't know. Maybe you try something out, right? Because here you don't really know. And then you actually measure or characterize this thing, which is what did you make? Because when you do these kind of things, it's not like, you get a really nice answer of you made XYZ. You actually get some high-dimensional measurements of sorts. So I don't know if you've guys seen the movie Project Hail Mary, but they had one. Allen Roush: Yeah. Rishabh Agarwal: Yeah, so they had this one device, which I think which was called S C E D S, which is one of the machines we have in our lab. But basically what it does is it is basically zooming onto the material. To a really, really small scale, shooting hydr like electron beams at this thing, and using that as trying to figure out the composition of the material. So I think if you remember that scene from that movie, it was about, there was this alien particle or something. And I think Rand Gosling had to figure out what was it made of. And then he was very sure that, well, it's not oxygen, hydrogen, and carbon only. There should be something else. But then he got really like upset that it was just oxygen, hydrogen, and carbon. But the thing he put that particle in was this machine called Scanning Electron Microscope. Basically, and then there is a way to analyze. So you get these high-dimensional observations thing, which gives you some idea about what you make. And then this is like an analyses problem. So this is the loop. And let's say you now you figure out, okay, you started with what do I want to make, you tried something to make it. Now you observe some sort of measurements of what happened. You analyze this to figure out what did you actually make, because maybe you didn't make the thing you wanted, and then you repeat this procedure, right? Because now you have an idea of if I tried XYZ, this has happened. And now the funny thing is, all of these have both physical bottlenecks and also intelligence bottlenecks. Well, I guess maybe the how-to-make part is the most intelligence bottlenecked. It's really thinking about, okay, what is the most amazing material that I should be trying to make? This is mostly an intelligence bottleneck problem. This is where human scientists or experts come in with their taste. It's like the same as us guys doing research or whatnot, where we're like, I think this is the idea which will really be an interesting idea. We should try this out. Cool. Now, the how-to-make part. again becomes both intelligence bottling. Well, you need machines to do this. you need a lab to actually mix powders, do some sort mix basically you shake them vigorously or something or mill them and then you heat them up. So you need furnaces for that. And maybe there are this is the simplest way, which is called shake and bake, I think. But there are other ways of making stuff. But you need equipment and robots and also technicians for this sort of thing because not everything can be automated either. so this is like one the physical side of things. And then there's the the intelligence side of things, which is well, what is the recipe? Like it's like a cooking recipe almost, except it doesn't help if you say, you wanna make a cake and I say, yeah, just mix some sugar, flour, mix them very well, butter also and then just put them in an oven and that's it. Like if you follow that, you're not going to make a cake. So so ba basically Ravid Shwartz-Ziv: But Rishabh Agarwal: sorry? Yeah. Ravid Shwartz-Ziv: but but this part like you basically you analyze the data, right? Rishabh Agarwal: No, no, no. This is basically you're saying it's like you said, I wanna make this amazing material. Now you have to ask, how do I make this material? Scientists will say, This is close to some other material in the literature, maybe this would Ravid Shwartz-Ziv: Mm-hmm. Rishabh Agarwal: work. So they're also making a guess based on their intuition and all the work they've done over their course of their life. But you can easily see, right, why this is also an intelligence problem. Because really it's trying to figure out something that how do I do something even though no one has done it before, And then so I I guess that this is the intelligence part of it. And then finally, the analysis problem is again funnily a intelligence problem, because it looks simple, it's analyses, right? But then one of the problems and I think this is what we released, blog post also, on this thing called XRD characterization. So basically what you get is you make a material, you put that in a machine. And now you get a bunch of these patterns that you have to figure out what exactly is this thing. And this is like a pattern-fitting problem. But you have the same classic overfitting issue as we have in machine learning, which is you can fit the pattern really well, but does it have physical plausibility? Does it make sense? Sort of thing. And then some of these patterns are really nasty, because a lot of this have you have to think about the previous experiments, the databases out there, you have to also run some simulations. So there's a lot of context that goes into figuring out this analysis problem. And I think. Some of these problems are nasty enough that it takes hours for our even the most expert scientists to just figure out. Because let's say you're trying to make something, you're expecting something, but that didn't happen. Something else happened, and this happens a lot, especially when you're trying more exploratory stuff in some sense. So so now you are bottlenecked again by intelligence. Because you just don't have enough scientist. let's you run 100 experiments, you kinda need like hundred scientists doing this analysis. Because it it's not just a analysis problem, it's also about all the context around it, which says, what was exactly run? Do I expect this to happen or that to happen? And what does the like the the measurement? So It's like think of this as the measurement tells you some information about reality, but it's not the complete information. And then there's a lot of intuition and I guess previous experience that goes into deciding how to interpret this measurement. So it's very yeah, it it it intuition can be thought of as mostly intelligence here in some sense. So this is the other part. So there are each of the parts, I guess except the hypothesis proposal. You need basically physical labs or labs to do something. So running this XRD thing, you need a machine which actually runs this thing. And the funny thing is I think the video I think our blog released some video or cool video of this machine measuring this. So the the I saw this this is likely a correlation. But the the I think the company which makes this machine, their stock price went up by like ten dollars. So they got like two billion dollars dollars more in like market Ravid Shwartz-Ziv: Mm. Rishabh Agarwal: cap. Cause there was excitement about, maybe AI is actually gonna use more. Because we need these machines at the end of the day to to have a lab. You need these, so for example, you might have heard these so there are machines even for things like if you put the material in this thing, it'll measure whether this is a superconductor or not. These machines do cost a lot, I think, but they can actually tell you clearly. So there's no doubt about this. You put your thing in this thing and it can tell you whether it is superconducting or not. So you can actually get away, rather than hypothesizing, might as well measure what happened. So now the name of the game is basically each part of this loop is intelligence button-linked. Once you figure out the capacity bottleneck, which is, well, if you have enough capacity to run experiments and enough machines to measure what is coming out of these experiments, someone needs to go in and analyze and then think about, what is the next experiment I want to run, right? And repeat this process of okay, this is the next experiment I want to run. What exactly do I do so that I make this successful? Okay, this is what happened, and you repeat. So this is like the loop that kind of human scientists carry out and has been carrying out forever. I think the most interesting example I know of is there was this guy called Bernd Matthias, I think, which made like more than 100 superconductors or something over his lifetime of like 30 years of research. He probably ran something around like 10,000, 20,000 experiments total. So he went through this loop 10,000 to 20,000 time. And he's probably the most productive person to do this. But now you can imagine that what we have is, let's say, if we have the same scientist, Matthias, but he can do I don't know, a million experiments, what would he have done? that's a counterfactual. and we don't know really the answer. We just know what exactly he did. But this is maybe the regime we are trying to go with AI, that if we have basically AI as smart as the existing scientists, right? Not even like smarter. We all want is like human-level intelligence, let's say, of the best possible scientist we can have. And if it can go through this loop, basically, a million times, then the hope is well, something really nice comes out of it, is the the expectation. now this is the high-level pitch. The details, and there's a lot of devils in the details, right? Like it's just like I'm just setting up the ground for why this is feasible, because I I I think typical academic labs do not have this kind of throughput, but they do have a lot of intelligence, right? Ravid Shwartz-Ziv: Why? Why w wait a second, why they don't have is it like the the machines? Is it like the coding agents that are running Rishabh Agarwal: Yeah yeah. Ravid Shwartz-Ziv: all over the agents that are running? w wha what is the missing part? Rishabh Agarwal: Yeah, it's it's both it's it's it's a matter of both capital, right? You need a lot of capital to set up these high throughput labs. And also a matter of their bottlenecked by intelligence. They don't have enough grad students. They only have like five to ten. You need like basically each experiment, as I just talked about, needs an intelligence to actually work through it, analyze it, think about it and repeat. So well, maybe the each grad student can have their agents and whatnot. So maybe it's a capital problem, but you can imagine that, right? Ravid Shwartz-Ziv: But it sounds that like correct me if I'm wrong, but like San you you said it like yeah, we are actually trying to to take like a university labs and raise more capital and just scale it up. Rishabh Agarwal: well, I I guess maybe that's the thing, right? Just scaling up is not enough. Because you could run a lot of really bad experiments. I think this is the kind of thing of we have so many AI researchers, but why do breakthroughs in AI only happen every two years? if you think about the big breakthroughs, it's not like we have gazillions of breakthroughs because of all these things. So I guess there is a big intelligence bottleneck. It's it's like once you have enough throughput or enough experiments. Then it all comes down to well, how smartly you run this and how much you learn from this. And maybe the amount of data produced is so much that I guess it is possible that humans may not have enough basically even working memory, sort of. Because it's easy to analyze tens of experiments. Like it's the same for us. Like if I just showed you hundred thousand experiments, what are you gonna do with it? Like, can you even read all the results of these hundred thousand experiments? Probably not. But AI doesn't have that limitation, right? So inequ but the thing is, well, we need some way to learn how we learn from these experiments is still very different from possibly how AI would learn from this, which is let's say through either next second prediction on the data or through reinforcement learning tasks on this data. But we do have a proof of concept that if you think about All the the coding even is a good thing, right? Like we tr we basically train the AI the same way. We train in on lots of coding data, and then we actually give it a lots and lots of these RL environments to basically practice its coding abilities and learn and sharpen. And we can see remarkable things happen if we do that, right? Like I so I think this is like the sort of bet I would say that maybe the company as a whole is making that, what if we apply this? Except rather than math and coding, we go to this harder domain of physical world. Would actually, would it would it have similar so maybe one outcome which I think might be feasible is it might accelerate to like how coding has accelerated a lot of us, right? Now it used to be the case then when you have an idea you have to actually implement it yourself and look through details and whatnot. Now you just all you need is like some really good idea and you can ask your codex or Claude, be like, hey, can you do this for me? So imagine the same thing, except for scientists now, that scientists actually get rid of some of these bottlenecks. So all they need is to come up with some interesting hypotheses. And then AI is the enabler. So so I I think it's like if we can compress 20 years of scientific discovery in two years, that that is a real possibility in my head still. Like it's harder to discover some really crazy things, but it's possible to accelerate existing scientists per se, like how coding has happened. And That I think I can believe 'cause AI if this is really thinking of AI as a tool per se. even if you think forget about superhuman coding, just how current AI models are used. and it's possible it can do things it can complement humans in interesting ways. Ravid Shwartz-Ziv: But but how how you imagine it? now AI get a lot of information and it analyze previous experiments and previous data and and make Rishabh Agarwal: Yeah, yeah. So so I think Ravid Shwartz-Ziv: these connections or like maybe it like it it finds like some hidden correlation in the data and like suggest better experiments. Rishabh Agarwal: Yeah, yeah. So I think it it's like yeah, so the way you can think about this is let's say even if in our own lab, right, we have all all these experiments that are run every day, and this is data that is not anywhere on the internet. Now, if you have an AI, even a frontier model, right? which is exposed to all this data, i it it's gonna be in context. the model will suddenly see all this new information and will in some sense I I guess learn. Or see that, I see. If this is if I do this, this happens, if I do this, this happens. It's it can get a lot, but still the model would actually have access to all this new information that'll change its per se. I guess I won't I don't want to use the word, but it's gonna change its world model of the world in the sense of basically like let's say the agent Put some sort of probabilities or some sort of ideas about, this is what would happen if I do this, or this is the thing that's gonna succeed. But once it sees the experiments we have in our lab, those will definitely change, right? Because it'll see, I guess these guys tried something similar and it didn't work. So maybe that is actually a bad idea. And I should propose something else, simple kind of things. And maybe there are deeper connections too. And the hope is. Basically, so this is the way I think I can imagine this kind of loop that AI looks at all the previous experiments, proposes some sort of hypotheses, maybe filters them down, surfaces only a few, and then the scientists say, okay, let's just go with these hypotheses. Then AI thinks about, what are the ways we can actually make this thing? and again, it could be human plus AI involvement or AI could also itself just propose this thing. And then there are ways to determine is this good enough and safe enough to run, and then you run it. And then there's the analysis problem back, which is once you run the thing, you have to analyze the result of this experiment of what happened. And then AI comes in. So it's like a lot of the work. analysis, for example, can be thought of as grunt work if you think about it, right? It's basically, but there's also a deeper work. there's the the raw analysis of what did actually happen? And then there's the analysis of okay, what should I do next? Or what what does this tell me really about the stuff? which AI can come in. And now I think there's a better one. Well, you can have a frozen frontier model which can learn all of this in context, which is let's say the AI model suddenly sees all the experiments in our lab and it starts producing things like, you should done Can they do this, or you should do this, or this could be promising. Or the classic way we have been training AI, more or less, which is how are we training all the frontier models? They are trained to compress the data and they are trained to achieve some goals. These are the two ways we are training them. We don't really, like we hadn't stopped. Like, it's not like we stopped last year and said, okay, let's just not pre like train these models. These are super smart models, they will just learn everything in context. No, we keep like spending more and more compute. in training and probably more and more data exposure in training of different domains, different modalities, that they are actually getting to the point. I saw this very interesting result that agents ended up solving net hack or something like this today. And I was very interested in this. Allen Roush: Thank you for bringing this up so much. I'm a hardcore gamer. I love NetHack. We talk about it on the episode. And yeah, they're solving like every game now. They also solved Rimworld recently. Rishabh Agarwal: Yeah, and I I think my hunch is it's very plausible the the model with like I think it was Astra, it was not trained on the specific game of NetHack, but it was trained on a lot of these really hard games, right? And it actually learned to generalize a bit that, this is how I kind of do these things, basically. And it's also like basically it's hard to predict when you spend the amount of compute that is being spent and the amount of data that is being spent in these models exactly We don't no one really knows outside these labs what is the distribution of data. And then what are the implications of is this a shallow generalization? Is this a deep generalization? Maybe Frontier Lab folks know. But we can only guess that assume they didn't train on attack, they might have. We also don't know this. But assuming if they did not, it seems like it might have just generalized from the behavior of, I know how to play a lot of these different games. Maybe they trained on Minecraft, because they had some work on it. And then it sort of generalizes a bit here, which is also amazing. Allen Roush: So so so I I I know a little bit about this because I've watched some of those streams. It's also been able to finish Rishabh Agarwal: Mm-hmm. Allen Roush: full real-time games like Fallout 3 and Fallout New Vegas, but specifically games that have pause functionality and the Fallout games are great because you can do the VATs mode to pause for shooting, and so that's what it's doing is VATs for all the combat. In NetHack's case, the thing about NetHack Ravid Shwartz-Ziv: Mm-hmm. Allen Roush: is that NetHack is has one of the best community wikis in the world and probably has since the eighties. So they yeah, Rishabh Agarwal: Yeah, there's a lot of language data. Allen Roush: they call it spoiled. Colors, right? And and so the the agent is constantly, if you look at the thinking traces, it's just looking up the the wiki entry for every little thing, Ravid Shwartz-Ziv: Yeah. Allen Roush: which is the way that humans Rishabh Agarwal: so it's like Allen Roush: have to play it too, if they actually want to get far, right? Without like a crazy good memory. But so I just want to add Rishabh Agarwal: Right. Yeah, so maybe Allen Roush: that net hack is a unique case, and so what where you could start testing this is there are mods for net hack that make it even more random with much more poor documentation, Ravid Shwartz-Ziv: Ha ha. Allen Roush: such as unnethack, slash etc. And if it can ascend Rishabh Agarwal: Even if you think about just NetHag and even learning from the wiki, it seems like it's able to do that better than humans, because maybe humans don't have the Allen Roush: Yes. Rishabh Agarwal: same search capabilities or working memories and whatnot. And maybe this brings to like, well, what makes empirical science that different? It is possibly a similar, like, well, like I see our scientists when they're coming up with something, they look at the literature quite a bit, right? And then they come up with something. It's possible the agent can do a better job at that and then propose something. And then s like maybe there is some sort of generality or similar similarities. Except maybe the bigger question is If I only train my agent on games like NetHack, will it play the game of doing science really well? I think I feel like the game of doing science requires doing some science also. Maybe this is proven wrong, it's unclear. But this is, let's say, you sort of bet off, we should learn from the data in the lab. Because coding became really better when we started learning on coding data. It didn't magically fall out of like, we were playing. Like I think, I don't know if you guys remember when because I did RL back in the day too, and we were playing these video games like Atari and whatnot. We'll basically learn to play like a single Atari game or a few of them, but then it'll not do anything other than that. the scale of this was very small. And it was really a shallow thing trained to just play simple Atari games. If you just put it in a bigger game, if you were to take an Atari agent to play NetHack, it'll just fail. Flat. Right. So so that's the kind of difference here that I'm seeing. And maybe the same bet you can say, well, maybe we we haven't tried this. So it's like I used to think then people were doing all the scaling stuff, it's like, yeah, it seems kinda like dumb, but I wanna see where this goes, because it's possible this dumb approach actually works. And then we have more compute every year. So it seems like because we have more and more compute, we keep scaling, unless I think we hit some practical limit, which will be some physical limit. But until we have that, it seems like If we can figure out how to convert compute into intelligence, is the game we have been playing. And we have been doing maybe not so best possible things, but it works. it does improve as we spend more compute onto something. And then alongside compute, we miss the fact that there is data too, right? And it does improve, especially on things where we have the data distribution close enough to the things we care about. In addition to other things. So I think this is maybe the bet of any sort of vertical AI company these days, I would say. And it makes sense. And now you can ask, well, it's like all the Frontier labs care about certain capabilities and they will really focus on that. But if that capability is not something, let's say you want, then you just have to wait for them to release a better model. And it's unclear if that will actually improve. Like I think, for example, I see when you see these releases of these new frontier models, they have a bunch of benchmarks. Sometimes performance goes down, especially I think if you look at Entropics post for OpenAI's models and vice versa, you'll see these results where the competitors models are not doing that well on some task. But that that's already an indication, right? it's not a Ravid Shwartz-Ziv: Mm. Rishabh Agarwal: net win that thing the thing is improving on everything. It's improving on specific things that I think people there really care about and a lot of other things, but it's not a given that everything would improve generally. Ravid Shwartz-Ziv: Yeah, i it was some like it was quite I don't know, the magic phenomena, right? That's like you improve on something and all the other capabilities also improved, right? Rishabh Agarwal: sorry, can you repeat again? the what is the magic part of it? Ravid Shwartz-Ziv: It's like that that it's it's something that you wouldn't expect, a priori that you improve in I don't know, some domains in a coding, right? And then suddenly it's getting better in also other domains. Rishabh Agarwal: Yeah, yeah, yeah. I think I have some th interest yeah, I think this is also a thing I we observed in our on lab tasks, right? it was a task of basically analyses and then you have to keep going, keep going. But then I realized some of the behavior is really just being persistent. Like rather than spending twenty minutes, you spend an hour and you maybe nail it. And I think this is the behavior I saw like w as models went from generation one to another, like I saw much more persistence, but seems like a general behavior they learned from solving these really long tasks, I would say, that they would just Ravid Shwartz-Ziv: Mm-hmm. Rishabh Agarwal: not give up easily. And I think that transferred to this mix of science plus coding kind of tasks, right? At the end of the day. So maybe that this is the thing. you're s you're actually getting really better at these domain-specific stuff, but at the same time, there are these general behaviors that are basically being developed because those are the ones that that that actually cause this improvement in the first place. So it's like, but at some point, I I guess like it's a good thought experiment. Let's say we just never, I think it's not gonna happen, but let's say we never touch some domain ever. Like we said, we'll never put any kind of data from this domain ever, and we'll keep training on LLMs on everything else. would be it's first of all, it's very hard to decide what is part of domain or not. But let's say if we just do this, would the LLMs actually like improve on this held out domain also? Unclear, right? Because but anyways, I think this is the kind of thing. The reason we no one does this is like, well, why would you do this other than for science? If you think about practicality, Ravid Shwartz-Ziv: Ha ha. Rishabh Agarwal: we want our LMMs to be really good at all sorts of things in some sense. and I I guess you guys mentioned, I think Orion mentioned world model here. it is part of this, right? It's possible. These are really huge models, and they are trained in two ways. One is to predict whatever the tokens in our data set are, so by an ex-token prediction, and the other is to achieve goals through reinforcement learning. And One thing I guess, for example, I think about is let's say one kind of RL task, which is very cool and which I think we think about as well. Can we predict the outcome of the experiment before running it? This is like the superhuman intelligence task. Because and the nice bit here is I don't have to depend on humans to get the reward. Because when I run the experiment, I know the outcome, right? So this is the bit of information. That you can leverage to train. And this is the place where maybe if you have enough of these, you can actually start thinking about superhuman predictions. Or for like I think this is related to the fact that AI is actually pretty good at forecasting. I think I've seen this thing that AI, AI forecaster, there was some championship or something where the best trained AI model beat all the best humans, although I don't know what exactly was the human baseline. So you can imagine now applying this to science. This is like a crazy extrapolation. Which is before I run the experiment, can AI do a good job at predicting how will this experiment succeed or what would happen, sort of thing? And now we are getting into the the regime of world modeling. except you can just do this as a treat, this as yet another RL task, right? You don't have to change anything about the architecture or anything per se. You're just saying I have this intelligent smart model and I'm training this to do this task. given everything that has happened so far, can you predict the experiment which I'm gonna run today? And if it does have this capability, well, we have really actually an amazing model that would be very useful because scientists really want this. Because otherwise they if they knew what would happen, they wouldn't run the experiment. They'll be this, okay, I guess let me run something else. Ravid Shwartz-Ziv: So so like co compare let's n l let's like today, compare to like an human expert, like for the intelligence parts of this pipeline, what do you think like is it better? Like are humans still better than our best models? Allen Roush: And and and and and Rishabh Agarwal: I think it's it's it's a mixed bag, to be honest. it feels like some tasks where, I guess I was saying talking about if you need a lot of persistence, the AI just needs to go in and ham and ham and ham, and basically eventually it might get there, and especially if there is some verifier signal in the loop, if there is some way to check how well it's doing, even if it's not perfect. current AI stuff is very good at using that and keep like hammering on this thing. While if you let's say ask it, hey, Just go to Chat GPT and say, hey, can you give me an amazing material that I should make? And it can do what I think, whatever, but it wouldn't come up with something that interesting per se, right? you have to like bring in scientific taste per se. And this is the part I think it's the same as for any other field too, even for coding. I think taste is still something that it's kind of vague, but it's not like AI is coming up with things to do right now. That'll I think humans are still guiding it, and maybe that'll change for coding. But for science, it seems like it like it's unclear if that would happen. it it's possible we want AI to have taste, right? But how do you develop taste is a a pretty hard problem. And I would say there might be other things okay, if it can predict outcomes of things. Because at the end, what is the learning signal? It's like you do something in the world, something happens. This is really the anchor per se. it it it's basically the main learning signal that humans also learn from, right? Because they update their view of, okay. I I guess this is how the world works. In this world maybe I should do something else per se. So so I think this is the kind of thing and it's it seems possible. Like I I guess I don't know. I have I don't have a good prior because I don't understand as much of like how the science part of this. But it seems like if we have enough data of this kind of like running experiments, seeing the outcome and If we have it's possible it'll only be good at let's say very, very specific domain or specific chemistry of things, right? Like if if you're always running data for like a specific chemistry which has A B C D, then maybe you'll really get good at that A B C D. But even that is very interesting, right? Like if you just run A B C D and then if you became good at A B C D E, like that seems like already a pretty good generalization test that can it actually learn more than what it was just trained on? And it's possible, because cause the underlying principles are more or less the same. I would say. And then of course there would be things like, well, not everything, like experiments you can run, but not everything can be run. And maybe you also need simulations here, right? And may maybe the best part I like about this is well, can we have simulations which can be learnt from experiments? Like because you run an experiment, it actually tells you what really happens, and simulation is some sort of approximation this. So if we can have our models access to these simulations, but they can actually improve over them because they can do some soft reasoning of, I think this simulation is wrong here. This is really what happens. So I should really take this with a grain of salt and continue. So so so I think it's yeah, but but I would say all I think about this is more or less we know how AI currently has progressed in some of the domains, and we know what is the recipe they have used. Now, question is if we just Blindly apply this recipe per se. The details would matter. But if we really are taking a dumb bet, like you can call this a really dumb bet, which is like you're saying, we know how AI really improved on math, and we know how AI really improved on coding. Can I actually do the same recipe per se of compute data and like keep doing what they are doing on some other scientific domain or field? how far could I go? I think is a good question to ask, right? And my hunch is once pe once let's say some of the simpler things are like solved or we have made enough progress, people will move on to harder problems. So like one reason, like one funny thing is when I was deciding whether to join this company or not, a lot of the things were people were saying, we will first develop superintelligence, and once we have it, then we will do this thing. Because it seems too hard. And that didn't make sense to me, per se, because it seems like, well, you can also do this thing Before you have superintelligence. Because it's possible the superintelligence may not be super in this domain, per se. So I would say it's like different takes on this kind of thing. that's why when you think about AI for science, there are two flavors of companies you can get these days. One is, we will automate AI research, and that AI researcher will automate all of science per se. Or you get the stake of, well, doing AI for science needs doing AI for science. So you just can't like keep doing AI for AI and hope that this AI will be really good for everything scientific. I think both are reasonable bets, per se. And it's only time that will tell what would happen. But I I feel like given I've been like an RL researcher, it feels like learning from outcomes or learning from reality has a better, like I feel better about this. Then Allen Roush: So what's w so what's the role for humans going forward then? Are we just diversity machines? Is that that it for the systems? And and are we even capable of that? do you think that these systems are super creative already too? Rishabh Agarwal: Actually, this is a very loaded question. Yeah, this is interesting. I guess if I think about how we use AI today, right? It seems like I personally still feel it's a very good enabler. I could do things but I wouldn't imagine earlier, right? But it still feels like I'm the one kind of driving it per se. I could have hundreds of agents running, but still me running this and saying, I want to do this. So this intent goal intention of I want to do this is something coming from me. It's possible AI can also come up with this, but then it's a question of how good this would be compared to humans. In the the world I have Or I think about, I feel like humans will still be there. Unclear about the exact rule, but it it feels like humans and AI might really be complementary, per se. Because the way we learn versus AI learn seems very different. Like, okay, just just like I do RL basically, have been doing RL so long, but I still feel RL is super dumb as a paradigm, if you think about it. It's like the dumbest paradigm anyone could have come up with. Ex it's very general, but the dumb part is. We do like these, I don't know, 10 hour rollouts. And then we say, hey, good. So it's like that's the kind of signal. And this is how things are really done, right? Like this is the really the learning signal. Like you wanna say more than just that. but it seems like the paradigm we are in, this is the signal we give, more or less. And then people will go into detail. So but really all I wanna say is, I think you did this part well and this part not well. I can't say that. 'Cause this will this this isn't there's not not any paradigm which actually basically lets me do this kind of thing. Well, you I at the end I have to still convert it to numbers because we are all doing gradient descent at the end. And i in a sense, I have to convert it to a scalar or you more more most of the time. So so it feels like the learning mechanisms of like we are having this conversation. You are learning a lot. I am learning a lot. We know what to learn from per s per se. you're kinda ignoring some of my statements. You're listening to some of the other and you're saying okay, I get this makes sense. Maybe this is useful. we don't have some of these mechanisms, because at the end RL was designed in the service of maximizing a reward, right? And then someone has to come up with the reward. So you convert everything to some sort of concise numerical thing that we We can learn from. And it's like, it's been really good. Like it has done so much progress has happened because of this. Ravid Shwartz-Ziv: Why? Why you think wh what is the reason it's so good? Rishabh Agarwal: I think there's a lot of basically even though the signal is sparse, maybe if you think about it, if you read these 10 hour rollouts, a lot of the time maybe there are five or ten really important decisions that matter per se. And I think I've seen these enough results that if we really just supervise these five or ten tokens, that's enough. So even though you're producing millions of tokens, there are these counterintuitive results in the literature which say, just supervising these 10 tokens is enough to actually do better, per se. So it's possible that all we are doing is basically finessing, like there's a lot of possible things the agent could do. But if we are just precise about, this is the one thing that it should do really well, and rest it, we are like, okay, whatever, it's fine. Either way, it works. It's that's one mechanism. The other is there's a lot of learning. if you do, let's say, for example, looking at this, let's say the agent calls a tool, it makes an error, you get something else. The agent sees that, it's all in context, right? And it'll do something better already. So there is learning within this 10 hour thingy. Except the way we are supervising is we just say, yeah, we ended up actually succeeding, so everything is good. So we can see some dumbness here, but overall it's possible that you end up doing good actions more than bad actions, and then you just end up like it's like a inefficient paradigm, but it works. It's inefficient in the sense that. It's like if I need to solve a 10 hour task, maybe I need to train on that 10 hour task. and i this this basically lo puts a like a lot of compute barrier, right? Like it's not easy to train a frontier model to which can compete with let's say open air anthropic on everything that they are caring about. It's possible you can compete on something else that they don't care as much about, but the amount of compute and data they're throwing at this, you are on an uneven ground. think about maybe an academic researcher with I don't know, 10 GPUs. Can you really compete on a frontier model training with 100,000 GPUs? Probably not, but you can compete on other kinds of things in some sense. So I would say it's like the inefficiency here of RL, in a sense, is an advantage to the current incumbents. Because because it's so inefficient and because it needs a lot of compute, it's actually it's harder to compete on the same space, but then you can go to a vertical space. I I think you've seen all these companies which go to a domain and say, we really care about this domain per se. And the hope is if we spend all the compute we have on this domain, maybe it ends up doing better than something which spends compute on all sorts of things. And this is just one of those N things they spend compute on. But but I do think the field is very ripe. there's a lot of potential innovations that can happen. Because the simple reason is this. I think RL as a paradigm was not developed for language models. LLMs did not exist really back then. It was developed as a general-purpose way to optimize any sort of reward. And this could be anything, right? Like it doesn't really say where does the reward come from. It could be any sort of scalar number. And then we have methods of optimizing this really well. But language is very different per se, because it adds a lot of richness. which and there are de definitely ideas in RL which touched on this, but I I think we will see. Like hopefully it probably some poor grad student working on a lab on 10 GPUs trying to figure out this thing and maybe they release something and this will get picked up. Ravid Shwartz-Ziv: So but for your problem if you have now I don't know, two hundred k GPUs, will you do something like fundamental different? You will I don't know pre-train the models, train from scratch, or you think the models like our current models are a good start? Rishabh Agarwal: Yeah, yeah, so I think it's the the compute game is well, if you have the compute, you'll think about where the data for training is. But what you'll do is you'll spurn all this compute on the specialized domain. Like for example, I keep telling people that, for the models we need for doing science, do we really need to know who Tom Cruise is? Probably not, right? But the all the pre-trained models do care about that, because they are trained on all sorts of information. So there is this hypothesis that maybe there is, and but then there's also the well, you need to understand English, right? You can't break that part. So but there is this hunch that if you can spend compute on the things we really care about, it is plausible that you end up better on this domain. You might be worse on all the other things, possibly. So and the other thing is it gives you like maybe, like I guess just as an example, I think we had this result where we said, we only used 1300. GPUs of previous generation, and we can end up doing better than Astra on a task that we care about because we had data from our lab for that. But It seemed like if we could just spend more compute on this, right? rather than 1300, if we have 13,000, let's say, then we can probably take on more data and more tasks. Or at least with the given amount of data, we can spend a lot more compute on the same data. And the expectation is that this shouldn't improve, because we do see the same RL scaling, right? That you see in a typical domain like math and coding. That is, you spend more compute. There's of course this thing I'm not talking about, which is how do you train on all this data? Like what kind of tasks are being created and what Not, but imagine like if that is let's say being done, because that's the other nasty part of this. Because science data is not as simple as your typical math problem or coding problem. I think there's a lot of taste involved here, and there's a lot of nuance here of what you can actually reliably train on. Like verifiable, I think someone said this thing to me and it's still trying to figure out exactly what they're meaning. But they said science is falsifiable, not verifiable. And the Allen Roush: yeah, Rishabh Agarwal: And I was like, hmm, Allen Roush: that's Rishabh Agarwal: how does this actually affect RL? Which is true. if I have a hypothesis, I can maybe tell is this a good one or a bad one, or is this does this make sense or agree with data? But it's really hard to give true verifiability to things, because we don't know unless we find it. So it's like the kind of thing of vacuum, that once you hit it, that you have hit it. But if you don't hit it, all is you don't hit it. And that's it. there's nothing really more than that I can say. So there's no ground truth answer most of the time. so I think it's it's it's challenging in its own way. And I and it's unclear how far this would go, but I would say let's bet on the things which has worked in the past on this kind of domain is the vibe. And there might be like more pressure to do different things, right? Because maybe this is the other thing I think people say that once you have constraints that actually do help you to figure out something maybe better, because this is maybe more critical for you than for someone else who don't have these constraints. Let's say if someone has unlimited compute, they would definitely think about things much more differently than if you have a finite amount of compute. And now it it's the game of well, which one would scale better? Because you don't want to have something that really works on your like I think this is coming back to the bitter lesson thing that we are putting in a lot of knowledge, but we still want systems that improve with more compute. And I suppose Th this is the thing that if you have something much more efficient, if your slope of improvement is much higher than existing things, that's great, right? Because you can convert the same compute to something much better. And this is where I think the name of the game is. Probably true also at Frontier Labs, but less so critical because of the abundance of compute there. except maybe they will face it too if they try to do one-month-long tasks. Ravid Shwartz-Ziv: Okay, I think we are out of time. anything else that you want to add? the record Rishabh Agarwal: I would say maybe the most yeah, I think maybe more people should do harder things. Cause it it's like it's definitely much more challenging day to day than being at just at a like a AI lab, right? 'Cause I don't need to have to worry about like all those sort of things understanding this kind of physics and whatnot and dealing with like all the messiness of it. But I I I guess this is Maybe I would say this is a bit different than other things. everyone's trying to pursue coding agents, everyone's trying to pursue RSI, everyone's trying to pursue the same thing. I I think it helps the same way it helps in science to have diversity. It's the same thing that maybe it will help if more people in AI branch out to more difficult things that seem really hard, but not impossible per se. And that will both benefit them and maybe the field overall. It it is very hard though, and it is annoying. That it's it's like if you it looks like we can be doing working on cool AI research, trying to make all that stuff, versus going to this. But but I think maybe working on a very hard domain tells you all the kind of problems that you will face that will require a different kind of approach or research is my expectation. or maybe we realize it's too hard and we go back to AI research. So again, it's like maybe that's the thing I'm gonna say to people. Ravid Shwartz-Ziv: Yeah, maybe we'll come to our podcast in a year and maybe you say that it's too hard. Who knows? Yeah. Rishabh Agarwal: É, é, bem, meu. O nós. Allen Roush: Ha ha ha. Ravid Shwartz-Ziv: Thank you so much for joining us. It was a real pleasure to have you. Rishabh Agarwal: Yeah. Allen Roush: Yeah. Rishabh Agarwal: Yeah. Nice chat. Cool. Allen Roush: Yeah, great pleasure. Ravid Shwartz-Ziv: Thank you so much. Rishabh Agarwal: Very fun. Bye guys.