Latest / The Information Bottleneck / RL Was Broken at Every Level - With Joseph Suarez (PufferAI)

Transcript

RL Was Broken at Every Level - With Joseph Suarez (PufferAI) · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the information bottleneck. And today we are going to talk about RL. We are going to talk about different types of RL, both like LLM style that we all know, and also like the more classical one, like the real RL. And for that, we have ⁓ Joseph. He is ⁓ the founder of ⁓ PufferAI ⁓ it's a lab that focused on RL, exactly these types of of ⁓ settings. Hey Joseph, thanks for coming. Joseph: Hello. Thanks for the invitation. Ravid Shwartz-Ziv: So let's start from the beginning. When you say RL, what do you actually mean? Joseph: I mean simply learning through interaction. I think that's probably the cleanest, simplest everyone would understand it definition. ⁓ I like to annoy people by saying, ⁓ yeah, offline RL isn't RL and other things like this. But ⁓ I think that they the cleanest definition is just interaction, learning through interaction. Ravid Shwartz-Ziv: So tell us like w what are the the seatings that the settings that you actually ⁓ think about when you learn from interaction? Do you think about games? ⁓ do you think about like robots? All of them. Allen Roush: Mm. Joseph: You Yeah, ⁓ either of them, right? Games, robots, simulators, definitely simulators. ⁓ we like games in our own research just because they're very interpretable. ⁓ kinda get this misconception that RL only works on games. ⁓ we really just like games because it's easy to see when something goes wrong when we're tuning our methods. The RL really doesn't care if it's controlling a robot or if it's playing a game or many other different types of tasks. ⁓ Yeah, I like I usually frame things in terms of games because it's a very clean way to think about a lot of problems. I do sometimes think of control problems separately. but it goes a lot it's a lot broader than that. We do stuff in finance, we do stuff in commerce, we did one thing in animation, ⁓ and of course there's stuff in games and ⁓ control problems as well. Allen Roush: So so I have a question. ⁓ the last time I seriously worked in reinforcement learning was closer to like 2020. and a lot of what I was exploring at the time was the open AI gym. ⁓ what would I I don't know if they still maintain that at all, right? But they had like really toy kind of game problems. ⁓ and what I remember at the time is that deep reinforcement learning was really bad because there were a lot of problems where ⁓ you would be like losing reward as you're trying to make a Joseph: Mm-hmm. Mm-hmm. Allen Roush: cart go up and down like a like a rink basically and then it gets a reward at the end after it's spent a very large amount of reward by ⁓ you know hitting a flag and this is sort of like not a not a useful or not useful like a like it's an adversarial problem compared to gradient based methods where the gradient kind of tries to push away from this thing that's you know leaking so much reward. ⁓ do you have opinions on this idea of like gr you know gradient based methods and its influence on reinforcement learning. Joseph: Yeah, it works and most of those problems that didn't work back then or at least were very fiddly. ⁓ those did work, they were just fiddly. ⁓ those are pretty much all just solved out of the box in seconds now. And on just one GPU with the same tiny networks we were using before, it was a small set of algorithmic changes and a small set of problem formum ch forming changes, but then mostly the code was just running a thousand times too slow, so things were unstable. Allen Roush: Hmm. Okay. Ravid Shwartz-Ziv: So what is maybe like because of our audience like mostly focused on LMs, maybe you can tell a bit like everyone knows RL in the LM sense, right? That like you you're doing like kind of like post training and you you can do like out at GFF or like PPO or like like I don't know, GPRO, like whatever, right? how it's different from RL in in your setting? Joseph: So I would say that the LLM setting is using a very weak form of RL by necessity because the problem really isn't well suited. ⁓ you can't really train value functions well. You have this really long sequence of tokens that you know is the token the action, is the whole sequence the action. You can't really do proper long form credit assignment, at least not with the RL itself. And you can scale it massively. And you can sort of make it work from that. And it's kind of funny how well that that does work. But ⁓ we have much more powerful methods in the setting that I'm used to. And also we have more data. That's one of the funny things is we have more data on one GPU than like OpenAI has with all of their GPUs. ⁓ you know, we can get up to petabytes of training data ⁓ just from SIM. So very, very high throughput ⁓ learning, much, much smaller models. RL does tend to favor smaller models. There's some empirical work on that as well. And ⁓ stronger algorithms. Not super fancy algorithms, but just like even being able to do PPO correctly makes a big difference. Allen Roush: What what do you think is the the key source then for ⁓ like robotics to improve? Because I've heard people talk about world models as synthetic data generators, which presumably would would hopefully, you know, make the the RL loop a little bit more obvious for, you know, maybe potentially like JEPA style architectures. Do you do you think that stuff like this is viable or do you really think it needs to maintain this paradigm you've articulated of very small models, ⁓ very, very strong rewards? Yeah. Joseph: It depends how you use them. the in the loop train the world model train the agent mostly doesn't work or at least is very inefficient. ⁓ a lot of this seems to me just to be band-aid patch over not having fast simulators and not wanting to do engineering to make your simulators decent. ⁓ I think world models themselves are likely to be very independently useful, though I am very suspicious of the like we're gonna generate infinite environments. From world models somehow. If it has a hope of working anywhere, I suppose it would be in robotics. it's a little bit obnoxious because you kind of lock yourself in to ⁓ a very expensive, scale-heavy approach when probably there are much more efficient ways to do things if you just have good fast SIMs. and you know, most places I've looked, the SIMs can just be made much faster than they are, and people kind of have not looked at them the right way. Robotics is a little bit different, but not too too far off. ⁓ I have a couple people I know that work in that space. So I I think that overall world models are definitely not that it's not even something we're considering, let me put it that way, for our applications. It's in the types of areas that we work in, it's something that would just would never make sense. ⁓ maybe for areas of robotics it makes sense, maybe for areas of like problems where you have agent interacting in real world, but ⁓ agent interacting in real world Allen Roush: Yeah. Joseph: That's not even the whole problem definition of RL, right? There are a lot of problems that are independently important that are entirely in sim where you can't just train a big world model. You need to have like coverage of the data. ⁓ yeah, I think that that pretty much sums it up. Allen Roush: Yeah. Ravid Shwartz-Ziv: So can you elaborate a bit about the problems that you actually want to solve? Like what are the biggest problems that you think like are the most important ones? Joseph: Yeah, I Yeah, so I mean there are few different phases to this. ⁓ I guess the first arc of puffer and what we're doing at the moment is just making all the normal RL methods work very, very well and stably. That is close to completed. I'd say we're within the next year or so. ⁓ we already have nice results. We're pretty much able to solve new hard problems within a few weeks worth of dev effort, experienced dev effort, ⁓ that would have taken months or a year before. ⁓ Ravid Shwartz-Ziv: What types of problems? Joseph: We use games mostly for research. We also have control problems. We have a fun demo with ⁓ with a drone. We've done stuff professionally in finance, commerce, and animation. We really don't have like one specific area of RL. The commonality is just problems where either there is a fast SIM or we can build a fast SIM. We build a lot of SIMs. ⁓ so we can have very high throughput data. And typically we look for problems where They have this flavor of being fiddly optimization problems where it's not just like a ⁓ common sense do this, right? Where it would be difficult to write down a solution, or you couldn't just take a language model or a vision language model and just have it instantly solve the problem. I mean, even something as ⁓ you know, one of the test environments we use, even something as simple as Sokobon, ⁓ well, not simple to solve, but simple to write down, like good luck getting the language model to do that, Sokobon. Ravid Shwartz-Ziv: Wait, what is it? Allen Roush: Yeah, I'm I'm a I'm a big fan as a as a net hack player, I instantly know. And I'm Ravid Shwartz-Ziv: I I'm not sure. I don't know what is it. Joseph: I was playing NetHack last night and we have a project on NetHack right now. Allen Roush: Yeah, yeah, I I'll ask about that in a second, but well explain it please, Sokaban. Joseph: Sokoban is a little puzzle game where you have to push boulders around in order to put different or boulders or boxes onto specific tiles. And ⁓ you can mess up the puzzle very easily such that it's unrecoverable by pushing a box to where you can't push it anymore up against a wall. And ⁓ you know, there's some usually really relatively short way to solve the puzzle, but they're very difficult. ⁓ they're difficult for people. Ravid Shwartz-Ziv: Please explain me what is it? Joseph: So I like this as a research problem. ⁓ and one of the reasons I like problems like this, and one of the cool things I can say having worked on the number of problems I have, it used to be that you'd have all these RL papers where you would have it working on one task and then the same method wouldn't work anywhere else. Pretty much if we make progress on Sokobon or even on mazes, that's another task, or twenty forty-eight. ⁓ the solution just transfers to everything else in Puffer and to our professional work as well. So that's been one of the ⁓ the cool findings. A lot of the tasks that we used back in the day, they actually were good tasks, just the setup wasn't right to get the results to transfer. Allen Roush: And and I I'll I'll just point out Sokobon was originally like a Japanese game, I think, in the like eight eighties and then NetHack, ⁓ the reason we bring up Net Hack is that there's so NetHack is this crazy large, ⁓ roguelike, ⁓ that has been in continuous development for like thirty plus years now, and one of its many dungeon branches that you actually really should complete quite like pretty much when you see it, because there's actually some really good loot at the very bottom. I think a a Lux Stone or ⁓ a wand of like like life s or sorry, ⁓ amulet of life saving as I recall, something like that, one or the other, maybe both. Might also be a bag of holding. Anyway, these are important items you need to like eventually like win the game or ascend, which is I I still haven't ever even done it, right? After you know playing forever. and And so you need to complete like five or six different levels. ⁓ I I guess my question then for for you is, you know, net hack is actually one of my favorite like games ever. I play it a lot. ⁓ I'm fascinated by people that study it. I've I've also seen it seems that a lot of people who do RL on it have to do very simplified variants 'cause the game's too hard. And like do you like wha what do you think we need to like make an agent like ascend in NetHack, like correctly without cheating or some other stuff? Joseph: So ⁓ we put our intern on this problem last week. We gave him a a large problem here. ⁓ yeah, we there's some good results we'll have in ⁓ in a few days. We're just kind of making sure stuff works the way we think it is. But I to solve that full game, it's not quite fair because there is some external information you need. ⁓ you basically you need to be able to re well, external information in the sense that you need to be able to read. Allen Roush: Ha ha ha ha. Yes you did. Joseph: Yeah, if you can't read, you're gonna have a hard time of knowing what you actually need to do. So I there are a couple spots in the game where you'd I think that here, if you gave us a small budget of like teleport from here to the next place that you obviously need to be that you can't read, or like somehow scaffold a couple things, I think it's possible that we could beat the whole thing. ⁓ I think without that, you're not gonna beat the whole thing with our methods, and that's kind of fine. ⁓ you know, one very similar thing, much easier but similar. Allen Roush: ⁓ yeah. Joseph: Pokemon also has Sokobon in it. ⁓ you know, a collab with Puffer beat all of Pokemon Red with RL before the LLMs did. Which was ⁓ yeah, which was that was a big one. It took a lot of scaffolding and engineering. ⁓ that's Dr. Rubinstein on Twitter, ⁓ X now did that. and yeah, Sokobon. There's a lot of scaffolding and such around Sokobon. I think now for Allen Roush: I didn't know that. Wow, that's a that's a big deal too. Joseph: Problems of that difficulty, like we could just train an agent on the Sokobon and it would just solve it in a few seconds. Those are not particularly hard Sokobon levels. Doing it in the context of a larger ⁓ a larger problem is more difficult. And then the main problem there was likewise the same thing with NetHack of having to read the instructions of okay, now you have to go back to this other part of the game. And if you don't know to do that, then there's nothing you can really do. So there's a lot of scaffolding for stuff like that. ⁓ I do think though that ⁓ you know, ⁓ this work was criticized for having a lot of engine it on that side. I think if Pokemon just ran at the speeds that our other ends do, it wouldn't eliminate all the engines, but it would eliminate at least two thirds of it. ⁓ 'cause Pokemon was running at least a hundred times slower than the other problems that we work on. Ravid Shwartz-Ziv: So w what what no, like ⁓ so what what types of like problems like you think are important? Like like okay, so let's assume that now we can have yeah. Allen Roush: Do you remember? Joseph: Yep. I yeah, I didn't answer that one. So longer term longer term, my real goal for RL is scientific simulation. Material science, molecular sim, biological sim. I have no knowledge or experience in these areas, but I don't think I particularly need to in order to think that it will be useful, because lots of areas of science use sim. They use sim in a very primitive way where they kind of just guess initial conditions based on their own experience, press the button and see if it works. And I would like to instead be able to specify outcome you want and then optimize towards that outcome efficiently, not with a super expensive SIM, but with a fast cheap SIM, then with ⁓ you know, checking the results with a slightly higher Fidelity SIM, and then checking the results of that with a really high Fidelity SIM. Something of that flavor, but in order for that to work at all, you need RL to be stable and consistent because you need to be able to optimize towards hard ⁓ objectives and fiddly problems. So the sort of phase one of puffers. Get RL stable enough that we can even play ball, and then look for problems that we can solve in ⁓ various areas of science. I'm just generally looking to where we can accelerate scientific research through this. I think it would be awesome if we found some useful problems in biological SIM. That's kind of my most ⁓ personally my highest interest area. That one will be hard. I think it's more likely that there will be a couple problems in material science. And ⁓ if not, we'll look from there. But there definitely will be problems that you can solve with this text. Ravid Shwartz-Ziv: Yeah, we we we talked like several weeks ago with ⁓ Daphne ⁓ Kohler and she has a drug ⁓ discovery company and she actually like she she she claimed that like the the main bottleneck and the main problem is like to actually do the experiments and like to get the results and this you basically can't ⁓ like speed up because you don't have a good enough ⁓ simulator. ⁓ okay so Joseph: Mm-hmm. We build a lot of s we build a lot of simulators and we specifically we build a lot of simulators in a way that they make sense for RL as well. Ravid Shwartz-Ziv: What does it mean? What does it mean to to have a simulator that makes sense for our L? Like what are the properties for a s good simulator? Joseph: Simulator has to be really fast. ⁓ you ideally want to be cutting as many corners as you can without compromising too much the result ⁓ being valid. Sometimes you can cut enough corners that it's not quite there, it's there most of the time, and you can make a slower sim to verify. ⁓ you want to have the observation and action data structured in a certain way. ⁓ There are, you know, a few different patterns that are common for different types of ends for how you format your obs and action data. You try to compress that data as much as possible. You don't want the observations to be huge. You kind of want to have it be the smallest amount of information you can. That is more so for infrastructure, not for learning difficulty. Just it's a lot of data. ⁓ I mentioned that we go through petabytes in SIM. That is that's actually true. You know, we go through petabytes of data ⁓ for the larger problems. So bandwidth matters. ⁓ then also at the same time you consider whether the sim is better on GPU or CPU. Most of our SIMs, almost all of them at the moment are CPU sims. It turns out that we have made CPU sims that are a hundred X faster than comparable GPU sims. ⁓ just because for the most part people were throwing GPU on problems that don't need to have a GPU sim. There are some problems that do have need GPU sims. ⁓ molecular SIM will be one of them most likely. aspects of robotics are another that's overdone there as well. So you need to consider that trade off appropriately. And yeah, it's pretty much just I think that the main thing if you want to just take one big thing is that your perf target is orders of magnitude higher than it is in any other application. Like many orders of magnitude higher. Ravid Shwartz-Ziv: But do you think like at the end, if you have a good simulator, like if you if you can build a good simulator, you it's kind of like solve the problem. Joseph: ⁓ the problem is not solved if you have a good sim, not at all. No no no no. You still need to know how to get from point A to point B in the simulator. That is still very hard. Ravid Shwartz-Ziv: But but do you think like it's harder than like to get a good simulator? Joseph: I hand you a simulator of Go is Go solved. Go is solved maybe if you ⁓ you know, give it a a top world class deep mine team and a few years and a whole bunch of compute, but you don't just have it solved because you have the simulator. Ravid Shwartz-Ziv: Okay. ⁓ fair enough. Joseph: Net hack. You have a simulator for net hack, right? Not solved, not remotely solved. And we can actually get this one to be pretty fast as well. Allen Roush: Yeah. ⁓ What about ⁓ I was gonna ask you because I I love games and and the idea of reinforcement learning applied to them, and I've even spent some time with AI in some of these games, but usually not RL. So one thing I was intimately familiar with was chess engines. and I guess I'm curious about two questions on that. One is what's your take on the idea of like the old school Monte Carlo tree search, alpha beta pruning, opening book and game table based, like how things were going before ⁓ a like RLHF ⁓ you know ki or ar or reinforcement learning in general just kind of ⁓ intervened there. And then what's your opinion on games that are like a bit different from ⁓ chess, for example Stratego where it's ⁓ partial information instead of full information? ⁓ do you think it's like more or less amicable to RL based techniques? Joseph: So a few questions there, right? ⁓ so for chess, we do have a project on chess as well at the moment. ⁓ though we're kind of gimping ourselves because we're trying to do it completely without search. ⁓ the yeah, no search at all. Yeah, the best agent is like twelve hundred elos so far, which is better than me. I'm terrible at chess. I've but you know it's Allen Roush: ⁓ Ravid Shwartz-Ziv: Me there. Joseph: Yeah, we need to it that definitely I honestly think that we can get a lot farther than that. And a lot of that is just the observation representations and stuff being a little bit tricky, ⁓ with the way chess works, as well as the action space is really obnoxious for ⁓ for chess. So I hoping we can get it to be higher than that. It if you have search available and you're trying to solve one specific problem, it makes sense to use search. I wouldn't say just use search, you know, use it in combination with RL. Is the whole deep mind line of work on that. There was re very recently a paper on Stratego. ⁓ I don't believe I don't think use search. I could look it up. But there was a very good recent paper on Stratego. ⁓ I will say that our for our purposes, we kind of don't care whether a game is full or partial information. Like that doesn't matter to us. ⁓ it matters to us in the same sense of like, does the partial information make the game or the task harder for a human to play? If so, then yes, but it's just general complexity. It's not specific to it being partial information. Yeah, we don't really depend on that in any way. Ravid Shwartz-Ziv: So okay, so like you have a good simulator and you have good ⁓ problems. And now like w what are broken in like current RL? Like you know, people are doing like RL for many years, right? And they tried in different settings and it wasn't really useful, right? like it wa like not a lot of people like actually used it, right? We can agree on that. Joseph: Well. There were definitely attempts professionally in a variety of companies for sure on different problems, more than you would expect, but I would say that there were a lot of failures, yeah. ⁓ Ravid Shwartz-Ziv: So why? Why w what is it what are the problems with current algorithms and and how you want to solve it? How it Joseph: See, that's the problem right there is assuming that it's the problem with the algorithm. That that's that's the issue right there, is that everybody assumed it was a problem with the algorithm. ⁓ the problem was catastrophic engineering failure. Like just catastrophic engineering failure. Every single portion of the stack was written in the worst possible way. Everything was over a thousand times slower than it should have been. That's not an exaggeration. Ravid Shwartz-Ziv: So what is the problem? Can you give us examples? Joseph: Everything from the M's themselves to the god awful Python wrappers on the M's to the vectorization and multithreading, copying data redundantly five times and not even accelerating stuff over single threaded, to the learners having a huge amount of overhead, to using tiny tiny batches of data, like to using a very few parallel M's. I mean every single part of the stack was broken. There was not a single part of the RL stack that was done correctly. It it was that simple. ⁓ it turns out that Yeah, we have better algorithms now with Puffer. ⁓ and that was a good chunk of s you know, what we've done, I would say. But the I'd say the biggest thing we've done is just made everything really fast. you know, for comparison, we will re implement like some of the classic arcade games from state, not pixels, but same game. And ⁓ it just the learning speed or the like the throughput you get through the same hardware is sometimes ten thousand X faster. ⁓ You literally solve these problems in seconds. ⁓ and that was even before no before we got new algorithms like ⁓ additions, it was maybe a thousand decks and we got another order of magnitude off of algorithm improvements. But it really was just engineering failure at every level. Allen Roush: Yeah. Th this is fascinating because, you know, normally I I get to I I'm a big fan of a specific type of retro gaming called ⁓ vector computing. Vector gaming, so the Vectrex is a relevant ⁓ old game console. And the problem is that they actually decouple the rendering from the processing because the rendering is being done with vectors and so it's based on your ⁓ your screen basically the the laser beam ⁓ and and and so, you know, they were able to get potential Potentially a thousand frames per second equivalents back in the eighties, but it depended on how much was going on in the screen. So you can't really like accelerate it. So I guess d do you think that, for example, there there are a few games, I'm sure you can emulate them, right? But if you I guess wanted to do it authentically, games that are ⁓ vector based that are kind of famous, for example, are like Battle Zone, Tempest Tempest. I don't well not Tempest, sorry. maybe Tempest, but ⁓ there's there's a few others. ⁓ the the the the old Star Wars one where you're doing like the trench run and everything from the eighties. So I'm just curious, are these any of the kind of games you guys are looking at? ⁓ or is it like even simpler stuff? Joseph: Mm. Mm-hmm. No, I mean we have complex tasks, but typically if something is slow and can't be made faster, we just rewrite it. Like we just rewrite the whole thing. ⁓ and that works more often than you would expect. Like it turns out you can just rewrite a lot of things. ⁓ we've rewritten some of the the main research M's first of all, because those have some parody with ⁓ the literature. We have some of the simpler t like the simpler games, ⁓ fully rewritten like What is it? Breakout and Duro. And Duro end up being more complex than you would expect. Pong. But they these are very, very simple tasks. I mean we use these for quick like three-second tasks. Breakout I think is solved in four seconds at the moment. for our more complex tasks ⁓ tasks, we have well actually there's some that aren't even games that scale really well. Like twenty-four ⁓ mazes scale really well, twenty forty-eight. Twenty forty-eight is a game. That one is very easy to implement. That scales very well as a research task for stuff that's like harder to implement in sim. We have one guy who has a ⁓ sixty or eighty, probably eighty now, eighty thousand line port of ⁓ some RuneScape private server code. And ⁓ yeah, we're solving some crazy tasks in rune yeah, yeah. So we have there's some results on the Inferno already that are just nuts, better than top half percent of players. Allen Roush: Yes. And you're talking OSRS, right? ⁓ thank goodness. Okay, yeah. So yeah, for those who don't know Yeah. So I I'm like pissed off because I didn't realize Jagex was still like enforcing anti-bot stuff. I I've botted on that game for like a year and a half on a member account and didn't get banned for so long. I was like level a hundred. I was so I I never got that high level. Joseph: Yeah. ⁓ yeah. That's that is my personal favorite game. ⁓ yeah. Don't break my favorite game, man. Allen Roush: Yeah. Well no, okay, I'm not selling stuff. I just don't have time to play that game the way I used to. What I'm literally only like crafting like anyway, I'm sorry, but okay, like Joseph: Yeah. I will have you know that I sat there for twenty hours and did my crafting by hand, thank you very much. Allen Roush: Yeah, me too as a kid, right? But anyway, okay okay, more serious question then. Joseph: Mm, this was like a this was a few months ago, man. This was after I was burnt out from releasing Puffer Four, okay? Yeah Yeah, I'm like a probably top one or two percent player. It's my favorite game. It's actually that that game has gotten given me more research ideas than any textbook or anything. Allen Roush: You're still finding time to play that game? ⁓ my ⁓ Wow. Wow. Okay. Okay. Okay, okay. Here's the question related to this. ⁓ Ravid Shwartz-Ziv: Like what? L like what? What r what type of research ideas? Joseph: The tick system of that game. Okay, sure. How did devs in the nineties get cities of players to run on the hardware that they had in the nineties? How did they produce like an interesting, engaging game on like nineties hardware? Ravid Shwartz-Ziv: Faul. Joseph: Well, they played to their limitations. They came up with a tick based system where the server has to update every point six seconds, not sixty times a second, and then they smoothed animations client side. They built mechanics that actually make sense around that system. They minimized the amount of state update that is required as a result of this. Like you can just build a sim in that way, and lo and behold, you'll b have it running at millions of steps per second if you implement it efficiently on modern hardware. Like that was one of the most important things. ⁓ in a lot of my research and my PhD was building around that exact system. And we still use that now all over the place with Puffer. There's a lot you can learn from ⁓ from old game dev 'cause you know people used to knew how people used to know how to make things fast 'cause they had to be. Allen Roush: So so I I I wanna go back to the RuneScape example for a moment because I think it's ⁓ related to other games, ⁓ this problem I have. Games are really based in time. ⁓ like for example, Castle Wars. I'm pretty sure nobody has played Castle Wars seriously in old school Runescape since like twenty thirteen. If Okay but like it's it's rare. You you know that's not like a common thing anymore, and you have to go on like some Discord or something like a private group when it used to be you know popping when I was a little kid. Joseph: There are a few. Ravid Shwartz-Ziv: Ha ha ha Joseph: Yeah. Allen Roush: How do you simulate old game like multiplayer games especially where it relied on large amounts of people opting into optional content? I worry about this like dead lost content not being like normally seen as accessible. Joseph: Well, if you wanna if you want to use it as a training task, you go implement it and then you train some agents on it from scratch. If you have some human data you can use it, but if you don't, whatever, you know, you just train agents on it from scratch. Allen Roush: So that means So so so you basically in a lot of cases have to write kind of add ons to the game, basically, like AI in places where it didn't exist, as in game AI, right. Like Joseph: ⁓ if you mean for actual games, yeah. It you know, we do some work in gaming. For a problem like that, you would probably use a combination of some amount of the human data that they have. ⁓ you still wouldn't wrap the game by the way. You would still write a sim for that specific task just to make it actually run it at an appreciable speed. And then you would do RL on that possibly with a little bit of human data to start it. It's a very common approach. And then we just get very good at building sims. Like it turns out that this is it's a lot easier to just build good sims than it is to attempt to make anything work at a thousandth of the speed it should be running. Allen Roush: And then there's there's another game I'm sure you've probably thought about, ⁓ Dwarf Fortress. What what do you think of that? Joseph: That one's tough. ⁓ that one's yeah. It's a little more sandboxy, which is annoying for it's kind of you know, I haven't played Dwarf Fortress enough. I've played it like once or twice. ⁓ I had this slide deck back in the day that was the Minecraft is not the way you solve AI slide deck. And it the point was like the game is very open-ended, there really aren't that many things that the game wants you to do, and it's mostly people having fun in a sandbox, which is not a good task definition. You get a little bit of that in the Factorio-esque or even Dwarf Fortress like style simulation games. You can definitely write down tasks. in those games and optimize towards them and use that as a you know a good task. So I also don't know how fast it runs. If you told me that Dwarf Fortress runs fast, if you told me that runs fast, I'd be like, sure, cool, we can do it. Allen Roush: Yeah, that's famously fame So so so famously Dwarf Fortress, it may have had most of its code ported to multi-cpu very recently, but famously it was written by two brothers, and I think they both were not like especially great devs, and this is their own description. And so most of that game was CPU bound for the majority of its history. It does not run well. And I was actually gonna ask Rimworld, ⁓ you can see Rimworld is like a modernized dwarf fortress in that it runs a lot better, written in I think like Objective C or something like that. and it has The objective of get off the planet. Now, obviously it's kind of optional, right? But I I'm wondering if that that becomes a better game for this. Joseph: Yeah. You're probably several orders of magnitude low on perf targets there. for context, we can at least get net hack on multis ⁓ multithreaded running like a few hundred thousand steps per second. ⁓ actually millions, but then we get bottlenecked by ⁓ we get bottlenecked by observation encodings. We're still working on that. Allen Roush: Yeah. Yeah. Yeah, you have no hope of that with these games. Anyway Joseph: Yeah, you'd ⁓ the thing is like really big games like that are not good targets to rewrite. ⁓ games that have like really, really fancy physics or really fancy, you know, advanced systems, but the core game is relatively straightforward to understand, those are very good targets because we just write the simple version and then apply some of our you know, our kung fu and it works. But yeah, really s like games like that that would just take ridiculous amounts of code to rewrite aren't as good. If they're open source it does kind of help. Like, I don't know, maybe we could just optimize like the existing dwarf fortress code, but that's probably a that'd be a real big project. Real big project. Allen Roush: ⁓ so I I wanna I wanna p Cataclysm Dark Days Ahead, the best open source zombie sim, I want AI agents playing it so bad. That's the one that gets me. It's it's the game that Project Zomboid was ⁓ based on the Yeah. And Yeah, please. I play that religiously and I claim it's like ⁓ the complexity of Dwarf Fortress in a d somewhat more net hack like experience. Joseph: What is this kind of ⁓ okay. Car. We can pretty much do any of these. We just can't do all of them, right? At least not unless we get a whole bunch of ⁓ new contributors. And I know, and they need to be experienced contributors as well. You can't like you need somebody who actually has some systems background. They can't just be like vibe coding the whole thing. That does not work. We've had several people try to do that. Yeah. Ravid Shwartz-Ziv: Maybe yeah, it's ⁓ so actually it's a good ⁓ question because like w what do you think about coding agents? How how good are coding agents on these problems? Like optimized s simulators or like ⁓ I don't know, optimize the games and ⁓ even like making more efficient ⁓ the the algorithms more efficient? Joseph: So I've regrettably started using them in the last few months. That's really the only time they've started to get okay. They're still incredibly dumb. they basically never come up with any useful ideas whatsoever. They will write large amounts of code on a very well-defined task quickly. ⁓ I typically will use them more so in research than anything of like just here's an idea I want to try out. It would take me a week to implement this fully myself. ⁓ let's just see if this will implement it in an hour. And if I get the positive result, ⁓ at least I know it works and I can spend a few days like actually implementing it correctly. Unfortunately, it not working is not like the negative result doesn't hold because most of the time it just messed something up. So they're useful in some ways. ⁓ they're also they're useful in porting stuff. You know, like optimizing old games, actually it's a very good use case. You still need to have somebody who knows what they're doing though. You can't just say, ⁓ yeah, port this or whatever. You need to actually put some very substantial guardrails that have require you to know some system architecture. ⁓ otherwise it just it makes a mess very, very quickly. I will also say that these definitely make my job a lot less enjoyable. I that that has definitely been the case. Sometimes more productive, definitely less enjoyable. Ravid Shwartz-Ziv: But why why you think it's the case? Because like it looks that it the consensus is that if you have a, you know, very like verifiable signal, right? you can use them. So you have the speed, right? Like you just want to make it faster. So why you think you Joseph: They write terrible code. Here pu ⁓ yeah, here's your kernel. Cool. Plus five thousand lines works on this one exact setting of arguments. You know, they just they will write very, very bad code. And ⁓ your project can very quickly get out of hand. ⁓ I mean it's very easy to write one kernel that's just bigger than the entirety of PufferLib. What am I supposed to do with that? Ravid Shwartz-Ziv: And do you think like you can const we can construct like enough like metrics that to actually can they can high high high climbing on these metrics, like that they will be generalized enough? Joseph: Well, you can use them for sure and like you can you can use them productively. ⁓ I don't think you're gonna be able to come up with like some system of guidelines that will let somebody who has no idea what they're doing get the result like get a good result out of them though. ⁓ that has kind of been consistent the whole way through, is they've slowly become more useful, if you know what you're doing, and they just kind of let people who don't know what they're doing make a mess even faster than before. I mean I'll tell you like one thing I did like this morning I was interested in trying out there's an asynchronous architecture that ⁓ I've wanted to play with for the last year, but I just haven't set aside a week. So I just like you know, I went through with one of these agents and I said, Okay, let's grab this exact thing from here, not anything else. Okay, here's the plan. Plan is dumb, okay, fix this, this and this, cool. Go go try this. And okay, cool, it is faster as expected, but it's not stable, so now I have to go through and see, is it not stable because it did something stupid, or is it not stable because the algorithm is fundamentally has some off-policy drift? And now I have to go figure that out. And ⁓ if it went and wrote five hundred lines of code, I can probably figure that out very easily. If it writes three thousand lines of code, then good luck. So, you know, useful sometimes, definitely not particularly smart. Allen Roush: Ha have you observed improvements as the models have gotten better? For example, I ha like the new Lunar Sol, I forget which one's the biggest, five point GPT five point six. Joseph: Hm I actually I went to use that this morning and then I was like, Alright, what version of this I look do I use? I looked at the charts and apparently these things just burn tokens like crazy for the same result. So I just went back to five five for now until that gets sorted. ⁓ I will say in previous improvements, I barely notice. Like I really have barely noticed. They got to a point where they were good enough to be useful, I don't know, near the start of the year maybe. they The differences between versions are just very, very exaggerated from what I've seen. ⁓ I don't know, it's just a lot of hype. ⁓ in fact it's like it's to the point that I almost don't like to be using these on my stream a ton because it just it gives people the wrong idea that like and like I get people asking, ⁓ do I actually need to program or stuff or or anything? And yes, yes you do, even more so than ever in fact. so like, yeah, they're useful. Don't use them as a crutch. They get better. They don't get as ⁓ they don't get it better as quickly as people are saying. There's a lot of marketing around the things. I could do my work just as well without them, though probably a bit slower. Ravid Shwartz-Ziv: So w what are the main ⁓ bottlenecks? What do you think are like the the main thing that you actually need to improve, let's say in the next year or like in the next five years? ⁓ and whi which one of them are like let's say solvable? Joseph: So ⁓ for RL generally, there are a few things on infrastructure. I actually think despite having puffer as fast as it is, there's still a bit more juice to squeeze out of it. I'd like to solve breakout in under three seconds instead of four and a half seconds. That would be fun. now when I say that it's not just like, ⁓ well what's the point in solving breakout, right? The same thing transfers to every other Envin puffer. So you know it's just a blanket perf improvement. I think there's still quite a bit more to squeeze out of that. But ⁓ That's like a nice to have, not an essential. The essential thing, there's still a couple algorithmic problems that bother me. It's possible we could just work around them in all cases, but I think that we at least want to know for sure whether there's a way to investigate or not. there's this really long form credit assignment problem and this state coverage problem that work way better now, but like you can very easily construct cases where stuff fails. ⁓ even with something as simple as just making a really big maze with a sparse reward. And you kind of need something GoExplory to fix that. You need something that relies on settable state, settable simulator state, as well as an uncertainty meth trick, an uncertainty heuristic. Ideally a learned one. ⁓ I actually started doing research on this a couple months ago and we got some very good results. ⁓ You know, we actually were able to solve some of the pathological failure cases. I haven't been able to get the results to transfer to some tasks where we would expect them to work yet. But the problem is that the algorithm kind of blows up in complexity quite quickly with these methods. So it's probably just a matter of more careful experimentation. ⁓ due to some professional ⁓ professional obligations with Puffer, we're kind of prioritizing getting stable self-play first now. And we're probably gonna switch the Puffer 5 and Puffer 6 updates to where the heavy algorithm research goes in Puffer 6. So probably closer to the end of this year, ⁓ I'll be fully working on that stuff. And now I'm working more on self-play and infra. but I think that that is relatively solvable from what I've seen. And I think it really only is a few more algorithm changes before we have something that's just generally robust and stable. There's also a little bit of just scientific scientific work we need to do or rather just experimental work on why do ⁓ why sometimes do our hyperparameter sweeps find really janky weird looking hyperparameters that somehow work better than anything we would expect. Like what's going on here. I'd like stuff to be a little bit more consistent and predictable in that way. ⁓ a few other small things, but I mean for the most part now, here, let me put it this way. Puffer is stable enough that whenever anybody comes up with a medium complexity, like a medium complexity problem, something that would take months of research ⁓ months of research to get any sort of science of life a few years ago. If Puffer doesn't work out of the box at least reasonably well, I just by default I assume that there's a simulator bug. And like 90% of the time there is a simulator bug. So we're getting very close, but Before we move off of using games as our core research tasks and like trying to do more of our own applied research in-house, ⁓ we wanna be like absolutely certain it's a simulator bug, right? We don't ever wanna be questioning whether there's a problem with the algorithm or the architecture or the setup, right? Because you lose some of the interpretability in other areas, right, that you have in games. So you just wanna reduce the error surface, ⁓ if you're gonna go there. Ravid Shwartz-Ziv: Yeah. And do you think like when you move to I don't know biological problems you will need to deal with more problems and like other like types of ⁓ like your other types of solution you you need to to think of? Joseph: I don't think so. I think it's mostly gonna be having to learn a lot about whatever space it is, if it's biological sim, having to learn myself a lot of ⁓ a lot of stuff in that area, as well as simulator techniques in that area and how do we make something that's fast and good enough and how do we know when we've broken the sim without the amount of ⁓ you know, without being able to just watch an agent play a game. Right. I think it's mostly gonna be stuff like that. From what we've seen it ⁓ again, RL really is not as brittle as it used to be once you have these ⁓ this higher throughput infrastructure. Like pretty much I will get I'll play around with an algorithm on mazes and it'll work everywhere else. Unless I do something that's obviously specialized to mazes. But any sort of like general architecture work or like general hyperprim op like hyperprem sweep optimizer work or whatever that I do pretty much just transfers across the board. Allen Roush: So Joseph: And includes to professional work as well, you know, 'cause our professional work is not just in games. Allen Roush: So, ⁓ just just kind of a a more, you know, question about the the field broadly, do you think that the current investment in massive GPU clusters, HBM pr producting capacity and et cetera, that people sometimes want to write off as a bubble? Do you think it's a bubble? Like does your experience with R L being so efficient lead you to believe that everybody else is doing it wrong? Like an L L ⁓ land? Joseph: Yeah. We're working on different problems. I mean sometimes you see people throwing LMs on stuff where they clearly don't belong, or thinking that it's going to solve everything where there are clearly pa places where there is zero progress and you'd expect there to be zero progress. I think there is probably a bubble in the same sense as that the internet was a bubble, where it was the most important thing ever, but still a bubble. ⁓ I the thing that just annoys me more than anything about that whole situation is just the hype in the marketing and just lack of scientific rigor around ⁓ a lot of stuff as a result of there being so much money in the space, more than the actual money in the space, I would say. I mean the nice thing is I've kind of found myself in an area where, hey, for whatever reason, there's very little progress. There's a bunch of hard engineering to do. There's a very c clear path to progress. I am not super dependent on ⁓ large amounts of compute, though it would be really, really nice if my ⁓ you know, two K GPUs were not suddenly four K, that'd be really, really nice. ⁓ so hopefully that'll get fixed at some point. ⁓ but yeah, other than that I think it's going to play out as you would expect. Ravid Shwartz-Ziv: But do you think like people that are working on LMs or like even R L for LMs can use your ⁓ like lessons or the the thing that you have found? Joseph: There's definitely stuff in there for sure. Yeah, there's definitely stuff in there for sure. Not everything is tied to there just being ⁓ you know, not being able to learn a value function. We also have a bunch of nice benchmarks for core ⁓ core research. We have architecture work on encodings that are useful that probably transfer. I mean you really you'd have to get deeper I'd have to get deeper into the LM side to know exactly what to use. But we really have a huge, huge amount of stuff. ⁓ like here, one thing that was actually kind of funny. is ⁓ if you're i if your infrastructure is bad, you end up thinking that you need larger models than you actually need. Because the big model runs at the same speed as the small model because of all the overhead. So if the big model's 1% better, even though it's like double the flops, you end up using the big model. So you know this it's kind of funny that our models have gotten smaller and smaller over time just because they're actually hardware efficient. So I think that some of it it sort of rewrites some of the scaling curves a little bit in that sense. Like some of the research that was dependent on ⁓ you know, wall clock measurement of ⁓ different model sizes is kinda have to go back in question. let me think what else. Ravid Shwartz-Ziv: But y you don't think but like you don't think that like scaling as we see it ⁓ today, you know, like with L L Ms is also applied in ⁓ on your problems. You don't think that like Joseph: Definitely there's an advantage. ⁓ it can be. But like there are many ways to scale, right? So I could have just I guess I could have just done the same thing I did by going and getting a bunch of money and running everything on a thousand GPUs and building a bunch of very clunky infrastructure to make that work. But you know, equally as well I can just make the code a thousand times faster and do it on one GPU. Or, you know, we have multi-GPU as well. We can do it on eight GPUs for some of the bigger problems. by the time we're on eight GPUs with an efficient sim. We're already getting to the scale of a trillion observations for some of the bigger problems that we work on. ⁓ for context, if you walk around at your typical RL conference, you'll see a lot of plots in the ones and tens of millions per experiment. And ⁓ you know, you'll see a couple people working on high perf RL that have billion step experiments. Maybe a very, very few people with more than that. But it just the scale is already there. It's just that, you know, it What does it matter if we scale up by making the code faster or using more GPUs, right? Allen Roush: Well na ⁓ from a ⁓ like power efficiency standpoint, are the newer GPUs like Vera Rubin or Blackwell significantly helping you guys in your work? Ravid Shwartz-Ziv: Okay. Joseph: Mm. So there's there's kind of this funny arbitrage where because we don't need all of the V RAM, we're really not V RAM bound at all. Our models are tiny. We can kinda just run everything on fifty nineties or forty nineties, which are a tenth of the price. And no, I've even run stuff on like ⁓ B two hundreds before and it's literally the same speed. ⁓ because we just we don't need we don't need the high bandwidth memory. ⁓ you know, we don't need Allen Roush: Well what about the low precision computing stuff? Is there any way to take advantage of that? Joseph: We do have our stuff in B float sixteen, which that was I like to be clear, that doesn't happen in RL. We were very, very proud to get that stable. Like we had to rewrite I i we threw away PyTorch completely, because that didn't work in PyTorch at all. We had to write our whole stack in CUDAC with like fused everything to get that to work cleanly. ⁓ I don't know if going lower than that makes any sense. I think probably not at the current model scale because you kind of have to optimize things even more to get to take advantage of the low precision. I guess one way to think of the ⁓ to think of the stuff we do, so there are two things that are really, really hard infrastructure wise, right? Making big models fast and making really small models fast. Those are both separately hard for different reasons. ⁓ we've gotten pretty good at making the small models fast, but like if you if we go to int four or like what not int four, nobody trains with that. If we go to like float eight, FP eight or something. Like I I don't sh the level of optimization required to actually take advantage of that would be basically impossible to do. That's some really hard engineering. But you know, we're we're already at i we get twenty plus mil step per second training with BF sixteen. I'm just hoping that in the future the consumer cards continue to be released and like continue to have tensor cores and stuff on them. And if not, then yeah, we'll eventually have to go to the more expensive chips and we'll have to retune some things, probably. ⁓ it actually is very nice to have a fast CPU co located. ⁓ I think was that Vera Rubin that was what was announced recently with a fast CPU. Yeah, that is actually very nice. ⁓ I have no idea if that will be I I highly doubt that that will be released at anything close to comparable in price to the hardware that we use now, but you know, one can hope. I mean, we can we typically for our bigger runs, if we just go grab a 8x fifty ninety box. Allen Roush: I think so, yes. Joseph: Like we can scale some research over a hundred million steps per second already. ⁓ you can technically go above that, you know, but there is only so much gradient noise. You can't infinitely scale batch size. You can scale the model up, but there are diminishing returns to that as well. in fact the model size scaling is way, way weaker in RL than in other areas because of the sparsity of the signal. So thus far they're just hasn't really been a need for it. ⁓ the newer chips I so the Blackwell chips would only really be important if we were on much, much larger models. And thus far we haven't found any problems where you get strong enough model size scaling to justify that. Plus also people love deploying small models, like you can run them on a toaster. Ravid Shwartz-Ziv: You can make your toaster ⁓ playing ⁓ a game. ⁓ and w what do you think about a multi-agent? Do you see it like as a fundamental setting the compared to single one? Joseph: So lot of my research has been multi agent. When you have like a vaguely cooperative esque task, it's generally not that much different from single agent. Kind of just works. ⁓ stuff gets annoying with competitive multi agent, mostly because you lose the ability to evaluate score accurately or to evaluate performance accurately, since everything becomes relative to the performance of other agents. That one is tough. You know, with a cooperative task, you at least have like a fixed goal typically to optimize against. ⁓ when you introduce competition, you don't because it's just how much better are you than the other agents. ⁓ This is what we've been doing lately with our own research is on self play and trying to see if we can make stuff robust. You can very it's annoying because if you're just gonna compare, you know, do I beat my past opponents if I does every checkpoint improve over the previous ones? Like you get all sorts of non transitive problems where you beat every previous checkpoint, but you're getting worse against an even very simple scripted opponent on even simple tasks. actually one of the the research tasks we've been using for self play lately is Robocode. I don't know if anybody remembers that. That was this like old Java library that they would use to introduce programming. We'd program a little tank to fight other people's scripted tanks. So we ported that to C. And we have it super fast. We can train that at 100 million steps per second if we want. And ⁓ we've been using that for self-play research. It's a really fun task. There are a lot of different strategies. It's strongly, strongly non-transitive. You know, A beats B, B beats C, ⁓ C beats A type stuff, all sorts of cycles and whatnot. and what we've found so far is that unfortunately a lot of the self-play lit is not as strong as expected. And ⁓ yes, you can get consistent improvements over your past checkpoints, but it's very, very hard to get like generalization to held out opponents, and there's not really a good way to get a ton of diversity into the gameplay agents without either hacking something domain specific or doing a ton of randomization or using human data. yeah, any one of these. But I do think this is we'll solve this eventually, right? Like we ac we already we can get decent self play agents. There's just Ravid Shwartz-Ziv: And like Joseph: They're easy to exploit for the time being. Ravid Shwartz-Ziv: And like do you see a scenario that like you have like thousands of agents with running on power law? Joseph: Yeah, you can do thousands of agents, no problem. I mean, what's the task that involves the thousands of w that involves thousands of agents, though? Ravid Shwartz-Ziv: I don't know if you want to simulate some like interactions, right? Like even like between can be between humans, can be bit like even between some molecules or whatever. Allen Roush: Yeah. Joseph: Yeah, so you get local interactions with those, right? I mean Neural MMO has a thousand agents in it. for reference that's very efficient as well because we typically will run, even if we have a single agent game, we'll run eight thousand copies of it. So to our infrastructure, one big environment with eight thousand agents looks about the same as eight thousand environments with ⁓ one agent each. I mean you mostly just have to be careful with stuff in the environment, like if it doesn't reset, can agents get stuck? Is there any ways for parts of the environments to become degenerate? Like you end up with much ⁓ many more sim challenges than actual learning challenges. And that was actually from my research as well in my PhD. I found that a lot. Like it wasn't even getting the learning to to work that was the hard thing. It was coming up with a way to pose the problem where all the agents couldn't like bunch up and screw up a certain part of the game or a certain part of the sim. Ravid Shwartz-Ziv: Mm. Allen Roush: Well so so it seems that agent-to-agent communication, at least with LLMs, is still a big problem. How are you guys handling it with thousands of agents? Joseph: Well, I mean they're they're not communicating with language, that's for sure. ⁓ like what type of tasks though, would do you like what type of tasks do they need to communicate? 'Cause you can do cooperative tasks without having communication. Allen Roush: See the problem is I'm such an L L ⁓ person that I always want to think of LLM based things to do, right? I don't know, I guess ant colony optimization, so like moving objects or or solving mazes, which which is you know, you've talked about B Yeah. Joseph: Yeah. Well, they don't they sure don't need to talk to each other in English, right? 'Cause unless I'm like I'm sorely misinformed, I don't think that the ants can do that. ⁓ Allen Roush: But isn't there some notion of communication? Joseph: Yeah, you can have I actually there was a guy that had an end like that where they could leave pheromone trails and that seemed like it sort of worked as well. like you don't need to code that into the algorithm though, right? I mean heck you can even have it in the action space. of like it leav there there was a literally a like leave trail or whatever action there. That works. A lot of the time you get a lot of tasks though that are cooperative are just done through like body language, kind of theory of mind type stuff implicitly. I mean if you look at the one of my favorite examples is if you look at the open AI five Dota agents, they had stronger coordination than any human team ever for their team fighting and stuff and it's just body language, knowing who needs to be where at what time. They're not actually communicating via language or anything. Allen Roush: I was gonna ask about that too. ⁓ you know, that that so I I play a lot of League of Legends and I'd heard that, you know, the the Dota stuff was so interesting because they had ⁓ APM caps, action per minute caps that tried to make it kind of like humans. But I really do worry that they couldn't simulate some other aspects. For example, as humans, you know, we have to like move our eyes and they have like eye tracking that you can do on what we're looking at on the screen. ⁓ and I do believe that computers are simultaneously effectively looking at all parts of the screen at all points in time, which which I I I guess I don't know, like do you what what can we do to try to make systems act even more human like in reinforced learning domains? Joseph: So y you kind of don't want to, right? The APM one, that one feels like okay, you know, we should control that because that's just unfair. But being able to just pay attention to more things at once, that just seems like a fundamental advantage. Why why would we cap that? Right? You know, if you get port perfect coordination because they just know exactly where they need to be relative to each other and they're paying perfect attention to each other at all times, that's an advantage of the method. It's not even it's not even like coded explicitly that way or anything. There's nothing in the code that tells the agents to do that. They just figure that out. Allen Roush: Well, I mean I g I claim it's for the same reason that there's a vibrant science within the chess engine world of trying to make chess engines believably bad. Right? That's the problem is that you can set them, you know, lower elos, but they start making blunders that are uncharacteristic. Right? Joseph: Well in that case then it you probably sh ⁓ if that's the goal then you probably should just be using human data more heavily. Like if your goal is to make it human like use human data. If your goal is to solve the problem, then do RL. If your goal is to solve the problem but take away some like cheap things that it can do, then put some constraints on the sim or whatever, like you do with the APM caps. They're not too too many things like that though. And I think I think Dota was actually a better task than League, because Dota is a slower game in many places. So it was kind of good that that wasn't like that wasn't as big it's still a big thing, but not as big of a thing in the first place. Allen Roush: It it's it's also a much more complicated game. You know, by the same principles around like, ⁓ you know, don't do door fortress, it's too big, I'm always shocked that Dota got all that stuff instead of League. Joseph: Yeah. Mm-hmm. Yeah. Dota's I that is my favorite result for for reference. That's my favorite result in the whole field. ⁓ open AI five result. Mm-hmm. ⁓ way more than that. It was twenty nineteen, seven years. Mm-hmm. Allen Roush: And and that happened a long time ago now, too. Wasn't that like four years ago or more? Yeah. Yeah. God. Dang. Too long. Joseph: We have a few results. I mean we're we're operating at a much smaller scale. We have a few results though that I'm very proud of that aren't quite on the same tier as that, but I think the Pokemon result was pretty awesome. That was a collab. you know, some of our internal results, I think that the the stuff I've seen on Runescape is just I didn't expect that to work. That's nuts. I guess better than the it's better at the game than I am. what else is there that's like ⁓ I think that Even some of the stuff ha it's very annoying in RL because sometimes the impressive results are not actually the ones that other people would think are impressive. Like I get way more excited when I see the agent solving a big maze perfectly than when I see it, you know, on some new arcade game. Because I know one of those is very hard and one of those is very easy, and one of those transfers to everything else and one of those doesn't. Like Yeah. So sometimes it's very funny. Twenty forty eight ends up just being harder than the vast majority of the other tasks in Puffer that look harder, for instance. Just getting a very high score on that game. And we have like really, really good experimental evidence for every single task we chuck stuff at as well. You know, we have thousand param ⁓ thousand experiment hyperparam sweeps fully filling in the frontier of cost versus performance for every possible experiment. So I think for bigger results though, to to your point, like When will we start going for bigger results? We're kind of starting to do that now. At the very least, we're feeling it out, right? We're trying to get some bigger ends in there. maybe it'll be net hack. Maybe it'll be you know one of the RuneScape adjacent things. maybe it'll be something else. I think there are a couple sort of intermediate steps. ⁓ we have crafter and craft exported. Those are sort of like some of the harder research grade ends that we can work on that. It's very similar to NetHack actually, just like easier. so there's that one. And you know, we have all sorts of people coming up with new tasks that they submit all the time as well. So Allen Roush: When are these ⁓ methods going to influence real game AI? and I'll I'll point out I'm aware of like Arc Raiders, you know, already having been influenced by reinforcement learning, but it seems like, you know, one game that I can name ⁓ that people actually play, right? Is there anything else? Do you think it's coming soon? Joseph: We currently do work professionally in gaming. Can't give you details on that, but yeah, we do we do work professionally and ⁓ I think it's yeah, obviously one of the big applications of it. I don't think it's gonna be like the biggest application long term, but it is sort of an obvious shorter term next few years application. Ravid Shwartz-Ziv: Okay, maybe let's talk we are almost out of time, but maybe let's talk a bit about the open science. You have a like very like popular livestream and also you are like trying to to do open source ⁓ some of your work. why? Why why why are you doing it? Joseph: Mm. 'Cause it needs to be solved. I I mean I pretty much just sat down and said, Okay, well, nobody else is going to solve this, so I have so I'm like this needs to be solved. This tech needs to exist. Like RL was the coolest thing ever in twenty nineteen when Dota was solved, and then people kind of just wrote slow code and stopped doing it. Ravid Shwartz-Ziv: But why you're like open source it and why you're doing the live stream Joseph: Well, I mean open source it so it exists and so it can actually be used everywhere. ⁓ I figure there's a like the business side I can make money either way. It's a heck of a lot more fun to keep it open source. You get it goes faster, you get better code. Live streams, th those kind of just started to keep me entertained while I was working, and then that ended up being a really good way for people to get support on the library, ⁓ for new contributors to ask questions and to get their PRs reviewed live. So I enjoy it, it makes my work more fun, people find it useful, and we get more contributions as a result of it. no, I I there's really not much value to me in publishing papers at this point. The review cycle's really long, everything's outdated by the time that it gets accepted like it gets accepted. Half the time you're getting reviewed by L LMs anyways. it it's just a lot of time investment. Ravid Shwartz-Ziv: But what about like upload the papers to archive? Joseph: Yeah, I mean I can do that. I could do that, but it's just much, much easier. And like I put the same information out with a fifth of the time investment just by doing blog posts. it's just a easier format to do stuff in. Like if you scroll through the articles tablet X, we have all the details of the major breakthroughs that we've made throughout every version of Puffer. There's stuff on the changes that we made to PPO. There's like the finding that Muan actually works amazingly well. for ⁓ small model RL, ⁓ the new architecture from Puffer Four that sets soda on everything, that's fully described there. It's just a much easier format and it reaches more people as well. Ravid Shwartz-Ziv: And you don't afraid it like other like companies or people will will use it to to their own goals and Joseph: I I have other stuff I can do if they do, but like no, so far no not at all. I mean the most effective way look, if anything, like they can see how good this stuff is there and the cheapest way for them to get that internally is to just hire us. Ravid Shwartz-Ziv: Okay. ⁓ makes sense. okay, we we're out of time. Do you have anything else that you want to to add? Joseph: Well, I mean all this stuff, all of my stuff on Puffer is at Puffer dot AI. It's all open source there. There's docs. ⁓ hopefully some of the people watching will get inspired and maybe build some M's, try their hand at it, 'cause it's one of the few areas of AI that you can actually do at home without needing a ton of compute. Which is kinda funny 'cause it used to be the other way around that RL took all the compute and language models were super cheap, but I guess this this is the way it's panned out. GitHub, Twitter and Discord are all linked there. That's where we do all of our stuff. And yeah, thank you very much for having me here. Ravid Shwartz-Ziv: Thank you so much for ⁓ coming. It was a great pleasure to have you. Joseph: Thank you. Allen Roush: Yeah, I loved I love this discussion so much.