Latest / The Information Bottleneck / World Models | John Langford (Microsoft AI Labs)

Transcript

World Models | John Langford (Microsoft AI Labs) · The Information Bottleneck

• Ravid Shwartz-Ziv: everyone, and welcome back to the information butternet podcast, and today we have a special guest, John Lengford. היי, John, תודה רבה אז כן, ג'ון הוא אחד מהליטים של AI Lab at Microsoft בניו יורק, אז יש לנו הרבה תושבים לדבר עליהם היום, ואני חושב שאנחנו יכולים להתחיל עם מודלות מודלות אז היינו כמה אפיזורים John Langford: yeah. Ravid Shwartz-Ziv: על מודלות מודלות מודלות, אבל אני חושב שכאילו הוא להגיד לנו מה קורה, מה אתה חושב על כך? ולמי אנחנו צריכים לחשוב על זה, או אולי למי אתה חושב על מיוחד כזה שבמקום העובדה עכשיו במיוחד ללמי העפיל של הוולדה. John Langford: So think world models make a huge amount of sense anytime you care about sample complexity. So if you want to succeed with a relatively small number of samples, you want to be using your existing samples well. And then i i it's easy to come up with situations where you may know sort of the the the rules of the game, the dynamics of the environment, but you don't necessarily know its implications. And then if you have a world model, you can you know, extrapolate to the implications and then you can learn a you can compile if you want a policy to deal with those implications or you can just do the extrapolation at at inference time. So this is the the fundamental capability that a world model gives you and then it makes a it could make an enormous difference in terms of the number of samples required to learn. The existing transformers I think often require, you know, three or four orders of magnitude more information than a human would in order to learn something similar. So that's a that's a huge gap. world models may cross one of those orders of magnitude, which creates you know fundamentally more capable systems. Ravid Shwartz-Ziv: אז, ומאלה אתה חושב שהמודדות האלה צריכים להיות מודדות אקספליסטיות? או כשאתה יודע, למשל, סטפאנו סואטה היה פה כאן כמה שנים לפני כשהוא אמר, תראו, הללו, יש מודדות אנחנו לא יודעים שזה לא אקספליסטי, יש משהו שזה מאוד מיוחד, אז אנחנו יכולים לעשות את John Langford: I think there's nothing wrong with explicit world models. I just I find them very inefficient, compared to what you might hope to do. most people are not very good at rendering what they view. I I can't actually draw very well. my my kids can draw probably draw better than me. I and so may maybe it's not necessary. There's a lot of extra work associated with being very explicit. If you want to model sensory information that c that can take a lot of effort. And so I I'm a bigger fan of more implicit approaches because those can satisfy the purpose that I talked about while being you know radically more efficient. Allen Roush: Well I I I have a question related to this then. something I I don't know what percentage of the population has it. I think it's called a fantasia, which is like mind blindness, right? It's like the inability to visualize objects coherently in their mind. And similarly, I think a significant percentage of the population does not have any notion of a v verbalized internal monologue, right? And if there's kind of spectrums of this. So I do you Do you think that this has any relation to kind of like world models needing to like be able to internally model physics correctly or not? John Langford: I wasn't aware of these facts. kinda interesting ones. My expectation is that implicit world models are far more fundamental than the ability to visualize an object. And so even animals have internal world models. that they effectively use. That's my expectation. Now proving that of course is a is kind of a challenging thing, but just as an example, my brother had a dog, I visited his house. The dog loved to pay f play fetch. I started trying to mess with the dog so there was a fence on the property and there were bushes and I tried to throw the thing the the the stick into a place which would be difficult for the dog to find. And the dog searched effectively to find it. So that suggests that the dog has it's it's not just like some sort of passive you know policy which acts based upon what it sees. It it actually has an internal model of the world. It it would it would try on this side of the fence for a while and then go around the other side of the fence and try for a while and and and things like this. So th that that capacity I think is pretty fundamental to what we think of as intelligence. Ravid Shwartz-Ziv: And do you think it should be this like world model like internal מודל שייך ללמוד, בגלל שסוריה גנגולי הייתה פה, אני יודע. שני חודשים לפני, כמו לאירוסיינטים מסטנפורד, הייתה דיברת כזאת ביומנים, נכון? כמה זמן, כמו תלמד? או כמו שצריך ללמוד? וכל הדברים האלה. מה הבנתי שלך? האם אתה חושב שאנחנו יכולים ללמוד את מפריעה או שאנחנו צריכים לתת איזה כאילו... פלייסאם פרייארז John Langford: I think learning is absolutely essential. If you I I can I can argue this in two ways. One way is the number of bits in the genome is actually not that great. and so and the number of bits that are in involved in like the the the divergence between us and chimpanzees is e is incredibly small. so you you can't imagine that a lot of intelligence really comes from evolution down that path. The other way to argue this is to say, look, I had a car at one time and I knew it was a control combustion engine. I took physics classes. but I I really didn't actually do car stuff until I I remembered that I was supposed to change the oil in my car. This is when I was a grad student. So I changed the oil, and turned out that I I flooded the engine with oil. That's bad. And and that at that point I I was like, okay, I'm gonna take things apart, get the oil out, and figure out what's going on. And I could actually figure things out based upon you know, the ph physics that I had and like, you know, this is the internal combustion engine, I have four chambers. All all these things could make sense because I had this kind of internal structure of how the world should work. And I got it I wicked out all the oil, I put it back together, I drove the car, and it worked. there's some smoke to be at the beginning, but it it still worked. and I I guess the point here is that there is no chance that I learned this evolutionarily. Allen Roush: So you Ravid Shwartz-Ziv: אין לזה Allen Roush: y you you mentioned the the like genetics kind of stuff and DNA. I I think that there's also kind of the study of like I I I think my understanding is the changes of how DNA expresses itself over time in the form of proteins in a living being called epigenetics, right? John Langford: Yeah. Allen Roush: And I'm I'm always fascinated by like it it seems like this part is not really mentioned much in the kind of analogues to AI. W what would be the analogue to AI here? John Langford: So n first of all, in terms of inheritance, epigenetics as far as I can tell is a very few bits. It it's like just a handful of bits. it's like how stressed is the mother or something like that. things like this. in terms of an analog I guess I think of the intelligence relevant part of the DNA as being an analog to like the specification of the architecture and the training process. Right. So we did some pre-training processes like six stages that we went through and there's a bunch of details that you specify for that and and and so and so forth. th that that seems like it's you know of a similar complexity to the the number of bits involved and sort of defining the architecture and things like that. Maybe you want to throw in sort of the compiled code to some extent, but not not too much. the epigenetic things Guess I'm less clear on I mean maybe you think of them as just still similar to the DNA in terms of or to the to the the the the configuration of these things. But yeah, I I don't know beyond that. Allen Roush: I applaud you for engaging with this. I know it's tough. Ravid Shwartz-Ziv: אוקיי, let's assume that, like, we convince everyone that world models are important and we should learn them. Now, tell us, how we should do it and why we shouldn't do it like Jepa style models. Whoever, like, didn't, don't, don't know, like, to people that don't know what is Jepa, like, you can go to the previous episodes, we talked to Trondel and Yon and others. אז תגיד לנו, למה, איך יכולים ללמוד את זה ומה הדברים שאנחנו צריכים להגיד את John Langford: So let me tell you first my sort of cartoon version of Jeppa, which is based upon the old co training paper. It's like you have two views of some data and you try to learn the representation which is kind of Which you derive from both views, which which is shared. And so you're trying to kind of maximize that mutual representation. and and so and there's there's many different ways to try to apply this. So what the strength of this approach is is that it's it's unsupervised, and that means you need no label information. Now the the weakness is that it's unsupervised, and that means that the process can end up spending a lot of compute on things which are not very relevant. and i the process can also converge something which is not very relevant. So I think a a common experience with Jeppo style o optimization is that it's it's a little undefined what you converge to. I mean if unless you actively impose constraints, you could just converge to your latent is always zero. and and then you get great agreement between the views but but maybe it's not so useful. so Ravid Shwartz-Ziv: This is why we put regularization, right? John Langford: absolutely yeah. and then you p play around with EMA sometimes in order to try to tricks people apply. But you have to apply these tricks. it's it's not quite as clean an optimization as as Transformers are, where you just have a ton of gradients and you just optimize the gradients and you go. and so I guess the style of implicit world model that I've been thinking about has more of this transformer flavor where you get a ton of gradients and you go. There's no like, do I have to impose these extra organization constraints or do we have to play with EMA? It doesn't happen. so the the style of implicit world model I'm thinking about is ones where you have a latent, that latent has two purposes. It's summarizing your your information from your past for the purpose of predicting the future. So it's your it's your belief state. And it is also the object that your policy operates on to predict actions. And you are in fact predicting actions. So assuming that we have actions available is an assumption. It's it's not always true. but in many cases it it is true or it can be made true relatively easily. So this is an example. If you have videos and you have with with no action information, Maybe you can label a small fraction of the videos with the actions that are being used. And then you can the process of of of automating that labeling process is much easier than the process of you know learning the world model itself. Because you're just trying to predict very local things. and then you can you can apply that. So people did this kind of thing with Minecraft, for example, and you know, it it as as good as having actual labels for all the actions. so I think you can often come up with actions i in this way or maybe actions are just available. Anytime you're playing with language You can think of language as actions, it's an intentional thing that the people do. That's an action. and then y you have very plentiful actions i whenever your words become language. Or whenever your words become the actions. So Ravid Shwartz-Ziv: אז מה הטעות האלה בלינגריג'? John Langford: the words. The tokens. Yeah. Ravid Shwartz-Ziv: The wars themself, tokens are the tokens, so each token is like the action. John Langford: Yeah, yeah. They're also observations. Actions can be observations, that's fine. It's kind of a collapse of the hierarchy. you don't have actions and observations really split in language modeling. You tend to have them together. However, if you go towards the end of post training processes, often you start splitting these things. You start saying, I really want to just have my losses from the actions that I'm going to be taking and not from the responses of the tool calls as an example. So so actions are often available and if you have actions available, suddenly the process of getting this implicit world model can be radically simpler. Because what you can do is you can say, look, I wanna have this I wanna learn this belief state with a transition operator to the next belief state and and a policy hanging off of it. And I can I can stick that into a transformer. This is the the next latent paper, which is out right now. And i i it it it does very interesting things. Like we get a factor of three reduction in extrapolation errors just from the base learned model, doing no extrapolation of any sort, ex no Monte Carlo simulation at all, compared to just an extent predictor. And it it gives us much better speculative execution than like existing MTP style transformers and and things like this. Ravid Shwartz-Ziv: למה אתה חושב שאנחנו רואים כאילו גרועה גדולה? רק בגלל שאנחנו מבטיחים ההצעה? John Langford: Well, predicting the action stabilizes the process of learning this implicit world model. So that that seems like that's it just makes a clean optimization. But why does the implicit world model help us? And here I'm going to agree with something. I think transformers do have an implicit world model built into them. So you can think of the key values as a an implicit world model. It's just a very loose implicit world model that grows with every token. And then if you go and you think about why that might be bad, there's an analogy I like to draw, which is to astronomy. In astronomy, it used to be the case that people could predict where the planets would be in the sky using using an epicycle system. I think this is very old. Ptolemy worked on it a long time ago. but you you could predict where the planets would be day by day in the sky. and then later people switched over to the heliocentric system. So you had Copernicus and Kepler and Newton coming up with the you know this process of orbits and and thinking about the sun as the as being at the center of the solar system and and and so forth. And then It turns out that if you go to Mars, the heliocentric system works and the epicycle system does not. You have to get new constants when you're at Mars. so that's that that's saying something fairly important about first of all, what you want to optimize for, a simple right system. And second of all that's also what science optimizes for. And so a a big difference between a transformer as they typically exist with, you know, tons of key values as a belief state. And the approach that we're taking is is we're learning a more compact belief state. And so it's a it's a compact, implicit belief state, which I think really drives this ability to have fewer extrapolation errors, to be able to go to Mars and not have to redo your astronomy. Th that that's kind of the wonderful thing that you're optimizing for. Ravid Shwartz-Ziv: So basically you think like compression is the key element John Langford: I think that's yeah. So I think that's a the key element compared it to just a vanilla transformer. Ravid Shwartz-Ziv: אהה John Langford: So if you find the simple right way Allen Roush: So John Langford: to explain your observations, that tends to generalize better. Allen Roush: You You you were using this analogy to like the the Galileo, you know, heliocentric all of these examples. there's this really cool book called Against Method by Paul Fierabend, who is I believe one of the students of Karl Popper who like heavily analyzed this. And his whole book is basically fighting rationalism, right? The idea that like science has rules you need to follow outside of like reproducibility. kind of standpoint, like methodological rules meaning. I interpret this as a really early version of the bitter lesson, right? do you do you kind of agree with that and even agree with like the bitter lesson today that we should make as few assumptions as possible about like problem spaces and how even AI systems might get to solutions? John Langford: Hm. So the bitter lesson that you're thinking about is that you want to remove your biases from a system so that it it's as free to learn as possible. Allen Roush: Yes. Yes. John Langford: And and then you just hit it with tons of data. Allen Roush: Yes. John Langford: So I I generally agree with the inclination. I I like learning. I I've been working in learning for quite a long time. and I've certainly been impressed by what Transformers can do. that surprised me. I think there was a real surprise. I think probably that was the biggest surprise of my career, just how good Transformers could be. Ravid Shwartz-Ziv: ‫אבל אולי אנחנו יכולים לדבר על זה, ‫מה, למה אנחנו באמת, ‫מה, אולי אתה חושב שהם לא יהיו כך טוב, ‫מה, ‫הם לא יהיו כל כך טוב, ‫מה, למה אתה חושב שהם באמת, ‫היא הזכת לגבי טרנספורמרים? John Langford: w wasn't that I didn't think that they aren't that good. It was more like all the previous learning albums that I played with would sort of limit out, in terms of how good they could be. for one reason or another. and my expectation when I first heard about Transformers was, these will limit out too, I just don't know quite where it'll limit out. And that turned out not to be true so well. I mean that they actually go much further than I expected. I think they don't go as far as you want because they're far less efficient at learning than than people are. But I I think th they go quite a long ways. Ravid Shwartz-Ziv: Why? What is your guess why they are so good? John Langford: So over the development of Transformers, there was a process of learning how to create deeper circuits that you could actually tune with gradient descent. So that alone was, I think, very powerful. that's where resonance came from and things like this. And then Transformers themselves, beyond just deep networks. Obviously the tension mechanism is very helpful. I feel like I don't have a a strong answer here. There's a combination of learning how to do gradient descent at depth and attention, which seems to be an amazing combination. So where by attention I mean sort of key value retrieval. So key value retrieval is essentially a a non-parametric operation. You can you can learn anything if you get enough representational capacity. And then having things stacked up in layers, of course, allows you to build circuits and circuits can represent far more things than the number of components that are in them. so you you get this kind of commentorial effect where you you have a representational capacity which grows very rapidly with the depth of the circuit. And you have the ability to do gradient descent over this and have the gradient descent actually make sense, which was not true when I was a grad student. and so th this this combinations of things seems to have really hit the magic. Ravid Shwartz-Ziv: אז אם צריך לדעת את העמדה של הפרפורמנה עכשיו בין המדע, הדאטה והאלגוריתם, John Langford: Yeah. Ravid Shwartz-Ziv: האופטימייזרים, מה אתה חושב, איך אתה תדע את זה? John Langford: Yeah. This is something that I have disputes with people about. there are people who are of the mindset that what really matters is compute and data. And, you know, research does not matter anymore. Go away, you're bothering me. and I guess what I would point at is the K three paper, which I I I read through parts of it. it's actually great. I it's it's fantastic. There's a bunch of things in the K three paper. The K through technical report, which are are real innovations. Th they claim in that paper that they're getting about a factor of two point five improvement in how rapidly they learn compared to the K two, which was already I think using Muon, if I recall correctly. Ravid Shwartz-Ziv: אהה John Langford: which is already an improvement over just the vanilla Adam W like approach. And there's probably other improvements there as well. And so I guess the question is if you take the two point five seriously, and I think you probably should, that's just that the value of research is half the GPUs they trained with. More than half the GPUs they trained with. And now how much is half the GPUs they trained with cost? And I don't know, but it's tr a two point eight trillion parameter model. That's probably a lot of GPUs g doing burning a lot of flops. I I think that There's substantial benefits to be had from thinking about and rethinking about how to do things better. exactly what that is is is harder to say. I mean you you have these empirical scaling laws with data and and compute. It's harder to have a scaling law with research, but the res returns to research I think are real and they're very substantial. I think the best functioning labs all have a substantial research element. Ravid Shwartz-Ziv: Do you feel it like it's research? Because there are some arguments, right? That like, today everything is kind of like engineering skills, right? Like to have to make it John Langford: Mm. Ravid Shwartz-Ziv: everything right, like more efficient, that is run like, I don't know, faster, like on a larger amount of GPUs and there is no like real research anymore in frontier labs. Do you agree with that? John Langford: I think that's a danger. I think that's a failure mode. I think that I mean my assumption is that I I don't really know w what's going on at OpenAI or Anthropic, but my assumption is that they have internal research teams that are big enough that are suggesting innovations and they're actually trying them out. The Kimmy folks clearly are. the the Deep Seek folks are are doing great things. Th th there's there's various innovations that seem to be coming out all over the place. my impression is that the the set of innovations which are in the public space is enormous. It's it's challenging to know which ones actually matter and which ones would matter for you or with whatever you're trying to do. But there's a lot of innovations and the innovations really move things along quite a bit. Allen Roush: So so I think related to that we should start talking about you know, the open versus closed models debate. obviously, so we've had two very high quality r model releases from China now, GLM five point two and Kini K three that are both within Stone Throw's distance of of Fable, Opus five, etc. and now this seems to have set off all sorts of crazy things going on. You know, w several open letters, some in s full support of open AI with Jensen Wong. And You know, the Mark Zuckerberg writing similar things and then others supporting gatekeeping and now, you know, and it the Dario having you know, throwing his hat in the ring, and then the whole rest of the scientific staff at most of the frontier labs signing on to possibly the most restrictive interpretation of them all, re with a recent letter. and then and then I guess finally in the backdrop of all this, we might even be seeing a pop or whatever in the bubble. with with AI hardware stocks and freefall, to the point of you know, serious crazy stuff going on in South Korea where they're over leveraged. So what do you make of all of this? John Langford: Fun times. Ravid Shwartz-Ziv: חחחח John Langford: I think that we owe a debt to Vernon Revenge, with this notion of a singularity. This was turned into I think a fantastic V C capital pitch where people are like, all we need to do is throw in compute and data and and w we'll have a singularity and we will be the first ones there and because we'll take off exponentially, everybody will be left behind and and so the only the investments in us will matter. This is this is a great VC pitch, right? I I think what we've experienced is different. and in particular what we've seen is y you don't actually get there just with data and compute. Maybe we need to have world models. Maybe there needs to some research. the history of AI is littered with people who thought that it was easier than it actually was. Ravid Shwartz-Ziv: חחח John Langford: And and you also see that the open world is coming fast. and still a little bit behind maybe, but only a little bit in in some areas. And I I think that It doesn't surprise me to see that people at OpenAI or Anthropic want to keep the value of their company and their job. but on the other hand, I feel like it's probably gonna end up better if AI is not owned by a single person, if it's owned by everyone. that's a better approximation to what would be valuable for the rest of the the future than if it's owned by a single person. So I I kind of like the open approach much more. Allen Roush: So so r I I want to also relate this to your prior assertion of nobody was ever successful with highly centralized power being held. I want to give a few counterexamples. One from public policy and governance in the form of Singapore with LKY Li Kai Yun. I I I think I I might be butchering the pronunciation, but he was extremely popular and is widely considered basically a benevolent dictator. And I also want to point out Python's governance until very recently was benevolent dictator for life, the creator of it, and I claim that most people would review both of these periods, right? Singapore's creation and really time period up to now and Python throughout its whole Ravid Shwartz-Ziv: ‫התקדם Allen Roush: history as being quite good. So obviously you know, one is a state, a sovereign state, and the other is a programming language. So maybe it is okay to have a little bit of centralized power. Do you do you see these as counterexamples or how do you respond to this? John Langford: N not really. both of these cases are organizations which existed in competition with other organizations. There was a time when Python was not obviously the language that everybody would converge to. And so having somebody who made decisions nobody say this is I think people there's a natural tendency for people to converge on a language. And so some language would it would would be what people converge to. It happened to be Python. There were a lot of other candidates back in the day. And so it doesn't surprise me that in the short term, somebody making decisions can be very effective. and that seems fine. I'm sure that within individual organizations at OpenA or Anthropic or whatever. people make decis this people who make decisions and and you have to have you have to have that that capacity to make a decision and act, which you don't get out of committee. So th th that seems fine. What's what would not be so fine is if think about it separately, supposed only Singapore existed. there was nothing beyond Singapore, all the rest of the world was gone. And then I think that the lack I mean it it it could be the case that you can have a benevolent d benevolent dic dictator for life but the next life there's no guarantees. and that has been the history of dictatorships, right? So you can have a good dictatorship. They can do good things. And I think particularly in a in a competitive situation Where they're they're they're essentially forced to run. Singapore cannot be run badly. It's a city-state. Right? If it's run badly, it's just going to lose its i national identity. Python cannot be run badly. There are lots of c even now there are competing languages. And if it w if people started making bad decisions for Python, other languages would would become more popular. So so that that that that existing in an equilibrium situation means that they they can't be run badly. And so the downsides that you run into with badly run dictatorships like Russia, maybe don't come into play. And the issue with transition of leadership, I mean maybe you're lucky you get a good dictatorship, but then there's the next guy, whoever that is And the process of how exactly you transition is not defined in this concept. And so the outcome could be essentially arbitrary. Ravid Shwartz-Ziv: אז אני עוד שאלה, האם אתה חייב בניו יורק, האם אתה חושב שהאמת שאתה לא חייב בגלל הסיליקון, כמו סן פרנסיסקו, חשבתי את ההדברות על אופן סרו ושהעגי אי-אנס עצמו וכל הדברים האלה? John Langford: I mean the answer in generically has to be yes, 'cause everything affects you. It's your environment, right? Ravid Shwartz-Ziv: אבל, כמו כן, אני אספר לך, כמו חושב שהסיליקון ואלי יש איזה צמד אקו-צמבר של אופיניאנס, שאיזה, אני לא יודע, מרגישים לטייפים האלה? John Langford: Right, possibly. I mean people develop echo chambers. would it affect me? I guess I'm maybe, but maybe not too. I'm I I guess I I'm contrary enough that I don't always take the stances of people around me. Ravid Shwartz-Ziv: And what do you think about self-improvement? Now everyone are talking about recursive self-improvement, the models will John Langford: Mm-hmm. Ravid Shwartz-Ziv: train themself the next version of them. Do you believe in it? like, again, Frank Ruter, for example, was here last week, and he said, basically, like, we did auto-ML, like, decade ago, right? מודלס חייבו את הפרמטרים והפרמטרים אחרים של מודלס אחרים וזה לא משהו נוסף ואולי שזה כמעט קצת יותר קצת יותר קצת אבל זה עדיין אותה שנייה אז קודם אתה חושב שזה יהיה המדע? ומה אתה חושב על נובל? John Langford: I mean I think the answer is yes, but there's a a tricky issue with goalposts here, which is which maybe address. right now the process of training a model is extremely computer resisted because You you don't it's not like you put in punch cards with the code on them and put in punch cards with the data. You have you know, your GitHub repo with your code and your you have your data sets and th is is complete orchestration in the computer side which is doing this. And then you're asking slight a slightly stronger question, which is if you if you given all of this computer is already there Can you automate just the the the the another step, which is maybe research into the process of finding better models? and I would have to say currently researchers are many of them, myself included. are heavily using models to assist with research. So it's already the case that these models can assist substantially with research. And I mean I went through a process when we were pre-training. We went through 2,000 experiments in 90 days and 500 patches. I was doing a large fraction of that myself. it's possible with these modern agents to do these things. And can I imagine going a little bit further to making that process be automated? It's not too hard to imagine. I think that in a year it may be the case that you can set up an agents, a system of agents, maybe, and some GPUs and tell it to look at archive and Build the best model it can and just allocate things and then and i i it can go. Does that completely replace humans in the process? I'm not sure. I it's hard to trust that that it would I mean, even now I find the agents often do things wrong and you have to correct them in various ways. They might overlook things, they might mistake make mistakes and so forth. But so Plausibly still oversight will be needed and interventions will be needed, but I could easily imagine that they is another order of magnitude of of automation and another order of magnitude the year after that and and so forth. Allen Roush: I I I have a bunch of other questions related to how we how we think about the field going forward. So, you know, a lot of why people talk about and discuss world models is ro like the the demand hopefully for physical AI systems and robotics. Do you think that establishing effective coherent world models and systems that can can simulate the world like that internally are the prerequisite for physical? physical AI systems and if so, is it because they're synthetic data generators or something else? John Langford: I think it's strongly aligned with physical AI. and I guess I'm I think synthetic data generation is a particular application that could be used here. maybe it's back up. one of the key differentiators between physical AI and language modeling is just the quantity of data that's available. You just don't have that much data. Available for a lot of physical AI systems. And so being able to learn rapidly is a much stronger limiting factor in in physical AI, in robots, and and so forth. And then how do you actually learn faster? Well world models are one of the approaches which is is very plausible. And the there's sort of several ways you can apply world models here. One way is as you were saying with with synthetic data, right? So if if you have actual world models that simulate sensory data, then you can create synthetic data and you can learn a policy in that synthetic data and and so forth. I I think that's that's it's it's clearly useful. There there's a lot of people who work at driving startups or whatever that do these kinds of things and i it it's very helpful. At the same time, that's the explicit world model. With the implicit world model, you can start doing Monte Carlo rollouts at entrance time if you do your world models right. And that can allow your system to engage in its own self-improvement and to in just to have a smarter inference, right? So if you can look ahead several steps, you can go, look, I'm gonna crash into another car if I go that direction, or at least I have a danger of that. I'll avoid that and and then you don't crash into the car. So that ability to kind of pull back from the a predicts projected future to the present action is what a world model gives you, if even an implicit world model. And that capacity is incredibly useful. I mean this is essentially defensive driving. Which I've been I've been teaching my daughter to drive, so we've been talk discussing this. Ravid: Let's talk a bit about the pre training. So you mentioned the that like you're doing some work on pre training and like and and about like how can we actually like use the like the the most efficient way in the in the data that we have. So maybe start with describing the problem or like why like what why we should like it looks at like pre training is like almost solved, like or so maybe describe What is the the question that you that you're asking? John Langford: So the problem that I faced was we had a cold pre-training pipeline off of an obsolete architecture. And I wanted to create a hot pipeline so that various research that I was working on could be tried out in a on a serious pipeline, basically. I I think there's a there's a a frustrating issue which is which people are running into, which is the The process of showing that your research is relevant is challenging when you don't have a a good pipeline to test against. Right? And I think there's Ravid: You mean when now you have an idea, like now I have an idea and I want to test if this idea is good or not. So I need John Langford: Yeah. Ravid: a good pipeline in order to to make it fast, right? To iterate over it fast. John Langford: Yeah, absolutely. Yeah. Yeah. So I I think if you go and you look at the optimization literature, it's an enormous number of papers which are all coming up with the next update rule, which is better than the last one. But they don't really go anywhere. which is kinda frustrating. Ravid: D do you want to mention some names? John Langford: No, I I think it's i if you follow optimization literature at all, you'll run into this. and I think that's that's more generally true across algorithms from machine learning. I could I could go back to NERUP's from a a decade ago or ICML a decade ago and you know, or two decades ago. There's lots of papers there which are, you know, this is a better learning algorithm. Doesn't get used, it doesn't get adopted. And why is that? Well, I think often you have weak baselines effectively. I I think th there's there's a question of relevance and there's a question of strength of the baseline. So Sometimes people are just doing different things and so you just need different solutions. That's that's it's it's it's important. But also, it's hard to I mean these debugging machine learning algorithms is one of the most subtle things you can try to debug. It it's it's it's much more difficult than a SIG fault, right? if it's a SAG fault, you know, you hit it with ball grind and you can figure out what the line of code's causing this. You can you can you you can source things. Even Heisenbugs, things where, you know, sometimes there's a race condition or something like that. Often there's a way to reason about this and you can you can sort it out. Machine learning algorithms things don't work and why? Don't know. th it's it's kind of challenging. i i d d debugging these things is i i is is is extremely challenging. And if you're asking people to implement other baselines they don't really care about, maybe maybe there's a bug in the in the baseline and you know it's just the way it is. it it's it's hard to be sure. Yeah. Ravid: Because you don't have incentive to do it, right? Like if you write a paper, right, like you want to show that your method is better, right? John Langford: Yeah, and I and I don't think it's it's necessarily it's it's not like a a hostile action or animal or or a deceptive action. It's just kind of Maybe you didn't spend the extra week debugging the the baseline compared to your own code, right? it it's hard to say, you should spend an extra week on this because, you know, maybe these people have finite weeks and It's it's just challenging. anyway, th the the point I'm trying to make is that you really want to have a a strong baseline to operate against in order to be sure that your own research is really doing something important, at least on an empirical basis. Theory is also fun. I like theory, but but if you're doing something empirical, it it's good to have a strong baseline. And that that's why I did it. Ravid: Okay, so tell us like what you're what you have done. John Langford: Yeah, so the the I had this a cold pipeline, old data sets. there was Ravid: What is a call pipeline? Let's start with that. John Langford: some system for doing something like pre training, but it was old and architectures were old and things like that and In a period of about ninety days we did about two thousand experiments five hundred patches, and we trained a four B model over five T tokens that is comparable to the Quinn three four B models in terms of performance. Right? So we we got the things up to a a reasonable level of performance. Ravid: So but like okay, so like you take like an old system and like you modernize it and like and and then like run a lot of of of experiments in order to like to achieve like the I don't know state of the art, like almost state of the art open source. what like why it's difficult? John Langford: Well, the thing which is challenging is there's probably a hundred thousand machine learning papers that you might want to at least consider the possibility of. and you have a finite amount of compute and you need to, you know, run the experiments and implement the papers and Decide on the outcomes and choose one piece of code which is going to be your the thing that you're going to do the big run with, right? And you don't have very much time to do it. And I think this is actually kind of I've been thinking about this kind of from a a theory perspective, right? So if you want to You want to aggregate improvements from a very large number of sources rapidly with a finite amount of compute to test things. And there's a question about how you actually allocate your compute to these different possibilities. I developed a system for doing this. I'm trying to actually at the moment to I I wanna formalize this. I think there's like a kind of theorems about science that we can prove here. A about if you have a finite set of resources, you need you have a lot of ideas for how to experiment. How do you actually experiment and under what conditions can you aggregate these things into that one solution that you're going to spend most of your compute on Ravid: So but like at at the end like do you think like there is like a I don't know empirical solution for that? You know, like right, like do do you need some metric in order to optimize, right? So like do you think like that we have a good metric for that? John Langford: Yeah, that's good question. there's a question of of alignment here, right? So th the story of transformers is you can optimize log loss, cross entropy, and it has huge effects. they're aligned with what you want later down the line. I I think that there's there's something to that. There's also limits to that. and so you kind of want to plug in additional steps beyond the to turn a a raw model into more of an agent-like model. and then you would like to evaluate against that. So I I don't feel like I have the ultimate metric sorted out in my head, but there's definitely times Ravid: But do you think John Langford: when I've hit the limits of cross entropy alone as an objective. Ravid: Mm-hmm. But so do you think like we need some other like proxies metrics and no information or like what things like that that are related to to our downstream task but we don't need actually like to to train the like the full network in order to get it? John Langford: I think you can often train with a much smaller network on much smaller quantities of data. So when I was doing this I was I had a one one thousandth C elicrome, which was my kind of basic experimental unit. and that was a a a big downsample both in parameters and in in data and tokens. that was designed not for maximum performance at that scale, but rather for maximum fidelity to what would happen at larger scale. Ravid: But like there are different different options here, right? Like you can say, okay, so like we have like the question of like what it will w what is like the the downstream performance of of like this like this method, right? Or I can then we can say okay, let's try to to I don't to to optimize it with some like proxies. But there are Questions about like okay, now I have a lot of methods that right that like there is some relationship between them. So like maybe I can unify some of them, right? And like some of them are related. So probably I don't need to sample like very like close related methods, so I can choose like from like e only one from these methods, for example, right? And there is also like the the question of like of like reproducibility or like how easy is it? I just saw someone that took I think it was iClear maybe 2026, it took like their oral papers, the papers that got oral on iClear, and tried using agents, tried to re-implement them. And to reproduce actually like the results, the actual results. Like till like I don't know, like one hundred GPUs or like something like like that. and I think he got maybe like twenty or thirty percent that actually like reproducible. and this is like with agents that that like we we we know much more right, like it's much easier today than I don't know, two or three years ago. so the this the question, like if like these methods are like actually like make the the the claim that that they they they said in the in the paper. So like there are different questions. Like what do you think are the important aspects here? John Langford: So twenty or thirty percent reproducibility through agents is actually a very interesting number because i and it's like a glass half full versus glass half empty kind of view here, right? So the empty view is that you know, why aren't the other seventy or eighty percent reproducible? And maybe it's because the agents suck, or maybe it's because the papers suck. Ravid: Yeah. John Langford: Is i hard to say. Expect there's a little bit of both. but the last half full view here is that twenty thirty percent of the papers are reproducible. And that means that twenty thousand of your hundred thousand papers could be automatically tested against your code base and any improvements that they could bring could be brought to bear in an automated fashion. That's an amazing number, right? I mean th that's a it's a huge number. It's it's far beyond the experiments that I was able to do in the last three months or so. Right, and I expect that Ravid: So Allen Roush: Do John Langford: number shifts over time 'cause the agents are getting better. Allen Roush: So I I just wanna quickly shift to to something that you're actually very famous for. My understanding is that you had a significant hand in the invention of CAPTCHA. John Langford: Yeah. Allen Roush: and that has been a big part of the internet since I was a little kid, right? and and probably even before I was born, probably had some some influence. I'd love to hear, you know, your description of CAPTCHA and sort of the developments up to now and your opinion on kind of modernized usage of CATCH and CAPTCHA style stuff on the internet, and especially how you think it interacts with like AI agents, identity access management, etc. John Langford: Yeah. captures was pretty fun. So Udi Manber, who was chief scientist at Yahoo at the time, visited us at Carnegie Mellon and he was complaining that they had this kind of free for use by users things that people would, you know, automate the use of and that would that would make it to where it was wasn't really economically viable for Yahoo to do. And so we thought about this for a while. I think it was Louis Van An and Manuel Blum and myself, who were thinking about this quite a bit the beginning. And th that's kind of where captures came up. the first captcha was called GIMPy. And the reason why it's called GIMPy was because The GIMP, as in the the image manipulation program, was scripted to do this. At the time, GIMP had terrible documentation, essentially none about how to script. And the way I built this was through a process of one-bit debugging. So I would I would like put Ravid: wow. John Langford: a script in and it would succeed or fail and you know y you would just kinda mutate things one bit at a time or w w one l one little bit at a time in order to make this work. but it worked and it it seemed to actually be useful. now to be c clear, there were a bunch of other similar ideas that were going on at the time. th th there was this notion of maybe what you want to do is you wanna have like a individual characters which are like rotated and th that was anyways, the point is that the the idea was in the air. The name came out of our group and our an im an initial implementation came out of our group. that was it was pretty robust. We had kind of these overlapping words on a distorted background, which would be very difficult for OCR at the time. So that that's our that's where captures came from. And I've been surprised in two ways. One of them is you still occasionally get hate mail about this because people don't like them. the other thing is I've been surprised how long they have lasted. because my I mean, I'm coming at this from a machine learning perspective. This is like Okay, there's a chal problems which are challenging for machine learning right now, twenty-five years ago or whatever, and it it turned out to be a concept which is still useful to some extent. I wouldn't overstate the extent though, because agents indeed and modern machine learning is pretty effective at getting around a lot of these things. Ravid: so but the were there other options that you considered back then? Or like good options? John Langford: Yeah, we considered a variety of options. Ravid: Good ones? John Langford: There were audio captions that we considered speech recognition at the time was pretty poor. And so if you had overlapping voices speaking, s people could decode that, but it would be quite challenging for a computer to decode that. Ravid: And d do you think today we have good ways to distinguish between humans and and robots, you know, like to to have like modern capture that are good? Allen Roush: Like like a von Kampf test or whatever they called it from Blade Runner to find the replicants. John Langford: Yeah. I think it's more challenging nowadays. Ravid: W why I I will try so why you think like actually for example tax, you know, that like models are s still like write text that you you like you really know that this is like AI generated text. Like why you think like we we didn't solve this problem, for example? Well we can le actually we can distinguish between AI generated text and real ones, human ones. John Langford: Actually th this is a a funny thing here. I my personal style tended to use an dash now and then. And now I have to avoid that Allen Roush: huh. John Langford: because yeah, the the the the they're used by the models. Ravid: Yeah. Allen Roush: Well well Microsoft Word inserts them and so all good writing. I'll read all sorts of books from like twenty seventeen with dashes everywhere. John Langford: Yeah. I it i I mean, obviously there's still a bit of a gap between what the models can do and what humans can do. I've been impressed by how much the models have improved. So it's hard to bet against them improving further. Ravid: Mm-hmm. Okay, l let's talk a bit about optimizers. I know that you have the like we talked about it before, you have some works in the this area. So and included there is like a huge improvement in the in the last years. maybe you can you can describe it a bit. Like let's start from the beginning. Like where like why what are the these like huge improvements and why we should care about them? John Langford: Hmm. So when I started out in machine learning, gradient descent was a thing. But it was done with SGD, basically, or even with batch gradient descent. So like full gradient over the entire data set. and I would say that the The people who could make green descent work, SGD work, were relatively few and they had certain heuristics to whiten signal, for example, and things like this, and it wasn't very well known. over time I think a lot of these techniques became very well known. I would say Leon Batu was one of the leaders here. He had a a piece of code which was just showing how to do SGD very rapidly. And so he he he could run on RCB one in like a fract a couple seconds or something like that. one of the things that I worked on for quite some time was this Volpal Wabbit learning algorithm, which it was kind of is it was an online learning system, maybe the first of its kind anywhere in the world. So it wasn't just like a demonstration. It was like you could use this as a as a learning tool. And so it was the first learning tool that would allow you to kind of use any amount of data to learn. Before that, everything was kind of like loaded all up into RAM, and then you do like batch operations in order to learn. So that that style of of online learning, online gradient descent, became common. And of course, many of the online update algorithms that people played around with and thought about at the time, we implemented and and and tried out and we improved things in various ways. In the process, I would say there were two papers related to Autograp that came out. There's one that the Autograad comes from and there's also one from Brendan McMahon, which were the w that's kind of the source of that's the Ada and Adam. so adaptive is what the auto stands for. and then that was augmented with the and then with the W, Adam W. The is a momentum, W is is the weight decay. each one was a slight tweak which improved things further. And and then people kinda fixed on that for a large to a large extent in deep learning. It's like, I'm gonna use Adam W and I'm just gonna be stuck on that. and then I could say about a year and a half ago, the modern Nano G PT speedrun was a real breakthrough because that was inarguably a strong baseline. People were really trying to optimize to make that transformer train as f as quickly as possible. And Muon made a big difference there. and then maybe a little bit later Normuon people's various variants people have done at this point. But the general message here is that the orthonormal update rules were a a a breakthrough. And then w how what is the breakthrough exactly? Well if you're trying to do these update rules, you're really governed by sort of the there's a speed limit which is connecting the learning rate times your principal eigenvalue. When you're trying to update a matrix. because if you try to go too fast, then you're you you you update too far and the system kind of destabilizes and doesn't work. And and now how do you get around the speed limit? Well you don't really get around the speed limit, but you somehow make the speed limit be much more uniformly enforced. So with your orthonormal update rules, what you do is you make sure your principal eigenvalue is the same eigenvalue as all of your other eigenvalues. And so you update all your other d directions as fast as you can update in the principal direction. And that ends up being maybe a factor of two improvement in speed when you optimize well with the orthonormal updates. Ravid: And Allen Roush: So I I I I also have questions. I I'm I also did not know you were the creator of Vopal Wabbit, which is always fun to say to people. I've always been fascinated by incremental and online learning and especially frustrated about the complete lack of you know, it feels like for example, everybody got so freaked out after Microsoft Tay and what 4chan did that it almost seems like o OpenAI and Anthropic haven't even tried, despite the fact that Transformers absolutely do support incremental training, at least in a technical sense, right? So w what what's your take on modern incremental learning? John Langford: I wouldn't bet the p it's not happening. I mean the the the terms of service for Anthropic changed I think about a year ago so that they can use whatever data by default in their training process. Allen Roush: But that's but that's after the fact for like D P O, right? You know, that somebody it's like being labelled and then it it gets put into the next I'm talking about training at inference time, right? John Langford: Yeah. And so if you just took your inference and you plug that in and literally trained on that, I don't expect the system to decay. And so it's important that you actually have a a selection process where you are saying, this is the kind of thing that I want. And this isn't not the kind of thing that I want. So it's like it's like a reinforcement learning element of things. So this could be DPO. But there needs to be some kind of entropic or entropy rejection process, which is which is going into the system in order to have Ravid: There is they they are theoretical. John Langford: so you don't you never want to learn from all your data in a in an imitative sense. You want to Actively selected against some things perhaps and actively select for some other things. Ravid: There are some theoretical work, right? From I think you're in Gao that said that you you can't train on like you can't do this loop like without any intervention because it like we just collapse. so so okay, but but l let's go back a bit like to to to optimizer. So but do you think like Moon is like a step forward like in terms of Like speed all only or like do you think this is something that it it learns like convert to to better solutions or like to solutions that generalize better or something like that? Do you think there is something like in normalized eigenvalues that that is not only like optimization speed boosting? John Langford: I mean it's certainly learning from less data. And so it is in some sense addressing that that core deficit that transformers have, they learn much slower than humans. So that that's a nice thing. Ravid: And and why I think it's happening. John Langford: I think that they they they're just that they're learning in these extra dimensions basically. So if you are doing normal gradient descent, maybe what you're doing is you're you're learning against your principal eigenvector and that you converge that and then you learn against your next eigenvector and you converge that and so forth. But here you you can learn in these other dimensions at the same time. But Ravid: And do you think it's like specially important in specific steps of the training process and the civ specific phases of the training process? and for example like pre-training versus post-training or like beginning versus the end? Do you think like we will we we will see like bigger impact? John Langford: Yeah, so I think there's bigger impact in the beginning because if you're very deep into the training process, you're already very near to the optima. If you're very near to the optimum, then it's kinda like there aren't that many extra dimensions to actually learn from. Ravid: Okay. I think we are almost out of time. do you have anything else that you want to add? to advertise, to to sell, hiring? John Langford: I don't have anything at the moment, but it's been good chatting. Ravid: Okay. So thank you so much. It Allen Roush: Okay. Ravid: was a real pleasure. Allen Roush: Good luck to everybody at Nurep's review buttles right now. John Langford: yes, we're we're doing some of that too. Ravid: Yeah. John Langford: Talk to you later. Goodbye. Ravid: Bye everyone. Thank you.