Latest / Dwarkesh Podcast / AI researchers debate how close we are to recursive self-improvement

Transcript

AI researchers debate how close we are to recursive self-improvement · Dwarkesh Podcast

• 0:00
  Speaker 1: Today, I'm chatting with three of my AI researcher friends,
• 0:03
  Speaker 1: from whom I learn a lot every time we talk,
• 0:05
  Speaker 1: and who also happen to be at somewhat open-ish labs
• 0:09
  Speaker 1: and companies, so you guys can actually say things on
• 0:11
  Speaker 1: the record. I'm joined by Baron Milic, who is the
• 0:14
  Speaker 1: CTO of Zyphra, which is developing open source models. John Shulman,
• 0:17
  Speaker 1: who is the chief scientist at Thinking Machines, previously the
• 0:20
  Speaker 1: co-founder of OpenAI, led the RLHF work that led to ChetchuPT.
• 0:24
  Speaker 1: And Charlie O'Neill, who is head of model training at BaseTen.
• 0:29
  Speaker 1: The first question I have, if we're in 2036, it's
• 0:31
  Speaker 1: been 10 years, and we don't have like billions of
• 0:35
  Speaker 1: crazy super intelligences that are running around that have like
• 0:38
  Speaker 1: radically transformed the world, what is the most likely reason
• 0:42
  Speaker 1: that that doesn't end up being the case? Other than
• 0:44
  Speaker 1: sort of exogenous political shocks or like there's a war
• 0:47
  Speaker 1: or they banned AI or something. But what is the
• 0:49
  Speaker 1: most likely technical reason that 2036 isn't like a crazy
• 0:53
  Speaker 1: alien super intelligence world?
• 0:55
  Speaker 2: I mean, like my reason would just be like, it's
• 0:58
  Speaker 2: got to be the sort of, like, there's been a
• 1:01
  Speaker 2: classic thing, almost like Marvec's paradox, right? Where like, we
• 1:03
  Speaker 2: see like, you know, we think of the AI being like,
• 1:05
  Speaker 2: if it can do this, it's going to be amazing, right?
• 1:07
  Speaker 2: Like if it can solve these hard math problems, if
• 1:09
  Speaker 2: it can win a chess, blah, blah, blah. And then
• 1:10
  Speaker 2: it solves these things, and then it's not that impactful. Obviously,
• 1:13
  Speaker 2: it's somewhat impactful, but not everything. If somehow that continues
• 1:17
  Speaker 2: and there's never the true spark of generalization that occurs,
• 1:21
  Speaker 2: I think that could lead to the AI just being
• 1:23
  Speaker 2: extremely good at everything that people put into a benchmark,
• 1:26
  Speaker 2: put into an environment, but there's still some persistent sim
• 1:29
  Speaker 2: to real, which is somehow blocking everything. I think this
• 1:32
  Speaker 2: is kind of unlikely. I think we do actually see
• 1:34
  Speaker 2: this kind of generalization even from RL in practice already.
• 1:37
  Speaker 2: But if it is just ridiculously hard to generalize meta-learning,
• 1:41
  Speaker 2: plus we don't solve container learning, it's just super hard
• 1:43
  Speaker 2: and impossible. This would be my default scenario in that case.
• 1:47
  Speaker 1: Yeah, I agree with that.
• 1:49
  Speaker 3: Humans have a lot of advantages over models now. And
• 1:54
  Speaker 3: each time a new model comes out, it'll catch up
• 1:59
  Speaker 3: in some of these areas. But you end up getting
• 2:02
  Speaker 3: bottlenecked by the places where the model is weaker and
• 2:05
  Speaker 3: where it has... worse judgment or, um, the models can't
• 2:10
  Speaker 3: check themselves well enough.
• 2:11
  Speaker 1: Yeah.
• 2:11
  Speaker 3: So, so there's this cycle that keeps repeating where people think, uh,
• 2:15
  Speaker 3: where a new model comes out and people are blown
• 2:17
  Speaker 3: away and they're like, this is it, this is the,
• 2:19
  Speaker 3: this is AGI, but then, uh, they use it a
• 2:21
  Speaker 3: bit and, and then it starts to feel dumb after
• 2:23
  Speaker 3: a month or so. So that cycle just might keep
• 2:26
  Speaker 3: going and it's hard to predict how many times it's,
• 2:29
  Speaker 3: it's going to repeat. And, uh, like right now you
• 2:31
  Speaker 3: don't get explosive growth, um, in capabilities because, uh, you
• 2:36
  Speaker 3: still get bottlenecked enough when you're trying to do research
• 2:38
  Speaker 3: and engineering that even if the model can write way
• 2:41
  Speaker 3: more code than a person, it doesn't make you like
• 2:44
  Speaker 3: a hundred times more productive. But yeah, so maybe they're
• 2:48
  Speaker 3: just more of these cycles than we would expect.
• 2:52
  Speaker 4: For me, it's like a question of how far off
• 2:55
  Speaker 4: like this global optimum of a learner you could have
• 2:58
  Speaker 4: on a chip is like the transformer plus like RL,
• 3:02
  Speaker 4: basically like the current recipe. So like, I think people
• 3:05
  Speaker 4: imagine that even once you have an agent which is
• 3:10
  Speaker 4: better than all humans at AI research, even if it's
• 3:12
  Speaker 4: like 0.1% better than all humans, then the fact that
• 3:15
  Speaker 4: you can run hundreds of thousands, if not millions of
• 3:17
  Speaker 4: these in parallel, you can run them much faster, like
• 3:20
  Speaker 4: chip's going to speed up, that's going to outweigh every
• 3:22
  Speaker 4: other bottleneck and you're eventually just going to.
• 3:24
  Speaker 5: Hit this very fast takeoff with recursive self-improvement.
• 3:27
  Speaker 4: I could imagine that if we continue along the trajectory
• 3:30
  Speaker 4: that we're currently on with that paradigm where it's basically
• 3:33
  Speaker 4: just like, Self-attention, RL, scaling up RL environments.
• 3:38
  Speaker 5: I guess if you.
• 3:39
  Speaker 4: Think about what happened with Moore's Law, we had this
• 3:42
  Speaker 4: very nice straight line and that held for a really,
• 3:44
  Speaker 4: really long time. But there were so many discrete discontinuities
• 3:49
  Speaker 4: and innovations that had to happen to keep that scaling
• 3:50
  Speaker 4: law going.
• 3:51
  Speaker 5: And the same thing has kind of happened with LLMs.
• 3:54
  Speaker 4: We had this pre-training scaling law and then that was
• 3:57
  Speaker 4: kind of hitting the diminishing returns. And then we came
• 3:59
  Speaker 4: up with RL and solved that. And then we got
• 4:01
  Speaker 4: this new diminishing returns algorithm curve to hit that made
• 4:04
  Speaker 4: it keep looking like a straight line going up. And
• 4:06
  Speaker 4: so if it requires another one of those discontinuities to solve,
• 4:10
  Speaker 4: I'm not sure that the current method of training LLMs
• 4:14
  Speaker 4: with these RL environments, even RSI-targeted RL environments, would be
• 4:18
  Speaker 4: able to discover that discontinuity. And if not, we're probably
• 4:23
  Speaker 4: going to hit this asymptotic curve where like.
• 4:25
  Speaker 1: Sorry, but do you think the discontinuity will be harder
• 4:28
  Speaker 1: than anything that's come since 2012?
• 4:31
  Speaker 4: If we had the answer to that, we'd kind of
• 4:33
  Speaker 4: have the ability to implement it. But maybe we should
• 4:37
  Speaker 4: distinguish between a discontinuity, which adds to the current paradigm.
• 4:40
  Speaker 5: Again, it's cumulative.
• 4:41
  Speaker 4: There's something beyond the RL that we have to discover,
• 4:43
  Speaker 4: and maybe they're capable of connecting the dots in that
• 4:47
  Speaker 4: straight line. Or, again, how far off the global optimum
• 4:51
  Speaker 4: are we? Do we have to go back and throw out...
• 4:53
  Speaker 4: you know, gradient descent and like neural nets in general.
• 4:56
  Speaker 4: And I don't think like if you continue to scale
• 4:58
  Speaker 4: up the current paradigm and LLM, no matter how many
• 5:00
  Speaker 4: LLMs you're running are capable of necessarily discovering that if
• 5:04
  Speaker 4: it's too far away.
• 5:05
  Speaker 1: Yeah. The only hope really is if deep learning just
• 5:09
  Speaker 1: can't get us to an AI, which is at least
• 5:14
  Speaker 1: can dominate human research and human development, including the human
• 5:18
  Speaker 1: ability to come up with new paradigms and so forth. Or, like,
• 5:21
  Speaker 1: I don't know, maybe humans would also never have discovered
• 5:24
  Speaker 1: the next learning architecture, but to the extent humans could
• 5:27
  Speaker 1: have discovered it eventually. But it just seems like, I
• 5:30
  Speaker 1: don't know, if you just look at the progress that's
• 5:33
  Speaker 1: happened in 2012 till now, and you just continue that on,
• 5:37
  Speaker 1: I mean, I know it's been powered by huge amounts
• 5:39
  Speaker 1: of compute scaling and so forth, but it would be
• 5:42
  Speaker 1: weird if, like, it just didn't get to the point
• 5:45
  Speaker 1: where it could, like, dominate humans, at least in R
• 5:48
  Speaker 1: & D, especially... Over the next few years, there's going
• 5:50
  Speaker 1: to be, Ryan Greenblatt was on the podcast recently, and
• 5:54
  Speaker 1: he made this point that you could imagine as AIs
• 5:56
  Speaker 1: get more and more capable and are capable of making
• 5:59
  Speaker 1: progress on simulations, which incentivize getting better at not only
• 6:04
  Speaker 1: AI R & D, but generally at science. So this
• 6:07
  Speaker 1: is a thing that all the labs are targeting, many
• 6:08
  Speaker 1: startups are targeting. Or another intuition pump is if you
• 6:11
  Speaker 1: look at the ELO score of chess bots since the 80s,
• 6:14
  Speaker 1: there's just like a very linear increase in ELO over time.
• 6:18
  Speaker 1: But there's this huge discontinuity as they cross the human
• 6:20
  Speaker 1: range of human experts always win against AIs to human
• 6:24
  Speaker 1: experts never win against AIs as this linear increase in
• 6:27
  Speaker 1: need will happen. And you could think, I agree with
• 6:29
  Speaker 1: your point that So far, AI capabilities have not been
• 6:33
  Speaker 1: that big of a deal in terms of their end
• 6:34
  Speaker 1: economic impact in the world. But that's because they're slowly
• 6:37
  Speaker 1: rising in ELO relative to humans.
• 6:40
  Speaker 2: Yeah, I mean, I agree. I mean, the only way
• 6:42
  Speaker 2: for this to not happen is if, as you said,
• 6:44
  Speaker 2: somehow asymptotes just before, basically, because we're already pretty close,
• 6:47
  Speaker 2: in my opinion, to where we'll start crossing the human
• 6:50
  Speaker 2: ELO score. And so we'll need to asymptote before that.
• 6:52
  Speaker 2: And that's the only way. In this scenario you posed,
• 6:55
  Speaker 2: we're like, somehow we're sitting here in 2035 and everything
• 6:57
  Speaker 2: is normal for this to happen, I think. I mean,
• 6:59
  Speaker 2: the only other way is, like, there's, like, some dramatic, like,
• 7:01
  Speaker 2: regulation on AI. It's like, this is kind of what
• 7:03
  Speaker 2: I see as, like, the most likely way for this
• 7:05
  Speaker 2: scenario to happen, actually, rather than the technical thing.
• 7:07
  Speaker 5: Yeah, I think there's different kinds of research.
• 7:09
  Speaker 4: There's, like, research where it's, like, the auto-research style where
• 7:12
  Speaker 4: the objective is already specified very cleanly.
• 7:15
  Speaker 5: Oh, for sure, yes. And you're optimizing that objective.
• 7:16
  Speaker 4: And I think everyone is picturing, like, if we continue
• 7:19
  Speaker 4: along this path of making pre-training loss go down, making
• 7:22
  Speaker 4: our environments go up, that's going to lead to improvement.
• 7:25
  Speaker 4: But maybe what Ryan is talking about is this much
• 7:27
  Speaker 4: more open-ended type of science, which is required for paradigm shifts,
• 7:31
  Speaker 4: where we can't specify the objective, and the AIs are
• 7:33
  Speaker 4: definitely not able to specify that objective either. We have
• 7:36
  Speaker 4: to be really, really careful about how we specify objectives
• 7:38
  Speaker 4: for any of these things.
• 7:39
  Speaker 1: And maybe your point is that the nature of the
• 7:41
  Speaker 1: breakthroughs that have happened since 2012 is that we have found...
• 7:45
  Speaker 1: In 2012, people weren't saying... I'm assuming, I don't know,
• 7:48
  Speaker 1: you guys were there. Or at least, John, you were there.
• 7:50
  Speaker 5: I was in primary school.
• 7:54
  Speaker 1: Actually, John, I'm curious about your wisdom of the ages
• 7:58
  Speaker 1: or wisdom of being in the trenches way back when.
• 8:01
  Speaker 1: But presumably a big breakthrough was realizing that next token
• 8:04
  Speaker 1: prediction is the... You wouldn't have thought that nano GPT
• 8:08
  Speaker 1: speedrun is the thing to be optimizing for in 2014.
• 8:11
  Speaker 3: Yeah.
• 8:12
  Speaker 1: But now that we have come to this new paradigm,
• 8:14
  Speaker 1: you wouldn't think to do a speed run on that
• 8:16
  Speaker 1: and have AIs get really good at that. But maybe
• 8:18
  Speaker 1: there's like a next inner loop to optimize that the
• 8:22
  Speaker 1: AIs wouldn't anticipate. And there's an outer loop of like
• 8:24
  Speaker 1: revenue or something that eventually should be strong, but it's
• 8:28
  Speaker 1: a very slow outer loop. Yeah.
• 8:30
  Speaker 3: In fact, I remember in the early OpenAI days having
• 8:33
  Speaker 3: the intuition that actually just like minimizing log loss wasn't
• 8:40
  Speaker 3: going to get you to intelligence, uh, because, uh, like
• 8:44
  Speaker 3: the important bits are accounting to, for such a small
• 8:47
  Speaker 3: fraction of the loss, uh, that like it was going
• 8:50
  Speaker 3: to be overwhelmed by noise. So just training a language model, uh,
• 8:55
  Speaker 3: on next token prediction just wasn't going to learn the
• 8:58
  Speaker 3: interesting things you wanted to learn. And, uh, we needed
• 9:01
  Speaker 3: to craft better objectives that would put more emphasis on
• 9:04
  Speaker 3: the important things. And, uh, like, You can make all
• 9:07
  Speaker 3: sorts of arguments for this and you could say, oh,
• 9:10
  Speaker 3: humans probably don't learn how to, like, we don't learn
• 9:15
  Speaker 3: how to model everything in our environment. We can't, most, like,
• 9:19
  Speaker 3: people can't create a photorealistic reproduction of some kind of
• 9:24
  Speaker 3: scene they've looked at. So there must be, we must
• 9:29
  Speaker 3: need a better objective. But then it turned out that
• 9:31
  Speaker 3: it just worked anyway.
• 9:32
  Speaker 1: As you were pointing out, the inner loop, even in
• 9:35
  Speaker 1: current AI research of, like, post-training benchmarks or whatever, it
• 9:39
  Speaker 1: doesn't necessarily translate into what users like.
• 9:41
  Speaker 3: Oh yeah. I mean, the whole field relies a lot
• 9:44
  Speaker 3: on generalization and it's very hard to, uh, predict when
• 9:47
  Speaker 3: you're going to get generalization or when you're going to
• 9:50
  Speaker 3: get some kind of out of distribution generalization. So we
• 9:52
  Speaker 3: know that if you train on the task you care about,
• 9:54
  Speaker 3: you're going to do better. But the most important advances
• 9:58
  Speaker 3: are often the types of generalization that we have no
• 10:05
  Speaker 3: right to expect. So for example, from just pre-training on
• 10:09
  Speaker 3: this very naive next token prediction objective, to various tasks
• 10:16
  Speaker 3: of interest that require understanding of the input in some
• 10:22
  Speaker 3: deep way, or learning some skill from pre-training that's very
• 10:27
  Speaker 3: rare and not very heavily represented. And then also generalization
• 10:34
  Speaker 3: from these verifiable tasks to less verifiable ones. This is
• 10:38
  Speaker 3: also a type of generalization that there's no reason to...
• 10:41
  Speaker 3: a priori to expect it.
• 10:44
  Speaker 1: So this is an interesting question because one intuition pump
• 10:47
  Speaker 1: that you could have for why you would see some
• 10:50
  Speaker 1: sort of singularity very rapidly without even scaling up the
• 10:54
  Speaker 1: inputs to AI progress that are not just AI labor
• 10:57
  Speaker 1: is that before every single experiment you run, that's like
• 11:00
  Speaker 1: a seven-figure experiment. You spend an equivalent amount of compute
• 11:03
  Speaker 1: on AI labor. And so you just have automated versions
• 11:07
  Speaker 1: of you guys spending a century thinking about what is
• 11:11
  Speaker 1: the optimal experiment to run, doing small-scale ablations, developing literally
• 11:17
  Speaker 1: a century's worth of theory. So going back even before
• 11:21
  Speaker 1: deep learning, before you decide what experiment to run, doing
• 11:24
  Speaker 1: extremely optimal setting up of the experiment, then you do
• 11:27
  Speaker 1: a century of thinking after the experiment is over. where
• 11:29
  Speaker 1: you're like analyzing what happened and what the next experiment
• 11:32
  Speaker 1: to run is.
• 11:33
  Speaker 3: Yeah. Well, I think if, if you think hard enough,
• 11:35
  Speaker 3: you probably could have, um, expected some of these things beforehand. Like, uh,
• 11:40
  Speaker 3: like there is probably some very clever way to do
• 11:42
  Speaker 3: a small scale experiment that'll let you build the theory
• 11:45
  Speaker 3: that then will generalize to the large scale experiments. So
• 11:48
  Speaker 3: I would expect that like we're nowhere near the ceiling
• 11:50
  Speaker 3: of how well you can do research. And like, I
• 11:54
  Speaker 3: would imagine a future where, um, where AI is like, um,
• 11:59
  Speaker 3: is doing a lot of analysis and theory building. Spending
• 12:05
  Speaker 3: a comparable amount of compute to the amount that you're
• 12:07
  Speaker 3: spending on the experiments themselves, doing various kinds of analysis
• 12:11
  Speaker 3: and building a theory around what we've seen so far.
• 12:15
  Speaker 4: I think there's really concrete examples of this when the
• 12:18
  Speaker 4: objective is well-specified. So again, all thinking can do is
• 12:21
  Speaker 4: update your posterior based on the bits that you've gotten
• 12:24
  Speaker 4: since you formed your prior. You can't gain any new
• 12:27
  Speaker 4: bits from just thinking. but when the objective is well
• 12:29
  Speaker 4: specified and there is like this data sitting around, like
• 12:32
  Speaker 4: I imagine there will be this big speed up in
• 12:34
  Speaker 4: the current paradigm we're in. And like, good example, this
• 12:35
  Speaker 4: is like, you know, if you've got an AI to
• 12:37
  Speaker 4: think about like the Kaplan scaling laws, like an AI
• 12:41
  Speaker 4: at this point would have noticed that like, you know, Oh,
• 12:43
  Speaker 4: like they've just taken these intermediate checkpoints and didn't account
• 12:46
  Speaker 4: for like the annealing.
• 12:47
  Speaker 5: And so like, this is wrong.
• 12:48
  Speaker 4: And like, that would have caught that like years earlier,
• 12:50
  Speaker 4: we would have made like, you know, progress, like, like
• 12:53
  Speaker 4: would have cut off a year or two of progress
• 12:54
  Speaker 4: just from that, like observation from an AI. Um, And like, again,
• 12:58
  Speaker 4: once the objective is well specified, which is like lower
• 13:00
  Speaker 4: pre-training loss or whatever, like there's many, many good examples
• 13:03
  Speaker 4: where if you just thought about it a bit more,
• 13:05
  Speaker 4: you would have been able to like cut down a
• 13:06
  Speaker 4: significant on things that you've done. So like mu P
• 13:09
  Speaker 4: and like how learning rate scales with like model size
• 13:12
  Speaker 4: and like realizing the model width is important in that
• 13:14
  Speaker 4: as well. Um, I feel like you can really back
• 13:18
  Speaker 4: out a lot of these things and cut off a
• 13:19
  Speaker 4: lot of hanging fruits. I would imagine a 10 times
• 13:21
  Speaker 4: speed up if our thing is just maximize the objective
• 13:24
  Speaker 4: we're currently on. But I don't see how that generalizes
• 13:27
  Speaker 4: at all to come up with the right objective in
• 13:29
  Speaker 4: the first place. Just thinking doesn't necessarily buy you the
• 13:32
  Speaker 4: right objective in.
• 13:32
  Speaker 5: The first place.
• 13:33
  Speaker 2: I mean, yeah, I think this is really the key
• 13:35
  Speaker 2: question to any kind of very rapid RSIs from current AIs.
• 13:39
  Speaker 2: It's like, how well can AIs generalize to learning their
• 13:41
  Speaker 2: own objectives? Because to have any kind of self-propelling automated loop,
• 13:45
  Speaker 2: you need the AI to propose objectives, optimize them, figure
• 13:48
  Speaker 2: that out, propose a new objective, and have this not
• 13:50
  Speaker 2: go off the rails at any point for a long,
• 13:52
  Speaker 2: long time. Coming back to Morvex Paradox, there might be
• 13:55
  Speaker 2: a case of Morvex Paradox where we think there's autonomy
• 13:58
  Speaker 2: and being self-encapsulated so we can think of what we
• 14:02
  Speaker 2: should do ourselves and then go do it and have
• 14:04
  Speaker 2: this loop. It's super easy because we always do this.
• 14:06
  Speaker 2: And obviously evolution needs to create creatures that can survive
• 14:08
  Speaker 2: on them by themselves for a long period of time.
• 14:10
  Speaker 2: And this just might be something that for some reason
• 14:12
  Speaker 2: is really hard for the AI in the same way
• 14:14
  Speaker 2: that locomotion stuff is really hard. Math is super easy
• 14:17
  Speaker 2: despite being super hard for us.
• 14:18
  Speaker 1: I don't know. Doesn't the time horizon increasing suggest that.
• 14:21
  Speaker 2: That's... Yeah, exactly. I mean, this is another possibility. But
• 14:24
  Speaker 2: I agree. There's no obvious evidence for this. In fact,
• 14:27
  Speaker 2: the fact that RLH is now super persistent and it's
• 14:30
  Speaker 2: quite easy to do this is kind of evidence against this.
• 14:32
  Speaker 2: But this would be potentially one of the reasons why
• 14:35
  Speaker 2: we just don't get this immediate takeoff is if this
• 14:38
  Speaker 2: is hard.
• 14:39
  Speaker 1: If you look back from 2012 till now, or maybe
• 14:43
  Speaker 1: from when you started doing your research till now, what
• 14:46
  Speaker 1: part of all the innovations that have happened since that time,
• 14:53
  Speaker 1: including purely engineering ones, including purely conceptual ones, what seems
• 14:58
  Speaker 1: like the thing that is the thing that would be
• 15:01
  Speaker 1: the last things humans would have to do before AIs
• 15:03
  Speaker 1: totally automate AI R & D?
• 15:05
  Speaker 2: Probably just iteratively asking the right questions. If you can
• 15:08
  Speaker 2: get the AI to do any experiment, but you need
• 15:10
  Speaker 2: to decide what experiments to do. And right now, I
• 15:12
  Speaker 2: think AIs are not very good at this compared to
• 15:15
  Speaker 2: coding the experiment at all. Whenever we talk about research,
• 15:18
  Speaker 2: they propose a bunch of miscellaneous things, which are very,
• 15:21
  Speaker 2: very tiny steps.
• 15:22
  Speaker 4: Or even going from DeepMind's approach of we're going to
• 15:24
  Speaker 4: solve intelligence by learning to play games at a superhuman level.
• 15:28
  Speaker 4: That's going to be the approach to one random researcher,
• 15:30
  Speaker 4: like Radford being like, I'm going to try and just
• 15:33
  Speaker 4: predict the next token off a very wide swath of data.
• 15:38
  Speaker 4: And then even once Radford had discovered that, it took
• 15:40
  Speaker 4: a while before people decided to scale it up because
• 15:42
  Speaker 4: we had to come up with the idea of scaling
• 15:44
  Speaker 4: walls and the fact that you could very reliably predict
• 15:46
  Speaker 4: these things.
• 15:47
  Speaker 3: Yeah. I would say that the last job for humans,
• 15:51
  Speaker 3: or the role for humans that will last the longest
• 15:55
  Speaker 3: is defining the objective and deciding what we actually want.
• 16:00
  Speaker 3: So in that vein, Something like deciding how the assistants
• 16:10
  Speaker 3: should behave or what it means to be helpful or
• 16:12
  Speaker 3: what's the objective when we're doing our all-from-human feedback is
• 16:16
  Speaker 3: one such thing. And then later, defining constitutions and model
• 16:22
  Speaker 3: specs is another one. And I think even if the
• 16:26
  Speaker 3: AIs can do all the technical work, we'll have to
• 16:28
  Speaker 3: still do a lot of that and decide what we
• 16:30
  Speaker 3: actually want.
• 16:31
  Speaker 6: Yeah.
• 16:31
  Speaker 1: Alignment is the final job.
• 16:33
  Speaker 5: Yeah.
• 16:33
  Speaker 3: Alignment is sort of the answer, but it's also, um,
• 16:36
  Speaker 3: alignment itself can be kind of decomposed into, um, into like, uh,
• 16:41
  Speaker 3: specification of the objective or figuring out what the right
• 16:43
  Speaker 3: objective should be. And then, uh, like actually, uh, like
• 16:48
  Speaker 3: achieving or optimizing the objective you've defined. And, uh, I
• 16:52
  Speaker 3: think the first one is not going to go away
• 16:55
  Speaker 3: anytime soon. And like, if I think about like a
• 16:59
  Speaker 3: post-training team and, uh, why you need a lot of
• 17:02
  Speaker 3: people to be on the team. It's just because there
• 17:06
  Speaker 3: are a lot of different areas where you have to
• 17:09
  Speaker 3: figure out how the model should behave. It would be
• 17:17
  Speaker 3: very hard to automate the whole thing just because someone
• 17:19
  Speaker 3: has to think about how should the model behave in
• 17:22
  Speaker 3: this area.
• 17:24
  Speaker 1: Jane Street started using Antithesis to test their software in
• 17:27
  Speaker 1: early 2025. And they were so impressed by the product
• 17:30
  Speaker 1: that they decided to invest in the company. I recently
• 17:32
  Speaker 1: caught up with Ron Minsky, who co-leads Jane Street's tech group,
• 17:36
  Speaker 1: to ask about how Antithesis actually plugs in.
• 17:38
  Speaker 6: The thing that I think is most impressive about Antithesis
• 17:41
  Speaker 6: is we started using it in a team that was
• 17:43
  Speaker 6: building high-assurance software and being really careful. And nonetheless, it
• 17:48
  Speaker 6: was able to shake out bugs that were otherwise going
• 17:50
  Speaker 6: to be really hard to find. And that's important both
• 17:53
  Speaker 6: because it helps make those systems more reliable, but also
• 17:56
  Speaker 6: because it helps the teams that build it to just
• 17:58
  Speaker 6: move faster.
• 17:59
  Speaker 1: This matters more and more as code production is increasingly automated.
• 18:02
  Speaker 6: I think in general, as we've been using agents more
• 18:05
  Speaker 6: and more, the key problem that you run into is
• 18:07
  Speaker 6: the verification bottleneck.
• 18:08
  Speaker 5: Just the time it takes from.
• 18:10
  Speaker 6: People to look at code and figure out, is that
• 18:12
  Speaker 6: actually something you want to accept in your production software?
• 18:16
  Speaker 6: And tools that make testing better are just incredibly helpful there.
• 18:19
  Speaker 6: They just ease the verification bottleneck and make it possible
• 18:22
  Speaker 6: for you to get more stuff done and move faster
• 18:24
  Speaker 6: because you can have more confidence that the code generated
• 18:27
  Speaker 6: by the agent is actually not introducing new problems.
• 18:30
  Speaker 1: To see how Antithesis fits into your development process, go
• 18:33
  Speaker 1: to antithesis.com. What is the story for why there isn't
• 18:42
  Speaker 1: huge consolidation in model providers? There's just so many things
• 18:45
  Speaker 1: that point to centralization here. If you step back over
• 18:51
  Speaker 1: the course of years, is there something that is going
• 18:53
  Speaker 1: to prevent that?
• 18:54
  Speaker 3: Yeah, I think distillation is the main thing that fights
• 18:57
  Speaker 3: against the centralizing force, because basically anything that can be
• 19:01
  Speaker 3: learned through RL can be distilled very easily, because it's
• 19:07
  Speaker 3: a small number of bits. It's something that you can
• 19:10
  Speaker 3: learn from a small amount of data. So if you
• 19:13
  Speaker 3: can get trajectories from the model that show a behavior,
• 19:17
  Speaker 3: you can easily distill it. So I think distillation is
• 19:21
  Speaker 3: one of the things that fight centralization.
• 19:26
  Speaker 1: There's also, I.
• 19:29
  Speaker 3: Mean, there is a possibility that there'll be company-specific models,
• 19:33
  Speaker 3: that it'll be possible to learn from deployment and have
• 19:38
  Speaker 3: a company continually improving its own model. And such a
• 19:43
  Speaker 3: system could be provided by the current oligopoly of model
• 19:46
  Speaker 3: providers or some other currently small company. But I think
• 19:51
  Speaker 3: that'll change the game a bit.
• 19:54
  Speaker 2: And I also want to point out that continual learning
• 19:56
  Speaker 2: doesn't stop distillation. Even if your model is improving every day,
• 19:59
  Speaker 2: people could be distilling it every day. So the loops
• 20:02
  Speaker 2: could just operate at the same pace.
• 20:03
  Speaker 1: Right, that makes sense. Okay, so copying model behavior... I
• 20:10
  Speaker 1: guess you need to know yourself what the right distribution
• 20:13
  Speaker 1: to prompt is in order to get the relevant model behavior.
• 20:15
  Speaker 3: Yeah, for just distilling with supervised learning, the prompt distribution
• 20:22
  Speaker 3: is extremely important. So it's very non-trivial to distill a model,
• 20:26
  Speaker 3: even if you have full access to it and have
• 20:28
  Speaker 3: the cot, the chain of thought and everything. Yeah, it's
• 20:32
  Speaker 3: non-trivial to distill all of the all of the useful
• 20:36
  Speaker 3: capabilities from it because you need to prompt the model
• 20:39
  Speaker 3: with something and you need to prompt it with like
• 20:43
  Speaker 3: realistic prompts. You need to have a really wide distribution
• 20:47
  Speaker 3: of realistic prompts. So yeah, one thing that's been coming
• 20:51
  Speaker 3: out recently is some of the some of the Chinese
• 20:55
  Speaker 3: companies are probably using these, uh, router services, which are
• 20:59
  Speaker 3: designed to allow people in China to use, uh, the
• 21:01
  Speaker 3: U S frontier models, which would otherwise be blocked in China. Uh,
• 21:05
  Speaker 3: but there are all these, uh, router or proxy services
• 21:08
  Speaker 3: that allow people in China to use these models mostly
• 21:10
  Speaker 3: for coding and, uh, and these router services are collecting
• 21:15
  Speaker 3: and selling some.
• 21:16
  Speaker 1: Of the data.
• 21:16
  Speaker 3: So I think this is, uh, like a very useful
• 21:19
  Speaker 3: data set for, uh, distillation because it gives you the
• 21:24
  Speaker 3: perfect prompt distribution.
• 21:25
  Speaker 2: I think this is one of those things where AIs
• 21:27
  Speaker 2: help a lot here. Like if you actually look at like,
• 21:29
  Speaker 2: you know, the frontier pipelines or say like the Chinese
• 21:31
  Speaker 2: models that they actually put in their papers, it's a
• 21:33
  Speaker 2: lot of like humans or like they get seed prompts
• 21:35
  Speaker 2: from somewhere, which is some combination of humans, this kind
• 21:37
  Speaker 2: of data. And then they like synthesize a vast coverage
• 21:40
  Speaker 2: from those seed prompts using their existing models or like
• 21:42
  Speaker 2: the other frontier models. And so it's like, you can
• 21:44
  Speaker 2: automate like an awful lot of this, like prompt distribution
• 21:47
  Speaker 2: gathering and like environment creation. It's just like humans need
• 21:49
  Speaker 2: to provide like increasingly few amounts of bits. It's like
• 21:51
  Speaker 2: the models get better.
• 21:53
  Speaker 1: Right. It still seems you're bottlenecked by like having a
• 21:58
  Speaker 1: service which has users or users are going through. Not necessarily.
• 22:01
  Speaker 2: I mean, like, yeah, that's obviously very helpful, but like
• 22:03
  Speaker 2: theoretically you can just think about like what users want
• 22:06
  Speaker 2: or like a.
• 22:06
  Speaker 1: Lot of tasks. The whole point is that we don't,
• 22:09
  Speaker 1: the user says, make me an application like this. Oh,
• 22:12
  Speaker 1: that didn't work. I actually want you to make this
• 22:13
  Speaker 1: new feature, but actually let's step back and do this
• 22:16
  Speaker 1: other thing. And capturing that whole trace is the, Or
• 22:19
  Speaker 1: to the extent you could have done that anyways, then
• 22:21
  Speaker 1: you just have, like, RSI.
• 22:23
  Speaker 2: Yeah, I mean, like, ultimately, like, if you have this, like,
• 22:26
  Speaker 2: fully automated loop, that is basically RSI, right? Like, the
• 22:28
  Speaker 2: AI is deciding the data, it's deciding the training, that
• 22:30
  Speaker 2: is the loop. But, yeah, I mean, like, it depends
• 22:32
  Speaker 2: how much human information you need. Like, at some point,
• 22:34
  Speaker 2: if you're just like, I want traces that look like this,
• 22:36
  Speaker 2: you prompt that to the model, the model will be
• 22:38
  Speaker 2: able to, like, come up with, like, a pretty good approximation.
• 22:40
  Speaker 1: But what if you want to do, like, make me
• 22:41
  Speaker 1: a really good politician and then just, like, anticipate de novo?
• 22:44
  Speaker 6: Yeah.
• 22:45
  Speaker 1: How would a discussion in the Senate halls go or something?
• 22:48
  Speaker 1: I just feel like there's going to be a lot
• 22:50
  Speaker 1: of things.
• 22:50
  Speaker 2: Ironically, this is actually, I think, easier for the distillers
• 22:53
  Speaker 2: than the frontier labs, right? Because the distillers are just like,
• 22:55
  Speaker 2: I want a good politician. They go to the frontier model.
• 22:57
  Speaker 2: The frontier model already knows how to be a good politician,
• 22:59
  Speaker 2: so it just generates those traces. Whereas if you actually
• 23:01
  Speaker 2: want to build the first model that does this, you
• 23:03
  Speaker 2: have to actually somehow get data on what politicians do
• 23:06
  Speaker 2: every day and build that. So it's actually much easier
• 23:09
  Speaker 2: to say, I want something like this, and then get
• 23:11
  Speaker 2: the AI to produce a billion variations. than to actually
• 23:14
  Speaker 2: create the thing like this to begin with.
• 23:15
  Speaker 4: I think you can actually make a really concrete prediction
• 23:17
  Speaker 4: based off this observation that the Chinese labs have this
• 23:21
  Speaker 4: router data. So I think the thing that gestated this
• 23:24
  Speaker 4: originally was I was saying, isn't it weird how Sonnet
• 23:27
  Speaker 4: 5 and Opus 5 are almost objectively worse models than
• 23:32
  Speaker 4: GLM 5.3, Kimi K3, even though they've had access to
• 23:36
  Speaker 4: not only distillation, but logic distillation from Mythos. And so
• 23:39
  Speaker 4: the counter here was that like, okay, the prompt distribution really,
• 23:41
  Speaker 4: really matters. Like you need to see what users are
• 23:44
  Speaker 4: doing so that you can distill like kind of these
• 23:46
  Speaker 4: behaviors and things in. I think the prediction from this
• 23:49
  Speaker 4: is that the frontier labs don't necessarily have much of
• 23:52
  Speaker 4: an advantage, if at all, in RL environments now. Because yes,
• 23:56
  Speaker 4: user distribution matters for general behavior and so on, but
• 24:00
  Speaker 4: the best measure of a capability is the very, very
• 24:02
  Speaker 4: hard RL environments you've made at the frontier. And so
• 24:06
  Speaker 4: if you have access to those RL environments as anthropic,
• 24:08
  Speaker 4: and you have access to logic distillation, and you've still
• 24:10
  Speaker 4: made a worse model, then.
• 24:12
  Speaker 5: Maybe, like.
• 24:13
  Speaker 1: That real-world deployment matters more than the environment.
• 24:15
  Speaker 5: Yeah.
• 24:15
  Speaker 1: That's really interesting. So, but they had to incentivize those
• 24:19
  Speaker 1: capabilities in the first place in Fable or the Frontier model.
• 24:24
  Speaker 1: And so it's weird that they can't incentivize them again with, like,
• 24:28
  Speaker 1: a smaller model or something.
• 24:30
  Speaker 4: Maybe, like, maybe we're just in this weird, like, uncanny
• 24:32
  Speaker 4: valley where, you know, like, actually trying to copy that
• 24:35
  Speaker 4: frontier model too much, like the student-teacher gap, whatever it is,
• 24:39
  Speaker 4: is just like too large. And like, I think people
• 24:41
  Speaker 4: made this point with Opus is it's like, the difference
• 24:43
  Speaker 4: between Opus 4.6 and Opus 5 is that Opus 5
• 24:46
  Speaker 4: really feels like it's got this like AI as a
• 24:48
  Speaker 4: judge checking every possible thing it's done.
• 24:50
  Speaker 5: And it's like, that's why it uses so many tokens.
• 24:52
  Speaker 4: It like tries to think about all these things, but
• 24:54
  Speaker 4: it doesn't necessarily have the big model smell of Fable
• 24:57
  Speaker 4: to know when to like stop doing that or like
• 24:59
  Speaker 4: when's a good path to go down or whatever.
• 25:00
  Speaker 1: The reach exceeds the grasp.
• 25:01
  Speaker 5: Yeah.
• 25:02
  Speaker 3: Yeah, I would offer a slightly different hypothesis. So I
• 25:06
  Speaker 3: would say there are a couple of different axes for
• 25:10
  Speaker 3: the environments you can create. And like one of them
• 25:12
  Speaker 3: is difficulty and the other is realism. It's sort of
• 25:16
  Speaker 3: easy to create or it's comparatively easy to create a
• 25:19
  Speaker 3: lot of difficult environments like that are just like. involve, uh,
• 25:26
  Speaker 3: like doing a much more complicated task or doing something
• 25:29
  Speaker 3: that requires a lot more cleverness. And, uh, you could,
• 25:32
  Speaker 3: you could say this is like the benchmarking distribution because
• 25:35
  Speaker 3: a lot of the most prominent benchmarks just involve doing
• 25:37
  Speaker 3: some very hard puzzle-like task, uh, that's easy to verify.
• 25:41
  Speaker 3: And then there's sort of, uh, like the realism axis
• 25:44
  Speaker 3: where you want the model to be good in the
• 25:46
  Speaker 3: realistic coding agent setting where there's like multiple back and
• 25:49
  Speaker 3: forth with the human and there's like multiple objectives. And, uh,
• 25:53
  Speaker 3: like I'd say, um, like the people, like the labs
• 25:58
  Speaker 3: who are crafting the model behavior for the first time
• 26:01
  Speaker 3: need to push in both directions and to get good
• 26:03
  Speaker 3: model behavior, you need to really push on the realism
• 26:06
  Speaker 3: axis and have like rubrics or some kind of human
• 26:10
  Speaker 3: feedback that's informing the reward function you use there. But
• 26:16
  Speaker 3: I think when, if you try to do distillation naively,
• 26:18
  Speaker 3: you end up just sort of matching the teacher on
• 26:21
  Speaker 3: the benchmarking distribution. And But if you don't have enough
• 26:25
  Speaker 3: of the environments that really exercise the capabilities in these trickier,
• 26:31
  Speaker 3: realistic settings, then you're not going to get those into
• 26:34
  Speaker 3: your student model. And I think maybe one thing that's
• 26:37
  Speaker 3: happening is the big models generalize better from the tricky,
• 26:43
  Speaker 3: narrow tasks to these sort of more realistic tasks. So
• 26:48
  Speaker 3: if you have a really good... realistic prompt distribution for distillation,
• 26:53
  Speaker 3: you can match the big model really well. But if
• 26:56
  Speaker 3: you only have this distribution of easily verifiable tasks, then
• 27:02
  Speaker 3: you can match the big model on all the benchmarks,
• 27:05
  Speaker 3: but you do worse on this broader distribution. So that
• 27:11
  Speaker 3: might even explain something about the smaller anthropic models like SONET5,
• 27:17
  Speaker 3: though it's hard to predict exactly what what they're doing
• 27:20
  Speaker 3: to post-train those models. It could also be that they're
• 27:23
  Speaker 3: always changing their post-training stack and they just got a
• 27:28
  Speaker 3: few things wrong in some of these models.
• 27:30
  Speaker 1: So they, I.
• 27:32
  Speaker 3: Don't know, they turned up something too high and created
• 27:37
  Speaker 3: some quirks that people really don't like. So it's really
• 27:40
  Speaker 3: easy to screw up post-training in some way that doesn't
• 27:43
  Speaker 3: show up in benchmarks.
• 27:44
  Speaker 2: I mean, just one other sort of very basic point
• 27:46
  Speaker 2: is just like the AI, like the frontier AI labs
• 27:49
  Speaker 2: buy all their data from their data companies. And like
• 27:51
  Speaker 2: the Chinese can also just buy the same data from
• 27:53
  Speaker 2: data companies.
• 27:53
  Speaker 5: And they are.
• 27:54
  Speaker 2: And like they are, exactly. There's a lot of people like,
• 27:56
  Speaker 2: you know, being annoyed about this, but like if they
• 27:58
  Speaker 2: have exactly the same data and like they can buy that,
• 28:00
  Speaker 2: they can also distill. It's like, it means it's quite
• 28:04
  Speaker 2: easy to like keep up really.
• 28:05
  Speaker 1: Yeah, yeah, yeah.
• 28:06
  Speaker 5: Okay.
• 28:06
  Speaker 1: The other question I had is how the first models
• 28:11
  Speaker 1: that are capable of automating AI R & D will
• 28:13
  Speaker 1: actually be trained Because there's a toy version, which is
• 28:17
  Speaker 1: this thing that Ryan was talking about, which is you
• 28:19
  Speaker 1: just have GPT-8 try to build GPT-3 size models that
• 28:24
  Speaker 1: are really good at like inner loop type challenges of
• 28:27
  Speaker 1: beating video games that require continual learning or just getting
• 28:31
  Speaker 1: to a certain loss with like the least amount of compute,
• 28:34
  Speaker 1: et cetera. But John, I think you had an interesting
• 28:36
  Speaker 1: point that maybe that's not the way it actually will
• 28:38
  Speaker 1: happen in practice. So I'd be curious about, by the
• 28:41
  Speaker 1: point in which you have EIs that are actually capable
• 28:43
  Speaker 1: of automating AR and D, how are they probably trained?
• 28:45
  Speaker 3: Yeah, I think we'll probably do some combination of learning
• 28:48
  Speaker 3: from human feedback to absorb like the researcher's taste and
• 28:53
  Speaker 3: just like creating a lot of practice environments which involve
• 28:57
  Speaker 3: like doing multi-step research projects.
• 28:59
  Speaker 1: So I think.
• 29:02
  Speaker 3: Yeah, people will in practice do some combination of those
• 29:05
  Speaker 3: two things and just each iteration like patch whatever seems
• 29:10
  Speaker 3: to be most broken in the last iteration. So like
• 29:14
  Speaker 3: researchers will be using the AIs a lot and will
• 29:20
  Speaker 3: notice that they have some consistent weaknesses and then those
• 29:23
  Speaker 3: things will either be patched by like collecting human feedback
• 29:26
  Speaker 3: or like creating environments.
• 29:29
  Speaker 5: Yeah, makes sense.
• 29:30
  Speaker 4: Maybe useful to think about this is like how much
• 29:32
  Speaker 4: of the lineage we roll back and then let self
• 29:34
  Speaker 4: play from there. Like I think in the limit like
• 29:37
  Speaker 4: you're picturing like, you know, just giving them like a
• 29:40
  Speaker 4: gpu and maybe neural nets or something and saying like
• 29:42
  Speaker 4: okay figure out how to train a model to like
• 29:45
  Speaker 4: do these particular tasks like the way it currently works
• 29:48
  Speaker 4: is like we go up to the very like edge
• 29:50
  Speaker 4: of the of the lineage and say okay like here
• 29:52
  Speaker 4: are the bugs like you know anthropic has found in
• 29:54
  Speaker 4: their training stack in the last few months we'll turn
• 29:55
  Speaker 4: those into environments like you need to train and get
• 29:57
  Speaker 4: better on the frontier. And so you obviously lock in
• 30:00
  Speaker 4: all the previous history of the lineage, but you could
• 30:02
  Speaker 4: imagine a world in which you roll back to like,
• 30:04
  Speaker 4: you know, before GRPO or something. And then you have
• 30:07
  Speaker 4: environments which like trying to get it to discover like
• 30:09
  Speaker 4: the best form to like RL models on.
• 30:12
  Speaker 5: And then maybe you roll further and further back.
• 30:14
  Speaker 4: But I think we will be still so compute bottlenecked
• 30:16
  Speaker 4: that like people will just keep like staying at the frontier.
• 30:18
  Speaker 6: Yeah.
• 30:19
  Speaker 4: And like diffing essentially the bugs and whatever improvements they
• 30:21
  Speaker 4: found since the last model version, turning those into training environments.
• 30:24
  Speaker 1: Which is also really good for having non-steel, like new
• 30:27
  Speaker 1: data between model generations. It's just, again, this is basically
• 30:31
  Speaker 1: continual learning within the AI lab of distilling the last
• 30:34
  Speaker 1: three months of AI research progress through environments and like
• 30:39
  Speaker 1: RLHF type stuff back into the model itself.
• 30:41
  Speaker 4: And it is distilling, right? And that's maybe why some
• 30:44
  Speaker 4: of us feel like it's asymptotic. It's like you're always
• 30:47
  Speaker 4: like just trying to get the last three months of progress.
• 30:50
  Speaker 4: And that progress is being contributed to by AI, of course,
• 30:52
  Speaker 4: but it also still has humans in the loop. And
• 30:54
  Speaker 4: it feels like, you know, you're just constantly inching closer
• 30:56
  Speaker 4: and closer to what the human researchers are, like, finding
• 30:59
  Speaker 4: and capable of doing.
• 30:59
  Speaker 5: Yeah.
• 31:00
  Speaker 2: I mean, the one thing I will say, though, is, like, obviously,
• 31:02
  Speaker 2: if you're just distilling on, like, trajectories, you can never
• 31:04
  Speaker 2: go above it. But environments can go quite a far
• 31:06
  Speaker 2: way above what the humans can do. Like, it's very
• 31:08
  Speaker 2: easy to design an environment that, like, no human can solve,
• 31:10
  Speaker 2: but the AI can obviously still try and solve it.
• 31:12
  Speaker 2: And so that would be the path to, like, go
• 31:13
  Speaker 2: ahead of just, like, what the human AI research is.
• 31:15
  Speaker 4: Do you have, like, an example of, like, in terms
• 31:17
  Speaker 4: of RSI or, like, you know, working on a train set.
• 31:20
  Speaker 1: But doing it even faster than a human speedrunner.
• 31:22
  Speaker 5: Yeah.
• 31:23
  Speaker 2: I mean, I feel like in AI research, especially, it's
• 31:24
  Speaker 2: very easy to define, like, goals, which, like, you know,
• 31:26
  Speaker 2: you could say, like, the loss needs to be, like, 1.3
• 31:28
  Speaker 2: or something, and, like, no human can get there, you know, now. But, like,
• 31:32
  Speaker 2: that's a very extremely measurable, verifiable task, and if the
• 31:35
  Speaker 2: AI gets there, then great.
• 31:37
  Speaker 1: Right, and building, like, a hundred million parameter model that
• 31:40
  Speaker 1: beats Minecraft, that's maybe too easy, but, like, beats a
• 31:43
  Speaker 1: much more complicated game or something.
• 31:45
  Speaker 4: Isn't it crazy that 100 million permanent models to beat
• 31:48
  Speaker 4: minecraft we're calling that too easy like imagine if you
• 31:50
  Speaker 4: said that like five years ago.
• 31:52
  Speaker 3: I would say a lot of research is not exactly
• 31:54
  Speaker 3: like that though where it's like hill climbing on a
• 31:57
  Speaker 3: well-defined goal it's sort of more like uh here's an
• 32:00
  Speaker 3: intuition we have about uh some way models should be
• 32:03
  Speaker 3: better and then we also have uh some idea for
• 32:06
  Speaker 3: an algorithm that seems to go a little bit in
• 32:09
  Speaker 3: this direction so let's come up with a task that
• 32:13
  Speaker 3: is uh sort of designed to show signs of life
• 32:16
  Speaker 3: on this approach and uh like see if we get
• 32:19
  Speaker 3: some uh get those signs of life and then if
• 32:22
  Speaker 3: we do we can make successively more realistic versions of
• 32:24
  Speaker 3: the task.
• 32:25
  Speaker 1: Right it's like a lot more guided by intuition and
• 32:28
  Speaker 1: then the the out the inner loop is to elicit
• 32:32
  Speaker 1: the or make tests for that intuition rather than like
• 32:37
  Speaker 1: the the the test itself leading to the insight.
• 32:40
  Speaker 3: Right, like you're not directly optimizing for the eventual objective
• 32:44
  Speaker 3: you care about or the practical production objective.
• 32:48
  Speaker 1: It's sort of you're.
• 32:52
  Speaker 3: Relaxing your objective a little bit. You're saying, yeah, let's
• 32:55
  Speaker 3: relax on the realism axis a little bit and find
• 32:58
  Speaker 3: some methods that actually work and then try to get
• 33:01
  Speaker 3: back to realism later after the method matures a little bit.
• 33:04
  Speaker 3: And then there's also more... there's research that's more oriented
• 33:09
  Speaker 3: towards explaining things and, like, developing a theory or a
• 33:14
  Speaker 3: sort of... Yeah, often we don't have, like, mathematical theories
• 33:18
  Speaker 3: in machine learning that are that predictive, but we have, like,
• 33:21
  Speaker 3: a lot of, like, more informal theories for what's going on.
• 33:26
  Speaker 2: Yeah, I mean, like, presumably the models will be trained on, like,
• 33:28
  Speaker 2: some combination of all of these tasks, and, like, some
• 33:30
  Speaker 2: will be very easily verifiable, some will be, like, LMS judge, or, like,
• 33:34
  Speaker 2: just ask the human, like, does this look reasonable?
• 33:36
  Speaker 5: And then you will.
• 33:37
  Speaker 2: The hope would be that these would all generalize to
• 33:40
  Speaker 2: these much sort of hard, sort of more vague, fuzzy
• 33:42
  Speaker 2: kind of tasks. And it probably will to some extent,
• 33:44
  Speaker 2: whether it generalizes enough that the loop can become self-sealing
• 33:48
  Speaker 2: without humans being in the loop at all, it's unclear.
• 33:50
  Speaker 1: Yeah, yeah, yeah. Maybe taking a step back, here's what
• 33:54
  Speaker 1: it seems to me that the plan for AI research
• 33:58
  Speaker 1: going forward is, and you tell me if you think
• 34:00
  Speaker 1: it's going to work or if you agree with this characterization.
• 34:04
  Speaker 1: The bet is that we will scale up our LVR
• 34:07
  Speaker 1: training across millions of diverse environments, across hundreds of different
• 34:11
  Speaker 1: kinds of domains. And what will emerge at the other
• 34:15
  Speaker 1: end is an agent which has learned these basic skills
• 34:19
  Speaker 1: or less than basic skills around being persistent, being able
• 34:23
  Speaker 1: to triage information and context, eventually having end-to-end optimization of
• 34:28
  Speaker 1: working with other agents and things like that. And such
• 34:31
  Speaker 1: an agent will be very sample efficient Within the context,
• 34:35
  Speaker 1: you know, you've done research on how you actually scale
• 34:36
  Speaker 1: up in context learning to make it like arbitrarily long,
• 34:39
  Speaker 1: but you keep scaling it up. And so what comes
• 34:41
  Speaker 1: out the other end will be something that basically functions
• 34:44
  Speaker 1: like a drop in remote worker over the course of
• 34:47
  Speaker 1: a week or a month. First of all, do you
• 34:49
  Speaker 1: agree that that is the bet the labs are making?
• 34:50
  Speaker 1: And second, is that enough? Like basically learning how to
• 34:54
  Speaker 1: learn within the simulacra, within a data center, and then
• 34:58
  Speaker 1: getting deployed into the real world, but not actually like
• 35:00
  Speaker 1: learning from real world deployment, only learning these meta skills
• 35:04
  Speaker 1: from the simulated environments in the data center.
• 35:07
  Speaker 5: Yeah, I think.
• 35:09
  Speaker 4: It's now hard to separate out, like, how much of
• 35:11
  Speaker 4: the lab's effort is going towards, like, direct RSI versus, like,
• 35:15
  Speaker 4: making generally intelligent models that they can continue to deploy
• 35:18
  Speaker 4: to collect revenue to fund the next big training run.
• 35:20
  Speaker 4: I think for the latter, like, yes, that's probably just
• 35:24
  Speaker 4: the bet they're making. Like, and it's very clear, like,
• 35:27
  Speaker 4: the pattern of, like, where these environments are going over
• 35:29
  Speaker 4: the last few years. I mean, like, Anthropic's lineage of
• 35:32
  Speaker 4: environments is, like, a very clear example of this. Like,
• 35:34
  Speaker 4: you know, first, like, we just focus on coding, and, like,
• 35:37
  Speaker 4: we're going to get really, really good at that, and
• 35:38
  Speaker 4: then the task horizon that we've got from coding, which
• 35:42
  Speaker 4: is probably the lowest hanging fruit in terms of, like,
• 35:44
  Speaker 4: data available on the internet to create environments, like, their
• 35:46
  Speaker 4: own internal stuff that they can turn into environments, then
• 35:49
  Speaker 4: we're going to generalize, we're going to go after finance next, and, like, literally, like,
• 35:52
  Speaker 4: just so much Excel data and all that sort of
• 35:54
  Speaker 4: stuff in the RL training, and then, you know, it's PowerPoints, it's, like,
• 35:58
  Speaker 4: this long tail of, like, the working economy, and, like,
• 36:00
  Speaker 4: that seemed to work really well. And like a lot
• 36:02
  Speaker 4: of the other labs and things, even the open source
• 36:04
  Speaker 4: labs have now realized that that was the correct bit.
• 36:06
  Speaker 1: But what is the implication from that? When I had
• 36:08
  Speaker 1: Dario on the podcast, the thing I asked him was,
• 36:10
  Speaker 1: if you truly expect models, which will be human-like in
• 36:12
  Speaker 1: their ability to learn on the job, why would you
• 36:15
  Speaker 1: try to bake in all these skills of like working
• 36:19
  Speaker 1: with PowerPoint or something? Wouldn't you just expect the model
• 36:21
  Speaker 1: to be able to pick that up on while it's deployed?
• 36:24
  Speaker 1: And so, yeah, there's multiple different explanations. One is just
• 36:26
  Speaker 1: that this is We expect models to get there soon,
• 36:29
  Speaker 1: but they're not there yet. So why not amortize these
• 36:32
  Speaker 1: skills into the model training? Another is that we're not
• 36:37
  Speaker 1: concentrated on making it really good at widely deployed work.
• 36:40
  Speaker 1: We just want it really good at RSI. And this
• 36:44
  Speaker 1: is just a way for us to get revenue so
• 36:46
  Speaker 1: that we can pour it back into a model that
• 36:47
  Speaker 1: is actually really good at doing RSI development. And then
• 36:51
  Speaker 1: once the singularity happens, the thing that comes out the
• 36:53
  Speaker 1: other end will be really good at all the things
• 36:55
  Speaker 1: which seem like bottlenecks to the current generation of models. Yeah, John,
• 36:58
  Speaker 1: I don't know if you have a taste on like what, how,
• 37:01
  Speaker 1: once you construe why there is so much task specific
• 37:04
  Speaker 1: knowledge in these models, if the path is like this
• 37:06
  Speaker 1: kind of generalization.
• 37:08
  Speaker 6: Yeah.
• 37:08
  Speaker 3: I mean, if the models were good enough at learning
• 37:10
  Speaker 3: in context, then in theory, you wouldn't be, you wouldn't
• 37:13
  Speaker 3: need to train them on finance. They would just be
• 37:16
  Speaker 3: able to figure out, read all the, all the books
• 37:19
  Speaker 3: on the fly and figure out how to, how to
• 37:22
  Speaker 3: do everything in the appropriate jurisdictions. Yeah, and you could
• 37:27
  Speaker 3: argue that you need to do a lot of this
• 37:30
  Speaker 3: domain-specific training just to make them more efficient. So even
• 37:35
  Speaker 3: if they were smart enough to figure this out on
• 37:37
  Speaker 3: the fly, you still might want to do a bunch
• 37:40
  Speaker 3: of RL and bake all these intuitions into the weights
• 37:45
  Speaker 3: so the model would be more efficient at runtime.
• 37:47
  Speaker 5: Yeah.
• 37:48
  Speaker 3: Yeah, I'd say in practice, it does seem like model
• 37:51
  Speaker 3: providers are going domain by domain and trying to strengthen
• 37:55
  Speaker 3: the models in the highest value domain. And I'd say
• 37:57
  Speaker 3: that that's one of the answers to why the models
• 38:00
  Speaker 3: have gotten so much better. It's just because the model
• 38:03
  Speaker 3: providers have covered a lot of the high value domains
• 38:06
  Speaker 3: and the most common types of skills.
• 38:10
  Speaker 2: I mean, I think another thing is just like, it's
• 38:12
  Speaker 2: not that expensive to do both at the same time, right?
• 38:14
  Speaker 2: Because like the models are massive, they can easily afford
• 38:16
  Speaker 2: in terms of their parameters to like learn everything. And
• 38:18
  Speaker 2: like there is likely some transfer and sort of even just,
• 38:20
  Speaker 2: even if like finance is not specifically like the information
• 38:23
  Speaker 2: is important for like RSI, just the general like meta
• 38:25
  Speaker 2: learning of like how to figure out what's important, how
• 38:27
  Speaker 2: to have taste, how to like do long horizon work
• 38:29
  Speaker 2: is potentially generalizable and like. there's not that much RSI
• 38:33
  Speaker 2: data in the world as well. It's kind of hard
• 38:35
  Speaker 2: to generate, and that requires a lot of effort. So
• 38:37
  Speaker 2: if you can amortize in this other data, get some
• 38:39
  Speaker 2: transfer from it, you already have masses of compute and
• 38:42
  Speaker 2: massive parameter space. So why not do that as well as, obviously,
• 38:44
  Speaker 2: the direct commercial incentive of selling a model?
• 38:47
  Speaker 1: MARK MANDELMANN That makes sense. Oh, yeah.
• 38:49
  Speaker 3: I'll add that there's one question about whether this current
• 38:53
  Speaker 3: paradigm of doing like SIM to real will be the
• 38:57
  Speaker 3: dominant one forever. So basically you look at what the
• 39:00
  Speaker 3: real world tasks are like, and then you try to
• 39:03
  Speaker 3: create a bunch of environments that can be simulated like
• 39:07
  Speaker 3: in the data center and you can do RL on them.
• 39:10
  Speaker 3: And I think obviously this has been very successful, but
• 39:14
  Speaker 3: it has a lot of weaknesses because a lot of
• 39:17
  Speaker 3: things are just kind of hard to simulate, especially if
• 39:20
  Speaker 3: they involve like interacting with a bunch of humans in
• 39:23
  Speaker 3: real time Yeah, so there's some question about whether Sim2Real
• 39:29
  Speaker 3: will be the dominant.
• 39:30
  Speaker 1: framework forever.
• 39:31
  Speaker 2: I think Sim2Real has to be the dominant framework while
• 39:33
  Speaker 2: sample efficiency is kind of low because right now you
• 39:36
  Speaker 2: need thousands and thousands of interactions with the humans and
• 39:38
  Speaker 2: no human is going to sit there and deal with this,
• 39:40
  Speaker 2: basically be in the loop of RL training. And so
• 39:43
  Speaker 2: we kind of have to simulate that now to get
• 39:44
  Speaker 2: the samples you need. But obviously if sample efficiency improves
• 39:47
  Speaker 2: a lot, You'd expect learning from deployment to become a
• 39:50
  Speaker 2: much bigger part of it.
• 39:51
  Speaker 3: Though there are also other things you could do, like
• 39:53
  Speaker 3: you can learn off policy, so you can take all
• 39:56
  Speaker 3: the traces, and even without re-simulating everything, you can potentially
• 40:00
  Speaker 3: learn something from them.
• 40:01
  Speaker 1: Jane Street just launched a new competition and it's their
• 40:04
  Speaker 1: most ambitious one yet. Design a protocol emulator ASIC. Basically,
• 40:08
  Speaker 1: if you have a chip that you want to test,
• 40:10
  Speaker 1: you can connect it to this ASIC and then this
• 40:12
  Speaker 1: ASIC will simulate realistic traffic. That way you can see
• 40:15
  Speaker 1: how the chip responds without having to plug it into
• 40:17
  Speaker 1: a live system. Jane Street is looking for flexible, general
• 40:20
  Speaker 1: purpose designs, not single protocol emulators. When I was chatting
• 40:24
  Speaker 1: with them, they suggested that I start off by trying
• 40:26
  Speaker 1: to implement what are apparently three very common protocols, UART, SPI,
• 40:31
  Speaker 1: and I2C. James Reid also mentioned that they hoped that
• 40:34
  Speaker 1: more ambitious designs will also tackle low-speed USB and Ethernet
• 40:39
  Speaker 1: and any other protocols that flex your chip's specific architecture. Importantly,
• 40:43
  Speaker 1: your design should be reprogrammable rather than smashing a bunch
• 40:47
  Speaker 1: of specific protocols onto a chip. If a new protocol
• 40:50
  Speaker 1: comes out after your ASIC is taped out, your chip
• 40:53
  Speaker 1: still needs to be able to handle it. How exactly
• 40:56
  Speaker 1: it does it is up to you, but there is
• 40:58
  Speaker 1: one hard constraint. Your design must target an open source
• 41:01
  Speaker 1: 130 nanometer process node. That's because Jane Street will pay
• 41:05
  Speaker 1: to tape out the most novel submissions and send the
• 41:07
  Speaker 1: physical copies to the winners. The competition is open till
• 41:11
  Speaker 1: January 18th, 2027, and working in teams is highly encouraged.
• 41:15
  Speaker 1: Go to janestreet.com slash thorcash to download the template code
• 41:20
  Speaker 1: and get started. I want to ask more about this
• 41:23
  Speaker 1: because it's sort of weird that you have 50% of
• 41:27
  Speaker 1: compute that's spent on inference that is not directly helping
• 41:31
  Speaker 1: the model become better. Like one of the key advantages
• 41:33
  Speaker 1: you'd expect eventually digital minds to have is unlike a
• 41:37
  Speaker 1: human who gets to have 50 years of real world experience,
• 41:41
  Speaker 1: a model will get to, through all its instances, will
• 41:43
  Speaker 1: get to experience, I don't know, millions of years of
• 41:47
  Speaker 1: deployment across all kinds of economically relevant work in the economy,
• 41:51
  Speaker 1: And right now that data is just not in a
• 41:53
  Speaker 1: meaningful sense, helping the model get better. Like it just
• 41:56
  Speaker 1: seems so obvious that eventually models should be able to
• 41:59
  Speaker 1: learn from this data. And once they do, you would
• 42:01
  Speaker 1: have something that almost feels like a widely deployed intelligence
• 42:04
  Speaker 1: explosion because the model is assimilating so much information across
• 42:08
  Speaker 1: all these deployed instances. But when do you expect this
• 42:10
  Speaker 1: kind of hive mind, kind of crazy shit to start happening?
• 42:13
  Speaker 2: I think broadly, at a very basic level, this is
• 42:15
  Speaker 2: already happening. Just in the next generation of models. So
• 42:18
  Speaker 2: right now, you can obviously take your deployment data and
• 42:20
  Speaker 2: put this in the pre-train or the mid-train of future models.
• 42:23
  Speaker 2: Especially if you do some kind of filtering or some
• 42:25
  Speaker 2: kind of judgment or annotation or recent synthesization of that.
• 42:28
  Speaker 1: How much do you think that explains the generation over
• 42:30
  Speaker 1: generation improvement?
• 42:31
  Speaker 2: I think it explains quite a bit.
• 42:33
  Speaker 1: I mean, this is....
• 42:35
  Speaker 2: I don't know whether the labs do this because, you know,
• 42:36
  Speaker 2: theoretically they claim not to train on people's data. But, like,
• 42:39
  Speaker 2: the Chinese 100% do. And, like, they definitely get this
• 42:41
  Speaker 2: advantage both, like, obviously deploying. This is basically what distillation is. Like,
• 42:45
  Speaker 2: they take out the models. They get some of their, like,
• 42:47
  Speaker 2: deployment data. They get some fraction of that by, like,
• 42:49
  Speaker 2: pinging the model. And then they train their next generation
• 42:51
  Speaker 2: of models on it. And they can suddenly do it
• 42:52
  Speaker 2: on their own models as well. Like, there's no reason
• 42:54
  Speaker 2: not to whatsoever.
• 42:55
  Speaker 4: I completely agree. Because I think if you zoom out
• 42:57
  Speaker 4: far enough, this is, like, definitely happening. Like, you're picturing this, like,
• 43:01
  Speaker 4: And we're all picturing this as what continual learning, like
• 43:03
  Speaker 4: the Holy Grail is, is this very, very organic live
• 43:06
  Speaker 4: loop of an individual model getting an experience and live
• 43:09
  Speaker 4: updating on the spot and learning from that. And a
• 43:12
  Speaker 4: lot of things break when you zoom into that level
• 43:14
  Speaker 4: of granularity. But yeah, the big labs are doing this.
• 43:16
  Speaker 4: The closed models are doing this. There's also early signs
• 43:19
  Speaker 4: of life of people using open source models doing this
• 43:22
  Speaker 4: at a much faster cadence. So a good example is
• 43:24
  Speaker 4: probably Composer. like you have some sort of model and
• 43:29
  Speaker 4: you are able to, or like, you know, Harvey's doing
• 43:31
  Speaker 4: the same thing with like legal agents. Like it is
• 43:36
  Speaker 4: getting very specific environments from the data that you have
• 43:39
  Speaker 4: for that particular task and things that like, you know,
• 43:41
  Speaker 4: users are complaining about and like all the feedback that
• 43:43
  Speaker 4: you're somehow extracting from like your specific deployments. And a
• 43:47
  Speaker 4: lot of these companies have the advantage over the big
• 43:48
  Speaker 4: labs in that they can use this data really, really well.
• 43:51
  Speaker 4: And then they will create environments. They will like, you know,
• 43:53
  Speaker 4: do a big post train of Kimmy K3, They will
• 43:56
  Speaker 4: go deploy it. They might do some online learning as well,
• 43:58
  Speaker 4: like Composer did online, basically, like, reinforce for a long time. So, yeah, like,
• 44:03
  Speaker 4: there's still a human in the loop. There's still a
• 44:05
  Speaker 4: human saying, okay, these are the signals we care about.
• 44:07
  Speaker 4: Here's how we're going to create environments from the data
• 44:08
  Speaker 4: that we have. And there's, like, still a longer cadence
• 44:11
  Speaker 4: than maybe the one that you're thinking of. But, like,
• 44:13
  Speaker 4: it really is happening. And, like, eventually that loop will become, like,
• 44:15
  Speaker 4: faster and faster.
• 44:16
  Speaker 1: I mean, the Composer thing is interesting because this is
• 44:18
  Speaker 1: where the model, like, in Cursor, people, like, press tab
• 44:22
  Speaker 1: or they don't press tab. on the next completion that
• 44:24
  Speaker 1: the model suggests. And based on that, every single day,
• 44:27
  Speaker 1: Composer gets better at, like, predicting the next.
• 44:29
  Speaker 5: So that was the old tab model.
• 44:31
  Speaker 4: Like, they actually did the same thing for the actual,
• 44:33
  Speaker 4: not just, like, the tab model, but the actual, like,
• 44:35
  Speaker 4: generative model.
• 44:35
  Speaker 1: Oh, that's interesting.
• 44:36
  Speaker 4: And it's hard because when you do online reinforcement learning,
• 44:40
  Speaker 4: you don't have groups, right? You just have one user
• 44:41
  Speaker 4: saying one thing, and then you get one rollout. And so, like,
• 44:44
  Speaker 4: you have a big variance reduction problem.
• 44:46
  Speaker 5: Yeah.
• 44:46
  Speaker 4: And, like, Cursor's kind of fuzzy answer to this was, like, oh,
• 44:49
  Speaker 4: you know, we have very good heuristics which are able
• 44:51
  Speaker 4: to estimate, like, like, how much better than average, like,
• 44:54
  Speaker 4: this response was or how much worse than average this
• 44:56
  Speaker 4: response was. And then they would do, like, this big,
• 44:58
  Speaker 4: you know, like, reinforce update. And then their solution to, like,
• 45:02
  Speaker 4: whether it got worse or not was, like, if it
• 45:04
  Speaker 4: improved on CursorBench, they would deploy the new model, like,
• 45:06
  Speaker 4: every five hours.
• 45:06
  Speaker 5: If it didn't, they would, like, throw that version out.
• 45:08
  Speaker 1: Interesting.
• 45:09
  Speaker 3: Yeah, I think your biggest problem is actually just not
• 45:11
  Speaker 3: knowing what the reward function should be from natural data.
• 45:14
  Speaker 3: And if you use some kind of superficial signal, like,
• 45:17
  Speaker 3: did they accept the the edit, that might get reward
• 45:23
  Speaker 3: hacked in some way.
• 45:24
  Speaker 1: But this seems like a bigger issue with the sim2real thing,
• 45:27
  Speaker 1: where the longer and longer horizon tests get, the harder
• 45:32
  Speaker 1: they are to simulate within a data center, right? It
• 45:34
  Speaker 1: seems to me already, potentially, even in coding, we're getting
• 45:38
  Speaker 1: to the point where there's not some year-long coding task
• 45:43
  Speaker 1: that doesn't eventually require you to talk to a client
• 45:46
  Speaker 1: or interact with the company or interact with users. And
• 45:50
  Speaker 1: if you think about the gamut of things we would
• 45:51
  Speaker 1: want AI to be capable of, you want, eventually, superintelligence
• 45:55
  Speaker 1: to be able to run a business. Or start a
• 45:57
  Speaker 1: new business and make it profitable. Or have a profitable
• 45:59
  Speaker 1: day trading in the markets. Or win a court case.
• 46:02
  Speaker 1: And these are all things which are very hard to
• 46:04
  Speaker 1: simulate in a data center. An inherent part of the
• 46:06
  Speaker 1: learning there is interacting with the real world. And so
• 46:09
  Speaker 1: maybe they may have to learn how to get better
• 46:11
  Speaker 1: at these things from the transfer between SIM to real.
• 46:14
  Speaker 1: But alternatively, maybe you do need weight updates from these
• 46:17
  Speaker 1: kinds of interactions. in order to get better at them.
• 46:21
  Speaker 1: And then if that is the case, if transfer isn't
• 46:23
  Speaker 1: strong enough and you do need weight updates, then the
• 46:25
  Speaker 1: fact that the models are quite sample inefficient is like
• 46:28
  Speaker 1: maybe a deeper problem. And the reason I'm curious about this,
• 46:32
  Speaker 1: I feel like by default, I don't see how you
• 46:35
  Speaker 1: don't get some kind of crazy recursive self-improvement within the
• 46:38
  Speaker 1: next 10 years. But the one reason why that might
• 46:40
  Speaker 1: not happen is in terms of like weight updates, The
• 46:44
  Speaker 1: sample efficiency of weight updates, they just seem way far
• 46:46
  Speaker 1: behind humans, right? Like plausibly millionfold behind humans in terms
• 46:50
  Speaker 1: of how much data a human sees from birth to
• 46:52
  Speaker 1: adulthood versus how much a model sees from, you know,
• 46:55
  Speaker 1: like cold start to like finishing training. And so, yeah,
• 47:00
  Speaker 1: this is all to say, first of all, is there
• 47:02
  Speaker 1: going to be a good transfer between simulations and extremely
• 47:04
  Speaker 1: long horizon, really complicated real shit that we want the
• 47:06
  Speaker 1: AI to do in the real world? And if not,
• 47:08
  Speaker 1: does that really mean that like the lack of sample
• 47:10
  Speaker 1: efficiency in these models comes to bite us?
• 47:13
  Speaker 4: I think maybe the way I'd break down like the
• 47:16
  Speaker 4: two types of tasks in which models get good and
• 47:18
  Speaker 4: models will like still continue to struggle is whether the
• 47:20
  Speaker 4: task is like cumulative or like you kind of have to,
• 47:23
  Speaker 4: you have this like non-stationary distribution you have to keep
• 47:26
  Speaker 4: learning and like relitigating a bunch of stuff. So like
• 47:29
  Speaker 4: maybe an example of a cumulative task might be RSI. Like,
• 47:32
  Speaker 4: It's theoretically possible to maybe have less than a million
• 47:35
  Speaker 4: token Python file, which from scratch trains a model that
• 47:39
  Speaker 4: is capable of recursive self-improvement. And every discovery that you
• 47:42
  Speaker 4: make is kind of a line in the sand that
• 47:44
  Speaker 4: you hold. If it's true that for RSI, we don't
• 47:47
  Speaker 4: need to discover a new attention variant or whatever, once
• 47:49
  Speaker 4: you've discovered attention and then once you've discovered a mixture
• 47:52
  Speaker 4: of experts and once you've discovered JRPO, you just add
• 47:54
  Speaker 4: that to the training stack and that's there.
• 47:56
  Speaker 5: And a good example of this is 5.6 or....
• 47:59
  Speaker 4: Training 5.6 terror or whichever one opening i told it
• 48:02
  Speaker 4: to train like it didn't have to go back and
• 48:04
  Speaker 4: discover attention like it basically probably would have called a
• 48:07
  Speaker 4: bunch of like scripts which is like pre-training.sh and post-training.sh
• 48:10
  Speaker 4: and just did that so like that's an example of
• 48:12
  Speaker 4: like a cumulative task i think the real world and
• 48:15
  Speaker 4: the reason like people are thinking so much about like
• 48:17
  Speaker 4: continuing learning is it's it's not really a cumulative task
• 48:19
  Speaker 4: like imagine like in a law firm you have an
• 48:22
  Speaker 4: agent acting as a legal associate.
• 48:25
  Speaker 5: That's a very non-stationary distribution. You have to be able to.
• 48:28
  Speaker 4: Fit in your context all the relationships between all the
• 48:30
  Speaker 4: important people at that company, which are also changing all
• 48:32
  Speaker 4: the time. You have all these implicit ways about how
• 48:35
  Speaker 4: things are done, where to find information, etc. That's not
• 48:39
  Speaker 4: as clean of an example of a cumulative hassle as
• 48:42
  Speaker 4: RSI is. So, I think that there will be this
• 48:45
  Speaker 4: breakdown between tiles, but you know, like if the labs
• 48:47
  Speaker 4: realize that and they, they do believe that RSI is
• 48:49
  Speaker 4: cumulative in the sense of like, we don't need to
• 48:50
  Speaker 4: go back and discover some brand new like architecture or whatever,
• 48:53
  Speaker 4: then maybe more and more effort and compute gets focused
• 48:56
  Speaker 4: on that versus the.
• 48:57
  Speaker 1: It's so unfortunate that RSI happened to be easier than. Yeah. Um, yeah.
• 49:02
  Speaker 1: I don't know if you guys have thoughts on this.
• 49:04
  Speaker 5: Yeah.
• 49:04
  Speaker 3: I would say there's, uh, like models, um, are today's
• 49:08
  Speaker 3: models are, um, weaker than humans in a lot of
• 49:11
  Speaker 3: different ways. And, uh, Some of them might have to
• 49:13
  Speaker 3: do with sample efficiency in a certain regime where, I mean,
• 49:19
  Speaker 3: in some regimes, models are very sample efficient, like learning
• 49:22
  Speaker 3: in context. But then there might be some like medium
• 49:25
  Speaker 3: length regime where they're less sample efficient because humans can
• 49:29
  Speaker 3: do some kind of weight update more efficiently than models.
• 49:34
  Speaker 3: So I think like being less sample efficient in certain
• 49:38
  Speaker 3: regimes might be one of the sources of weakness, but
• 49:41
  Speaker 3: then I think there are other sources of weaknesses that
• 49:43
  Speaker 3: are completely different than that. For example, uh, having lower
• 49:47
  Speaker 3: diversity of thought than humans or, um, yeah, being bad
• 49:52
  Speaker 3: at certain kinds of long horizon judgments. Uh, I mean,
• 49:55
  Speaker 3: I think a lot of, um, what people call taste is, uh,
• 49:59
  Speaker 3: is something about, um, behavior that, um, works in the
• 50:03
  Speaker 3: long run and that people have realized, uh, works in
• 50:07
  Speaker 3: the long run.
• 50:07
  Speaker 1: Uh, not everything, but like some, some.
• 50:11
  Speaker 3: Aspect of taste, like, especially for something like software engineering,
• 50:14
  Speaker 3: like I think a lot of taste is like, what
• 50:16
  Speaker 3: are the systems that are going to be maintainable and, uh,
• 50:20
  Speaker 3: work well in the long run of this project. So, uh, yeah,
• 50:25
  Speaker 3: I think the weaknesses of humans, which limit, um, RSI
• 50:30
  Speaker 3: along with other things, uh, are, um, yeah, there's a
• 50:34
  Speaker 3: variety of them and some of them are related to
• 50:35
  Speaker 3: sample efficiency and some of them aren't.
• 50:37
  Speaker 4: Maybe an interesting thought experiment is if you were able
• 50:40
  Speaker 4: to give a model a context window of, I don't know,
• 50:43
  Speaker 4: a trillion tokens or whatever you would have needed to
• 50:45
  Speaker 4: fit in your experience prior to, let's say, RLHF. And
• 50:50
  Speaker 4: it's got all that experience in the context window and
• 50:52
  Speaker 4: it has the same sample efficiency and in-context learning ability
• 50:55
  Speaker 4: as it does at a million tokens. Do you think
• 50:58
  Speaker 4: taste is then solved? Would it be able to make
• 51:00
  Speaker 4: the same judgments that you did? Or is there something
• 51:03
  Speaker 4: fundamentally missing apart from just a longer context window with
• 51:05
  Speaker 4: the same sample efficiency?
• 51:07
  Speaker 3: Yeah, I mean, it would have to be trained to
• 51:08
  Speaker 3: learn from that context. So I'm not sure. Yeah, either
• 51:14
  Speaker 3: it would have to be trained to learn the right.
• 51:18
  Speaker 3: update to make from that context.
• 51:21
  Speaker 4: So you don't think you can just dump it all in,
• 51:23
  Speaker 4: your whole life research experience?
• 51:25
  Speaker 2: I mean, you still need the data to train it
• 51:27
  Speaker 2: long context, right? Even if you could theoretically get a
• 51:29
  Speaker 2: trillion context, you would need a trillion lengths of data
• 51:32
  Speaker 2: to train it.
• 51:33
  Speaker 5: Right now, if you have 10K context, you can't dump
• 51:34
  Speaker 5: it in a million. I'm just asking if you had that.
• 51:37
  Speaker 5: I think, yes.
• 51:38
  Speaker 2: I mean, this really just comes down to the question
• 51:40
  Speaker 2: of how meta-learnable is taste from shorter Horizon episodes. And like,
• 51:44
  Speaker 2: I feel like there's no obvious reason it's super long.
• 51:46
  Speaker 2: Cause like humans somehow develop taste with not having many
• 51:48
  Speaker 2: long episodes. Like we don't live to be like 10,000.
• 51:51
  Speaker 2: We have like, you know, we, we develop pretty quickly.
• 51:54
  Speaker 3: Right.
• 51:54
  Speaker 2: And so like, you know, if you think about like,
• 51:55
  Speaker 2: even like in a PhD, the difference between like a
• 51:57
  Speaker 2: first year PhD student and like a final, like postdoc
• 52:00
  Speaker 2: or something, that's like five years maybe. And they've only
• 52:02
  Speaker 2: done like, maybe like 10, 50, 30 research projects in total,
• 52:05
  Speaker 2: but somehow they develop taste quite quickly from like a
• 52:08
  Speaker 2: relatively short succession of like small things. And so like
• 52:10
  Speaker 2: theoretically it's, possible to develop it like that. The AI
• 52:14
  Speaker 2: obviously will have vastly more experience in which to develop taste,
• 52:16
  Speaker 2: like meta-learn it. And there's like, how well does that
• 52:19
  Speaker 2: generalize to like really long horizon things? Is I think
• 52:21
  Speaker 2: the question, which I think is really unsolved at this point.
• 52:23
  Speaker 1: Like we don't know. Going back to this question, eventually
• 52:28
  Speaker 1: there should be a regime where AIs are learning a
• 52:30
  Speaker 1: ton from each individual instance of deployment that they have.
• 52:33
  Speaker 1: Where currently you could say there's a meta fuzzy process
• 52:37
  Speaker 1: by which models do improve for deployment, but I feel
• 52:39
  Speaker 1: like it's a very weak very weak feedback loop. Do
• 52:43
  Speaker 1: you see this around the horizon where there's this like
• 52:45
  Speaker 1: hive mind kind of learning that's very rapid? And if so,
• 52:50
  Speaker 1: how exactly does it happen?
• 52:52
  Speaker 3: Actually, I would say that around will we get a
• 52:55
  Speaker 3: hive mind that learns from all of its deployment experience?
• 52:59
  Speaker 3: I mean, a big part of that is actually about
• 53:01
  Speaker 3: incentives rather than being a technical question. So like companies
• 53:06
  Speaker 3: aren't going to want to have the model provider learn
• 53:09
  Speaker 3: from all of their deployment, because that might just reduce
• 53:14
  Speaker 3: the advantage of their business.
• 53:16
  Speaker 4: I think that maybe the economics of this will pressure,
• 53:20
  Speaker 4: not necessarily weight updates to one big, like common shared model,
• 53:23
  Speaker 4: but like kind of like modules that get subbed in.
• 53:26
  Speaker 4: So like a very obvious example, this is Allura, but
• 53:28
  Speaker 4: it might be something else like, you know, there's been
• 53:30
  Speaker 4: a lot of work to try and fit like an
• 53:32
  Speaker 4: arbitrary context length into a fixed size.
• 53:34
  Speaker 5: Like this is all the linear attention stuff and all
• 53:36
  Speaker 5: that sort of stuff.
• 53:36
  Speaker 4: And like cartridges, which are, essentially KBK is trained to
• 53:40
  Speaker 4: be very, very compressed KBK just to fit in a
• 53:42
  Speaker 4: lot of information. That's another example of like, you know,
• 53:45
  Speaker 4: something that like companies may be willing to sign up
• 53:47
  Speaker 4: for if that's get subbed into the model.
• 53:49
  Speaker 5: And it's not like actually changing the base underlying model itself.
• 53:51
  Speaker 4: So like there's many different versions of like learning from,
• 53:55
  Speaker 4: from your data in real time. And like the latter
• 53:57
  Speaker 4: ones are not really helping the big labs because they
• 53:59
  Speaker 4: are just these modules. But I think the, like the,
• 54:02
  Speaker 4: like the, the economic pressure will force like the labs
• 54:05
  Speaker 4: to go down that path first before they can embark
• 54:07
  Speaker 4: on this.
• 54:08
  Speaker 2: Like, you know, Which economic pressure though? Because I feel
• 54:11
  Speaker 2: like even if you have like a bunch of cartridges
• 54:12
  Speaker 2: or laws or whatnot, you can still just like take
• 54:15
  Speaker 2: all these traces and just like distill this, dump this
• 54:17
  Speaker 2: to the pre-training of like your next generation of bloggers.
• 54:19
  Speaker 4: So it may be a more indirect form of learning
• 54:21
  Speaker 4: that the big labs are getting and that's obviously still
• 54:23
  Speaker 4: really valuable to them. But I can't imagine a world
• 54:26
  Speaker 4: in which we start off with like, you know, we're
• 54:27
  Speaker 4: going to just like directly train this one big model
• 54:29
  Speaker 4: on like all the exact data.
• 54:31
  Speaker 2: No, I think it will definitely like go through stages
• 54:33
  Speaker 2: because I mean, this is assuming there's like one discontinuous
• 54:35
  Speaker 2: event where it's like suddenly we fix like wait updates continuously.
• 54:38
  Speaker 2: And like in practice, I think it's much more likely
• 54:39
  Speaker 2: to be like the cartridges and stuff allow you to
• 54:42
  Speaker 2: specialize in deployment. Then you generate traces, you put that
• 54:44
  Speaker 2: in your model, like three months later, you come out
• 54:46
  Speaker 2: with a model which is better at this stuff. You
• 54:47
  Speaker 2: specialize it again and you like consolidate it again. And
• 54:50
  Speaker 2: then eventually we'll just like make this leap faster and faster.
• 54:52
  Speaker 2: So instead of like every three months, we release a model.
• 54:54
  Speaker 2: Now it's, like, every week and then every, like, day
• 54:56
  Speaker 2: and then every hour, at which point we basically have
• 54:57
  Speaker 2: obviously solved it.
• 54:58
  Speaker 4: Yeah, and I think this is a good point as
• 55:00
  Speaker 4: well because you asked, like, kind of how far off
• 55:03
  Speaker 4: the current paradigm we are from being able to do this.
• 55:06
  Speaker 4: We've done a bit of research on this and people
• 55:07
  Speaker 4: have done a lot of research. Like, at a really
• 55:10
  Speaker 4: large scale, like, when you wash out enough noise and
• 55:12
  Speaker 4: you have large enough patches, like, this outer loop process
• 55:14
  Speaker 4: of like putting data into mid training, creating our own environments.
• 55:17
  Speaker 4: Like it does work in like some sort of continual
• 55:19
  Speaker 4: learning regime. But the problem is like when you zoom
• 55:21
  Speaker 4: in close enough at like a micro level, it's like
• 55:23
  Speaker 4: I've got one model and I'm trying to update it
• 55:25
  Speaker 4: again for like a law firm or something. And I'm
• 55:27
  Speaker 4: trying to do that very continuously, like with a relatively
• 55:30
  Speaker 4: small amount of data, like all the methods kind of
• 55:33
  Speaker 4: break down a bit. So like if I SFT the
• 55:36
  Speaker 4: model on just like, you know, successful traces off policy
• 55:40
  Speaker 4: on policy, Like eventually in the very iterative regime, like
• 55:43
  Speaker 4: when you're doing like, you know, hundreds of these micro updates,
• 55:46
  Speaker 4: you see both catastrophic forgetting, you see forgetting of like
• 55:49
  Speaker 4: previous information I've learned on top of the base model
• 55:51
  Speaker 4: that was much earlier on. And I see degradation of
• 55:54
  Speaker 4: general like use, general capabilities. You know, on policy distillation
• 55:59
  Speaker 4: seems to like push this horizon out a little bit,
• 56:01
  Speaker 4: but it still eventually succumbs to the same thing. Um,
• 56:04
  Speaker 4: and RL is not very good at like, it is
• 56:07
  Speaker 4: good at like getting capabilities in, but it's not as
• 56:09
  Speaker 4: good as getting like knowledge in and like just this
• 56:11
  Speaker 4: very explicit knowledge of like, oh, okay. Like this person
• 56:14
  Speaker 4: does this at this law firm. And like, this is
• 56:15
  Speaker 4: a very specific process we find.
• 56:17
  Speaker 5: And you have to put in a lot of compute
• 56:18
  Speaker 5: to create the right environments to get the knowledge.
• 56:20
  Speaker 1: Do you think that the fundamental issue here, why you
• 56:23
  Speaker 1: get worse at these other skills or there's forgetting and stuff,
• 56:26
  Speaker 1: do you think it's fundamentally an issue of capacity or
• 56:28
  Speaker 1: it's an issue of techniques?
• 56:30
  Speaker 5: A little bit of both.
• 56:31
  Speaker 4: I think SFT and even on-policy distillation can be way
• 56:36
  Speaker 4: too destructive.
• 56:38
  Speaker 5: The reason RL is so nice is because.
• 56:40
  Speaker 4: It changes a very, very small amount about the model
• 56:44
  Speaker 4: and there's a lot of evidence for why this is
• 56:45
  Speaker 4: the case. And so it kind of just tweaks it
• 56:49
  Speaker 4: in this very, very, very small loss value to get
• 56:51
  Speaker 4: it into the right point. But that also then limits
• 56:54
  Speaker 4: what you can do with RL, like how much you
• 56:55
  Speaker 4: can actually change the model.
• 56:56
  Speaker 1: Whereas you're saying the reason this isn't the winner-take-all potentially
• 57:00
  Speaker 1: is that it's just very hard to distill that much
• 57:02
  Speaker 1: information into the base model.
• 57:03
  Speaker 5: Without ruining something in an interface.
• 57:06
  Speaker 2: It's easy to distill it into a different base model.
• 57:08
  Speaker 2: This is why I think it's mostly technique. It's not
• 57:10
  Speaker 2: like It's definitely not like there isn't capacity. If you
• 57:14
  Speaker 2: had some modeling with all this data and you take
• 57:16
  Speaker 2: nearly the same size model and pre-trained it from scratch
• 57:17
  Speaker 2: with all of this stuff in mid-training, it will be better.
• 57:20
  Speaker 2: And I think that's a lot of what's happening today.
• 57:22
  Speaker 2: And so it's very much like there's a bottleneck that
• 57:24
  Speaker 2: stops us from just keeping training the same model forever
• 57:26
  Speaker 2: versus just getting all the data from the old model
• 57:28
  Speaker 2: and training a new model from scratch. And this is
• 57:30
  Speaker 2: exactly what Charlie was saying, some combination of plasticity and
• 57:33
  Speaker 2: catastrophic forgetting. If you just naively train on non-stationary data
• 57:38
  Speaker 2: because you're adding new data as you go. Basically, this
• 57:39
  Speaker 2: is messing with the data distribution, so the old stuff
• 57:42
  Speaker 2: is just forgotten. And we don't really have good methods
• 57:44
  Speaker 2: to stop that from happening.
• 57:45
  Speaker 1: MARK MANDELMANN And so maybe at the limit, you're just
• 57:47
  Speaker 1: bottlenecked by retraining the model from scratch with all this
• 57:49
  Speaker 1: new information.
• 57:50
  Speaker 2: MARK MANDELMANN Yes, which, of course, is very expensive. Training
• 57:53
  Speaker 2: a model from scratch is expensive.
• 57:54
  Speaker 1: MARK MANDELMANN But you're going to do that anyways. I mean,
• 57:55
  Speaker 1: not necessarily.
• 57:56
  Speaker 2: I mean, like, maybe eventually if you have continual learning,
• 57:58
  Speaker 2: you never train a new model. You just, like, just
• 58:00
  Speaker 2: have a model and it keeps learning and, like, expanding, right?
• 58:02
  Speaker 1: But there might be, like, some deep technical reason why
• 58:04
  Speaker 1: that's very difficult because of these, like... I mean, that's
• 58:06
  Speaker 1: the question. That's the question, yeah.
• 58:08
  Speaker 4: I think we have pushed back, like, how much from
• 58:10
  Speaker 4: scratch we need to do. Like, it is definitely possible
• 58:12
  Speaker 4: now to take, like, the pre-trained base and, like, do
• 58:15
  Speaker 4: very good mid-training on top of that, like, kind of
• 58:17
  Speaker 4: continuously plus some RL from, like, different checkpoints that are
• 58:20
  Speaker 4: later on in the training and, like, that's looking more
• 58:23
  Speaker 4: like continual learning but it's certainly not the case of, like,
• 58:25
  Speaker 4: take the most recent model, apply a couple of very
• 58:27
  Speaker 4: small updates, and iteratively never lose it.
• 58:30
  Speaker 1: I'm a bit confused because isn't this literally what happens
• 58:32
  Speaker 1: during training or during post-training or something? You have a
• 58:36
  Speaker 1: model that's already gone through so much training and then
• 58:39
  Speaker 1: you distill some fork that's been further RL'd or something.
• 58:42
  Speaker 1: Isn't that literally what happens?
• 58:44
  Speaker 4: It's still at a large enough scale, I think, that
• 58:46
  Speaker 4: you're washing out a lot of the noise. And you're
• 58:48
  Speaker 4: not just focused on one distribution, which, as Baron said,
• 58:51
  Speaker 4: is like, you know, that is now a very... If
• 58:53
  Speaker 4: you're just focusing on one task, right?
• 58:55
  Speaker 1: Oh, sure. I mean, in the eventual regime, you'd be doing...
• 58:58
  Speaker 1: I don't know, there's billions of deployed instances. You're, like, doing...
• 59:02
  Speaker 1: You're learning from all of them at once, and so
• 59:05
  Speaker 1: hopefully there's some washing out of noise and stuff from that, right?
• 59:08
  Speaker 5: MARK MANDELBACHER- Maybe that scale, yeah. PAUL LEWISOHN- Yeah, I mean.
• 59:11
  Speaker 2: I think definitely, as I was saying, you can do
• 59:13
  Speaker 2: continual mid-training for a long time. And you can roll
• 59:15
  Speaker 2: back to a checkpoint, give any mid-training data. But at
• 59:18
  Speaker 2: the same time, you can't do this indefinitely. If you
• 59:20
  Speaker 2: just keep continual mid-training the same base forever, It just
• 59:23
  Speaker 2: like get, it does, it's sort of asymptote at some point.
• 59:25
  Speaker 2: Like you can't just learn new stuff in that base.
• 59:27
  Speaker 2: And this is why people end up training new bases.
• 59:29
  Speaker 2: Like otherwise you would just keep training the same base forever.
• 59:32
  Speaker 1: Whenever I finish recording an interview, I immediately brain dump
• 59:35
  Speaker 1: all my thoughts into Slack. Things like, what was the
• 59:37
  Speaker 1: most interesting? And what should get cut? This ensures that
• 59:39
  Speaker 1: my editors have all the context they need to start
• 59:42
  Speaker 1: editing the episode. But it's not like these brain dumps
• 59:44
  Speaker 1: have any clear timestamps and my unedited recordings are many
• 59:47
  Speaker 1: hours long. It can take a ton of editor time
• 59:50
  Speaker 1: to even find the exact moments that I was referencing.
• 59:53
  Speaker 1: So we decided to try adding a Grokbot producer to
• 59:57
  Speaker 1: our chat. Now, whenever one of my editors posts a
• 59:59
  Speaker 1: rough cut of the episode, Grokbot opens the transcript on
• 1:00:02
  Speaker 1: its own computer and starts working, usually before I've even
• 1:00:05
  Speaker 1: seen the message. It takes the notes that I dropped
• 1:00:07
  Speaker 1: into Slack and it highlights the relevant snippets in the transcript.
• 1:00:10
  Speaker 1: It also uses a big case file that I've compiled
• 1:00:12
  Speaker 1: with all my preferences, so it can suggest potential edits.
• 1:00:15
  Speaker 1: And when it's done, it sends me its top clip
• 1:00:17
  Speaker 1: candidates so that I can review everything from my phone.
• 1:00:20
  Speaker 1: This has worked really well. Being able to send informal
• 1:00:22
  Speaker 1: messages like I'm texting my editor and then having the
• 1:00:25
  Speaker 1: transcript immediately reflect my preferences has just been so helpful.
• 1:00:29
  Speaker 1: Try Grokbot yourself at x.ai slash bot. Okay, let's talk
• 1:00:34
  Speaker 1: a bit about data now. So I'm generally interested in
• 1:00:37
  Speaker 1: this question of how much of AI progress is just
• 1:00:39
  Speaker 1: explained by data progress. It doesn't mean it will be
• 1:00:43
  Speaker 1: necessarily hard to automate, but that's a separate question. So
• 1:00:46
  Speaker 1: is there some data distribution which if you trained current
• 1:00:51
  Speaker 1: architectures on would result in a super intelligence that totally
• 1:00:56
  Speaker 1: dominates human experts across every single field?
• 1:00:58
  Speaker 5: Are we talking about pre-training plus post-training environments as well?
• 1:01:03
  Speaker 5: I think the existence of this is obvious. It's just
• 1:01:05
  Speaker 5: whether we can create the right environment.
• 1:01:08
  Speaker 2: In the trivial case, we could just train it to
• 1:01:09
  Speaker 2: output the Python file, which trains the actual super intelligence.
• 1:01:12
  Speaker 2: Just have that memorized in the weights.
• 1:01:14
  Speaker 4: Yes, there's probably, like, a ladder of RL environments that
• 1:01:18
  Speaker 4: is possible to construct such that you would get an
• 1:01:22
  Speaker 4: AI researcher which is at least as good as a
• 1:01:25
  Speaker 4: human researcher, but the effort to climb each successive rung grows, like,
• 1:01:30
  Speaker 4: kind of exponentially, and that's going to be the two
• 1:01:34
  Speaker 4: things that you have to trade off against as to whether, like,
• 1:01:36
  Speaker 4: how fast we're going to hit, like, that final rung
• 1:01:38
  Speaker 4: where it's better. I think that's fairly clear, and I
• 1:01:41
  Speaker 4: think there's, like, We're still relatively early in our environment creation.
• 1:01:46
  Speaker 4: There's a lot of asymmetries that we exploit in order
• 1:01:47
  Speaker 4: to create good environments. So one of those asymmetries, which
• 1:01:51
  Speaker 4: we've talked about before, is there's environments where it's easier
• 1:01:55
  Speaker 4: to go backwards than forwards. And what I mean by
• 1:01:57
  Speaker 4: that is it's very easy to define this complex data
• 1:02:00
  Speaker 4: generating process.
• 1:02:01
  Speaker 5: And this is this kind of latent variable you keep
• 1:02:02
  Speaker 5: hidden from the model.
• 1:02:04
  Speaker 4: You can generate arbitrarily complex environments, and the model has
• 1:02:07
  Speaker 4: to do a lot of irreducible token spend and irreducible
• 1:02:11
  Speaker 4: work to figure out what that data generating process was.
• 1:02:14
  Speaker 4: There's asymmetries in terms of like, you know, you can
• 1:02:16
  Speaker 4: inject information from the real world. So like Anthropic finds
• 1:02:18
  Speaker 4: a bug through like, you know, tens of thousands of
• 1:02:21
  Speaker 4: human and LLMLs combined and like turn that into a very,
• 1:02:24
  Speaker 4: very neat environment, which a single LLML could theoretically find
• 1:02:27
  Speaker 4: within like, you know, a few million tokens. So, like,
• 1:02:30
  Speaker 4: there's all these asymmetries which we're cherry-picking and, like, we're
• 1:02:32
  Speaker 4: counting on, like, kind of this task horizon generalization. But
• 1:02:36
  Speaker 4: I think, yeah, again, there's just going to hit diminishing
• 1:02:38
  Speaker 4: returns at some point. Like, at some point, there's diminishing
• 1:02:40
  Speaker 4: returns and how hard it is to create those environments
• 1:02:42
  Speaker 4: in the first place, like, coming up with them because
• 1:02:44
  Speaker 4: you can't necessarily just, like... have these really, like, these
• 1:02:48
  Speaker 4: processes where it's easier to go backwards than forwards. Like,
• 1:02:50
  Speaker 4: you actually have to sit down and construct, like, something
• 1:02:52
  Speaker 4: that looks, like, with humans, like, you know, a long
• 1:02:55
  Speaker 4: enough time horizon, like, it's going to be a really
• 1:02:57
  Speaker 4: complex task to create. And then it's also going to be, like,
• 1:03:00
  Speaker 4: the compute and time bottlenecks for the agent to actually
• 1:03:02
  Speaker 4: do those tasks. So, like, I think you're just going
• 1:03:05
  Speaker 4: to start seeing this, like, curve to flatten out.
• 1:03:07
  Speaker 3: I saw something about how someone fine-tuned the Taki model,
• 1:03:11
  Speaker 3: which is only trained on data up to 1930, on this, like.
• 1:03:15
  Speaker 3: Modern coding agent data, and it did better than Claude
• 1:03:20
  Speaker 3: 3 Opus on SweeBench. So this model that has no
• 1:03:25
  Speaker 3: knowledge of code whatsoever can be fine-tuned on a moderate
• 1:03:28
  Speaker 3: amount of data and behave better as a coding agent
• 1:03:33
  Speaker 3: than this much larger pre-trained model is pretty crazy. And
• 1:03:36
  Speaker 3: it kind of shows you that once you have an
• 1:03:38
  Speaker 3: example of the right expert behavior, it's actually surprisingly easy
• 1:03:43
  Speaker 3: to copy that into a like relatively weak.
• 1:03:47
  Speaker 4: Model yeah but a counter example to like that kind
• 1:03:51
  Speaker 4: of is there was a paper recently where they trained
• 1:03:52
  Speaker 4: it up to like fifth grade maths um and like
• 1:03:55
  Speaker 4: also like primary school like english and stuff so it
• 1:03:58
  Speaker 4: was like a decent language model and they tried to
• 1:04:00
  Speaker 4: rl it to do like you know late high school
• 1:04:02
  Speaker 4: and college maths and the gap was just too large
• 1:04:04
  Speaker 4: like they couldn't get it to climb at all like
• 1:04:06
  Speaker 4: but if you did like successive wrong so like you
• 1:04:08
  Speaker 4: know you're seven maths and then you're eight maths and
• 1:04:10
  Speaker 4: so on like you could obviously climb to year 12. So, like, again,
• 1:04:14
  Speaker 4: it's just, like, what is the distance between the rungs
• 1:04:16
  Speaker 4: on those ladders and how hard is it to create?
• 1:04:18
  Speaker 2: Yeah, and this just comes back to, like, the RL
• 1:04:19
  Speaker 2: signal problem. Like, RL is not very good at, like,
• 1:04:21
  Speaker 2: exploring right now. And so if the model can't, like,
• 1:04:24
  Speaker 2: get in, like, you know, 128 rollouts, it's very unlikely
• 1:04:27
  Speaker 2: to get signal to, like, progress. And this is why, like,
• 1:04:29
  Speaker 2: in RL we need, like, curricula, whereas, like, in pre-training
• 1:04:31
  Speaker 2: we don't because, like, that's not a problem for pre-training
• 1:04:33
  Speaker 2: at all.
• 1:04:34
  Speaker 5: Yeah, and, again, pre-training data is different to post-training data.
• 1:04:39
  Speaker 5: And I imagine.
• 1:04:40
  Speaker 4: as we continue on, like, yeah, humans will be involved
• 1:04:42
  Speaker 4: less and less, but that doesn't change the fact that
• 1:04:44
  Speaker 4: you're bottlenecked on like how much signal you can extract
• 1:04:47
  Speaker 4: from the real world. So like, there's a lot of
• 1:04:50
  Speaker 4: signal in the world and that's true. Like, you know,
• 1:04:51
  Speaker 4: there's people doing like spreadsheet tasks, there's people doing like
• 1:04:55
  Speaker 4: legal tasks and all this sort of stuff. But, you know,
• 1:04:58
  Speaker 4: the capability frontier of where the models are at now,
• 1:05:00
  Speaker 4: like how many bits in the world actually like really
• 1:05:03
  Speaker 4: relevant to like improving the model's capabilities? Like, how many
• 1:05:07
  Speaker 4: new maths problems are being solved that are just beyond
• 1:05:09
  Speaker 4: the reach or grasp of the current models? How many
• 1:05:12
  Speaker 4: new coding problems are being created or solved that are
• 1:05:14
  Speaker 4: beyond the reach of the current models? And I think
• 1:05:16
  Speaker 4: that's why the diminishing returns kicks in because even the
• 1:05:18
  Speaker 4: world as a whole is not giving you the bits
• 1:05:21
  Speaker 4: that are useful for tipping you into the next basin
• 1:05:23
  Speaker 4: of capability.
• 1:05:25
  Speaker 2: Yeah, I totally agree with this. It's really a question
• 1:05:27
  Speaker 2: of where the signal is coming from. And so the
• 1:05:29
  Speaker 2: signal doesn't... In pre-training, the signal is already in Common Core.
• 1:05:33
  Speaker 2: For the tasks that you care about in pre-training, The
• 1:05:36
  Speaker 2: problem is, it's not just getting signal at all, it's
• 1:05:38
  Speaker 2: filtering out all the noise that exists. And that's quite
• 1:05:40
  Speaker 2: an automatable process. But as the models get better, as
• 1:05:43
  Speaker 2: we enter mid-training and post-training, the signal just doesn't exist
• 1:05:46
  Speaker 2: anywhere in the original data we have. No amount of
• 1:05:48
  Speaker 2: filtering will get the... There's no hidden proof of the
• 1:05:52
  Speaker 2: Millennium Prize problem sitting in Common Core. We can just
• 1:05:54
  Speaker 2: filter until we see it. And so at that point,
• 1:05:56
  Speaker 2: you have to get bits some other way, either from
• 1:05:58
  Speaker 2: humans directly asking them to write out their reasoning, or
• 1:06:02
  Speaker 2: by creating environments where humans decide what environment should be created.
• 1:06:05
  Speaker 2: what the objectives of these environments are, or some kind
• 1:06:09
  Speaker 2: of training on the human data that exists in deployment.
• 1:06:11
  Speaker 2: You have to get the bits from somewhere.
• 1:06:13
  Speaker 1: Yeah, yeah. There's a question of how much of the
• 1:06:15
  Speaker 1: progress in pre-training is being driven by data. I did
• 1:06:18
  Speaker 1: this investigation with Jerry Han, who's a student at Princeton,
• 1:06:21
  Speaker 1: where we basically trained all the recipes from 2019 till now,
• 1:06:26
  Speaker 1: pairwise with all the data sets from 2019 to now.
• 1:06:31
  Speaker 1: So you're seeing like GPT-2 on the newest data set,
• 1:06:34
  Speaker 1: like ultra fine web, and you train Delphi, which is
• 1:06:37
  Speaker 1: the newest training recipe or the open source training recipe
• 1:06:39
  Speaker 1: on like the pile or some old data set. And
• 1:06:42
  Speaker 1: you do like the whole grid and you see the
• 1:06:46
  Speaker 1: getting to some level of capabilities, how much less compute
• 1:06:49
  Speaker 1: does it take across this grid? And you see that
• 1:06:54
  Speaker 1: the data seems to explain like nine X of a
• 1:06:57
  Speaker 1: compute efficiency gain, But the architecture improvements explains like a
• 1:07:01
  Speaker 1: 3x compute efficiency gain at a very small scale. And
• 1:07:04
  Speaker 1: so to the extent that that is true at large scale,
• 1:07:07
  Speaker 1: that most of the pre-training compute efficiency gains are coming
• 1:07:09
  Speaker 1: from better data, how much can that continue? Like, can
• 1:07:13
  Speaker 1: you keep just filtering data more and more and building
• 1:07:16
  Speaker 1: more and more synthetic data until, yeah, do you have
• 1:07:19
  Speaker 1: a sense of how much this kind of pre-training progress
• 1:07:21
  Speaker 1: can continue?
• 1:07:22
  Speaker 4: I think my prior is that like, again, the low-hanging
• 1:07:26
  Speaker 4: fruit is, like, somewhat exhausted with, like, we got the
• 1:07:29
  Speaker 4: internet as this big block, and, like, it's not like
• 1:07:32
  Speaker 4: the internet is necessarily, like, growing at the same, like, rate.
• 1:07:36
  Speaker 5: All the useful stuff on the internet is growing at
• 1:07:37
  Speaker 5: the same rate.
• 1:07:37
  Speaker 4: So, like, we've probably got, like, a bunch of, like, 0.1%
• 1:07:40
  Speaker 4: loss drops to go, but, like, definitely not as many
• 1:07:43
  Speaker 4: as have currently occurred. But, like, that's also really interesting that, like,
• 1:07:46
  Speaker 4: you know, you find this, like, what, cumulative, like, 27
• 1:07:48
  Speaker 4: times improvement across both. I think, like, it was Epoch
• 1:07:52
  Speaker 4: or someone who estimated, like, three times a year since 2019,
• 1:07:56
  Speaker 4: which would imply something like three to the seventh, like
• 1:07:58
  Speaker 4: over 2,000 times improvement. So where's that missing 100 times
• 1:08:03
  Speaker 4: or whatever coming from? That probably gives you a good
• 1:08:06
  Speaker 4: signal of how much of this is post-training.
• 1:08:08
  Speaker 1: I think the explanation has to be that a lot
• 1:08:11
  Speaker 1: of the compute efficiency gains are scale dependent. And we're
• 1:08:14
  Speaker 1: studying an extremely small scale. And that raises the question of,
• 1:08:19
  Speaker 1: do the data compute efficiency gains or the algorithmic compute
• 1:08:22
  Speaker 1: efficiency gains have more scale dependence. I don't know if
• 1:08:26
  Speaker 1: you guys were prior on that. We just didn't have
• 1:08:27
  Speaker 1: enough computer to investigate that question. I mean, like, just naively, right?
• 1:08:30
  Speaker 4: Like, theoretically, the scale dependence of the architecture is, like,
• 1:08:35
  Speaker 4: fairly well known.
• 1:08:35
  Speaker 1: Yeah.
• 1:08:36
  Speaker 5: And, like, you can fit a straight line to it.
• 1:08:38
  Speaker 4: Whereas, like, I would have no idea how to do
• 1:08:40
  Speaker 4: that for, like, combining pre-training plus post-training data and mid-training data.
• 1:08:44
  Speaker 2: I don't know. I feel like data is actually more, like,
• 1:08:46
  Speaker 2: more important with scale. Like, I feel like architectures are
• 1:08:49
  Speaker 2: kind of, like, a one-time, like, you know, an architect.
• 1:08:51
  Speaker 2: I feel like Combining, saying just like an X percent
• 1:08:54
  Speaker 2: efficiency gain is kind of misleading, because like what an
• 1:08:56
  Speaker 2: architecture does is like letting you reach like a qualitatively
• 1:08:59
  Speaker 2: new regime which you couldn't reach with the old architecture
• 1:09:00
  Speaker 2: and then within that regime obviously the data is like
• 1:09:03
  Speaker 2: the primary thing determining it but like you know if
• 1:09:05
  Speaker 2: we say didn't have like even like gqa we're doing
• 1:09:07
  Speaker 2: like full attentional day we wouldn't be able to do
• 1:09:09
  Speaker 2: like a million it would be like ridiculous expense to
• 1:09:11
  Speaker 2: do a million context and then like because of that
• 1:09:13
  Speaker 2: we couldn't we could never use the data which is
• 1:09:15
  Speaker 2: like actually at a million context and so we couldn't
• 1:09:17
  Speaker 2: get these capabilities even though like if you just do
• 1:09:19
  Speaker 2: a naive like how much does this do at like
• 1:09:21
  Speaker 2: 2K context where the architecture isn't unlocking anything, then like
• 1:09:24
  Speaker 2: the data, you know, that will look much more important
• 1:09:26
  Speaker 2: than in some sense it is, right? It's unclear to
• 1:09:28
  Speaker 2: me that these things are like really just like multiplicative gains.
• 1:09:30
  Speaker 5: In this way.
• 1:09:32
  Speaker 2: I see.
• 1:09:33
  Speaker 1: Sorry, but then what does this take away for the
• 1:09:34
  Speaker 1: scale dependence of data?
• 1:09:36
  Speaker 2: So, I mean, on scale dependence, I think like, A
• 1:09:39
  Speaker 2: lot of the mid-training and post-training data we have now
• 1:09:42
  Speaker 2: actually gets better with scale. Because a lot of it,
• 1:09:45
  Speaker 2: like the very long context horizon environment stuff, really requires
• 1:09:48
  Speaker 2: big models to be able to make use of them.
• 1:09:50
  Speaker 2: And if you try and train your 100 million parameter
• 1:09:53
  Speaker 2: model on three bench traces, it's not going to get anywhere.
• 1:09:55
  Speaker 2: It's not going to show you the same kind of
• 1:09:57
  Speaker 2: improvement that you would get if you train an actual
• 1:09:59
  Speaker 2: sensible size model on it.
• 1:10:00
  Speaker 4: And it's hard as well now because so many of
• 1:10:03
  Speaker 4: the architecture changes, like you look at Kimi, for instance,
• 1:10:06
  Speaker 4: or DeepSeek, they're doing these architectural modifications with not just
• 1:10:10
  Speaker 4: like dropping the pre-training loss in mind, but like for instance,
• 1:10:12
  Speaker 4: how the models are going to be used in the
• 1:10:14
  Speaker 4: real world. So like the inference efficiency, like having some
• 1:10:17
  Speaker 4: form of compressed attention in the deep seek models is
• 1:10:19
  Speaker 4: not necessarily geared around, you know, this is fundamentally like improvement.
• 1:10:23
  Speaker 5: It's just like, okay, we're considering how the models are
• 1:10:25
  Speaker 5: going to be used.
• 1:10:25
  Speaker 1: Right, right, right. Well, one question I'm curious about to
• 1:10:28
  Speaker 1: understand the future is how parameter scaling will go as
• 1:10:31
  Speaker 1: we're getting into more of a RL heavy regime. Like,
• 1:10:35
  Speaker 1: I don't know. I don't know how fast historically, Yeah,
• 1:10:37
  Speaker 1: you can look at sort of open source architectures and
• 1:10:39
  Speaker 1: see how fast parameters have been scaling. And maybe it's
• 1:10:42
  Speaker 1: like roughly 2x every year for frontier open source models.
• 1:10:46
  Speaker 1: And to the extent that like even frontier closed source
• 1:10:48
  Speaker 1: models have like 100b or 200b active parameters. Do you
• 1:10:52
  Speaker 1: think that like keeps 2x in year over year or
• 1:10:54
  Speaker 1: now that we're in an RL regime where you also
• 1:10:55
  Speaker 1: want to conserve compute on rollouts? And also, maybe there
• 1:11:00
  Speaker 1: is like a threshold effect where you have enough capacity.
• 1:11:02
  Speaker 1: And at that point, increasing parameters arbitrarily doesn't matter as much.
• 1:11:06
  Speaker 1: Do you guys have a sense of In 2030, how
• 1:11:08
  Speaker 1: many active parameters will a frontier model have?
• 1:11:10
  Speaker 4: Yeah, I think for the next few years, we're going
• 1:11:12
  Speaker 4: to be like, because we're so focused on doing longer
• 1:11:15
  Speaker 4: and longer horizon rollouts for RL, where inference efficiency matters
• 1:11:20
  Speaker 4: a lot, it feels like the moles aren't necessarily saturated
• 1:11:24
  Speaker 4: on their ability to do that, where the bottleneck is
• 1:11:25
  Speaker 4: still the environments.
• 1:11:27
  Speaker 5: And so we might see a little bit of a plateau.
• 1:11:29
  Speaker 4: I have a feeling that Mythos and And the GPT
• 1:11:33
  Speaker 4: models are much smaller than the 10 trillion parameter range
• 1:11:35
  Speaker 4: that people are talking about. Even just naively comparing to
• 1:11:39
  Speaker 4: the open source models, you can probably back out that conclusion.
• 1:11:42
  Speaker 4: So yeah, probably for the next few years, I wouldn't
• 1:11:45
  Speaker 4: imagine a huge growth in the number of parameters. But again,
• 1:11:48
  Speaker 4: there's so many different things to trade off here. You
• 1:11:51
  Speaker 4: decide the size of your model based on how much
• 1:11:54
  Speaker 4: pre-training data you have and then the difficulty of the
• 1:11:56
  Speaker 4: RL environments that you've got to train on. And you
• 1:11:59
  Speaker 4: ideally want to get to the optimal point where you
• 1:12:02
  Speaker 4: can get a decent pass at one or something on
• 1:12:05
  Speaker 4: the hardest environments you have. And it wouldn't make sense
• 1:12:07
  Speaker 4: to make a bigger model pass there because then you're
• 1:12:09
  Speaker 4: just paying much more inference slots when.
• 1:12:11
  Speaker 5: You need to. So there's a lot of inputs to this.
• 1:12:14
  Speaker 5: It depends on how quickly McCore and then in-house these
• 1:12:17
  Speaker 5: guys can scale up the complexity of the RL environments
• 1:12:19
  Speaker 5: they're training on.
• 1:12:20
  Speaker 3: I would expect the models to keep getting bigger just
• 1:12:23
  Speaker 3: because people are scaling up compute and the GPUs are
• 1:12:25
  Speaker 3: getting bigger. But I would say exactly how much they
• 1:12:28
  Speaker 3: get bigger depends a bit on the scaling laws in
• 1:12:31
  Speaker 3: non-obvious ways. So one thing is that I think data
• 1:12:36
  Speaker 3: efficiency is going to be a bigger driver than compute
• 1:12:38
  Speaker 3: efficiency of the exact architectures people use. Now that we're
• 1:12:43
  Speaker 3: getting to the regime where we're sort of running low
• 1:12:47
  Speaker 3: on high-quality pre-training data, so that might affect how sparse
• 1:12:51
  Speaker 3: you want to make the model. And then I also
• 1:12:54
  Speaker 3: think we don't understand sparsity that well And it's like
• 1:12:58
  Speaker 3: parameters are a different resource than active parameters. And sparsity
• 1:13:05
  Speaker 3: has definitely increased a bit, but it's not clear that
• 1:13:09
  Speaker 3: it's going to keep increasing without bound. There might be
• 1:13:11
  Speaker 3: some kind of sweet spot. There's an argument that sparsity
• 1:13:15
  Speaker 3: should make data efficiency worse because you might have to
• 1:13:18
  Speaker 3: learn the same thing on multiple experts. So that's debatable.
• 1:13:24
  Speaker 3: So I think we don't... I don't think we... We
• 1:13:27
  Speaker 3: have a good enough theory of scaling laws that we
• 1:13:30
  Speaker 3: really understand why sparsity is helping and how much it'll
• 1:13:34
  Speaker 3: help and if that'll plateau at some point at a
• 1:13:37
  Speaker 3: certain level of sparsity.
• 1:13:39
  Speaker 1: Can you spell out exactly what the implication of data
• 1:13:42
  Speaker 1: efficiency would be? It sounds like you'd say, well, there
• 1:13:45
  Speaker 1: should be less sparsity, but what are the other implications
• 1:13:48
  Speaker 1: on parameter scaling?
• 1:13:49
  Speaker 3: I guess, uh, just that the scaling law, you're, you're
• 1:13:51
  Speaker 3: not necessarily looking for the most, uh, you're not trying
• 1:13:54
  Speaker 3: to optimize computer efficiency. So you have all your, um,
• 1:13:57
  Speaker 3: choices you can make on the architecture and, uh, each
• 1:14:00
  Speaker 3: of these gives you a different scaling law. And then
• 1:14:02
  Speaker 3: like traditionally you would look at some kind of envelope
• 1:14:05
  Speaker 3: based on compute. Uh, so you would look at performance
• 1:14:07
  Speaker 3: versus compute and take the envelope of, uh, like the, yeah, uh,
• 1:14:13
  Speaker 3: the best models. Uh, but, uh, like if we're making
• 1:14:17
  Speaker 3: that decision based on data, so it's like, yeah, we're
• 1:14:20
  Speaker 3: sort of assuming we can spend a lot of compute
• 1:14:25
  Speaker 3: and like we're sort of data is on our X
• 1:14:28
  Speaker 3: axis instead of compute, then we just get a different
• 1:14:35
  Speaker 3: set of Optima or a different set of models that
• 1:14:37
  Speaker 3: are on that frontier. Yeah.
• 1:14:39
  Speaker 4: And I also don't think that we've necessarily like, you know,
• 1:14:42
  Speaker 4: doubled like the size of the models every year for
• 1:14:45
  Speaker 4: the last few years. Like, the people have been training, like,
• 1:14:48
  Speaker 4: 1 trillion parameter models for at least a few years. Like,
• 1:14:50
  Speaker 4: there was even an open-source one called Falcon, but, like,
• 1:14:53
  Speaker 4: Liam from Periodic Labs, like, I think, posted yesterday on
• 1:14:55
  Speaker 4: Twitter about how, like, an early experiment at OpenAI was, like,
• 1:14:58
  Speaker 4: training a 1 trillion parameter model that was very, very sparse.
• 1:15:00
  Speaker 3: Oh, yeah, that was what they did before OpenAI. That was, like,
• 1:15:03
  Speaker 3: at Google, the switch transformer, yeah.
• 1:15:05
  Speaker 4: So, like, it was, like, you know, very, very good at, like, knowledge,
• 1:15:09
  Speaker 4: but terrible at reasoning because it was so sparse. And so, like, yeah,
• 1:15:12
  Speaker 4: it feels like we've been playing in this, like, 100
• 1:15:15
  Speaker 4: billion to up to 2 trillion parameter range for at
• 1:15:18
  Speaker 4: least a little bit.
• 1:15:18
  Speaker 5: And it certainly hasn't been. This is a nice linear increase, yeah.
• 1:15:22
  Speaker 2: I mean, I feel like there's two things. So as
• 1:15:23
  Speaker 2: Charlie was saying, inference efficiency is super important for RL rollouts.
• 1:15:27
  Speaker 2: And so this will really push down active parameters quite
• 1:15:30
  Speaker 2: a lot. And then I think the total parameters really
• 1:15:32
  Speaker 2: depends a lot on the hardware as well. So you
• 1:15:35
  Speaker 2: really need to get very high memory bandwidth and VRAM
• 1:15:38
  Speaker 2: size to actually be able to solve multi-trillion parameter models.
• 1:15:41
  Speaker 2: And so like, you know, right now, you know, people
• 1:15:42
  Speaker 2: still are using a lot of like H100s and stuff.
• 1:15:44
  Speaker 2: And so as everyone moves to GBs and then VARubens
• 1:15:47
  Speaker 2: will get like more actual, like the ability to scale
• 1:15:50
  Speaker 2: and like actually serve and like do like large RL
• 1:15:52
  Speaker 2: inputs at like different, at larger scales. The data question
• 1:15:55
  Speaker 2: I think is interesting because naively like larger models are
• 1:15:58
  Speaker 2: much more sample efficient in like the actual data points.
• 1:16:00
  Speaker 2: And so like, even if you're like not saturating the model,
• 1:16:02
  Speaker 2: it's still better to go bigger because like the models,
• 1:16:05
  Speaker 2: larger models generalize better and like get to a better
• 1:16:07
  Speaker 2: loss for the same amount of data. And so right
• 1:16:09
  Speaker 2: now I think we're kind of have a lot of
• 1:16:11
  Speaker 2: data and like that's not the constraint rather than computing.
• 1:16:13
  Speaker 2: So we're having like smaller models, which are like very insufficient.
• 1:16:16
  Speaker 2: But if computers are no longer the bottleneck, it might
• 1:16:18
  Speaker 2: come back to larger models, which are like sort of under-saturated,
• 1:16:21
  Speaker 2: but like they have this generalization ability because they're much larger.
• 1:16:24
  Speaker 1: If you just look at like the basic Gentile scaling
• 1:16:28
  Speaker 1: law and you just maximize out parameters, it actually decreases
• 1:16:33
  Speaker 1: the amount of data you need to get to the
• 1:16:34
  Speaker 1: same loss very little. If you go to infinity on parameters,
• 1:16:38
  Speaker 1: the amount of data you need, I think, goes down
• 1:16:40
  Speaker 1: less than 10x, just because of the nature of the
• 1:16:43
  Speaker 1: power line.
• 1:16:43
  Speaker 2: But we're now on the way-too-much-data side of the Chinchilla laws, right?
• 1:16:46
  Speaker 2: So right now, we over-train more as we put into Chinchilla,
• 1:16:48
  Speaker 2: and so we could easily get back to a point
• 1:16:50
  Speaker 2: to which, as we're running out of data, we move
• 1:16:51
  Speaker 2: back to the Chinchilla optimal point, or even a bit
• 1:16:54
  Speaker 2: on the over-training, under-training model side.
• 1:16:57
  Speaker 4: But surely, even with these new chips that come online
• 1:16:59
  Speaker 4: and stuff, like we're just going to be so compute
• 1:17:01
  Speaker 4: bottlenecked for the next few years that that won't necessarily.
• 1:17:03
  Speaker 1: Be the case.
• 1:17:03
  Speaker 2: I mean, this could well, yeah, this depends on, like,
• 1:17:05
  Speaker 2: the ratio you have, like, training and inference to compute, really.
• 1:17:07
  Speaker 2: It's like, if you're super bottlenecked on data and not
• 1:17:09
  Speaker 2: on compute, you should go bigger. If you're super bottlenecked
• 1:17:11
  Speaker 2: on compute, you should always go smaller.
• 1:17:13
  Speaker 5: And then, like, yeah.
• 1:17:14
  Speaker 2: But you can also use computer-generated synthetic data, so it's, like,
• 1:17:17
  Speaker 2: one of these very hard things to predict.
• 1:17:19
  Speaker 3: Yeah, I think part of the reason it took people
• 1:17:22
  Speaker 3: so long to figure out the scaling laws in the
• 1:17:23
  Speaker 3: first place was that if you don't get all these
• 1:17:25
  Speaker 3: things right, then you don't get such a clean relationship.
• 1:17:28
  Speaker 1: And, like, the.
• 1:17:30
  Speaker 3: The beautiful straight lines on graphs hide a lot of
• 1:17:33
  Speaker 3: complexity on how you have to make sure to scale
• 1:17:37
  Speaker 3: every hyperparameter the right way or parametrize your optimizer in
• 1:17:41
  Speaker 3: a way that scales and where you don't have to
• 1:17:44
  Speaker 3: change your hyperparameters as you change.
• 1:17:46
  Speaker 5: The model size.
• 1:17:46
  Speaker 4: And bugs have their own clean scaling laws as well, right?
• 1:17:49
  Speaker 4: Like with Kaplan, forgetting the cosine annealing thing or even
• 1:17:53
  Speaker 4: just not considering embedding parameters, I think. And so that
• 1:17:57
  Speaker 4: messed up the estimate at smaller models because embedding parameters
• 1:18:00
  Speaker 4: are a decent size of the model.
• 1:18:03
  Speaker 1: A bit on RL. So I feel like a year ago,
• 1:18:07
  Speaker 1: a lot of people were making this argument that RL
• 1:18:11
  Speaker 1: will not be super successful at scaling for models. I think, John,
• 1:18:14
  Speaker 1: you wrote a research paper where you were pointing out
• 1:18:17
  Speaker 1: that models learn one bit per episode. When you RL,
• 1:18:21
  Speaker 1: they basically learn, did I get the answer right or
• 1:18:22
  Speaker 1: did I get it wrong? And I wrote some blog
• 1:18:25
  Speaker 1: posts earlier this year where I was like, It's even
• 1:18:26
  Speaker 1: worse than that because when the pass rate is low
• 1:18:28
  Speaker 1: and the models are very unlikely to get the answer right,
• 1:18:30
  Speaker 1: it learns almost nothing at all from an RL episode.
• 1:18:35
  Speaker 1: But I look at the models today, and they seem
• 1:18:38
  Speaker 1: pretty smart, and it seems to be the result of
• 1:18:40
  Speaker 1: scaling up RL. Baron, you had a post, I think,
• 1:18:42
  Speaker 1: a few weeks ago where you were trying to explain
• 1:18:44
  Speaker 1: what's going on. But why has RL been more successful
• 1:18:47
  Speaker 1: than one would have naively thought?
• 1:18:50
  Speaker 2: I mean, so I think the success of RL comes
• 1:18:52
  Speaker 2: down to a bunch of different things. So first, I
• 1:18:54
  Speaker 2: think what is slightly underestimated is actually the mid-training. So
• 1:18:57
  Speaker 2: an awful lot of what we see as successes of
• 1:18:59
  Speaker 2: RL actually comes from very, very good mid-training data, which
• 1:19:02
  Speaker 2: is basically where we're essentially doing pre-training, but on synthetic
• 1:19:06
  Speaker 2: reasoning data and the kind of environments that get the
• 1:19:09
  Speaker 2: model warm started for RL. And so this actually takes
• 1:19:11
  Speaker 2: the model almost 80% of the way to the final
• 1:19:13
  Speaker 2: RL checkpoint often. And then what RL does on top
• 1:19:16
  Speaker 2: of that is it does a lot of essentially tweaking
• 1:19:19
  Speaker 2: to the policy. This is one of the reasons why
• 1:19:21
  Speaker 2: it doesn't need as many bits as you would naively think.
• 1:19:23
  Speaker 2: It doesn't have to learn all of these behaviors from scratch.
• 1:19:26
  Speaker 2: It needs just a few bits from these episodes, which
• 1:19:28
  Speaker 2: you do get. And then the other thing that I
• 1:19:30
  Speaker 2: really point out in my blog is that these bits
• 1:19:33
  Speaker 2: are actually extremely high signal compared to regular pre-training, which
• 1:19:36
  Speaker 2: is why you need RL at all versus just SFTing
• 1:19:39
  Speaker 2: on successful reasoning traces.
• 1:19:40
  Speaker 1: Because it's exactly the bits about how to get the
• 1:19:43
  Speaker 1: answer right. Well, there's two things.
• 1:19:44
  Speaker 2: So yes, one, it's exactly the bits about how to
• 1:19:46
  Speaker 2: get the answer right. But this is not exactly how
• 1:19:49
  Speaker 2: you think of it because in SFT, you have a trace, right?
• 1:19:52
  Speaker 2: You have a bunch of math reasoning and then the
• 1:19:54
  Speaker 2: answer at the end. the bit is still there, like
• 1:19:56
  Speaker 2: it's still SFT on the answer token. So that bit
• 1:19:58
  Speaker 2: is still there. What's important is that the objective ignores
• 1:20:00
  Speaker 2: all the other bits. So in SFT, you like have like,
• 1:20:03
  Speaker 2: you know, to try and match like the exact reasoning
• 1:20:05
  Speaker 2: tokens that the model produces. So you're essentially getting like
• 1:20:08
  Speaker 2: too many bits about like the exact way this other
• 1:20:10
  Speaker 2: model you're training on reasons. For RL, you only get
• 1:20:13
  Speaker 2: the one bit. And that means that that signal is
• 1:20:16
  Speaker 2: not drowned out in the noise of all the other
• 1:20:17
  Speaker 2: bits the model has. And so that's what really, it's
• 1:20:20
  Speaker 2: really a super dramatic increase into the signal-to-noise ratio during training,
• 1:20:24
  Speaker 2: which is why RL is so dramatically efficient in terms
• 1:20:26
  Speaker 2: of steps.
• 1:20:27
  Speaker 1: I don't know if you guys have thoughts on that.
• 1:20:30
  Speaker 4: Yeah, there's been so much debate about what RL does
• 1:20:33
  Speaker 4: to the model versus mid-training or SOT or whatever. And like,
• 1:20:38
  Speaker 4: you know, everyone talks about how, you know, parser one
• 1:20:39
  Speaker 4: will go up, but parser 256 will go down. Like,
• 1:20:42
  Speaker 4: very rare correct reasoning traces will be, like, downweighted and
• 1:20:45
  Speaker 4: kind of, like, outweighed by a gradient signal from, like.
• 1:20:47
  Speaker 5: Easier kind of reasoning traces.
• 1:20:49
  Speaker 4: And I think the simple, like, way to view RL
• 1:20:51
  Speaker 4: now is that if you have a large enough, like,
• 1:20:54
  Speaker 4: a large enough amount of compute to sample a large
• 1:20:56
  Speaker 4: enough group size such that your probability of getting a
• 1:21:00
  Speaker 4: bunch of correct answers is, like, past some, like, not
• 1:21:03
  Speaker 4: insignificant probability, then, like, it will be upweighted. And, like,
• 1:21:06
  Speaker 4: to Veran's point, like, basically mid-training and more pre-training, like
• 1:21:12
  Speaker 4: the pass at one, the starting point for RL, like
• 1:21:14
  Speaker 4: scales in the log number of pre-training tokens.
• 1:21:16
  Speaker 1: You can ask very basic questions. I guess that answer
• 1:21:20
  Speaker 1: makes sense, and maybe there's empirical research which shows that
• 1:21:25
  Speaker 1: this is what's happening. But then I just look at
• 1:21:28
  Speaker 1: the models themselves, and I don't know what's happened. Maybe
• 1:21:33
  Speaker 1: you can give me a sense of what is the
• 1:21:34
  Speaker 1: basis of the AI progress over the last year. But
• 1:21:37
  Speaker 1: if it's... Yeah, maybe it's just upweighting the policies, which
• 1:21:44
  Speaker 1: we're going to do the correct thinking anyways. But it
• 1:21:46
  Speaker 1: just seems like qualitatively, the models have gotten so much
• 1:21:49
  Speaker 1: more capable. And anyways, maybe there's nothing to... There's no
• 1:21:54
  Speaker 1: inherent contradiction there. But how do we square the relatively...
• 1:21:59
  Speaker 1: Small impact this take would imply that RL would have
• 1:22:02
  Speaker 1: from the actual qualitative capabilities the model seemed to be gaining.
• 1:22:05
  Speaker 2: So, like, one thing I want to point out here
• 1:22:07
  Speaker 2: is that, like, it doesn't necessarily imply that RL has
• 1:22:09
  Speaker 2: a small, like, effect, right? Even if you have a
• 1:22:12
  Speaker 2: few bits and, like, you only change the parameters a
• 1:22:13
  Speaker 2: small amount, like, the actual impact on, like, function space
• 1:22:16
  Speaker 2: the model lands, like, the input to output mapping can
• 1:22:18
  Speaker 2: still be, like, super dramatic. You know, even if it's, like, even, like,
• 1:22:20
  Speaker 2: one bit can change, like, your function space a lot
• 1:22:22
  Speaker 2: and it can, like, rule out, like, half the hypothesis space,
• 1:22:25
  Speaker 2: which is huge. So, like, I don't think it's necessarily
• 1:22:26
  Speaker 2: the case it's, like, small amounts of bits, small amounts
• 1:22:28
  Speaker 2: of RL, once you're starting from a really good point,
• 1:22:31
  Speaker 2: means that you don't have dramatic impacts in behavior. At
• 1:22:33
  Speaker 2: least not necessarily.
• 1:22:34
  Speaker 5: I think it comes down to two things.
• 1:22:36
  Speaker 4: I think the first thing is that everyone was hoping
• 1:22:38
  Speaker 4: that RL would generalize this reasoning across all these different domains.
• 1:22:42
  Speaker 4: And I don't think we necessarily got this horizontal generalization.
• 1:22:46
  Speaker 4: Just training on math doesn't necessarily make you the greatest coder.
• 1:22:49
  Speaker 4: You do have to do RL.
• 1:22:50
  Speaker 5: On code environments.
• 1:22:51
  Speaker 4: I think what we did get, though, is horizon generalization.
• 1:22:55
  Speaker 4: The models just learned how to use more tokens for
• 1:22:59
  Speaker 4: longer and still make progress on some sort of task.
• 1:23:01
  Speaker 4: And so you can train on environments where they get
• 1:23:04
  Speaker 4: longer and longer and longer and then put them into
• 1:23:06
  Speaker 4: a completely new environment. And yes, they may not have
• 1:23:08
  Speaker 4: generalized the reasoning patterns which allow them to do well
• 1:23:10
  Speaker 4: in that environment, but they've at least generalized the ability
• 1:23:12
  Speaker 4: to continue on that task for longer, which is correlated
• 1:23:15
  Speaker 4: with success. I think there's a paper called Edge Bench
• 1:23:18
  Speaker 4: which showed that the rate at which models can work
• 1:23:21
  Speaker 4: for longer is, like, doubling every three months. And so
• 1:23:23
  Speaker 4: that's a clear evidence of generalization. And I think, like,
• 1:23:25
  Speaker 4: the final way to think about it is, like, in pre-training,
• 1:23:29
  Speaker 4: there's this idea of, like, quanta. So you have this
• 1:23:31
  Speaker 4: very smooth, like, pre-training loss curve. And when you actually
• 1:23:35
  Speaker 4: look at what's happening in the model, like, the model
• 1:23:37
  Speaker 4: is learning all these, like, very discrete, like, tasks. And there's, like,
• 1:23:39
  Speaker 4: all these, like, emergent points. There's, like, kind of a
• 1:23:42
  Speaker 4: phase transition. Like, it didn't have induction heads. Now it
• 1:23:45
  Speaker 4: has induction heads. And there's, like, tens of thousands, millions, probably, like,
• 1:23:48
  Speaker 4: hundreds of millions of these things. And you average them
• 1:23:50
  Speaker 4: all together and you get this very smooth loss curve.
• 1:23:53
  Speaker 4: I think to an extent, a similar thing is happening
• 1:23:56
  Speaker 4: for RL. There is this very slow outer loop, as
• 1:23:59
  Speaker 4: Baron mentioned, of we will train a model and then
• 1:24:01
  Speaker 4: RL it, and then the next model iteration of training,
• 1:24:05
  Speaker 4: we will dump a bunch of these synthetic reasoning traces
• 1:24:07
  Speaker 4: into the mid-training data. We're hitting all these quanta for
• 1:24:10
  Speaker 4: all these different tasks. On an individual task level, it
• 1:24:14
  Speaker 4: may look like a phase transition and you're suddenly going
• 1:24:16
  Speaker 4: from a 0.5% pass rate to a 90% pass rate
• 1:24:19
  Speaker 4: on a particular finance task or Excel task or whatever,
• 1:24:23
  Speaker 4: but you average all these things together and plus the
• 1:24:25
  Speaker 4: horizon generalization, you kind of would go, wow, we've got
• 1:24:28
  Speaker 4: qualitatively better models.
• 1:24:29
  Speaker 2: Yeah. I mean, I think a lot of this as
• 1:24:31
  Speaker 2: well is just like, I think RL does generalize a bit.
• 1:24:34
  Speaker 2: Suddenly you get some transfer between math and code or
• 1:24:36
  Speaker 2: puzzles and math and this kind of stuff. Also, just
• 1:24:39
  Speaker 2: the sheer amount of environments I think the people are
• 1:24:41
  Speaker 2: targeting is just vastly greater. So before, when you tried
• 1:24:44
  Speaker 2: to do some task, which you do in your daily life,
• 1:24:47
  Speaker 2: two years ago, the labs wouldn't really care about this.
• 1:24:49
  Speaker 2: They wouldn't train the model for it. And now, it's
• 1:24:51
  Speaker 2: just so much broader. They have a lot of environments
• 1:24:53
  Speaker 2: targeting this specific thing.
• 1:24:55
  Speaker 1: Earlier in the conversation, we were talking about RL in
• 1:24:56
  Speaker 1: the context of causing this entropy collapse or just concentrating
• 1:25:01
  Speaker 1: probability on solutions the base model had already done and
• 1:25:05
  Speaker 1: causing relatively sparse updates in the policy. But when I
• 1:25:09
  Speaker 1: think about.
• 1:25:11
  Speaker 1: I think there's also another story about RL, which is
• 1:25:16
  Speaker 1: going back to the Atari games and then AlphaGo coming
• 1:25:19
  Speaker 1: up with Move 37, the super creative move, that because
• 1:25:21
  Speaker 1: it was never initialized on human data, it can think
• 1:25:24
  Speaker 1: in ways that humans are not even thinking and come
• 1:25:26
  Speaker 1: up with extremely creative solutions. Do you have a sense
• 1:25:30
  Speaker 1: on when we should expect or if we should expect
• 1:25:32
  Speaker 1: RL on LLMs to result in things like Move 37,
• 1:25:36
  Speaker 1: just extreme creativity, even beyond human creativity, because there's just
• 1:25:43
  Speaker 1: de novo initialization of intelligence?
• 1:25:45
  Speaker 2: I mean, so a couple of things here, like first off,
• 1:25:47
  Speaker 2: I think that the AlphaGo is using MCTS, which obviously
• 1:25:50
  Speaker 2: does like more exploration and like stuff than regular policy gradients.
• 1:25:54
  Speaker 2: But I kind of also think that like RL doesn't
• 1:25:56
  Speaker 2: necessarily like reduce the creativity. And like, I mean, even
• 1:25:59
  Speaker 2: if we look, I think, you know, this is obviously qualitative,
• 1:26:01
  Speaker 2: but if we look at like the, you know, the
• 1:26:03
  Speaker 2: open air hugging phase incident, like these models were coming
• 1:26:05
  Speaker 2: up with like multiple zero days at a time to
• 1:26:07
  Speaker 2: like break out of the sandbox. And like, this is
• 1:26:09
  Speaker 2: clearly like some level of like move 37 creativity, I
• 1:26:12
  Speaker 2: think already. which we just get from just like the
• 1:26:14
  Speaker 2: general generalization properties of the LMs. Like, I don't think
• 1:26:17
  Speaker 2: it's definitely not the case of like RLs, like totally
• 1:26:19
  Speaker 2: destroying their like entropy.
• 1:26:20
  Speaker 6: Yeah.
• 1:26:21
  Speaker 2: Especially on long horizons.
• 1:26:23
  Speaker 1: Yeah.
• 1:26:23
  Speaker 3: I mean, one thing that people call creativity is just
• 1:26:26
  Speaker 3: solving hard search problems. So, and so that's like, like
• 1:26:31
  Speaker 3: move 37 is obviously an example of that, or like
• 1:26:35
  Speaker 3: writing some kind of poem that satisfies a ton of
• 1:26:37
  Speaker 3: different constraints. So that's something AI is obviously going to
• 1:26:41
  Speaker 3: be extremely good at. if trained for it. Then there's
• 1:26:47
  Speaker 3: another way in which the models, like the diversity of
• 1:26:51
  Speaker 3: their outputs is a lot lower after RL and they
• 1:26:55
  Speaker 3: sort of develop these ticks. And like, even though the
• 1:26:58
  Speaker 3: models seem like they're good at writing, when you do
• 1:27:00
  Speaker 3: some kind of like distributional analysis, you find that like
• 1:27:04
  Speaker 3: they're reusing certain themes like all the time and they're
• 1:27:10
  Speaker 3: using the same character names all the time. So there's actually, um,
• 1:27:13
  Speaker 3: it's not like you're getting the same kind of diversity
• 1:27:15
  Speaker 3: that you get when you, like from human authors, you're
• 1:27:18
  Speaker 3: sort of getting one really good, uh, like style. So
• 1:27:21
  Speaker 3: I think that, um, like that kind of diversity has
• 1:27:24
  Speaker 3: definitely been like cut.
• 1:27:27
  Speaker 5: Down by RL a lot.
• 1:27:28
  Speaker 3: And in fact, uh, now, oh yeah, since we were
• 1:27:31
  Speaker 3: talking about distillation earlier, that's sort of, uh, something, yeah,
• 1:27:36
  Speaker 3: one thing that's happening is that so many people are distilling, uh,
• 1:27:40
  Speaker 3: mostly from Claude that like, like all the open weight models, right?
• 1:27:43
  Speaker 3: The same way as Claude and use the same, like
• 1:27:46
  Speaker 3: have the same ticks. So this seems kind of concerning
• 1:27:49
  Speaker 3: to me that we're having this, like this monoculture emerge.
• 1:27:54
  Speaker 2: Again, I don't think this is like fundamental to RLs,
• 1:27:57
  Speaker 2: like a method though. And same with distillation, like even
• 1:27:59
  Speaker 2: with distillation, like you're just training on the data. It's like,
• 1:28:01
  Speaker 2: just because your data is not like super broad, that
• 1:28:04
  Speaker 2: doesn't mean like the training method itself is somehow wrong.
• 1:28:06
  Speaker 5: It's like a problem with the data.
• 1:28:07
  Speaker 2: And I think a lot of instance, like, The RL
• 1:28:10
  Speaker 2: entropy collapse is basically due to exploitation of fairly simple
• 1:28:14
  Speaker 2: verifiers when you don't have a huge diversity of environments.
• 1:28:17
  Speaker 2: Because for instance, the writing, I think, the writing is
• 1:28:19
  Speaker 2: presumably graded by some judge and the judge has some
• 1:28:21
  Speaker 2: specific text and the model is learning to award hack
• 1:28:24
  Speaker 2: the judge and that's why it collapses. But this is
• 1:28:26
  Speaker 2: really a problem with the judge, it's not a problem
• 1:28:28
  Speaker 2: with RL in general.
• 1:28:29
  Speaker 1: Okay, super rapid fire predictions about the future. So I
• 1:28:34
  Speaker 1: want timelines on the following couple questions. By when do
• 1:28:39
  Speaker 1: we have models which you can, here's what it feels
• 1:28:45
  Speaker 1: like to a user. You basically hire them as a
• 1:28:48
  Speaker 1: drop-in remote worker for all kinds of white collar work,
• 1:28:53
  Speaker 1: not just coding, but I don't know, video editing, law, paralegal, etc.,
• 1:28:59
  Speaker 1: like it's like literally an actual remote worker, but like
• 1:29:02
  Speaker 1: full computer use with like literally a month of seamless
• 1:29:05
  Speaker 1: learning and operation and executing on like complex projects and
• 1:29:10
  Speaker 1: it required interacting with other people, et cetera, et cetera.
• 1:29:12
  Speaker 1: It's like everything a human worker could do over a month.
• 1:29:15
  Speaker 4: If you like mandated to use like a browser or whatever,
• 1:29:17
  Speaker 4: rather than like these, again, the firm setting up the
• 1:29:20
  Speaker 4: information to be like programmatically accessible, like maybe a couple
• 1:29:23
  Speaker 4: of years, but if it's not like browser-based, like it
• 1:29:26
  Speaker 4: can send Slack messages, it can do all this stuff,
• 1:29:28
  Speaker 4: but I'd still probably say around a year.
• 1:29:30
  Speaker 2: Yeah, I mean, I would say maybe like for the
• 1:29:32
  Speaker 2: like full generality, maybe like three years, but I think
• 1:29:36
  Speaker 2: to Charlie's point, we will end up with like a
• 1:29:37
  Speaker 2: lot of people like making their organizations easier for the
• 1:29:40
  Speaker 2: AIs to use. And so you get like 80, 90%
• 1:29:43
  Speaker 2: of the way there before that.
• 1:29:45
  Speaker 1: Sorry, but the thing that's, the diff between one year
• 1:29:47
  Speaker 1: and three years, there was just literally like.
• 1:29:49
  Speaker 2: Like, I think there's going to be like a long
• 1:29:51
  Speaker 2: tail of like miscellaneous stuff, which like some human can do,
• 1:29:53
  Speaker 2: which like will take them all.
• 1:29:54
  Speaker 5: It's like quite a while to do.
• 1:29:55
  Speaker 1: Are you thinking of sort of computer stuff or like
• 1:29:57
  Speaker 1: basic cognitive capabilities?
• 1:29:59
  Speaker 2: I mean, I think this really comes down to a
• 1:30:01
  Speaker 2: question of how quickly can we solve this kind of
• 1:30:03
  Speaker 2: online learning and whether we can get 80%, 90% of
• 1:30:07
  Speaker 2: the way there with compaction and writing files to yourself
• 1:30:09
  Speaker 2: and stuff. And that's my big uncertainty.
• 1:30:11
  Speaker 1: I really don't know.
• 1:30:13
  Speaker 4: And maybe an example of something that I wouldn't be
• 1:30:15
  Speaker 4: good at is if I have to yell at someone
• 1:30:17
  Speaker 4: to get something at work or really push someone to
• 1:30:20
  Speaker 4: get something done, the model isn't just going to do that.
• 1:30:22
  Speaker 5: It's just going to be too nice. Yeah.
• 1:30:24
  Speaker 3: I'd say there's a wide variation in quality of, uh,
• 1:30:27
  Speaker 3: human remote workers. So if you, if you try to
• 1:30:30
  Speaker 3: hire someone like off of Upwork to do a software
• 1:30:32
  Speaker 3: engineering project, there's going to be a huge variation. It's
• 1:30:35
  Speaker 3: like often quite hard to get them to do like,
• 1:30:38
  Speaker 3: to do a good job or, uh, like pay attention
• 1:30:40
  Speaker 3: to all the feedback you're getting. And, uh, like I
• 1:30:44
  Speaker 3: would guess that in some cases, uh, like it'll be worse.
• 1:30:47
  Speaker 3: Like the, the pre AI, um, version of this, uh,
• 1:30:50
  Speaker 3: was worse than what you can get now from existing AI. Uh,
• 1:30:54
  Speaker 3: So I think it might end up being a little
• 1:30:57
  Speaker 3: complicated because maybe to some extent we already have this
• 1:31:01
  Speaker 3: like for some like not so high quality of work,
• 1:31:04
  Speaker 3: but then like, then it's obviously like, we're not, yeah,
• 1:31:08
  Speaker 3: we're not matching human level in certain like higher quality,
• 1:31:11
  Speaker 3: like forms of work. So, but I basically agree with
• 1:31:16
  Speaker 3: Charlie and Baron that maybe, yeah, we'll, yeah, we'll have
• 1:31:19
  Speaker 3: some version of this in a year.
• 1:31:20
  Speaker 1: So that's like, okay.
• 1:31:21
  Speaker 3: And it will be able to do, maybe we'll have
• 1:31:24
  Speaker 3: that form factor and it'll be able to do some
• 1:31:28
  Speaker 3: things really well, some things not so well, and things
• 1:31:31
  Speaker 3: will be improving from there.
• 1:31:33
  Speaker 4: Like, we ship the goalposts based on the very long
• 1:31:34
  Speaker 4: tail all the time. Like, I feel like you've used
• 1:31:37
  Speaker 4: this example before of, like, doing your taxes or something. Like,
• 1:31:40
  Speaker 4: this year I literally just, like, told Codex to, like,
• 1:31:42
  Speaker 4: go get everything I needed to do and send it
• 1:31:44
  Speaker 4: to the accountant. And, like, there was this massive list
• 1:31:46
  Speaker 4: of stuff it had to use computers to click through and, like,
• 1:31:48
  Speaker 4: download some stuff.
• 1:31:49
  Speaker 5: And it did it. It was, like, fine.
• 1:31:50
  Speaker 2: It was perfect.
• 1:31:51
  Speaker 5: So, like, I don't know. a lot of this stuff
• 1:31:53
  Speaker 5: it can.
• 1:31:54
  Speaker 1: Already do yeah okay um give you 10x total productivity
• 1:31:58
  Speaker 1: uplift basically if you if it takes you a year
• 1:32:02
  Speaker 1: to make a breakthrough now you make a breakthrough every month.
• 1:32:05
  Speaker 3: I think i would just refuse to give you a
• 1:32:07
  Speaker 3: scaler on this uh like like we might already be
• 1:32:10
  Speaker 3: past that in some like types of work uh like
• 1:32:14
  Speaker 3: let's say you're just trying to prove uh You're trying
• 1:32:17
  Speaker 3: to do certain types of math.
• 1:32:21
  Speaker 1: For you as AI researchers, trying to advance the state
• 1:32:26
  Speaker 1: of AI research, how much are AI researchers sped up
• 1:32:29
  Speaker 1: or uplifted?
• 1:32:31
  Speaker 5: Somewhere between five and ten years.
• 1:32:33
  Speaker 1: Oh, really? That's far away.
• 1:32:34
  Speaker 2: Really think it's longer than for a general remote worker?
• 1:32:37
  Speaker 1: Yeah.
• 1:32:38
  Speaker 5: Interesting.
• 1:32:39
  Speaker 1: I think I'm realizing you probably have very different definitions
• 1:32:42
  Speaker 1: of fully general remote worker. I could have specified that.
• 1:32:44
  Speaker 2: Yeah, this is true. Because I mean, like, yeah, Because obviously, like,
• 1:32:47
  Speaker 2: an AI researcher can be a remote worker.
• 1:32:48
  Speaker 5: And so, like... Yeah, no, I'm picturing, like, you know,
• 1:32:51
  Speaker 5: normal white-collar work over the period of a month.
• 1:32:54
  Speaker 6: Yeah.
• 1:32:54
  Speaker 5: I think it starts to diverge a little bit past
• 1:32:56
  Speaker 5: a month.
• 1:32:56
  Speaker 1: Like, a very competent white-collar worker, but not necessarily, like,
• 1:32:59
  Speaker 1: a super creative researcher.
• 1:33:00
  Speaker 3: I would say, like, two years.
• 1:33:02
  Speaker 1: Two years?
• 1:33:03
  Speaker 5: Yeah.
• 1:33:03
  Speaker 1: 10x?
• 1:33:03
  Speaker 2: Okay.
• 1:33:04
  Speaker 1: How are you, Bert?
• 1:33:05
  Speaker 2: I can kind of see that, actually, because, like, it
• 1:33:07
  Speaker 2: really is just, like... Right now, it's already, like, definitely
• 1:33:10
  Speaker 2: more than 10x of, like, coding stuff, and so it's, like,
• 1:33:13
  Speaker 2: If it can do even one or two loops of
• 1:33:15
  Speaker 2: experimental feedback, that would actually be massive already.
• 1:33:17
  Speaker 1: So 10x uplift of AI researchers within two years? If
• 1:33:22
  Speaker 1: you just plug it into a very naive model of
• 1:33:25
  Speaker 1: AI progress and how much is coming from AI researchers
• 1:33:28
  Speaker 1: and there's a 10x increase in their productivity. Yeah, you
• 1:33:32
  Speaker 1: have radically accelerated pace of AI progress starting two years
• 1:33:35
  Speaker 1: from now.
• 1:33:36
  Speaker 2: Yeah, I mean, I think this will mean that AI
• 1:33:38
  Speaker 2: progress doesn't get bottlenecked on... AI researchers' ability to run
• 1:33:41
  Speaker 2: small experiments that get bottlenecked.
• 1:33:42
  Speaker 5: On other things.
• 1:33:43
  Speaker 1: Of course, of course. But it just happens 10x faster.
• 1:33:45
  Speaker 5: For sure, yeah.
• 1:33:46
  Speaker 1: Which is a huge deal. And that also helps the
• 1:33:48
  Speaker 1: next thing, which gives you 100x speed up, happens sooner, etc.
• 1:33:52
  Speaker 5: Yeah, I'm happy to stick with longer on that one.
• 1:33:55
  Speaker 1: And what's the crux?
• 1:33:58
  Speaker 4: My capacity to absorb information and make the Bayesian optimal
• 1:34:01
  Speaker 4: decision on the next experiment.
• 1:34:03
  Speaker 1: Yeah, makes sense.
• 1:34:04
  Speaker 2: Yeah, I mean, I'm assuming that like you can delegate
• 1:34:06
  Speaker 2: some of this to the AI. So like the AI
• 1:34:08
  Speaker 2: is becoming decent at like deciding, you know, it's run
• 1:34:10
  Speaker 2: this experiment, it's got this result, it runs like the
• 1:34:11
  Speaker 2: next experiment. And then if it can run like two
• 1:34:13
  Speaker 2: or three experiments in a row without like crashing, then
• 1:34:15
  Speaker 2: like that is actually a big update in like uplift.
• 1:34:18
  Speaker 1: And okay, final question. An AI which is, which dominates
• 1:34:25
  Speaker 1: top human experts across every single field of work that
• 1:34:28
  Speaker 1: can be done over a computer. So not only AI research,
• 1:34:33
  Speaker 1: but all cognitive work. And not just like short horizon work,
• 1:34:36
  Speaker 1: but like literally if it takes like three years or something,
• 1:34:39
  Speaker 1: the AI will still do better than humans.
• 1:34:41
  Speaker 2: So this is basically just like ASI.
• 1:34:43
  Speaker 6: Yeah.
• 1:34:43
  Speaker 5: Okay.
• 1:34:44
  Speaker 3: I would say like three or four years.
• 1:34:45
  Speaker 1: The fuck? I mean, that doesn't seem wrong.
• 1:34:50
  Speaker 3: I mean, I would say like AI is obviously getting
• 1:34:54
  Speaker 3: more attention. So it's like one of the harder things,
• 1:34:58
  Speaker 3: but it's like... a lot of energy is being put
• 1:35:01
  Speaker 3: into it. And it's also, uh, like not one of
• 1:35:03
  Speaker 3: the hardest things for AI. Cause it's like real involves
• 1:35:05
  Speaker 3: a lot of code and, and math, which models are
• 1:35:08
  Speaker 3: really good at, uh, maybe for things that involve like
• 1:35:11
  Speaker 3: 3d and like spatial stuff and physical stuff. I think, uh, that,
• 1:35:16
  Speaker 3: that will take a little longer. Um, so, um, especially
• 1:35:20
  Speaker 3: if it's not like, yeah, if it's like mechanical engineering
• 1:35:22
  Speaker 3: or something and it's not getting, like, the most attention
• 1:35:25
  Speaker 3: right now.
• 1:35:26
  Speaker 1: That might take a little longer. But it also doesn't
• 1:35:27
  Speaker 1: include fields where there is relatively little data because of
• 1:35:30
  Speaker 1: the nature of the field. And it has to, like,
• 1:35:32
  Speaker 1: learn that data on the fly. So, for example, it
• 1:35:35
  Speaker 1: has to become superhuman at, like, being an engineer at
• 1:35:40
  Speaker 1: TSMC or something.
• 1:35:41
  Speaker 3: Oh, yeah, so you would have to assume that, like,
• 1:35:45
  Speaker 3: the onboarding, yeah, you can give the AI the same
• 1:35:47
  Speaker 3: onboarding material and, oh, yeah, then there's some, like, something
• 1:35:52
  Speaker 3: has to be solved about, like, sort of longer horizon learning.
• 1:35:57
  Speaker 5: I'd say 5 to 10.
• 1:35:58
  Speaker 1: Basically, yeah. You think automating AI research is like ASI
• 1:36:04
  Speaker 1: complete or something?
• 1:36:05
  Speaker 5: Yeah, I think so.
• 1:36:07
  Speaker 4: Yeah, I think there's so many things in the world which, like,
• 1:36:10
  Speaker 4: even if you have some sort of memory system external
• 1:36:12
  Speaker 4: to the model, and even if, like, context length grows
• 1:36:15
  Speaker 4: a little bit, like, there are just fundamentally things, like,
• 1:36:18
  Speaker 4: even if you could research the information or write notes yourself, like,
• 1:36:20
  Speaker 4: you'd need more than a mental context field to do it.
• 1:36:23
  Speaker 2: Yeah, I mean, I kind of agree in the five-year range,
• 1:36:26
  Speaker 2: at least for the stuff that labs are focusing on.
• 1:36:28
  Speaker 2: But I think there's going to be a long tail
• 1:36:30
  Speaker 2: of stuff which the AI could theoretically go out and
• 1:36:33
  Speaker 2: learn about, but no one has bothered to do it.
• 1:36:35
  Speaker 2: The computer hasn't been allocated to that, so that might
• 1:36:38
  Speaker 2: take longer for literally every single human expert.
• 1:36:41
  Speaker 1: That's right, but by this I also included the ability
• 1:36:43
  Speaker 1: to learn as fast as a human, a new domain.
• 1:36:45
  Speaker 2: I mean, I think that's not necessarily necessary, actually, because
• 1:36:48
  Speaker 2: the AI will have vastly greater experience than any human.
• 1:36:51
  Speaker 4: Right.
• 1:36:51
  Speaker 1: Thanks for doing this, guys. I feel like this was
• 1:36:53
  Speaker 1: a great format for getting different experts to disagree and
• 1:36:56
  Speaker 1: debate and discuss things together. It was very productive.
• 1:36:58
  Speaker 5: Thanks for having us.