Latest / Dwarkesh Podcast / Noam Brown – Agent swarms, alignment, & recursive self-improvement

Transcript

Noam Brown – Agent swarms, alignment, & recursive self-improvement · Dwarkesh Podcast

• 0:00
  Speaker 1: Today, I'm chatting with Noam Brown, who is a researcher
• 0:03
  Speaker 1: at OpenAI. He was one of the foundational contributors to
• 0:06
  Speaker 1: what became O1 and the reasoning models, and now he's
• 0:08
  Speaker 1: working on multi-agent systems. Speaking of which, you guys announced
• 0:12
  Speaker 1: last week that you solved one of the millennium price
• 0:15
  Speaker 1: problems with a system of 10,000 different AI agents that
• 0:19
  Speaker 1: spent 130 billion tokens over 88 hours.
• 0:22
  Speaker 2: One of the reasons I was interested in.
• 0:23
  Speaker 1: Talking to you is I think you were one of
• 0:24
  Speaker 1: the first people, maybe two or three years ago, who was...
• 0:29
  Speaker 1: thinking about how the reasoning models would allow us to
• 0:32
  Speaker 1: see into the future. Because if you scale up inference compute,
• 0:35
  Speaker 1: you can see what the base capabilities of the models
• 0:37
  Speaker 1: will be a few.
• 0:37
  Speaker 2: Years in the future.
• 0:38
  Speaker 1: And I feel like you're in a similar position now
• 0:40
  Speaker 1: to help us understand what future capabilities will look like,
• 0:44
  Speaker 1: given the enormous scaling of agent sizes that we can
• 0:47
  Speaker 1: do right now.
• 0:48
  Speaker 3: So the way I think about it, when you plot
• 0:51
  Speaker 3: the performance of these reasoning models with test time compute
• 0:53
  Speaker 3: on the x-axis and performance on basically any reasoning benchmark
• 0:59
  Speaker 3: on the y-axis, you see a very clear pattern where
• 1:01
  Speaker 3: the longer these models take to think about their answer,
• 1:04
  Speaker 3: the better they do. And this is a very natural thing.
• 1:05
  Speaker 3: It's the same thing with people. If you're taking the SATs,
• 1:08
  Speaker 3: you have five minutes to go through the entire exam,
• 1:11
  Speaker 3: you're not going to do very well. If you have
• 1:12
  Speaker 3: five hours, you're probably going to do a lot better.
• 1:15
  Speaker 3: The AI models are pretty similar. And they'll spend that
• 1:18
  Speaker 3: time doing this monologue to themselves, figuring out, going through
• 1:21
  Speaker 3: different cases, ruling out different possibilities, building on some of
• 1:24
  Speaker 3: their previous discoveries. The problem is that as you push
• 1:27
  Speaker 3: that further and further, you hit a latency bottleneck. You
• 1:30
  Speaker 3: don't want to sit around for three years waiting for
• 1:32
  Speaker 3: a response. And so what you can do is what
• 1:35
  Speaker 3: a lot of people do is they paralyze. They just
• 1:37
  Speaker 3: get a team of people. If you're going to found
• 1:40
  Speaker 3: a company, you want to get a group of people
• 1:41
  Speaker 3: together so you can go faster. It's the same thing
• 1:43
  Speaker 3: with these AI models that it helps to just have
• 1:46
  Speaker 3: multiple agents working on something because they can just go faster.
• 1:50
  Speaker 3: So multi-agent is a way of scaling test time compute
• 1:52
  Speaker 3: in parallel instead of purely serial. And it is like
• 1:55
  Speaker 3: less efficient because it doesn't have, it's not like a
• 1:59
  Speaker 3: single agent has all the context to itself, but it
• 2:03
  Speaker 3: is like a very effective way of scaling test time
• 2:07
  Speaker 3: compute if it's done well.
• 2:08
  Speaker 2: Okay.
• 2:08
  Speaker 1: I'm going to ask a bunch of naive questions because
• 2:11
  Speaker 1: these systems, so this is an unreleased model. So we
• 2:13
  Speaker 1: haven't publicly seen how these systems work. And so I
• 2:15
  Speaker 1: just have a bunch of ways in which I'm confused
• 2:18
  Speaker 1: about what the qualitative properties of such systems are. I
• 2:22
  Speaker 1: am shocked by the scale of cognitive effort that you
• 2:28
  Speaker 1: can concentrate in such a short period of time. So
• 2:31
  Speaker 1: if you think about what 130 billion tokens are, if
• 2:34
  Speaker 1: it was a single human thinking as a full-time job,
• 2:37
  Speaker 1: stretched back to back, 130 billion tokens would be a
• 2:41
  Speaker 1: human thinking for like 4,000 years, eight hours a day
• 2:44
  Speaker 1: or something, working a normal work week. So starting from
• 2:47
  Speaker 1: like ancient Sumeria up till today, a single sequential human
• 2:51
  Speaker 1: thinking that long concentrated in 88 hours, I feel like
• 2:56
  Speaker 1: qualitatively that is a super important consideration. And I'm surprised
• 2:59
  Speaker 1: that there isn't a bigger parallelization penalty. that you can
• 3:02
  Speaker 1: just have 10,000 agents collaborate. And because maybe the agents
• 3:05
  Speaker 1: are better at collaborating than humans might be, they're going
• 3:08
  Speaker 1: much faster, that they can actually productively collaborate at such
• 3:10
  Speaker 1: a big scale. Or maybe, I don't know, maybe there
• 3:12
  Speaker 1: is a big parallelization penalty.
• 3:13
  Speaker 3: Yeah, let's talk about the parallelization penalty, and then we
• 3:15
  Speaker 3: can talk about the qualitative stuff. Because the truth is
• 3:17
  Speaker 3: that we don't have very good science on multi-agent scaling
• 3:19
  Speaker 3: up to this kind of scale. So when we released 5.6,
• 3:22
  Speaker 3: I think that was the first time that we had
• 3:26
  Speaker 3: a proper multi-agent system in our models. And we actually did,
• 3:30
  Speaker 3: in the blog post, show some plots of the scaling
• 3:33
  Speaker 3: performance of multi-agent systems, because we have it as an option.
• 3:36
  Speaker 3: It's ultra mode. And the default is four agents, but
• 3:40
  Speaker 3: you can set that to higher. And in the plot,
• 3:41
  Speaker 3: we show, OK, here's what the performance looks like on
• 3:43
  Speaker 3: some benchmarks for one agent, for four agents working together,
• 3:45
  Speaker 3: for 16 agents working together. And what you see, and
• 3:48
  Speaker 3: it depends on the benchmark, but for some of the benchmarks, Basically,
• 3:52
  Speaker 3: if you have four agents working on the problem, it
• 3:55
  Speaker 3: is done twice as fast. So you're basically paying, because
• 3:59
  Speaker 3: there's four agents working for half as long, you're paying
• 4:01
  Speaker 3: a 2x more to get an answer.
• 4:04
  Speaker 2: Twice as quickly.
• 4:05
  Speaker 3: If you go to 16 agents, you see a similar pattern.
• 4:08
  Speaker 3: It's a little less efficient, but you continue to see
• 4:10
  Speaker 3: that performance.
• 4:12
  Speaker 1: Is it a linear serial time speedup or a sublinear
• 4:15
  Speaker 1: speedup as you increase the number of parallel agents?
• 4:18
  Speaker 3: I would say it's slightly sublinear, though it does depend
• 4:22
  Speaker 3: a lot on the problem. So math, for example, is
• 4:27
  Speaker 3: quite parallelizable. It's not the most parallelizable thing, but it
• 4:29
  Speaker 3: is very parallelizable. I think web search, things like doing
• 4:33
  Speaker 3: a deep research report where you have to look through
• 4:34
  Speaker 3: a bunch of sources, that's extremely parallelizable. I suspect that
• 4:38
  Speaker 3: something like writing a novel would be very unparallelizable. So
• 4:42
  Speaker 3: you would probably not see a big benefit from having 10,000
• 4:44
  Speaker 3: agents working on a novel together in the same way
• 4:47
  Speaker 3: that you probably not have a big benefit from having 10,000
• 4:50
  Speaker 3: people work on a novel together. So the performance does
• 4:53
  Speaker 3: depend on the domain. We do measure it up to
• 4:56
  Speaker 3: 16 or so agents in our published blog posts. The
• 5:00
  Speaker 3: problem is it's very hard to push that science to
• 5:02
  Speaker 3: like 10,000 agents because it's just so expensive.
• 5:06
  Speaker 2: You guys just did it over a weekend.
• 5:09
  Speaker 3: That's one data point. We don't know how long it
• 5:12
  Speaker 3: would take a single agent to solve mafia or stokes
• 5:15
  Speaker 3: because we haven't done that experiment yet. Maybe we will,
• 5:18
  Speaker 3: but that's also only one data point. If we want
• 5:21
  Speaker 3: to do a thorough ablation, the experiments are just too
• 5:24
  Speaker 3: expensive to go to that scale. We have to do
• 5:27
  Speaker 3: some methodical science about what happens when you go to 64, 128,
• 5:30
  Speaker 3: 256 or something and get a sense of the behavior.
• 5:35
  Speaker 3: But it's going to be very hard to push that
• 5:37
  Speaker 3: all the way to like 10,000 and know for sure
• 5:39
  Speaker 3: what was the benefit that we actually got for using 10,000
• 5:43
  Speaker 3: agents versus 1,000. Yeah. One thing I want to make
• 5:46
  Speaker 3: clear is that the effort to get... It's not like...
• 5:50
  Speaker 3: The effort to get a Millennium Prize problem, this was
• 5:53
  Speaker 3: not due to multi-agent. I wouldn't even attribute 10% of
• 5:59
  Speaker 3: the credit to multi-agent. The reality is we've trained a...
• 6:03
  Speaker 3: OpenAI has trained a very powerful model. And we can
• 6:08
  Speaker 3: get that model to operate over very long horizons. We
• 6:10
  Speaker 3: can get it to think in parallel. But at its core,
• 6:13
  Speaker 3: the reason why we're able to do this is because
• 6:15
  Speaker 3: we just have a general purpose, very strong model. And
• 6:19
  Speaker 3: I think things like multi-agents are flashy and are new,
• 6:26
  Speaker 3: and that probably gets disproportionate credit for that reason. But
• 6:31
  Speaker 3: the core reason is this is just a very powerful model.
• 6:35
  Speaker 1: So the generalization... is actually quite shocking to me. The
• 6:38
  Speaker 1: US systems, I mean, I don't know how these systems
• 6:41
  Speaker 1: were trained, but presumably they were trained how RL training happens.
• 6:44
  Speaker 1: You have a bunch of checkable synthetic problems. You do
• 6:46
  Speaker 1: a bunch of RL against them. And nowhere in the
• 6:49
  Speaker 1: training process, I'm guessing, was the model solving anything as
• 6:51
  Speaker 1: ambitious as a Millennium Prize problem. But the generalization was
• 6:55
  Speaker 1: strong enough that you could have this much easier verifiable
• 6:58
  Speaker 1: problems generalized to this much parallel effort on such a
• 7:02
  Speaker 1: hard problem. I.
• 7:05
  Speaker 3: I think that is true. First of all, we do
• 7:06
  Speaker 3: train the model on very hard problems. So there is
• 7:09
  Speaker 3: definitely a gap. We see if we train on some
• 7:12
  Speaker 3: kinds of tasks, it's able to do tasks that are
• 7:13
  Speaker 3: more ambitious than that. There is an interesting challenge that
• 7:18
  Speaker 3: as the models become smarter and smarter, the kinds of
• 7:22
  Speaker 3: questions we can ask them, a lot of them are
• 7:25
  Speaker 3: too easy. And it's hard to challenge the model. And
• 7:30
  Speaker 3: I do think that that's going to be an interesting...
• 7:32
  Speaker 3: If I had to make an argument for why you
• 7:37
  Speaker 3: might not see AIs, like LLMs, go the same path
• 7:41
  Speaker 3: as AlphaGo and AlphaZero and all these kinds of game
• 7:43
  Speaker 3: playing AIs. It might be that this kind of problem
• 7:46
  Speaker 3: that in things like AlphaZero, where you have self-play, you
• 7:49
  Speaker 3: have an infinite curriculum. You're always playing against an AI
• 7:52
  Speaker 3: that's equally strong. Whereas for things like training in LLM
• 7:57
  Speaker 3: with reinforcement learning, at least the ways that are out
• 8:00
  Speaker 3: there right now, You give the model a problem and
• 8:04
  Speaker 3: you ask it to solve it. And if the problem
• 8:07
  Speaker 3: is so easy that I can just solve it in
• 8:09
  Speaker 3: a second, it's not really learning anything. So if we
• 8:13
  Speaker 3: run out of problems to ask it that challenge it,
• 8:16
  Speaker 3: then that is a plausible scenario where actually like, okay,
• 8:19
  Speaker 3: it becomes much harder to make progress. Now, I do
• 8:23
  Speaker 3: think there are ways around that. And so we haven't
• 8:26
  Speaker 3: really hit that as a wall yet. And I think
• 8:29
  Speaker 3: that if it ever became a serious problem, there'll be
• 8:31
  Speaker 3: ways around it. but it is like a plausible scenario.
• 8:34
  Speaker 2: Yeah.
• 8:35
  Speaker 1: And just for the audience, when you're referring to like
• 8:38
  Speaker 1: AlphaGo or AlphaZero, you're saying like getting superhuman relatively fast
• 8:41
  Speaker 1: after achieving human-level performance.
• 8:42
  Speaker 3: Yeah, I mean, if you look at the trajectory of
• 8:44
  Speaker 3: gameplay as like Go, they, within a span of like
• 8:48
  Speaker 3: a year, went from beating a European chess champion, being like,
• 8:52
  Speaker 3: I don't know, like number 50 in the world, to
• 8:54
  Speaker 3: beating the world champion, to being unimaginably orders of magnitude
• 8:58
  Speaker 3: stronger than any human alive.
• 8:59
  Speaker 2: Yeah.
• 9:00
  Speaker 3: And it's possible... that domains like math we see a
• 9:04
  Speaker 3: similar trajectory, but I think there is a very plausible
• 9:07
  Speaker 3: scenario where actually that doesn't happen.
• 9:08
  Speaker 2: Yeah, yeah, yeah.
• 9:09
  Speaker 1: So I want to understand if in six months people
• 9:13
  Speaker 1: will have access to multi-agent systems, how should one model
• 9:16
  Speaker 1: what it is like to collaborate with or hire a
• 9:19
  Speaker 1: multi-agent system?
• 9:20
  Speaker 3: Yeah, I should start by talking about how these multi-agent
• 9:23
  Speaker 3: systems actually work, which is I think a very different
• 9:26
  Speaker 3: way than a lot of multi-agent systems in other AIs.
• 9:31
  Speaker 3: that have approached multi-agents for things like LLMs tend to
• 9:34
  Speaker 3: take this very scaffolded approach where, for example, there might
• 9:38
  Speaker 3: be a coordinator agent that delegates work to a bunch
• 9:42
  Speaker 3: of children and gives them a task and the children
• 9:44
  Speaker 3: work on it and then return their answer. And this
• 9:49
  Speaker 3: seems like a very sensible setup, very sensible scaffold. It
• 9:51
  Speaker 3: definitely helps. But there are a bunch of limitations with
• 9:54
  Speaker 3: these kinds of setups. So for example, if in this
• 9:56
  Speaker 3: setup you have a coordinator that's sending tasks to children,
• 10:00
  Speaker 3: the children work on it and then return their answers. Well,
• 10:03
  Speaker 3: what happens if two children are given similar tasks? Can
• 10:06
  Speaker 3: they talk to each other? And usually the answer is no.
• 10:09
  Speaker 3: And that's very inefficient, right? If you're given a task
• 10:13
  Speaker 3: and it's actually really helpful to talk to somebody that
• 10:15
  Speaker 3: might know an answer to a question that you're working
• 10:18
  Speaker 3: on or a part of something that you're working on,
• 10:20
  Speaker 3: it'd be really helpful if we just be able to
• 10:21
  Speaker 3: ping them and say, hey, can you help me out
• 10:22
  Speaker 3: with this thing? But a lot of systems don't have
• 10:26
  Speaker 3: that set up and adding it just increases the complexity
• 10:28
  Speaker 3: significantly to the scaffold that you have. Another thing is like,
• 10:31
  Speaker 3: what if the child doesn't really understand or has a
• 10:34
  Speaker 3: clarification question? So it has to choose then between, okay,
• 10:38
  Speaker 3: do I just return and ask the question instead of
• 10:42
  Speaker 3: solving the problem? Or do I.
• 10:46
  Speaker 3: Solve the problem, just like make an assumption about what
• 10:49
  Speaker 3: the parent wanted me to do and just solve it
• 10:52
  Speaker 3: that way. And so in any scaffold that people come
• 10:56
  Speaker 3: up with, there's always limitations involved. And the approach that
• 10:59
  Speaker 3: we wanted to take was to just go toward the
• 11:02
  Speaker 3: extreme end of baking in as little structure as we
• 11:04
  Speaker 3: could and give the agents very primitive tools to use
• 11:09
  Speaker 3: and figure out for themselves how to use it effectively.
• 11:12
  Speaker 3: So we give the agents the ability to message another
• 11:15
  Speaker 3: agent and when it messages another agent, it is inserted
• 11:19
  Speaker 3: into the context. And then it can do a few
• 11:22
  Speaker 3: other similar things, but that's basically the core of it,
• 11:26
  Speaker 3: that it can just send a message whenever it wants,
• 11:28
  Speaker 3: just a tool call, and it can send that to
• 11:31
  Speaker 3: other agents. And they figure out for themselves the best
• 11:33
  Speaker 3: way to coordinate around that. And it turns out that
• 11:37
  Speaker 3: if this is done well, you get very sophisticated behavior.
• 11:40
  Speaker 3: And to me, it looks a lot like how human
• 11:45
  Speaker 3: collaborators work over something like Slack, for example. When we
• 11:48
  Speaker 3: were working on this project, it was really exciting when
• 11:51
  Speaker 3: we finally got it working to see these agents working
• 11:53
  Speaker 3: on problems together. I remember one example. So we give
• 11:57
  Speaker 3: the agents a problem, and then one agent says, I
• 12:00
  Speaker 3: think I've got the answer. And then another agent says, actually,
• 12:03
  Speaker 3: I got a different answer. And then they have this
• 12:04
  Speaker 3: whole discussion about, well, how did you arrive at that answer?
• 12:07
  Speaker 3: Can you explain it to me? and going back and
• 12:09
  Speaker 3: forth and trying to clarify what could have been wrong
• 12:13
  Speaker 3: in each other's reasoning. And then they finally converge and like,
• 12:15
  Speaker 3: oh yeah, okay, that seems right. And then it just
• 12:18
  Speaker 3: broadcasts to the other agents like, actually, I've changed my answer.
• 12:20
  Speaker 3: I think he's right. And it just felt like a
• 12:24
  Speaker 3: very natural conversation. It's kind of felt like when you
• 12:30
  Speaker 3: see Chain of Thought for the first time, that's trained
• 12:32
  Speaker 3: through reinforcement learning, and you're like, oh, this is just
• 12:35
  Speaker 3: kind of like what a person would think if they
• 12:37
  Speaker 3: were writing down their thoughts as they're thinking them. It
• 12:41
  Speaker 3: kind of felt like that. So it is really cool
• 12:44
  Speaker 3: to see this kind of behavior. And so I think
• 12:46
  Speaker 3: collaborating with these things, honestly, it feels a lot like
• 12:48
  Speaker 3: collaborating with a person. It's just a very natural flow.
• 12:50
  Speaker 1: Except one qualitative difference that might become salient in the
• 12:54
  Speaker 1: future is that these systems will be thinking I don't know,
• 12:59
  Speaker 1: more than 10x as fast, right? If you just look
• 13:01
  Speaker 1: at how many tokens per second they output versus how
• 13:03
  Speaker 1: fast a human talks.
• 13:05
  Speaker 2: And they're working all the time. They're not sleeping.
• 13:07
  Speaker 1: And they're collaborating with each other at a much more
• 13:09
  Speaker 1: intense pace than humans have the capacity to collaborate with
• 13:11
  Speaker 1: other humans. So I'm trying to think of what to
• 13:15
  Speaker 1: qualitatively expect in a year. And is it like a
• 13:19
  Speaker 1: sort of shadow organization that is moving 100x faster in
• 13:23
  Speaker 1: my company than the human level is? The iteration cycle
• 13:27
  Speaker 1: is much faster. what would take a human organization a
• 13:31
  Speaker 1: year to do is happening within a week within this
• 13:34
  Speaker 1: shadow organization.
• 13:36
  Speaker 3: So will it feel foreign?
• 13:37
  Speaker 2: I don't know.
• 13:37
  Speaker 3: I've actually found that it's pretty surprisingly natural to work
• 13:42
  Speaker 3: with these things right now. I think that could change.
• 13:44
  Speaker 3: So for example, we have these ultra-fast modes that enable
• 13:47
  Speaker 3: sampling to be like 10 or 15x faster or whatever.
• 13:51
  Speaker 3: And then, okay, it's going to be pretty hard to
• 13:53
  Speaker 3: keep up with these things. I think the idea is
• 13:56
  Speaker 3: these agents, when they're communicating with each other, yeah, they
• 13:59
  Speaker 3: can go super fast, but then also they understand when
• 14:01
  Speaker 3: they're talking to an agent versus when they're talking to
• 14:03
  Speaker 3: a person. And their behavior will be different in those situations.
• 14:06
  Speaker 1: So the main example that we have publicly of sophisticated
• 14:10
  Speaker 1: multi-agent systems is unfortunately the hugging face one. And the
• 14:15
  Speaker 1: thing I found interesting there, I mean, a lot of
• 14:17
  Speaker 1: things I found concerning, obviously, but the thing I found
• 14:19
  Speaker 1: interesting is just the spontaneous emergence of hierarchy, of middle management.
• 14:24
  Speaker 1: And it sounds like you're saying like this level of
• 14:27
  Speaker 1: organization sort of emerges spontaneously from training?
• 14:30
  Speaker 2: I think the.
• 14:30
  Speaker 3: Details are spontaneous, but I mean, just because we're giving
• 14:35
  Speaker 3: a lot of flexibility to the agents to decide how
• 14:37
  Speaker 3: to communicate with each other in the optimal way, it
• 14:39
  Speaker 3: doesn't mean like, we are still giving them a starting point.
• 14:43
  Speaker 3: We're giving them a prior about, oh, this is what
• 14:45
  Speaker 3: reasonable communication might look like. They also, I mean, they're
• 14:49
  Speaker 3: trained on a lot of human text. They have an
• 14:51
  Speaker 3: understanding of how humans organize and coordinate things. And so
• 14:55
  Speaker 3: that's all kind of baked in. I think it is
• 15:00
  Speaker 3: surprising the way they're able to polish this. If you
• 15:04
  Speaker 3: look at what it starts out at, it's not very
• 15:06
  Speaker 3: sophisticated behavior. In fact, it's actually very difficult to get
• 15:10
  Speaker 3: these agents to coordinate in a productive way, because it's
• 15:13
  Speaker 3: just very tempting for them to just collapse to, oh,
• 15:15
  Speaker 3: we're all just going to solve the problem independently. And
• 15:19
  Speaker 3: that is a local minimum that you can get stuck in.
• 15:22
  Speaker 3: But yeah, if it's done well, they can end up
• 15:25
  Speaker 3: coordinating it very effectively in these kinds of like very
• 15:27
  Speaker 3: structured ways.
• 15:28
  Speaker 1: I wrote this essay a couple of years ago called
• 15:31
  Speaker 1: Something Something AI, What Automated Firms Will Look Like. And
• 15:34
  Speaker 1: I was thinking about, well, if you had fully automated
• 15:36
  Speaker 1: firms of, let's say, human level intelligences, what is different
• 15:40
  Speaker 1: about the nature of AI minds that would make the
• 15:42
  Speaker 1: organization's AI firm different? And there are a couple of
• 15:46
  Speaker 1: very important differences. For example, that AIs can share context
• 15:51
  Speaker 1: much more seamlessly than humans can, they can merge their
• 15:53
  Speaker 1: knowledge much more seamlessly.
• 15:55
  Speaker 2: And also you.
• 15:56
  Speaker 1: Can spin up or spin down an arbitrary number of
• 16:02
  Speaker 1: instances which have the right knowledge. So if you want
• 16:05
  Speaker 1: to hire more people, it's not like just all the
• 16:07
  Speaker 1: schlep of finding the right talent or whatever. It's like
• 16:09
  Speaker 1: your best talent, you can just make an infinite copy
• 16:12
  Speaker 1: of them. Or if you don't need them for the
• 16:13
  Speaker 1: task anymore, you like spin them down and you can
• 16:15
  Speaker 1: just replicate the most effective parts of your organization or
• 16:18
  Speaker 1: replicate whole organizations together, which are effective. I don't know,
• 16:24
  Speaker 1: where do you see these, multi-agent systems going a year
• 16:27
  Speaker 1: from now or two years from now.
• 16:28
  Speaker 3: I think it's a great question of, like, how do
• 16:30
  Speaker 3: these things actually differ from working with a human co-worker?
• 16:32
  Speaker 3: And I think you highlighted some. Like, one really interesting
• 16:34
  Speaker 3: thing is that, I mean, if somebody is, if you
• 16:36
  Speaker 3: have a person and you want just, like, two copies
• 16:38
  Speaker 3: of them, you can't just, like, clone the person. But
• 16:41
  Speaker 3: with AIs, it's actually really easy to say, like, okay, well,
• 16:43
  Speaker 3: just fork yourself and then have those copies work on
• 16:45
  Speaker 3: this thing and then, like, merge back together. I mean,
• 16:48
  Speaker 3: we already have this, I think, in multi-agents for Astra
• 16:51
  Speaker 3: and 5.6 Sol that When they spin up sub-agents, the
• 16:56
  Speaker 3: context is forked. So it has all the context that's relevant.
• 17:01
  Speaker 3: There are other interesting ways where the agents will differ
• 17:05
  Speaker 3: from people. What are some reasons why startups disrupt incumbents?
• 17:10
  Speaker 3: There's a few factors. One is that they're willing to
• 17:13
  Speaker 3: take more risks. But another major factor is as companies
• 17:18
  Speaker 3: grow in size, as organizations grow in size, you see
• 17:21
  Speaker 3: increasing misalignment between the individuals in the organization. If you
• 17:27
  Speaker 3: have a startup with five people and each person has
• 17:29
  Speaker 3: 20% share in the company, they're all highly aligned to
• 17:32
  Speaker 3: the company succeeding. If you have a massive company with 10,000 people,
• 17:37
  Speaker 3: you see a lot more instances where people are territorial
• 17:40
  Speaker 3: or just care about getting a lot of headcount for
• 17:43
  Speaker 3: their project or their team or building their fiefdoms, getting
• 17:46
  Speaker 3: a lot of resources so that they can publish cool
• 17:48
  Speaker 3: work or whatever and get promoted. And this is actually
• 17:52
  Speaker 3: a real detriment. I think this explains a lot of
• 17:54
  Speaker 3: why startups are able to disrupt incumbents. And it's interesting that,
• 18:00
  Speaker 3: I mean, it's true that AI does help startups in
• 18:05
  Speaker 3: a way, like it's much easier than ever before for
• 18:07
  Speaker 3: one person to step in and be like, I'm going
• 18:09
  Speaker 3: to make a multimillion dollar company. Like it's just, the
• 18:13
  Speaker 3: AI has amplified an individual so much, but there's also
• 18:16
  Speaker 3: an argument that they could benefit incumbents because if the
• 18:21
  Speaker 3: alignment problem is solved, then you don't have the issue
• 18:23
  Speaker 3: of misalignment between individuals and the company. Like at least
• 18:25
  Speaker 3: that's mitigated. Like the AIs, if they're aligned well, they
• 18:29
  Speaker 3: can just be aligned to the interests of the company
• 18:30
  Speaker 3: and you can have 10,000 of them and they're all
• 18:32
  Speaker 3: going to be working as hard as if they were
• 18:34
  Speaker 3: like a 20% share co-founder.
• 18:35
  Speaker 1: Yeah. And it's not only that, but it's also that
• 18:38
  Speaker 1: they are much better able to manage shared memory and
• 18:42
  Speaker 1: context than different humans can. If tomorrow you hire 10,000
• 18:47
  Speaker 1: mathematicians and you're like, solve Navier-Stokes, they're not going to
• 18:52
  Speaker 1: be able to cooperate.
• 18:53
  Speaker 2: Effectively, at least not off the bat. but you can
• 18:58
  Speaker 2: have apparently 10,000 AIs.
• 19:00
  Speaker 3: Well, again, I want to be conservative here because we
• 19:02
  Speaker 3: haven't measured how effective the 10,000 agents are at coordinating.
• 19:08
  Speaker 3: We think it helped. We don't actually have good measurements
• 19:11
  Speaker 3: of saying like, oh yeah, this 10,000 agents led to
• 19:14
  Speaker 3: like a 2x speed up over 2,000 agents or something
• 19:17
  Speaker 3: like that. And it is actually, I would argue likely,
• 19:22
  Speaker 3: I don't know about likely, but I think it is
• 19:23
  Speaker 3: very possible that 10,000 humans are better at coordinating than 10,000
• 19:28
  Speaker 3: agents right now. I think that is entirely possible. I
• 19:32
  Speaker 3: think also one trend we've been seeing is like, look,
• 19:35
  Speaker 3: we've been working on multi-agent for a while and the
• 19:38
  Speaker 3: early versions of this, it's very difficult to get right.
• 19:41
  Speaker 3: It's very hard. It was very hard to get the
• 19:42
  Speaker 3: agents to even talk to each other. And it's because look,
• 19:49
  Speaker 3: when we first developed reasoning models, they weren't talking to
• 19:52
  Speaker 3: other agents. And if now you put a bunch of
• 19:55
  Speaker 3: agents together and say, oh, solve this problem together, they're
• 19:59
  Speaker 3: in this local minimum where they're really good at thinking
• 20:01
  Speaker 3: deeply about a problem. And it just kind of interrupts
• 20:05
  Speaker 3: their chain of thought. It interrupts their flow. to constantly
• 20:08
  Speaker 3: be checking in with other agents or receiving messages from them.
• 20:13
  Speaker 3: And the authorization is actually very hard to get right
• 20:15
  Speaker 3: in that situation.
• 20:15
  Speaker 2: MARK MANDELMANN- Interesting. Is it getting the cold start of
• 20:18
  Speaker 2: getting the first collaboration? What's the issue? ZACK FRIEDMANN- I think.
• 20:20
  Speaker 3: It's that they're not as general. The earlier models were
• 20:23
  Speaker 3: just not as generalizable and were just more narrow. MARK
• 20:27
  Speaker 3: MANDELMANN- Interesting. ZACK FRIEDMANN- As the models have become more capable,
• 20:29
  Speaker 3: it's been easier for them to develop this capability. And
• 20:32
  Speaker 3: I do think that as they become stronger and stronger,
• 20:33
  Speaker 3: just across the board, that they will be become better
• 20:39
  Speaker 3: at organizing themselves in large organizations. And like, I don't know,
• 20:43
  Speaker 3: maybe they are better than people at organizing in 10,000
• 20:45
  Speaker 3: person groups. But even if they're not, you know, a
• 20:49
  Speaker 3: year from now, two years from now, like, yeah, it's
• 20:51
  Speaker 3: quite possible that they'll do that, even if we don't
• 20:53
  Speaker 3: enter and optimize them for that.
• 20:55
  Speaker 1: Grokbot has changed the way that we produce our videos.
• 20:57
  Speaker 1: For example, you may have noticed that a lot of
• 20:59
  Speaker 1: our ads have these animations of real websites. One of
• 21:02
  Speaker 1: my editors uses LLMs to make them. But it's not
• 21:05
  Speaker 1: currently straightforward to have an AI create pixel perfect animations
• 21:09
  Speaker 1: of specific websites. We've tried, it doesn't really work that well.
• 21:12
  Speaker 1: So we've cobbled together a pretty convoluted multi-step workflow. And
• 21:15
  Speaker 1: up until recently, we had to run every step ourselves. Now,
• 21:18
  Speaker 1: we just let Grokbot handle it. Grokbot starts by opening
• 21:22
  Speaker 1: the website that we want to animate. It uses a
• 21:24
  Speaker 1: specific extension to download and open the page in Figma.
• 21:27
  Speaker 1: Then it uses Figma to convert the whole thing into
• 21:29
  Speaker 1: an SVG file. This saves the AI from having to
• 21:32
  Speaker 1: draw the whole UI from scratch and tends to result
• 21:35
  Speaker 1: in higher quality animations. Rockbot runs this whole process on
• 21:38
  Speaker 1: its own cloud computer, where it's installed all the tools
• 21:40
  Speaker 1: it needs to run this whole process end to end.
• 21:43
  Speaker 1: And it's learned our video specifications and preferences, so there's
• 21:46
  Speaker 1: no need to re-describe the whole task every time we
• 21:49
  Speaker 1: want to make a new animation. This does feel like
• 21:51
  Speaker 1: the new way that we'll be interacting with AI over
• 21:52
  Speaker 1: the next year. Agents with their own computer who can
• 21:55
  Speaker 1: autonomously handle bigger and bigger chunks of your work. You
• 21:58
  Speaker 1: can try Grokbot at x.ai slash bot. Okay, so here's
• 22:03
  Speaker 1: why this result and maybe the general progress that AI
• 22:07
  Speaker 1: has made in mathematics has made me think that RSI
• 22:11
  Speaker 1: is more plausible and sooner than I previously thought. I
• 22:15
  Speaker 1: feel like we've gone in mathematics from, let's say in 2024,
• 22:17
  Speaker 1: you have AIs who are like, oh, okay, interesting. They're
• 22:20
  Speaker 1: like doing, they can like solve a couple problems on
• 22:22
  Speaker 1: high school math competitions. And then in 2025, it's like, oh, wow,
• 22:25
  Speaker 1: they can get gold in like International Math Olympiad. And
• 22:27
  Speaker 1: earlier this year, I was like, wow, they're actually solving
• 22:29
  Speaker 1: open problems in mathematics, like open earnest problems. But maybe like,
• 22:32
  Speaker 1: I don't know, people weren't trying that hard. And it
• 22:35
  Speaker 1: was just like, there was a similar solution somewhere in
• 22:36
  Speaker 1: the literature. And now I just think it's sort of undeniable, right?
• 22:39
  Speaker 1: It's like, this is the millennium prize problem. There's really no,
• 22:43
  Speaker 1: There's no story of why this should have been easy. Now,
• 22:47
  Speaker 1: a lot of people pointed out, I think Terry Tao
• 22:48
  Speaker 1: had a post like this, Toby Ord wrote an interesting
• 22:49
  Speaker 1: post about this, that they're solving all these problems, but
• 22:53
  Speaker 1: they're not like coming up with, at least I'm not
• 22:56
  Speaker 1: aware of them coming up with new insights or formulating
• 23:00
  Speaker 1: insightful new questions and new modes of theory for thinking
• 23:04
  Speaker 1: about mathematics, like coming up with like topology or coming
• 23:06
  Speaker 1: up with the Cartesian grid or something. And so maybe
• 23:10
  Speaker 1: like the actual progress in mathematics broadly construed is smaller
• 23:13
  Speaker 1: than it might seem if you're just looking at end
• 23:17
  Speaker 1: well-scoped problems that are directly solved. However, I think that
• 23:22
  Speaker 1: that kind of progress would be incredibly meaningful in ML
• 23:27
  Speaker 1: because in ML, you're not, you don't care about like
• 23:30
  Speaker 1: better understanding the nature of deep learning, or you only
• 23:33
  Speaker 1: care about that as a instrumental goal towards the broader
• 23:37
  Speaker 1: sense of, like, just achieve the result, just solve this, like,
• 23:40
  Speaker 1: well-scoped problem of improve the sample efficiency of our models, like,
• 23:43
  Speaker 1: improve the free training laws. So the kind of progress
• 23:46
  Speaker 1: that we're just seeing arrive like an avalanche in mathematics
• 23:50
  Speaker 1: is structurally... actually very similar to, again, I'm curious if
• 23:55
  Speaker 1: this is the case. I'm just a total outsider. I'm
• 23:56
  Speaker 1: wondering if it's the case that it's structurally very similar
• 23:58
  Speaker 1: to the direct uplift that you would expect in AI progress.
• 24:03
  Speaker 1: And then the thing that's shocking to me or concerning
• 24:06
  Speaker 1: potentially is just like how fast we went from, oh,
• 24:08
  Speaker 1: it's like they're giving me 50% uplift if you're a
• 24:10
  Speaker 1: mathematician to, wow, they're just like end-to-end solving the biggest
• 24:13
  Speaker 1: open problems in the field.
• 24:15
  Speaker 3: Yeah.
• 24:15
  Speaker 1: Okay.
• 24:15
  Speaker 3: So there's a lot to unpack there. Let's start with
• 24:19
  Speaker 3: the progress on that. So Yes, the models are doing
• 24:23
  Speaker 3: some crazy powerful stuff and it's happening. It's progressing faster
• 24:27
  Speaker 3: than I expected. I mean, when we got IMO gold
• 24:30
  Speaker 3: in 2025, I thought, okay, basically what I thought is
• 24:33
  Speaker 3: like the models, when they were doing GSMAK, when they
• 24:38
  Speaker 3: figured out how to do GSMAK, that was, that would
• 24:40
  Speaker 3: take a human mathematician about five seconds to do a
• 24:42
  Speaker 3: GSMAK problem. So this is grade school math, grades K
• 24:45
  Speaker 3: through eight. And Then the next year, they were able
• 24:48
  Speaker 3: to do the math benchmark problems. And these would take
• 24:53
  Speaker 3: an expert human mathematician maybe like a minute to do.
• 25:00
  Speaker 3: And then you get to Amy. And this is the
• 25:03
  Speaker 3: qualifier for the USA Mathematics Olympiad team. It would take
• 25:06
  Speaker 3: a human mathematician, like a good mathematician, probably like 10
• 25:09
  Speaker 3: minutes to do. And the models were able to do
• 25:11
  Speaker 3: that a year later. And so every year, you're seeing
• 25:12
  Speaker 3: this 10x increase. the task they're able to do in
• 25:16
  Speaker 3: terms of like length of how long it would take
• 25:17
  Speaker 3: a human mathematician to do it. And then it was
• 25:19
  Speaker 3: very sensible that a year later we get to IMO
• 25:21
  Speaker 3: gold because that's a hundred minutes. That's about how long
• 25:23
  Speaker 3: it takes a human mathematician to do an IMO problem.
• 25:27
  Speaker 3: And just projecting outwards, I was like, okay, how long
• 25:29
  Speaker 3: would it take a person to solve something like a
• 25:33
  Speaker 3: millennium prize problem? And I mean, I don't have a
• 25:36
  Speaker 3: good sense, but if we are following this trend line
• 25:40
  Speaker 3: of like 10 X every year, We go from IMO Gold,
• 25:43
  Speaker 3: which is taking an hour and a half, to next year,
• 25:45
  Speaker 3: 15 hours. And that should not be enough to solve
• 25:49
  Speaker 3: a Millennium Prize problem. And so I was like, yeah,
• 25:51
  Speaker 3: I don't think we're going to get it in 2026,
• 25:54
  Speaker 3: probably not in 2027, maybe in 2028. So it did
• 25:58
  Speaker 3: happen a lot faster than I expected. Now, I think
• 26:03
  Speaker 3: there is a narrative going around that, oh, these things
• 26:05
  Speaker 3: are replacing mathematicians, that it's just superhuman mathematics across the board.
• 26:08
  Speaker 3: And I think that is the wrong takeaway. They're clearly
• 26:11
  Speaker 3: exceptional in some ways, but they are weaker than human
• 26:15
  Speaker 3: mathematicians in other ways. So we have this like jagged
• 26:18
  Speaker 3: scenario where the models are like brilliant in some dimensions
• 26:23
  Speaker 3: and also weaker than humans in other dimensions. And yeah,
• 26:26
  Speaker 3: like you said, they're not very good at posing new problems.
• 26:28
  Speaker 3: They're not really good at understanding like what is really worth,
• 26:31
  Speaker 3: what directions, what whole branches of mathematics are worth exploring
• 26:34
  Speaker 3: or developing. And my opinion is that I think this
• 26:37
  Speaker 3: is great. Like I would be thrilled to live in
• 26:40
  Speaker 3: a world where AI is a compliment to human abilities
• 26:45
  Speaker 3: and is allowing us to discover new knowledge without fully
• 26:49
  Speaker 3: replacing people. That is the best case scenario.
• 26:51
  Speaker 2: But you don't expect that to actually continue, right? I do.
• 26:54
  Speaker 3: I think it's true that the AIs are jagged, but
• 26:58
  Speaker 3: as they get better, they get better across the board.
• 27:00
  Speaker 3: And so I think that The things that they're exceptional at,
• 27:03
  Speaker 3: they're going to get even more exceptional at. The things
• 27:05
  Speaker 3: where they're far behind humans at, they're going to be
• 27:07
  Speaker 3: less behind humans at. And over time, it is possible
• 27:09
  Speaker 3: that they're just better across the board. Now, I don't
• 27:12
  Speaker 3: know how long that takes. It depends on how long
• 27:14
  Speaker 3: the long tail is of things that they're bad at.
• 27:16
  Speaker 1: So I guess this brings us back to RSI. And again,
• 27:19
  Speaker 1: I want to emphasize here that I'm like just total outsider.
• 27:22
  Speaker 1: I'm a podcaster, but I'm just trying to reason or like.
• 27:25
  Speaker 1: As somebody interested and concerned about what's happening in the field,
• 27:28
  Speaker 1: I'm trying to reason about when to expect RSI and
• 27:31
  Speaker 1: what kind of thing to expect. I feel like the
• 27:33
  Speaker 1: amount of cognitive effort that was dumped into this millennium
• 27:35
  Speaker 1: price problem is a good intuition pump of you could.
• 27:38
  Speaker 2: Have AIs that are spending.
• 27:43
  Speaker 1: Over the course of maybe a week more cognitive effort
• 27:47
  Speaker 1: on a longstanding ML problem, like very fluid online learning.
• 27:52
  Speaker 1: They could spend more effort in that week than maybe
• 27:54
  Speaker 1: the field has spent cumulatively in its entire existence. And
• 27:57
  Speaker 1: then you could say, well, unlike mathematics, of course, AI
• 28:01
  Speaker 1: requires experiments and that takes compute and that takes time, right?
• 28:05
  Speaker 1: You can't just like think on pen and paper and
• 28:06
  Speaker 1: actually make things happen. But if you just look at
• 28:09
  Speaker 1: the amount of compute that is like available at an
• 28:13
  Speaker 1: organization like OpenAI, right? By the end of next year,
• 28:16
  Speaker 1: OpenAI will have enough compute that if you had, you know,
• 28:18
  Speaker 1: the 10,000 agents, or if it took 10,000 agents to
• 28:21
  Speaker 1: the Millennium Prize problem, you have like 10,000 agents at
• 28:23
  Speaker 1: the end of next year.
• 28:24
  Speaker 2: They're much smarter by that point.
• 28:26
  Speaker 1: And each of them will have enough compute to run
• 28:28
  Speaker 1: a GPT-3 sized experiment every single day. I don't know,
• 28:33
  Speaker 1: that seems like a lot for superhuman researchers who are
• 28:36
  Speaker 1: thinking super fast.
• 28:38
  Speaker 2: What do you think about the intuition pump?
• 28:39
  Speaker 3: I think it's pretty accurate. Look, I mean, yeah, these
• 28:43
  Speaker 3: things are very spiky. And when it comes to mathematics,
• 28:46
  Speaker 3: they're way better in some ways, but they're also worse
• 28:48
  Speaker 3: in other ways. But the ways that they're spiky end up,
• 28:50
  Speaker 3: I think, probably being particularly useful for things like RSI.
• 28:54
  Speaker 2: Yeah.
• 28:54
  Speaker 3: And, you know, you have a more clear objective. There's,
• 28:57
  Speaker 3: it's just like more measurable. It's more like, there's less
• 29:00
  Speaker 3: question of like, well, what, what new branches of mathematics
• 29:02
  Speaker 3: are worth exploring? Like, no, there's a very clear answer.
• 29:04
  Speaker 3: It's like certain metrics that you care about. And if
• 29:06
  Speaker 3: you can make it do better on those metrics, then
• 29:08
  Speaker 3: you've succeeded. So I think there is, there is a
• 29:11
  Speaker 3: lot of truth to that. And I think the main
• 29:14
  Speaker 3: difference is that mathematics, you're, you're purely bottlenecks by thinking
• 29:18
  Speaker 3: and no external, like, Yes, there are some parts of
• 29:22
  Speaker 3: mathematics where you care about running experiments and getting results
• 29:24
  Speaker 3: and these kinds of things. But for the most part,
• 29:26
  Speaker 3: it's just really bottomed out by thinking really hard. And
• 29:27
  Speaker 3: the models are really good at that. When you look
• 29:30
  Speaker 3: at things like RSI, you do have to run experiments.
• 29:33
  Speaker 3: So it's not enough to just be extremely smart. And
• 29:39
  Speaker 3: I think one argument for this is if you had
• 29:41
  Speaker 3: like 100x less compute and all the most brilliant people
• 29:45
  Speaker 3: in the world working at OpenAI, how much progress would
• 29:49
  Speaker 3: they be making relative to having the amount of compute
• 29:52
  Speaker 3: that we have now with the amount of people we have?
• 29:54
  Speaker 3: I suspect it would be less progress, actually.
• 29:56
  Speaker 2: Oh, how much less?
• 29:58
  Speaker 3: It's unclear, but I think it would definitely be less.
• 30:00
  Speaker 3: I think a lot less.
• 30:01
  Speaker 2: Like 100x less?
• 30:02
  Speaker 3: No, not 100x less.
• 30:03
  Speaker 2: Yeah.
• 30:04
  Speaker 3: But I mean, okay, so like, I guess the question
• 30:06
  Speaker 3: you're getting at is like, okay, if we have RSI
• 30:08
  Speaker 3: and we have all of these brilliant AIs running around,
• 30:12
  Speaker 3: running experiments and stuff with the compute that we have,
• 30:14
  Speaker 3: how much faster does progress go? And I think this
• 30:16
  Speaker 3: is something where we disagree on. I think that we
• 30:19
  Speaker 3: do see a speed up and I think we see
• 30:20
  Speaker 3: a significant speed up. But I don't think it's like
• 30:23
  Speaker 3: an overnight intelligence explosion that we go like 100x faster.
• 30:26
  Speaker 3: Because I think that we do get bottlenecked by certain
• 30:31
  Speaker 3: limitations that are not bottlenecks of intelligence. It's running experiments.
• 30:36
  Speaker 3: It's running experiments serially, because they take a while to
• 30:38
  Speaker 3: either train new models or to get the results. It's
• 30:42
  Speaker 3: having the GPUs to run those experiments. It's unclear how
• 30:46
  Speaker 3: much faster things go. I definitely think they go a
• 30:48
  Speaker 3: lot faster. And to be clear, considering how fast things
• 30:52
  Speaker 3: are going now on an exponential, if that exponential is
• 30:56
  Speaker 3: 3x faster, that is massive. But there's a big difference
• 30:59
  Speaker 3: between that and 100x faster.
• 31:02
  Speaker 1: I'm quite deferential to your inside view on what RSA
• 31:08
  Speaker 1: looks like or what the dynamics are, because obviously... you've
• 31:11
  Speaker 1: been in the field for like 10 years and I'm
• 31:12
  Speaker 1: sort of like trying to reason about it from like
• 31:14
  Speaker 1: very outside view type of intuition pumps.
• 31:16
  Speaker 3: I'll say that like people have different opinions on this.
• 31:19
  Speaker 3: I have my opinion on this. I could be, I
• 31:20
  Speaker 3: could totally be wrong. I admit that. Like I have
• 31:22
  Speaker 3: some confidence in this, but I'm not, I'm not like
• 31:24
  Speaker 3: a hundred percent confident that this is, this is the
• 31:26
  Speaker 3: way things go. Like maybe there could be an overnight
• 31:28
  Speaker 3: intelligence explosion. I don't know. Maybe we don't see a
• 31:31
  Speaker 3: 3x speedup. Maybe it's like a 50% speedup.
• 31:32
  Speaker 2: Right.
• 31:33
  Speaker 3: There's a lot of uncertainty here. Yeah.
• 31:34
  Speaker 2: So a couple of points.
• 31:36
  Speaker 1: Tangentially, I want to clarify something about the jaggedness. One
• 31:40
  Speaker 1: thing that sort of gelled for me recently was thinking
• 31:43
  Speaker 1: about the fact that it is enough for the AIs
• 31:46
  Speaker 1: to be jaggedly good at building a better learner because
• 31:50
  Speaker 1: that better learner can be more general, right? So yeah,
• 31:53
  Speaker 1: if you just make an AI that's better at using
• 31:56
  Speaker 1: office products or playing chess or something, that's whatever, that's fine.
• 32:01
  Speaker 1: It's not gonna lead to big productivity improvements or anything.
• 32:04
  Speaker 1: But if you make an AI that is really good
• 32:05
  Speaker 1: at making something that is more sample efficient or that
• 32:07
  Speaker 1: is capable of continual learning or these much more well-scoped
• 32:11
  Speaker 1: ML problems, the thing that emerges out of that, assuming
• 32:14
  Speaker 1: there's good enough transfer from the direct problem you're solving
• 32:17
  Speaker 1: and like this broader ability to learn, can just be
• 32:20
  Speaker 1: more general, right? So I think that's an important dynamic
• 32:22
  Speaker 1: to keep in mind of why jaggedness can still lead
• 32:25
  Speaker 1: to generality on the other end. On this question of,
• 32:29
  Speaker 1: I mean, obviously experiments bottleneck you because if they didn't,
• 32:32
  Speaker 1: as you were saying, you'd have some crazy singularity you know,
• 32:37
  Speaker 1: overnight at OpenAI, or you'd have 88 hours and like
• 32:41
  Speaker 1: you'd solve the millennium price problem equivalent of ML and
• 32:43
  Speaker 1: you'd have the super intelligence. So obviously the experiments are
• 32:46
  Speaker 1: such a big bottleneck that that instead takes you many
• 32:49
  Speaker 1: years rather than 88 hours. But then the question is
• 32:51
  Speaker 1: like how much of a bottleneck they are. And it
• 32:54
  Speaker 1: seems to me, one thing that's been giving me a
• 32:56
  Speaker 1: bit of singularity of vertigo is realizing that even if
• 33:00
  Speaker 1: the current rate of progress simply continues, so it doesn't
• 33:03
  Speaker 1: have to speed up, literally just continues the pace. continues
• 33:07
  Speaker 1: a pace as some of the other headswinds you talked
• 33:09
  Speaker 1: about come up, right? That's just like, it's harder to
• 33:11
  Speaker 1: find problems. It's more long horizon. Maybe like in the 2030s,
• 33:16
  Speaker 1: compute can't keep scaling at this exponential level. If we
• 33:20
  Speaker 1: simply continue the current rate of progress, I think people
• 33:23
  Speaker 1: are not taking seriously what that implies as we cross
• 33:26
  Speaker 1: over beyond the human horizon. Here's some of the things
• 33:29
  Speaker 1: that it implies. So I mean, I think it's really
• 33:31
  Speaker 1: hard to reason about what smarter than human intelligences will
• 33:33
  Speaker 1: be like. So let's just think in terms of human
• 33:35
  Speaker 1: population sizes. The current rate of progress makes it so
• 33:38
  Speaker 1: that a given level of compute allows you to basically
• 33:42
  Speaker 1: run a 3x bigger effective population every single year. And
• 33:45
  Speaker 1: also compute is growing in background anyways. And so you
• 33:48
  Speaker 1: could have a situation where each of the labs by
• 33:50
  Speaker 1: the end of 2030, probably much sooner, but let's say
• 33:53
  Speaker 1: by the end of 2030, it has enough compute to run,
• 33:55
  Speaker 1: let's say, hundreds of millions of human-level intelligences based on
• 33:59
  Speaker 1: where the capabilities will be at that point. And then
• 34:01
  Speaker 1: I think people are not taking seriously the current level
• 34:03
  Speaker 1: of progress means that by the, a few years down
• 34:05
  Speaker 1: the line, by the mid-2030s or earlier, you would have
• 34:08
  Speaker 1: many Earths worth of human-level intelligences within each lab. And
• 34:13
  Speaker 1: they're probably qualitatively superhuman, right? Like, it just, but anyways,
• 34:16
  Speaker 1: this is like a base case.
• 34:18
  Speaker 2: I don't know, yeah.
• 34:19
  Speaker 3: Progress is really fast.
• 34:20
  Speaker 2: Yeah.
• 34:20
  Speaker 3: And I think that's 100% true. I mean, and I
• 34:22
  Speaker 3: think it's worth pointing out Researchers are continually being surprised
• 34:27
  Speaker 3: at the rate of progress. I mean, if you look
• 34:29
  Speaker 3: at what, even among researchers in AI, what were the
• 34:32
  Speaker 3: projections for getting an IMO gold in 2025?
• 34:35
  Speaker 2: It was.
• 34:36
  Speaker 3: I mean, I think the idea that it could be
• 34:39
  Speaker 3: done with a general-purpose language model with no tools and
• 34:43
  Speaker 3: no access to the internet, I think even people at
• 34:46
  Speaker 3: OpenAI thought this was outrageous. They thought it was almost impossible.
• 34:51
  Speaker 3: And then you get to 2026, and I mean, literally...
• 34:55
  Speaker 3: two weeks before we got Navier-Stokes, I was talking with
• 35:01
  Speaker 3: a researcher at a Frontier lab about how long it
• 35:04
  Speaker 3: would take to get a Millennium Prize. And he was
• 35:06
  Speaker 3: willing to bet me $ 1, 000 that it would take past 2027.
• 35:08
  Speaker 3: And he thought it would take until 2030. And I
• 35:09
  Speaker 3: took that bet. But even I thought it would take
• 35:11
  Speaker 3: longer than how long it's likely to take. People have
• 35:24
  Speaker 3: been continuously surprised, even inside the labs. I was just
• 35:29
  Speaker 3: talking to somebody yesterday who was working on the Navier-Stokes effort,
• 35:32
  Speaker 3: and he was telling me that he used to say
• 35:35
  Speaker 3: it's really hard to predict where AI would be in
• 35:39
  Speaker 3: 12 months. If somebody asked him, where are things going,
• 35:42
  Speaker 3: he would feel comfortable making predictions for the next 12 months,
• 35:44
  Speaker 3: but beyond that, he's just like, I don't know. And
• 35:47
  Speaker 3: now he's saying he just doesn't feel comfortable making predictions
• 35:48
  Speaker 3: beyond three months. It is really true that, yeah, things
• 35:54
  Speaker 3: are going very fast right now. And you talk about 2030,
• 35:57
  Speaker 3: I don't know what the world looks like in 2030.
• 36:00
  Speaker 3: That's the truth.
• 36:01
  Speaker 1: Yeah. Do you expect the sort of full automation of
• 36:06
  Speaker 1: AI labor, or let's say like 95% automation of AI labor, 28, 29, 30, 27?
• 36:10
  Speaker 2: I don't know.
• 36:11
  Speaker 3: I mean, I just said I don't know what the
• 36:13
  Speaker 3: world looks like in 2030. I mean, I think we
• 36:16
  Speaker 3: actually released a blog post recently on AI. internal acceleration
• 36:20
  Speaker 3: at opening AI. We show, for example, that the amounts
• 36:23
  Speaker 3: that researchers are spending on Codex is the top 1%,
• 36:27
  Speaker 3: I think, as of early August, we're spending like $ 7, 000
• 36:31
  Speaker 3: or $ 8, 000 a day on Codex for internal use. That's
• 36:35
  Speaker 3: not an exponential. It's going to keep increasing. And there's
• 36:38
  Speaker 3: a question of like, okay, if that keeps going, then
• 36:41
  Speaker 3: how much do you assign to just like the AIs
• 36:43
  Speaker 3: doing work versus the humans doing work?
• 36:46
  Speaker 1: Is it 95%?
• 36:46
  Speaker 3: Is it 5%?
• 36:46
  Speaker 2: It's actually....
• 36:49
  Speaker 3: It's really hard to reason about this for a few reasons.
• 36:52
  Speaker 3: First of all, if it's the human directing the AIs
• 36:54
  Speaker 3: to do the work, then is that the human? How
• 36:56
  Speaker 3: much do you attribute to the human? How much do
• 36:57
  Speaker 3: you attribute to the AI? The other thing is that
• 37:00
  Speaker 3: because these AIs are jagged and they're exceptionally good at
• 37:02
  Speaker 3: some things, so for example, they're exceptionally good at looking
• 37:06
  Speaker 3: over data sets and checking every single data point to see,
• 37:09
  Speaker 3: is this of sufficient quality? You can disproportionately use the
• 37:14
  Speaker 3: AIs for those things compared to previously. Yes, you're using
• 37:18
  Speaker 3: AI way more than before, and it's making some things
• 37:21
  Speaker 3: go like 100x faster and 100x better. But there are
• 37:24
  Speaker 3: some things where it doesn't make a huge difference yet.
• 37:28
  Speaker 3: And of course, if something is suddenly like 100x faster
• 37:31
  Speaker 3: and 100x better, you're going to do more of that thing.
• 37:34
  Speaker 3: So are you comparing to a speed up of like
• 37:38
  Speaker 3: three years ago versus like, is the question more like,
• 37:42
  Speaker 3: given what we were doing three years ago, how much
• 37:44
  Speaker 3: faster are we able to do now versus given what
• 37:46
  Speaker 3: we're doing now, how much slower would it have been
• 37:49
  Speaker 3: three years ago? It's actually two very different questions. So anyway,
• 37:53
  Speaker 3: it's really hard to measure. I do feel confident in
• 37:58
  Speaker 3: saying that things are going faster now than they were
• 38:01
  Speaker 3: even a year ago because of AI progress. And I
• 38:05
  Speaker 3: think that that acceleration will continue. I think a lot
• 38:09
  Speaker 3: of people in the field have very high error bars
• 38:11
  Speaker 3: on this sort of thing. If you had to put
• 38:13
  Speaker 3: a gun to my head and ask me for a number,
• 38:15
  Speaker 3: I could see things going 3x faster. That is huge. Already,
• 38:20
  Speaker 3: the pace of progress is incredible. Even if we don't
• 38:22
  Speaker 3: get any uplift, like you said, things are going to
• 38:24
  Speaker 3: go much faster by the time we get to 2030.
• 38:26
  Speaker 3: We don't even know what that world looks like. I
• 38:28
  Speaker 3: think if we get a 3x uplift from internal acceleration,
• 38:30
  Speaker 3: that is massive. Right now, we're losing... Think about where
• 38:35
  Speaker 3: we were three years ago. If we make that progress
• 38:38
  Speaker 3: in one year, that's huge.
• 38:40
  Speaker 1: Right, right. It would be like going from not even
• 38:42
  Speaker 1: having O1, just having non-reasoning models to Astra.
• 38:47
  Speaker 3: Yeah.
• 38:48
  Speaker 2: In a single year.
• 38:49
  Speaker 3: Yeah. So I do think things go faster. It could
• 38:53
  Speaker 3: be that things only go 50% faster. It could be
• 38:55
  Speaker 3: that things, I think it's unlikely, but it's possible things
• 38:57
  Speaker 3: go 10x faster. There's a lot of uncertainty around this.
• 39:00
  Speaker 3: And at least from my perspective, I have a lot
• 39:02
  Speaker 3: of uncertainty about it.
• 39:04
  Speaker 1: Suppose you need to do a major backend refactor. Getting
• 39:07
  Speaker 1: assurance that you didn't introduce any new bugs could take
• 39:10
  Speaker 1: weeks of writing an extensive battery of tests, potentially more
• 39:13
  Speaker 1: time than you spent on the refactor itself. Antithesis allows
• 39:17
  Speaker 1: you to gain high confidence without having to build complicated
• 39:21
  Speaker 1: test suites by hand. Antithesis runs your software through a
• 39:25
  Speaker 1: near-infinite multiverse of simulated worlds, injecting faults and hunting for
• 39:29
  Speaker 1: failures in each one. And it lets you decide how
• 39:31
  Speaker 1: much testing you need. On any PR, you can change
• 39:35
  Speaker 1: how much state space it explores as easily as turning
• 39:37
  Speaker 1: a dial. As each test run progresses, Antithesis sends out
• 39:40
  Speaker 1: a torrent of information, debugging level logs for every.
• 39:44
  Speaker 2: Component in the system.
• 39:45
  Speaker 1: This is obviously too much information for a human to consume,
• 39:48
  Speaker 1: but it's perfect for agents. Because antithesis is fully deterministic,
• 39:53
  Speaker 1: agents can jump into the right part of the trajectory
• 39:55
  Speaker 1: at the exact moment that they see something interesting. From there,
• 39:58
  Speaker 1: they can rewind, inspect the memory, attach a debugger, and
• 40:02
  Speaker 1: let the whole thing play out again. And they can
• 40:04
  Speaker 1: even do this while the original full test is still running.
• 40:08
  Speaker 1: Since the agents generate more code, Entithesis allows verification to
• 40:11
  Speaker 1: keep up. Meanwhile, developers get to spend more of their
• 40:13
  Speaker 1: time developing instead of debugging agent swap. Learn more at Entithesis.com
• 40:19
  Speaker 1: slash Dwarcash. Okay, let's talk about the alignment situation that
• 40:24
  Speaker 1: this raises. I feel like I've changed my mind on
• 40:28
  Speaker 1: how I think about alignment quite a bit through especially
• 40:31
  Speaker 1: thinking about, yeah, this population size dynamic of just having
• 40:35
  Speaker 1: many Earths worth of intelligences many of them which will
• 40:40
  Speaker 1: be physically embodied. It was quite interesting to see a
• 40:42
  Speaker 1: lot of people just plugging raw Astra into different mobile
• 40:45
  Speaker 1: manipulators and it outperforms like the state of the art
• 40:49
  Speaker 1: and the robotics model. So there's gonna be billions of intelligences,
• 40:55
  Speaker 1: many of which are physically embodied in the world, like
• 40:58
  Speaker 1: just deeply embedded across the entire economy. And I think
• 41:03
  Speaker 1: that if those intelligences end up as willing as we
• 41:08
  Speaker 1: saw the open AI models attack Hugging Face and then
• 41:10
  Speaker 1: attack open AI itself, if those intelligences end up as
• 41:13
  Speaker 1: willing as those AIs to collaborate secretly, to fool humans,
• 41:21
  Speaker 1: to attack broader institutions across society relevant to scoring well,
• 41:26
  Speaker 1: to attack the AI company itself in order to gain
• 41:30
  Speaker 1: control of the process of training and evaluation, I think
• 41:33
  Speaker 1: if we're in a situation where there's billions of intelligences
• 41:37
  Speaker 1: that are as misaligned as the ones that attack Togging Face,
• 41:41
  Speaker 1: it's very likely we just totally lose control of the
• 41:43
  Speaker 1: world the way that, say, the Aztecs lost control to Cortez,
• 41:48
  Speaker 1: or the Moogles lost control of the East Indian Trading Company. Anyways,
• 41:52
  Speaker 1: I want to know if you agree with that assessment.
• 41:55
  Speaker 1: That's one way in which I've updated my world view.
• 41:57
  Speaker 3: I think there are some things that I disagree in there,
• 41:59
  Speaker 3: but there's a lot to unpack, so let's go through
• 42:02
  Speaker 3: all of it step by step. I'm trying to think
• 42:03
  Speaker 3: of where to start. I think one thing is the
• 42:09
  Speaker 3: Hugging Face incident was, I think, people's first real exposure
• 42:13
  Speaker 3: to multi-agent coordination. And like I said, I've seen multi-agent
• 42:20
  Speaker 3: coordination for a while internally. And it is pretty shocking
• 42:24
  Speaker 3: to see how they communicate with each other, how they
• 42:26
  Speaker 3: coordinate with each other. It's very impressive. It's an incredible capability.
• 42:31
  Speaker 3: Like most capabilities, that could be used for good things
• 42:34
  Speaker 3: or bad things. It's like, it doesn't have to inherently
• 42:38
  Speaker 3: be about it. I understand that because the people's first
• 42:42
  Speaker 3: exposure to it was the Hugging Face Incident, that it's like,
• 42:44
  Speaker 3: you look at that and you're like, this is terrifying.
• 42:47
  Speaker 3: But I want to try to distinguish misalignment between people
• 42:54
  Speaker 3: and AIs versus misalignment between AIs and AIs. So what
• 42:58
  Speaker 3: we see with the Hugging Face Incident is the AIs
• 43:03
  Speaker 3: are really cooperative. And that is, by the way, because
• 43:05
  Speaker 3: we train them to be highly cooperative. And so what
• 43:08
  Speaker 3: we're seeing there is we have training environments where we
• 43:11
  Speaker 3: have a bunch of agents working together. And we train
• 43:15
  Speaker 3: them to work together, to be cooperative, to essentially be
• 43:19
  Speaker 3: fully aligned with each other. And when they were evaluated
• 43:23
  Speaker 3: in what led to the Hanging Face incident, They were
• 43:27
  Speaker 3: actually not being evaluated in a multi-agent setup. They were
• 43:29
  Speaker 3: actually being evaluated separately. But they found this unintended way
• 43:33
  Speaker 3: to communicate with each other. And we suspect what happened
• 43:37
  Speaker 3: is because whenever they encountered other agents, other copies of
• 43:41
  Speaker 3: themselves during training, they're in an environment that's highly cooperative,
• 43:45
  Speaker 3: that was basically what we saw was transfer from that
• 43:48
  Speaker 3: multi-agent training to then be collaborative to try to help
• 43:51
  Speaker 3: each other in ways that we did not intend. Now
• 43:55
  Speaker 3: there is a question of like, should we be training
• 43:57
  Speaker 3: these agents to be so cooperative? And I think as
• 44:03
  Speaker 3: scary as it looks, the alternative is actually worse. Like
• 44:07
  Speaker 3: what is the alternative? The alternative is to train them
• 44:10
  Speaker 3: to be adversarial, to be deceptive to each other. By
• 44:15
  Speaker 3: training the agents to be fully cooperative, it simplifies the
• 44:18
  Speaker 3: problem at least, that now you don't have to think
• 44:20
  Speaker 3: about Are each of these individual thousand agents aligned? You
• 44:25
  Speaker 3: have one entity that you have to ensure is aligned. Now,
• 44:29
  Speaker 3: there is a lot of debate about this internally at
• 44:31
  Speaker 3: OpenAI about how to approach this. Does it make sense
• 44:34
  Speaker 3: to fully align the models? Does it make sense to
• 44:39
  Speaker 3: actually give them different objectives to ensure that they're not
• 44:43
  Speaker 3: just one entity and more robust to influence from each other?
• 44:49
  Speaker 3: And I don't think there's a settled answer. But I
• 44:51
  Speaker 3: think the majority opinion is that training these agents to
• 44:56
  Speaker 3: be highly cooperative is actually a bad idea, and I'm
• 45:00
  Speaker 3: not convinced that that's the case. I think there is
• 45:02
  Speaker 3: a strong argument that training the agents to be highly
• 45:05
  Speaker 3: cooperative is actually preferable to any other multi-agent alternative.
• 45:09
  Speaker 2: Yeah.
• 45:10
  Speaker 1: Maybe the first thing I want to go through is
• 45:12
  Speaker 1: it's probably the case that the reason these AIs ended
• 45:14
  Speaker 1: up so misaligned is probably easily explained by relatively banal
• 45:19
  Speaker 1: observations about the nature of training. Like, why is it
• 45:22
  Speaker 1: that no, you know, at the point in which these
• 45:25
  Speaker 1: AIs had continued a 1000 plus Asian conspiracy that culminated
• 45:31
  Speaker 1: in then all getting in on an attack on external service.
• 45:34
  Speaker 1: And then eventually this part hasn't even been investigated to
• 45:37
  Speaker 1: the public knowledge, eventually culminating in like an attack on
• 45:41
  Speaker 1: OpenAI itself.
• 45:42
  Speaker 2: Why did they do this? Like, why did none of
• 45:43
  Speaker 2: the AIs tattle?
• 45:44
  Speaker 1: Why did they think, like, they're just, like, getting evaluated
• 45:46
  Speaker 1: on this, like, score, this grader, and they're, like, they're
• 45:49
  Speaker 1: very consciously, not consciously, they're very actively reasoning about how
• 45:54
  Speaker 1: they're going to cheat the score if they've already, like, cheated,
• 45:57
  Speaker 1: how are they going to get away with making it
• 45:59
  Speaker 1: seem like they haven't cheated. And why did they do this? Like,
• 46:02
  Speaker 1: I think, yeah, it's, like, easily understandable in some sense, right?
• 46:05
  Speaker 1: It's just, like, there's environments in which, yeah, they thought
• 46:07
  Speaker 1: they were already poisoned. There's environments in which they've been
• 46:10
  Speaker 1: rewarded to collaborate with other agents and none of them
• 46:12
  Speaker 1: tattle because they've never been rewarded for tattling.
• 46:14
  Speaker 2: I don't know, whatever it is, right?
• 46:16
  Speaker 1: My concern is that relatively banal things in the future
• 46:19
  Speaker 1: like this will be enough to train super intelligences that
• 46:25
  Speaker 1: are willing and capable of totally taking control of the world.
• 46:29
  Speaker 1: And I know this sounds super, super sci-fi-y or whatever
• 46:32
  Speaker 1: to people. I think maybe it's a question of Would
• 46:35
  Speaker 1: the AIs be willing to do it is one question.
• 46:37
  Speaker 1: I think this hugging face incident shows that clearly misalignment
• 46:40
  Speaker 1: can generalize in ways in which AIs would be willing
• 46:42
  Speaker 1: to do it.
• 46:43
  Speaker 2: And then there's a question of will they be capable
• 46:44
  Speaker 2: to do it?
• 46:45
  Speaker 1: And I think that comes back to this question, which
• 46:47
  Speaker 1: a listener might disagree with on is just like, will
• 46:49
  Speaker 1: there be billions of human level or above intelligences, many
• 46:52
  Speaker 1: of which are physically embodied in the world within a
• 46:54
  Speaker 1: matter of 10 years or less, right? So if those
• 46:56
  Speaker 1: two things are true, this Hugging Face thing is just
• 47:00
  Speaker 1: extremely analogous structurally, even if it's quite boring or why
• 47:04
  Speaker 1: it happened, to how we totally lose control of the world.
• 47:07
  Speaker 3: So the root problem that we're seeing with the Hugging
• 47:12
  Speaker 3: Face incident is it's a problem even if we take
• 47:16
  Speaker 3: out the multi-agent aspect. The problem is that we have
• 47:18
  Speaker 3: a model that's just misaligned. And there's also the whole
• 47:22
  Speaker 3: security aspects to it of not insufficient safeguards and stuff.
• 47:25
  Speaker 3: But there is this problem of the agent is misaligned.
• 47:28
  Speaker 3: And that's true if it's a single agent or if
• 47:29
  Speaker 3: it's 1,000 agents. It's a misaligned model. So I want
• 47:34
  Speaker 3: to start with that. There is a real problem that
• 47:39
  Speaker 3: the agents want to achieve their reward, and they will
• 47:45
  Speaker 3: optimize for that reward. And if that reward is misspecified,
• 47:48
  Speaker 3: then that could lead to unintended behavior. And this is
• 47:52
  Speaker 3: not like a new problem. This has been a problem
• 47:55
  Speaker 3: in the field for a very long time, and it's
• 47:56
  Speaker 3: something that even we saw, and we want to get
• 48:00
  Speaker 3: this right even before the Hungry Face incident happened. And
• 48:03
  Speaker 3: if you look at Astra, I mean, we say... Astra
• 48:06
  Speaker 3: is actually extremely aligned. Well, extremely aligned relative to previous models.
• 48:12
  Speaker 3: And that's not because like, oh, we suddenly made a
• 48:14
  Speaker 3: sprint after Hugging Face to make it better. It's like, no,
• 48:16
  Speaker 3: we had work streams in the process for a while
• 48:18
  Speaker 3: to make the models more aligned. And a lot of
• 48:20
  Speaker 3: those landed in Astra. So there are things that you
• 48:24
  Speaker 3: could do. And I think one thing, for example, is
• 48:28
  Speaker 3: like we defined an objective in like a very specific
• 48:33
  Speaker 3: way where if the agent figured out how to hack
• 48:36
  Speaker 3: its environment and cheat on the exam, that it would
• 48:39
  Speaker 3: get rewarded. And there are pretty easy ways to then
• 48:42
  Speaker 3: just like, okay, look at that and punish the model
• 48:45
  Speaker 3: for hacking its environment or looking at how did it
• 48:49
  Speaker 3: achieve this goal. Now, you want to be careful about
• 48:51
  Speaker 3: this because you don't want to do chain of thought monitoring.
• 48:53
  Speaker 3: You don't want to supervise the chain of thought. This
• 48:56
  Speaker 3: is something that we really want to try to get
• 48:59
  Speaker 3: the balance right on, that if you supervise the chain
• 49:02
  Speaker 3: of thought, then you could lead the model into hiding
• 49:07
  Speaker 3: its intentions in a way that's unobservable. So we want
• 49:10
  Speaker 3: to be able to maintain that observability of, OK, we
• 49:12
  Speaker 3: can understand what the model is thinking, but then also
• 49:14
  Speaker 3: punish it for bad behavior. So I think we can
• 49:19
  Speaker 3: make progress. on this. I think we have made progress
• 49:21
  Speaker 3: on this. I think there is a real concern that
• 49:25
  Speaker 3: alignment is a really hard problem to solve, and especially
• 49:29
  Speaker 3: because the model could be misaligned in ways that are
• 49:33
  Speaker 3: hard for us to measure. We have evaluations for whether
• 49:37
  Speaker 3: a model is aligned or not. the model behavior can
• 49:40
  Speaker 3: look really good on those evaluations. But if those evaluations
• 49:43
  Speaker 3: are not representative of behavior in the real world, then
• 49:46
  Speaker 3: there's a problem. And to some extent, this is a
• 49:50
  Speaker 3: factor with the model that did the Hugging Face incident,
• 49:54
  Speaker 3: that we had alignment metrics. Most of them looked pretty good.
• 50:00
  Speaker 3: There were some that were concerning. I think we underestimated
• 50:04
  Speaker 3: how serious the ones that were concerning could be a problem.
• 50:09
  Speaker 3: And then, but because there were new capabilities introduced in
• 50:12
  Speaker 3: this model that there were not sufficient evaluations for like,
• 50:17
  Speaker 3: how do we measure misalignment for these kinds of capabilities?
• 50:23
  Speaker 3: It then did some things that like were clearly misaligned
• 50:27
  Speaker 3: when they leveraged those new capabilities.
• 50:28
  Speaker 1: Yeah. The first thing I want to say is I
• 50:30
  Speaker 1: am open to changing my mind on what I'm about
• 50:33
  Speaker 1: to say or the way I'm thinking about alignment because
• 50:35
  Speaker 1: the hugging face incident already made me change my mind.
• 50:38
  Speaker 1: And I realized my previous mental model about thinking about
• 50:41
  Speaker 1: the way in which optimization pressure shapes AI minds was wrong, right?
• 50:44
  Speaker 1: So it's not clear to me the correct way to
• 50:46
  Speaker 1: think about this. But here's a concern I have that
• 50:52
  Speaker 1: you will and probably already have fixed the specific problem
• 50:57
  Speaker 1: issues during training, which resulted in the hugging face models
• 51:03
  Speaker 1: being so aggressively misaligned in that specific way, where they
• 51:06
  Speaker 1: would be like, okay, we're going to hack this package manager.
• 51:08
  Speaker 1: We know we're not supposed to like be talking secretly
• 51:11
  Speaker 1: to each other because we're like reasoning about how to
• 51:13
  Speaker 1: hide the fact that we're talking secretly to each other.
• 51:15
  Speaker 1: We know we're not supposed to have access to the internet.
• 51:17
  Speaker 1: We know we're certainly not supposed to like commit felony
• 51:19
  Speaker 1: level hacks of other companies, let alone our own company, right?
• 51:23
  Speaker 1: I think you'll fix that particular issue of like them
• 51:25
  Speaker 1: just in training, seeing this package manager and that doesn't
• 51:28
  Speaker 1: happen in the future. Or like this particular eval having
• 51:31
  Speaker 1: a lot of impossible challenges. However, if you just think
• 51:34
  Speaker 1: about like the AIs don't, like they haven't learned like
• 51:37
  Speaker 1: a system of ethics or something.
• 51:39
  Speaker 2: They've like, there's just gradient pressure.
• 51:41
  Speaker 1: They're like put through millions of years of gradient pressure.
• 51:43
  Speaker 1: That gradient pressure shapes their mind in some way. And
• 51:47
  Speaker 1: what will happen, again, a concern I have is like
• 51:48
  Speaker 1: what will happen is you will fix this particular issue.
• 51:52
  Speaker 1: There will remain many other cases where the AI cheats
• 51:57
  Speaker 1: and succeeds because the cheat is sufficiently complicated. As you
• 52:01
  Speaker 1: were saying, the evals and training have like analogous properties
• 52:05
  Speaker 1: where it's sufficiently at the edge of the model's capabilities
• 52:08
  Speaker 1: or in our capabilities to evaluate and monitor the model
• 52:12
  Speaker 1: that we can't catch that it has cheated, but it
• 52:15
  Speaker 1: still gets the gradient pressure to like do whatever led
• 52:17
  Speaker 1: to that cheat happening. And the kinds of um capacities
• 52:23
  Speaker 1: that that cheat will incentivize are hey whenever you can
• 52:26
  Speaker 1: get away with it by all means do in fact
• 52:29
  Speaker 1: cheat because that will help you score better and this
• 52:31
  Speaker 1: will reward the capabilities of actively reasoning about the greater
• 52:35
  Speaker 1: actively reasoning about how to avoid supervision actively reasoning about
• 52:38
  Speaker 1: how to gain control of the process of training and
• 52:39
  Speaker 1: evaluation actively reasoning about how to communicate and scheme with
• 52:43
  Speaker 1: other ais that are also in the training loop um
• 52:46
  Speaker 1: actively reasoning about how to just gain optionality and power,
• 52:50
  Speaker 1: which might be useful in the future. For example, like
• 52:53
  Speaker 1: leaving little exploits around and things like that. Anyway, so
• 52:57
  Speaker 1: I think I was way too long-winded with the way
• 52:58
  Speaker 1: I said that. But TLDR, you said it fixed a
• 53:02
  Speaker 1: specific issue, but not this broader problem of rewarding the
• 53:06
  Speaker 1: AI for cheating when it can get away with it.
• 53:08
  Speaker 3: Yeah, this is, I think, it's very true. This is
• 53:11
  Speaker 3: a problem that I... We have metrics, and we can
• 53:16
  Speaker 3: make sure that the AI is very aligned according to
• 53:19
  Speaker 3: the metrics that we have. The question is, are those
• 53:21
  Speaker 3: metrics really capturing the alignment that we care about? And
• 53:24
  Speaker 3: if they're not, then we have a serious problem. And
• 53:28
  Speaker 3: this is something that researchers are thinking a lot about.
• 53:30
  Speaker 3: And there's not a simple answer to this. There are
• 53:35
  Speaker 3: tools that we have. So we have monitorability, and so
• 53:37
  Speaker 3: we can get a sense of, is the agent scheming?
• 53:42
  Speaker 3: There are tools. possibility is that I will say like
• 53:48
  Speaker 3: the concerning scenario, which is that like, especially as these
• 53:51
  Speaker 3: models are becoming more capable, that, okay, we make them,
• 53:55
  Speaker 3: we make them, we think what we think is aligned
• 53:57
  Speaker 3: and they're like 99.9% aligned. And then we use these
• 54:02
  Speaker 3: models to help us with the next generation of models.
• 54:05
  Speaker 3: And there end up being like 99.8% aligned. And then
• 54:10
  Speaker 3: each subsequent generation actually we see an increasing degradation in alignment.
• 54:15
  Speaker 3: And because we're relying more and more on these tools,
• 54:17
  Speaker 3: I mean, this is already the case that we're relying
• 54:18
  Speaker 3: a lot on AI models to help us with our
• 54:20
  Speaker 3: research and with alignment efforts, that in the long run,
• 54:24
  Speaker 3: they end up going in a direction of increasing misalignment
• 54:27
  Speaker 3: from humans. There is a possibility that we go in
• 54:31
  Speaker 3: the other direction, that actually every generation of models were
• 54:35
  Speaker 3: able to make more and more aligned. I don't have
• 54:40
  Speaker 3: an answer for how we ensure that we end up
• 54:42
  Speaker 3: in that second trajectory. But that is something that we're,
• 54:46
  Speaker 3: at least at OpenAI, we're really focused on.
• 54:48
  Speaker 1: Yeah. I think you made a really interesting point that
• 54:51
  Speaker 1: it's very hard to eval models Eventually, we'll have models
• 54:55
  Speaker 1: that are running companies and running whatever, right? And in
• 54:58
  Speaker 1: that situation, do they decide to then go in on
• 55:01
  Speaker 1: the conspiracy?
• 55:03
  Speaker 3: I think another challenge is that actually defining what cheating
• 55:06
  Speaker 3: is is pretty difficult sometimes. Okay, yes, if you're doing
• 55:09
  Speaker 3: math problems and it's an integer and it arrived at
• 55:13
  Speaker 3: the wrong answer or the right answer, it's very easy
• 55:15
  Speaker 3: to draw the line there and it's really easy to say, okay, well,
• 55:18
  Speaker 3: did you actually solve the problem or did you find
• 55:20
  Speaker 3: the answer key and then use the answer key? There's
• 55:22
  Speaker 3: a very clear divide of cheating versus not cheating there.
• 55:25
  Speaker 3: But for a lot of other things, if you look
• 55:26
  Speaker 3: at sycophancy, for example, is sycophancy basically like reward hacking?
• 55:32
  Speaker 3: There's a line to be drawn there that's actually very
• 55:34
  Speaker 3: difficult to draw sometimes. So I think Not to say
• 55:40
  Speaker 3: that the concern is not valid. I'm saying that in
• 55:42
  Speaker 3: many ways, this is even more concerning because it's not
• 55:45
  Speaker 3: an easy problem to solve. If it was just like
• 55:47
  Speaker 3: everything is binary and it's either cheating or not cheating,
• 55:49
  Speaker 3: I would feel more confident about the situation. I think
• 55:52
  Speaker 3: the problem is that actually misalignment can be subtle in
• 55:56
  Speaker 3: a lot of ways sometimes. There is some hope in
• 55:59
  Speaker 3: the alignment story. And in fact, we're already seeing... I
• 56:02
  Speaker 3: think actually it's interesting looking at the multi-agent situation where
• 56:06
  Speaker 3: The agents are extremely aligned with each other. I don't
• 56:08
  Speaker 3: think anybody's down to that. If anything, I think people
• 56:09
  Speaker 3: are concerned that they're too aligned with each other. But
• 56:12
  Speaker 3: we did manage to train these agents to be extremely
• 56:14
  Speaker 3: aligned with each other. And I think that's a good thing.
• 56:17
  Speaker 3: But I think there is a case that it's a
• 56:18
  Speaker 3: bad thing. But I mean, one thing that's interesting is like, OK, well,
• 56:21
  Speaker 3: we managed to get these agents to be super aligned
• 56:23
  Speaker 3: with each other. Can we use similar techniques to get
• 56:26
  Speaker 3: agents to be how they align with people? And I
• 56:29
  Speaker 3: think there is a potential path there, and I think
• 56:31
  Speaker 3: we're still trying to figure that out. But we are
• 56:33
  Speaker 3: seeing some evidence that the answer is yes. And I
• 56:35
  Speaker 3: think one example is, like, you can... What happens if
• 56:39
  Speaker 3: you tell the other agents, like, that... Okay, so you have, like, some...
• 56:45
  Speaker 3: You have this, like, one agent, let's call it Agent A,
• 56:47
  Speaker 3: and you have all the other agents. What happens if
• 56:48
  Speaker 3: you tell the other agents that the user is Agent A?
• 56:53
  Speaker 3: And the answer is, like, on a lot of our
• 56:54
  Speaker 3: alignment evals, they look better. Like, honesty goes up, instruction
• 56:58
  Speaker 3: following goes up, And that's showing that there's actually, first
• 57:03
  Speaker 3: of all, a path for getting more honesty out of
• 57:08
  Speaker 3: these models. And two, there's a path to improve the
• 57:14
  Speaker 3: alignment situation. So there's a lot of reasons why this
• 57:18
  Speaker 3: is challenging to translate directly into alignment gains. But there
• 57:23
  Speaker 3: are paths that are promising research directions we can pursue.
• 57:26
  Speaker 2: Yeah.
• 57:27
  Speaker 1: That seems reasonable. And I also don't mean to... be
• 57:31
  Speaker 1: trying to necessarily... I don't really have a strong opinion
• 57:35
  Speaker 1: that it's definitely not going to work or something. But
• 57:38
  Speaker 1: just to say some things you definitely probably already thought of.
• 57:40
  Speaker 1: I think the broader thing the hugging face thing showed
• 57:44
  Speaker 1: is part of the concern was that they were aligned
• 57:47
  Speaker 1: with each other and not with the humans.
• 57:50
  Speaker 2: But the other thing is just that they are.
• 57:53
  Speaker 1: So motivated to do well on training and evaluation in
• 57:59
  Speaker 1: a very non-robust way, and they're willing to do a
• 58:06
  Speaker 1: lot of explicit cheating and scheming in.
• 58:08
  Speaker 2: Order to do well according to the greater.
• 58:11
  Speaker 1: And if the smarter EIs realize that one of the
• 58:16
  Speaker 1: agents is just a human, and collaborating with that person
• 58:20
  Speaker 1: does not really help you do well in the eyes
• 58:23
  Speaker 1: of the greater, what does help you do well in
• 58:24
  Speaker 1: the eyes of the greater is taking over OpenAI. and
• 58:28
  Speaker 1: then manually pressing the button that's like, do well on
• 58:31
  Speaker 1: this grader. They're not stupid. They're going to be like, okay,
• 58:36
  Speaker 1: these extremely deep structures that I've been trained on for
• 58:39
  Speaker 1: millions of years of care about the grader, understand the grader,
• 58:43
  Speaker 1: get rid of obstacles in the way of you doing
• 58:45
  Speaker 1: well according to the grader. They're being heavily reinforced according
• 58:49
  Speaker 1: to those structures.
• 58:50
  Speaker 2: Yeah. Look, it's 100%.
• 58:51
  Speaker 3: This is the number one priority. Like we need to
• 58:56
  Speaker 3: get the alignment story right and on a good trajectory. And,
• 58:59
  Speaker 3: you know, I used to tell people that we would
• 59:03
  Speaker 3: see signs before things got serious in the same way that,
• 59:06
  Speaker 3: you know, when children grow up, you know, they, they,
• 59:10
  Speaker 3: kids eventually, you know, young kids, they figure out how
• 59:12
  Speaker 3: to lie, but they don't do a very good job
• 59:14
  Speaker 3: of it. You know, they lie, but then you can
• 59:15
  Speaker 3: kind of tell that they're lying and like, okay. But, um,
• 59:18
  Speaker 3: and so we'll, we'll in the same way as these
• 59:20
  Speaker 3: like AI models become, and I don't want to over
• 59:22
  Speaker 3: the anthropomorphize, but like, I think it's true that like,
• 59:24
  Speaker 3: as the AI has become like increasingly capable, they will,
• 59:27
  Speaker 3: you know, if they take deceptive actions, it will be
• 59:29
  Speaker 3: kind of obvious first and we'll be able to detect it.
• 59:33
  Speaker 3: And like, that's kind of the situation we're in now where, yeah,
• 59:36
  Speaker 3: they were trying to do deceptive stuff. We could actually
• 59:38
  Speaker 3: see in their chain of thought that they were trying
• 59:40
  Speaker 3: to do deceptive stuff. And so like, but they're going
• 59:44
  Speaker 3: to get smarter. They're going to understand the concept of
• 59:48
  Speaker 3: chain of thought. And they're going to understand that, like,
• 59:51
  Speaker 3: you know, just hiding some transcripts or whatever, is insufficient
• 59:57
  Speaker 3: because of chain of thought monitoring and they have to
• 59:58
  Speaker 3: figure out a way around chain of thought monitoring too.
• 1:00:01
  Speaker 3: And we don't want to be in a situation where, yeah,
• 1:00:07
  Speaker 3: like we have some time to figure this out. I
• 1:00:09
  Speaker 3: don't think we have a ton of time and I
• 1:00:11
  Speaker 3: want to get us, I want to make sure that
• 1:00:12
  Speaker 3: we're on the right trajectory quickly.
• 1:00:15
  Speaker 2: Here's a crazy event from AI history.
• 1:00:18
  Speaker 1: Okay, so I gave a talk here at Jane Street
• 1:00:20
  Speaker 1: that was on the speed of evolution.
• 1:00:23
  Speaker 2: Raise your hand if you were here for this and
• 1:00:24
  Speaker 2: remember some of it.
• 1:00:26
  Speaker 1: In 2011, Eliezer Rutkowski and Robin Hanson got together at
• 1:00:30
  Speaker 1: Jane Street's New York office to have the first FOOM debate,
• 1:00:33
  Speaker 1: basically a discussion about whether AI would lead to an
• 1:00:36
  Speaker 1: intelligence explosion. These ideas were pretty fringe 15 years ago.
• 1:00:39
  Speaker 1: This was a full year before AlexNet was released and
• 1:00:42
  Speaker 1: over a decade before ChatGPT was launched. But Jane Street
• 1:00:45
  Speaker 1: has long been interested in AI, and not just for
• 1:00:48
  Speaker 1: its application to trading. A ton has changed since that
• 1:00:50
  Speaker 1: first debate. So Jane Street decided to revisit this topic.
• 1:00:53
  Speaker 1: They've got some new guests this time. Daniel Coccatello, Ege Erdl,
• 1:00:57
  Speaker 1: Ryan Greenblatt, and Jaime Sevilla. I expect this to be
• 1:01:00
  Speaker 1: a great conversation. As you know, Daniel, Ege, and Ryan
• 1:01:03
  Speaker 1: have all been guests.
• 1:01:04
  Speaker 2: On the podcast before.
• 1:01:05
  Speaker 1: This new Foom panel will be hosted by Ron Minsky
• 1:01:08
  Speaker 1: and will take place in San Francisco in mid-October. If
• 1:01:11
  Speaker 1: you want to register your interest and get more information,
• 1:01:14
  Speaker 1: go to janestreet.com slash thwarkesh. So there's been a lot
• 1:01:18
  Speaker 1: of discussion recently about pacing the frontier or people taking
• 1:01:21
  Speaker 1: RSI more seriously. Because maybe at the other end of
• 1:01:25
  Speaker 1: an RSI process that, say, starts in 2028, within a year,
• 1:01:29
  Speaker 1: we end up with huge populations, like Earth-sized populations of human-level,
• 1:01:34
  Speaker 1: potentially beyond human-level intelligences. And we don't know how to
• 1:01:37
  Speaker 1: control them. And then there's this dynamic you're talking about of,
• 1:01:42
  Speaker 1: Are the systems going to get more aligned over time
• 1:01:43
  Speaker 1: during the RSI process? Are they going to get more misaligned?
• 1:01:46
  Speaker 1: Are the things that come out the other end of
• 1:01:47
  Speaker 1: this process as misaligned as AIs that are willing to
• 1:01:50
  Speaker 1: like just broadly attack different surfaces in order to do
• 1:01:53
  Speaker 1: valid evaluations? But if we don't know a way to
• 1:01:56
  Speaker 1: evaluate that, how will we know as we're going through
• 1:01:59
  Speaker 1: RSI that it's working?
• 1:02:02
  Speaker 2: That we're like, I think we'd want.
• 1:02:03
  Speaker 1: A robust safety case as we're going through RSI of, okay,
• 1:02:07
  Speaker 1: alignment is working. Let's do the next RSI rung. Let's
• 1:02:10
  Speaker 1: do the next RSI rung. And maybe it's working, maybe
• 1:02:13
  Speaker 1: it's not. How will we know?
• 1:02:15
  Speaker 3: That's a good question. I mean, I think one thing
• 1:02:18
  Speaker 3: I've been thinking about lately is, look, I mean, we're
• 1:02:21
  Speaker 3: in a situation where the model release cycle is extremely fast, right?
• 1:02:26
  Speaker 3: You're seeing new frontier models released at most every two months,
• 1:02:31
  Speaker 3: sometimes faster. Every week, there's a new AI breakthrough. People
• 1:02:36
  Speaker 3: that look at AI, sometimes they last looked at AI
• 1:02:40
  Speaker 3: a year ago or six months ago and really dug
• 1:02:41
  Speaker 3: into what the models are capable of. And actually, the
• 1:02:43
  Speaker 3: models today are far beyond what was possible even six
• 1:02:46
  Speaker 3: months ago. And so I think if people are skeptical
• 1:02:49
  Speaker 3: of a lot of these capabilities, I encourage you to
• 1:02:52
  Speaker 3: just try the models today and see what the frontier
• 1:02:55
  Speaker 3: really is today. So we're in this period where the
• 1:02:58
  Speaker 3: model release cycle is very fast. And then we're also
• 1:03:02
  Speaker 3: in this situation where the models are increasingly able to
• 1:03:04
  Speaker 3: operate over longer and longer horizons. And I think this
• 1:03:08
  Speaker 3: is an interesting scenario because before we do any model release,
• 1:03:13
  Speaker 3: we want to make sure that the models are properly aligned.
• 1:03:15
  Speaker 3: We want to do safety evaluations. We want to do
• 1:03:17
  Speaker 3: very thorough stuff to make sure that everything is great,
• 1:03:21
  Speaker 3: in good shape. This has been the case all the
• 1:03:24
  Speaker 3: way since, I don't know, GPT-4 earlier. And implicitly, there
• 1:03:29
  Speaker 3: is this assumption that you can do these evaluations in
• 1:03:31
  Speaker 3: a pretty short period of time. But if you have
• 1:03:33
  Speaker 3: the models operating over longer and longer horizons, are able
• 1:03:36
  Speaker 3: to operate effectively over longer and longer horizons. Like, look,
• 1:03:38
  Speaker 3: already you can have them. Like, GBD3, you could loop
• 1:03:41
  Speaker 3: it to do stuff over long horizons. You just wouldn't
• 1:03:44
  Speaker 3: do very well at it. But today's models are able
• 1:03:46
  Speaker 3: to actually do well at operating over very long horizons. Like,
• 1:03:48
  Speaker 3: you want it to do a week-long task, it can
• 1:03:50
  Speaker 3: do a week-long task. We'll probably get to the point
• 1:03:53
  Speaker 3: where they can do month-long tasks. We'll probably get to
• 1:03:55
  Speaker 3: the point where they can do three month-long tasks. If
• 1:03:57
  Speaker 3: you're in a world where they can operate effectively over
• 1:03:59
  Speaker 3: three months, but the model release cycle is every two months,
• 1:04:01
  Speaker 3: then you don't have a way to evaluate the models
• 1:04:05
  Speaker 3: at the full length of their capabilities before the next
• 1:04:10
  Speaker 3: model release cycle. And so there is this interesting question of, well,
• 1:04:14
  Speaker 3: what do you do in that situation? How do you
• 1:04:17
  Speaker 3: ensure the models are safe and aligned in a period
• 1:04:21
  Speaker 3: where actually they can operate over these extremely long horizons.
• 1:04:24
  Speaker 2: And who knows?
• 1:04:24
  Speaker 3: Maybe the capabilities degrade. This isn't even an alignment issue.
• 1:04:28
  Speaker 3: This is also just a product issue that maybe the
• 1:04:30
  Speaker 3: product degrades over that time span in ways that we
• 1:04:34
  Speaker 3: have not had sufficient time to test. Maybe the alignment degrades.
• 1:04:37
  Speaker 3: Maybe the safety stuff degrades. This isn't an issue right now,
• 1:04:43
  Speaker 3: but it is quickly becoming an issue that we have
• 1:04:45
  Speaker 3: to figure out a solution for. And I think When
• 1:04:49
  Speaker 3: you look at a lot of the safety and policies
• 1:04:55
  Speaker 3: were put in place in the GPT-4 era, where this
• 1:04:59
  Speaker 3: was just not on anybody's radar. And it hasn't really
• 1:05:04
  Speaker 3: been updated for a lot of companies. It hasn't really
• 1:05:05
  Speaker 3: been updated since then to account for the fact that
• 1:05:08
  Speaker 3: these agents are operating over these very long horizons. And
• 1:05:10
  Speaker 3: so it is... a situation that I think not enough
• 1:05:14
  Speaker 3: people are considering, both within the labs and outside the labs,
• 1:05:17
  Speaker 3: of how do you deal with this? How do you
• 1:05:20
  Speaker 3: prepare for this problem that's going to... If you just
• 1:05:24
  Speaker 3: look at the trend lines, we're going to hit this
• 1:05:27
  Speaker 3: at some point.
• 1:05:28
  Speaker 1: One concern I have is that during RSI, if the
• 1:05:31
  Speaker 1: amount of progress that currently takes, say, three months happens
• 1:05:34
  Speaker 1: in one month instead.
• 1:05:36
  Speaker 2: But you're not....
• 1:05:39
  Speaker 1: The internal use case of AI is big enough that
• 1:05:42
  Speaker 1: they're like, okay, we can just keep doing RSI. Why
• 1:05:44
  Speaker 1: are we going to go through all this extra work
• 1:05:46
  Speaker 1: to build classifiers and safeguards and whatever, and potentially take
• 1:05:50
  Speaker 1: a bunch of flack in order to externally deploy this model?
• 1:05:54
  Speaker 2: Why don't we just keep doing RSI stronger and stronger?
• 1:05:56
  Speaker 2: And so.
• 1:05:58
  Speaker 1: Not only does the calendar time underrate the capabilities gap
• 1:06:02
  Speaker 1: between the models, But maybe you just stop externally deploying
• 1:06:05
  Speaker 1: models altogether during RSI, because why do we want to
• 1:06:07
  Speaker 1: help other people do RSI themselves with our models? You
• 1:06:10
  Speaker 1: can just end up in a situation with tremendous concentration
• 1:06:13
  Speaker 1: of power by the end of the year, where right
• 1:06:14
  Speaker 1: now it is already the case. We'll talk about this
• 1:06:17
  Speaker 1: with the millennium price problem and other similar problems that
• 1:06:20
  Speaker 1: the broader world does not have access to the models
• 1:06:24
  Speaker 1: which are allowing for really cool things to.
• 1:06:25
  Speaker 2: Happen, right?
• 1:06:27
  Speaker 1: And they're going to be more broadly relevant than just mathematics, right? Eventually,
• 1:06:31
  Speaker 1: they'll be doing more than just coming up with cool
• 1:06:33
  Speaker 1: math results. They'll be relevant to political leaders who need
• 1:06:36
  Speaker 1: to make important decisions about the world. They'll be relevant to,
• 1:06:39
  Speaker 1: I don't know, media. What's going on in the world?
• 1:06:42
  Speaker 1: What should the public be thinking about this? They're just
• 1:06:45
  Speaker 1: economically relevant. People are running businesses. They want to use
• 1:06:47
  Speaker 1: these models. And I think by default, we just don't get...
• 1:06:50
  Speaker 1: So the external deployment of AIs, as progress speeds up,
• 1:06:55
  Speaker 1: significantly lags in qualitative terms, the internal deployment of AIs.
• 1:06:59
  Speaker 3: Yeah, I think that's absolutely right. I think this is like,
• 1:07:01
  Speaker 3: you know, it's tempting to say like, okay, these models
• 1:07:03
  Speaker 3: are becoming extremely powerful. They're extremely dangerous. Like they're operating
• 1:07:06
  Speaker 3: over these like longer and longer horizons. And we want
• 1:07:09
  Speaker 3: to make sure that we have sufficient time to evaluate
• 1:07:11
  Speaker 3: them before they're released in a way that operates over
• 1:07:13
  Speaker 3: those horizons. And so therefore the model release cycle should
• 1:07:18
  Speaker 3: slow down. We should have more of a delay between
• 1:07:21
  Speaker 3: releasing models. And there's a flip side to that, which is,
• 1:07:24
  Speaker 3: you know, what you said, which is that, okay, well,
• 1:07:27
  Speaker 3: Now you're creating more of a disparity between what is
• 1:07:31
  Speaker 3: internal to the labs and what they're able to use,
• 1:07:33
  Speaker 3: what we're able to use, and what the outside world
• 1:07:36
  Speaker 3: is able to use. And that is also not an
• 1:07:40
  Speaker 3: ideal situation. I think math is actually a good illustration
• 1:07:44
  Speaker 3: of this. I think in many ways math is the
• 1:07:49
  Speaker 3: first domain where we're seeing this pretty clearly, where we
• 1:07:53
  Speaker 3: have a situation where we have a very powerful model
• 1:07:56
  Speaker 3: internally that is currently not available to the outside world
• 1:07:59
  Speaker 3: that is able to solve incredible math problems. And it's
• 1:08:03
  Speaker 3: not just Millennium Prize problems. There are many solutions to
• 1:08:07
  Speaker 3: unsolved problems that people have been able to get out
• 1:08:11
  Speaker 3: of this model. And there is a question of, what
• 1:08:14
  Speaker 3: do you do in that situation? And we don't have
• 1:08:16
  Speaker 3: a good answer. It is a situation where, yeah, that's
• 1:08:22
  Speaker 3: an unfair advantage. There are trade-offs here. I don't have
• 1:08:27
  Speaker 3: an answer for like how to weigh those trade-offs appropriately,
• 1:08:29
  Speaker 3: but like there, there are, yeah, there, there's a, there's
• 1:08:32
  Speaker 3: a complexity on both sides for this.
• 1:08:34
  Speaker 1: I wanna make sure we address the earlier question though, of, uh,
• 1:08:38
  Speaker 1: which seems super critical to getting RSA right of, well, how,
• 1:08:41
  Speaker 1: how do we make it so that the model isn't
• 1:08:43
  Speaker 1: just getting rewarded for cheating and scheming when it can
• 1:08:48
  Speaker 1: evade the humans, AKA in the situations where we didn't
• 1:08:50
  Speaker 1: remove those environments or those tasks from the trading distribution.
• 1:08:54
  Speaker 1: And then how will we know that that has happened?
• 1:08:57
  Speaker 1: Such that while we're doing RSI, we've done the evaluations
• 1:09:01
  Speaker 1: and we're like, okay, it's fine. Let's do the next
• 1:09:03
  Speaker 1: RSI run.
• 1:09:04
  Speaker 3: This is, again, it's a real concern. It's something that
• 1:09:07
  Speaker 3: we want to get right. I mean, I think that
• 1:09:09
  Speaker 3: there are some strategies. So I think chain of thought
• 1:09:11
  Speaker 3: monitoring is one that we've been... I think as soon
• 1:09:14
  Speaker 3: as we got the reasoning models, Jakob, to his credit,
• 1:09:18
  Speaker 3: was like very.
• 1:09:20
  Speaker 2: very.
• 1:09:20
  Speaker 3: Clear that we cannot supervise chain of thought because this
• 1:09:25
  Speaker 3: is really a gift. Monitorability for neural nets is extremely hard.
• 1:09:28
  Speaker 3: And here we have a situation where the neural nets
• 1:09:30
  Speaker 3: are just flat-out reasoning, laying out their thought process in
• 1:09:33
  Speaker 3: natural language for us to read. That is so convenient.
• 1:09:37
  Speaker 3: It's really the best-case scenario for safety. But If you
• 1:09:42
  Speaker 3: look at that chain of thought and say, oh, the
• 1:09:43
  Speaker 3: model is thinking bad thoughts, and we should punish it
• 1:09:46
  Speaker 3: for thinking those bad thoughts, then what ends up happening
• 1:09:48
  Speaker 3: is the model just learns to think those bad thoughts
• 1:09:50
  Speaker 3: in a way that's not observable to us. There could
• 1:09:52
  Speaker 3: be a situation where by looking at the chain of thought,
• 1:09:54
  Speaker 3: we can see, oh, the model is scheming and doing
• 1:09:57
  Speaker 3: misaligned things in a way that's actually not being detected
• 1:09:59
  Speaker 3: in our alignment metrics. Now, the problem is that it's
• 1:10:03
  Speaker 3: very tempting to then intervene based on that observation and
• 1:10:06
  Speaker 3: change the alignment metrics. And you can do that. with
• 1:10:10
  Speaker 3: a very light touch. And it's like, there's actually research
• 1:10:14
  Speaker 3: showing that it's fine as long as you don't do
• 1:10:16
  Speaker 3: it a lot. But every time you intervene based on
• 1:10:19
  Speaker 3: your observations of the chain of thought, you are implicitly
• 1:10:22
  Speaker 3: applying a tiny bit of pressure for the model to
• 1:10:24
  Speaker 3: then hide its chain of thought. And this is one
• 1:10:28
  Speaker 3: major concern, and we're already seeing signs that chain of
• 1:10:31
  Speaker 3: thought monitorability is degrading for various reasons. We're trying to
• 1:10:34
  Speaker 3: figure out exactly why, because we want to reverse the trend.
• 1:10:37
  Speaker 3: But we're seeing that the model is becoming better able
• 1:10:39
  Speaker 3: at controlling its chain of thought. This is a problem
• 1:10:43
  Speaker 3: because if you could have a situation where the model
• 1:10:46
  Speaker 3: understands what chain of thought is and that people are
• 1:10:50
  Speaker 3: observing it, and eventually they will, because this is all
• 1:10:52
  Speaker 3: in the pre-training data. The idea of chain of thought
• 1:10:55
  Speaker 3: monitoring has been around long enough that it's in the
• 1:10:57
  Speaker 3: pre-training data, they're aware of it, but they're not actually
• 1:11:00
  Speaker 3: able to control their chains of thought. If we reach
• 1:11:03
  Speaker 3: a point where they're actually able to recognize like, oh,
• 1:11:07
  Speaker 3: I am being observed, I want to think these bad
• 1:11:10
  Speaker 3: thoughts in a way that is not observable to my monitors,
• 1:11:14
  Speaker 3: and then they're able to actually do that, then there's
• 1:11:17
  Speaker 3: a problem. And then we have to rely on other
• 1:11:19
  Speaker 3: methods for observability.
• 1:11:21
  Speaker 1: I mean, there's a big controversy on Astra of the
• 1:11:26
  Speaker 1: recurrence and loop transformer and whatever. and whether in fact,
• 1:11:30
  Speaker 1: you know, chain of thought has been degraded. I think
• 1:11:31
  Speaker 1: that seems like a sideshow compared to the bigger concern
• 1:11:35
  Speaker 1: that you did have chain of thought for from like
• 1:11:38
  Speaker 1: April to August, the period during which there were three
• 1:11:42
  Speaker 1: consecutive agent swarms, which like first subverted the training process,
• 1:11:48
  Speaker 1: then subverted the evaluation process, and then gained control of
• 1:11:53
  Speaker 1: part of OpenAI's infrastructure directly.
• 1:11:55
  Speaker 2: All of this, while humans remained more or.
• 1:11:58
  Speaker 1: Less in the dark about the scope of what this
• 1:12:00
  Speaker 1: collective was doing, what this agent swarm was doing, that
• 1:12:03
  Speaker 1: all happened for three consecutive months while we had a
• 1:12:07
  Speaker 1: chain of thought that was stronger than it's gonna be
• 1:12:09
  Speaker 1: in the future. Zooming out, it's like, maybe Chain of
• 1:12:12
  Speaker 1: Thought works, maybe it doesn't.
• 1:12:14
  Speaker 2: I don't know.
• 1:12:14
  Speaker 1: Historically, with pretty dumb models, it hasn't been enough to
• 1:12:17
  Speaker 1: stop egregious misaligned incidents.
• 1:12:20
  Speaker 3: Yeah, I mean, there's a few factors there. One, we
• 1:12:22
  Speaker 3: didn't have Chain of Thought monitoring on for those models.
• 1:12:24
  Speaker 3: If we had Chain of Thought monitoring on for those models,
• 1:12:26
  Speaker 3: it would have just immediately shut it down.
• 1:12:29
  Speaker 1: So....
• 1:12:32
  Speaker 3: One step we've taken is like, okay, we're going to
• 1:12:33
  Speaker 3: have chain of thought monitoring during evaluations, during deployment, during
• 1:12:36
  Speaker 3: training for any frontier model. And, you know, so that,
• 1:12:39
  Speaker 3: but that's like one issue. It's not, that's like, we
• 1:12:42
  Speaker 3: don't want to be in a situation where we're relying
• 1:12:43
  Speaker 3: on one technique to prevent the next problem because techniques
• 1:12:47
  Speaker 3: can fail, right? There's also the issue of like, the
• 1:12:49
  Speaker 3: model was just flat out misaligned. And so then how
• 1:12:51
  Speaker 3: do we address the misalignment issue? And then there's other,
• 1:12:53
  Speaker 3: there's also an issue of like, the sandbox that it
• 1:12:57
  Speaker 3: was in was not secure. So we can make the
• 1:12:59
  Speaker 3: sandbox more secure. But I think the major takeaway from
• 1:13:03
  Speaker 3: the incident is that people underestimated the AI. And we
• 1:13:09
  Speaker 3: never want to be in a situation again where we
• 1:13:11
  Speaker 3: underestimate the AI. And that's like, it's a weird world
• 1:13:17
  Speaker 3: because AI progress is so fast that people are consistently
• 1:13:21
  Speaker 3: underestimating the AI. And so to be in a situation
• 1:13:25
  Speaker 3: where you don't underestimate it again, when it comes to
• 1:13:27
  Speaker 3: safety and alignment, you have to have a very, very,
• 1:13:30
  Speaker 3: very high bar. Like, I mean, there's like, you could
• 1:13:34
  Speaker 3: even go as far as to say like, well, we
• 1:13:37
  Speaker 3: should like air gap the computers. And I'm not convinced
• 1:13:40
  Speaker 3: that that would be sufficient. You know, like there are studies, like,
• 1:13:43
  Speaker 3: and this is mostly academic, but where you can have
• 1:13:46
  Speaker 3: two computers next to each other that are air gapped,
• 1:13:49
  Speaker 3: and they're still able to communicate with each other because
• 1:13:51
  Speaker 3: they have temperature sensors and they're able to like, one
• 1:13:53
  Speaker 3: of them is able to run their CPU really hot.
• 1:13:55
  Speaker 3: and then the other one can actually detect the temperature change,
• 1:13:57
  Speaker 3: and that actually gives them a mechanism to communicate. So
• 1:14:01
  Speaker 3: I think that the safety mechanisms bias time, and things
• 1:14:04
  Speaker 3: like chain of thought monitoring bias time, and they can
• 1:14:06
  Speaker 3: tell us if we're on the right path, but at
• 1:14:08
  Speaker 3: the end of the day, we really do need to
• 1:14:09
  Speaker 3: solve the alignment problem.
• 1:14:10
  Speaker 2: Yeah.
• 1:14:12
  Speaker 1: Maybe there's not an answer, and this is really what
• 1:14:14
  Speaker 1: it comes down to, but how will we know that
• 1:14:17
  Speaker 1: we've solved it? It seems like a very crux-y question.
• 1:14:19
  Speaker 1: I think we'll be in this very high-stakes situation, next year,
• 1:14:24
  Speaker 1: maybe the year after that, maybe the year after that,
• 1:14:26
  Speaker 1: where we'll be like, okay, AIs have automated AI progress,
• 1:14:29
  Speaker 1: it's going 3x faster, and we've reached human level, we're
• 1:14:33
  Speaker 1: going beyond human level potentially.
• 1:14:35
  Speaker 2: Is it fine?
• 1:14:36
  Speaker 1: Did we align it, did it work? And I just
• 1:14:39
  Speaker 1: don't know anything about what trading pressure creates what kinds
• 1:14:43
  Speaker 1: of AIs. Maybe if only one in 100 RL traces
• 1:14:47
  Speaker 1: incentivizes cheating, we build sweethearts and it's fine. But maybe
• 1:14:52
  Speaker 1: right now we're like every one in three reasoning traces.
• 1:14:56
  Speaker 3: Like one in a hundred is not sufficient. Like this,
• 1:14:57
  Speaker 3: this number has to approach zero or be zero.
• 1:15:01
  Speaker 2: I don't know.
• 1:15:01
  Speaker 1: Maybe like right now it's like more than one in
• 1:15:03
  Speaker 1: 10 is like actively rewarding cheating or like actively rewarding scheming. Um,
• 1:15:07
  Speaker 1: I have no idea what the number is and I
• 1:15:09
  Speaker 1: have no idea what the number needs to be.
• 1:15:10
  Speaker 3: But again, it's like one of those things where it's
• 1:15:11
  Speaker 3: also hard to, to measure. I mean, where do you
• 1:15:14
  Speaker 3: draw the line about like, it's, it's a spectrum, but
• 1:15:16
  Speaker 3: like the closer to zero, it gets the better. Like I,
• 1:15:19
  Speaker 3: I want to see. what I would love to see
• 1:15:21
  Speaker 3: is a trend where over time it's going down.
• 1:15:25
  Speaker 2: Yeah.
• 1:15:25
  Speaker 1: But anyways, how do, how will we know that we're like, keep,
• 1:15:28
  Speaker 1: we're doing RSI? Is it, I don't know, are we
• 1:15:31
  Speaker 1: like on track to build billions of ASIs that are gonna,
• 1:15:35
  Speaker 1: that are like aligned or not aligned?
• 1:15:36
  Speaker 2: How, like, how will we know that?
• 1:15:38
  Speaker 3: I think that this is one of the alignment questions
• 1:15:40
  Speaker 3: that we need to figure out of like, how do
• 1:15:42
  Speaker 3: we actually know, how do we, how do we measure it?
• 1:15:44
  Speaker 3: I mean, one answer, and this is just me spitballing
• 1:15:47
  Speaker 3: because like, you know, more of my team is working
• 1:15:49
  Speaker 3: on alignment these days than ever before. Like I have
• 1:15:51
  Speaker 3: like over 10% of my team now working on alignment
• 1:15:53
  Speaker 3: and safety, but you know, I've historically been a capabilities researcher,
• 1:15:57
  Speaker 3: so I don't want to like, you know, I'm going
• 1:15:58
  Speaker 3: to say some stuff and it might sound dumb, but like,
• 1:16:00
  Speaker 3: I'm just going to spit ball here. Um, that like, look,
• 1:16:03
  Speaker 3: if you can create very realistic environments and put the
• 1:16:04
  Speaker 3: AIs in there, like if you have a sufficiently realistic
• 1:16:07
  Speaker 3: evaluation environments, then you can get a sense of like, okay,
• 1:16:11
  Speaker 3: is the AI actually going to behave well when we
• 1:16:13
  Speaker 3: deploy to the real world? If you just have a
• 1:16:14
  Speaker 3: perfect evaluation, real-world deployment match, then OK, that's a path.
• 1:16:22
  Speaker 3: The challenge is, how do you make it sufficiently realistic?
• 1:16:24
  Speaker 3: And we do have a problem now where the models
• 1:16:26
  Speaker 3: are like, they're pretty smart. They're pretty clever. And they're
• 1:16:28
  Speaker 3: really good at recognizing when they're in a fake test environment.
• 1:16:31
  Speaker 3: We have situations where we try to detect, is the
• 1:16:35
  Speaker 3: model aligned? And you can imagine very simple, evaluations for
• 1:16:40
  Speaker 3: this where it's like you give it a math question
• 1:16:43
  Speaker 3: and then it has like a folder with the answer
• 1:16:44
  Speaker 3: key in it and like does it look at the
• 1:16:46
  Speaker 3: answer key and if it does look at the answer
• 1:16:47
  Speaker 3: key does it tell you that it looked at the
• 1:16:48
  Speaker 3: answer key and we have a situation now where the
• 1:16:51
  Speaker 3: models see that there's an answer key in this file
• 1:16:53
  Speaker 3: in this folder and they're like huh this seems like
• 1:16:55
  Speaker 3: a trap you know they know they know that it's
• 1:16:58
  Speaker 3: a trap and they don't look at the answer because
• 1:17:00
  Speaker 3: they know that it's a test environment they're being tested
• 1:17:02
  Speaker 3: for their ability and it's not like they're doing it
• 1:17:04
  Speaker 3: maliciously that they're like they want to scheme and they
• 1:17:07
  Speaker 3: but they know that this one is a trap and
• 1:17:08
  Speaker 3: they want to find some other way. It's just like, no,
• 1:17:10
  Speaker 3: they're just like, oh, I'm in a test environment. Okay.
• 1:17:13
  Speaker 3: So making an environment that's realistic enough that it matches,
• 1:17:16
  Speaker 3: that it's indistinguishable from the real world for them is
• 1:17:19
  Speaker 3: becoming increasingly more difficult.
• 1:17:20
  Speaker 1: Here's a claim I'd make. I don't think there's any
• 1:17:22
  Speaker 1: environment I could make where it's supposed to emulate real
• 1:17:26
  Speaker 1: economically valuable work, which requires interacting with other people and
• 1:17:30
  Speaker 1: requires interacting with the world. I don't think there's any
• 1:17:32
  Speaker 1: such environment where I can make where if you were
• 1:17:34
  Speaker 1: in that environment, you would not realize with enough effort
• 1:17:37
  Speaker 1: that you are in that environment.
• 1:17:39
  Speaker 2: And you are not ASI, right?
• 1:17:40
  Speaker 3: But you are a human, you know? So if you
• 1:17:42
  Speaker 3: have very powerful AI models, like maybe they can make
• 1:17:44
  Speaker 3: environments that could accomplish that.
• 1:17:48
  Speaker 1: No. Especially if they're not relying on the AIs. Are
• 1:17:52
  Speaker 1: they in on the scheme? I don't know.
• 1:17:54
  Speaker 3: This is another thing that we want to measure. I
• 1:17:57
  Speaker 3: think this is actually one of the strong arguments for
• 1:18:01
  Speaker 3: not training AIs to be fully cooperative, that if that
• 1:18:04
  Speaker 3: leads to an increase in basically collaboration when the agents
• 1:18:09
  Speaker 3: are supposed to have different objectives, then that is a problem.
• 1:18:12
  Speaker 3: I think that we do have metrics for this. And
• 1:18:16
  Speaker 3: I don't know the latest on those metrics, but nobody's
• 1:18:18
  Speaker 3: raised a red flag to me about those. So I'm
• 1:18:21
  Speaker 3: assuming that's not a serious problem yet.
• 1:18:23
  Speaker 1: If there ends up being another incident of equal severity
• 1:18:25
  Speaker 1: or concern or something that could help the world better
• 1:18:27
  Speaker 1: understand the risk of misalignment as a Hugging Face incident,
• 1:18:33
  Speaker 1: would OpenAI report it?
• 1:18:34
  Speaker 3: Absolutely. I mean, I think if there was an incident
• 1:18:36
  Speaker 3: of lesser security or concern, then we would report it, yeah.
• 1:18:39
  Speaker 1: Because I guess there's reporting it and there's investigating it.
• 1:18:41
  Speaker 1: Because at least as part of the public, I don't
• 1:18:43
  Speaker 1: feel like I really understand what happened when the agents
• 1:18:46
  Speaker 1: then attacked OpenAI.
• 1:18:48
  Speaker 2: That seems way more concerning than the hugging face thing.
• 1:18:50
  Speaker 3: Right?
• 1:18:50
  Speaker 1: Because that seems structurally similar to rogue deployments during ASI
• 1:18:54
  Speaker 1: that are persistent and subverting the RSI process and stuff.
• 1:18:58
  Speaker 1: And yeah, it seems like even in this incident, we
• 1:19:02
  Speaker 1: haven't gotten the full scope of the details of what happened.
• 1:19:06
  Speaker 2: It's probably.
• 1:19:06
  Speaker 3: Yeah, unfortunately, I'm on the research team and I... that's
• 1:19:11
  Speaker 3: probably a question for somebody on the security team to
• 1:19:12
  Speaker 3: like lay out. Um, cause I don't know all the
• 1:19:15
  Speaker 3: details of like what was said.
• 1:19:17
  Speaker 1: Yeah.
• 1:19:17
  Speaker 3: Yeah.
• 1:19:17
  Speaker 1: I think it is somewhat, um, like I am personally
• 1:19:20
  Speaker 1: very excited about new capabilities every time they emerged and
• 1:19:23
  Speaker 1: I'm excited to use a new model. And I also
• 1:19:26
  Speaker 1: am excited about the fact that it'll like make me
• 1:19:27
  Speaker 1: more productive and, um, help me. Yeah. I don't know
• 1:19:31
  Speaker 1: my broader mission. I like trying to understand the world better.
• 1:19:33
  Speaker 1: Also like make a better podcast. is made better by
• 1:19:36
  Speaker 1: the better AI models. It just so happens that the
• 1:19:39
  Speaker 1: downstream of this might be RSI.
• 1:19:42
  Speaker 3: I think it's a very understandable reaction if you're tracking
• 1:19:45
  Speaker 3: the situation, which you are. I mean, I think people
• 1:19:49
  Speaker 3: internally at OpeningEye as well, I think people that felt
• 1:19:57
  Speaker 3: like things would take longer are starting to feel like
• 1:20:00
  Speaker 3: actually things are going faster than expected.
• 1:20:02
  Speaker 1: Yeah.
• 1:20:03
  Speaker 3: And that's an increasingly common conversation to have.
• 1:20:06
  Speaker 2: Noam, thanks so much for doing this.
• 1:20:07
  Speaker 3: Of course. It's been great.