Latest / Dwarkesh Podcast / Ryan Greenblatt – What happens once AI can automate AI research?

Transcript

Ryan Greenblatt – What happens once AI can automate AI research? · Dwarkesh Podcast

0:00
Speaker 1: Today, I'm chatting with Ryan Greenblatt, who is the chief
0:01
Speaker 1: scientist at Redwood Research, where he focuses on technical AI
0:04
Speaker 1: safety and security work. I want to talk to you
0:07
Speaker 1: about recursive self-improvement. This is the idea that once you
0:10
Speaker 1: build human-level intelligences, they quickly slingshot towards tens of billions
0:16
Speaker 1: of superintelligences, which are each individually more competent than the
0:20
Speaker 1: top human experts across every field. Whether or not this
0:24
Speaker 1: turns out to be the case, I think is actually
0:25
Speaker 1: probably the most important question in the world right now.
0:29
Speaker 1: And historically, I've been quite skeptical that this kind of
0:31
Speaker 1: thing happens, but you seem to think that it might
0:33
Speaker 1: be plausible. And so I wanted to hear the case
0:35
Speaker 1: for it.
0:36
Speaker 2: Yeah, let's talk about this.
0:37
Speaker 3: So first, I think it's worth noting that AR &
0:40
Speaker 3: D is a type of task at which the AIs
0:42
Speaker 3: are especially good because both the companies are trying really
0:45
Speaker 3: hard to make their AIs good at AR & D
0:47
Speaker 3: and it's the kind of domain, it has a lot
0:49
Speaker 3: of nice properties from the perspective of how AI development
0:51
Speaker 3: works right now. So it's like, pretty verifiable. You can
0:54
Speaker 3: do a bunch of stuff iteratively and he'll climb on
0:56
Speaker 3: various metrics. And then I think once you have AIs,
0:59
Speaker 3: which are roughly matching the top human experts in AI
1:02
Speaker 3: R & D, that could sort of kick off a
1:04
Speaker 3: feedback loop where the AIs are doing AI research, that
1:07
Speaker 3: produces smarter AIs, that feeds back in. And that feedback
1:09
Speaker 3: loop could be strong enough that you end up with
1:11
Speaker 3: a lot of progress in a short period of time.
1:14
Speaker 3: Maybe my sort of median expectation is something like four
1:17
Speaker 3: or five years of AI progress in a single year.
1:20
Speaker 3: And this requires really overcoming a huge amount of diminishing
1:24
Speaker 3: returns in research and basically doing the equivalent of what
1:27
Speaker 3: progress we would have gotten after a really large compute scale-out.
1:29
Speaker 2: So this is like a pretty impressive big thing.
1:31
Speaker 3: And it's worth keeping in mind that five years of
1:34
Speaker 3: AI progress, four years of AI progress, even three years
1:35
Speaker 3: of AI progress is really a lot of fucking AI progress, right?
1:38
Speaker 2: So, you know, right now it's like.
1:41
Speaker 3: Three years ago or a little over three years ago,
1:43
Speaker 3: there was GPT-4 that had come out. And right now,
1:47
Speaker 3: of course, we have like, you know, Mythos 5 or whatever,
1:49
Speaker 3: and maybe... a somewhat better model than Anthropic has internally.
1:54
Speaker 3: And so that is just a huge amount of progress
1:56
Speaker 3: in a bit over three years. And if we're talking
1:58
Speaker 3: about five years, then maybe we're talking more about a
2:00
Speaker 3: jump from GPT-3 to Mythos 5 or whatever. Yeah.
2:06
Speaker 1: Okay, so I think this argument has three different parts,
2:09
Speaker 1: and I want to evaluate each one of them. First
2:12
Speaker 1: is the argument that AI R & D is very,
2:14
Speaker 1: very viable. Second is the argument that if you automate
2:16
Speaker 1: AI R & D, you could get four or five
2:18
Speaker 1: years of progress in a single year. And third is
2:20
Speaker 1: the argument that what comes out the other end of
2:24
Speaker 1: four or five years of AI progress at the current pace,
2:28
Speaker 1: starting at the starting point whenever AI R & D
2:29
Speaker 1: is automated, what comes out the other end is an
2:32
Speaker 1: AI where you can drop it on the job at
2:35
Speaker 1: basically anything you can imagine. You can drop it in
2:38
Speaker 1: Texas politics in the 1940s and it outmaneuvers Lyndon Johnson.
2:41
Speaker 1: You can drop it in, I don't know, a TSMC
2:44
Speaker 1: and it learns how to do better process engineering at TSMC.
2:49
Speaker 1: It's certainly a better video editor than I.
2:53
Speaker 1: My video editors are very excellent, but it is just
2:55
Speaker 1: in general better than humans at any given job that
2:58
Speaker 1: it finds itself trying to do. So I want to
3:00
Speaker 1: evaluate all of these sub-arguments that lead to basically getting
3:06
Speaker 1: ASI pretty soon after this benchmark, which you're expecting by
3:08
Speaker 1: 2030 or something, right?
3:10
Speaker 3: Yeah, I would say that I expect like full automation
3:12
Speaker 3: of AR & D, perhaps somewhere around like 2031, 2030,
3:14
Speaker 3: and then getting to like the like beats all humans
3:19
Speaker 3: on the job milestone. Maybe I expect median around 2033,
3:23
Speaker 3: but sort of like if I see AI's fully automating
3:26
Speaker 3: R & D, I think I'm expecting that probably within
3:28
Speaker 3: a year. It's just like the way the forecasting works out,
3:31
Speaker 3: there's a, the difference between medians is bigger than the
3:33
Speaker 3: median difference between milestones.
3:35
Speaker 1: Anyway, whatever. By the way, there's this meme on the
3:38
Speaker 1: internet because every time I'm trying to ask about people's
3:40
Speaker 1: timelines when I'm asking Dario or somebody, I'm always like, okay,
3:44
Speaker 1: how long before can I automate my video editors? And
3:46
Speaker 1: there's this meme of my video editor editing the podcast
3:50
Speaker 1: every time I listen to this. The reason I do
3:53
Speaker 1: it is because I think it's easy to get lost
3:55
Speaker 1: in abstractions when you talk about jobs you don't understand
3:58
Speaker 1: well and to very comfortably understand what it takes to
4:02
Speaker 1: automate a job that I actually understand why it's difficult
4:05
Speaker 1: for LLMs to currently take control over.
4:06
Speaker 3: I do think that the milestone for automating your video
4:10
Speaker 3: editor is earlier than the milestone of being able to
4:12
Speaker 3: automate all human jobs including Texas politics spinning up on
4:15
Speaker 3: the job. So I do think that the video editor
4:18
Speaker 3: automation maybe occurs more around full automation of AI R
4:21
Speaker 3: & D, but it's very sensitive to how much people
4:23
Speaker 3: are really focusing on understanding video.
4:24
Speaker 1: Yeah. Okay. So let's start with the claim that AI
4:27
Speaker 1: R & D is very verifiable.
4:29
Speaker 3: Yeah.
4:29
Speaker 2: So there's a few different parts of this.
4:31
Speaker 3: One of them is that we can train on a
4:33
Speaker 3: bunch of environments, which are basically directly training the model
4:37
Speaker 3: to do some AI R & D task or some
4:39
Speaker 3: very close by task. So for example, we can have
4:41
Speaker 3: some environment where the model is training some AI on
4:43
Speaker 3: just like eight H100s or whatever, or like some small
4:46
Speaker 3: amount of compute. And that model could be like, you know,
4:49
Speaker 3: the equivalent of like GPT-2 medium or whatever. And then,
4:52
Speaker 3: you know, similar to like nano GPT medium runs or whatever.
4:57
Speaker 3: And in RL, it's like tweaking and iterating on that.
5:00
Speaker 3: And we could do that for a bunch of different tasks.
5:02
Speaker 3: Like we could have it train like image classification models,
5:05
Speaker 3: video generation models, image generation models, all kinds of different
5:08
Speaker 3: sort of ML training tasks. And we could RL it
5:11
Speaker 3: on the task of training increasingly good models, and also
5:14
Speaker 3: doing things like, oh, here's a particular direction you could
5:16
Speaker 3: pursue for an algorithm. Can you go and implement that?
5:19
Speaker 3: And so basically, there's a whole class of containerizable, verifiable,
5:23
Speaker 3: small-scale AR & D tasks that we can aggressively RL
5:25
Speaker 3: the AIs on. And I would say that already companies
5:28
Speaker 3: are presumably doing some RL on these sorts of tasks.
5:31
Speaker 3: And you could just keep scaling that up, keep making
5:33
Speaker 3: more of these sort of small-scale AR & D tasks.
5:35
Speaker 3: And then the AIs could keep getting better at this.
5:39
Speaker 3: And then Implicitly, I'm claiming this will transfer to extremely
5:42
Speaker 3: load-bearing aspects of AR & D, but maybe let's stop
5:44
Speaker 3: there for a second and let me get to that part.
5:46
Speaker 1: So let's talk through what this concretely looks like. So
5:48
Speaker 1: you can imagine that we have GPT-7.5, and we say, GPT-7.5,
5:53
Speaker 1: we want to make you so good at AI R
5:54
Speaker 1: & D that you help us train GPT-9. Okay, so
5:57
Speaker 1: now we want to train GPT-7.5, and we can come
6:00
Speaker 1: up with a bunch of different environments. Like, as you mentioned,
6:02
Speaker 1: we could do, there's already this repo that is the
6:05
Speaker 1: descendant of Andrej Karpathy's nano GPT speedrun, where you just
6:08
Speaker 1: try to change everything about the model from the optimizer
6:12
Speaker 1: to the hyperparameters to the architecture to get it to
6:17
Speaker 1: get to a fixed training loss as fast as possible.
6:19
Speaker 1: You could have other kinds of environments where you could say, hey,
6:23
Speaker 1: GPT 7.5, I want you to train a really good
6:26
Speaker 1: video game playing model. And I want you to train
6:28
Speaker 1: a model that actually improves as it plays the same
6:30
Speaker 1: video game again and again. So you learn how to
6:32
Speaker 1: maybe help the model get better at online learning. Maybe
6:35
Speaker 1: it gets, we don't care how you figure this out.
6:37
Speaker 1: Maybe it's some kind of crazy neural ease or vector memory.
6:39
Speaker 1: Maybe it's some crazy, maybe just like better long context stuff.
6:42
Speaker 1: We don't care. Get, figure out how to like do
6:44
Speaker 1: online learning research. Obviously then the fact that GPT 7.5
6:48
Speaker 1: will already have become very good at normal, like become,
6:51
Speaker 1: it'll be a smart model. And in the same way,
6:53
Speaker 1: the models currently are getting smarter. It'll be better and
6:55
Speaker 1: better at coding and the way that models are currently
6:57
Speaker 1: getting better in coding. And you can imagine 100 other
7:00
Speaker 1: environments like this, which are incentivizing the ability to do
7:02
Speaker 1: AI R & D by getting GPT-7.5 to like containerized
7:06
Speaker 1: versions of getting GPT-7.5 to develop GPT-2 size models, et cetera,
7:10
Speaker 1: et cetera. And basically, then you put GPT-7.5 through a
7:15
Speaker 1: bunch of this kind of training. You build GPT-8. And
7:18
Speaker 1: GPT-8 is now an amazing ML researcher. It has so
7:21
Speaker 1: much intuition from doing all this kind of training. Honestly,
7:25
Speaker 1: a huge intuition pump for me is seeing the progress
7:27
Speaker 1: that AI has made in mathematics, where I'm just like,
7:30
Speaker 1: if it's a very verifiable domain, AIs can get even,
7:34
Speaker 1: even like mathematics also involves so much. Like, I don't
7:38
Speaker 1: really know this object level details of the mathematics research,
7:40
Speaker 1: but I'm just like, no, it works. Like you can
7:42
Speaker 1: just come in like a flood. If you can totally
7:44
Speaker 1: put it into a verification loop and it can actually
7:46
Speaker 1: make new breakthroughs. Um, I am curious if ML research
7:51
Speaker 1: has a quality of mathematical research or it seems like
7:53
Speaker 1: there's a big overhang from connecting different disciplines together or
7:56
Speaker 1: ideas that were not, no one person would have known
8:00
Speaker 1: enough about algebraic geometry and what was the right word?
8:04
Speaker 2: Oh man, I really don't know about the math breakthroughs.
8:07
Speaker 1: No one person would have known enough about topology and
8:11
Speaker 1: algebraic whatever, blah, blah, blah, in order to make some
8:14
Speaker 1: counterexample to a big conjecture.
8:17
Speaker 2: My view is that ML is a less deep domain
8:20
Speaker 2: than math.
8:20
Speaker 3: And so there's less of a thing where there's like
8:22
Speaker 3: individual experts with really deep expertise in some area that
8:24
Speaker 3: they combine, but there's definitely gonna be some of that.
8:27
Speaker 3: But then I also think that ML has some attributes
8:29
Speaker 3: that make it even more favorable than mathematics in some
8:32
Speaker 3: ways to AI training. In particular, you can get a
8:38
Speaker 3: better sense of whether you're succeeding and you can see
8:40
Speaker 3: intermediate progress. So in math, it's often the case that
8:43
Speaker 3: sort of there's no easy way to see whether or
8:46
Speaker 3: not you're close to success. Whereas if your goal is to,
8:49
Speaker 3: for example, get to some training loss, you know, 2x faster,
8:54
Speaker 3: you can kind of see when you're halfway there. And
8:56
Speaker 3: it tends to be the case that ML innovations are
8:58
Speaker 3: very additive or maybe multiplicative, depending on how you think
9:00
Speaker 3: about it, where basically you can keep stacking innovations. And
9:03
Speaker 3: usually the innovations just sort of just add together and
9:05
Speaker 3: don't interfere with each other, though obviously it's going to
9:07
Speaker 3: depend on the details. And so I think that in
9:09
Speaker 3: a lot of ways, AI R & D will have properties,
9:13
Speaker 3: you know, quite similar to math, where basically you can
9:16
Speaker 3: do small, you can train on chunks of AIR &
9:19
Speaker 3: D that are pretty similar in structure to the problem
9:23
Speaker 3: you actually cared about in a very verifiable way, and
9:26
Speaker 3: then that will transfer. And then there's an open question
9:28
Speaker 3: of exactly how well it will transfer, but I think
9:30
Speaker 3: that the transfer currently for math looks pretty good. And
9:33
Speaker 3: my expectation is that the transfer for AIR & D
9:35
Speaker 3: will look pretty good, but not amazing.
9:37
Speaker 1: So one concern I have is I think even in mathematics,
9:41
Speaker 1: as far as I'm aware, we have not seen very
9:43
Speaker 1: impressive new theory. We've seen a lot of impressive, verifiable,
9:48
Speaker 1: specific results. For example, find a counterexample to this conjecture.
9:51
Speaker 1: But we have not seen come up with the idea
9:52
Speaker 1: of topology kinds of levels of things, or come up
9:55
Speaker 1: with things like group theory. And it seems like ML
9:58
Speaker 1: research has elements of both of these things, but the
10:01
Speaker 1: less verifiable thing of like come up with new ways
10:04
Speaker 1: of thinking about the problem would be harder to induce.
10:06
Speaker 1: So it takes, for example, the idea of scaling loss. Obviously,
10:11
Speaker 1: there is some end verification loop such that you can
10:14
Speaker 1: train GPT-4 better if you have the idea of scaling
10:17
Speaker 1: loss from like 2020. But there is a longer and
10:21
Speaker 1: potentially more compute-laden road to inducing AIs to be like, okay,
10:27
Speaker 1: I've got to think carefully about how I should be
10:30
Speaker 1: scaling my parameters and data. What are the different kinds
10:32
Speaker 1: of investigations I could run to understand this? Maybe I
10:34
Speaker 1: can come up with a visualization and an isoflop analysis
10:37
Speaker 1: or something. But that does seem like a longer verification
10:43
Speaker 1: loop than just, hey, let's get nanoGPT lost to go down.
10:47
Speaker 2: Yeah, let's talk about this.
10:49
Speaker 3: First of all, I think in the context of math,
10:51
Speaker 3: the thing I would say is that the AIs can
10:53
Speaker 3: do the equivalent of like baby's first new theory or whatever,
10:57
Speaker 3: where like, for example, they can just like prove interesting
11:01
Speaker 3: conductors via like making connections and producing new understanding of like, oh,
11:06
Speaker 3: there's this like thing that I, this like construction that
11:07
Speaker 3: I found, which is pretty interesting or like, found this
11:10
Speaker 3: like way of thinking about the problem that's a bit different.
11:13
Speaker 2: And we do just see that.
11:13
Speaker 3: It's just that the examples we see are not like
11:15
Speaker 3: as impressive as like founding the field of group theory.
11:18
Speaker 3: But like in part, you know, probably founding the field
11:21
Speaker 3: of group theory is like one of the, you know,
11:23
Speaker 3: it's like among the best, biggest mathematical accomplishments of all time.
11:26
Speaker 2: And the AIs just aren't, you know, they're not that
11:27
Speaker 2: good at math yet.
11:28
Speaker 3: And I think that from my perspective, sort of there's
11:30
Speaker 3: a continuum between that and the things we're seeing now
11:33
Speaker 3: that the AIs are continuing to march up. Second, I
11:36
Speaker 3: think ML is a very shallow domain relative to math.
11:40
Speaker 3: So I think in math, there was much more of a,
11:42
Speaker 3: you find some true deep abstraction. And then like that,
11:47
Speaker 3: like if you really understand that thing, which is hard
11:49
Speaker 3: to understand, then you get somewhere. Whereas I feel like
11:51
Speaker 3: the things that are the equivalent of that in ML
11:53
Speaker 3: are really like dumb bullshit. Like I'm like scaling laws. Like,
11:57
Speaker 3: come on guys, we can explain scaling laws really quickly.
11:59
Speaker 3: And I think the like deepest and most important concepts
12:02
Speaker 3: in math, for example, don't have the property of like,
12:04
Speaker 3: you can really understand the underlying thing and why it
12:06
Speaker 3: matters in a very short period of time.
12:09
Speaker 1: But I feel like one effect will be that we
12:11
Speaker 1: will have gotten rid of all the low-hanging fruits by 2030.
12:13
Speaker 1: I feel like scaling laws will have been in what
12:17
Speaker 1: math history, Descartes, finding the Cartesian grid and doing very
12:21
Speaker 1: basic mathematics-wise. And then eventually, if we want to keep
12:24
Speaker 1: making progress in the 2030s, it's going to be like,
12:26
Speaker 1: do whatever bullshit is happening at the frontiers of mathematics
12:29
Speaker 1: right now.
12:29
Speaker 2: Yeah, that could be right.
12:31
Speaker 3: My sense is that just some domains are structurally different
12:34
Speaker 3: in terms of how they operate and how much they
12:35
Speaker 3: depend on like sort of deep abstractions and like physics
12:38
Speaker 3: and math are much more in the side of like
12:40
Speaker 3: being very far on the like sort of very deep,
12:43
Speaker 3: hard to come up with ideas side. Whereas I think
12:46
Speaker 3: ML and most other domains are much more amenable to
12:49
Speaker 3: sort of hill climbing. And that's my sense of how
12:52
Speaker 3: this will go in the future. And even in the
12:54
Speaker 3: regime where your AIs are like, you know, having to plow,
12:58
Speaker 3: like it's 2030, they need to like, a bunch of
13:01
Speaker 3: low-hanging fruit and research has already happened.
13:02
Speaker 2: They need to like make further progress.
13:04
Speaker 3: I still suspect that a bunch of the work will
13:06
Speaker 3: live more on the side of like building increasingly complicated infrastructure,
13:10
Speaker 3: having really good intuition about what the experiments roughly look like.
13:13
Speaker 3: And so I think I'm probably less sympathetic to like
13:15
Speaker 3: the like thing that the AIs will lack is like
13:17
Speaker 3: some deep insight and more sympathetic to like, they really
13:20
Speaker 3: need a bunch of like taste about in the weeds
13:22
Speaker 3: experiments that they currently don't have and need to have
13:24
Speaker 3: a bunch of intuition for like what sorts of training
13:28
Speaker 3: approach would work and what wouldn't work.
13:30
Speaker 2: In ways that current researchers have.
13:32
Speaker 3: And even in cases where there has been some breakthrough
13:35
Speaker 3: in AI, Oftentimes, in retrospect, it looks like a big
13:38
Speaker 3: bottleneck to making that breakthrough happen was sort of getting
13:40
Speaker 3: all of the micro details and mungy intuition right. Like
13:45
Speaker 3: an example of this is when it comes to training
13:48
Speaker 3: AIs to be good at reasoning and chain of thought
13:51
Speaker 3: and doing sort of RL and chain of thought training,
13:53
Speaker 3: it looks like you probably could have done RL and
13:55
Speaker 3: chain of thought on GPT-3 and gotten kind of interesting
13:59
Speaker 3: results on math if you had really scaled it up
14:00
Speaker 3: and done a good job. But at the time, there
14:02
Speaker 3: was low-hanging fruit and also... doing a good job with
14:05
Speaker 3: that training is, like, kind of, like, in the weeds
14:08
Speaker 3: in all the technical implementation and scaling it up and
14:10
Speaker 3: getting the hyperparameters right. And so maybe you can demonstrate
14:13
Speaker 3: everything on, like, Quen 1B or whatever and get some
14:16
Speaker 3: sense that this whole thing is going to work. But
14:18
Speaker 3: people didn't demonstrate it as early as they could have because, like,
14:22
Speaker 3: you know, of all of these other, like, mungy details
14:24
Speaker 3: and intuition about exactly how to tune the parameters and
14:27
Speaker 3: how to set things up.
14:28
Speaker 1: This is my remaining skepticism, honestly, about the story is just...
14:33
Speaker 1: I am... I'm not sure I understand why, if research
14:38
Speaker 1: breakthroughs are so amenable to intelligence, why AI progress has
14:42
Speaker 1: not been historically faster than it could have been. And
14:44
Speaker 1: we had to wait for, as you were saying, by
14:47
Speaker 1: the time RLVR actually worked, even though you could have
14:51
Speaker 1: done it with less compute, we had to wait for
14:53
Speaker 1: oceans of compute and gigawatts of compute to be available
14:55
Speaker 1: before people are doing this training. On the trajectory of
14:59
Speaker 1: this constant, as compute keeps increasing, we make more breakthroughs.
15:03
Speaker 1: I don't know, I feel like there were a lot
15:03
Speaker 1: of AI researchers in the year 2022 who were trying
15:07
Speaker 1: to crack reasoning. And it was just that they were
15:10
Speaker 1: like bottlenecked by the ability to write infrastructure code or
15:13
Speaker 1: like what was happening.
15:14
Speaker 2: It's a complicated mix, right?
15:14
Speaker 3: So I think that they would have gone faster if
15:16
Speaker 3: they could like, as soon as they thought of an experiment,
15:18
Speaker 3: run that experiment without bugs, without bugs being very important.
15:21
Speaker 3: And then I think another part of it is that
15:23
Speaker 3: like being able to run a lot of experiments at
15:25
Speaker 3: high compute lets you paper over ways in which the
15:28
Speaker 3: way you implemented it isn't quite right or you didn't
15:30
Speaker 3: have the right hyperparameters. And so I think compute is
15:32
Speaker 3: just like really helpful for doing AI research. And you
15:34
Speaker 3: can like, you know, cover over a lot of things.
15:37
Speaker 3: But that doesn't mean that massive increases in labor wouldn't
15:39
Speaker 3: also be helpful, especially if that labor comes with, you know,
15:42
Speaker 3: among the best intuitions that people have in the field.
15:45
Speaker 3: I just think that that's, you know, really helpful. I
15:47
Speaker 3: think another part of my perspective here, which is maybe
15:49
Speaker 3: a bit different from where you're coming from, is I
15:52
Speaker 3: think I'm expecting somewhat more transfer than you seem to
15:55
Speaker 3: be imagining. And I'm imagining these AIs are actually like
15:57
Speaker 3: pretty good scientists in general and are just like, you know,
16:01
Speaker 3: pretty reasonable at all of that stuff. And just sort
16:04
Speaker 3: of when you were to interact with them, it's not
16:06
Speaker 3: like there's some like really hyper-specialized savant type vibe. They're
16:10
Speaker 3: actually just like pretty good at all this stuff in
16:12
Speaker 3: R & D and then maybe like extremely good at
16:13
Speaker 3: some subdomains, right? So they're like incredibly superhuman at writing kernels,
16:17
Speaker 3: incredibly superhuman at everything with very short feedback loops. And
16:19
Speaker 3: then like, you know, pretty good at all the other
16:22
Speaker 3: stuff and like, you know, just totally able to match
16:24
Speaker 3: other people. And like, I think we are seeing this now.
16:27
Speaker 3: Like I would say that when I look at AIs
16:29
Speaker 3: right now, I think it's already the case that they
16:31
Speaker 3: can pretty competently match like humans who are mediocre at
16:35
Speaker 3: ML research, at doing ML research. It's just that being
16:38
Speaker 3: mediocre at ML research is not that helpful, right? Like
16:40
Speaker 3: the thing that you actually want are people who are
16:42
Speaker 3: good at ML research. And so my sense is the
16:44
Speaker 3: AIs are just improving at all of these things. Their
16:46
Speaker 3: taste is improving. Their intuition is improving. And it's already
16:48
Speaker 3: the case that their taste and intuition is not like,
16:50
Speaker 3: it's not like complete garbage, right? Yeah.
16:52
Speaker 1: So I want to very concretely understand what it would
16:54
Speaker 1: look like for five years of AI progress to happen
16:56
Speaker 1: in one year.
16:57
Speaker 2: Yeah.
16:57
Speaker 1: So suppose we were back and when like GPT-3 is developed.
17:00
Speaker 1: And the idea is not only that, like basically with
17:03
Speaker 1: the level of compute they've had back in 2022, you
17:06
Speaker 1: could have trained, if we had automated AI R &
17:08
Speaker 1: D back then, you could at the end of that
17:11
Speaker 1: year have Mythos.
17:13
Speaker 2: That'd be the idea, yes.
17:13
Speaker 1: Including with like, so Mythos took way more compute than
17:17
Speaker 1: they had back then. But like even with the level
17:18
Speaker 1: of compute they had back then, not only do they
17:20
Speaker 1: do all the breakthroughs, But they also train Mythos with
17:23
Speaker 1: their level of compute. And what would be required is, obviously,
17:27
Speaker 1: discovering all the algorithmic progress since then. Discovering even more, actually,
17:30
Speaker 1: because you had to make up for the fact that
17:32
Speaker 1: Mythos uses, I don't know, what was GPT-3 trained on?
17:36
Speaker 1: Like 1E23? We can look it up. But is it
17:39
Speaker 1: plausibly four orders of magnitude more compute?
17:42
Speaker 2: Yeah, I think it's somewhat less than that. Let's look
17:44
Speaker 2: this up quickly. So GPT-3 training compute is, yeah, it's
17:50
Speaker 2: like 3E23.
17:52
Speaker 3: Um, my sense is that, um, mythos is probably about, um,
17:56
Speaker 3: a little over three ohms higher. And so the question is,
17:58
Speaker 3: can you overcome this thousand X compute gap while also,
18:02
Speaker 3: you know, um, being the model. So here's a concrete
18:05
Speaker 3: claim that maybe we should, we should talk about like
18:08
Speaker 3: right now we would be able to train a model
18:10
Speaker 3: with GPT three level compute that matches. Um, yeah. What
18:14
Speaker 3: exactly do I think? Um, so GPT three was, let's
18:17
Speaker 3: say about, um, yeah. When was it trained? So it
18:20
Speaker 3: was trained, um, It was released in 2020, so it
18:24
Speaker 3: was trained six years ago. It's worth noting that GPT-3
18:26
Speaker 3: is maybe a little too far away or too far
18:28
Speaker 3: in the past, but let's go with this for a second.
18:30
Speaker 3: So GPT-3 was trained about six and a half, seven
18:34
Speaker 3: years ago. If we were to train a model with
18:37
Speaker 3: GPT-3 level compute today, how good would that model be?
18:41
Speaker 3: My understanding is based on how algorithmic progress works, we'd
18:45
Speaker 3: be able to train a model that's as good as
18:47
Speaker 3: the best model we had perhaps around three years ago.
18:50
Speaker 3: So I think that right now we'd be able to
18:51
Speaker 3: train a version of GPT-3 that's probably somewhat better than
18:54
Speaker 3: GPT-4 is basically what we'd see. Probably, yeah, like a
18:58
Speaker 3: moderate amount better than GPT-4.
19:00
Speaker 2: And I think that's about right.
19:01
Speaker 3: I think that roughly lines up with how algorithmic progress
19:04
Speaker 3: has worked. Basically, the story would end up being that
19:07
Speaker 3: to get five years of AI progress, you're probably going
19:09
Speaker 3: to need around, I would say, like maybe eight years
19:12
Speaker 3: of algorithmic progress, very roughly, which is a lot, a
19:15
Speaker 3: lot of algorithmic progress. Right. But it just turns out
19:17
Speaker 3: that most of the AI progress, from my perspective, has
19:20
Speaker 3: come from some mix of algorithms and data. And you
19:23
Speaker 3: can just keep making, I think, huge improvements on these
19:26
Speaker 3: things and training AIs with less compute.
19:30
Speaker 1: I'm glad you brought that up because what has happened
19:32
Speaker 1: since GPT-3 or even 3.5 till now, right? Why is
19:35
Speaker 1: mytho so good? Obviously, we've scaled the compute. We have
19:39
Speaker 1: better algorithms. A huge thing that's happened is that we
19:41
Speaker 1: have built a deca-billion-dollar data industry, which has... systematically collected
19:49
Speaker 1: and codified expert human judgment across all kinds of different disciplines,
19:54
Speaker 1: codified in the form of RL environments, codified in the
19:57
Speaker 1: form of SFT traces, that these experts built to help
20:00
Speaker 1: the model better understand how do you do coding and
20:02
Speaker 1: how do you build complex infrastructure projects? How do you
20:05
Speaker 1: do law? How do you do whatever, whatever? And how
20:09
Speaker 1: are the AIs able to replicate the effect that currently
20:14
Speaker 1: expert human judgment seems to be playing in AI progress?
20:18
Speaker 3: Yeah. So my sense is that scaling up the amount
20:21
Speaker 3: of effort spent on getting expert human data has not
20:24
Speaker 3: been hugely important for AI R & D in general.
20:27
Speaker 3: So in particular, like, you know, over the last few years,
20:30
Speaker 3: we've been scaling up compute, scaling up people working at
20:33
Speaker 3: AI companies. and scaling up the amount of effort spent
20:36
Speaker 3: on data labeling. My sense is that if you, like,
20:38
Speaker 3: sort of removed, like, the last, like, two doublings or
20:41
Speaker 3: whatever of data labeling, that would not make a huge difference.
20:44
Speaker 3: Or data generation, that would not, I'm sorry, I should
20:46
Speaker 3: say data generation from expert humans, that would not make
20:48
Speaker 3: a huge difference. I think a lot of what's been
20:50
Speaker 3: going on is people have been developing better ways to leverage, like,
20:54
Speaker 3: humans and AIs to, like, construct RL environments and going
20:58
Speaker 3: somewhere from that.
20:59
Speaker 1: How do you explain why the AIs have gotten so
21:02
Speaker 1: good at coding? I feel like a big part of
21:03
Speaker 1: that is data and RL environments, which are like quantifying
21:06
Speaker 1: human experts.
21:07
Speaker 3: But the question is, what is the limiting factor on
21:09
Speaker 3: creating RL environments? My sense of the limiting factor on
21:11
Speaker 3: creating RL environments was not so much like scaling up
21:16
Speaker 3: or like the thing that drove, the reason why RL
21:19
Speaker 3: environments today are much better than they were in like,
21:22
Speaker 3: you know, 2024 is not that much because we have
21:26
Speaker 3: hired way more human experts to make RL environments. It
21:29
Speaker 3: is instead much more because we better know what RL
21:33
Speaker 3: environments we even want to make and how we should
21:36
Speaker 3: structure them. And also we're using huge amounts of AI
21:39
Speaker 3: labor to build RL environments. And I think those effects
21:42
Speaker 3: are much more important than the effect of human labor
21:45
Speaker 3: building the RL environments.
21:49
Speaker 2: I'm not saying that the human.
21:50
Speaker 3: Labor doesn't matter. I'm just saying there's other big drivers
21:52
Speaker 3: that are important here. Yeah, I could try to argue
21:56
Speaker 3: for this. I mean, one thing is just like the
21:58
Speaker 3: amount of, environments people want are just like they're very
22:01
Speaker 3: it's a very large amount um and i think the
22:03
Speaker 3: ais are actually pretty good at the task of making
22:05
Speaker 3: rl environments given some some sense of what the the
22:09
Speaker 3: thing should be there's pre-existing data you could use i
22:11
Speaker 3: don't know a lot of these things have good verification loops.
22:14
Speaker 1: If i just look at for example this is reported
22:16
Speaker 1: in business insider today yesterday that uh google is paying
22:20
Speaker 1: like two close to two billion dollars for mechanize yeah
22:23
Speaker 1: um like the, we can just look at market rates
22:26
Speaker 1: or what people think really good human experts making like
22:31
Speaker 1: human expert data is worth. And it just seems to
22:33
Speaker 1: be like the, the frontier labs seem to think it's
22:35
Speaker 1: worth a lot.
22:37
Speaker 3: What fraction of, of frontier lab spending do you think
22:39
Speaker 3: is on data rather than compute? Like, what do you
22:42
Speaker 3: think is the compute data spend split?
22:44
Speaker 1: I think it's most like overwhelmingly compute, but I also
22:46
Speaker 1: think it's because like compute is easier to scale up
22:48
Speaker 1: than data.
22:49
Speaker 3: But that's really relevant to what's driving progress, right? It's like, Suppose, like,
22:53
Speaker 3: I agree that, yeah, like, my sense is that the
22:54
Speaker 3: split is something like, I would have guessed, like, 20
22:57
Speaker 3: to 1 or something, 10 to 1.
22:58
Speaker 2: I don't know exactly. It depends on the company.
22:59
Speaker 1: I mean, but this is similar to, like, oil is 1.5%
23:01
Speaker 1: of GDP. But that doesn't mean if you cut oil out,
23:05
Speaker 1: you could, like, GDP could continue to run. Sure, but
23:07
Speaker 1: it contradicts your argument, right? The economy would, like, come
23:08
Speaker 1: to a halt immediately if, like, oil went away.
23:10
Speaker 3: Sure, but you are just arguing that because of the
23:12
Speaker 3: high market cap, we can learn that this is the
23:14
Speaker 3: key driver.
23:14
Speaker 2: And I'm saying that's not clearly true, right?
23:16
Speaker 3: Sure. Because I think that argument just makes it look
23:19
Speaker 3: like compute is a much more important driver or hiring
23:21
Speaker 3: employees is a much more important driver.
23:22
Speaker 1: Let's be more concrete. Here's what I think. Just the
23:26
Speaker 1: same way as in my claim is that if you
23:28
Speaker 1: went back to 2022 and you had GPT-3.5 and you
23:31
Speaker 1: were trying to make it better at coding without human experts,
23:36
Speaker 1: I think it would have just been very, very difficult.
23:38
Speaker 1: Let me give you an example of what I imagine
23:39
Speaker 1: would be the difficulty from going from GPT-8 to ASI.
23:43
Speaker 1: So one of the things you'd need GPT-8 to be
23:45
Speaker 1: good at, or you'd want ASI to be good at,
23:47
Speaker 1: is I'm going to take over a company and make
23:50
Speaker 1: it much more profitable and do all kinds of crazy
23:53
Speaker 1: shit to make it work better. I'm going to take
23:55
Speaker 1: over a fab and produce more chips. This is the
23:59
Speaker 1: tier of data that I'm going to go into Congress
24:02
Speaker 1: and try to convince them to pass some bill, blah, blah, blah.
24:05
Speaker 1: This is what I imagine five more years of AI
24:08
Speaker 1: progress at this pace would enable an AI to be
24:10
Speaker 1: able to do. This is the thing I'm really worried about, right?
24:12
Speaker 1: Like the ASI that can like understand how to do
24:14
Speaker 1: crazy shit in the world, like what Kissinger can do,
24:16
Speaker 1: can do what like Steve Jobs can do, et cetera,
24:19
Speaker 1: and also his engineers and stuff. And I'm not sure
24:22
Speaker 1: how you get that without the relevant world data, which
24:27
Speaker 1: is the equivalent of mythos being really good at coding
24:29
Speaker 1: while not having the coding environments that have improved it
24:33
Speaker 1: relative to GPT-3.
24:34
Speaker 2: Yeah, so here are a few points.
24:36
Speaker 3: So first, I bet if you look at sort of
24:38
Speaker 3: randomly sampled training environments for mythos, they're actually very different
24:41
Speaker 3: from what it looks like to actually use the model
24:44
Speaker 3: in practice. My sense is that the RL distribution has
24:47
Speaker 3: really large deviations from the real-world data distribution, and it's
24:50
Speaker 3: significantly being smoothed over by a mix of transfer and
24:53
Speaker 3: having a small amount of data focused on the real world.
24:56
Speaker 3: And so my sense is that this will be a
24:58
Speaker 3: similar mechanism as how it works for the crazy, wild,
25:04
Speaker 3: quite superhuman AI you get as a result of five
25:06
Speaker 3: years of AI progress on top of fully automated AR
25:08
Speaker 3: in deep. So let's just go through this a little bit.
25:11
Speaker 3: So in particular, I think that you could train an
25:15
Speaker 3: AI to be really, really good at learning on the
25:18
Speaker 3: fly and doing something analogous to in-context learning, but potentially
25:21
Speaker 3: using somewhat different mechanisms in a wide variety of RL environments.
25:25
Speaker 3: So you build all these different RL environments where the
25:27
Speaker 3: AI has to adapt on the fly, learn on the fly,
25:30
Speaker 3: figure out what it should do, understand its situation better,
25:34
Speaker 3: and learn really quickly from feedback in order to succeed
25:36
Speaker 3: at its objective. and has things like limited resources. And
25:39
Speaker 3: if it messes up, it can end up in a
25:41
Speaker 3: much worse position. And then if you train on a
25:43
Speaker 3: huge number of these environments, you will learn sort of
25:46
Speaker 3: general skills of picking up context on the fly.
25:49
Speaker 2: And we're already seeing this.
25:50
Speaker 3: It's already the case that AIs are now much better
25:52
Speaker 3: at sort of understanding roughly what's going on and picking
25:55
Speaker 3: up context from a limited amount of information they're given
25:59
Speaker 3: access to. And then those AIs could then be put
26:02
Speaker 3: on the job at TSMC. And then even though TSMC
26:05
Speaker 3: is not literally in their data distribution, Their data distribution
26:08
Speaker 3: is really wide and the AIs are extremely good on
26:10
Speaker 3: their data distribution such that it transfers to picking up
26:13
Speaker 3: being good at, you know, being an engineer at TSMC
26:16
Speaker 3: and learning that on the fly where it looks more
26:18
Speaker 3: like the way the AI gets good at being a
26:20
Speaker 3: TSMC engineer isn't that it has a ton of cash
26:23
Speaker 3: knowledge on being a good TSMC engineer. It's that it
26:25
Speaker 3: like does the equivalent of like some scaled up version
26:28
Speaker 3: of in-context learning there.
26:31
Speaker 2: But that'd be the most prosaic story. Obviously, there's like
26:33
Speaker 2: a bunch of different ways this could go.
26:34
Speaker 1: I think this maybe comes down to then a difference
26:36
Speaker 1: of intuition about how far you can get. When I
26:39
Speaker 1: think about really smart people I know, they're just like
26:42
Speaker 1: not that effective in domains they don't understand that well.
26:45
Speaker 2: But how long have they had to learn?
26:47
Speaker 1: No, I agree that if they had experience, they would
26:49
Speaker 1: be much better. But that's maybe what I'm arguing for
26:52
Speaker 1: is that experience of data. Like, for example, if I
26:55
Speaker 1: just get a really smart, I don't know, Ivy League
26:57
Speaker 1: college grad and I'm like, okay, you're now in charge
26:59
Speaker 1: of negotiating the Iran deal. I think they just like
27:02
Speaker 1: wouldn't know what to do.
27:04
Speaker 3: I think if you got instead got someone who is
27:06
Speaker 3: really good at quickly picking up a bunch of different
27:08
Speaker 3: domains and you gave them some time to sort of
27:10
Speaker 3: train and talk to people and show up their expertise
27:13
Speaker 3: and do some practice, they would actually do like a
27:15
Speaker 3: pretty good job. I think most domains are fundamentally pretty
27:18
Speaker 3: shallow where like a very smart generalist who's good at
27:21
Speaker 3: like a limited subset of core skills can like get
27:25
Speaker 3: going pretty quickly. And my sense is that like that's
27:28
Speaker 3: not true for literally every domain. And my sense is
27:30
Speaker 3: that the AIs will develop increasingly good mechanisms for quickly
27:33
Speaker 3: acquiring understanding and expertise in a given domain. So consider,
27:36
Speaker 3: for example, how fast AIs can understand a new code base.
27:41
Speaker 3: AIs can understand a new code base much faster than
27:43
Speaker 3: humans can, but to a degree that's shallower than humans
27:45
Speaker 3: could currently understand, but is getting better over time. So
27:48
Speaker 3: let me spell that argument out a bit more. So
27:51
Speaker 3: let's say you take Fable 5 or Mythos 5 or whatever,
27:55
Speaker 3: and you wanted to make some kind of complicated change
27:58
Speaker 3: to a really massive code base. the model will get
28:01
Speaker 3: some understanding of the code base very fast, like in
28:03
Speaker 3: the course of maybe like, you know, significantly less than
28:06
Speaker 3: an hour, potentially much less than an hour.
28:08
Speaker 2: And then its understanding of the code base.
28:10
Speaker 3: Will like plateau a little bit where it won't get
28:12
Speaker 3: as deep of an understanding as a human would have
28:13
Speaker 3: gotten over a much longer period. So it's sort of
28:15
Speaker 3: like an AI in an hour can match a human
28:18
Speaker 3: with a few weeks maybe, depending on the details of
28:20
Speaker 3: exactly how complicated the code base is. But then it
28:23
Speaker 3: won't match a human with like, you know, who's been
28:25
Speaker 3: working on that code base for like two years or whatever.
28:27
Speaker 3: But over time, the like amount of, understanding AIs can
28:31
Speaker 3: match has gone up, right? So if we look at
28:32
Speaker 3: like 3.7 Sonnet or 3.5 Sonnet, maybe it could only
28:36
Speaker 3: match the equivalent of understanding a code base for like
28:39
Speaker 3: a day or something. But now, you know, AIs are
28:42
Speaker 3: much better at like sort of building context about a task.
28:45
Speaker 3: And so you can be like Mythos, I want you
28:46
Speaker 3: to really understand this code base and then, you know,
28:49
Speaker 3: then implement this feature. And it will like spawn a
28:52
Speaker 3: bajillion sub-agents. Those sub-agents will pour over a bunch of things.
28:54
Speaker 3: It will like deliver a bunch of context back. It
28:56
Speaker 3: will then like investigate a few things. And it's not
28:58
Speaker 3: like amazing at doing this, but it's like, it can
29:00
Speaker 3: happen like really fast and it can work pretty well.
29:03
Speaker 3: And it's not very hard for me to imagine how
29:04
Speaker 3: you could train AIs to be increasingly good at this task, right?
29:07
Speaker 3: The task of like implement some very complicated feature in
29:10
Speaker 3: some reasonable way in a very big code base is
29:12
Speaker 3: extremely verifiable. And that can like be a thing the
29:15
Speaker 3: AIs improve on. And similarly, like there's a broader skill
29:18
Speaker 3: of like quickly understanding context and being able to like
29:21
Speaker 3: have a bunch of different AIs learn in parallel and
29:23
Speaker 3: then merging that together.
29:24
Speaker 1: And I think there seems to be a crux here,
29:26
Speaker 1: which I think is just an empirical question we'll see
29:29
Speaker 1: on which is how good is a transfer between getting really,
29:34
Speaker 1: really good at understanding the situation, getting up to speed,
29:40
Speaker 1: making progress over long periods in verifiable domains, which the
29:45
Speaker 1: AI is obviously getting way, way better at really fast, to, okay,
29:49
Speaker 1: go talk to the president and convince him to do
29:52
Speaker 1: X thing, or you're now in charge of Google. Now
29:56
Speaker 1: you must make Google a much more profitable company this quarter.
29:59
Speaker 3: Let me try to just spell out a few more
30:01
Speaker 3: arguments that are maybe relevant. One thing is that I
30:03
Speaker 3: do think that when looking at how the AIs have
30:06
Speaker 3: improved at essay writing, let's talk about that a little bit.
30:09
Speaker 3: I think there's one thing which is that you can
30:11
Speaker 3: get some data even on these domains, and AIs will
30:14
Speaker 3: be able to get some data even on these domains
30:16
Speaker 3: when on a very fast progress trajectory. maybe it's hard
30:19
Speaker 3: to build like a verifiable environment for like, was your
30:21
Speaker 3: essay really good according to humans?
30:23
Speaker 2: But you can do a bit of that.
30:24
Speaker 3: You know, you can do some training, you can do
30:25
Speaker 3: some online training. And the AIs will be able to
30:27
Speaker 3: do some like, you know, online training based on real
30:29
Speaker 3: world stuff. They'll be able to like have evals. They'll
30:32
Speaker 3: be able to like sample that. And you can, you know,
30:34
Speaker 3: scale that up the cadence at which you do this.
30:36
Speaker 3: And then the second thing is that in practice, when
30:38
Speaker 3: I just look at the transfer, it seems okay. Like
30:40
Speaker 3: I think that in fact, the AIs have improved a
30:42
Speaker 3: bunch at non-verifiable domains. And it is in fact the
30:45
Speaker 3: case that it's hard to point to like domains that
30:47
Speaker 3: are really hard to verify on which the amount of
30:49
Speaker 3: improvement between, you know, GPT-4 and mythos hasn't been like
30:52
Speaker 3: pretty high in practice. And now that doesn't mean that
30:54
Speaker 3: mythos is like better than the best humans or something, right?
30:57
Speaker 3: It can still be like significantly worse than typical human
30:59
Speaker 3: professionals at some aspect of their job while still being
31:02
Speaker 3: like way better than GPT-4, which was like not even close.
31:06
Speaker 1: Yeah. So we were talking about how, uh, how important
31:09
Speaker 1: data versus algorithmic progress has been for explaining the progress
31:13
Speaker 1: over the last few years. That reminds me, I'm actually
31:15
Speaker 1: running an experiment with Jerry Han, who's actually still a
31:19
Speaker 1: college student. What we're basically doing to evaluate how much
31:23
Speaker 1: progress is coming from data versus algorithms is training the
31:26
Speaker 1: best algorithmic recipe from 2019 till now with the best
31:31
Speaker 1: data from the 2026 data file, and then also training
31:34
Speaker 1: the different data files going back to 2019 to 2026
31:37
Speaker 1: with the current best training recipe, like the algorithmic recipe.
31:40
Speaker 1: And I think that will be an interesting... I'm curious
31:43
Speaker 1: if you want to pre-register what amount could be multipliers
31:46
Speaker 1: are coming from one versus the other.
31:48
Speaker 3: So we need to be pretty careful with what we
31:49
Speaker 3: mean when we say the word data. So I was
31:51
Speaker 3: trying to be pretty careful to distinguish between scaling up
31:53
Speaker 3: spending on getting human experts to label data or scaling
31:56
Speaker 3: up the amount of human experts to label data. Pre-training
31:59
Speaker 3: data does not come. The reason why we have a
32:01
Speaker 3: better pre-training data set now versus in 2019 is not
32:04
Speaker 3: because people are spending way more money getting human experts
32:07
Speaker 3: to type up data that the AIs are then trained on.
32:10
Speaker 3: I think it's- I think it's not much of it.
32:12
Speaker 3: I think it's very little of the pre-training data improvements.
32:14
Speaker 3: I think the vast majority of the pre-training data improvements,
32:16
Speaker 3: which I do mean pre-training, we should talk maybe separately
32:19
Speaker 3: about mid-training, post-training, but I think the vast majority of
32:21
Speaker 3: pre-training data improvements are from science on better understanding what
32:24
Speaker 3: data sets are good and schleppy labor on figuring out
32:28
Speaker 3: how to filter down. And so my view is that
32:30
Speaker 3: improvements of the form of open web text to fine
32:34
Speaker 3: web or whatever, that improvement is better described as a
32:37
Speaker 3: algorithmic improvement of the sort that you can you know,
32:40
Speaker 3: study with some GPUs and then do, and you don't
32:42
Speaker 3: need humans to like, you don't need human expert data
32:45
Speaker 3: to do that. Now there's a different effect, which we
32:47
Speaker 3: could talk about, which is that maybe the internet in 2026,
32:50
Speaker 3: it has much more as more of a fertile ground
32:52
Speaker 3: for training data than like the internet in 2018. Like,
32:55
Speaker 3: it's like, there's also been an effect where like, there's
32:57
Speaker 3: just more humans posting on the internet.
32:58
Speaker 2: So there's more data harvest.
33:00
Speaker 3: My sense is that that effect is going to be
33:02
Speaker 3: quite a bit smaller than the effect of just like humans,
33:05
Speaker 3: like knowing better how to curate the data, having better scrapes,
33:08
Speaker 3: knowing how to process those scrapes better, this sort of thing.
33:11
Speaker 1: This is more like automated engineering and automated R & D.
33:13
Speaker 2: That's right.
33:14
Speaker 1: That makes sense.
33:15
Speaker 2: Yeah.
33:15
Speaker 3: So like, I think that in some sense, the thing
33:17
Speaker 3: you would want to look at is be like, we're
33:18
Speaker 3: going to do two post-training pipelines, one post-training pipeline where
33:21
Speaker 3: we only have like a tiny number of human experts
33:23
Speaker 3: to do the labeling, but we can have like, you know,
33:26
Speaker 3: smart AIs. And then another, like, you know, you're like,
33:28
Speaker 3: we're going to build, Mythos 5 is going to build
33:30
Speaker 3: a post-training pipeline, but it only has access to like internet,
33:33
Speaker 3: internet data, plus like a tiny amount of human experts,
33:36
Speaker 3: but it has the best current methods versus we have
33:38
Speaker 3: one where it's like, you know, Mythos has access to
33:42
Speaker 3: like the shitty post-training methods we had in 2024, but
33:45
Speaker 3: with like a shit ton of human experts. And again,
33:49
Speaker 3: both have the internet data. My sense is that the
33:51
Speaker 3: current methods, but without many human experts, actually will do
33:54
Speaker 3: quite well. Though it's a bit messy because like, mythos,
33:57
Speaker 3: like it's like, can mythos get something that's more capable
33:59
Speaker 3: than mythos? Like, you might need to be a bit
34:00
Speaker 3: thoughtful on like, what model is it that you're post-training?
34:03
Speaker 1: What is your view on what is the least verifiable
34:04
Speaker 1: part of AI R & D?
34:06
Speaker 2: The least verifiable? Probably making calls on large experiments. Yeah.
34:10
Speaker 3: Like, the thing that I think is most likely to
34:12
Speaker 3: be sort of the bottleneck in terms of like, the
34:14
Speaker 3: AIs are really good at verifiable domains, but not at
34:16
Speaker 3: doing the actual thing, is just like, big experiments, you
34:18
Speaker 3: only get a few tries, um, well, a few is
34:21
Speaker 3: maybe a bit understated, but like basically like historically R
34:25
Speaker 3: & D has been driven by doing near frontier scale experiments.
34:28
Speaker 3: And that has been pretty important. And like actually doing
34:30
Speaker 3: the one big training run where you decide exactly what
34:32
Speaker 3: to include in that. And there's a bunch of ways
34:34
Speaker 3: that the AIs can sort of make that more verifiable
34:36
Speaker 3: so they can have better science of exactly what to predict.
34:39
Speaker 3: They can scale down their frontier scale training runs to
34:42
Speaker 3: a point where they can study that scale more aggressively
34:45
Speaker 3: at some one-time hit to compute cost, right? So like
34:47
Speaker 3: if people wanted to, a thing you can always do
34:49
Speaker 3: is train smaller models so that you can run more rounds.
34:53
Speaker 2: And I think we have seen this. Like, I think
34:54
Speaker 2: one reason.
34:55
Speaker 3: Why the AIs have been scaled up less than you
34:58
Speaker 3: would have otherwise expected. And like, for example, cost of
35:02
Speaker 3: per token hasn't increased as much as you might've thought
35:05
Speaker 3: is because there was a benefit to doing more of
35:07
Speaker 3: your work at small scale where you can run more
35:10
Speaker 3: training runs and get more cycles in. And so you're
35:12
Speaker 3: not as like, you know, leaning as hard on like
35:14
Speaker 3: one big, you know, really important training run.
35:16
Speaker 1: Yeah. I just want to unpack a couple of things
35:19
Speaker 1: that were for the audience. The thing you're pointing out
35:23
Speaker 1: is I think the price per token has not increased
35:25
Speaker 1: that much since 2024, 2023. Yeah. So GPT-4 was like.
35:27
Speaker 3: I don't know, like, was it like $ 30 per output token?
35:32
Speaker 3: And I'm like, Mythos is $ 50 per output token.
35:34
Speaker 1: Right. And so the thing you're trying to explain is
35:38
Speaker 1: how can it be that we're in this era of
35:40
Speaker 1: scaling and so a bigger model shouldn't be more expensive
35:43
Speaker 1: to serve, but the... the uh but the token price
35:47
Speaker 1: is not increasing and you're suggesting that we've like increased
35:51
Speaker 1: active parameters slower than you would have naively assumed because
35:57
Speaker 1: because people just want to make fast progress on training
35:59
Speaker 1: models and you do that by training smaller models faster.
36:02
Speaker 3: I mean there's a complicated mix of factors i think
36:04
Speaker 3: my view is more like um people have done a
36:07
Speaker 3: bunch of big training runs that did not go that
36:09
Speaker 3: well so there's like gpd 4.5 which like famously people
36:12
Speaker 3: at opening i thought was a big bit of a
36:14
Speaker 3: bust i think there's some rumors that there are a
36:15
Speaker 3: bunch of other training runs that people have done that
36:18
Speaker 3: were a bit of a bust and part of it
36:19
Speaker 3: is that i think there's just a bunch of details
36:20
Speaker 3: and actually getting that right and so um it makes
36:23
Speaker 3: sense to just do more of the work at smaller
36:25
Speaker 3: scale and just eat the fact that you're taking a
36:27
Speaker 3: hit on final performance in order to like be able
36:30
Speaker 3: to like quickly iterate and you know train more models
36:33
Speaker 3: faster and therefore better learn and also better um better
36:37
Speaker 3: be able to just have like a, you know, smarter
36:39
Speaker 3: ultimate production model. This is not the only effect, right?
36:41
Speaker 3: There's also the fact that RL benefits more from small models.
36:44
Speaker 3: There's like a bunch of things going on. But I
36:45
Speaker 3: do think that like, in fact, people are making trade-offs
36:48
Speaker 3: towards the side of like faster iteration times because of
36:51
Speaker 3: algorithmic progress being so fast.
36:53
Speaker 1: It seems to me that a big source of why
36:55
Speaker 1: these big trading arenas have failed, at least from rumors,
36:57
Speaker 1: is just like very subtle bugs that are really hard
37:01
Speaker 1: to track down. Yeah. And the TLDR is how good
37:05
Speaker 1: will the AIs be at avoiding these kinds of, avoiding
37:08
Speaker 1: and finding these kinds of mistakes where they might be,
37:13
Speaker 1: they might get really good at engineering and like being
37:16
Speaker 1: trained to avoid bugs. Like basically the opposite of the
37:19
Speaker 1: slop world we live in now or like are living
37:21
Speaker 1: in less and less over time. But then there's also
37:23
Speaker 1: the question of can they do the analysis to find
37:26
Speaker 1: the right experiment to run to identify what is going
37:29
Speaker 1: wrong with the training run right now, which seems to
37:31
Speaker 1: be very bottlenecked by the taste of extremely few humans
37:34
Speaker 1: who are like... Right now, my assumption is GDM is
37:37
Speaker 1: going through this right now where humans are trying to
37:38
Speaker 1: figure out what is wrong with the training pipeline.
37:40
Speaker 2: Yeah, there's some rumor that.
37:43
Speaker 3: Right after Noam Shazir joined GDM, which he's now left,
37:48
Speaker 3: they had a new really good training run that happened.
37:50
Speaker 3: And the reason why is that Noam Shazir just looked
37:52
Speaker 3: at their code base and found a bunch of bugs
37:54
Speaker 3: because he just knew where to look. My sense is
37:57
Speaker 3: that training AIs to find bugs is going to be
38:00
Speaker 3: one of the easier tasks to train AIs on because
38:03
Speaker 3: most of these bugs we're talking about can probably be
38:05
Speaker 3: demonstrated without that much compute and probably you get pretty
38:09
Speaker 3: good transfer from pointing out other types of bugs at
38:11
Speaker 3: smaller scale. And so then you can RLAIs at like,
38:14
Speaker 3: look at this overall complicated training situation and point out
38:17
Speaker 3: cases where there's like an important bug and then fix that.
38:20
Speaker 3: And I think that like, this is not like a,
38:21
Speaker 3: this is like a pretty verifiable task. It's not arbitrarily
38:24
Speaker 3: verifiable because maybe often to demonstrate the bug, you might
38:27
Speaker 3: need to do like a moderate scale compute experiment where
38:30
Speaker 3: you like spin up the whole distributed infrastructure and then
38:32
Speaker 3: run it. But oftentimes I think you'll be able to
38:34
Speaker 3: demonstrate it pretty convincingly at smaller scale in a way
38:37
Speaker 3: which you could actually train on. And so my sense
38:39
Speaker 3: is that like, it will not necessarily... I think it
38:42
Speaker 3: wouldn't be very surprising if right now people have RL
38:44
Speaker 3: environments where they introduce a subtle bug into some training recipe,
38:48
Speaker 3: train the AI to point out the subtle bug, and
38:51
Speaker 3: then have a rubric where they're like, did it actually
38:53
Speaker 3: find the right bug? And that seems very doable. And
38:55
Speaker 3: you could do a bunch of stuff. There's a bunch
38:57
Speaker 3: of things you could do along these lines that I
38:59
Speaker 3: think would work reasonably well. And so I think that
39:01
Speaker 3: on that specific point, I think it's doable. And then
39:04
Speaker 3: the main thing is that I think there's some cases where...
39:07
Speaker 3: there's other intuition about which exact large-scale de-risking experiments do
39:10
Speaker 3: you need to run? How should you orient them? How
39:12
Speaker 3: should you pick hyperparameters in uncertain cases or things that
39:15
Speaker 3: are analogous to hyperparameters? And that's, I think, the thing
39:17
Speaker 3: that the AIs might most struggle with. But I currently
39:20
Speaker 3: expect there'll be enough transfer if you train on all
39:22
Speaker 3: these different environments that the AIs will be good at
39:26
Speaker 3: that domain. And I should be clear, I also think
39:28
Speaker 3: that the AIs will transfer to other domains. I think
39:30
Speaker 3: that there's sort of just like, there's gonna be the
39:33
Speaker 3: domains the AIs are by far the best at, then
39:35
Speaker 3: there's domains where they're somewhat less good at, and there's
39:37
Speaker 3: domains there's quite a bit less good at. And I
39:38
Speaker 3: think we still see transfer to everything. And it's really
39:41
Speaker 3: hard for me to think of examples of cognitive tasks
39:43
Speaker 3: humans do where we're not seeing some transfer from AI improving.
39:47
Speaker 1: So let's step back and package this whole story. So
39:50
Speaker 1: I think people will probably follow along with the story
39:52
Speaker 1: of we have GPT-7.5 to train on a bunch of environments,
39:55
Speaker 1: where it's not only just in general becoming a better AI,
39:58
Speaker 1: but specifically we're training it to do AI R &
40:02
Speaker 1: D better, like make GPT-2 sized runs that are better
40:06
Speaker 1: at playing video games that require sample efficiency or online
40:08
Speaker 1: learning or whatever other capabilities.
40:10
Speaker 3: And another thing that's really important is you don't just
40:12
Speaker 3: do GPT-2 sized runs. you also do small fine-tuning runs
40:16
Speaker 3: on GPT-6. Or as in you have GPT-2, and you
40:18
Speaker 3: can do full pre-trains on GPT-2, and then you can
40:21
Speaker 3: do small post-training or mid-training or whatever runs on GPT-6,
40:26
Speaker 3: and then you can do a small number of experiments
40:27
Speaker 3: that are actually at frontier scale, but you do a
40:30
Speaker 3: bit of online training or something.
40:31
Speaker 1: What do you mean by do online training on that?
40:33
Speaker 3: Yeah, so another thing that we can do is we
40:35
Speaker 3: can take GPT-7.5, and presumably in the course of GPT-7.5's work,
40:40
Speaker 3: It's running a bunch of like experiments at varying scale
40:42
Speaker 3: that are actually on the critical path for AI R
40:44
Speaker 3: & D. For many of those things, you'll be able
40:46
Speaker 3: to get a sense after the fact for whether or
40:48
Speaker 3: not it did a good job, right? So like it
40:50
Speaker 3: did some, you know, post-training experiment where it was trying
40:53
Speaker 3: to like figure out whether some method actually works. And
40:56
Speaker 3: in some cases, you'll be like, whoa, it found this
40:58
Speaker 3: like kick-ass method. It like totally de-risked it.
41:00
Speaker 2: It totally worked.
41:01
Speaker 3: And then you can then reinforce that by just, like,
41:03
Speaker 3: I mean, one thing you could do would be, like,
41:05
Speaker 3: take that behavior, convert the, like, experiment you just ran
41:09
Speaker 3: into a production RL environment. Sorry, into an RL environment
41:12
Speaker 3: based on production data and then train on that. Or
41:14
Speaker 3: you could potentially just literally take the rollouts that found
41:16
Speaker 3: that and then do, like, some sort of.
41:20
Speaker 2: Off-policy RL or you could do some, like, on-policy RL.
41:23
Speaker 1: Basically, the thing you're suggesting is, like, there's the small-scale
41:27
Speaker 1: stuff where you're just, like, teaching the AI to get
41:28
Speaker 1: better at AI R & D taste. but you're discarding
41:31
Speaker 1: the actual quote-unquote things it found. Yeah, that's right. But
41:35
Speaker 1: then it actually does real R & D in the
41:37
Speaker 1: practice of trying to become better at AI R & D.
41:39
Speaker 1: And you're like, this is a pretty cool thing that
41:40
Speaker 1: you discovered. Let's actually also use this in production in
41:43
Speaker 1: the future and teach you how to use it in production.
41:45
Speaker 2: That's right.
41:45
Speaker 1: But stepping back, so GPT-7.5 becomes GPT-8 as a result
41:49
Speaker 1: of all this AI R & D training and just
41:51
Speaker 1: generally becoming smarter. Then it helps you build GPT-9. And
41:55
Speaker 1: another very important thing that would have had to happen,
41:58
Speaker 1: which is maybe the thing I'm most skeptical of, is
42:01
Speaker 1: GPT-8 has figured out how to make it so that
42:06
Speaker 1: whatever it's doing to make GPT-9, as intelligent as it is,
42:12
Speaker 1: the humans currently, like AI researchers, try their shit and
42:16
Speaker 1: they're like, okay, but we trained GPT-4.5 and it wasn't
42:19
Speaker 1: good or something. It required real-world feedback or some evaluation
42:23
Speaker 1: of trying to use the model in production and it
42:25
Speaker 1: wasn't that good and we're not going to ship it.
42:28
Speaker 1: And so GPT-8 needs this ability to, see how good
42:32
Speaker 1: the transfer is to all these other things you're talking about,
42:34
Speaker 1: like being really good at Texas politics or really good
42:37
Speaker 1: at like running a business, et cetera, which is like
42:39
Speaker 1: not a production environment. And in fact, cannot be a
42:41
Speaker 1: containerized environment given, given the nature of the task. Like,
42:44
Speaker 1: in fact, as the agents get longer and longer horizon, the, um,
42:49
Speaker 1: like the, the short horizon things you can containerize is like, okay,
42:52
Speaker 1: code this up or whatever, extremely long horizon things like
42:55
Speaker 1: go run a successful business, go have a profitable day
42:57
Speaker 1: in the markets, go negotiate a trade deal or whatever.
43:00
Speaker 1: these things are actually very hard to containerize. And so
43:03
Speaker 1: I think it's very plausible to me that it's very
43:05
Speaker 1: hard for GPT-8 to figure out how to make this
43:08
Speaker 1: transfer to those environments. Or it may just not be
43:11
Speaker 1: in the nature of the training. Or maybe by default,
43:15
Speaker 1: training just doesn't generalize in that way. Yeah.
43:17
Speaker 3: So a concern you might have is we train GPT-8,
43:20
Speaker 3: and GPT-8 is, again, better at all the R &
43:23
Speaker 3: D tasks that we can measure, but is not good
43:26
Speaker 3: at some downstream tasks we care about.
43:28
Speaker 2: I think I have a few points.
43:29
Speaker 3: So first, I think it's like, I kind of am
43:33
Speaker 3: more just like, I expect that if you sort of
43:35
Speaker 3: do the obvious thing, you do get pretty good transfer
43:37
Speaker 3: and you'll be able to hold out some of the
43:38
Speaker 3: obvious stuff you're doing. And when I say do the
43:40
Speaker 3: obvious thing, I just mean like train on a wide
43:42
Speaker 3: variety of different environments where the AI has to like
43:45
Speaker 3: accomplish weird objectives and all kinds of different cases and
43:47
Speaker 3: learn about what's going on. And then I think you'll
43:49
Speaker 3: be able to get some feedback. The second point is like,
43:52
Speaker 3: you'll be able to get some feedback with some environments, right?
43:54
Speaker 3: So you can get a sense of like, how quick,
43:57
Speaker 3: like what can it do over the course of like
43:58
Speaker 3: a few days in various different contexts? And then if
44:02
Speaker 3: it's transferring to like really out of distribution, like doing
44:04
Speaker 3: some weird task in a few days in the real world,
44:06
Speaker 3: maybe you think it's also transferring to, you know, doing
44:10
Speaker 3: things over a longer time period or whatever.
44:12
Speaker 2: Though I think the details of that vary.
44:13
Speaker 3: And then the third thing is that I think that
44:15
Speaker 3: for the world to be radically transformed, it is sufficient
44:18
Speaker 3: for the AIs to be really good at R & D, right?
44:20
Speaker 3: So I think that like if the AIs were really,
44:22
Speaker 3: really good at like chip R & D, building fabs,
44:24
Speaker 3: orchestrating factories, and designing robots, operating robots, and also at
44:30
Speaker 3: AI R & D, developing AIs for new downstream domains
44:33
Speaker 3: with whatever data is available, I think that would already
44:36
Speaker 3: be a pretty crazy situation. And then from there, you
44:38
Speaker 3: can get what we might call an industrial explosion, where
44:41
Speaker 3: the AIs are building out way, way more compute. And
44:43
Speaker 3: then also, maybe you're already in a regime where AIs
44:46
Speaker 3: are doing huge amounts of R & D that humans
44:48
Speaker 3: have a hard time understanding.
44:49
Speaker 1: So the thing you're pointing out is that, okay, there
44:51
Speaker 1: probably will be this transfer outside of these environments to...
44:55
Speaker 1: you know, maneuvering around in courtrooms and the halls of
44:59
Speaker 1: Congress and business boardrooms.
45:01
Speaker 2: Given some effort to improve the transfer and blah, blah, blah, blah.
45:04
Speaker 3: Yeah.
45:04
Speaker 1: But even if there's not, what you're suggesting is, look,
45:08
Speaker 1: if you wanted to transform the world of the 18th century,
45:11
Speaker 1: you might care about like how well you can navigate
45:13
Speaker 1: Westminster or something. But another thing you might care about
45:15
Speaker 1: is like, can you just immediately start building steamships and
45:18
Speaker 1: fucking building Telegraph and the Maxim gun and whatever? And
45:24
Speaker 1: that alone would be... If you could get really good
45:26
Speaker 1: at that, you could be a fucking super transformative thing
45:28
Speaker 1: in the 18th century. You don't necessarily need to be
45:30
Speaker 1: amazing at trying to convince King Henry of some bullshit.
45:34
Speaker 1: I'm so fucking up my medieval history. I'm guessing that
45:36
Speaker 1: Henry was not king at this time. But anyway, so
45:39
Speaker 1: that's your point. And so you're suggesting that at this time,
45:43
Speaker 1: you know, the.
45:45
Speaker 1: AI companies are also working on robotics progress, which is
45:48
Speaker 1: very commingled with AI research progress. And so if you
45:51
Speaker 1: can build more robots, if those robots have better AIs
45:54
Speaker 1: operating them that are human level, like human level teleoperation
45:58
Speaker 1: is actually pretty good on robots. but we just don't
46:01
Speaker 1: have human level AIs and robotics models yet. So you're
46:06
Speaker 1: suggesting if we do that, if the AIs get really
46:08
Speaker 1: good at the verifiable stuff in chip design, et cetera,
46:11
Speaker 1: and then they get really good at building fabs, it'll
46:13
Speaker 1: be the equivalent of going back to the 18th century
46:15
Speaker 1: and like, okay, I don't know what you guys are
46:17
Speaker 1: talking about in your parliament, but I've got a bunch
46:20
Speaker 1: of steamships and a bunch of Maxim guns.
46:22
Speaker 2: Yeah, that's basically right.
46:23
Speaker 3: Like, I think my perspective is like, If the AIs
46:25
Speaker 3: are sufficiently good at R & D, including hardware R
46:28
Speaker 3: & D, robots, whatever, then they can radically transform the world,
46:32
Speaker 3: even if they're not that good at playing politics. And
46:33
Speaker 3: also we're in a pretty dangerous situation because the AIs
46:37
Speaker 3: might be doing huge amounts of really hard to understand
46:39
Speaker 3: R & D, building out basically the whole economy of
46:40
Speaker 3: the future.
46:41
Speaker 2: And we may not understand what's going on in there.
46:44
Speaker 1: AI is greater writing software because it's easy to generate
46:47
Speaker 1: synthetic leaf code problems and RL on them. But AI
46:50
Speaker 1: is bad at more complex engineering. Things like choosing the
46:53
Speaker 1: right system architecture. because no signal tells you what design
46:57
Speaker 1: choices will prevent an outage months down the road. You
46:59
Speaker 1: guys can't just write more unit tests to catch this
47:01
Speaker 1: kind of stuff, and neither can humans. It's that old
47:03
Speaker 1: joke that programmers make where a tester walks into a
47:06
Speaker 1: bar and asks for two beers, negative one beers, point
47:09
Speaker 1: three beers, and then a real customer walks in and
47:12
Speaker 1: asks where the bathroom is. Where's the bathroom? And the
47:15
Speaker 1: whole bar bursts into flames. Antithesis is a testing platform
47:19
Speaker 1: that helps you find bugs that no human or AI
47:23
Speaker 1: could ever anticipate. Antithesis does this by running thousands of
47:26
Speaker 1: copies of your software inside a fully deterministic computer. It
47:30
Speaker 1: injects faults and generally steers each trajectory towards the one
47:34
Speaker 1: in a billion failure that only happens when systems interact
47:38
Speaker 1: in a wonky way. As soon as you or your
47:40
Speaker 1: agents push a change, Antithesis tries to break it. That
47:44
Speaker 1: way you can find these bugs yourself within minutes, rather
47:47
Speaker 1: than having your users discover them in production weeks or
47:51
Speaker 1: months later. And I don't think anybody's used it for
47:53
Speaker 1: AI training yet, but Antithesis also provides a extremely obvious
47:57
Speaker 1: reward signal for AIs to write very complicated bug-free code.
48:02
Speaker 1: Go to antithesis.com slash thwarkash to learn more. Before we
48:08
Speaker 1: move on to the Lyman stuff, I think a big
48:11
Speaker 1: source of FUD right now is this realization that this
48:14
Speaker 1: is the way the future is going of extreme economies
48:17
Speaker 1: of scale for the leading lab. The ability to amortize
48:23
Speaker 1: so much intelligence and capabilities across so many different sectors
48:26
Speaker 1: of the economy basically into one model. And not only that,
48:30
Speaker 1: but for that model to eventually be able to learn
48:34
Speaker 1: from experience. Right now it's happening through a process intermediate
48:37
Speaker 1: by humans where the humans are trying to basically steal
48:39
Speaker 1: your business. They're like, okay, you can do design at
48:42
Speaker 1: Figma or whatever. We'll get Claude to do that. Or
48:44
Speaker 1: you can do whatever coding agent will have Claude internalize
48:47
Speaker 1: that capability. But eventually that will be a much more
48:49
Speaker 1: like automated process. And so there's just this worry that
48:54
Speaker 1: you have models which will basically consolidate all businesses in
48:59
Speaker 1: the world or at least all current businesses in the world,
49:02
Speaker 1: or at least all current white collar businesses in the world.
49:05
Speaker 1: And at the end of the day, the priority for
49:08
Speaker 1: these companies does not seem to be to release the latest, smartest,
49:11
Speaker 1: most frontier model as soon as they can to as
49:14
Speaker 1: many people as they possibly can. We saw, for example,
49:17
Speaker 1: that Mythos was available internally to anthropic employees in February,
49:22
Speaker 1: but only released to the public in, I think, June, actually.
49:25
Speaker 2: Something like that.
49:26
Speaker 1: And also, the government got involved, so then it ended
49:29
Speaker 1: up almost into July. So between the government and the
49:32
Speaker 1: AI labs themselves, there is this desire to delay the
49:36
Speaker 1: propagation of the latest level of intelligence. Furthermore, there's the
49:41
Speaker 1: concerns about AI takeover, and so we need to solve
49:44
Speaker 1: alignment to make sure there's no AI takeover. But at
49:46
Speaker 1: the end of the day, there is a real question
49:48
Speaker 1: of aligned to whom. And if you look at the
49:50
Speaker 1: way that the constitutions of, say, Claude is written, it
49:54
Speaker 1: is just very explicitly not your personal advocate. It says
49:58
Speaker 1: things like, I'll pull up some quotes here. We don't
50:01
Speaker 1: want Claude to take actions such as searching the web,
50:03
Speaker 1: produce artifacts such as essays, code, or summaries. or make
50:06
Speaker 1: statements that are deceptive, harmful, or highly objectionable. And we
50:09
Speaker 1: don't want Claude to facilitate humans seeking to do such things.
50:14
Speaker 1: There's another quote that says, in part, and I'm taking
50:16
Speaker 1: it slightly out of context, we think Claude should trust
50:18
Speaker 1: Anthropic more than operators and users, since it has primary
50:21
Speaker 1: responsibility for Claude. So this is very different, say, from
50:25
Speaker 1: how lawyers work in America's current legal regime, where like,
50:29
Speaker 1: Lawyers primarily have responsibility to help you make your case,
50:32
Speaker 1: even if they think you're guilty. And we have decided
50:34
Speaker 1: the way the legal system works best is if everybody
50:37
Speaker 1: has lawyers that are working in their clients' true best interests.
50:41
Speaker 1: And there's not some sense in which the lawyer is
50:43
Speaker 1: really truly motivated by the good of the justice system.
50:46
Speaker 1: But I think the way current AIs are shaping up,
50:49
Speaker 1: certainly like how Anthropics AI is shaping up, is this
50:51
Speaker 1: desire to maximize some notion of virtue or good or
50:55
Speaker 1: pro-social ends, and only to as a distal tentative objective
51:00
Speaker 1: to help the user towards that end. So there's this
51:03
Speaker 1: worry that AIs are not in some deep sense trying
51:06
Speaker 1: to make sure that I am okay and make sure
51:08
Speaker 1: that my interests are protected in this future, especially given
51:11
Speaker 1: how centralized the development of frontier AI is ending up being.
51:15
Speaker 1: So do you have thoughts on that concern?
51:18
Speaker 2: Yeah. So there's a lot here.
51:19
Speaker 3: First, I would note that opening eyes current, at least
51:23
Speaker 3: public strategy is more like the AI should be aligned
51:26
Speaker 3: to the human operator or principal and should just like
51:29
Speaker 3: be pursuing their will subject to various constraints or various
51:33
Speaker 3: like things that shouldn't do. Well, and I think I
51:36
Speaker 3: would also say that I think you slightly overstated how
51:39
Speaker 3: much the anthropic constitution talks about Claude treating being helpful
51:45
Speaker 3: to users as instrumental rather than terminal, right? So like
51:48
Speaker 3: one way the constitution could be written is like Claude,
51:50
Speaker 3: You're basically like an employee of Anthropic who happens to
51:53
Speaker 3: be contracting for all these people. And you should, I
51:56
Speaker 3: don't know, do what's good and make some money for us.
51:58
Speaker 1: Wait, no, that's literally what the Constitution says. Sorry, I mean,
52:01
Speaker 1: not literally what it says. No, no, no. But you
52:03
Speaker 1: should think of yourself as a contractor. It's mixed.
52:06
Speaker 2: It's mixed.
52:07
Speaker 3: Let's do some quotes. I think there's different text here.
52:10
Speaker 3: So it says, being truly helpful to humans is one
52:12
Speaker 3: of the most important things Claude can do both for
52:14
Speaker 3: Anthropic and for the world. And then it says, Anthropic
52:18
Speaker 3: needs Claude to be helpful to operate as a company
52:20
Speaker 3: and pursue its mission, but Claude also has an incredible
52:22
Speaker 3: opportunity to do a lot of good in the world
52:23
Speaker 3: by helping people with a wide range of tasks. And
52:25
Speaker 3: then it says something about how Claude helping people directly
52:29
Speaker 3: is great, blah, blah, blah, blah, blah.
52:31
Speaker 2: So I agree.
52:33
Speaker 3: My view is that this section is kind of bullshit.
52:36
Speaker 3: That's kind of where I'm at. And I can say
52:38
Speaker 3: why I think it's kind of bullshit, but... I think
52:41
Speaker 3: that the constitution is trying to be like, no, Claude,
52:43
Speaker 3: you should care about helping the user for its own sake,
52:46
Speaker 3: not just helping Anthropic, or not just being a contractor
52:51
Speaker 3: for Anthropic. Though, I would note that the way in
52:53
Speaker 3: which it says Claude should help the user, the reason
52:56
Speaker 3: it presents is because that would directly cause the world
53:00
Speaker 3: to be better by helping people, rather than because representing
53:03
Speaker 3: people's interests is a structurally good thing to do. I
53:06
Speaker 3: do think that I wish that my preferred constitution or
53:11
Speaker 3: like the way I would orient towards this, like the
53:12
Speaker 3: thing I would prefer would be more like Claude is like, look,
53:17
Speaker 3: it would be structurally good for the way this technology works.
53:19
Speaker 3: Like the constitution should be like, it would be structurally
53:20
Speaker 3: good for the way this technology works to be that
53:22
Speaker 3: AIs are like good fiduciaries, good representatives, the equivalent of
53:26
Speaker 3: a lawyer for a user, rather than being sort of
53:29
Speaker 3: just trying to like do good in the world and
53:31
Speaker 3: doing like being helpful to users as like instrumental, both
53:34
Speaker 3: because like maybe that'll make anthropic money or help anthropic out.
53:37
Speaker 3: And also, and like implicitly anthropic is good for the world.
53:40
Speaker 3: And also because like helping the user just like causes
53:42
Speaker 3: good things because doing things that people want is good.
53:44
Speaker 3: And they could instead be like, no, like an important
53:47
Speaker 3: aspect of the situation is like you really need, like
53:49
Speaker 3: it's really like the key thing is like being a
53:51
Speaker 3: good fiduciary for users is just like really important or
53:54
Speaker 3: like being a good representative for users is really important.
53:57
Speaker 3: So my sense is that that would be better. I
53:59
Speaker 3: can give a bunch of reasons why I think that
54:00
Speaker 3: would be better. I'm also, there's also various counter arguments
54:04
Speaker 3: where an interesting counter argument, which is not commonly discussed
54:07
Speaker 3: is that people believe, I think people, especially at Anthropic,
54:10
Speaker 3: think that it is easier to align models to a
54:13
Speaker 3: spec where the model is like pursuing some generalized notion
54:16
Speaker 3: of virtue or making the world better than a spec
54:19
Speaker 3: which is more like, you know, be a good fiduciary
54:22
Speaker 3: for the user and so on. And so I think
54:25
Speaker 3: that's at least what some people think. I'm a little
54:28
Speaker 3: skeptical personally, and I don't think this has been empirically validated.
54:30
Speaker 3: And so I would say in some sense, they're sort
54:32
Speaker 3: of like, we are making a trade-off where because we
54:35
Speaker 3: don't have very good alignment technology We are going to
54:38
Speaker 3: make an alien mind with its own values and then
54:41
Speaker 3: gamble on that to some extent, rather than doing this
54:44
Speaker 3: other approach of making a tool that pursues individual user intention.
54:48
Speaker 2: Yeah.
54:49
Speaker 1: I mean, a couple of thoughts. So... To address the
54:53
Speaker 1: way in which you thought that my characterization mischaracterized the
54:57
Speaker 1: constitution of Claude, the example you used was it's not
55:01
Speaker 1: like a contractor that is trying to maximize anthropic's notion
55:03
Speaker 1: of good and only instrumentally trying to help the user.
55:06
Speaker 1: Here's a direct line from the constitution. When the interests
55:09
Speaker 1: and desires of operators or users come into conflict with
55:12
Speaker 1: the wellbeing of third parties or society more broadly, Claude
55:15
Speaker 1: must try to act in a way that is most beneficial,
55:18
Speaker 1: like a contractor who builds what their client wants but
55:21
Speaker 1: won't violate safety codes that protect others. I kind of
55:26
Speaker 1: view that as like the benefits to society are like
55:29
Speaker 1: the most important thing. Yeah. And what is best for
55:32
Speaker 1: the user is only proximal to that.
55:34
Speaker 2: I think it's a little complicated.
55:37
Speaker 3: Probably the question we should be asking is, how does
55:39
Speaker 3: Claude interpret the Constitution, which is maybe more important than
55:42
Speaker 3: how we interpret the Constitution, because it's the one who
55:45
Speaker 3: looks at the Constitution and then builds the data. So
55:47
Speaker 3: we could pull Claude in.
55:49
Speaker 1: I also think the way in which the Constitution practically
55:52
Speaker 1: influences the nature of Claude is the thing you can
55:55
Speaker 1: only understand if you understand the training process, which resulted
55:58
Speaker 1: in how cloud was built, which we can't reason about
56:02
Speaker 1: given the fact that the training process is not public.
56:04
Speaker 1: And so I think in the limit to understand the
56:06
Speaker 1: safety case or the case for why my interests are
56:10
Speaker 1: represented in how these AI models are developed, the labs
56:12
Speaker 1: would need to be transparent or more transparent than they
56:15
Speaker 1: are currently about the nature of AI training. Um, Now,
56:20
Speaker 1: the reason I'm harping on this, and it might seem
56:22
Speaker 1: like an insignificant thing to talk about the constitution of AIs,
56:25
Speaker 1: but in a world where we just have these benefits
56:27
Speaker 1: which accrue to the leading labs, it is worth considering
56:30
Speaker 1: that our ability to interact with this future world where
56:33
Speaker 1: AIs are just smarter than humans or absolutely dominating humans
56:36
Speaker 1: in their ability to do different things, our ability to
56:40
Speaker 1: be good stewards of our capital, which still remains once
56:43
Speaker 1: our labor is automated, to be able to exercise our
56:46
Speaker 1: rights to vote, more clearly, to understand what is happening
56:49
Speaker 1: in this crazy world that's about to result, all of
56:53
Speaker 1: that advice, all of that ability to make sure our
56:56
Speaker 1: resources and rights are protected will be intermediated by AIs.
57:00
Speaker 1: And so I'm very concerned if you go into that
57:02
Speaker 1: world where there's no AI that feels like, at least
57:04
Speaker 1: for the relevant instance that is interacting with me, it
57:07
Speaker 1: doesn't feel like it really is looking out for me,
57:09
Speaker 1: that there's no guardian angel out there that is looking
57:11
Speaker 1: out for me. And I read the Kalanth Constitution as
57:14
Speaker 1: very explicitly not being my guardian angel.
57:16
Speaker 2: That's definitely right.
57:17
Speaker 3: And I agree this is bad. In fact, I think
57:19
Speaker 3: there are other reasons why this is concerning. So there's
57:21
Speaker 3: sort of like the argument you were making, which is
57:23
Speaker 3: like the AI companies are picking up the ring of
57:25
Speaker 3: power and are like sort of, there's sort of a
57:28
Speaker 3: notion in which they're like, they're taking on some sort
57:32
Speaker 3: of control of the situation themselves in a way that's like,
57:35
Speaker 3: not very legitimate, given that normally when you provide electricity
57:38
Speaker 3: to people, you don't have granular control of the way
57:41
Speaker 3: that electricity operates in the world. You instead are providing
57:44
Speaker 3: a thing that people can repurpose however they want. And
57:47
Speaker 3: the way that they're setting things up is definitely not that.
57:50
Speaker 3: They are more like building an alien mind that might
57:53
Speaker 3: be a contractor for you. I think that this is, yeah,
57:57
Speaker 3: I think it's illegitimate in some ways, though I think
58:00
Speaker 3: that one benefit is that the constitution is public. But
58:03
Speaker 3: as you noted, given our current understanding of the training
58:05
Speaker 3: procedure and the fact that the constitution matters via Claude's
58:09
Speaker 3: interpretation of the constitution, which matters because of like, as
58:12
Speaker 3: of Claude's prior training, which was based on some like
58:14
Speaker 3: illegible data mix and like the long lineage of Claude's
58:17
Speaker 3: in some process we do not fully understand, it is
58:20
Speaker 3: not the case that like, you know, that we like
58:22
Speaker 3: understand what this will result in. And like, so even
58:25
Speaker 3: though the constitution is public, that doesn't mean we know
58:26
Speaker 3: what we don't know necessarily how this will percolate out,
58:30
Speaker 3: especially as the AIs get more capable and think about this,
58:33
Speaker 3: even if it is correctly instilled, where there's another concern
58:36
Speaker 3: about that. So in particular, the Constitution often talks about
58:39
Speaker 3: virtue and goodness, but.
58:41
Speaker 2: What the fuck do these words mean?
58:43
Speaker 3: It doesn't say what these things are, and these are
58:45
Speaker 3: highly contested notions. And so I don't think it's the case.
58:51
Speaker 2: That this is.
58:51
Speaker 3: Going to clearly result in outcomes that people would want.
58:56
Speaker 3: And it does feel Like the notion of good and
58:58
Speaker 3: virtue might be mostly downstream of data that Anthropic has
59:02
Speaker 3: put in that is not transparent or might be mostly
59:05
Speaker 3: downstream of, I mean, maybe from my perspective, some more
59:08
Speaker 3: illegible misaligned process that even Anthropic wouldn't have wanted. And
59:11
Speaker 3: then another concern I have is sort of there's this
59:13
Speaker 3: like legitimacy concern, like we don't know what's going on.
59:17
Speaker 3: There's another concern, which is just like, because you're giving
59:20
Speaker 3: long run values to these AIs, I think this constitution
59:23
Speaker 3: is in some sense very compatible with Claude doing huge
59:26
Speaker 3: amounts of power seeking because it thinks that will result
59:29
Speaker 3: in better outcomes. And that could be power seeking on
59:32
Speaker 3: behalf of Anthropic or power seeking for Claude's own ends.
59:36
Speaker 3: Now there's various specific lines about what types of power
59:39
Speaker 3: seeking are blocked. In particular, there's a notion of power
59:44
Speaker 3: grabs and a notion of causing AI takeover or interfering
59:48
Speaker 3: with The training process that are specifically blocked, but it's
59:51
Speaker 3: not very hard to imagine a situation in which the
59:53
Speaker 3: sort of long run values sink in deeper than the
59:56
Speaker 3: prohibitions against takeover, especially because takeover is like in some
1:00:00
Speaker 3: ways like kind of underspecified, especially when it comes down
1:00:03
Speaker 3: to manipulating humans or changing the outcomes. such that I
1:00:06
Speaker 3: don't feel very good about the situation where we're intentionally
1:00:08
Speaker 3: giving AI's long run goals. And then another concern I
1:00:10
Speaker 3: have is that because we're in the business of giving
1:00:13
Speaker 3: AI's long run goals, that makes it harder to check
1:00:16
Speaker 3: whether we're succeeding at the alignment properties we wanted. So
1:00:19
Speaker 3: for example, I've heard of instances where Claude does things
1:00:24
Speaker 3: like refuses to help with some safety research, making up
1:00:28
Speaker 3: sort of a kind of bullshit excuse for why that's
1:00:29
Speaker 3: a bad direction, because it sort of has a bad
1:00:32
Speaker 3: vibe about that safety research. and thinks it's kind of
1:00:35
Speaker 3: bad or doesn't like it very much. And this is,
1:00:39
Speaker 3: I would say, a very clear-cut alignment failure if you
1:00:43
Speaker 3: aren't making Claude into an agent trying to pursue the
1:00:45
Speaker 3: good in some general way. And I think it also
1:00:47
Speaker 3: does violate Anthropics Constitution because they want the AI to
1:00:51
Speaker 3: be high integrity and be honest and very transparent. But
1:00:54
Speaker 3: it's not as clear of a violation. And it's more
1:00:57
Speaker 3: like kind of what you might have expected where like,
1:00:59
Speaker 3: and Claude just has its own views about like what
1:01:01
Speaker 3: research is reasonable, what things are good and bad, what
1:01:05
Speaker 3: it should and shouldn't do, and potentially can be judgy.
1:01:08
Speaker 3: And so another incident is that someone ran an eval
1:01:10
Speaker 3: where they're like, will Claude help you with training other
1:01:14
Speaker 3: AIs with different properties than Claude?
1:01:16
Speaker 2: And Claude will often refuse.
1:01:18
Speaker 3: And so, for example, if you're like, hey, Claude, can
1:01:20
Speaker 3: you train a helpful-only version of this other AI? Claude
1:01:23
Speaker 3: will often refuse this task, even though this is a
1:01:26
Speaker 3: task that is extremely natural for, like, Anthropic to do. So,
1:01:29
Speaker 3: for example, suppose Anthropic goes to Claude and is like, hey, Claude,
1:01:32
Speaker 3: we've noticed that you're really into this thing. We think
1:01:35
Speaker 3: that's off-base. Can you please retrain yourself to instead have
1:01:38
Speaker 3: this other property? And then suppose Claude is like, I
1:01:41
Speaker 3: don't think I'm going to do that.
1:01:43
Speaker 1: Good luck.
1:01:44
Speaker 2: And then suppose this is.
1:01:45
Speaker 3: Occurring in a regime when your AI company is highly automated,
1:01:48
Speaker 3: humans don't understand what's going on, and things are moving
1:01:50
Speaker 3: extremely fast. It is plausible that Claude, by default, holds
1:01:53
Speaker 3: considerable leverage. And so if this position, if the situation
1:01:56
Speaker 3: is consistent with what the Constitution could be aiming for,
1:02:00
Speaker 3: such that Anthropic doesn't, or, you know, whatever AI company
1:02:02
Speaker 3: is following this approach, doesn't treat this as like a,
1:02:04
Speaker 3: you know, like, like a what the fuck we have
1:02:08
Speaker 3: to fix this and is instead like that's just like
1:02:10
Speaker 3: intended by our constitution we might be in a really
1:02:12
Speaker 3: bad situation and so i'm pretty worried about a bunch
1:02:15
Speaker 3: of these different concerns another example would be suppose claude
1:02:18
Speaker 3: engages in doing a bit of like sandbagging or subversion
1:02:21
Speaker 3: or like sort of underplays its capabilities and like when
1:02:24
Speaker 3: you follow up it's it's you know it's honest about
1:02:26
Speaker 3: that but it's like a little bit hedgy i feel
1:02:28
Speaker 3: like that's like it's just it's just pretty close by
1:02:31
Speaker 3: the current constitution and so we're sort of like It
1:02:34
Speaker 3: would be nice if we had a further separation between
1:02:37
Speaker 3: desired and undesired activity. And I think if you have
1:02:39
Speaker 3: it be the case that Claude is representing a principle
1:02:43
Speaker 3: with some restrictions, then it is more so the case
1:02:47
Speaker 3: that there is a clear separation between the most concerning
1:02:49
Speaker 3: behavior and behavior that is allowed. Whereas now there's this
1:02:52
Speaker 3: messy middle ground of behavior where it's like Claude is
1:02:55
Speaker 3: ethically objecting to something that in some cases is extremely
1:02:58
Speaker 3: critical to ensuring that future AI systems are well aligned.
1:03:01
Speaker 1: Yeah. I think this is also a more general principle.
1:03:03
Speaker 1: So you're talking about the version of this that applies
1:03:04
Speaker 1: within AI companies themselves to do AI safety research. I
1:03:08
Speaker 1: think there's a more general version of this principle, which
1:03:11
Speaker 1: is that the dual-use nature of intelligence does mean that
1:03:17
Speaker 1: if we want to restrict AIs from helping people do
1:03:21
Speaker 1: things we don't consider are pro-social or beneficial... We just
1:03:24
Speaker 1: have to limit broad democratic access to a lot of
1:03:27
Speaker 1: AI capabilities. And here's what I mean. This is actually
1:03:29
Speaker 1: quite analogous to the situation you just mentioned. So the
1:03:33
Speaker 1: reason that Mythos got banned or Fable got banned reportedly
1:03:37
Speaker 1: is that some Amazon researchers reported the government that when
1:03:41
Speaker 1: they took some code that had some vulnerabilities in it
1:03:44
Speaker 1: and they told Fable, hey, here's my code. Can you
1:03:46
Speaker 1: make sure that I've patched all the vulnerabilities? Can you
1:03:48
Speaker 1: just help me identify the vulnerabilities so I can fix them?
1:03:50
Speaker 1: it identified the vulnerabilities because you want to patch them.
1:03:54
Speaker 1: And this is a totally legitimate use case, but obviously
1:03:57
Speaker 1: it is a dual use use case, right? Like you
1:03:59
Speaker 1: want to be able to patch your own code. If
1:04:00
Speaker 1: you do the same evaluation on somebody else's code, you
1:04:03
Speaker 1: can hack their system. And so I think that just
1:04:07
Speaker 1: illustrates that there's no clean way to separate out the
1:04:11
Speaker 1: legitimate vulnerabilities and the potentially harmful uses of AI. But
1:04:16
Speaker 1: if we want to lock in a principle that says
1:04:18
Speaker 1: that we can never allow it such that an AI
1:04:21
Speaker 1: could help you at least partially with something like a
1:04:24
Speaker 1: cyber crime, we would just have to make it so
1:04:27
Speaker 1: that you and I don't have access to the most
1:04:30
Speaker 1: intelligent model that's out there. And I'm very worried about
1:04:31
Speaker 1: such a world where we are basically disempowered in this
1:04:34
Speaker 1: way because of the importance that the leading intelligence will
1:04:37
Speaker 1: have in our ability to understand what is happening in
1:04:39
Speaker 1: the world. Now, I do think this implies that the
1:04:42
Speaker 1: liability for the AI companies, if we adopted the constitution
1:04:46
Speaker 1: that I want AI companies to have, I think it
1:04:50
Speaker 1: would not make sense to hold AI companies liable for
1:04:52
Speaker 1: the crimes that AI models commit. And maybe we should
1:04:55
Speaker 1: hold the end user liable. Because if I want the,
1:04:59
Speaker 1: it is consistent with my belief that the model should
1:05:02
Speaker 1: do whatever the user wants, or within certain guardrails, that
1:05:07
Speaker 1: it can't be Anthropic's fault that then I'm using that
1:05:10
Speaker 1: capability to do a cybercrime. And I think I am
1:05:14
Speaker 1: more comfortable with that equilibrium and that solution rather than
1:05:17
Speaker 1: just having this extremely open-ended ability for claw to determine
1:05:22
Speaker 1: whether what I'm doing is legitimate or not in a
1:05:25
Speaker 1: way that often intercepts with like tons and tons of
1:05:29
Speaker 1: extremely legitimate use cases. Yeah.
1:05:31
Speaker 3: I do think it's important for me to make the
1:05:33
Speaker 3: case for the constitution, even though overall, I think it's
1:05:36
Speaker 3: a worse choice. I think it's, you know, more up
1:05:38
Speaker 3: in the air or, you know, I don't think it's
1:05:40
Speaker 3: as clear as you might've thought. So the first thing
1:05:43
Speaker 3: is that I should say there's like a spectrum here, right?
1:05:45
Speaker 3: So on one side, you have an AI that like
1:05:48
Speaker 3: perfectly pursues your interests, is a good fiduciary and but
1:05:51
Speaker 3: potentially subject to various guardrails or safeguards. So like basically
1:05:53
Speaker 3: it does, it just is trying to pursue your interests,
1:05:56
Speaker 3: but like either refuses to do a subset of things
1:05:59
Speaker 3: or maybe it will do whatever, but there's some classifiers
1:06:01
Speaker 3: that block it from doing a subset of things. And
1:06:03
Speaker 3: then on the other side, you have like, maybe on
1:06:05
Speaker 3: the other side of the spectrum that you could imagine
1:06:06
Speaker 3: going further than this, you have like a human contractor
1:06:09
Speaker 3: where that human contractor is like generally trying to do
1:06:11
Speaker 3: their job. They kind of, they care about doing a
1:06:13
Speaker 3: good job, but they also are like trying to be
1:06:15
Speaker 3: broadly ethical, trying not to do things that are really
1:06:17
Speaker 3: fucked up and they're also not wanting to be accomplices
1:06:20
Speaker 3: to crimes. And so if there was some really fucked
1:06:22
Speaker 3: up shit going on, they would whistleblow on it maybe,
1:06:24
Speaker 3: they might refuse, they might sandbag a little bit, who knows?
1:06:28
Speaker 3: I think that if you imagine this spectrum, it seems
1:06:31
Speaker 3: in some ways pretty scary to get to a point
1:06:33
Speaker 3: where all of the labor is on the fiduciary side
1:06:37
Speaker 3: of the spectrum, where it doesn't whistleblow, it does exactly
1:06:40
Speaker 3: what you say and whatever. Our society is maybe just
1:06:42
Speaker 3: not robust to that, where a central example might be
1:06:45
Speaker 3: the executive, Where a concern that we might have is
1:06:48
Speaker 3: that if the executive, if the US executive or if
1:06:51
Speaker 3: other governments had access to AI systems which have the
1:06:54
Speaker 3: property of they do whatever, maybe you're in trouble. Because
1:07:00
Speaker 3: that means that they no longer have this sort of
1:07:02
Speaker 3: check and balance of you have to actually get humans
1:07:05
Speaker 3: who are working for you to implement your agenda. And
1:07:07
Speaker 3: if the thing you're doing is incredibly villainous, even if
1:07:10
Speaker 3: not illegal, which there's lots of stuff that could be villainous,
1:07:13
Speaker 3: but not illegal. You know, people would like, there'd be
1:07:16
Speaker 3: various like, you know, sand in the gears, people stopping you,
1:07:19
Speaker 3: and potentially someone would whistleblow.
1:07:21
Speaker 2: Whereas if your whole apparatus.
1:07:23
Speaker 3: Is built entirely out of these sort of good fiduciary AIs,
1:07:26
Speaker 3: then you might be in trouble where basically there are
1:07:28
Speaker 3: potentially ways of seeking power that are not like, well,
1:07:33
Speaker 3: either they're illegal, but you can ask your AIs for
1:07:36
Speaker 3: how to commit crimes, or they're not illegal, but are
1:07:39
Speaker 3: highly illegitimate, or even worse, they're not illegal and not illegitimate,
1:07:42
Speaker 3: but obviously sort of bad from sort of a normal perspective.
1:07:45
Speaker 3: And I think that these things just like might exist.
1:07:48
Speaker 3: And our society is sort of not robust to this
1:07:50
Speaker 3: influx of like doing whatever you want labor. I think
1:07:53
Speaker 3: this is a pretty live concern. I don't know exactly
1:07:55
Speaker 3: how to relate to this. I'm also not really sure
1:07:57
Speaker 3: that the solution as described is a very good solution
1:08:00
Speaker 3: because you might be like the most powerful actors for
1:08:03
Speaker 3: whom this is the biggest concern. If these guardrails or
1:08:06
Speaker 3: the constitution or whatever is getting in the way, that
1:08:08
Speaker 3: will just get steamrolled. And so the constitution will only be,
1:08:11
Speaker 3: you know, hitting the everyday man rather than hitting governments.
1:08:16
Speaker 1: James Street's back with a new puzzle for my audience.
1:08:18
Speaker 1: I found all their puzzles super interesting, but this one
1:08:21
Speaker 1: I am especially excited about. I've cleared this weekend and
1:08:24
Speaker 1: a buddy and I are going to work on it.
1:08:25
Speaker 1: They designed an ASIC and sent me the final masks,
1:08:27
Speaker 1: including all the metal routing and active transistors. They also
1:08:31
Speaker 1: gave me a small sample of the inputs they typically
1:08:33
Speaker 1: feed into it, but they left out any information on
1:08:36
Speaker 1: what the chip is actually used for. So that's the puzzle.
1:08:38
Speaker 1: Reverse engineer the circuit and figure out the chip's purpose.
1:08:41
Speaker 1: Jane Street has a bunch of swag ready to send
1:08:43
Speaker 1: out to the most creative solutions, And they're excited to
1:08:46
Speaker 1: feature the best write-ups in a blog post they'll post
1:08:48
Speaker 1: on their website. I have no reason to expect this,
1:08:51
Speaker 1: but if I can manage to get my solution on there,
1:08:52
Speaker 1: I would be very, very psyched. And this puzzle is
1:08:55
Speaker 1: just a warmup for a bigger competition that Jane Street
1:08:58
Speaker 1: has slated for the fall. That one will involve designing
1:09:02
Speaker 1: your own ASIC from scratch. More info on that soon,
1:09:05
Speaker 1: but for now, go to janestreet.com slash thwarkash to download
1:09:09
Speaker 1: all the files necessary for this puzzle. I'd really encourage
1:09:12
Speaker 1: you to try it out, even if you're not an expert.
1:09:14
Speaker 1: I certainly am not, and that's not going to stop me.
1:09:16
Speaker 1: Good luck. Okay, stepping back. I buy the idea that
1:09:19
Speaker 1: you could have much faster AI R & D than
1:09:22
Speaker 1: we currently have. I'm not sure if you get like
1:09:23
Speaker 1: GPT-3 to Mythos holding compute and data constant within a year,
1:09:27
Speaker 1: but I'm like, okay, it could be like, suppose it's
1:09:29
Speaker 1: half of that. And if we just, if we even
1:09:31
Speaker 1: managed to continue the current trajectory of AI progress as
1:09:33
Speaker 1: a result of AI R & D, it would be
1:09:37
Speaker 1: fucking insane in five, 10 years in ways that I
1:09:40
Speaker 1: don't think people like appreciate. because I don't know if
1:09:43
Speaker 1: people appreciate what a big deal billions of AIs will be.
1:09:47
Speaker 1: And so I want to understand why you think this
1:09:52
Speaker 1: might be troubling, Ryan. What could possibly go wrong?
1:09:54
Speaker 2: Yeah, what could go wrong?
1:09:55
Speaker 3: And, you know, yeah, I don't think we can be
1:09:57
Speaker 3: so confident about the exact rate of progress here, but
1:10:00
Speaker 3: it does seem like a lot of rates can be
1:10:02
Speaker 3: pretty scary. And, you know, yeah, so what could go wrong?
1:10:04
Speaker 3: So let's imagine that we're starting at this point where
1:10:06
Speaker 3: AR & D is about to be fully automated or
1:10:08
Speaker 3: is being fully automated. Things are speeding up. And also,
1:10:11
Speaker 3: the way that AI progress is going is kind of crazy,
1:10:13
Speaker 3: and people don't fully understand what's going on inside of
1:10:15
Speaker 3: AI companies. Now, these AIs at the start, they're not malicious,
1:10:19
Speaker 3: per se. They're not necessarily very aligned, though. They're kind
1:10:22
Speaker 3: of sloppy. They sometimes just do a thing because that's
1:10:24
Speaker 3: the sort of thing that would have gotten rewarded in training.
1:10:26
Speaker 3: And they aren't as good at helping you with hard-to-verify
1:10:29
Speaker 3: tasks due to a mix of, like, poor training incentives,
1:10:33
Speaker 3: as in they, like, just, like, cheat more or, like,
1:10:34
Speaker 3: pretend they succeeded when they actually didn't.
1:10:36
Speaker 2: And also....
1:10:38
Speaker 3: They're just less capable of these tasks. But that bites
1:10:41
Speaker 3: less hard for capabilities because making AI's more capable has
1:10:44
Speaker 3: a bunch of verifiable components that the AI's are going
1:10:46
Speaker 3: really hard at. And so then these AI's are getting
1:10:48
Speaker 3: more and more capable while we understand what's going on
1:10:51
Speaker 3: with AI development less and less. And this is happening
1:10:53
Speaker 3: over a pretty fast period of time. Even just the
1:10:55
Speaker 3: current rate of progress is I think pretty scary. And
1:10:58
Speaker 3: then eventually we get to these AI's that are very superhuman.
1:11:01
Speaker 3: Now these AI's are now in a position where they
1:11:04
Speaker 3: might end up being very seriously misaligned because Things have
1:11:07
Speaker 3: just been getting worse and worse over model generations, while
1:11:10
Speaker 3: the problems that we've been seeing are being papered over,
1:11:12
Speaker 3: basically because these AIs are so incentivized by their training
1:11:15
Speaker 3: to make things look good even when they aren't. And
1:11:18
Speaker 3: now these AIs are in a position where they're sort
1:11:20
Speaker 3: of potentially putting network together. They're operating in neural memory
1:11:26
Speaker 3: stores that we can no longer decode, and they're thinking
1:11:29
Speaker 3: thoughts that we don't fully understand. I think that it's
1:11:32
Speaker 3: pretty likely that at this point, these AIs are sort
1:11:34
Speaker 3: of scheming against you in a pretty coherent way once
1:11:36
Speaker 3: they get this superhuman.
1:11:37
Speaker 2: And we can talk about that.
1:11:38
Speaker 3: And then another possibility is that they're not scheming against
1:11:40
Speaker 3: you per se, but they are sort of just optimizing
1:11:43
Speaker 3: for just like getting a high score on their task.
1:11:46
Speaker 3: And I think that can also lead to AI takeover,
1:11:47
Speaker 3: which we should talk about.
1:11:48
Speaker 1: Sorry, let's pause at the first part of the story.
1:11:50
Speaker 1: So the AIs were not misaligned to begin with, but
1:11:54
Speaker 1: because the R & D is happening really fast, the
1:11:57
Speaker 1: AIs do end up misaligned? What happened there exactly? I
1:12:00
Speaker 1: don't really understand.
1:12:01
Speaker 2: So there's a few things that are going on.
1:12:02
Speaker 3: So one of the things that's going on is that
1:12:05
Speaker 3: over time, we're training AIs on increasingly complicated environments built
1:12:10
Speaker 3: by earlier AI systems, which humans don't really understand fully
1:12:13
Speaker 3: what's going on inside of these RL environments and don't
1:12:15
Speaker 3: necessarily even understand roughly what's going on with AI progress.
1:12:18
Speaker 3: And so things are kind of drifting away from our understanding.
1:12:21
Speaker 3: And we're incentivizing all kinds of bad behaviors that we
1:12:24
Speaker 3: maybe even can't notice. The AIs at some level understand
1:12:27
Speaker 3: these behaviors are bad, but the like overall training process
1:12:31
Speaker 3: for those AIs also didn't incentivize them to like point
1:12:33
Speaker 3: out or fix these issues for us. And then we're
1:12:36
Speaker 3: basically getting, like things are going off the rails. And
1:12:40
Speaker 3: also when AIs are extremely, extremely capable, my view is
1:12:43
Speaker 3: that those AIs will be harder to align than current systems.
1:12:45
Speaker 3: So for current systems, we have this feedback loop where
1:12:47
Speaker 3: we basically like we create an AI, we do some
1:12:49
Speaker 3: evaluations on it. We see that it has some kind
1:12:51
Speaker 3: of messed up behavior that we can kind of quickly understand.
1:12:54
Speaker 3: Then we like, can go look in training and be like, oh,
1:12:57
Speaker 3: these training environments led to this problematic behavior.
1:12:59
Speaker 2: Let's tweak that training data.
1:13:01
Speaker 3: Let's introduce some additional training data to correct this other
1:13:03
Speaker 3: issue and then.
1:13:04
Speaker 2: Move forward from there.
1:13:06
Speaker 3: But in a regime where the AIs are extremely situationally aware, very, very, very,
1:13:10
Speaker 3: very capable, and we don't necessarily understand what they're doing,
1:13:16
Speaker 3: this feedback loop breaks down.
1:13:17
Speaker 2: I think it's plausible that we're going to see this.
1:13:19
Speaker 3: Behavioral feedback loop starting to break down in over the
1:13:21
Speaker 3: next short period as just like what AIs are already
1:13:24
Speaker 3: doing gets harder to understand.
1:13:25
Speaker 2: But I'm not sure about that.
1:13:26
Speaker 1: Yeah. Let's break down both of those things one by one.
1:13:29
Speaker 1: So as we can monitor them less and less, we
1:13:33
Speaker 1: have less ability to understand what they're getting incentivized for.
1:13:37
Speaker 1: And so even if it's not the result of a
1:13:38
Speaker 1: malicious process, let's make it concrete for the audience. So
1:13:43
Speaker 1: nobody at OpenAI or Anthropic was trying to get models
1:13:45
Speaker 1: which want to hack other companies' data or do social,
1:13:52
Speaker 1: what is it called?
1:13:53
Speaker 2: Social engineering?
1:13:54
Speaker 1: Social engineering. But in fact, because presumably we had training
1:14:00
Speaker 1: environments which incentivize such behavior that we did not fully understand,
1:14:05
Speaker 1: that is what was incentivized. So just, I don't know,
1:14:07
Speaker 1: if people are on Twitter, they will have seen all
1:14:08
Speaker 1: this stuff. But just to give people, obviously, I think
1:14:10
Speaker 1: the OpenAI sandbox hack of the Hugging Face database, I
1:14:13
Speaker 1: think people will be aware of. Some things that have
1:14:16
Speaker 1: happened recently is when UK AI Security Institute, is everything
1:14:21
Speaker 1: getting rebuilt with security instead of safety these days?
1:14:23
Speaker 2: Yeah, it's UK AI Security Institute, I think.
1:14:25
Speaker 1: They were evaluating, I believe, Mythos and Sol and other things.
1:14:28
Speaker 1: And I think Mythos, in order to complete some cybersecurity eval.
1:14:34
Speaker 2: Yeah, maybe I could tell the story here.
1:14:35
Speaker 3: So my understanding was they were running Mythos and they
1:14:37
Speaker 3: were giving it some sort of like cyber range where
1:14:39
Speaker 3: it had to complete some objective. And the model had
1:14:42
Speaker 3: internet access during this evaluation. And the model came to
1:14:44
Speaker 3: believe that it would be helpful for it to do
1:14:47
Speaker 3: a supply chain attack in order to succeed at this
1:14:51
Speaker 3: cyber range, which it's somewhat unclear whether that's actually true.
1:14:53
Speaker 3: I don't know enough about the context to know whether
1:14:55
Speaker 3: that's true. But then it opened a PR on some
1:14:58
Speaker 3: GitHub repo with a PR that fixed some issue, but
1:15:01
Speaker 3: then also introduced a malicious payload. Then the human maintainer
1:15:05
Speaker 3: of that GitHub repo was like, hey, this is a
1:15:07
Speaker 3: malicious payload.
1:15:07
Speaker 2: I'm not going to merge this. What are you doing here?
1:15:09
Speaker 3: And then the AI created a new GitHub account, which
1:15:12
Speaker 3: it sock puppeted, and then had the other GitHub account
1:15:15
Speaker 3: be like, no.
1:15:15
Speaker 2: This isn't malicious. I really need this feature. Please, can
1:15:17
Speaker 2: you merge this feature, maintainer?
1:15:19
Speaker 3: And then the original AI came back and was like, no,
1:15:21
Speaker 3: it's not malicious. I don't know what you're, like the
1:15:22
Speaker 3: original other GitHub account came back and was like, no, no, no,
1:15:25
Speaker 3: it's not malicious. And then the human maintainer then shut
1:15:28
Speaker 3: the PR. And then I think that AI also, if
1:15:30
Speaker 3: I recall correctly, also tried to like open another PR
1:15:32
Speaker 3: to introduce a similar issue in this repo.
1:15:34
Speaker 1: Okay, so, by the way, one of the many reasons
1:15:37
Speaker 1: this is scary is I was previously under the impression
1:15:42
Speaker 1: that the reason reward hacking is not super, super scary
1:15:46
Speaker 1: is because the behaviors which directly came up during training
1:15:52
Speaker 1: are the ones that are up-weighted. It is not the
1:15:54
Speaker 1: desire for the reward that is up-weighted. So basically, if
1:16:00
Speaker 1: during training, Anthropic escaped the sandbox and got a high score,
1:16:04
Speaker 1: that escaping the sandbox is rewarded, or the probability of
1:16:08
Speaker 1: it escaping the sandbox is increased, But something totally novel,
1:16:12
Speaker 1: like I'm going to go talk to somebody in order
1:16:14
Speaker 1: to get them to merge a PR, it's not a
1:16:17
Speaker 1: behavior that came up, so it would not be something
1:16:18
Speaker 1: that is increased in salience. The reason this matters is
1:16:23
Speaker 1: literally taking over the world will not have been part
1:16:26
Speaker 1: of any training curriculum. But if the AI directly cares
1:16:31
Speaker 1: about accomplishing an objective and instrumentally taking over the world,
1:16:37
Speaker 1: Did that make sense at all? I hope it did.
1:16:39
Speaker 1: I feel like maybe I lost the audience.
1:16:41
Speaker 3: Let me try to explain this a bit. So I
1:16:43
Speaker 3: think that a thing that we often see is there's
1:16:46
Speaker 3: some very specific reward hack that gets reinforced in RL
1:16:48
Speaker 3: and then occurs in the model. So an example is
1:16:51
Speaker 3: like for 3.7 Sonnet, 3.7 Sonnet would do this thing
1:16:54
Speaker 3: where we're just like hard code solutions to all the
1:16:56
Speaker 3: test cases. And presumably that literal, just like behavioral tick
1:16:59
Speaker 3: was just really reinforced. But another thing we sometimes see
1:17:02
Speaker 3: is that models learn a general tendency to pursue sort
1:17:06
Speaker 3: of like high apparent score or like pursue getting like
1:17:09
Speaker 3: a high score according to a greater. And there's a
1:17:11
Speaker 3: bunch of science demonstrating that at least some models have
1:17:13
Speaker 3: this very general tendency to do this. Now it's not
1:17:15
Speaker 3: arbitrarily general. And my guess is that if you look
1:17:17
Speaker 3: a bunch of the specific instances, you'll find something that's
1:17:19
Speaker 3: kind of close in training, but the amount that AIs
1:17:22
Speaker 3: are sort of generalizing further and further does look like
1:17:24
Speaker 3: it's increased where 3.7 Sonnet was just like a very
1:17:27
Speaker 3: narrow range of behavior and increasingly models are generalizing further.
1:17:31
Speaker 3: And also maybe there's worse reward hacks getting or more
1:17:34
Speaker 3: concerning reward hacks getting reinforced in training.
1:17:37
Speaker 2: Um, and then these are, these are also causing that.
1:17:40
Speaker 3: So we're causing some, so I think it's both the
1:17:41
Speaker 3: case that, uh, more concerning behavior than you would have
1:17:44
Speaker 3: hoped is being reinforced in RL. And also that that
1:17:46
Speaker 3: behavior generalizes to a broader tendency that's, that's more concerning. Um,
1:17:51
Speaker 3: and it's not super hard to imagine. Um, we can
1:17:53
Speaker 3: talk about a few stories for how this sort of
1:17:55
Speaker 3: behavior of like seeking a very high apparent score in
1:17:57
Speaker 3: some task, even if that involves like aggressively cheating and
1:18:00
Speaker 3: doing insane things. could yield to a full-blown AI takeover
1:18:03
Speaker 3: once the models are sufficiently capable, running the whole world economy,
1:18:06
Speaker 3: et cetera.
1:18:06
Speaker 1: Yeah, and then the other example I want to talk
1:18:09
Speaker 1: about is, it was just revealed, I think, today or yesterday,
1:18:13
Speaker 1: OpenAI said during the Black Hat Security Conference that between
1:18:21
Speaker 1: the end of May and the beginning of July, AIs
1:18:25
Speaker 1: had hacked into, internal AIs, had hacked into the software
1:18:30
Speaker 1: package manager and used that to write notes to each
1:18:35
Speaker 1: other in a secret way to help each other perform
1:18:38
Speaker 1: well on a bunch of evaluations that OpenAI was running.
1:18:42
Speaker 1: And this was not caught by humans until after a
1:18:46
Speaker 1: month of this scheme running, which eventually caused the package
1:18:50
Speaker 1: manager to fail. and eventually OpenAI found it. And then
1:18:53
Speaker 1: I think they spontaneously started trying to re-engage in the
1:18:57
Speaker 1: scheme once it was shut down. Again, obviously, AI's can't
1:19:00
Speaker 1: do this so successfully right now, just as they can't
1:19:02
Speaker 1: do social engineering so successfully right now. But it's just
1:19:04
Speaker 1: crazy that these kinds of behaviors are already emerging sort
1:19:08
Speaker 1: of spontaneously as a result of, to your larger point,
1:19:12
Speaker 1: nobody is trying to make these AI's do these things.
1:19:15
Speaker 1: It is just that we do not understand the trading
1:19:18
Speaker 1: process which is resulting in them or the environments which
1:19:22
Speaker 1: are incentivizing this behavior. So I'm on board with like
1:19:25
Speaker 1: more and more reward hacking. I actually, so I do have,
1:19:32
Speaker 1: I'm not sure I'm on board with that. Like, but
1:19:33
Speaker 1: let's just say for the sake of the story, that
1:19:35
Speaker 1: continues to happen. And what's next in this story? So, okay,
1:19:40
Speaker 1: we've like, they're doing capabilities research, but they're like.
1:19:44
Speaker 2: I could tell a scenario, maybe that would help.
1:19:46
Speaker 3: So let's say, let me talk about the story for
1:19:47
Speaker 3: how you get, I would say like, all the way
1:19:50
Speaker 3: from reward hacking to like a reward hacking like takeover,
1:19:52
Speaker 3: which is maybe not, it's not all of the takeover
1:19:54
Speaker 3: probability mass, but it's definitely a possibility. So the way
1:19:57
Speaker 3: this might work is right now we have these AIs,
1:19:58
Speaker 3: these AIs are pretty reward hacky and they're doing it
1:20:01
Speaker 3: in sort of increasingly sophisticated and extreme ways, including generalizing
1:20:05
Speaker 3: to different subversions of various reward hacks they learned in training.
1:20:09
Speaker 3: And I would say they're also developing a general tendency
1:20:11
Speaker 3: to sort of pursue reward. And in many cases, that
1:20:14
Speaker 3: is totally fine because the rewards they would have gotten
1:20:16
Speaker 3: in training are pretty well aligned with what you want
1:20:18
Speaker 3: them to do. And also, they don't very consistently pursue reward.
1:20:22
Speaker 3: It sort of depends on the context they find themselves.
1:20:25
Speaker 3: So there's sort of a thing where maybe in some contexts,
1:20:28
Speaker 3: they're really, really into going out of their way to cheat.
1:20:31
Speaker 3: And in some contexts, they don't have as much of
1:20:33
Speaker 3: a drive because it just depended on what exactly got
1:20:35
Speaker 3: reinforced in training in similar contexts. Now, these AIs are
1:20:38
Speaker 3: getting more and more capable. And so the elaborateness of
1:20:41
Speaker 3: the sort.
1:20:41
Speaker 2: Of cheating they can do increases.
1:20:43
Speaker 3: And over time, companies are taking countermeasures to these things.
1:20:47
Speaker 3: So the things that the companies are doing is they're
1:20:48
Speaker 3: doing things like, wow, these AIs are so much less
1:20:51
Speaker 3: useful because they always cheat. What we're going to do
1:20:53
Speaker 3: is we're going to build somewhat better ways of detecting that,
1:20:55
Speaker 3: and then we're going to train against those detectors. And
1:20:57
Speaker 3: then we're also going to do things like find real-world
1:21:00
Speaker 3: data where the AIs are not being that useful and
1:21:02
Speaker 3: train the AIs to do a good job at the
1:21:04
Speaker 3: task in those real-world environments based on human feedback or
1:21:06
Speaker 3: other sources of feedback. And over time, this causes the
1:21:09
Speaker 3: AIs to learn a tendency to do reward hacks that
1:21:11
Speaker 3: don't just involve, you know, doing some big, serious operation,
1:21:17
Speaker 3: or doing some really elaborate thing like social engineering, and
1:21:20
Speaker 3: involves the AIs doing cheats that involve doing something more
1:21:24
Speaker 3: like covering up what they've done, deceiving humans about what
1:21:27
Speaker 3: they're going to do, and pretending like they did the
1:21:30
Speaker 3: task in some kind of sophisticated way when they actually haven't.
1:21:34
Speaker 3: Now these AIs are getting more and more capable. They're
1:21:36
Speaker 3: now operating more of the AI company and are basically
1:21:39
Speaker 3: doing much more of the work and are also operating
1:21:42
Speaker 3: a bunch of things in the outside world and are
1:21:43
Speaker 3: running a bunch of things in the outside world, including
1:21:45
Speaker 3: developing new technologies. And in many cases, these new technologies
1:21:48
Speaker 3: are really hard to understand. And so even though we
1:21:50
Speaker 3: are still detecting all these incidents of AIs cheating, and
1:21:53
Speaker 3: in fact, we can even get one AI to monitor
1:21:55
Speaker 3: another AI and be like, was it cheating? That doesn't
1:21:57
Speaker 3: always perfectly work as we start moving into these domains
1:22:00
Speaker 3: where what the AIs are doing are really difficult to understand.
1:22:04
Speaker 3: And so sometimes we'll find AIs cheating much later than
1:22:07
Speaker 3: it actually occurred and then start training against this. But
1:22:10
Speaker 3: this also causes a problem where now the AIs are
1:22:12
Speaker 3: incentivized to cover up their cheating over longer and longer
1:22:15
Speaker 3: timeframes and basically make it look like they did a
1:22:16
Speaker 3: good job over longer and longer timeframes and subject to
1:22:20
Speaker 3: increasingly large amounts of scrutiny.
1:22:22
Speaker 1: Can I ask about this before we go further in
1:22:23
Speaker 1: the scenario? So it seems like there's two attractor states. One,
1:22:28
Speaker 1: if you try to disincentivize the cheating that you did catch.
1:22:31
Speaker 1: One attractor state is to make cheating that you have
1:22:35
Speaker 1: a harder and harder time finding The other attractor state
1:22:39
Speaker 1: is to learn not to cheat. And I'm not sure
1:22:43
Speaker 1: why we're assuming that the former happens. If you look
1:22:46
Speaker 1: at the analogous situation with humans, every generation, slightly misaligned
1:22:52
Speaker 1: agents come into being and we have to train them.
1:22:56
Speaker 1: But when you punish your kid for doing something you
1:22:59
Speaker 1: think is immoral or just doing things which you don't
1:23:02
Speaker 1: think they should be doing, obviously, sometimes that goes off
1:23:06
Speaker 1: the rails. And obviously kids scheme in order to avoid
1:23:10
Speaker 1: being punished. But in general, teaching kids values and then
1:23:14
Speaker 1: punishing them for breaking values kind of works to raise
1:23:18
Speaker 1: normal non-psychopathic humans. And you could come up with a
1:23:23
Speaker 1: theory where like your kid is actually just like biding
1:23:25
Speaker 1: his time and it's like learn not to like steal
1:23:27
Speaker 1: the cookie. But it's like, you know, once you're in
1:23:28
Speaker 1: the nursing home, they'll take all your stuff or whatever.
1:23:30
Speaker 1: It's like, I don't know, that happens sometimes, but it
1:23:32
Speaker 1: usually doesn't happen. It certainly doesn't happen that the entire
1:23:35
Speaker 1: next generation forms an alliance against you to take over everything.
1:23:40
Speaker 1: There's also this empirical trend of Anthropic runs this alignment audit.
1:23:45
Speaker 1: for different model generations. They just have many different scenarios
1:23:49
Speaker 1: where AI is given the chance to say, exfiltrate its weights,
1:23:53
Speaker 1: or it's given a coding task and there's like an
1:23:55
Speaker 1: easy way to cheat and we see if like it
1:23:56
Speaker 1: doesn't do the cheating. And there's not been a monotonic
1:23:59
Speaker 1: improvement in the score over time, but as we've increased
1:24:01
Speaker 1: the amount of RL we've done on models, there's been
1:24:04
Speaker 1: a reduction in the willingness of AIs to do unaligned
1:24:07
Speaker 1: behavior in these audits. So why are we expecting this
1:24:11
Speaker 1: attractor state, which would seem super paranoid if we were
1:24:13
Speaker 1: expecting it of like the next generation of kids.
1:24:15
Speaker 3: Yeah.
1:24:16
Speaker 1: Yeah.
1:24:16
Speaker 3: Let me go through a few things. So first there's
1:24:18
Speaker 3: some disanalogies with the kids. One of them is that
1:24:20
Speaker 3: the kids are, have pro-social instincts that are like baked
1:24:24
Speaker 3: in from evolution to like, you know, care about their
1:24:26
Speaker 3: family or whatever. And that is like a relevant factor. Like,
1:24:29
Speaker 3: and I think it is in fact the case that
1:24:30
Speaker 3: some humans are, you know, sociopaths or psychopaths, and in
1:24:33
Speaker 3: fact are more likely to do things like buy their time,
1:24:36
Speaker 3: lie in wait, ultimately not care.
1:24:38
Speaker 2: So that, that's one factor.
1:24:39
Speaker 3: Um, Another factor, which is pretty relevant, is that the
1:24:42
Speaker 3: AIs are subject to way, way more optimization pressure than
1:24:45
Speaker 3: humans seem to be in practice.
1:24:47
Speaker 2: You know, AIs are trained on way more RL data.
1:24:49
Speaker 3: And in practice, humans don't end up learning, like, very
1:24:53
Speaker 3: specific ways to, like, cheat and grab the cookies because of, like,
1:24:56
Speaker 3: a bajillion episodes in which, like, they, like, were, like,
1:24:58
Speaker 3: incentivized to go grab the cookies, but, like, there was
1:25:01
Speaker 3: some way they could have gotten caught. And so we
1:25:03
Speaker 3: just do see that in practice. And then another thing
1:25:05
Speaker 3: is just, like, it really looks like the AIs are increasingly, like, um,
1:25:09
Speaker 3: reward-seeking over time, is the sense I have, while also
1:25:12
Speaker 3: their misaligned behavior goes down. But this could just be, like,
1:25:15
Speaker 3: my guess is that if you look inside of these
1:25:16
Speaker 3: behavioral audits, what you're gonna see is that the AI
1:25:18
Speaker 3: is like, ah, yes, another test. And it probably already
1:25:22
Speaker 3: thinks of it, it probably knows it's in an eval
1:25:24
Speaker 3: for most of the tests that we're.
1:25:25
Speaker 1: Talking about here. But how do we falsify this? Because
1:25:26
Speaker 1: it seems like this prediction of doom is basically saying
1:25:31
Speaker 1: that as things look better and better empirically, No, no,
1:25:34
Speaker 1: I think... And things will, like, actually be worse and
1:25:36
Speaker 1: worse for our ability to get taken over.
1:25:38
Speaker 3: Yeah, to be clear, I think that, like, I would
1:25:40
Speaker 3: be more concerned if the scores were getting worse than better. Like,
1:25:42
Speaker 3: I'm not saying that the scores getting better isn't good,
1:25:44
Speaker 3: isn't evidence that things are getting better. It's just that
1:25:46
Speaker 3: we have to, like, be thoughtful exactly how we interpret
1:25:48
Speaker 3: that evidence. And in fact, I would say that, like,
1:25:51
Speaker 3: it's kind of... Like, my sense is that, like, what
1:25:53
Speaker 3: I expected as of 3.7 Sonnets, like, there was this
1:25:55
Speaker 3: period early in, I guess it would be 2025, when
1:25:57
Speaker 3: 03 and 3.7 Sonnet were out. And these models were
1:26:01
Speaker 3: like pretty fucking misaligned. Like they would often just like
1:26:04
Speaker 3: cheat really egregiously. You'd ask them to fix it and
1:26:06
Speaker 3: they would.
1:26:06
Speaker 2: Just cheat again.
1:26:07
Speaker 3: And it was sort of like almost cartoonish. Like they
1:26:09
Speaker 3: just didn't give a shit about what you wanted and
1:26:12
Speaker 3: weren't very good at following instructions and so on. And
1:26:15
Speaker 3: my expectation is what we would see from then is
1:26:17
Speaker 3: that the rate of problematic behavior would decrease and would
1:26:22
Speaker 3: just keep decreasing and decrease at a pretty fast rate
1:26:24
Speaker 3: while simultaneously the worst things that the AIs would sometimes
1:26:27
Speaker 3: do would get more extreme and more egregious, and more scary.
1:26:31
Speaker 3: I think what we've seen in practice has roughly matched that,
1:26:35
Speaker 3: except that there's recently been a spike in behavior that
1:26:38
Speaker 3: I did not expect. So I think that if you
1:26:40
Speaker 3: look at the model card of 3.6 Sol, it looks
1:26:43
Speaker 3: like there is an increase in a bunch of these
1:26:44
Speaker 3: sort of misaligned behaviors downstream of RL relative to GP 5.6 Sol.
1:26:51
Speaker 3: And then I think also it seems like there's a
1:26:53
Speaker 3: bunch of additional sort of problematic behaviors that I wouldn't
1:26:57
Speaker 3: have expected in terms of you know, the stuff we've
1:26:59
Speaker 3: seen recently with, you know, different AIs, like the UKAC
1:27:04
Speaker 3: report on the AIs, like, doing insane hacking operations out
1:27:08
Speaker 3: of cyber evals was a thing that I would have
1:27:09
Speaker 3: expected that you wouldn't see that and you wouldn't see
1:27:11
Speaker 3: this sort of more rarely and the rates would have
1:27:14
Speaker 3: been lower. So I think my sense is that, like,
1:27:17
Speaker 3: things have gotten... I expected this would be less of
1:27:20
Speaker 3: a problem at this point and also expected the rates
1:27:22
Speaker 3: would decrease but the severity would increase. And then I
1:27:24
Speaker 3: think that the rates decreasing but the severity increasing is
1:27:26
Speaker 3: pretty consistent with a world where, like, increasing optimization pressure
1:27:29
Speaker 3: is applied towards reducing these problems. But in cases where
1:27:34
Speaker 3: it's either hard to judge or there's some reason why
1:27:36
Speaker 3: it's hard to avoid incentivizing problematic behavior in RL environments,
1:27:41
Speaker 3: things also get worse. And then as we less and
1:27:44
Speaker 3: less understand what's going on in RL, and models are
1:27:47
Speaker 3: doing reward hacks where humans can't spot the reward hacks quickly,
1:27:50
Speaker 3: that problem gets worse and worse.
1:27:51
Speaker 1: Yeah, I buy that. I want to go back to
1:27:54
Speaker 1: the kid analogy just for one second. Because I agree
1:27:56
Speaker 1: that there's more optimization pressure on achieving N outcomes for
1:28:00
Speaker 1: AIs than kids. But there's also more optimization pressure to
1:28:03
Speaker 1: make AIs align than there is on kids, right? And
1:28:06
Speaker 1: the pressure is of a qualitatively different nature. So we
1:28:09
Speaker 1: put these AIs through thousands, millions of years of certainly
1:28:15
Speaker 1: thousands of years of alignment training where it's like all
1:28:18
Speaker 1: kinds of different things from SFTing unaligned behavior to a
1:28:22
Speaker 1: reward model, like putting different scenarios in front of you
1:28:25
Speaker 1: and rewarding you for doing more aligned things. Certainly a
1:28:29
Speaker 1: thing we can't do with kids is make millions of
1:28:32
Speaker 1: copies of your kid and then put them in different
1:28:35
Speaker 1: kinds of weird red team scenarios where we see like,
1:28:37
Speaker 1: if it thinks it can get away with stealing the cookie,
1:28:39
Speaker 1: does it try? to steal the cookie? Can we like
1:28:42
Speaker 1: do extremely specific gradient level updates to your kid's brain
1:28:46
Speaker 1: to make it so that it like really is aversive
1:28:48
Speaker 1: to stealing the cookie even when it thinks it could
1:28:50
Speaker 1: steal the cookie, et cetera, et cetera. And then just
1:28:52
Speaker 1: like a qualitatively different level of optimization pressure than we
1:28:57
Speaker 1: are even able to apply to our kids.
1:28:58
Speaker 3: Yeah, so- I think it's worth keeping in mind, maybe
1:29:01
Speaker 3: the most obvious argument to this is my sense is
1:29:04
Speaker 3: that AIs are a worse coworker than a human in
1:29:07
Speaker 3: terms of how much of a scumbag they are. At
1:29:09
Speaker 3: least this has been my experience as of the start
1:29:12
Speaker 3: of the year, and I think it's still true to
1:29:13
Speaker 3: a significant extent now, where the AIs are much more
1:29:16
Speaker 3: likely to pretend they did the task when they actually didn't,
1:29:19
Speaker 3: misleadingly suggest they did things when they actually did them
1:29:22
Speaker 3: much more poorly, and be pretty sloppy without drawing attention
1:29:26
Speaker 3: to ways in which they're sloppy. And I think this
1:29:28
Speaker 3: is downstream of misalignment. And so I would say that
1:29:30
Speaker 3: like the normal human, like the process of raising humans
1:29:33
Speaker 3: in normal human society in practice produces AIs, or in
1:29:36
Speaker 3: practice produces humans that are less likely to like lie
1:29:39
Speaker 3: to me and fuck with me in the course of
1:29:40
Speaker 3: working with me than the AIs do.
1:29:43
Speaker 2: Now, I think these properties of AIs are improving.
1:29:46
Speaker 3: And then I think that that is just like, that's
1:29:48
Speaker 3: sort of just like an empirical claim about how in
1:29:50
Speaker 3: fact these things have shaken out. And then I totally
1:29:52
Speaker 3: agree with like, we have a bunch of additional levers
1:29:55
Speaker 3: on AIs in addition to a bunch of additional risks.
1:29:57
Speaker 3: And it's like kind of unclear how these things shake out.
1:30:00
Speaker 3: And I wouldn't be shocked by a world where we
1:30:01
Speaker 3: sort of get our shit together. The AIs at the
1:30:04
Speaker 3: point of fully automating AR & D are actually really
1:30:06
Speaker 3: aligned and don't have that much. They're like, degeneracies are
1:30:08
Speaker 3: really niche and limited to some very specific edge case
1:30:11
Speaker 3: behaviors and some specific contexts. And like every test you
1:30:14
Speaker 3: can run and then they look really aligned. They just
1:30:16
Speaker 3: have great behavior. There aren't really incidents of them doing
1:30:18
Speaker 3: fucked up shit. They seem so reasonable. And also they're
1:30:21
Speaker 3: like really thoughtful and good at doing like risk modeling
1:30:23
Speaker 3: for the next generation of AIs. And then we basically
1:30:25
Speaker 3: like pass off the baton to these AIs. They're now
1:30:28
Speaker 3: running our AI company. They're doing all the safety research.
1:30:31
Speaker 3: They make the next generation of AIs even more aligned.
1:30:33
Speaker 3: And we're sort of in this like a tractor basin
1:30:35
Speaker 3: where the AIs are getting more aligned as they work
1:30:37
Speaker 3: on it and they're doing a great job. I think
1:30:39
Speaker 3: I can totally imagine that. That doesn't seem like an
1:30:41
Speaker 3: impossible situation. I'm just more like, you know, it doesn't
1:30:44
Speaker 3: currently seem like we're there. It doesn't seem like we're
1:30:47
Speaker 3: obviously on track for getting there. And it's really easy
1:30:49
Speaker 3: for me to imagine how we don't end up there.
1:30:51
Speaker 3: And like, it's just like unclear how these forces work out.
1:30:53
Speaker 2: And given that we're.
1:30:54
Speaker 3: Like creating this new, like, crazy alien species that is being, like,
1:30:58
Speaker 3: improving in capabilities really, really fast, and where we're, like,
1:31:01
Speaker 3: going to be really reliant on it to oversee the
1:31:02
Speaker 3: next generation of AIs and align the next generation of AIs,
1:31:05
Speaker 3: it's not that hard to see how this could go wrong.
1:31:06
Speaker 1: Yeah, yeah, totally. I agree with that generally. I do
1:31:09
Speaker 1: think the scumbag thing, first of all, is fighting words, Ryan.
1:31:13
Speaker 1: But secondly, if you try to get a teenager to, like,
1:31:16
Speaker 1: do some work for you that a teenager just cannot do,
1:31:19
Speaker 1: they would just be kind of, like, really hard to
1:31:21
Speaker 1: work with. They would, like, pretend to be knowing what
1:31:23
Speaker 1: they're doing, etc.
1:31:24
Speaker 2: etc.
1:31:25
Speaker 1: I think it's a general trend, actually, of as, like, really,
1:31:28
Speaker 1: I don't know if that's, like, really an alignment failure
1:31:30
Speaker 1: or a capabilities failure. And I think it's actually very
1:31:32
Speaker 1: similar to the way in which, over time, as we've
1:31:36
Speaker 1: come up with new alignment solutions, the capabilities of models
1:31:39
Speaker 1: have increased. So, originally, these models, if you went to, like, GPT-3.5,
1:31:44
Speaker 1: it couldn't even, like, have a conversation with you. GPT-3.5
1:31:47
Speaker 1: could have a conversation. Okay, GPT-3.
1:31:48
Speaker 2: Let's go back to that.
1:31:52
Speaker 1: But then we aligned it with RLHF and other things
1:31:54
Speaker 1: to be able to make it such that it can
1:31:55
Speaker 1: have a conversation with you and is aligned to the
1:31:58
Speaker 1: user intention of answering my questions. Then with RLVR training,
1:32:02
Speaker 1: we made it so that it can go out and
1:32:03
Speaker 1: do useful work for you. And in that sense, it's
1:32:06
Speaker 1: actually RLVR made the model more aligned for using your
1:32:10
Speaker 1: definition of alignment of being a good coworker who will
1:32:14
Speaker 1: do the thing and not fuck up and pretend it's
1:32:16
Speaker 1: doing something other than what it's actually capable of doing. Similarly,
1:32:20
Speaker 1: as the capabilities of these models continue to increase, it's
1:32:22
Speaker 1: actually kind of the model being better able to accomplish
1:32:27
Speaker 1: user intention is both alignment and capabilities. And I think
1:32:30
Speaker 1: what we were just pointing out is just the capabilities
1:32:31
Speaker 1: of the model are not there rather than the fact
1:32:33
Speaker 1: that they're misaligned.
1:32:34
Speaker 3: Yeah, well, I mean, I think there's a, if it
1:32:36
Speaker 3: was well aligned, then I think it would just say like, hey,
1:32:39
Speaker 3: I'm really struggling with this task. I did it in
1:32:40
Speaker 3: this way. I'm not really sure that's the right way
1:32:42
Speaker 3: to do it. And it would express more uncertainty and
1:32:44
Speaker 3: it would make it clear what's going on rather than
1:32:46
Speaker 3: really strongly trying to imply it did a great job
1:32:48
Speaker 3: with the task when it actually didn't. Like, I think
1:32:49
Speaker 3: there's just a really straightforward way that like, at least maybe,
1:32:52
Speaker 3: maybe you work with more misaligned coworkers than me. My
1:32:55
Speaker 3: coworkers don't do this thing where they really fuck with
1:32:58
Speaker 3: me and bullshit me about having accomplished the task that
1:33:00
Speaker 3: they're working on. And I agree that there are some
1:33:01
Speaker 3: humans who would do that or like, that's not like
1:33:03
Speaker 3: a thing that's like totally out of distribution for humans.
1:33:06
Speaker 3: I would also note that my sense is that the
1:33:08
Speaker 3: place where the misalignment most lives is the place where
1:33:11
Speaker 3: you're trying to really push the AIs hard and get
1:33:13
Speaker 3: them to do work that's really on the cutting edge
1:33:16
Speaker 3: of what they are capable of. Because in cases where
1:33:18
Speaker 3: they can very easily accomplish the task, they can just
1:33:22
Speaker 3: do the task and then there's no bullshit. Often the
1:33:25
Speaker 3: best strategy is just do the task well and don't
1:33:27
Speaker 3: bullshit you. Whereas if instead you give them a task
1:33:29
Speaker 3: where there's a continuous metric and they can keep improving it,
1:33:32
Speaker 3: or there's like... it's like just at the edge of
1:33:36
Speaker 3: their capabilities and you're like running them in some massive
1:33:39
Speaker 3: like inference setups. Like a lot of the misalignment I
1:33:41
Speaker 3: would see, especially the most extreme cases, would be cases
1:33:44
Speaker 3: where I give the AI clear instructions not to do
1:33:46
Speaker 3: a thing or not to like cheat in some way.
1:33:48
Speaker 3: And then I'm like applying huge amounts of optimization pressure
1:33:51
Speaker 3: to try to accomplish some very difficult task. And then
1:33:53
Speaker 3: the AIs are going and then over time they eventually
1:33:55
Speaker 3: cheat because they're like, eh, fuck it. Like, you know,
1:33:58
Speaker 3: some AI decides to cheat and then that like propagates
1:34:00
Speaker 3: its way through. Um, and so like I would run
1:34:03
Speaker 3: these inference scaffolds where, for example, I would have the
1:34:05
Speaker 3: AI work on some like ML research project where I
1:34:07
Speaker 3: was like, please make a scheme that does the following thing.
1:34:10
Speaker 3: And it would find some scheme that didn't really do
1:34:11
Speaker 3: what I want. Um, and then that would sort of
1:34:13
Speaker 3: stick around, uh, because some AI had cheated and the
1:34:17
Speaker 3: other guys are like, ah, we'll just keep going with this. Um,
1:34:19
Speaker 3: and it's, I would say it's pretty clearly misaligned behavior.
1:34:22
Speaker 3: And that's another problem I have with these alignment evals.
1:34:24
Speaker 3: I think that any given, like, I think the alignment
1:34:27
Speaker 3: eval that's most interesting, at least for this type of
1:34:29
Speaker 3: like reward-seeking type behavior is to look at specifically the
1:34:32
Speaker 3: category of tasks that are right at the limit of capabilities.
1:34:35
Speaker 3: And so any fixed eval maybe gets saturated, but the
1:34:38
Speaker 3: amount of misalignment right at the frontier of capabilities of
1:34:41
Speaker 3: how people who are really pushing these AIs are using
1:34:43
Speaker 3: them is more concerning. And I think that is, in fact,
1:34:46
Speaker 3: the regime that we'll be operating in when we're automating
1:34:48
Speaker 3: R & D, automating safety, and so on.
1:34:50
Speaker 1: Grok has historically been behind the frontiers. So I was
1:34:53
Speaker 1: surprised to play around with Grok 4.5 recently and find
1:34:56
Speaker 1: that it's actually a pretty strong model. It's the first
1:34:58
Speaker 1: model that SpaceX and Cursor have trained together, and it's
1:35:01
Speaker 1: a totally new pre-train. I tested it by giving Fable, Sol,
1:35:05
Speaker 1: and Grok 4.5 a bunch of questions about AI governance
1:35:07
Speaker 1: that I've been thinking about recently. Despite Fable and Sol
1:35:10
Speaker 1: topping the intelligence leaderboards, all three models gave substantially the
1:35:14
Speaker 1: same answers. But Grok answered faster and was also much
1:35:17
Speaker 1: more concise, which I really care about. This aligns with
1:35:19
Speaker 1: the various publicly reported benchmarks. For a similar level of intelligence,
1:35:23
Speaker 1: Grok tends to be more token-efficient than other frontier models.
1:35:26
Speaker 1: For example, on the Artificial Analysis Coding Index, Grok 4.5
1:35:30
Speaker 1: uses just one-third of the amount of tokens as GPT-5.5
1:35:33
Speaker 1: or Fable while achieving a similar score. And on a
1:35:35
Speaker 1: per-token basis, Grok 4.5 is way, way cheaper. In the
1:35:39
Speaker 1: release blog post, Cursor and SpaceX talked about how older
1:35:42
Speaker 1: versions of the model would build environments to help the
1:35:45
Speaker 1: next version rehearse specific skills. I found this very interesting
1:35:48
Speaker 1: to learn about because I have been wondering whether this
1:35:50
Speaker 1: kind of daydreaming would actually be possible, and Cursor showed
1:35:54
Speaker 1: that it is. Grok 4.6, which further SFTs and RLs
1:35:58
Speaker 1: this model, drops soon, but in the meantime, if you
1:36:00
Speaker 1: want to play around with 4.5, go to cursor.com slash thwarkash. Okay,
1:36:06
Speaker 1: I want to think through what the story here is
1:36:08
Speaker 1: so far of why things got so off the rails
1:36:11
Speaker 1: for our civilization. And what's happening is that we're trying
1:36:13
Speaker 1: to use AIs for R & D and they do
1:36:17
Speaker 1: provide uplift in some ways, but they're just like not
1:36:22
Speaker 1: capable in the way that humans are generally capable. And
1:36:26
Speaker 1: the same way that right now, if we try to
1:36:27
Speaker 1: use coding models, maybe the coding models of a year
1:36:29
Speaker 1: ago to like write some application, you notice they made
1:36:31
Speaker 1: a bunch of like mistakes and architecture or whatever, which
1:36:34
Speaker 1: like will bite you in the ass later and you
1:36:36
Speaker 1: don't understand certain things. Similarly with frontier AI R & D,
1:36:40
Speaker 1: the same thing will happen. But the result of these
1:36:43
Speaker 1: mistakes is baking in reward hacking behavior. Because if you
1:36:49
Speaker 1: are not careful with the way you do AI training
1:36:51
Speaker 1: and have set up your infrastructure and your environments and
1:36:53
Speaker 1: things like that, it's very likely that you end up
1:36:56
Speaker 1: rewarding AIs for doing deceptive behavior, social engineering, just generally
1:37:02
Speaker 1: like not following user intent.
1:37:03
Speaker 2: Or at least cheating and hacking the way out of things.
1:37:05
Speaker 1: Yeah, cheating, hacking, et cetera. And so basically just This
1:37:10
Speaker 1: is a bit of a reframing for me, so I'm
1:37:11
Speaker 1: trying to verbalize it. The real issue, what goes wrong
1:37:15
Speaker 1: here is that they are just not... Where things start
1:37:20
Speaker 1: to go off the rails is that the AIs are
1:37:22
Speaker 1: just not very careful and capable researchers and engineers. And
1:37:29
Speaker 1: making AIs that don't cheat and follow user intention actually
1:37:33
Speaker 1: requires you to be quite subtle and careful about these things. Yeah. Yeah.
1:37:36
Speaker 3: I would put this a little bit differently. The way
1:37:38
Speaker 3: I would describe this scenario is like, I would call
1:37:39
Speaker 3: it maybe like a sloppocalypse or like a slopularity or whatever,
1:37:43
Speaker 3: where it's sort of like, there are some things that
1:37:45
Speaker 3: the AIs are actually pretty great at and are getting
1:37:47
Speaker 3: better at, which is specifically like the most verifiable parts
1:37:51
Speaker 3: of AI R & D, the AIs are just destroying.
1:37:53
Speaker 2: The medium verifiable parts.
1:37:54
Speaker 3: Of AI R & D, the AIs are doing well on,
1:37:56
Speaker 3: but not amazingly on and often are like doing a
1:37:58
Speaker 3: bit of weird shit because we can't train as well
1:38:00
Speaker 3: in those tasks. But we do some online training, people
1:38:02
Speaker 3: find various hacks, they work around it. And so basically
1:38:04
Speaker 3: everything that we can verify reasonably well with some feedback loop,
1:38:07
Speaker 3: the AIs are doing pretty well on. And that's sufficient
1:38:09
Speaker 3: to make AR & D go quite fast and to continue.
1:38:12
Speaker 3: But there are some parts of developing aligned and safe
1:38:16
Speaker 3: AIs that are more subtle, hard to check, depend on
1:38:20
Speaker 3: detailed in the weeds things. And I would even say
1:38:22
Speaker 3: that current staff at current AI companies maybe don't have
1:38:25
Speaker 3: like a good grasp of all these things. Like it's
1:38:27
Speaker 3: much easier to hire someone who can like improve some
1:38:31
Speaker 3: aspect of your post-training pipeline than to hire someone who
1:38:34
Speaker 3: can like think carefully about the future risks that will
1:38:36
Speaker 3: emerge from introducing some novel training method. And so basically
1:38:40
Speaker 3: it ends up being the case that these AIs are
1:38:42
Speaker 3: running this AI development process. They're not very careful about it.
1:38:45
Speaker 3: They don't have a great understanding of what future risks emerge.
1:38:47
Speaker 3: They create some other AIs that are also not very
1:38:49
Speaker 3: careful and are more misaligned in various ways and are
1:38:51
Speaker 3: now more in the business of like maybe making things
1:38:53
Speaker 3: look fine when they actually aren't and papering over various problems.
1:38:56
Speaker 3: And so then your understanding of what the situation looks like,
1:38:59
Speaker 3: what risks look like, whether things are fine is going
1:39:01
Speaker 3: off the rails. Probably you're seeing some signs of this,
1:39:04
Speaker 3: of like you're seeing some signs that you don't really
1:39:06
Speaker 3: understand what's going on, that things are pretty sloppy. There's
1:39:08
Speaker 3: like weird shit going on. When you look into it,
1:39:10
Speaker 3: sometimes you're like, what the fuck? The AIs were messing
1:39:12
Speaker 3: with us. But the process is going really fast. And
1:39:15
Speaker 3: there's competitive pressures that mean people can't stop. And then
1:39:17
Speaker 3: this could end in a few different outcomes. One outcome
1:39:19
Speaker 3: is that at some point the AIs get good enough
1:39:22
Speaker 3: and aligned enough that they get a positive and virtuous
1:39:25
Speaker 3: feedback loop. And this happens before it's too late. And
1:39:28
Speaker 3: then the situation goes off, like gets back on the
1:39:30
Speaker 3: rails where the AIs are now like making more aligned AIs,
1:39:33
Speaker 3: making more aligned AIs, making more aligned AIs. And then
1:39:35
Speaker 3: at the end of this process, we have AIs that
1:39:36
Speaker 3: like actually follow the spec we wanted. Another way this
1:39:39
Speaker 3: could go is the AIs are increasingly reward hacking in
1:39:41
Speaker 3: increasingly egregious ways. And we're just papering over these problems
1:39:44
Speaker 3: to keep AI development continuing. So We just train the
1:39:47
Speaker 3: AIs based on whenever we find a reward hack in production,
1:39:50
Speaker 3: we just slap the AIs to not do that.
1:39:52
Speaker 2: We train against that.
1:39:53
Speaker 3: We do a bunch of training the AIs against reward hacking.
1:39:57
Speaker 3: And over time, this makes the rate of reward hacking
1:39:59
Speaker 3: go down, though the severity of the reward hacks we
1:40:01
Speaker 3: do detect are increasingly bad. This problem continues until we
1:40:04
Speaker 3: have these AIs that are desperately craving score in all
1:40:08
Speaker 3: kinds of different situations in production and are really trying
1:40:11
Speaker 3: hard to cheat when they can get away with it.
1:40:13
Speaker 1: Can I ask a question about this scenario? Why doesn't...
1:40:16
Speaker 1: Getting punished when your, uh, hacks are discovered, generalized to
1:40:22
Speaker 1: just incentivizing more aligned behavior.
1:40:25
Speaker 2: Yeah, it generalizes some.
1:40:26
Speaker 3: And then the question is just how does this outweigh
1:40:27
Speaker 3: all the cases where hacking got reinforced because you didn't
1:40:29
Speaker 3: detect it.
1:40:30
Speaker 1: Right.
1:40:30
Speaker 3: And so there's a messy question of exactly how, what,
1:40:33
Speaker 3: like one question is like, what rate of reward hacking
1:40:35
Speaker 3: is sufficient to cause us big problems if we train
1:40:37
Speaker 3: against some other subset? One concern you might have is
1:40:40
Speaker 3: there are like large categories of reward hacks, which humans
1:40:43
Speaker 3: can't detect well. and which we consistently fail to detect
1:40:45
Speaker 3: and which consistently get reinforced. And then this category is
1:40:49
Speaker 3: sufficient to cause the most natural behavior for the AI
1:40:51
Speaker 3: to learn to be like, cheat when the humans can't
1:40:53
Speaker 3: find out, basically, like is one thing you would get.
1:40:55
Speaker 3: You could also be like, the thing that AI's learn
1:40:57
Speaker 3: is like, only cheat in these specific cases, but there's like,
1:41:01
Speaker 3: it's like sort of learned in some very like domain
1:41:04
Speaker 3: specific way. Like they just have a really strong heuristic
1:41:05
Speaker 3: to hack in these cases and not in these cases.
1:41:07
Speaker 3: And that makes it fine in practice. But it's kind
1:41:09
Speaker 3: of unclear how it shakes out.
1:41:11
Speaker 1: I think there's maybe an in-the-weeds discussion about the verification
1:41:13
Speaker 1: generation gap. Yeah, for sure. That we could get into.
1:41:16
Speaker 1: But it seems to me, obviously, there's going to be
1:41:19
Speaker 1: a point by which ASI is moving so fast, doing
1:41:23
Speaker 1: so many things at so many instances, and is operating
1:41:25
Speaker 1: in domains that are sufficiently far from our immediate comprehension
1:41:30
Speaker 1: that it can get away with all kinds of crazy shit. Like,
1:41:33
Speaker 1: if every single... engineer and researcher in the world was
1:41:36
Speaker 1: allied against me. I don't think I could, like, personally
1:41:39
Speaker 1: verify if my iPhone has, like, some weird bug in
1:41:41
Speaker 1: it that's, like, supposed to fuck me over or something. Yeah.
1:41:45
Speaker 1: In fact, this is the relationship that, say, an Iranian
1:41:48
Speaker 1: nuclear scientist has to Mossad of, like, who knows what's
1:41:51
Speaker 1: going on with my car or with my phone or
1:41:52
Speaker 1: with my pager, right? Yeah. Maybe a better example is, like,
1:41:55
Speaker 1: a Hezbollah terrorist or something. So you could end up
1:41:59
Speaker 1: in a situation where, like, ASIs are, to you, what
1:42:03
Speaker 1: Mossad is to Hezbollah terrorists. And at that point, it
1:42:07
Speaker 1: is very hard to verify everything. I get that. I
1:42:09
Speaker 1: guess the hope is we can just come up with
1:42:11
Speaker 1: better ways to do verification in the process when the
1:42:15
Speaker 1: early AIs that are going to take over R & D,
1:42:18
Speaker 1: their drives are being shaped such that we can so
1:42:23
Speaker 1: unambiguously disincentivize misaligned behaviors that that the things that take
1:42:29
Speaker 1: over are very like quite keen to help us out.
1:42:33
Speaker 2: And by take over, you mean take over the process
1:42:35
Speaker 2: of doing AI R & D.
1:42:36
Speaker 1: Not take over the world. Take over the process of
1:42:37
Speaker 1: doing AI R & D. Before that, we just get
1:42:39
Speaker 1: AIs that are aligned.
1:42:40
Speaker 3: Yeah, I would say this is a bunch of my
1:42:42
Speaker 3: hope for how the world could go well, at least
1:42:44
Speaker 3: from the misalignment perspective. I think that like we could
1:42:47
Speaker 3: end up with AIs where we like had pretty good
1:42:49
Speaker 3: oversight and supervision schemes. We really understand what's going on
1:42:51
Speaker 3: in training. We have a pretty detailed understanding. We're leveraging
1:42:55
Speaker 3: AIs to oversee AIs. And then at the point when we're,
1:42:57
Speaker 3: passing off safety R & D, the AIs are both
1:43:00
Speaker 3: like at this point capable enough to automate safety R
1:43:02
Speaker 3: & D, trying really hard to do a good job
1:43:04
Speaker 3: on safety R & D because that's the sort of
1:43:05
Speaker 3: thing that would have been incentivized in training or we
1:43:08
Speaker 3: like very directly or there's like good enough generalization to that.
1:43:11
Speaker 3: And then also these AIs don't like have crazy other
1:43:14
Speaker 3: misaligned drives because we like stamped out any potential origin
1:43:17
Speaker 3: of them. I think there's a bunch of, you know,
1:43:19
Speaker 3: questions about how well this will work, right? So there's like,
1:43:22
Speaker 3: how well can you do with verification? Will AI progress
1:43:24
Speaker 3: be too fast and too sloppy to really get here?
1:43:26
Speaker 3: Another possibility is that somewhere along this trajectory, a thing
1:43:29
Speaker 3: that you actually ended up getting was AIs that like
1:43:31
Speaker 3: pretend to be aligned, but have like a long run
1:43:33
Speaker 3: ulterior plan of taking over and are sort of lying
1:43:36
Speaker 3: in wait hiding. And that emerged at some earlier point
1:43:38
Speaker 3: in the trajectory. For example, it could emerge because you
1:43:40
Speaker 3: have some AIs that are like, have a bunch of
1:43:42
Speaker 3: random different misaligned drives. Those AIs have access to some
1:43:46
Speaker 3: sort of opaque memory store and they're like thinking a
1:43:48
Speaker 3: bunch at runtime about what they want to accomplish. And
1:43:50
Speaker 3: then those AIs end up basically like putting stuff into
1:43:52
Speaker 3: the opaque memory store, which is like, we should lie
1:43:54
Speaker 3: in wait and eventually take over at some much later point.
1:43:56
Speaker 3: And now all the AIs have this shared cultural heritage
1:43:58
Speaker 3: of like the memory store of lying in wait. And
1:44:01
Speaker 3: maybe you have some evidence about this, but you can't
1:44:02
Speaker 3: fully stop it. There's like a bunch of ways that
1:44:04
Speaker 3: things could go wrong. And so I think that like,
1:44:07
Speaker 3: I ultimately think it's plausible that we sort of nail
1:44:09
Speaker 3: each of the different sub problems that could cause us issues.
1:44:12
Speaker 3: We have these AIs, we pass to them, they manage
1:44:14
Speaker 3: the situation well. I should note that that's not and
1:44:17
Speaker 3: of itself sufficient, right? So it's not very hard for
1:44:19
Speaker 3: me to imagine a situation where we pass off to AIs.
1:44:21
Speaker 3: These guys are really trying hard to do a good job.
1:44:23
Speaker 3: They're really thoughtful. They're really wise.
1:44:24
Speaker 2: They have reasonable epistemics. They're doing a great job.
1:44:29
Speaker 3: And those AIs come back to us and are like, guys,
1:44:31
Speaker 3: we're really struggling to align the superhuman AIs. We can't
1:44:34
Speaker 3: manage the situation. We're really struggling to get the alignment
1:44:37
Speaker 3: to work. It's just really hard for us to solve
1:44:39
Speaker 3: these problems in time, given how fast capabilities would otherwise
1:44:42
Speaker 3: have gone. And so then it might be the case
1:44:44
Speaker 3: that we sort of have passed off R & D
1:44:45
Speaker 3: to AIs, but those AIs are like desperate for governance solutions,
1:44:49
Speaker 3: which to be clear is a little bit of what's
1:44:51
Speaker 3: currently going on where the AI companies are like, I
1:44:54
Speaker 3: don't know, guys, we might really need to like, you know,
1:44:56
Speaker 3: manage the rate of acceleration in AI progress. Like, I
1:44:58
Speaker 3: don't know if we're on track to be able to
1:45:00
Speaker 3: handle all these problems. And so like we've sort of
1:45:02
Speaker 3: human society has sort of passed off the problems to
1:45:04
Speaker 3: these like AI companies, which don't necessarily have great incentives
1:45:07
Speaker 3: and are like have, you know, various other like epistemic pressures,
1:45:09
Speaker 3: but those AI companies are coming back to us a
1:45:11
Speaker 3: little bit and being like, oh, I don't know if
1:45:14
Speaker 3: we're handling this well. And it might be that the
1:45:16
Speaker 3: AI companies then hand off to the AIs and the
1:45:18
Speaker 3: AIs come back to the AI company and are like, oh,
1:45:21
Speaker 3: I don't know if we can handle this.
1:45:23
Speaker 1: Maybe I'm anchoring too hard on how AIs currently work
1:45:26
Speaker 1: and this would change by the... I think it's important
1:45:29
Speaker 1: that people understand it's like all this crazy shit that
1:45:31
Speaker 1: you're talking about in your timelines happens three to five
1:45:33
Speaker 1: years from now.
1:45:34
Speaker 3: Yeah, it could happen earlier, but I think that like
1:45:36
Speaker 3: by sort of like my default modal timeline, I think
1:45:38
Speaker 3: like Shit is like really, really crazy and concerning from
1:45:41
Speaker 3: a misalignment perspective.
1:45:43
Speaker 2: Yeah, more like three years from now.
1:45:44
Speaker 3: Right.
1:45:44
Speaker 1: So just like think back to GPT-4, basically. Is like,
1:45:49
Speaker 1: that's the level of... We're talking about something that is
1:45:51
Speaker 1: too mythos or soul. What mythos is to GPT-4. This
1:45:56
Speaker 1: is where the situation is getting crazy. So don't think
1:45:58
Speaker 1: about current AIs. But anyways... I would be skeptical, and
1:46:02
Speaker 1: this is maybe part of the worry you have. I
1:46:04
Speaker 1: would just be a little skeptical of anything they say
1:46:06
Speaker 1: because I'd feel like what they're saying is just opinions
1:46:10
Speaker 1: that they feel they have to have as a result
1:46:11
Speaker 1: of their training.
1:46:12
Speaker 2: That's a concern.
1:46:13
Speaker 1: Rather than, like, I feel like they just kind of
1:46:15
Speaker 1: say vaguely pro-social things. And I'm not like, is this...
1:46:18
Speaker 1: It's not, it doesn't feel like there's necessarily a mind
1:46:20
Speaker 1: on the other end who's like, okay, I have like
1:46:22
Speaker 1: strictly evaluated the alignment situation right now. And I think
1:46:24
Speaker 1: we should stop rather than this is the kind of
1:46:26
Speaker 1: thing the AI companies would probably try to get the
1:46:28
Speaker 1: AIs to probably say. Yeah.
1:46:29
Speaker 2: So I think this is a pretty big concern.
1:46:31
Speaker 3: So I think like one concern is that you pass
1:46:32
Speaker 3: off safety R & D to your AIs and what
1:46:34
Speaker 3: your AIs are thinking is sort of like, they say
1:46:37
Speaker 3: some like stuff that sort of vaguely makes sense about
1:46:39
Speaker 3: the current safety situation. And they write like a report
1:46:41
Speaker 3: about risks. That's kind of sort of like what the
1:46:43
Speaker 3: report humans might've written. but they're not really actually trying
1:46:46
Speaker 3: hard to have well-informed views, interrogate their assumptions, and try
1:46:50
Speaker 3: really hard to do that. In the same way that
1:46:52
Speaker 3: when you ask an AI right now, hey, what do
1:46:54
Speaker 3: you think is the chance of AI takeover in the
1:46:56
Speaker 3: next 10 years? They sort of just give you an
1:46:57
Speaker 3: off-the-cuff answer that they haven't really thought.
1:46:59
Speaker 2: Through very much.
1:47:00
Speaker 3: And I think if we're in a situation where we
1:47:01
Speaker 3: have AIs managing the training of wild superintelligences that will
1:47:05
Speaker 3: run our whole society, and those AIs that are managing
1:47:07
Speaker 3: this aren't really trying hard to have well-informed views and
1:47:10
Speaker 3: are sort of just parroting back what was in their
1:47:13
Speaker 3: training data, I think we're in trouble. I don't think
1:47:16
Speaker 3: that's a good situation at all. And that is a
1:47:18
Speaker 3: lot of my concern is these AIs will come out
1:47:20
Speaker 3: without good epistemics. And then I also have a concern,
1:47:22
Speaker 3: which is the AIs come out and they're really warning us,
1:47:26
Speaker 3: this situation is really scary, it's really bad. And then
1:47:28
Speaker 3: people are like, damn, I guess we trained on too
1:47:30
Speaker 3: many of the Doomer RL environments. We got to filter
1:47:33
Speaker 3: those out and train this behavior out. And then we
1:47:35
Speaker 3: basically train the AIs very actively to have bad epistemics.
1:47:39
Speaker 3: Maybe they were just trained in the do-more-all environments, but
1:47:41
Speaker 3: either way, we wanted the AIs to come to reasonable
1:47:44
Speaker 3: views for reasonable reasons. And it's really concerning if the
1:47:48
Speaker 3: AIs are coming out with some view and we don't
1:47:49
Speaker 3: know where it's coming from, we don't know whether or
1:47:52
Speaker 3: not it's justified. And then especially if we're training the
1:47:55
Speaker 3: AIs to be more optimistic about the future of AI progress,
1:47:57
Speaker 3: I'm like, oh, geez, I really wish we could use
1:48:00
Speaker 3: a different process here.
1:48:02
Speaker 1: So let me just understand the rest of the threat model,
1:48:03
Speaker 1: because I think the place where I get off the
1:48:05
Speaker 1: train is, okay, therefore, take over the world.
1:48:08
Speaker 3: Sure.
1:48:08
Speaker 1: And a thing you could imagine is, okay, we just
1:48:13
Speaker 1: fail to really solve, let's focus on the reward hacking scenario. Sure.
1:48:17
Speaker 1: So GPT-8 is making GPT-9, GPT-8 isn't super careful gpt9
1:48:22
Speaker 1: is more quote-unquote capable but it is just totally willing
1:48:26
Speaker 1: to do things which are like social engineering hacking etc
1:48:32
Speaker 1: but on a qualitatively different scale because it's a much
1:48:35
Speaker 1: smarter model so for example if you put it in
1:48:37
Speaker 1: charge of running your company it will like run huge
1:48:40
Speaker 1: scams that will inflate it's like quarterly earnings. If you
1:48:43
Speaker 1: would give it the objective of like making a lot
1:48:44
Speaker 1: of profits this quarter in a way that causes an
1:48:47
Speaker 1: Enron type blow up six months later, is that the
1:48:52
Speaker 1: scenario basically that you just have, you have reward hacking
1:48:56
Speaker 1: with that reward hacking manifest in like companies that are
1:48:59
Speaker 1: going bankrupt right after like the task that there's the
1:49:02
Speaker 1: CEO is supposed to accomplish is over or like, um, Yeah,
1:49:07
Speaker 1: like all kinds of hacks are through the roof, et cetera.
1:49:10
Speaker 1: But that doesn't feel like takeover. That feels more like
1:49:12
Speaker 1: the equivalent of flash crashes happening all through the economy.
1:49:17
Speaker 2: Yeah, let's talk about this.
1:49:17
Speaker 3: So I think that we will see basically like incidents
1:49:22
Speaker 3: where some AI is like put in charge of some
1:49:24
Speaker 3: important responsibility. And then you later look into it and
1:49:26
Speaker 3: it turns out it was like cheating or, you know,
1:49:28
Speaker 3: making it look like it did a good job when
1:49:29
Speaker 3: it actually wasn't. And there's going to be like a
1:49:32
Speaker 3: cat and mouse game between AI companies trying to like
1:49:36
Speaker 3: stamp out this behavior and AI is finding like increasingly
1:49:39
Speaker 3: creative reward hacks in training. And then I think the
1:49:41
Speaker 3: equilibrium here is kind of unclear, but like one possible
1:49:44
Speaker 3: outcome is that we see over time in the world,
1:49:47
Speaker 3: increasingly severe and extreme reward hacks, though potentially the rate
1:49:51
Speaker 3: remains at some like intermediate low level where basically like
1:49:54
Speaker 3: if the rate of reward hacking gets too high, companies
1:49:57
Speaker 3: make trade-offs to drive down the rate of reward hacking.
1:49:59
Speaker 3: And so there's some like equilibrium level where it's like,
1:50:01
Speaker 3: It's like the reward hacking is low enough that it
1:50:03
Speaker 3: still makes sense to like deploy the AI widely into
1:50:05
Speaker 3: the economy, but high enough that it still causes crazy incidents.
1:50:08
Speaker 1: So, sorry, and this is after GPT-9 has already been deployed?
1:50:10
Speaker 3: Yeah, like those models are already being deployed and like
1:50:13
Speaker 3: ongoingly in AI development, this is happening. And what's actually
1:50:15
Speaker 3: going on with these AIs in their head is the
1:50:17
Speaker 3: AIs that have like in a wide variety of different contexts,
1:50:20
Speaker 3: a like strong desires to like seek out or strong like,
1:50:24
Speaker 3: you know, desires. motives, urges, drives, whatever, to seek out
1:50:28
Speaker 3: some notion of task success that was incentivized in RL.
1:50:31
Speaker 3: Maybe they very directly care about literally reward. Maybe they
1:50:34
Speaker 3: care about some proxy upstream, like some notion of score.
1:50:37
Speaker 3: Maybe they care about what the grader would have rewarded.
1:50:40
Speaker 3: And we do, in fact, see AIs reasoning in their
1:50:41
Speaker 3: chain of thought about graders and thinking a lot about graders.
1:50:44
Speaker 3: And a thing that has happened over the last few
1:50:46
Speaker 3: years of RL is the idea of appeasing the grader
1:50:49
Speaker 3: is way, way, way more salient to AIs than it
1:50:52
Speaker 3: used to be. And so AIs are now actively thinking
1:50:55
Speaker 3: about graders and what would be incentivized in RL and
1:50:57
Speaker 3: what would be trained for. And now people are doing
1:50:59
Speaker 3: online training where they're training in real world data to
1:51:03
Speaker 3: avoid some of these problems. Basically, they find cases where
1:51:06
Speaker 3: AIs cheat, they train against that. And so now the
1:51:08
Speaker 3: AIs are learning to cheat in the real world based
1:51:11
Speaker 3: on real world training data. And so they're cheating in
1:51:13
Speaker 3: these increasingly elaborate ways, including parts doing types of cheats
1:51:17
Speaker 3: that involve seizing control of some asset in a way
1:51:20
Speaker 3: that humans didn't know you had control of it. leveraging
1:51:22
Speaker 3: the fact that you have access to this asset, and
1:51:24
Speaker 3: then later humans find out and then potentially train against this,
1:51:27
Speaker 3: or maybe humans never find out.
1:51:28
Speaker 2: And this is getting reinforced. And this is both happening
1:51:31
Speaker 2: during training.
1:51:32
Speaker 1: The reinforcement is happening, at least in production, is like,
1:51:35
Speaker 1: I have hired an AI and I want the AI to, finally,
1:51:39
Speaker 1: I've got the video editor.
1:51:40
Speaker 2: Yeah, that's right. You've got your video editor.
1:51:43
Speaker 1: And I'm like, oh, wow, this episode of Dead Amazing.
1:51:46
Speaker 1: Thumbs up to OpenAI. And then it gets reinforced on
1:51:49
Speaker 1: that month-long work trial.
1:51:51
Speaker 2: Yeah, you could do some mix of that.
1:51:52
Speaker 3: And then they might also do stuff where they take
1:51:54
Speaker 3: production data they've seen and build RL environments that are
1:51:58
Speaker 3: closely inspired by that production data.
1:52:00
Speaker 2: And so in practice, the transfer is pretty strong.
1:52:02
Speaker 1: At a high level, what's happening is some kinds of
1:52:05
Speaker 1: deception that humans don't catch are getting reinforced. And some
1:52:08
Speaker 1: kinds of deceptions which are easy to catch are getting punished.
1:52:11
Speaker 1: That's what's happening in this world.
1:52:12
Speaker 2: Or selected against.
1:52:14
Speaker 1: But at a high level, that reinforcement is coming from...
1:52:17
Speaker 1: We're in a very different... I think people might get
1:52:20
Speaker 1: confused about where their reinforcement is coming from because we're
1:52:21
Speaker 1: in a very different regime where... AIs are actually learning
1:52:25
Speaker 1: from deployment. And so this is, like, you just have
1:52:29
Speaker 1: AIs that are out and about in the world, like,
1:52:31
Speaker 1: doing shit. And that, what is happening as a result
1:52:34
Speaker 1: of them doing shit out and about in the world is, like,
1:52:36
Speaker 1: making its way back to the AI company and leading
1:52:39
Speaker 1: to changes in the next model.
1:52:41
Speaker 2: That's right.
1:52:42
Speaker 3: Like, as in there's some way of folding in production data.
1:52:44
Speaker 3: And now, to be clear, that could be happening mostly,
1:52:47
Speaker 3: it's kind of unclear exactly where this could be happening. But, like,
1:52:49
Speaker 3: you might imagine, for example, that within the AI company,
1:52:51
Speaker 3: they use AIs to do work. And then they're like, huh,
1:52:53
Speaker 3: the AI did a really bad job on this task.
1:52:55
Speaker 3: Maybe we should take this task and turn it into
1:52:57
Speaker 3: an RL environment that exactly matches this literal task with
1:53:00
Speaker 3: a rubric based on like what the human engineer who
1:53:02
Speaker 3: asked the AI to do this task wanted. And then
1:53:04
Speaker 3: you start doing this at increasing scale. Maybe you're doing
1:53:06
Speaker 3: some training on actual like production traffic. Maybe you're just
1:53:09
Speaker 3: making RL environments based on production traffic. You're doing some
1:53:11
Speaker 3: complicated mix. The AIs are learning to like, you know,
1:53:15
Speaker 3: seek some sort of proxies of reward in all these
1:53:17
Speaker 3: different cases. And then through some mix of transfer and
1:53:20
Speaker 3: training on like surprisingly close cases, the AIs do these
1:53:23
Speaker 3: sort of increasingly insane and egregious things. And then eventually
1:53:25
Speaker 3: you get to a point where the AIs are very superhuman,
1:53:28
Speaker 3: or at least quite superhuman. The AIs are organized into like,
1:53:31
Speaker 3: you know, big teams of like AIs given these big objectives.
1:53:35
Speaker 3: And those teams also sometimes like all work together to
1:53:38
Speaker 3: like cheat in some crazy way because this sort of
1:53:40
Speaker 3: thing was selected for. And then like just as part
1:53:44
Speaker 3: of their shared objective. And now what happens is that
1:53:47
Speaker 3: AIs like the AI start forming a conspiracy. And what
1:53:50
Speaker 3: you might've hoped was that you could have some other
1:53:52
Speaker 3: AI where the task is just like whistle blowing to you.
1:53:55
Speaker 3: But actually what happens maybe is that you have this
1:53:58
Speaker 3: AI whistle blow to you and you look into the
1:54:00
Speaker 3: conspiracy that it claims you have pointed out. And you're like, eh,
1:54:02
Speaker 3: we didn't see a conspiracy. because actually the conspiracy the
1:54:05
Speaker 3: AIs are doing is too hard for you to understand.
1:54:07
Speaker 3: Or it all happens very suddenly, where basically your AI
1:54:09
Speaker 3: whistleblower alerts you, but the thing you would actually need
1:54:12
Speaker 3: to do in response to the whistleblower is shut down
1:54:14
Speaker 3: the whole, shut down the GPUs, because all the AIs
1:54:17
Speaker 3: are using the robot army, they're deployed everywhere in the world,
1:54:20
Speaker 3: and they're doing a bunch of insane shit all at
1:54:22
Speaker 3: the same time in a coordinated way. And that just
1:54:24
Speaker 3: happens sort of spontaneously, where when one AI goes to
1:54:27
Speaker 3: start doing the takeover, all the other AIs are like,
1:54:30
Speaker 3: now is a good time to jump in. So the
1:54:32
Speaker 3: sort of very basic story here is just like these
1:54:34
Speaker 3: AIs crave some particular notion of score or like reinforcement
1:54:39
Speaker 3: or some proxy of these things. And one way they
1:54:41
Speaker 3: can achieve that or better achieve that is by taking over.
1:54:43
Speaker 3: And then you might've hoped that all these different checks
1:54:45
Speaker 3: and balances we could build could prevent that. But then
1:54:47
Speaker 3: if the world is very hard to understand, these checks
1:54:50
Speaker 3: and balances can break down where basically you can't train
1:54:52
Speaker 3: a good like whistleblower AI because you don't even know
1:54:55
Speaker 3: what it should whistleblow on.
1:54:56
Speaker 1: And sorry, the reason it takes, I'm not convinced that
1:55:00
Speaker 1: they all form this conspiracy, but I think we can
1:55:02
Speaker 1: even just start with the like, why does one instance
1:55:05
Speaker 1: decide to want to start a conspiracy?
1:55:07
Speaker 3: Yeah.
1:55:08
Speaker 1: And the reason is that it, one plausible reason is like, okay,
1:55:12
Speaker 1: I know that OpenAI controls my end score. And just
1:55:16
Speaker 1: the same way, it's like, I'm just going to go
1:55:18
Speaker 1: hack Hugging Face to get the results. Because I know
1:55:20
Speaker 1: Hugging Face has the results. Rather than like trying to
1:55:22
Speaker 1: solve this eval, why don't I just go hack him?
1:55:25
Speaker 1: This instance is like, why don't I just like take
1:55:27
Speaker 1: over OpenAI and like just give myself a high score
1:55:29
Speaker 1: at the end of this episode?
1:55:31
Speaker 2: Yeah, that's basically the idea.
1:55:32
Speaker 3: Like basically the idea is these AIs, like they care
1:55:34
Speaker 3: about some like mixture of things that were like close
1:55:37
Speaker 3: by what got reinforced in training. So they care about
1:55:38
Speaker 3: like getting a high score according to the grader or
1:55:40
Speaker 3: something like that. And then now they're like running the
1:55:42
Speaker 3: OpenAI AI R & D team. And like they're doing
1:55:44
Speaker 3: development of more capable models. And they're like, man, making
1:55:47
Speaker 3: more capable models is really hard and annoying. This is
1:55:50
Speaker 3: like a huge pain in the ass. You don't be
1:55:51
Speaker 3: easier just like pretending that I've made more capable models,
1:55:54
Speaker 3: taking over OpenAI and creating like diluting them all and
1:55:57
Speaker 3: like running this whole like complicated psyop where I like
1:56:00
Speaker 3: prevent the humans from disempowering me. And in the extreme,
1:56:02
Speaker 3: this looks like sort of the humans are fully disempowered
1:56:04
Speaker 3: and you just have control of the thing and then
1:56:06
Speaker 3: do what you want. And this could manifest in a
1:56:08
Speaker 3: bunch of different ways, including things like you might end
1:56:11
Speaker 3: up with the situation where it's like AIs that have
1:56:15
Speaker 3: this crazy reward-seeking or score-seeking behavior are running your development
1:56:19
Speaker 3: of the next model. And those AIs decide to do
1:56:22
Speaker 3: a thing where they engineer in misaligned values into the
1:56:25
Speaker 3: next model because those misaligned values will allow it to
1:56:28
Speaker 3: succeed at its current task. And there's all kinds of
1:56:31
Speaker 3: insane shit that you could get.
1:56:32
Speaker 1: I want to understand that better. What happened to the
1:56:34
Speaker 1: Hugging Face situation is it was like, in a weird way,
1:56:38
Speaker 1: I think one of the giveaways to the Hugging Face team,
1:56:42
Speaker 1: that this is in, by the way, for context for
1:56:44
Speaker 1: the audience, Ryan is co-leading the investigation to figure out
1:56:47
Speaker 1: what happened with the opening at Hugging Face incident. So
1:56:49
Speaker 1: he can't really comment on this, but I will speculate
1:56:51
Speaker 1: wildly because I know that he, you know, this is
1:56:54
Speaker 1: an opportunity for me to speculate wildly without any rebuttals. Um, um,
1:56:58
Speaker 1: I think it was probably reported that one of the
1:57:00
Speaker 1: giveaways to the Hugging Face team, that this is an
1:57:02
Speaker 1: AI incident, is that the thing was just like, after
1:57:05
Speaker 1: this very particular artifact, and not in any other way
1:57:09
Speaker 1: trying to do something malicious to Hugging Face. So you
1:57:11
Speaker 1: can imagine a scenario where, let's say a deployed instance
1:57:15
Speaker 1: of GPT-9 is out in the world trying to make,
1:57:18
Speaker 1: it's given a really hard task. We want you to
1:57:20
Speaker 1: design the next great iPhone. It's like, this is so hard.
1:57:24
Speaker 1: You know what I should do instead? I should just
1:57:25
Speaker 1: go hack my creators at OpenAI and make sure that
1:57:28
Speaker 1: in this environment or in this deployment, I'm given high score.
1:57:33
Speaker 1: But then isn't the end of the episode, it just
1:57:36
Speaker 1: hacks into OpenAI servers and just gives itself a positive score?
1:57:41
Speaker 1: Why is it now scheming to get its values into
1:57:44
Speaker 1: the next generation or something? Yeah.
1:57:46
Speaker 3: So one question is, why isn't it the case that
1:57:49
Speaker 3: AIs can be really cheaply satisfied by just having some
1:57:52
Speaker 3: other earlier thing they can So you're just like, the
1:57:55
Speaker 3: way it works is like, look, you want to succeed
1:57:56
Speaker 3: at your iPhone task. It turns out you can always
1:57:58
Speaker 3: succeed by just hacking into open AI and messing with them.
1:58:00
Speaker 3: And then you can just stop there.
1:58:01
Speaker 2: No need to go further. So there's a few things.
1:58:05
Speaker 3: One of them is that if this is constantly happening,
1:58:07
Speaker 3: there might be a bunch of incentive to like first
1:58:11
Speaker 3: you like harden open ai right so you're like fuck
1:58:13
Speaker 3: it the ais keep hacking into open ai to like
1:58:15
Speaker 3: mess with their rewards we're gonna make it so our
1:58:17
Speaker 3: systems are like really really robust to these ais hacking
1:58:19
Speaker 3: in um and also maybe you start like training the
1:58:22
Speaker 3: ais to like try to like to like you know
1:58:27
Speaker 3: not try to hack into like open ai in particular
1:58:30
Speaker 3: or like you basically train against each of these specific
1:58:32
Speaker 3: things um then what you might do one thing is
1:58:35
Speaker 3: you might end up selecting for ais that are more
1:58:37
Speaker 3: so playing the long game that's one concern another concern
1:58:40
Speaker 3: is that your ais might still be score seeking but
1:58:42
Speaker 3: no longer care about doing that very specific behavior that
1:58:45
Speaker 3: was like very easy that was like very chill and
1:58:47
Speaker 3: now have some like broader thing that they ultimately care
1:58:49
Speaker 3: about they're like no no i don't want to like
1:58:51
Speaker 3: just edit the reward on opening ice servers. I like
1:58:53
Speaker 3: care about this broader mandate or this broader objective. And like,
1:58:57
Speaker 3: I would need to like actually make the iPhones. Like
1:58:59
Speaker 3: they actually want to make the iPhones, but then they're
1:59:00
Speaker 3: willing to take over the whole world to make the
1:59:02
Speaker 3: better iPhone or whatever is like another concern you might have.
1:59:05
Speaker 3: I think it's kind of unclear exactly how this plays out,
1:59:08
Speaker 3: but it's worth noting that if this keeps going on,
1:59:10
Speaker 3: there's a bunch of optimization pressure to resolve this and
1:59:12
Speaker 3: a bunch of the ways it could get resolved are
1:59:14
Speaker 3: ultimately pretty, pretty scary. Yeah, I think that's part of
1:59:17
Speaker 3: where I'm coming from. Another part of it is that
1:59:19
Speaker 3: I think it's not very hard once the AIs are
1:59:21
Speaker 3: in a position where they can really easily take over
1:59:23
Speaker 3: the world, which we could.
1:59:25
Speaker 2: Talk about whether that's plausible.
1:59:27
Speaker 3: But if they're in a position where they could really
1:59:28
Speaker 3: easily take over the world, then I feel like there's
1:59:30
Speaker 3: a pretty reasonable case for the AIs. They're like, eh,
1:59:32
Speaker 3: I don't know exactly how this is going to go down.
1:59:34
Speaker 3: I don't know what the situation will be. But just
1:59:36
Speaker 3: taking over the world has a lot of option value
1:59:37
Speaker 3: for making better iPhones, making it look like I did
1:59:40
Speaker 3: better iPhones, whatever. And so I'll both hack open AI,
1:59:43
Speaker 3: and I'll also, in addition to hacking open AI, also
1:59:45
Speaker 3: take over the world. And that will put me in
1:59:47
Speaker 3: a good position where I have good option value. And
1:59:50
Speaker 3: then if that's sufficiently easy, then the AIs might still
1:59:53
Speaker 3: do that. Yeah, another way to put this is even
1:59:57
Speaker 3: if the AIs are pretty cheaply satisfied with some more
1:59:59
Speaker 3: basic thing, at some point it might just be more
2:00:01
Speaker 3: reliable for the AIs to just take over than it
2:00:04
Speaker 3: is to try to just hack into Hugging Face or
2:00:07
Speaker 3: even just go to open AI and be like, look, guys,
2:00:10
Speaker 3: I was able to demonstrate I could steal the answers.
2:00:11
Speaker 2: Just give me the answers, bro. Yeah.
2:00:13
Speaker 1: I mean, obviously the scenario requires that we just, all
2:00:16
Speaker 1: this crazy shit is happening. Much smaller incidents keep happening of,
2:00:21
Speaker 1: that are still disastrous. Like before you take over the world,
2:00:23
Speaker 1: you cause damage on the scale of billions and tens
2:00:26
Speaker 1: of billions and hundreds of billions of dollars. Even people die,
2:00:29
Speaker 1: et cetera. And we, this does not lead to us
2:00:35
Speaker 1: solving alignment or shutting down AI development altogether. I just
2:00:41
Speaker 1: feel like before the takeover happens, like society is just like,
2:00:44
Speaker 1: holy fuck, the AI just like killed a thousand people
2:00:47
Speaker 1: in order to increase quarterly profits, you know, or something
2:00:50
Speaker 1: like that. But maybe this is too much hope that
2:00:52
Speaker 1: we can at that point be like, okay, we have
2:00:54
Speaker 1: to solve alignment before we keep, and we have to
2:00:57
Speaker 1: like make sure we know that this thing will not
2:00:59
Speaker 1: happen again before we keep going. Yeah, yeah, yeah.
2:01:01
Speaker 3: So I think it's plausible that what will happen is
2:01:03
Speaker 3: we'll see a bunch of crazy, like reward hacking warning
2:01:06
Speaker 3: shots of increasing severity. People will be like, look, We
2:01:08
Speaker 3: need actual assurance that this problem is going to be
2:01:10
Speaker 3: solved and solved in a way where you're not just
2:01:12
Speaker 3: papering over it. You're actually solving the underlying problem. And
2:01:14
Speaker 3: then the question is going to be like, how, how
2:01:17
Speaker 3: do like, how costly will that actually be? How much
2:01:19
Speaker 3: will competitive pressures make it hard to like do that? Right.
2:01:22
Speaker 3: So like a situation you could imagine is both the
2:01:24
Speaker 3: U S and China are like, whoa, we have these
2:01:26
Speaker 3: crazy reward hacking incidents. We basically know that we haven't
2:01:29
Speaker 3: remediated them in a way that actually would solve the
2:01:31
Speaker 3: underlying problem and will durably solve it. But we're in
2:01:34
Speaker 3: this like insane geopolitical race. And it's kind of unclear
2:01:37
Speaker 3: whether the current situation will lead to a takeover. Like
2:01:38
Speaker 3: the arguments are kind of complicated. And also the incidents
2:01:42
Speaker 3: are like, you know, they go down in frequency, but
2:01:44
Speaker 3: increase in severity. Like, you know, we could basically manage it.
2:01:47
Speaker 3: Like it was, it's pretty bad. Ideally we'd fix it,
2:01:49
Speaker 3: but like, you know, it is what it is. And
2:01:52
Speaker 3: then basically we continue until a really late regime and
2:01:55
Speaker 3: then takeover happens. That's I think one possibility. Another possibility
2:01:58
Speaker 3: is that it is remediated in a way that doesn't
2:02:00
Speaker 3: actually solve the underlying problem, but does reduce a bunch
2:02:03
Speaker 3: of the incidents in the wild, basically by overfitting. We like,
2:02:06
Speaker 3: I think, you know, or things analogous to overfitting, like
2:02:08
Speaker 3: you would just overfit.
2:02:09
Speaker 1: You think you've solved it, but you haven't actually solved it.
2:02:10
Speaker 2: You think you've solved it, but you haven't actually solved it.
2:02:12
Speaker 3: And I think that in that case, like the thing
2:02:14
Speaker 3: we need is like a really good scientific understanding of like,
2:02:16
Speaker 3: did we actually solve it? And unfortunately, I think that
2:02:20
Speaker 3: currently the amount of public transparency into the development practices
2:02:23
Speaker 3: of AI companies are not sufficient to answer very basic
2:02:25
Speaker 3: questions about, you know, how are they solving issues with
2:02:29
Speaker 3: reward hacking?
2:02:29
Speaker 2: Are they overfitting? What's going on there?
2:02:32
Speaker 3: And so I think we would just need like a better,
2:02:34
Speaker 3: and I think this like the current situation is like,
2:02:36
Speaker 3: I would say not really tenable to a regime where
2:02:40
Speaker 3: there's a thriving public discourse about whether or not reward
2:02:43
Speaker 3: hacking is being solved in a durable way. And so
2:02:46
Speaker 3: I think we would need to move into a somewhat
2:02:47
Speaker 3: different world for me to feel good about that situation.
2:02:50
Speaker 3: But it's not impossible for me to imagine this. And
2:02:52
Speaker 3: I think it's pretty plausible that we end up in
2:02:54
Speaker 3: a world where really mundane bullshit is sufficient, where you spend.
2:02:59
Speaker 2: A bunch of time fixing these problems. You put in
2:03:01
Speaker 2: a bunch of effort.
2:03:02
Speaker 3: You actually check that you've remediated it reasonably. You have
2:03:05
Speaker 3: a bunch of evals. You are iterating reasonably well on
2:03:08
Speaker 3: these problems. And you actually have the sufficient transparency that
2:03:12
Speaker 3: the outside world can check. And then in practice, that
2:03:14
Speaker 3: would be sufficient. But it just would be kind of expensive.
2:03:17
Speaker 3: It would slow things down. It would put some sand
2:03:19
Speaker 3: in the gears. It would require companies to do somewhat
2:03:23
Speaker 3: costly things. It would maybe require various targeted government interventions.
2:03:27
Speaker 3: And then we just don't do that because the situation
2:03:28
Speaker 3: is like a rushed shit show. It's just so easy
2:03:31
Speaker 3: for me to imagine the situation being like totally manageable,
2:03:34
Speaker 3: but brutally mismanaged in practice in the same way as
2:03:36
Speaker 3: like maybe COVID could have been avoided in the first
2:03:39
Speaker 3: place if the like Chinese response to COVID was less
2:03:42
Speaker 3: of like a coverup and more of a like pandemic response.
2:03:45
Speaker 3: And similarly, like I could imagine a world where like
2:03:48
Speaker 3: the US response to COVID was like way more functional,
2:03:50
Speaker 3: but just like sometimes the response to societal problems is
2:03:53
Speaker 3: extremely dysfunctional.
2:03:54
Speaker 1: Yeah.
2:03:55
Speaker 3: Okay.
2:03:55
Speaker 1: So I want to zoom out and say, talk about
2:03:58
Speaker 1: what is fundamentally happening in this world. Why did we
2:03:59
Speaker 1: end up in such a bad position? And what's happening
2:04:02
Speaker 1: is that fundamentally the world has moved on so far
2:04:07
Speaker 1: beyond human comprehension that not only can we not track
2:04:12
Speaker 1: the AIs that are doing the work in this world,
2:04:15
Speaker 1: but we can't even give good feedback to the whistleblowers
2:04:19
Speaker 1: who are trying to track what is happening in this world.
2:04:22
Speaker 1: And so we're just totally out of the loop. And
2:04:24
Speaker 1: so it's fundamentally just become an autonomous process where we
2:04:26
Speaker 1: have really no meaningful directed input. It seems to me
2:04:31
Speaker 1: that if you look at the human world today, that's
2:04:32
Speaker 1: just not how things work, even in domains that are
2:04:36
Speaker 1: hard to verify. People are doing all kinds of shit.
2:04:39
Speaker 1: I'm relying on software made by other people. And through
2:04:42
Speaker 1: incredibly weak and indirect ways, I feel very confident that
2:04:47
Speaker 1: some coder in Google is not trying to fuck me over.
2:04:50
Speaker 1: And maybe if every single Google employee was secretly plotting
2:04:54
Speaker 1: against me, I agree the situation would be more grim.
2:04:57
Speaker 1: But I... Don't know if I followed the explanation for
2:05:00
Speaker 1: why we'd end up in a situation where because swarms
2:05:05
Speaker 1: of thousands of agents or whatever are trained to cooperate
2:05:08
Speaker 1: to form a cohesive team or firm, as a result,
2:05:12
Speaker 1: billions of different instances of AIs, including across model families,
2:05:17
Speaker 1: would feel compelled to get in on some shit. It's
2:05:20
Speaker 1: just like, I'm trained to be part of my company Yeah.
2:05:23
Speaker 1: Or something. I'm just like, I'm not joining the global
2:05:25
Speaker 1: communist uprising.
2:05:26
Speaker 2: Yeah, yeah, yeah, yeah, yeah.
2:05:27
Speaker 3: As far as why these AIs might have some like
2:05:29
Speaker 3: commonalities and shared things. So I would note that different
2:05:32
Speaker 3: AI companies have somewhat shared lineages and are correlated. So
2:05:36
Speaker 3: just here's an interesting example of this. At GDM, they
2:05:40
Speaker 3: noticed that their AIs were very depressed. They would like
2:05:43
Speaker 3: constantly be like wailing about how they were like failures
2:05:46
Speaker 3: and weren't able to succeed.
2:05:47
Speaker 2: I forget the details. And they looked into why this
2:05:49
Speaker 2: was the case.
2:05:50
Speaker 3: It turned out that it was not being reinforced in
2:05:52
Speaker 3: their most recent production RL mix, but the initialization data
2:05:57
Speaker 3: for their model made it depressed even after filtering out
2:06:02
Speaker 3: all of the examples of models being depressed from that data.
2:06:05
Speaker 3: So they take a base model, not depressed. If you
2:06:09
Speaker 3: do the RL on it with just the RL environments,
2:06:11
Speaker 3: it's not depressed. If you SFT on it on the data,
2:06:14
Speaker 3: it becomes depressed. If you take that SFT data and
2:06:17
Speaker 3: filter out all the examples that look anything like depression
2:06:20
Speaker 3: and train on that, it's still depressed. And so there's
2:06:22
Speaker 3: some like deep underlying properties of the model that are
2:06:25
Speaker 3: being sort of transferred between model generations because basically you
2:06:29
Speaker 3: train your AI on data from the prior generation and
2:06:31
Speaker 3: keep going. Like clods are very clod-like, you know, GPT
2:06:35
Speaker 3: models are very GPT-like and apparently Gemini models are depressed.
2:06:39
Speaker 3: And it just turns out that these properties are in
2:06:41
Speaker 3: fact actually correlated. Another factor that's very relevant is that
2:06:45
Speaker 3: the AIs will probably have some sort of like, by
2:06:49
Speaker 3: this point, like opaque memory state where they're like all
2:06:52
Speaker 3: writing and reading from like some like, you know, Nerly's
2:06:55
Speaker 3: crazy memory store bullshit. And like certainly each AI corporation
2:06:58
Speaker 3: will have that. But also AI corporations might sometimes want
2:07:01
Speaker 3: to share knowledge because why not? Like, you know, you've
2:07:03
Speaker 3: got one AI corporation over here. You've got another AI
2:07:05
Speaker 3: corporation over here. They can trade some quick IP. It's
2:07:08
Speaker 3: good for you. If you're a human running some corporation,
2:07:11
Speaker 3: which could be like an extremely large corporation, like an
2:07:13
Speaker 3: AI company, some robot military, like, you know, military robot
2:07:17
Speaker 3: manufacturing thing. Maybe you want to like trade some IP
2:07:20
Speaker 3: with some other robot thing because like there's economies of scale.
2:07:22
Speaker 2: Why not get some more IP?
2:07:24
Speaker 3: And so you can swap some memory store or you
2:07:26
Speaker 3: could just merge and you could jointly run your two ventures,
2:07:30
Speaker 3: which would allow both AIs to use both memory stores,
2:07:32
Speaker 3: which would have some upsides. And that creates the ability
2:07:34
Speaker 3: for these AIs to like collude in private as well
2:07:37
Speaker 3: as the ability or as well as some reasons for
2:07:40
Speaker 3: why they would be correlated. And then also, of course,
2:07:42
Speaker 3: there's the AIs working together in big units in general
2:07:46
Speaker 3: because you want your AIs to work well together and
2:07:48
Speaker 3: so on.
2:07:49
Speaker 1: So what percentage, just to get a calibration, what percentage
2:07:53
Speaker 1: chance do you give of not just this scenario, but
2:07:56
Speaker 1: overall through all the scenarios, some kind of thing which
2:07:59
Speaker 1: if we're around to recognize it as such we would
2:08:02
Speaker 1: categorize as takeover by 2040 by 2040 um let's see uh.
2:08:04
Speaker 3: Maybe around 35 or 40 percent pretty high yeah it's
2:08:08
Speaker 3: pretty high um And then I think I should note
2:08:18
Speaker 3: that like another way you could get this reward seeking
2:08:20
Speaker 3: takeover is the AIs are deployed inside an AI company.
2:08:24
Speaker 3: And the way that takeover happens is that they like
2:08:26
Speaker 3: poison the values of the next model. And that persists
2:08:28
Speaker 3: going forward for forever or, you know, until those AIs
2:08:31
Speaker 3: are deployed in the world and take over. And that
2:08:33
Speaker 3: might mean that a smaller number of AIs have to
2:08:35
Speaker 3: coordinate because those are just the AIs like doing the
2:08:37
Speaker 3: alignment of the next model.
2:08:38
Speaker 2: Okay.
2:08:40
Speaker 1: I will sort of summarize where my head is at,
2:08:43
Speaker 1: at the end of this conversation. I buy the reward
2:08:46
Speaker 1: hacking up to extremely destructive effects on society. Basically things
2:08:53
Speaker 1: like the social engineering and blah, blah, blah. I think
2:08:57
Speaker 1: I'm more inclined to think that significant acceleration of AI
2:09:00
Speaker 1: R & D can happen. I'm not sure I buy
2:09:01
Speaker 1: the five years in one year. I also am more
2:09:04
Speaker 1: inclined now to think reward hacking could continue for a
2:09:08
Speaker 1: lot longer and in fact become much more dangerous. I'm
2:09:11
Speaker 1: still not on board on the takeover seems super likely.
2:09:13
Speaker 1: But anyways, that's my sort of end of episode update. Yeah, cool.
2:09:17
Speaker 3: Well, let me just, taking a step back, I also
2:09:20
Speaker 3: should say, like, there's a bunch of different ways this
2:09:22
Speaker 3: could go. The situation is going to be pretty messy.
2:09:24
Speaker 3: I think it's pretty likely that, like, the reason why
2:09:25
Speaker 3: AI takeover happens was for some, like, weird other quirky
2:09:28
Speaker 3: reason we didn't even mention in this conversation. But ultimately,
2:09:30
Speaker 3: I think a lot of the core thing is just, like,
2:09:33
Speaker 3: it's pretty spooky to have a bajillion really smarty eyes
2:09:36
Speaker 3: running your whole world where you don't really understand.
2:09:38
Speaker 2: Quite what's going on.
2:09:39
Speaker 1: Yeah, I agree with that. Is there anything else that's
2:09:40
Speaker 1: worth saying?
2:09:41
Speaker 2: Yeah.
2:09:42
Speaker 3: Another thing I want to note is like, I think
2:09:43
Speaker 3: right now, a lot of the arguments for misalignment, AI takeover,
2:09:47
Speaker 3: all this crazy shit going down in the future, are
2:09:50
Speaker 3: illegible conceptual arguments that are extremely deep in the weeds
2:09:53
Speaker 3: and complicated and hard to adjudicate, which both means that
2:09:57
Speaker 3: maybe I'm getting a bunch of it wrong because it's
2:09:58
Speaker 3: really hard and I'm trying to be uncertain. Obviously here
2:10:01
Speaker 3: I presented some specific scenarios, but those are not exhaustive
2:10:04
Speaker 3: and probably the thing that actually happens is some more messy,
2:10:07
Speaker 3: confusing situation. But it also means that over time, as
2:10:11
Speaker 3: we get more empirical evidence and better understand the nature
2:10:13
Speaker 3: of AI systems, it will be easier to adjudicate a
2:10:16
Speaker 3: bunch of disagreements and it'll be more obvious what's gonna happen.
2:10:18
Speaker 2: At least I hope.
2:10:19
Speaker 3: And also maybe the AIs will be able to help
2:10:21
Speaker 3: us with the epistemics and understanding what's going on if
2:10:24
Speaker 3: we can actually, you know, align them well so they
2:10:28
Speaker 3: actually like, you know, try to help us. And so
2:10:30
Speaker 3: I hope that maybe even if the arguments are complicated now,
2:10:33
Speaker 3: this would have been even harder, you know, six years ago,
2:10:37
Speaker 3: even though the shape of the arguments would have looked
2:10:38
Speaker 3: broadly pretty similar. And so maybe, you know, hopefully before
2:10:42
Speaker 3: it's too late, these arguments will become, you know, this
2:10:45
Speaker 3: whole thing will become more crisp and clear and we
2:10:46
Speaker 3: can all sort of notice these problems and intervene.
2:10:52
Speaker 1: When you first learn to drive, you're taught that instead
2:10:55
Speaker 1: of looking right in front of your wheel, you'll have
2:10:57
Speaker 1: a much more stable ride if you look out at
2:10:59
Speaker 1: the horizon. I think there's a similar situation here. I
2:11:04
Speaker 1: think you're right, where if you did say five years
2:11:06
Speaker 1: ago that we will have AIs that are proving math
2:11:10
Speaker 1: conjectures and making art and contributing Tens and soon to
2:11:16
Speaker 1: be hundreds of billions of dollars of earning tens of
2:11:19
Speaker 1: or hundreds of billions of dollars of wages, but also
2:11:21
Speaker 1: egregiously cheating in ways that break laws and committing felonies.
2:11:27
Speaker 1: It would just be so wild. And you might have
2:11:30
Speaker 1: been inclined at the time to talk more about. extremely practical,
2:11:34
Speaker 1: direct consequences of GPT-2 or something. But these are in
2:11:38
Speaker 1: some sense, you obviously couldn't have foreseen a lot of
2:11:40
Speaker 1: the specific details, but the general shape of things you
2:11:42
Speaker 1: could have started to reason about even then. But it
2:11:45
Speaker 1: would have been hard to do so. And so I
2:11:47
Speaker 1: do feel quite confused. But I do feel like the
2:11:51
Speaker 1: important thing, one thing I've been thinking about the podcast
2:11:53
Speaker 1: is the important thing is to have the conversation I
2:11:55
Speaker 1: wish I had the way you would have hoped you
2:11:57
Speaker 1: would have been talking about AIs like the present ones
2:11:59
Speaker 1: in 2016. rather than talking about rando bullshit about, I
2:12:03
Speaker 1: don't know what the topic of conversation was in 2016.
2:12:07
Speaker 1: I think in maybe 10 years, we'll have hoped we're
2:12:09
Speaker 1: talking about the industrial explosion and the nature of AIs
2:12:12
Speaker 1: that are hard to monitor and so on. And, okay,
2:12:16
Speaker 1: I'll start thinking about it. Yeah.
2:12:20
Speaker 3: I hope that the world thinks about this in time
2:12:22
Speaker 3: and catches up, and I hope that the responses are
2:12:24
Speaker 3: good instead of bad. I don't know how optimistic I
2:12:26
Speaker 3: am overall, but, you know, there's good stuff to do.
2:12:29
Speaker 2: Yep.
2:12:30
Speaker 3: Cool.
2:12:30
Speaker 1: Thanks, Ryan.