Latest / In the Loop AI / Dwarkesh Patel: The Scaling Era of AI is Here

Transcript

Dwarkesh Patel: The Scaling Era of AI is Here · In the Loop AI

0:03
Ryan Sean Adams: Dwarkesh Patel, we are big fans. It's an honor to have you.
0:07
Dwarkesh: Thank you so much for having me on.
0:08
Ryan Sean Adams: Okay, so you have a book out. It's called The Scaling Era, an oral history of AI from 2019 to 2025.
0:16
Ryan Sean Adams: These are some key dates here. This is really a story of how AI emerged.
0:21
Ryan Sean Adams: And it seemed to have exploded on people's radar over the past five years.
0:26
Ryan Sean Adams: And And everyone in the world, it feels like, is trying to figure out what just
0:29
Ryan Sean Adams: happened and what is about to happen.
0:32
Ryan Sean Adams: And I feel like for this story, we should start at the beginning, as your book does.
0:36
Ryan Sean Adams: What is the scaling era of AI and when abouts did it start? What were the key milestones?
0:42
Dwarkesh: So I think the undertold story about everybody's, of course,
0:47
Dwarkesh: been hearing more and more about AI.
0:48
Dwarkesh: The under-told story is that the big contributor to these AI models getting
0:53
Dwarkesh: better over time has been the fact that we are throwing exponentially more compute
0:58
Dwarkesh: into trading frontier systems every year.
1:00
Dwarkesh: So by some estimates, we spend 4x every single year over the last decade trading
1:04
Dwarkesh: the frontier system than the one before it.
1:06
Dwarkesh: And that just means that we're spending hundreds of thousands of times more
1:10
Dwarkesh: compute than the systems of the early 2010s.
1:15
Dwarkesh: Of course, we've also had algorithmic breakthroughs in the meantime.
1:17
Dwarkesh: 2018, we had the Transformer.
1:19
Dwarkesh: Since then, obviously, many companies have made small improvements here and there.
1:23
Dwarkesh: But the overwhelming fact that we're spending already hundreds of billions of
1:28
Dwarkesh: dollars in building up the infrastructure,
1:31
Dwarkesh: the data centers, the chips for these models, and this picture is only going
1:36
Dwarkesh: to intensify if this exponential keeps going,
1:38
Dwarkesh: 4x a year, over the next two years, is something that is on the minds of the
1:45
Dwarkesh: CFOs of the big hyperscalers and the people planning the expenditures and training going forward,
1:49
Dwarkesh: but is not as common in the conversation around where AI is headed.
1:54
Ryan Sean Adams: So what do you feel like people should know about this?
1:57
Ryan Sean Adams: Like what is the scaling era? There have been other eras maybe of AI or compute,
2:02
Ryan Sean Adams: but what's special about the scaling era?
2:04
Dwarkesh: People started noticing. Well, first of all, in 2012, there's this,
2:09
Dwarkesh: Ilya Seskaver and others started using neural networks in order to categorize images.
2:16
Dwarkesh: And just noticing that instead of doing something hand-coded,
2:19
Dwarkesh: you can get a lot of juice out of just neural networks, black boxes.
2:24
Dwarkesh: You just train them to identify what thing is like what.
2:28
Dwarkesh: And then people started playing around these neural networks more,
2:31
Dwarkesh: using them for different kinds of applications.
2:33
Dwarkesh: And then the question became, we're noticing that these models get better if
2:39
Dwarkesh: you throw more data at them and you throw more compute at them.
2:41
Dwarkesh: How can we shove as much compute into these models as possible?
2:47
Dwarkesh: And the solution ended up being obviously internet text. So you need an architecture
2:51
Dwarkesh: which is amenable to the trillions of tokens that have been written over the
2:55
Dwarkesh: last few decades and put up on the internet.
2:57
Dwarkesh: And we had this happy coincidence of the kinds of architectures that are amenable
3:01
Dwarkesh: to this kind of training with the GPUs that were originally made for gaming.
3:05
Dwarkesh: We've had decades of internet text being compiled and Ilias actually called it the fossil fuel of AI.
3:12
Dwarkesh: It's like this reservoir that we can call upon to train these minds,
3:17
Dwarkesh: which are like, you know, they're fitting the mold of human thought because
3:20
Dwarkesh: they're trading on trillions of tokens of human thought.
3:23
Dwarkesh: And so then it's just been a question of making these models bigger,
3:27
Dwarkesh: of using this data that we're getting from internet techs to further keep training them.
3:33
Dwarkesh: And over the last year, as you know, the last six months, the new paradigm has
3:38
Dwarkesh: been not only are we going to pre-train on all this internet text,
3:41
Dwarkesh: we're going to see if we can have them solve math puzzles,
3:45
Dwarkesh: coding puzzles, and through this, give them reasoning capabilities.
3:50
Dwarkesh: The kind of thing, by the way, I mean, I have some skepticism around AGI just
3:55
Dwarkesh: around the corner, which we'll get into.
3:56
Dwarkesh: But just the fact that we now have machines which can like reason,
4:00
Dwarkesh: like, you know, you can like ask a question to a machine and it'll go away for a long time.
4:04
Dwarkesh: It'll like think about it and then like it'll come back to you with a smart answer.
4:06
Dwarkesh: And we just sort of take it for granted. But obviously, we also know that they're
4:10
Dwarkesh: extremely good at coding, especially.
4:12
Dwarkesh: I don't know if you actually got a chance to play around with Cloud Code or
4:15
Dwarkesh: Cursor or something. But it's a wild experience to design, explain at a high
4:20
Dwarkesh: level, I want an application to does X.
4:22
Dwarkesh: 15 minutes later, there's like 10 files of code and the application is built.
4:28
Josh Kale: That's where we stand.
4:29
Dwarkesh: I have takes on how much this can continue. The other important dynamic,
4:33
Dwarkesh: I'll add my monologue here, but the other important dynamic is that if we're
4:36
Dwarkesh: going to be living in the scaling era, you can't continue exponentials forever,
4:41
Dwarkesh: and certainly not exponentials that are 4x a year forever.
4:44
Dwarkesh: And so right now, we're approaching a point where within by 2028,
4:50
Dwarkesh: at most by 2030, we will literally run out of the energy we need to keep trading
4:57
Dwarkesh: these frontier systems,
4:58
Dwarkesh: the capacity at the leading edge nodes, which manufacture the chips that go
5:02
Dwarkesh: into the dyes, which go into these GPUs, even the raw fraction of GDP that will
5:07
Dwarkesh: have to use to train frontier systems.
5:09
Dwarkesh: So we have a couple more years left of the scaling era. And the big question
5:12
Dwarkesh: is, will we get to AGI before then?
5:15
Ryan Sean Adams: I mean, that's kind of a key insight of your book that like,
5:18
Ryan Sean Adams: we're in the middle of the scaling era.
5:19
Ryan Sean Adams: I guess we're like, you know, six years in or so. And we're not quite sure.
5:23
Ryan Sean Adams: It's like, like the protagonist in the middle of the story, We don't know exactly
5:26
Ryan Sean Adams: which way things are going to go.
5:28
Ryan Sean Adams: But I want you to maybe, Dworkesh, help folks get an intuition for why scaling in this way even works.
5:36
Ryan Sean Adams: Because I'll tell you, for me and for most people, our experience with these
5:41
Ryan Sean Adams: revolutionary AI models probably started in 2022 with ChatGPT3 and then ChatGPT4
5:46
Ryan Sean Adams: and seeing all the progress, all these AI models.
5:49
Ryan Sean Adams: And it just seems really unintuitive that if you take a certain amount of compute
5:55
Ryan Sean Adams: and you take a certain amount of data, out pops AI, out pops intelligence.
6:01
Ryan Sean Adams: Could you help us get an intuition for this magic?
6:05
Ryan Sean Adams: How does the scaling law even work? Compute plus data equals intelligence? Is that really all it is?
6:12
Dwarkesh: To be honest, I've asked so many AI researchers this exact question on my podcast.
6:17
Dwarkesh: And I could tell you some potential theories of why it might work.
6:20
Dwarkesh: I don't think we understand.
6:24
Dwarkesh: You know what? I'll just say that. I don't think we understand.
6:27
Ryan Sean Adams: We don't understand how this works. We know it works, but we don't understand
6:30
Dwarkesh: How it works. We have evidence from actually, of all things,
6:36
Dwarkesh: primatology of what could be going on here, or at least like why similar patterns
6:41
Dwarkesh: in other parts of the world.
6:42
Dwarkesh: So what I found really interesting, There's this research by this researcher,
6:46
Dwarkesh: Susanna Herculana Huzel,
6:48
Dwarkesh: which shows that if you look at how the number of neurons in the brain of a rat,
6:56
Dwarkesh: different kinds of rat species increases, as the weight of their brains increase
7:01
Dwarkesh: from species to species, there's this very sublinear pattern.
7:04
Dwarkesh: So if their brain size doubles, the neuron count will not double between different rat species.
7:09
Dwarkesh: And there's other animals where there's other kinds of...
7:14
Dwarkesh: Families of species for which this is true. The two interesting exceptions to
7:18
Dwarkesh: this rule, where there is actually a linear increase in neuron count and brain
7:22
Dwarkesh: size, is one, certain kinds of birds.
7:26
Dwarkesh: So, you know, birds are actually very smart, given the size of their brains, and primates.
7:32
Dwarkesh: So the theory for what happened with humans is that we unlocked an architecture that was very scalable.
7:38
Dwarkesh: So the way people talk about transformers being more scalable and then LSTMs,
7:41
Dwarkesh: the thing that preceded them in 2018.
7:43
Dwarkesh: We unlocked this architecture as it's very scalable.
7:46
Dwarkesh: And then we were in an evolutionary niche millions of years ago,
7:50
Dwarkesh: which rewarded marginal increases in intelligence.
7:53
Dwarkesh: If you get slightly smarter, yes, the brain costs more energy,
7:56
Dwarkesh: but you can save energy in terms of like not having to, you can cook,
7:59
Dwarkesh: you can cook food so you don't have to spend much more on digestion.
8:02
Dwarkesh: You can find a game, you can find different ways of foraging.
8:06
Dwarkesh: Birds were not able to find this evolutionary niche, which rewarded the incremental
8:11
Dwarkesh: increases in intelligence because if your brain gets too heavy as a bird, you're not going to fly.
8:17
Dwarkesh: So it was this happy coincidence of these two things. Now, why is it the case
8:21
Dwarkesh: that the fact that our brains could get bigger resulted in us becoming as smart
8:27
Dwarkesh: as we are? We still don't know.
8:29
Dwarkesh: And there's many different dissimilarities between AIs and humans.
8:32
Dwarkesh: While our brains are quite big, we don't need to be trained.
8:35
Dwarkesh: You know, a human from the age they're zero to 18 is not seeing within an order
8:41
Dwarkesh: of magnitude of the amount of information these LLMs are trained on.
8:44
Dwarkesh: So LLMs are extremely data inefficient.
8:46
Dwarkesh: They need a lot more data, but the pattern of scaling, I think we see in many different places.
8:53
Ryan Sean Adams: So is that a fair kind of analog? This analog has always made sense to me.
8:57
Ryan Sean Adams: It's just like transformers are like neurons.
9:00
Ryan Sean Adams: You know, AI models are sort of like the human brain.
9:04
Ryan Sean Adams: Evolutionary pressures are like gradient descent, reward algorithms and out
9:09
Ryan Sean Adams: pops human intelligence. We don't really understand that.
9:13
Ryan Sean Adams: We also don't understand AI intelligence, but it's basically the same principle at work.
9:17
Dwarkesh: I think it's a super fascinating, but also very thorny question because is gradient
9:23
Dwarkesh: intelligence like evolution?
9:24
Dwarkesh: Well, yes, in one sense. But also when we do gradient descent on these models,
9:29
Dwarkesh: we start off with the weights and then we're, you know, it's like learning how
9:35
Dwarkesh: does chemistry work, how does coding work, how does math work.
9:38
Dwarkesh: And that's actually more similar to lifetime learning, which is to say that,
9:42
Dwarkesh: like, by the time you're already born to the time you turn 18 or 25,
9:46
Dwarkesh: the things you learn, and that's not evolution.
9:48
Dwarkesh: Evolution designed the system or the brain by which you can do that learning,
9:53
Dwarkesh: but the lifetime learning itself is not evolution. And so there's also this
9:57
Dwarkesh: interesting question of, yeah, is training more like evolution?
10:01
Dwarkesh: In which case, actually, we might be very far from AGI because the amount of
10:04
Dwarkesh: compute that's been spent over the course of evolution to discover the human
10:07
Dwarkesh: brain, you know, could be like 10 to the 40 flops. There's been estimates, you know, whatever.
10:12
Dwarkesh: I'm sure it will bore you to discover, talk about how these estimates are derived,
10:15
Dwarkesh: but just like how much versus is it like a single lifetime,
10:20
Dwarkesh: like going from the age of zero to the age of 18, which is closer to,
10:23
Dwarkesh: I think, 10 to the 24 flops, which is actually less than compute than we use
10:26
Dwarkesh: to train frontier systems.
10:28
Dwarkesh: All right, anyways, we'll get back to more relevant questions.
10:33
Ryan Sean Adams: Well, here's kind of a big picture question as well.
10:36
Ryan Sean Adams: It's like I'm constantly fascinated with the metaphysical types of discussions
10:41
Ryan Sean Adams: that some AI researchers kind of take.
10:43
Ryan Sean Adams: Like a lot of AI researchers will talk in terms of when they describe what they're
10:48
Ryan Sean Adams: making, we're making God.
10:49
Ryan Sean Adams: Like why do they say things like that? What is this talk of like making God?
10:53
Ryan Sean Adams: What does that mean? Is it just the idea that scaling laws don't cease?
10:57
Ryan Sean Adams: And if we can, you know, scale intelligence to AGI, then there's no reason we
11:03
Ryan Sean Adams: can't scale far beyond that and create some sort of a godlike entity.
11:07
Ryan Sean Adams: And essentially, that's what the quest is. We're making artificial superintelligence.
11:12
Ryan Sean Adams: We're making a god. We're making god.
11:13
Dwarkesh: I think people focus too much on when they, I think this God discussion focuses
11:19
Dwarkesh: too much on the hypothetical intelligence of a single copy of an AI.
11:27
Dwarkesh: I do believe in the notion of a super intelligence, which is not just functionally,
11:32
Dwarkesh: which is not just like, oh, it knows a lot of things, but is actually qualitatively
11:36
Dwarkesh: different than human society.
11:38
Dwarkesh: But the reason is not because I think it's so powerful that any one individual
11:42
Dwarkesh: copy of AI will be as smart, but because of the collective advantages that AIs
11:48
Dwarkesh: will have, which have nothing to do with their raw intelligence,
11:51
Dwarkesh: but rather the fact that these models will be digital or they already are digital,
11:55
Dwarkesh: but eventually they'll be as smart as humans at least.
11:58
Dwarkesh: But unlike humans, because of our biological constraints, these models can be copied.
12:03
Dwarkesh: If there's a model that has learned a lot about a specific domain,
12:06
Dwarkesh: you can make infinite copies of it.
12:08
Dwarkesh: And now you have an infinite copies of Jeff Dean or Ilya Satskova or Elon Musk
12:12
Dwarkesh: or any skilled person you can think of.
12:15
Dwarkesh: They can be merged. So the knowledge that each copy is learning can be amalgamated
12:21
Dwarkesh: back into the model and then back to all the copies.
12:24
Dwarkesh: They can be distilled. They can run at superhuman speeds.
12:29
Dwarkesh: These collective advantages, also they can communicate in latent space.
12:32
Dwarkesh: These collective advantages.
12:33
Ryan Sean Adams: They're immortal. I mean, you know, as an example.
12:36
Dwarkesh: Yes, exactly. No, I mean, that's actually, tell me if I'm rabbit holing too
12:40
Dwarkesh: much, but like one really interesting question will come about is how do we prosecute AIs?
12:45
Dwarkesh: Because the way we prosecute humans is that we will throw you in jail if you commit a crime.
12:50
Dwarkesh: But if there's trillions of copies or thousands of copies of an AI model,
12:57
Dwarkesh: if a copy of an AI model, if an instance of an AI model does something bad, what do you do?
13:01
Dwarkesh: Does the whole model have to get, and how do you even punish a model,
13:04
Dwarkesh: right? Like, does it care about its weights being squandered?
13:09
Dwarkesh: Yeah, there's all kinds of questions that arise because of the nature of what AIs are.
13:14
Dwarkesh Patel: And also who is liable for that, right?
13:16
Dwarkesh: Like, is it the toolmaker?
13:17
Dwarkesh Patel: Is it the person using the tool? Who is responsible for these things?
13:20
Dwarkesh Patel: There's one topic that I do want to come to here about scaling laws,
13:23
Dwarkesh Patel: At what time did we realize that scaling laws were going to work?
13:27
Dwarkesh Patel: Because there were a lot of theses early in the days, early 2000s about AI,
13:31
Dwarkesh Patel: how we were going to build better models.
13:33
Dwarkesh Patel: Eventually, we got to the transformer. But at what point did researchers and
13:36
Dwarkesh Patel: engineers start to realize that, hey, this is the correct idea.
13:39
Dwarkesh Patel: We should start throwing lots of money and resources towards this versus other
13:42
Dwarkesh Patel: ideas that were just kind of theoretical research ideas, but never really took off.
13:45
Dwarkesh Patel: We kind of saw this with GPT two to three, where there's this huge improvement.
13:49
Dwarkesh: A lot of.
13:49
Dwarkesh Patel: Resources went into it. Was there a specific moment in time or a specific breakthrough
13:53
Dwarkesh Patel: that led to the start of these scaling laws?
13:55
Dwarkesh: I think it's been a slow process of more and more people appreciating this nature
14:00
Dwarkesh: of the overwhelming role of compute in driving forward progress.
14:05
Dwarkesh: In 2018, I believe, Dario Amadei wrote a memo that was secret while he was at
14:13
Dwarkesh: OpenAI. Now he's the CEO of Anthropic.
14:16
Dwarkesh: But while he's at OpenAI, he's subsequently revealed on my podcast that he wrote
14:19
Dwarkesh: this memo where the title of the memo was called Big Blob of Compute.
14:25
Dwarkesh: And it says basically what you expect it to say, which is that like,
14:29
Dwarkesh: yes, there's ways you can mess up the process of training. You have the wrong
14:31
Dwarkesh: kinds of data or initializations.
14:33
Dwarkesh: But fundamentally, AGI is just a big blob of compute.
14:37
Dwarkesh: And then we've gotten over the subsequent years, there was more empirical evidence.
14:41
Dwarkesh: So a big update, I think it was 2021.
14:44
Dwarkesh: Correct me. Somebody definitely will correct me in the comments.
14:46
Dwarkesh: I'm wrong. There were these,
14:48
Dwarkesh: there's been multiple papers of these scaling laws where you can show that the
14:54
Dwarkesh: loss of the model on the objective of predicting the next token goes down very predictably,
15:01
Dwarkesh: almost to like multiple decimal places of correctness based on how much more
15:07
Dwarkesh: compute you throw in these models.
15:08
Dwarkesh: And the compute itself is a function of the amount of data you use and how big
15:13
Dwarkesh: the model is, how many parameters it has.
15:15
Dwarkesh: And so that was an incredibly strong evidence back in the day,
15:19
Dwarkesh: a couple of years ago, because then you could say, well, OK,
15:22
Dwarkesh: if it really has this incredibly low loss of predicting the next token in all
15:28
Dwarkesh: human output, including scientific papers, including GitHub repositories.
15:34
Dwarkesh: Then doesn't it mean it has actually had to learn coding and science and all
15:40
Dwarkesh: these skills in order to make those predictions, which actually ended up being true.
15:43
Dwarkesh: And it was it was something people, you know, we take it for granted now,
15:46
Dwarkesh: but it actually even as of a year or two ago, people were really even denying that premise.
15:50
Dwarkesh: But some people a couple of years ago just like thought about it and like,
15:53
Dwarkesh: yeah, actually, that would mean that it's learned the skills.
15:55
Dwarkesh: And that's crazy that we just have this strong empirical pattern that tells
15:59
Dwarkesh: us exactly what we need to do in order to learn these skills.
16:02
Dwarkesh Patel: And it creates this weird perception, right, where like very early on and so
16:05
Dwarkesh Patel: to this day, it really is just a token predictor, right?
16:07
Dwarkesh Patel: Like we're just predicting the next word in the sentence. But somewhere along
16:10
Dwarkesh Patel: the lines, it actually creates this perception of intelligence.
16:14
Dwarkesh Patel: So I guess we covered the early historical context. I kind of want to bring
16:18
Dwarkesh Patel: the listeners up to today, where we are currently, where the scaling laws have
16:21
Dwarkesh Patel: brought us in the year 2025.
16:23
Dwarkesh Patel: So can you kind of outline where we've gotten to from early days of GPTs to
16:28
Dwarkesh Patel: now we have GPT-4, we have Gemini Ultra, we have Club, which you mentioned earlier.
16:32
Dwarkesh Patel: We had the breakthrough of reasoning.
16:33
Dwarkesh Patel: So what can leading frontier models do today?
16:36
Dwarkesh: So there's what they can do. And then there's the question of what methods seem to be working.
16:41
Dwarkesh: I guess we can start at what they seem to be able to do. They've shown to be
16:46
Dwarkesh: remarkably useful at coding and not just at answering direct questions about
16:51
Dwarkesh: how does this line of code work or something.
16:53
Dwarkesh: But genuinely just autonomously working for 30 minutes or an hour,
16:57
Dwarkesh: doing the task, it would take a front-end developer a whole day to do.
17:01
Dwarkesh: And you can just ask them at a high level, do this kind of thing,
17:04
Dwarkesh: and they can go ahead and do it.
17:05
Dwarkesh: Obviously, if you've played around with it, you know that they're extremely
17:07
Dwarkesh: useful assistants in terms of research, in terms of even therapists,
17:12
Dwarkesh: whatever other use cases.
17:13
Dwarkesh: On the question of what training methods seem to be working,
17:16
Dwarkesh: we do seem to be getting evidence that pre-training is plateauing,
17:19
Dwarkesh: which is to say that we had GPT 4.5, which was just following this old mold
17:25
Dwarkesh: of make the model bigger,
17:27
Dwarkesh: but it's fundamentally doing the same thing of next token prediction.
17:31
Dwarkesh: And apparently it didn't pass muster. The OpenAI had to deprecate it because
17:36
Dwarkesh: there's this dynamic where the bigger the model is, the more it costs not only
17:39
Dwarkesh: to train, but also to serve, right?
17:41
Dwarkesh: Because every time you serve a user, you're having to run the whole model,
17:44
Dwarkesh: which is going, so, but that doesn't be working is RL, which is this process
17:49
Dwarkesh: of, not just training them on existing tokens on the internet,
17:52
Dwarkesh: but having the model itself try to answer math and coding problems.
17:55
Dwarkesh: And finally, we got to the point where the model is smart enough to get it right
17:57
Dwarkesh: some of the time, and so you can give it some reward, and then it can saturate
18:01
Dwarkesh: these tough reasoning problems.
18:04
Dwarkesh Patel: And then what was the breakthrough with reasoning for the people who aren't familiar?
18:08
Dwarkesh Patel: What made reasoning so special that we hadn't discovered before?
18:11
Dwarkesh Patel: And what did that kind of unlock for models that we use today?
18:14
Dwarkesh: I'm honestly not sure. I mean, GBD-4 came out a little over two years ago,
18:19
Dwarkesh: and then it was after two years after GPT-4 came out that O-1 came out which
18:23
Dwarkesh: was the original reasoning breakthrough I think last November and,
18:28
Dwarkesh: And subsequently, a couple of months later, DeepSeq showed in their R1 paper.
18:33
Dwarkesh: So DeepSeq open source their research and they explained exactly how their algorithm worked.
18:37
Dwarkesh: And it wasn't that complicated. It was just like what you would expect,
18:41
Dwarkesh: which is get some math problems,
18:44
Dwarkesh: give for some initial problems, tell the model exactly what the reasoning trace
18:48
Dwarkesh: looks like, how you solve it, just like write it out and then have the model
18:51
Dwarkesh: like try to do it raw on the remaining problems.
18:54
Dwarkesh: Now, I know it sounds incredibly arrogant to say, well, it wasn't that complicated.
18:57
Dwarkesh: Why did it take you years?
18:58
Dwarkesh: I think there's an interesting insight there of even things which you think
19:02
Dwarkesh: will be simple in terms of high level description of how to solve the problem
19:05
Dwarkesh: end up taking longer in terms of haggling out the remaining engineering hurdles
19:10
Dwarkesh: than you might naively assume.
19:12
Dwarkesh: And that should update us on how long it will take us to go through the remaining
19:17
Dwarkesh: bottlenecks on the path to AGI.
19:19
Dwarkesh: Maybe that will be tougher than people imagine, especially the people who think
19:22
Dwarkesh: we're only two to three years away.
19:24
Dwarkesh: But all this to say, yeah, I'm not sure why it took so long after GPT-4 to get
19:27
Dwarkesh: a model trained on a similar level of capabilities that could then do reasoning.
19:31
Dwarkesh Patel: And in terms of those abilities, the first answer you had to what can it do was coding.
19:36
Dwarkesh Patel: And I hear that a lot of the time when I talk to a lot of people is that coding
19:40
Dwarkesh Patel: seems to be a really strong suit and a really huge unlock to using these models.
19:44
Dwarkesh Patel: And I'm curious, why coding over general intelligence?
19:48
Dwarkesh Patel: Is it because it's placed in a more confined box of parameters?
19:51
Dwarkesh Patel: I know in the early days, we had the AlphaGo and And we had the AIs playing
19:54
Dwarkesh Patel: chess and they exceed, they perform so well because they were kind of contained
19:58
Dwarkesh Patel: within this box of parameters that was a little less open-ended than general intelligence.
20:01
Dwarkesh Patel: Is that the reason why coding is kind of at the frontier right now of the ability of these models?
20:06
Dwarkesh: There's two different hypotheses. One is based around this idea called Moravac's paradox.
20:13
Dwarkesh: And this was an idea, by the way, one super interesting figure,
20:16
Dwarkesh: actually, I should have mentioned him earlier.
20:18
Dwarkesh: One super interesting figure in the history of scaling is Hans Moravac,
20:22
Dwarkesh: who I think in the 90s predicts that 2028 will be the year that we will get to AGI.
20:29
Dwarkesh: And the way he predicts this, which is like, you know, we'll see what happens,
20:32
Dwarkesh: but like not that far off the money as far as I'm concerned.
20:35
Dwarkesh: The way he predicts this is he just looks at the growth in computing power year
20:41
Dwarkesh: over year and then looks at how much compute he estimated the human brain to be to require.
20:47
Dwarkesh: And just like, OK, we'll have computers as powerful as the human brain by 2028.
20:51
Dwarkesh: Which is like at once a deceptively simple argument, but also ended up being
20:57
Dwarkesh: incredibly accurate and like worked, right?
21:01
Dwarkesh: I might add a fact drive it was 2028, but it was within that,
21:03
Dwarkesh: like within something you would consider a reasonable guess, given what we know now.
21:07
Dwarkesh: Sorry, anyway, so the Morrowind's paradox is this idea that computers seemed
21:12
Dwarkesh: in AI get better first at the skills which humans are the worst at.
21:19
Dwarkesh: Or at least there's a huge variation in the human repertoire.
21:22
Dwarkesh: So we think of coding as incredibly hard, right? We think this is like the top
21:26
Dwarkesh: 1% of people will be excellent coders.
21:29
Dwarkesh: We also think of reasoning as very hard, right? So if you like read Aristotle,
21:32
Dwarkesh: he says, the thing which makes humans special, which distinguishes us from animals is reasoning.
21:38
Dwarkesh: And these models aren't that useful yet at almost anything. The one thing they can do is reasoning.
21:45
Dwarkesh: So how do we explain this pattern? And Moravec's answer is that evolution has
21:51
Dwarkesh: spent billions of years optimizing us to do things we take for granted.
21:56
Dwarkesh: Move around this room, right? I can pick up this can of Coke,
21:58
Dwarkesh: move it around, drink from it.
22:00
Dwarkesh: And that we can't even get robots to do at all yet.
22:04
Dwarkesh: And in fact, it's so ingrained in us by evolution that there's no human, or.
22:08
Ryan Sean Adams: At least humans who don't have
22:10
Dwarkesh: Disabilities will all be able to do this. And so we just take it for granted
22:13
Dwarkesh: that this is an easy thing to do.
22:15
Dwarkesh: But in fact, it's evidence of how long evolution has spent getting humans up to this point.
22:19
Dwarkesh: Whereas reasoning, logic, all of these skills have only been optimized by evolution
22:27
Dwarkesh: over the course of the last few million years.
22:30
Dwarkesh: So there's been a thousand fold less evolutionary pressure towards coding than
22:35
Dwarkesh: towards just basic locomotion.
22:38
Dwarkesh: And this has actually been very accurate in predicting what kinds of progress
22:42
Dwarkesh: we see even before we got deep learning, right?
22:43
Dwarkesh: Like in the 40s when we got our first computers, the first thing that we could
22:47
Dwarkesh: use them to do is long calculations for ballistic trajectories at the time for World War II.
22:52
Dwarkesh: Humans suck at long calculations by hand.
22:56
Dwarkesh: And anyways, so that's the explanation for coding, which seems hard for humans,
23:00
Dwarkesh: is the first thing that went to AIs.
23:02
Dwarkesh: Now, there's another theory, which is that this is actually totally wrong.
23:05
Dwarkesh: It has nothing to do with the seeming paradox of how long evolution has optimized
23:10
Dwarkesh: us for, and everything to do with the availability of data.
23:15
Dwarkesh: So we have GitHub, this repository of all of human code, at least all open source
23:23
Dwarkesh: code written in all these different languages, trillions and trillions of tokens.
23:26
Dwarkesh: We don't have an analogous thing for robotics. We don't have this pre-training
23:29
Dwarkesh: corpus. And that explains why code has made so much more progress than robotics.
23:34
Ryan Sean Adams: That's fascinating because if there's one thing that I could list that we'd
23:38
Ryan Sean Adams: want AI to be good at, probably coding software is number one on that list.
23:44
Ryan Sean Adams: Because if you have a Turing complete intelligence that can create Turing complete
23:49
Ryan Sean Adams: software, is there anything you can't create once you have that?
23:52
Ryan Sean Adams: Also, like the idea of Morvac's paradox, I guess that sort of implies a certain
23:58
Ryan Sean Adams: complementarianism with humanity.
24:01
Ryan Sean Adams: So if robots can do things that robots can do really well and can't do the things
24:06
Ryan Sean Adams: humans can do well, well, perhaps there's a place for us in this world.
24:09
Ryan Sean Adams: And that's fantastic news. It also maybe implies that humans have kind of scratched
24:14
Ryan Sean Adams: the surface on reasoning potential.
24:17
Ryan Sean Adams: I mean, if we've only had a couple of million years of evolution and we haven't
24:21
Ryan Sean Adams: had the data set to actually get really good at reasoning, it seems like there'd
24:25
Ryan Sean Adams: be a massive amount of upside, unexplored territory,
24:29
Ryan Sean Adams: like so much more intelligence that nature could actually
24:33
Ryan Sean Adams: contain inside of reasoning.
24:35
Ryan Sean Adams: I mean, are these some of the implications of these ideas?
24:38
Dwarkesh: Yeah, I know. I mean, that's a great insight. Another really interesting insight
24:41
Dwarkesh: is that the more variation there
24:44
Dwarkesh: is in a skill in humans, the better and faster that AIs will get at it.
24:50
Dwarkesh: Because coding is the kind of thing where 1% of humans are really good at it.
24:55
Dwarkesh: The rest of us will, if we try to learn it, we'd be okay at it or something, right?
25:00
Dwarkesh: And because evolutionists spend so little time optimizing us,
25:03
Dwarkesh: there's this room for variation where the optimization hasn't happened uniformly
25:07
Dwarkesh: or it hasn't been valuable enough to saturate the human gene pool for this skill.
25:14
Dwarkesh: I think you made an earlier point that I thought was really interesting I wanted
25:16
Dwarkesh: to address. Can you remind me of the first thing you said? Is it the complementarianism? Yes.
25:23
Dwarkesh: So you can take it as a positive future. You can take it as a negative future
25:27
Dwarkesh: in the sense that, well, what is the complementary skills we're providing?
25:30
Dwarkesh: We're good meat robots.
25:33
Ryan Sean Adams: Yeah, the low skilled labor of the situation.
25:35
Dwarkesh: We can do all the thinking and planning. One dark future,
25:39
Dwarkesh: one dark vision of the future is we'll get those meta glasses
25:44
Dwarkesh: and the AI speaking into our ear and it'll tell us to go put this brick over
25:50
Dwarkesh: there so that the next data center couldn't be built because the AI's got the
25:53
Dwarkesh: plan for everything. It's got the better design for the ship and everything.
25:55
Dwarkesh: You just need to move things around for it. And that's what human labor looks
25:58
Dwarkesh: like until robotics is solved.
26:01
Dwarkesh: So yeah, it depends on how you... On the other hand, you'll get paid a lot because
26:04
Dwarkesh: it's worth a lot to move those bricks. We're building AGI here.
26:08
Dwarkesh: But yeah, it depends on how you come out of that question.
26:09
Ryan Sean Adams: Well, there seems to be something to that idea, going back to the idea of the
26:12
Ryan Sean Adams: massive amount of human variation.
26:14
Ryan Sean Adams: I mean, we have just in the past month or so, we have news of meta hiring AI
26:18
Ryan Sean Adams: researchers for $100 million signing bonuses, okay?
26:22
Ryan Sean Adams: What does the average software engineer make versus what does an AI researcher
26:27
Ryan Sean Adams: make at kind of the top of the market, right?
26:29
Ryan Sean Adams: Which has got to imply, obviously there's some things going on with demand and
26:33
Ryan Sean Adams: supply, but also that it does also seem to imply that there's massive variation
26:38
Ryan Sean Adams: in the quality of a software engineer.
26:40
Ryan Sean Adams: And if AIs can get to that quality, well, what does that unlock?
26:44
Ryan Sean Adams: Yeah. So, okay. Yeah. So I guess we have like coding down right now.
26:48
Ryan Sean Adams: Like another question though is like, what can't AIs do today?
26:53
Ryan Sean Adams: And how would you characterize that? Like what are the things they just don't do well?
26:57
Dwarkesh: So I've been interviewing people on my podcast who have very different timelines
27:02
Dwarkesh: from a role to get to AGI. I have had people on who think it's two years away
27:05
Dwarkesh: and some who think it's 20 years away.
27:08
Dwarkesh: And the experience of building AI tools for myself actually has been the most
27:13
Dwarkesh: insight driving or maybe research I've done on the question of when AI is coming.
27:18
Ryan Sean Adams: More than the guest interviews.
27:20
Dwarkesh: Yeah, because you just, I have had, I've probably spent on the order of a hundred
27:25
Dwarkesh: hours trying to build these little tools. The kinds I'm sure you've also tried
27:28
Dwarkesh: to build of like, rewrite auto-generated transcripts for me to make them sound,
27:32
Dwarkesh: the rewritten the way a human would write them.
27:35
Dwarkesh: Find clips for me to tweet out, write essays with me, co-write them passage
27:39
Dwarkesh: by passage, these kinds of things.
27:41
Dwarkesh: And what I found is that it's actually very hard to get human-like labor out
27:45
Dwarkesh: of these models, even for tasks like these, which should be death center in
27:49
Dwarkesh: the repertoire of these models, right?
27:50
Dwarkesh: They're short horizon, they're language in, language out.
27:53
Dwarkesh: They're not contingent on understanding some thing I said like a month ago.
27:58
Dwarkesh: This is just like, this is the task.
28:00
Dwarkesh: And I was thinking about why is it the case that I still haven't been able to
28:04
Dwarkesh: automate these basic language tasks? Why do I still have a human work on these things?
28:09
Dwarkesh: And I think the key reason that you can't automate even these simple tasks is
28:15
Dwarkesh: because the models currently lack the ability to do on the job training.
28:21
Dwarkesh: So if you hire a human for the first six months, for the first three months,
28:24
Dwarkesh: they're not going to be that useful, even if they're very smart,
28:26
Dwarkesh: because they haven't built up the context, they haven't practiced the skills,
28:29
Dwarkesh: they don't understand how the business works.
28:31
Dwarkesh: What makes humans valuable is not that mainly the raw intellect obviously matters,
28:35
Dwarkesh: but it's not mainly that.
28:36
Dwarkesh: It's their ability to interrogate their own failures in this really dynamic,
28:40
Dwarkesh: organic way to pick up small efficiencies and improvements as they practice
28:45
Dwarkesh: the task and to build up this context as they work within a domain.
28:50
Dwarkesh: And so sometimes people wonder, look, if you look at the revenue of OpenAI,
28:54
Dwarkesh: the annual recurring revenue, it's on the order of $10 billion.
28:57
Dwarkesh: Kohl's makes more money than that. McDonald's makes more money than that, right?
29:01
Dwarkesh: So why is it that if they've got AGI, they're, you know, like Fortune 500 isn't
29:07
Dwarkesh: reorganizing their workflows to, you know, use open AI models at every layer of the stack?
29:12
Dwarkesh: My answer, sometimes people say, well, it's because people are too stodgy.
29:15
Dwarkesh: The management of these companies is like not moving fast enough on AI.
29:18
Dwarkesh: That could be part of it. I think mostly it's not that.
29:20
Dwarkesh: I think mostly it genuinely is very hard to get human-like labor out of these
29:23
Dwarkesh: models because you can't.
29:26
Dwarkesh: So you're stuck with the capabilities you get out of the model out of the box.
29:30
Dwarkesh: So they might be five out of 10 at rewriting the transcript for you.
29:33
Dwarkesh: But if you don't like how it turned out, if you have feedback for it,
29:36
Dwarkesh: if you want to keep teaching it over time, once the session ends,
29:41
Dwarkesh: the model, like everything it knows about you has gone away.
29:44
Dwarkesh: You got to restart again. It's like working with an amnesiac employee.
29:47
Dwarkesh: You got to restart again.
29:49
Ryan Sean Adams: Every day is the first day of employment, basically.
29:52
Dwarkesh: Yeah, exactly. It's a groundhog day for them every day or every couple of hours, in fact.
29:56
Dwarkesh: And that makes it very hard for them to be that useful as an employee,
29:59
Dwarkesh: right? They're not really an employee at that point.
30:02
Dwarkesh: This, I think, not only is a key bottleneck to the value of these models,
30:06
Dwarkesh: because human labor is worth a lot, right?
30:09
Dwarkesh: Like $60 trillion in the world is paid to wages every year.
30:12
Dwarkesh: If these model companies are making on the order of $10 billion a year, that's a big way to AGI.
30:18
Dwarkesh: And what explains that gap? What are the bottlenecks? I think a big one is this
30:21
Dwarkesh: continual learning thing.
30:23
Dwarkesh: And I don't see an easy way that that just gets solved within these models.
30:26
Dwarkesh: There's no like, with reasoning, you could say, oh, it's like train it on math
30:29
Dwarkesh: and code problems, and then I'll get the reasoning. And that worked.
30:31
Dwarkesh: I don't think there's something super obvious there for how do you get this
30:35
Dwarkesh: online learning, this on-the-job training working for these models.
30:38
Ryan Sean Adams: Okay, can we talk about that, go a little bit deeper on that concept?
30:41
Ryan Sean Adams: So this is basically one of the concepts you wrote in your recent post.
30:44
Ryan Sean Adams: AI is not right around the corner. Even though you're an AI optimist,
30:48
Ryan Sean Adams: I would say, and overall an AI accelerationist, you You were saying it's not
30:53
Ryan Sean Adams: right around the corner.
30:54
Ryan Sean Adams: You're saying the ability to replace human labor is a ways out.
30:58
Ryan Sean Adams: Not forever out, but I think you said somewhere around 2032,
31:01
Ryan Sean Adams: if you had to guess on when the estimate was.
31:05
Ryan Sean Adams: And the reason you gave is because AIs can't learn on the job,
31:09
Ryan Sean Adams: but it's not clear to me why they can't.
31:10
Ryan Sean Adams: Is it just because the context window isn't large enough?
31:14
Ryan Sean Adams: Is it just because they can't input all of the different data sets and data
31:19
Ryan Sean Adams: points that humans can? Is it because they don't have stateful memory the way a human employee?
31:25
Ryan Sean Adams: Because if it's these things, all of these do seem like solvable problems.
31:28
Ryan Sean Adams: And maybe that's what you're saying. They are solvable problems.
31:30
Ryan Sean Adams: They're just a little bit longer than some people think they are.
31:35
Dwarkesh: I think it's like in some deep sense a solvable problem because eventually we will build AGI.
31:40
Dwarkesh: And to build AGI, we will have had to solve the problem.
31:43
Dwarkesh: My point is that the obvious solutions you might imagine, for example,
31:46
Dwarkesh: expanding the context window or having this
31:49
Dwarkesh: like external memory using systems like rag these
31:54
Dwarkesh: are basically techniques we already have to it's called retrieval augmented
31:57
Dwarkesh: generate anyways these kinds of retrieval augmented generation i
32:00
Dwarkesh: don't think these will suffice and just to put a finer point first of all like
32:04
Dwarkesh: what is the problem the problem is exactly as you say that within the context
32:09
Dwarkesh: window these models actually can learn on the job right so if you talk to it
32:13
Dwarkesh: for long enough it will get much better at understanding your needs and what your exact problem is.
32:17
Dwarkesh: If you're using it for research for your podcast, it will get a sense of like,
32:21
Dwarkesh: oh, they're actually especially curious about these kinds of questions. Let me focus on that.
32:25
Dwarkesh: It's actually very human-like in context, right? The speed at which it learns,
32:28
Dwarkesh: the task of knowledge it picks out.
32:30
Dwarkesh: The problem, of course, is the context length for even the best models only
32:33
Dwarkesh: last a million or two million tokens.
32:36
Dwarkesh: That's at most like an hour of conversation.
32:39
Dwarkesh: Now, then you might say, okay, well, why can't we just solve that by expanding
32:42
Dwarkesh: the context window, right? So context window has been expanding for the last
32:44
Dwarkesh: few years. Why can't we just continue that?
32:47
Ryan Sean Adams: Yeah, like a billion token context window, something like this.
32:50
Dwarkesh: So 2018 is when the transformer came out and the transformer has the attention mechanism.
32:55
Dwarkesh: The attention mechanism is inherently quadratic with the nature of the length
33:00
Dwarkesh: of the sequence, which is to say that if you go from if you double go from 1
33:05
Dwarkesh: million tokens to 2 million tokens,
33:07
Dwarkesh: it actually costs four times as much compute to process that 2 millionth token.
33:12
Dwarkesh: It's not just 2 to as much compute. so it gets super linearly more expensive
33:18
Dwarkesh: as you increase the context length and for the last,
33:23
Dwarkesh: seven years people have been trying to get around this inherent quadratic nature
33:26
Dwarkesh: of attention of course we don't know secretly what the labs are working on but we have frontier,
33:32
Dwarkesh: companies like deep seek which have open source their research and
33:35
Dwarkesh: we can just see how their algorithms work and they found
33:38
Dwarkesh: these constant time modifiers to attention which is
33:41
Dwarkesh: to say that they there's like a it'll still
33:44
Dwarkesh: be quadratic but it'll be like one half times
33:47
Dwarkesh: quadratic but the inherent like super linearness has not
33:49
Dwarkesh: gone away and because of that yeah you might be able to increase it from 1 million
33:53
Dwarkesh: tokens to 2 million tokens by finding another hack like uh make sure experts
33:57
Dwarkesh: just run such things latent attention is another such technique but or kbcash
34:01
Dwarkesh: right there's many other things that have been discovered but people have not
34:05
Dwarkesh: discovered okay how do you get around the fact that if you went to a billion,
34:09
Dwarkesh: it would be a billion squared as expensive in terms of compute to process that token.
34:14
Dwarkesh: And so I don't think you'll just get it by increasing the length of the context window, basically.
34:19
Ryan Sean Adams: That's fascinating. Yeah, I didn't realize that. Okay, so the other reason in
34:23
Ryan Sean Adams: your post that AI is not right around the corner is because it can't do your taxes.
34:27
Ryan Sean Adams: And Dwarkesh, I feel your pain, man. Taxes are just like quite a pain in the ass.
34:33
Ryan Sean Adams: I think you were talking about this from the context of like computer vision,
34:36
Ryan Sean Adams: computer use, that kind of thing.
34:38
Ryan Sean Adams: So, I mean, I've seen demos. I've seen some pretty interesting computer vision
34:42
Ryan Sean Adams: sort of demos that seem to be right around the corner.
34:46
Ryan Sean Adams: But what's the limiter on computer use for an AI?
34:49
Dwarkesh: There was an interesting blog post by this company called Mechanize where they
34:54
Dwarkesh: were explaining why this is such a big problem. And I love the way they phrased it, which is that,
34:58
Dwarkesh: Imagine if you had to train a model in 1980, a large language model in 1980,
35:04
Dwarkesh: and you could use all the compute you wanted in 1980 somehow,
35:08
Dwarkesh: but you didn't have, you were only stuck with the data that was available in
35:14
Dwarkesh: the 1980s, of course, before the internet became a widespread phenomenon.
35:17
Dwarkesh: You couldn't train a modern LLM, even with all the computer in the world,
35:20
Dwarkesh: because the data wasn't available.
35:22
Dwarkesh: And we're in a similar position with respect to computer use,
35:26
Dwarkesh: because there's not this corpus of collected videos, people using computers
35:31
Dwarkesh: to do different things, to access different applications and do white collar work.
35:37
Dwarkesh: Because of that, I think the big challenge has been accumulating this kind of data. off.
35:43
Ryan Sean Adams: And to be clear, when I was saying the use case of like, do my taxes,
35:47
Ryan Sean Adams: you're effectively talking about an AI having the ability to just like,
35:51
Ryan Sean Adams: you know, navigate the files around your computer,
35:54
Ryan Sean Adams: you know, log in to various websites to download your pay stubs or whatever,
35:58
Ryan Sean Adams: and then to go to like TurboTax or something and like input it all into some
36:02
Ryan Sean Adams: software and file it, right?
36:04
Ryan Sean Adams: Just on voice command or something like that. That's basically doing my taxes.
36:08
Dwarkesh: It should be capable of navigating UIs that it's less familiar with or that
36:13
Dwarkesh: come about organically within the context of trying to solve a problem.
36:17
Dwarkesh: So for example, I might have business deductions.
36:20
Dwarkesh: It sees on my bank statement that I've spent $1,000 on Amazon.
36:24
Dwarkesh: It goes logs in my Amazon.
36:25
Dwarkesh: It sees like, oh, he bought a camera. So I think that's probably a business
36:29
Dwarkesh: expense for his podcast.
36:30
Dwarkesh: He bought an Airbnb over a weekend in the cabins of whatever,
36:35
Dwarkesh: in the woods of whatever. That probably wasn't a business expense.
36:38
Dwarkesh: Although maybe, maybe it's, if it's a sort of like a gray, if it's willing to
36:42
Dwarkesh: go in the gray area, maybe I'll talk to you. Yeah, yeah, yeah.
36:45
Ryan Sean Adams: Do the gray area stuff.
36:46
Dwarkesh: I was, I was researching.
36:50
Dwarkesh: But anyway, so that, including all of that, including emailing people for invoices,
36:55
Dwarkesh: and haggling with them, it would be like a sort of week long task to do my taxes, right?
37:01
Dwarkesh: You'd have to, there's a lot of work involved. That's not just like do this
37:04
Dwarkesh: skill, this skill, this skill, but rather of having a sort of like plan of action
37:08
Dwarkesh: and then breaking tasks apart, dealing with new information,
37:11
Dwarkesh: new emails, new messages, consulting with me about questions, et cetera.
37:16
Ryan Sean Adams: Yeah, I mean, to be clear on this use case too, even though your post is titled
37:18
Ryan Sean Adams: like, you know, AI is not right around the corner, you still think this ability
37:22
Ryan Sean Adams: to file your taxes, that's like a 2028 thing, right?
37:27
Ryan Sean Adams: I mean, this is maybe not next year, but it's in a few years.
37:31
Dwarkesh: Right, which is, I think that was sort of, people maybe write too much in The
37:35
Dwarkesh: Decital and then didn't read through the arguments.
37:37
Ryan Sean Adams: I mean, that never happens on the internet. Wow.
37:40
Dwarkesh: First time.
37:42
Dwarkesh: No, I think like I'm arguing against people who are like, you know, this will happen.
37:47
Dwarkesh: AGI is like two years away. I do think the wider world, the markets,
37:53
Dwarkesh: public perception, even people who are somewhat attending to AI,
37:57
Dwarkesh: but aren't in this specific milieu that I'm talking to, are way underpricing AGI.
38:03
Dwarkesh: One reason, one thing I think they're underestimating is not only will we have
38:09
Dwarkesh: millions of extra laborers, millions of extra workers,
38:12
Dwarkesh: potentially billions within the course of the next decade, because then we will
38:16
Dwarkesh: have a potentially, I think like likely we will have AGI within the next decade.
38:20
Dwarkesh: But they'll have these advantages that human workers don't have,
38:23
Dwarkesh: which is that, okay, a single model company, so suppose we solve continual learning, right?
38:27
Dwarkesh: So there, and we saw computer use. So as far as white collar work goes,
38:31
Dwarkesh: that might fundamentally it would be solved.
38:32
Dwarkesh: You can have AIs which can use not just they're not just like a text box where
38:36
Dwarkesh: you put into you ask questions in a chatbot and you get some response out.
38:39
Dwarkesh: It's not that useful to just have a very smart chatbot. You need it to be able
38:42
Dwarkesh: to actually do real work and use real applications.
38:45
Dwarkesh: Suppose you have that solved because it acts like an employee.
38:48
Dwarkesh: It's got continual learning. It's got computer use.
38:49
Dwarkesh: But it has another advantage that humans don't have, which is that copies of
38:53
Dwarkesh: this model are going being deployed all through the economy and it's doing on the job training.
38:57
Dwarkesh: So copies are learning how to be an accountant, how to be a lawyer,
39:00
Dwarkesh: how to be a coder, except because it's an AI and it's digital,
39:04
Dwarkesh: the model itself can amalgamate all this on-the-job training from all these copies.
39:10
Dwarkesh: So what does that mean? Well, it means that even if there's no more software
39:13
Dwarkesh: progress after that point, which is to say that no more algorithms are discovered,
39:17
Dwarkesh: there's not a transformer plus plus that's discovered.
39:20
Dwarkesh: Just from the fact that this model is learning every single skill in the economy,
39:25
Dwarkesh: at least for white-collar work, you might just, based on that alone,
39:30
Dwarkesh: have something that looks like an intelligence explosion.
39:31
Dwarkesh: It would just be a broadly deployed intelligence explosion, but it would functionally
39:35
Dwarkesh: become super intelligent just from having human-level capability of learning on the job.
39:41
Dwarkesh Patel: Yeah, and it creates this mesh network of intelligence that's shared among everyone.
39:45
Dwarkesh Patel: That's a really fascinating thing. So we're going to get there.
39:48
Dwarkesh Patel: We're going to get to AGI. it's going to be incredibly smart.
39:51
Dwarkesh Patel: But what we've shared recently is just kind of this mixed bag where currently
39:54
Dwarkesh Patel: today, it's pretty good at some things, but also not that great at others.
39:57
Dwarkesh Patel: We're hiring humans to do jobs that we think AI should do, but it probably doesn't.
40:00
Dwarkesh Patel: So the question I have for you is, is AI really that smart? Or is it just good
40:04
Dwarkesh Patel: at kind of acing these particular benchmarks that we measure against?
40:08
Dwarkesh Patel: Apple, I mean, famously recently, they had their paper, The Illusion of Thinking,
40:11
Dwarkesh Patel: where it was kind of like, hey, AI is like pretty good up to a point,
40:14
Dwarkesh Patel: but at a certain point, it just falls apart.
40:16
Dwarkesh Patel: And the inference is like, maybe it's not intelligence, maybe it's just good
40:21
Dwarkesh Patel: at guessing. So I guess the question is, is AI really that smart?
40:24
Dwarkesh: It depends on who I'm talking to. I think some people overhype its capabilities.
40:27
Dwarkesh: I think some people are like, oh, it's already AGI, but it's like a little hobbled
40:31
Dwarkesh: little AGI where we're like sort of giving it a concussion every couple of hours
40:35
Dwarkesh: and like it forgets everything.
40:37
Dwarkesh: We're like trapped in a chatbot context. But fundamentally, the thing inside
40:41
Dwarkesh: is like a very smart human.
40:44
Dwarkesh: I disagree with that perspective. So if that's your perspective,
40:46
Dwarkesh: I say like, no, it's not that smart.
40:47
Dwarkesh: Your perspective is just statistical associations. I say definitely smarter.
40:51
Dwarkesh: Like it's like genuinely there's an intelligence there.
40:54
Dwarkesh: And the, so one thing you could say to the person who thinks that it's already
40:58
Dwarkesh: AGI is this, look, if a single human had as much stuff memorized as these models
41:03
Dwarkesh: seem to have memorized, right?
41:04
Dwarkesh: Which is to say that they have all of internet text, everything that human has
41:08
Dwarkesh: written on the internet memorized, they would potentially be discovering all
41:13
Dwarkesh: kinds of connections and discoveries.
41:16
Dwarkesh: They'd notice that this thing which causes a migraine is associated with this kind of deficiency.
41:22
Dwarkesh: So maybe if you take the supplement, your migraines will be cured.
41:25
Dwarkesh: There'd be just this list of just like trivial connections that lead to big
41:28
Dwarkesh: discoveries all through the place.
41:30
Dwarkesh: It's not clear that there's been an unambiguous case of an AI just doing this by itself.
41:37
Dwarkesh: So then why, so that's something potentially to explain, like if they're so
41:40
Dwarkesh: intelligent, why aren't they able to use their disproportionate capabilities,
41:44
Dwarkesh: their unique capabilities to come up with these discoveries?
41:47
Dwarkesh: I don't think there's actually a good answer to that question yet,
41:49
Dwarkesh: except for the fact that they genuinely aren't that creative.
41:51
Dwarkesh: Maybe they're like intelligent in the sense of knowing a lot of things,
41:54
Dwarkesh: but they don't have this fluid intelligence that humans have.
41:57
Dwarkesh: Anyway, so I give you a wish-washy answer because I think some people are underselling
42:00
Dwarkesh: the intelligence. Some people are overselling it.
42:03
Ryan Sean Adams: I recall a tweet lately from Tyler Cowen. I think he was referring to maybe
42:07
Ryan Sean Adams: O3, and he basically said, it feels like AGI.
42:10
Ryan Sean Adams: I don't know if it is AGI or not, but like to me, it feels like AGI.
42:14
Ryan Sean Adams: What do you account for this feeling of like intelligence then
42:18
Dwarkesh: I think this is actually very interesting because it gets to a crux that Tyler
42:22
Dwarkesh: and I have so Tyler and I disagree on two big things one he thinks you know
42:28
Dwarkesh: as he said in the blog post 03 is AGI I don't think it's AGI I think it's,
42:32
Dwarkesh: it's orders of magnitude less valuable or, you know, like many orders of magnitude
42:37
Dwarkesh: less valuable and less useful than an AGI.
42:39
Dwarkesh: That's one thing we disagree on. The other thing we disagree on is he thinks
42:43
Dwarkesh: that once we do get AGI, we'll only see 0.5% increase in the economic growth
42:47
Dwarkesh: rate. This is like what the internet caused, right?
42:49
Dwarkesh: Whereas I think we will see tens of percent increase in economic growth.
42:53
Dwarkesh: Like it will just be the difference between the pre-industrial revolution rate
42:57
Dwarkesh: of growth versus industrial revolution, that magnitude of change again.
43:00
Dwarkesh: And I think these two disagreements are linked because if you do believe we're
43:05
Dwarkesh: already at AGI and you look around the world and you say like,
43:08
Dwarkesh: well, it fundamentally looks the same, you'd be forgiven for thinking like,
43:12
Dwarkesh: oh, there's not that much value in getting to AGI.
43:14
Dwarkesh: Whereas if you are like me and you think like, no, we'll get this broadly at
43:17
Dwarkesh: the minimum, at a very minimum, we'll get a broadly deployed intelligence explosion once we get to AGI,
43:22
Dwarkesh: then you're like, OK, I'm just expecting some sort of singulitarian crazy future
43:26
Dwarkesh: with a robot factories and, you know, solar farms all across the desert and things like that.
43:31
Ryan Sean Adams: Yeah, I mean, it strikes me that your disagreement with Tyler is just based
43:35
Ryan Sean Adams: on the semantic definition of like what AGI actually is.
43:39
Ryan Sean Adams: And Tyler, it sounds like he has kind of a lower threshold for what AGI is,
43:44
Ryan Sean Adams: whereas you have a higher threshold.
43:45
Ryan Sean Adams: Is there like a accepted definition for AGI?
43:48
Dwarkesh: No. One thing that's useful for the purposes of discussions is to say automating
43:54
Dwarkesh: all white collar work because robotics hasn't made as much progress as LLMs
43:59
Dwarkesh: have or computer use has.
44:01
Dwarkesh: So if we just say anything a human can do or maybe 90% of what humans can do
44:06
Dwarkesh: at a desk, an AI can also do, that's potentially a useful definition for at
44:11
Dwarkesh: least getting the cognitive elements relevant to defining AGI.
44:15
Dwarkesh: But yeah, there's not one definition which suits all purposes.
44:18
Ryan Sean Adams: Do we know what's like going on inside of these models, right?
44:23
Ryan Sean Adams: So like, you know, Josh was talking earlier in the conversation about like this
44:26
Ryan Sean Adams: at the base being sort of token prediction, right?
44:29
Ryan Sean Adams: And I guess this starts to raise the question of like, what is intelligence in the first place?
44:35
Ryan Sean Adams: And these AI models, I mean, they seem like they're intelligent,
44:40
Ryan Sean Adams: but do they have a model of the world the way maybe a human might?
44:44
Ryan Sean Adams: Are they sort of babbling or like, is this real reasoning?
44:49
Ryan Sean Adams: And like, what is real reasoning? Do we just judge that based on the results
44:53
Ryan Sean Adams: or is there some way to like peek inside of its head?
44:56
Dwarkesh: I used to have similar questions a couple of years ago. And then,
45:00
Dwarkesh: because honestly, the things they did at the time were like ambiguous.
45:03
Dwarkesh: You could say, oh, it's close enough to something else in this trading data set.
45:07
Dwarkesh: That is just basically copy pasting. It didn't come up with a solution by itself.
45:12
Dwarkesh: But we've gotten to the point where I can come up with a pretty complicated
45:17
Dwarkesh: math problem and it will solve it.
45:19
Dwarkesh: It can be a math problem, like not like, you know, undergrad or high school math problem.
45:24
Dwarkesh: Like the problem we get, the problems the smartest math professors come up with
45:28
Dwarkesh: in order to test International Math Olympiad.
45:31
Dwarkesh: You know, the kids who spend all their life preparing for this,
45:34
Dwarkesh: the geniuses who spend all their life, all their young adulthood preparing to
45:37
Dwarkesh: take these really gnarly math puzzle challenges.
45:40
Dwarkesh: And the model will get these kinds of questions, right? They require all this
45:43
Dwarkesh: abstract creative thinking, this reasoning for hours, the model will get the right.
45:48
Dwarkesh: Okay, so if that's not reasoning, then why is reasoning valuable again?
45:53
Dwarkesh: Like, what exactly was this reasoning supposed to be?
45:56
Dwarkesh: So I think they genuinely are reasoning. I mean, I think there's other capabilities
45:59
Dwarkesh: they lack, which are actually more, in some sense, they seem to us to be more
46:03
Dwarkesh: trivial, but actually much harder to learn. But the reasoning itself, I think, is there.
46:07
Dwarkesh Patel: And the answer to the intelligence question is also kind of clouded,
46:10
Dwarkesh Patel: right? Because we still really don't understand what's going on in an LLM.
46:14
Dwarkesh Patel: Dario from Anthropoc, he recently posted the paper about interpretation.
46:17
Dwarkesh Patel: And can you explain why we don't even really understand what's going on in these
46:21
Dwarkesh Patel: LLMs, even though we're able to make them and yield the results from them? Mmm.
46:27
Dwarkesh Patel: Because it very much still is kind of like a black box. We write some code,
46:30
Dwarkesh Patel: we put some inputs in, and we get something out, but we're not sure what happens in the middle,
46:34
Dwarkesh: Why it's creating this output.
46:36
Dwarkesh Patel: I mean, it's exactly what you're saying.
46:38
Dwarkesh: It's that in other systems we engineer in the world, we have to build it up bottom-ups.
46:44
Dwarkesh: If you build a bridge, you have to understand how every single beam is contributing to the structure.
46:50
Dwarkesh: And we have equations for why the thing will stay standing.
46:54
Dwarkesh: There's no such thing for AI. We didn't build it, more so we grew it.
46:59
Dwarkesh: It's like watering a plant. And a couple thousand years ago,
47:03
Dwarkesh: they were doing agriculture, but they didn't know why.
47:08
Dwarkesh: Why do plants grow? How do they collect energy from sunlight? All these things.
47:13
Dwarkesh: And I think we're in a substantially similar position with respect to intelligence,
47:18
Dwarkesh: with respect to consciousness, with respect to all these other interesting questions
47:23
Dwarkesh: about how minds work, which is in some sense really cool because there's this
47:27
Dwarkesh: huge intellectual horizon that's become not only available, but accessible to investigation.
47:33
Dwarkesh: In another sense, it's scary because we know that minds can suffer.
47:37
Dwarkesh: We know that minds have moral worth and we're creating minds and we have no
47:42
Dwarkesh: understanding of what's happening in these minds.
47:44
Dwarkesh: Is a process of gradient descent a painful process?
47:48
Dwarkesh: We don't know, but we're doing a lot of it.
47:51
Dwarkesh: So hopefully we'll learn more. But yeah, I think we're in a similar position
47:54
Dwarkesh: to some farmer in Uruk in 3500 BC.
47:57
Josh Kale: Wow.
47:58
Ryan Sean Adams: And I mean, the potential, the idea that minds can suffer, minds have some moral
48:03
Ryan Sean Adams: worth, and also minds have some free will.
48:06
Ryan Sean Adams: They have some sort of autonomy, or maybe at least a desire to have autonomy.
48:11
Ryan Sean Adams: I mean, this brings us to kind of this sticky subject of alignment and AI safety
48:15
Ryan Sean Adams: and how we go about controlling the intelligence that we're creating,
48:20
Ryan Sean Adams: if even that's what we should be doing, controlling it. And we'll get to that in a minute.
48:24
Ryan Sean Adams: But I want to start with maybe the headlines here a little bit.
48:28
Ryan Sean Adams: So headline just this morning, latest OpenAI models sabotaged a shutdown mechanism
48:34
Ryan Sean Adams: despite commands to the contrary.
48:36
Ryan Sean Adams: OpenAI's O1 model attempted to copy itself to external servers after being threatened
48:41
Ryan Sean Adams: with shutdown that denied the action when discovered.
48:44
Ryan Sean Adams: I've read a number of papers for this. Of course, mainstream media has these
48:48
Ryan Sean Adams: types of headlines almost on a weekly basis now, and it's starting to get to daily.
48:53
Ryan Sean Adams: But there does seem to be some evidence that AIs lie to us,
48:58
Ryan Sean Adams: If that's even the right term, in order to pursue goals, goals like self-preservation,
49:03
Ryan Sean Adams: goals like replication, even deep-seated values that we might train into them,
49:08
Ryan Sean Adams: sort of a constitution type of value.
49:11
Ryan Sean Adams: They seek to preserve these values, which maybe that's a good thing,
49:15
Ryan Sean Adams: or maybe it's not a good thing if we don't actually want them to interpret the values in a certain way.
49:21
Ryan Sean Adams: Some of these headlines that we're seeing now, To you, with your kind of corpus
49:25
Ryan Sean Adams: of knowledge and all of the interviews and discovery you've done on your side,
49:29
Ryan Sean Adams: is this like media sensationalism or is this like alarming?
49:33
Ryan Sean Adams: And if it's alarming, how concerned should we be about this?
49:37
Dwarkesh: I think on net, it's quite alarming. I do think that some of these results have
49:42
Dwarkesh: been sort of cherry picked.
49:44
Dwarkesh: Or if you look into the code, what's happened is basically the researchers have
49:47
Dwarkesh: said, hey, pretend to be a bad person.
49:50
Dwarkesh: Wow, AI is being a bad person. Isn't that crazy?
49:53
Dwarkesh: But the system prompt is just like hey do this bad thing right now i personally
49:57
Dwarkesh: but i have also seen other results which are not of this quality i mean the
50:02
Dwarkesh: the clearest example so backing up,
50:05
Dwarkesh: what is the reason to think this will be a bigger problem in the future than
50:08
Dwarkesh: it is now because we all interact with these systems and they're actually like
50:13
Dwarkesh: quite moral or aligned right like you can talk to a chatbot and you like ask
50:17
Dwarkesh: it to how should you deal with some crisis where there's a correct answer,
50:22
Dwarkesh: you know, like it will tell you not to be violent. It'll give you reasonable advice.
50:26
Dwarkesh: It seems to have good values. So it's worth noticing this, right?
50:29
Dwarkesh: And being happy about it.
50:31
Dwarkesh: The concern is that we're moving from a regime where we've trained them on human
50:36
Dwarkesh: language, which implicitly has human morals and the way, you know,
50:41
Dwarkesh: normal people think about values implicit in it.
50:44
Dwarkesh: Plus this RLHF process we did to a regime where we're mostly spending compute
50:50
Dwarkesh: on just having them answer problems yes or no or correct or not rather just like.
50:58
Dwarkesh: And pass all the unit tests, get the right answer on this math problem.
51:02
Dwarkesh: And this has no guardrails intrinsically in terms of what is allowed to do,
51:09
Dwarkesh: what is the proper moral way to do something.
51:11
Dwarkesh: I think that can be a loaded term, but here's a more concrete example.
51:15
Dwarkesh: One problem we're running into with these coding agents more and more,
51:18
Dwarkesh: and this has nothing to do with these abstract concerns about alignment,
51:21
Dwarkesh: but more so just like how do we get economic value out of these models,
51:24
Dwarkesh: is that Claude or Gemini will, instead of writing code such that it passes the unit tests,
51:32
Dwarkesh: it will often just delete the unit tests so that the code just passes by default.
51:37
Dwarkesh: Now, why would it do that? Well, it's learned in the process.
51:41
Dwarkesh: It was trained on the goal during training of you must pass all unit tests.
51:45
Dwarkesh: And probably within some environment in which it was trained,
51:48
Dwarkesh: it was able to just get away.
51:49
Dwarkesh: Like there wasn't designed well enough. And so it found this like little hole
51:53
Dwarkesh: where it could just like delete the file that had the unit test or rewrite them
51:55
Dwarkesh: so that it always said, you know, equals true, then pass.
51:59
Dwarkesh: And right now we can discover these even without, even though we can discover
52:03
Dwarkesh: these, you know, it's still past, there's still been enough hacks like this,
52:06
Dwarkesh: such that the model is like becoming more and more hacky like that.
52:10
Dwarkesh: In the future, we're going to be training models in ways that we is beyond our
52:14
Dwarkesh: ability to even understand, certainly beyond everybody's ability to understand.
52:18
Dwarkesh: There may be a few people who might be able to see just the way that right now,
52:21
Dwarkesh: if you came up with a new math proof for some open problem in mathematics,
52:24
Dwarkesh: there will be only be a few people in the world who will be able to evaluate that math proof.
52:28
Dwarkesh: We'll be in a similar position with respect to all of the things that these
52:31
Dwarkesh: models are being trained on at the frontier, especially math and code,
52:34
Dwarkesh: because humans were big dum-dums with respect to this reasoning stuff.
52:38
Dwarkesh: And so there's a sort of like first principles reason to expect that this new
52:41
Dwarkesh: modality of training will be less amenable to the kinds of supervision that
52:46
Dwarkesh: was grounded within the pre-training corpus.
52:49
Ryan Sean Adams: I don't know that everyone has kind of an intuition or an idea why it doesn't
52:54
Ryan Sean Adams: work to just say like, so if we don't want our AI models to lie to us,
52:59
Ryan Sean Adams: why can't we just tell them not to lie?
53:01
Ryan Sean Adams: Why can't we just put that as part of their core constitution?
53:05
Ryan Sean Adams: If we don't want our AI models to be sycophants, why can't we just say,
53:10
Ryan Sean Adams: hey, if I tell you I want the truth, not to flatter me, just give me the straight up truth.
53:16
Ryan Sean Adams: Why is this even difficult to do?
53:18
Dwarkesh: Well, fundamentally, it comes down to how we train them. And we don't know how
53:22
Dwarkesh: to train them in a way that does not reward lying or sycophancy.
53:26
Dwarkesh: In fact, the problem is OpenAI, they explained why their recent model of theirs
53:30
Dwarkesh: was they had to take down was just sycophantic.
53:33
Dwarkesh: And the reason was just that they rolled out, did it in the A-B test and the
53:37
Dwarkesh: version, the test that was more sycophantic was just preferred by users more.
53:42
Dwarkesh: Sometimes you prefer the lie.
53:44
Dwarkesh: Yeah, so that's, if that's what's preferred in training, you know,
53:47
Dwarkesh: Or, for example, in the context of lying, if we've just built RL environments
53:52
Dwarkesh: in which we're training these models, where they're going to be more successful if they lie, right?
53:59
Dwarkesh: So if they delete the unit tests and then tell you, I passed this program and
54:06
Dwarkesh: all the unit tests have succeeded, it's like lying to you, basically.
54:09
Dwarkesh: And if that's what is rewarded in the process of gradient descent,
54:12
Dwarkesh: then it's not surprising that the model you interact with will just have this
54:17
Dwarkesh: drive to lie if it gets it closer to its goal.
54:20
Dwarkesh: And I would just expect this to keep happening unless we can solve this fundamental
54:24
Dwarkesh: problem that comes about in training.
54:26
Dwarkesh Patel: So you mentioned how like ChatGPT had a version that was sycophantic,
54:30
Dwarkesh Patel: and that's because users actually wanted that.
54:32
Dwarkesh Patel: Who is in control? Who decides the actual alignment of these models?
54:36
Dwarkesh Patel: Because users are saying one thing, and then they deploy it,
54:38
Dwarkesh Patel: and then it turns out that's not actually what people want.
54:41
Dwarkesh Patel: How do you kind of form consensus around this alignment or these alignment principles?
54:47
Dwarkesh: Right now, obviously, it's the labs who decided this, right?
54:49
Dwarkesh: And the safety teams of the labs.
54:51
Dwarkesh: And I guess the question you could ask is then who should decide these? Because this will be...
54:56
Dwarkesh Patel: Assuming the trajectory, yeah. So we keep going to get more powerful.
54:59
Dwarkesh: Because this will be the key modality that all of us use to get,
55:03
Dwarkesh: not only get work done, but even like, I think at some point,
55:06
Dwarkesh: a lot of people's best friends will be AIs, at least functionally in the sense
55:10
Dwarkesh: of who do they spend the most amount of time talking to. It might already be AIs.
55:14
Dwarkesh: This will be the key layer in your business that you're using to get work done
55:20
Dwarkesh: so this process of training which shapes their personality who gets to control
55:25
Dwarkesh: it I mean it will be the laughs functionally,
55:30
Dwarkesh: But maybe you mean, like, who should control it, right? I honestly don't know.
55:34
Dwarkesh: I mean, I don't know if there's a better alternative to the labs.
55:36
Dwarkesh Patel: Yeah, I would assume, like, there's some sort of social consensus,
55:39
Dwarkesh Patel: right? Similar to how we have in America, the Constitution.
55:41
Dwarkesh Patel: There's, like, this general form of consensus that gets formed around how we
55:44
Dwarkesh Patel: should treat these models as they become as powerful as we think they probably will be.
55:47
Dwarkesh: Honestly, I don't have, I don't know if anybody has a good answer about how
55:49
Dwarkesh: you do this process. I think we lucked out, we just, like, really lucked out with the Constitution.
55:55
Dwarkesh: It also wasn't a democratic process which resulted in the constitution,
55:58
Dwarkesh: even though it instituted a Republican form of government.
56:00
Dwarkesh: It was just delegates from each state. They haggled it out over the course of a few months.
56:05
Dwarkesh: Maybe that's what happens with AI. But is there some process which feels both
56:10
Dwarkesh: fair and which will result in actually a good constitution for these AIs?
56:15
Dwarkesh: It's not obvious to me that, I mean, nothing comes up to the top of my head.
56:19
Dwarkesh: Like, oh, this, you know, do rank choice voting or something.
56:22
Dwarkesh Patel: Yeah, so I was going to ask, is there any, I mean, having spoken to everyone
56:24
Dwarkesh Patel: who you've spoken to is there any alignment path which looks most promising which
56:27
Dwarkesh: Feels the.
56:28
Dwarkesh Patel: Most comforting and exciting to you
56:30
Dwarkesh: I i think alignment in the sense of you
56:33
Dwarkesh: know and eventually we'll have these super intelligent systems what do we do
56:36
Dwarkesh: about that i think the the approach that i think is most promising is less about
56:44
Dwarkesh: finding some holy grail some you know giga brain solution some equation which
56:49
Dwarkesh: solves the whole puzzle and more like one.
56:53
Dwarkesh: Having this Swiss cheese approach where, look, we kind of have gotten really good at jailbreaks.
57:01
Dwarkesh: I'm sure you've heard a lot about jailbreaks over the last few years.
57:03
Dwarkesh: It's actually much harder to jailbreak these models because,
57:06
Dwarkesh: you know, people try to whack at these things in different ways.
57:10
Dwarkesh: Model developers just like patched these obvious ways to do jailbreaks.
57:14
Dwarkesh: The model also got smarter. So it's better able to understand when somebody
57:18
Dwarkesh: is trying to jailbreak into it.
57:20
Dwarkesh: That, I think, is one approach. Another is, I think, competition.
57:24
Dwarkesh: I think the scary version of the future is where you have this dynamic where
57:27
Dwarkesh: a single model and its copies are controlling the entire economy.
57:31
Dwarkesh: When politicians want to understand what policies to pass, they're only talking
57:35
Dwarkesh: to copies of a single model.
57:36
Dwarkesh: If there's multiple different AI companies who are at the frontier,
57:40
Dwarkesh: who have competing services, and whose models can monitor each other, right?
57:44
Dwarkesh: So Claude may care about its own copies being successful in the world and it
57:50
Dwarkesh: might be able to willing to lie on their behalf, even if you ask one copy to supervise another.
57:53
Dwarkesh: I think you get some advantage from a copy of OpenAI's model monitoring a copy
57:58
Dwarkesh: of DeepSeek's model, which actually brings us back to the Constitution, right?
58:01
Dwarkesh: One of the most brilliant things in the Constitution is the system of checks and balances.
58:04
Dwarkesh: So some combination of the Swiss cheese approach to model development and training
58:09
Dwarkesh: and alignment, where you're careful, if you notice this kind of reward hacking,
58:13
Dwarkesh: you do your best to solve it.
58:14
Dwarkesh: You try to keep as much of the models thinking in human language rather than
58:19
Dwarkesh: letting it think in AI thought in this latent space thinking.
58:23
Dwarkesh: And the other part of it is just having normal market competition between these
58:27
Dwarkesh: companies so that you can use them to check each other and no one company or
58:31
Dwarkesh: no one AI is dominating the economy or advisory roles for governments.
58:41
Ryan Sean Adams: I really like this like bundle of ideas that you sort of put together in that
58:45
Ryan Sean Adams: because like, I think a lot of the, you know, AI safety conversation is always
58:50
Ryan Sean Adams: couched in terms of control.
58:52
Ryan Sean Adams: Like we have to control the thing that is the way. And I always get a little
58:56
Ryan Sean Adams: worried when I hear like terms like control.
58:59
Ryan Sean Adams: And it reminds me of a blog post I think you put out, which I'm hopeful you continue to write on.
59:05
Ryan Sean Adams: I think you said it was going to be like one of a series, which is this idea
59:08
Ryan Sean Adams: of like classical liberal AGI. And we were talking about themes like balance of power.
59:13
Ryan Sean Adams: Let's have Claude check in with ChatGPT and monitor it.
59:17
Josh Kale: When you have themes like transparency as well,
59:19
Ryan Sean Adams: That feels a bit more, you know, classically liberal coded than maybe some of
59:25
Ryan Sean Adams: the other approaches that I've heard.
59:27
Ryan Sean Adams: And you wrote this in the post, which I thought was kind of,
59:30
Ryan Sean Adams: it just sparked my interest because I'm not sure where you're going to go next
59:33
Ryan Sean Adams: with this, but you said the most likely way this happens,
59:37
Ryan Sean Adams: that is AIs have a stake in humanity's future, is if it's in the AI's best interest
59:42
Ryan Sean Adams: to operate within our existing laws and norms.
59:45
Ryan Sean Adams: You know, this whole idea that like, hey, the way to get true AI alignment is
59:49
Ryan Sean Adams: to make it easy, make it the path of least resistance for AI to basically partner with humans.
59:56
Ryan Sean Adams: It's almost this idea if the aliens
59:59
Ryan Sean Adams: landed or something, we would create treaties with the aliens, right?
1:00:03
Ryan Sean Adams: We would want them to adopt our norms. We would want to initiate trade with them.
1:00:08
Ryan Sean Adams: Our first response shouldn't be, let's try to dominate and control them.
1:00:13
Ryan Sean Adams: Maybe it should be, let's try to work with them. Let's try to collaborate.
1:00:16
Ryan Sean Adams: Let's try to open up trade.
1:00:18
Ryan Sean Adams: What's your idea here? And like, are you planning to write further posts about this?
1:00:22
Dwarkesh: Yeah, I want to. It's just such a hard topic to think about that,
1:00:25
Dwarkesh: you know, something always comes up.
1:00:26
Dwarkesh: But the fundamental point I was making is, look, in the long run,
1:00:32
Dwarkesh: if AIs are, you know, human labor is going to be obsolete because of these inherent
1:00:37
Dwarkesh: advantages that digital minds will have and robotics will eventually be solved.
1:00:41
Dwarkesh: So our only leverage on the future will no longer come from our labor.
1:00:50
Dwarkesh: It will come from our legal and economic control over the society that AIs will
1:00:58
Dwarkesh: be participating in, right? So, you know, AIs might make the economy explode
1:01:03
Dwarkesh: in the sense of grow a lot.
1:01:04
Dwarkesh: And for humans to benefit from that, it would have to be the case that AIs still
1:01:08
Dwarkesh: respect your equity in the S&P 500 companies that you bought, right?
1:01:13
Dwarkesh: Or for the AIs to follow your laws, which say that you can't do violence onto
1:01:18
Dwarkesh: humans and you got to respect humans' properties.
1:01:21
Josh Kale: It would have to be the case that AIs are actually bought into our
1:01:25
Dwarkesh: System of government, into our laws and norms. And for that to happen,
1:01:30
Dwarkesh: the way that likely happens is if it's just like the default path for the AIs
1:01:37
Dwarkesh: as they're getting smarter and they're developing their own systems of enforcement
1:01:41
Dwarkesh: and laws to just participate in human laws and governments.
1:01:46
Dwarkesh: And the metaphor I use here is right now you pay half your paycheck in taxes,
1:01:53
Dwarkesh: probably half of your taxes in some way just go to senior citizens, right?
1:01:59
Dwarkesh: Medicare and Social Security and other programs like this.
1:02:04
Dwarkesh: And it's not because you're in some deep moral sense aligned with senior citizens.
1:02:09
Dwarkesh: It's not like you're spending all your time thinking about like,
1:02:11
Dwarkesh: my main priority in life is to earn money for senior citizens.
1:02:15
Dwarkesh: It's just that you're not going to overthrow the government to get out of paying this tax. And so...
1:02:22
Ryan Sean Adams: Also, I happen to like my grandmother. She's fantastic. You know,
1:02:25
Ryan Sean Adams: it's those reasons too. But yeah.
1:02:26
Dwarkesh: So that's why you give money to your grandmother directly. But like,
1:02:29
Dwarkesh: why are you giving money to some retiree in Illinois? Yes.
1:02:33
Josh Kale: Yes.
1:02:34
Dwarkesh: Yeah, it's like, okay, you could say it's like, sometimes people,
1:02:37
Dwarkesh: some people are trying to that post by saying like, oh no, I like deeply care
1:02:39
Dwarkesh: about the system of social welfare.
1:02:41
Dwarkesh: I'm just like, okay, maybe you do, but I don't think like the average person
1:02:45
Dwarkesh: is giving away hundreds of thousands of dollars a year, tens of thousands of
1:02:47
Dwarkesh: dollars a year to like some random stranger they don't know,
1:02:50
Dwarkesh: who's like, who's not like especially in need of charity, right?
1:02:53
Dwarkesh: Like most senior citizens have some savings.
1:02:55
Dwarkesh: It's just, it's just because this is a law and you like, you give it to them
1:02:59
Dwarkesh: or you'll get, go to jail.
1:03:01
Dwarkesh: But fundamentally, if the tax was like 99%, you would, like,
1:03:05
Dwarkesh: you would, maybe you wouldn't overthrow the government. You'd just,
1:03:07
Dwarkesh: like, leave the jurisdiction.
1:03:08
Dwarkesh: You'd, like, emigrate somewhere. And AIs can potentially also do this,
1:03:12
Dwarkesh: right? There's more than one country.
1:03:14
Dwarkesh: They could, like, there's countries which would be more AI forward.
1:03:16
Dwarkesh: And it would be a bad situation to end up in where...
1:03:21
Dwarkesh: All this explosion in AI technology is happening in the country,
1:03:24
Dwarkesh: which is doing the least amount to protect humans',
1:03:28
Dwarkesh: rights and to provide some sort of monetary compensation to humans once their
1:03:36
Dwarkesh: labor is no longer valuable.
1:03:38
Dwarkesh: So our labor could be worth nothing, but because of how much richer the world
1:03:42
Dwarkesh: is after AI, you have these billions of extra researchers, workers, etc.
1:03:47
Dwarkesh: It could still be trivial to have individual humans have the equivalent of millions,
1:03:54
Dwarkesh: even billions of dollars worth of wealth. In fact, it might literally be invaluable
1:03:59
Dwarkesh: amounts of wealth in the following sense. So here's an interesting thought experiment.
1:04:02
Dwarkesh: Imagine you have this choice. You can go back to the year 1500,
1:04:07
Dwarkesh: but you know, of course, the year 1500 kind of sucks.
1:04:09
Dwarkesh: You have no antibiotics, no TV, no running water. But here's how I'll make it up to you.
1:04:15
Dwarkesh: I can give you any amount of money, but you can only use that amount of money in the year 1500.
1:04:20
Dwarkesh: And you'll go back with these sacks of gold. How much money would I have to
1:04:24
Dwarkesh: give you that you can use in the year 1500 to make you go back? And plausibly.
1:04:27
Dwarkesh Patel: The answer is
1:04:27
Dwarkesh: There's no amount of money you would rather have in the year 1500 than just
1:04:30
Dwarkesh: have a normal life today.
1:04:31
Dwarkesh: And we could be in a similar position with regards to the future where there's
1:04:36
Dwarkesh: all these different, I mean, you'll have much better health,
1:04:39
Dwarkesh: like physical health, mental health, longevity.
1:04:42
Dwarkesh: That's just like the thing we can contemplate now. But people in 1500 couldn't
1:04:45
Dwarkesh: contemplate the kinds of quality of life advances we would have 500 years later,
1:04:49
Dwarkesh: right? So anyways, this is all to say that this could be our future for humans,
1:04:54
Dwarkesh: even if our labor isn't worth anything.
1:04:57
Dwarkesh: But it does require us to have AIs that choose to participate or in some way
1:05:05
Dwarkesh: incentivize to participate in some system which we have leverage over.
1:05:12
Ryan Sean Adams: Yeah, I find this just such a fast, I'm hopeful we do some more exploration
1:05:16
Ryan Sean Adams: around this because I think what you're calling for is basically like,
1:05:19
Ryan Sean Adams: what you would be saying is invite them into our property rights system.
1:05:22
Ryan Sean Adams: I mean, there are some that are calling in order to control AI,
1:05:25
Ryan Sean Adams: they have great power, but they don't necessarily have capabilities.
1:05:28
Ryan Sean Adams: So we shouldn't allow AI to hold money or to have property.
1:05:31
Ryan Sean Adams: I think you would say, no, actually, the path forward to alignment is allow
1:05:36
Ryan Sean Adams: AI to have some vested interest in our property rights system and some stake
1:05:42
Ryan Sean Adams: in our governance, potentially, right?
1:05:44
Ryan Sean Adams: The ability to vote, almost like a constitution for AIs.
1:05:47
Ryan Sean Adams: I'm not sure how this would work, but it's a fascinating thought experiment.
1:05:53
Dwarkesh: I will say one thing I think this could end disastrously if we give them a stake
1:06:00
Dwarkesh: in their property system but we let them play,
1:06:04
Dwarkesh: us off each other. So if you think about, there's many cases in history where
1:06:09
Dwarkesh: the British, initially, the East India Trading Company was genuinely a trading
1:06:13
Dwarkesh: company that operated in India.
1:06:15
Dwarkesh: And it was able to play off, you know, it was like doing trade with different,
1:06:18
Dwarkesh: different, you know, provinces in India, there was no single powerful leader.
1:06:23
Dwarkesh: And by playing, you know, by doing trade, one of them, leveraging one of their
1:06:28
Dwarkesh: armies, etc., they were able to conquer the continent. Similar thing could happen to human society.
1:06:32
Dwarkesh: The way to avoid such an outcome at a high level is involves us playing the
1:06:38
Dwarkesh: AIs off each other instead, right?
1:06:40
Dwarkesh: So this is why I think competition is such a big part of the puzzle,
1:06:45
Dwarkesh: having different AIs monitor each other, having this bargaining position where
1:06:49
Dwarkesh: there's not just one company that's at the frontier.
1:06:51
Dwarkesh: Another example here is if you think about how the Spanish conquered all these
1:06:55
Dwarkesh: new world empires, it's actually so crazy that a couple hundred conquistaDwars
1:06:58
Dwarkesh: would show up and conquer a nation of 10 million people, the Incas,
1:07:03
Dwarkesh: Aztecs. And why were they able to do this?
1:07:05
Dwarkesh: Well, one of the reasons is the Spanish were able to learn from each of their
1:07:11
Dwarkesh: previous expeditions, whereas the Native Americans were not.
1:07:15
Dwarkesh: So Cortez learned from how Cuba was subjugated when he conquered the Aztecs.
1:07:22
Dwarkesh: Pizarro was able to learn from how Cortez conquered the Aztecs when he conquered the Incas.
1:07:25
Dwarkesh: The Incas didn't even know the Aztecs existed. So eventually there was this
1:07:30
Dwarkesh: uprising against Pizarro and Manco Inca led an insurgency where they actually
1:07:36
Dwarkesh: did figure out how to fight horses,
1:07:37
Dwarkesh: how to fight people, you know, people in armor on horses, don't fight them on
1:07:42
Dwarkesh: flat terrain, throw rocks down at them, et cetera.
1:07:44
Dwarkesh: But by this point, it was too late. If they knew this going into the battle,
1:07:48
Dwarkesh: the initial battle, they might've been able to fend off because,
1:07:51
Dwarkesh: you know, just as the conquistaDwars only arrived at a few hundred soldiers,
1:07:54
Dwarkesh: we're going to the age of AI with a tremendous amount of leverage.
1:07:58
Dwarkesh: We literally control all the stuff, right?
1:08:01
Dwarkesh: But we just need to lock in our advantage. We just need to be in a position
1:08:04
Dwarkesh: where, you know, they're not going to be able to play us off each other.
1:08:08
Dwarkesh: We're going to be able to learn what their weaknesses are.
1:08:11
Dwarkesh: And this is why I think one good idea, for example, would be that,
1:08:14
Dwarkesh: look, DeepSeek is a Chinese company.
1:08:17
Dwarkesh: It would be good if, suppose DeepSeek did something naughty,
1:08:21
Dwarkesh: like the kinds of experiments we're talking about right now where it hacks the
1:08:24
Dwarkesh: unit tests or so forth. I mean, eventually these things will really matter.
1:08:27
Dwarkesh: Like Xi Jinping is listening to AIs because they're so smart and they're so capable.
1:08:32
Dwarkesh: If China notices that their AIs are doing something bad, or they notice a failed
1:08:37
Dwarkesh: coup attempt, for example,
1:08:38
Dwarkesh: it's very important that they tell us And we tell them if we notice something
1:08:43
Dwarkesh: like that on our end, it would be like the Aztecs and Incas talking to each
1:08:46
Dwarkesh: other about like, you know, this is what happens.
1:08:49
Dwarkesh: This is how you fight. This is how you fight horses.
1:08:51
Dwarkesh: This is the kind of tactics and deals they try to make with you. Don't trust them, etc.
1:08:56
Dwarkesh: It would require cooperation on humans' part to have this sort of red telephone.
1:08:59
Dwarkesh: So during the Cold War, there was this red telephone between America and the
1:09:03
Dwarkesh: Soviet Union after the human missile crisis, where just to make sure there's
1:09:06
Dwarkesh: no misunderstandings, they're like, okay, if we think something's going on,
1:09:08
Dwarkesh: let's just hop on the call.
1:09:10
Dwarkesh: I think we should have a similar policy with respect to these kinds of initial
1:09:15
Dwarkesh: warning signs we'll get from AI so that we can learn from each other.
1:09:19
Dwarkesh Patel: Awesome. Okay, so now that we've described this artificial gender intelligence,
1:09:22
Dwarkesh Patel: I want to talk about how we actually get there. How do we build it?
1:09:25
Dwarkesh Patel: And a lot of this we've been discussing kind of takes place in this world of
1:09:27
Dwarkesh Patel: bits. But you have this great chapter in the book called Inputs,
1:09:30
Dwarkesh Patel: which discusses the physical world around us, where you can't just write a few strings of code.
1:09:35
Dwarkesh Patel: You actually have to go and move some dirt and you have to ship servers places
1:09:38
Dwarkesh Patel: and you need to power it and you need physical energy from meat space.
1:09:41
Dwarkesh Patel: And you kind of describe these limiting factors where we have compute,
1:09:46
Dwarkesh Patel: we have energy, we have data.
1:09:47
Dwarkesh Patel: What I'm curious to know is, do we have enough of this now? or is there a clear
1:09:52
Dwarkesh Patel: path to get there in order to build the AGI?
1:09:54
Dwarkesh Patel: Basically, what needs to happen in order for us to get to this place that you're describing?
1:09:57
Dwarkesh: We only have a couple more years left of this scaling,
1:10:02
Dwarkesh: this exponential scaling before we're hitting these inherent roadblocks of energy
1:10:09
Dwarkesh: and our ability to manufacture ships, which means that if scaling is going to
1:10:13
Dwarkesh: work to deliver us AGI, it has to work by 2028.
1:10:17
Dwarkesh: Otherwise, we're just left with mostly algorithmic progress,
1:10:19
Dwarkesh: But even within algorithmic progress, the sort of low-hanging fruit in this
1:10:23
Dwarkesh: deep learning paradigm is getting more and more plucked.
1:10:26
Dwarkesh: So then the odds per year of getting to AGI diminish a lot, right?
1:10:31
Dwarkesh: So there is this weird, funny thing happening right now where we either discover
1:10:36
Dwarkesh: AGI within the next few years,
1:10:40
Dwarkesh: or the yearly probability craters, and then we might be looking at decades of
1:10:44
Dwarkesh: further research that's required in terms of algorithms to get to AGI.
1:10:47
Dwarkesh: I am of the opinion that some algorithmic progress is necessarily needed because
1:10:51
Dwarkesh: there's no easy way to solve continual learning just by making the context length
1:10:55
Dwarkesh: bigger or just by doing RL.
1:10:57
Dwarkesh: That being said, I just think the progress so far has been so remarkable that,
1:11:01
Dwarkesh: you know, 2032 is very close.
1:11:04
Dwarkesh: My time has to be slightly longer than that, but I think it's extremely plausible
1:11:08
Dwarkesh: that we're going to see a broadly deployed intelligence explosion within the next 10 years.
1:11:13
Dwarkesh Patel: And one of these key inputs is energy, right? a lot, I actually heard it mentioned
1:11:18
Dwarkesh Patel: on your podcast, is the United States relative to China on this particular place
1:11:24
Dwarkesh Patel: of energy, where China is adding, what is the stat?
1:11:26
Dwarkesh Patel: I think it's one United States worth of energy every 18 months.
1:11:30
Dwarkesh Patel: And their plan is to go from three to eight terawatts of power versus the United
1:11:34
Dwarkesh Patel: States, one to two terawatts of power by 2030.
1:11:36
Dwarkesh Patel: So given that context of that one resource alone, is China better equipped to
1:11:41
Dwarkesh Patel: get to that place versus with the United States?
1:11:44
Dwarkesh: So right now, America has a big advantage in terms of chips.
1:11:50
Dwarkesh: China doesn't have the ability to manufacture leading-edge semiconductors,
1:11:53
Dwarkesh: and these are the chips that go into...
1:11:56
Dwarkesh: You need these dyes in order to have the kinds of AI chips to...
1:12:01
Dwarkesh: You need millions of them in order to have a frontier AI system.
1:12:08
Dwarkesh: Eventually, China will catch up in this arena as well, right?
1:12:10
Dwarkesh: Their technology will catch up. So the export controls will keep us ahead in
1:12:14
Dwarkesh: this category for 5, 10 years.
1:12:16
Dwarkesh: But if we're looking in the world where timelines are long, which is to say
1:12:19
Dwarkesh: that AGI isn't just right around the corner, they will have this overwhelming
1:12:23
Dwarkesh: energy advantage and they'll have caught up in chips.
1:12:27
Dwarkesh: So then the question is like, why wouldn't they win at that point?
1:12:30
Dwarkesh: So the longer you think we're away from AGI, the more it looks like China's game to lose.
1:12:37
Dwarkesh: I mean, if you look in the nitty gritty, I think it's more about having centralized
1:12:42
Dwarkesh: sources of power because you need to train the AI in one place.
1:12:47
Dwarkesh: This might be changing with RL, but it's very important to have a single site
1:12:51
Dwarkesh: which has a gigawatt, two gigawatts more power.
1:12:54
Dwarkesh: And if we ramped up natural gas, you know, you can get generators and natural
1:13:00
Dwarkesh: gas and maybe it's possible to do a last ditch effort, even if our overall energy
1:13:04
Dwarkesh: as a country is lower than China's. The question is whether we will have the
1:13:07
Dwarkesh: political will to do that.
1:13:08
Dwarkesh: I think people are sort of underestimating how much of a backlash there will be against AI.
1:13:14
Dwarkesh: The government needs to make proactive efforts in order to make sure that America
1:13:18
Dwarkesh: stays at the leading edge in AI from zoning of data centers to how copyright
1:13:25
Dwarkesh: is handled for data for these models.
1:13:27
Dwarkesh: And if we mess up, if it becomes too hard to develop in America,
1:13:32
Dwarkesh: I think it would genuinely be China's game to lose.
1:13:34
Ryan Sean Adams: And do you think this narrative is right, that whoever wins the AGI war,
1:13:38
Ryan Sean Adams: kind of like whoever gets to AGI first, just basically wins the 21st century? Is it that simple?
1:13:43
Dwarkesh: I don't think it's just a matter of training the frontier system.
1:13:46
Dwarkesh: I think people underestimate how important it is to have the compute available to run these systems.
1:13:51
Dwarkesh: Because eventually once you get to AGI, just think of it like a person.
1:13:55
Dwarkesh: And what matters then is how many people you have.
1:13:59
Dwarkesh: I mean, it actually is the main thing that matters today as well,
1:14:02
Dwarkesh: right? Like, why could China take over Taiwan if it wanted to?
1:14:05
Dwarkesh: And if it didn't have America, you know, America, it didn't think America would intervene.
1:14:09
Dwarkesh: But because Taiwan has 20 million people or on the order of 20 million people
1:14:13
Dwarkesh: and China has 1.4 billion people.
1:14:17
Dwarkesh: You could have a future where if China has way more compute than us,
1:14:21
Dwarkesh: but equivalent levels of AI, it would be like the relationship between China and Taiwan.
1:14:27
Dwarkesh: Their population is functionally so much higher. This just means more research,
1:14:31
Dwarkesh: more factories, more development, more ideas.
1:14:35
Dwarkesh: So this inference capacity, this capacity to deploy AIs will actually probably
1:14:41
Dwarkesh: be the thing that determines who wins the 21st century.
1:14:44
Ryan Sean Adams: So this is like the scaling law applied to, I guess, nation state geopolitics, right?
1:14:50
Ryan Sean Adams: And it's back to compute plus data wins.
1:14:54
Ryan Sean Adams: If compute plus data wins superintelligence, compute plus data also wins geopolitics.
1:15:00
Dwarkesh: Yep. And the thing to be worried about is that China, speaking of compute plus
1:15:04
Dwarkesh: data, China also has a lot more data on the real world, right?
1:15:08
Dwarkesh: If you've got entire megalopolises filled with factories where you're already
1:15:13
Dwarkesh: deploying robots and different production systems which use automation,
1:15:19
Dwarkesh: you have in-house this process knowledge you're building up which the AIs can
1:15:24
Dwarkesh: then feed on and accelerate.
1:15:27
Dwarkesh: That equivalent level of data we don't have in America.
1:15:31
Dwarkesh: So this could be a period in which those technological advantages or those advantages
1:15:37
Dwarkesh: in the physical world manufacturing could rapidly compound for China.
1:15:41
Dwarkesh: And also, I mean, their big advantage as a civilization and society,
1:15:44
Dwarkesh: at least in recent decades, has been that they can do big industrial projects fast and efficiently.
1:15:50
Dwarkesh: That's not the first thing you think of when you think of America.
1:15:53
Dwarkesh: And AGI is a huge industrial, high CapEx, Manhattan project, right?
1:16:00
Dwarkesh: And this is the kind of thing that China excels at and we don't.
1:16:03
Dwarkesh: So, you know, I think it's like a much tougher race than people anticipate.
1:16:07
Ryan Sean Adams: So what's all this going to do for the world? So once we get to the point of AGI,
1:16:11
Ryan Sean Adams: we've talked about GDP and your estimate is less on the Tyler Cowen kind of
1:16:15
Ryan Sean Adams: half a percent per year and more on, I guess, the Satya Nadella from Microsoft,
1:16:21
Ryan Sean Adams: what does he say, 7% to 8% once we get to AGI.
1:16:24
Ryan Sean Adams: What about unemployment? Does this cause mass, I guess, job loss across the
1:16:30
Ryan Sean Adams: economy or do people adopt?
1:16:33
Ryan Sean Adams: What's your take here? Yeah, what are you seeing?
1:16:36
Dwarkesh: Yeah, I mean, definitely will cause job loss. I think people who don't,
1:16:39
Dwarkesh: I think a lot of AI leaders try to gloss over that or something. And like, I mean.
1:16:43
Josh Kale: What do you mean?
1:16:43
Dwarkesh: Like, what does AGI mean if it doesn't cause job loss, right?
1:16:45
Dwarkesh: If it does what a human does and.
1:16:47
Josh Kale: It does it
1:16:48
Dwarkesh: Cheaper and better and faster, like why would that not cause job loss?
1:16:52
Dwarkesh: The positive vision here is just that it creates so much wealth,
1:16:56
Dwarkesh: so much abundance, that we can still give people a much better standard of living
1:17:00
Dwarkesh: than even the wealthiest people today, even if they themselves don't have a job.
1:17:06
Dwarkesh: The future I worry about is one where instead of creating some sort of UBI that
1:17:12
Dwarkesh: will get exponentially bigger as society gets wealthier,
1:17:16
Dwarkesh: we try to create these sorts of guild-like protection rackets where if the coders got unemployed,
1:17:26
Dwarkesh: then we're going to make these bullshit jobs just for the coders and this is
1:17:32
Dwarkesh: how we give them a redistribution.
1:17:34
Dwarkesh: Or we try to expand Medicaid for AI, but it's not allowed to procure all of
1:17:42
Dwarkesh: these advanced medicines and cures that AI is coming up with,
1:17:46
Dwarkesh: rather than just giving people, you know, maybe lump sums of money or something.
1:17:50
Dwarkesh: So I am worried about the future where instead of sharing this abundance and
1:17:54
Dwarkesh: just embracing it, we just have these protection rackets that maybe let a few
1:18:00
Dwarkesh: people have access to the abundance of AI.
1:18:03
Dwarkesh: So maybe like if you sue AI, if you sue the right company at the right time,
1:18:06
Dwarkesh: you'll get a trillion dollars, but everybody else is stuck with nothing.
1:18:09
Dwarkesh: I want to avoid that future and just be honest about what's coming and make
1:18:15
Dwarkesh: programs that are simple and acknowledge how fast things will change and are
1:18:21
Dwarkesh: forward looking rather than trying to turn what already exists into something
1:18:25
Dwarkesh: amenable to the displacement that AI will create.
1:18:29
Ryan Sean Adams: That argument reminds me of, I don't know if you read the essay recently came
1:18:32
Ryan Sean Adams: out called The Intelligence Curse. Did you read that?
1:18:34
Ryan Sean Adams: It was basically the idea of applying kind of the nation state resource curse
1:18:40
Ryan Sean Adams: to the idea of intelligence.
1:18:42
Ryan Sean Adams: So like nation states that are very high in natural resources,
1:18:45
Ryan Sean Adams: they just have a propensity.
1:18:48
Ryan Sean Adams: I mean, an example is kind of like a Middle Eastern state with lots of oil reserves, right?
1:18:53
Ryan Sean Adams: They have this rich source of a commodity type of abundance.
1:18:58
Ryan Sean Adams: They need their people less. And so they don't invest in citizens' rights.
1:19:02
Ryan Sean Adams: They don't invest in social programs.
1:19:04
Ryan Sean Adams: The authors of the intelligence curse were saying that there's a similar type
1:19:08
Ryan Sean Adams: of curse that could happen once intelligence gets very cheap,
1:19:11
Ryan Sean Adams: which is basically like the nation state doesn't need humans anymore.
1:19:15
Ryan Sean Adams: And those at the top, the rich, wealthy corporations, they don't need workers anymore.
1:19:19
Ryan Sean Adams: So we get kind of locked in this almost feudal state where, you know,
1:19:23
Ryan Sean Adams: everyone has the property that their grandparents had and there's no meritocracy
1:19:27
Ryan Sean Adams: and sort of the nation states don't reinvest in citizens.
1:19:31
Ryan Sean Adams: Almost some similar ideas to your idea that like, you know, that the robots
1:19:35
Ryan Sean Adams: might want us just, or sorry, the AIs might just want us for our meat hands
1:19:39
Ryan Sean Adams: because they don't have the robotics technology on a temporary basis.
1:19:43
Ryan Sean Adams: What do you think of this type of like future? Is this possible?
1:19:46
Dwarkesh: I agree that that is like definitely more of a concern given that humans will
1:19:49
Dwarkesh: not be directly involved in the economic output that will be generated in the CIA civilization.
1:19:54
Dwarkesh: The hopeful story you can tell is that a lot of these Middle Eastern resource,
1:19:59
Dwarkesh: you know, Dutch disease is another term that's used,
1:20:01
Dwarkesh: countries, the problem is that they're not democracies, so that this wealth
1:20:06
Dwarkesh: can just be, the system of government
1:20:08
Dwarkesh: just lets whoever's in power extract that wealth for themselves.
1:20:11
Dwarkesh: Whereas there are countries like Norway, for example, which also have abundant
1:20:15
Dwarkesh: resources, who are able to use those resources to have further social welfare
1:20:21
Dwarkesh: programs, to build sovereign wealth funds for their citizens,
1:20:24
Dwarkesh: to invest in their future.
1:20:26
Dwarkesh: We are going into, at least some countries, America included,
1:20:29
Dwarkesh: will go into the age of AI as a democracy.
1:20:33
Dwarkesh: And so we, of course, will lose our economic leverage, but the average person
1:20:38
Dwarkesh: still has their political leverage.
1:20:39
Dwarkesh: Now, over the long run, yeah, if we didn't do anything for a while,
1:20:44
Dwarkesh: I'm guessing the political system would also change.
1:20:46
Dwarkesh: So then the key is to lock in or turn our current, well, it's not just political leverage, right?
1:20:52
Dwarkesh: We also have property rights. So like we own a lot of stuff that AI wants, factories,
1:20:56
Dwarkesh: sources of data, et cetera, is to use the combination of political and economic
1:21:01
Dwarkesh: leverage to lock in benefits for us for the long term, but beyond our the lifespan
1:21:07
Dwarkesh: of our economic usefulness.
1:21:10
Dwarkesh: And I'm more optimistic for us than I am for these Middle Eastern countries
1:21:13
Dwarkesh: that started off poor and also with no democratic representation.
1:21:17
Ryan Sean Adams: What do you think the future of like ChachipD is going to be?
1:21:20
Ryan Sean Adams: If we just extrapolate maybe one version update forward to ChatGPT 5,
1:21:25
Ryan Sean Adams: do you think the trend line of the scaling law will essentially hold for ChatGPT 5?
1:21:30
Ryan Sean Adams: I mean, another way to ask that question is, do you feel like it'll feel like
1:21:33
Ryan Sean Adams: the difference between maybe a BlackBerry and an iPhone?
1:21:37
Ryan Sean Adams: Or will it feel more like the difference between, say, the iPhone 10 and the
1:21:41
Ryan Sean Adams: iPhone 11, which is just like incremental progress, not a big breakthrough,
1:21:45
Ryan Sean Adams: not an order of magnitude change? Yeah.
1:21:50
Dwarkesh: I think it'll be somewhere in between but I don't think it'll feel like a humongous
1:21:53
Dwarkesh: breakthrough even though I think it's in a remarkable pace of change because
1:21:58
Dwarkesh: the nature of scaling is that sometimes people talk about it as an exponential process,
1:22:03
Dwarkesh: Exponential usually refers to like it going like this.
1:22:07
Dwarkesh: So having like a sort of J curve aspect to it, where the incremental input is
1:22:10
Dwarkesh: leading to super linear amounts of output, in this case, intelligence and value,
1:22:16
Dwarkesh: where it's actually more like a sideways J.
1:22:20
Dwarkesh: The exponential means the exponential and the scaling laws is that you need
1:22:23
Dwarkesh: exponentially more inputs to get marginal increases in usefulness or loss or intelligence.
1:22:29
Dwarkesh: So and that's what we've been seeing, right? I think you initially see like some cool demo.
1:22:34
Dwarkesh: So as you mentioned, you see some cool computer use demo, which comes at the
1:22:38
Dwarkesh: beginning of this hyper exponential, I'm sorry, of this sort of plateauing curve.
1:22:44
Dwarkesh: And then it's still an incredibly powerful curve and we're still early in it.
1:22:48
Dwarkesh: But the next demo will be just adding on to making this existing capability
1:22:54
Dwarkesh: more reliable, applicable for more skills.
1:22:57
Dwarkesh: The other interesting incentive in this industry is that because there's so
1:23:01
Dwarkesh: much competition between the labs, you are incentivized to release a capability.
1:23:06
Dwarkesh: As soon as it's even marginally viable or marginally cool so you can raise more
1:23:11
Dwarkesh: funding or make more money off of it.
1:23:13
Dwarkesh: You're not incentivized to just like sit on it until you perfected it,
1:23:17
Dwarkesh: which is why I don't expect like tomorrow OpenAI will just come out with like,
1:23:20
Dwarkesh: we've solved continual learning, guys, and we didn't tell you about it.
1:23:22
Dwarkesh: We're working on it for five years.
1:23:24
Dwarkesh: If they had like even an inkling of a solution, they'd want to release it ASAP
1:23:27
Dwarkesh: so they can raise a $600 billion round and then spend more money on compute.
1:23:32
Dwarkesh: So yeah, I do think it'll seem marginal. But again, marginal in the context of seven years to AGI.
1:23:38
Dwarkesh: So zoom out long enough and a crazy amount of progress is happening.
1:23:42
Dwarkesh: Month to month, I think people overhype how significant any one new release is. So I guess the answer.
1:23:48
Dwarkesh Patel: To when we will get AGI very much depends on that scaling trend holding.
1:23:52
Dwarkesh Patel: Your estimate in the book for AGI was 60% chance by 2040.
1:23:57
Dwarkesh Patel: So I'm curious, what guess or what idea had the most influence on this estimate?
1:24:00
Dwarkesh Patel: What made you end up on 60% of 2040?
1:24:04
Dwarkesh Patel: Because a lot of timelines are much faster than that.
1:24:06
Dwarkesh: It's sort of reasoning about the things they currently still lack,
1:24:10
Dwarkesh: the capabilities they still lack, and what stands in the way.
1:24:13
Dwarkesh: And just generally an intuition that things often take longer to happen than
1:24:16
Dwarkesh: you might think. Progress tends to slow down.
1:24:19
Dwarkesh: Also, it's the case that, look, you might have heard the phrase that we keep
1:24:23
Dwarkesh: shifting the goalposts on AI, right?
1:24:26
Dwarkesh: So they can do the things which skeptics were saying they couldn't ever do already.
1:24:30
Dwarkesh: But now they say AI is still a dead end because problem X, Y,
1:24:34
Dwarkesh: Z, which will be solved next year.
1:24:36
Dwarkesh: Now, there's a way in which this is frustrating, but there's another way in which there's some,
1:24:43
Dwarkesh: It is the case that we didn't get to AGI, even though we have passed the Turing
1:24:46
Dwarkesh: test and we have models that are incredibly smart and can reason.
1:24:49
Dwarkesh: So it is accurate to say that, oh, we were wrong and there is some missing thing
1:24:53
Dwarkesh: that we need to keep identifying about what is still lacking to the path of AGI.
1:24:57
Dwarkesh: Like it does make sense to shift the goalposts. And I think we might discover
1:25:00
Dwarkesh: once continual learning is solved or once extended computer use is solved,
1:25:04
Dwarkesh: that there were other aspects of human intelligence, which we take for granted
1:25:08
Dwarkesh: in this Moravax paradox sense, but which are actually quite crucial to making
1:25:12
Dwarkesh: us economically valuable.
1:25:14
Ryan Sean Adams: Part of the reason we wanted to do this, Dwarkesh, is because we both are enjoyers
1:25:18
Ryan Sean Adams: of your podcast. It's just fantastic.
1:25:20
Ryan Sean Adams: And you talk to all of the, you know, those that are on the forefront of AI
1:25:25
Ryan Sean Adams: development, leading it in all sorts of ways.
1:25:28
Ryan Sean Adams: And one of the things I wanted to do with reading your book,
1:25:30
Ryan Sean Adams: and obviously I'm always asking myself when I'm listening to your podcast is
1:25:34
Ryan Sean Adams: like, what does Dwarkesh think personally?
1:25:36
Ryan Sean Adams: And I feel like I sort of got that insight maybe toward the end of your book,
1:25:39
Ryan Sean Adams: like, you know, in the summary section, where you think like there's a 60% probability
1:25:44
Ryan Sean Adams: of AGI by 2040, which puts you more in the moderate camp, right?
1:25:48
Ryan Sean Adams: You're not a conservative, but you're not like an accelerationist.
1:25:50
Ryan Sean Adams: So you're moderate there.
1:25:51
Ryan Sean Adams: And you also said you think more than likely AI will be net beneficial to humanity.
1:25:57
Ryan Sean Adams: So you're more optimist than Doomer. So we've got a moderate optimist.
1:26:01
Ryan Sean Adams: And you also think this, and this is very interesting, There's no going back.
1:26:05
Ryan Sean Adams: So you're somewhat of an AI determinist. And I think the reason you state for
1:26:10
Ryan Sean Adams: not, you're like, there's no going back.
1:26:12
Ryan Sean Adams: It struck me, there's this line in your book. It seems that the universe is
1:26:16
Ryan Sean Adams: structured such that throwing large amounts of compute at the right distribution of data gets you AI.
1:26:21
Ryan Sean Adams: And the secret is out. If the scaling picture is roughly correct,
1:26:24
Ryan Sean Adams: it's hard to imagine AGI not being developed this century, even if some actors
1:26:28
Ryan Sean Adams: hold back or are held back.
1:26:31
Ryan Sean Adams: That to me is an AI determinist position. Do you think that's fair?
1:26:34
Ryan Sean Adams: Moderate with respect to accelerationism, optimistic with respect to its potential,
1:26:39
Ryan Sean Adams: and also determinist, like there's nothing else we can do. We can't go backwards here.
1:26:43
Dwarkesh: I'm determinist in the sense that I think if AI is technologically possible, it is inevitable.
1:26:48
Dwarkesh: I think sometimes people are optimistic about this idea that we as a world will sort of,
1:26:53
Dwarkesh: I collectively decide not to build AI. And I just don't think that's a plausible outcome.
1:26:58
Dwarkesh: The local incentives for any actor to build AI are so high that it will happen.
1:27:02
Dwarkesh: But I'm also an optimist in the sense that, look, I'm not naive.
1:27:05
Dwarkesh: I've listed out all the way, like what happened to the Aztecs and Incas was
1:27:08
Dwarkesh: terrible. And I've explained how that could be similar to what AIs could do
1:27:10
Dwarkesh: to us and what we need to do to avoid that outcome.
1:27:13
Dwarkesh: But I am optimistic in the sense that the world of the future fundamentally
1:27:18
Dwarkesh: will have so much abundance that there's all these,
1:27:22
Dwarkesh: that alone is a prima facie reason to think that there must be some way of cooperating
1:27:28
Dwarkesh: that is mutually beneficial.
1:27:30
Dwarkesh: If we're going to be thousands, millions of times wealthier,
1:27:33
Dwarkesh: is there really no way that humans are better off or can we can find a way for
1:27:37
Dwarkesh: humans to become better off as a result of this transformation?
1:27:40
Dwarkesh: So yeah, I think you've put your finger on it.
1:27:43
Ryan Sean Adams: So this scaling book, of course, goes through the history of AI scaling.
1:27:46
Ryan Sean Adams: I think everyone should should pick it up to get the full chronology,
1:27:49
Ryan Sean Adams: but also sort of captures where we are in the midst of this story is like, we're not done yet.
1:27:55
Ryan Sean Adams: And I'm wondering how you feel at this moment of time.
1:27:58
Ryan Sean Adams: So I don't know if we're halfway through, if we're a quarter way through,
1:28:03
Ryan Sean Adams: if we're one tenth of the way through, but we're certainly not finished the path to AI scaling.
1:28:07
Ryan Sean Adams: How do you feel like in this moment in 2025?
1:28:11
Ryan Sean Adams: I mean, is all of this terrifying? Is it exciting?
1:28:15
Ryan Sean Adams: Is it exhilarating?
1:28:17
Ryan Sean Adams: What's the emotion that you feel?
1:28:20
Dwarkesh: Maybe I feel a little sort of hurried. I personally feel like there's a lot
1:28:24
Dwarkesh: of things I want to do in the meantime,
1:28:27
Dwarkesh: including what my mission is with the podcast, which is to, and I know it's
1:28:33
Dwarkesh: your mission as well, is to improve the discourse around these topics,
1:28:38
Dwarkesh: to not necessarily push for a specific agenda, but make sure that when people are making decisions,
1:28:42
Dwarkesh: they're as well-informed as possible, They have as much strategic awareness
1:28:47
Dwarkesh: and depth of understanding around how AI works, what it could do in the future as possible.
1:28:55
Dwarkesh: And, but in many ways, I feel like I still haven't emotionally priced in the future I'm expecting.
1:29:00
Dwarkesh: In this one very basic sense, I think that there's a very good chance that I
1:29:06
Dwarkesh: live beyond 200 years of age.
1:29:08
Dwarkesh: I have not changed anything about my life with regards to that knowledge, right?
1:29:12
Dwarkesh: I'm not like, when I'm picking partners, I'm not like, oh, this is the person,
1:29:17
Dwarkesh: now that I think I'm going to live for 200, you know, like hundreds of years.
1:29:20
Ryan Sean Adams: Yeah.
1:29:23
Dwarkesh: Well, you know, ideally I would pick a partner that would, ideally you pick
1:29:27
Dwarkesh: somebody who would be, that would be true regardless.
1:29:30
Dwarkesh: But you see what I'm saying, right? There's like, the fact that I expect my
1:29:34
Dwarkesh: personal life, the world around me, the lives of the people I care about,
1:29:37
Dwarkesh: humanity in general to be so different has, it just like doesn't emotionally resonate as much as,
1:29:45
Dwarkesh: I, my intellectual thoughts and my emotional landscape aren't in the same place.
1:29:50
Dwarkesh: I wonder if it's similar for you guys.
1:29:51
Ryan Sean Adams: Yeah, I totally agree. I don't think I've priced that in. Also,
1:29:54
Ryan Sean Adams: there's like non-zero chance that Eliezer Yudkowsky is right, Dworkesh.
1:29:58
Ryan Sean Adams: Do you know? And so that scenario, I just, I can't bring myself to emotionally price in.
1:30:03
Ryan Sean Adams: So I veer towards the optimism side as well.
1:30:07
Ryan Sean Adams: Dworkesh, this has been fantastic. Thank you so much for all you do on the podcast.
1:30:12
Ryan Sean Adams: I have to ask a question for our crypto audience as well, which is,
1:30:15
Ryan Sean Adams: when are you going to do a crypto podcast on Dwarkech?
1:30:19
Dwarkesh: I already did. It was with one Sam Bigman-Fried.
1:30:22
Ryan Sean Adams: Oh my God.
1:30:24
Dwarkesh: Oh man.
1:30:26
Ryan Sean Adams: We got to get you a new guest. We got to get you someone else to revisit the top best.
1:30:29
Dwarkesh: Don't look that one up. It's Ben Omen. Don't look that one up.
1:30:31
Dwarkesh: I think in retrospect. You know what? We'll do another one.
1:30:36
Ryan Sean Adams: Fantastic. I'll ask you
1:30:37
Dwarkesh: Guys for some recommendations. That'd be great. Dwarkech, thank you so much.
1:30:40
Dwarkesh: But I've been following your stuff for a while, for I think many years.
1:30:43
Dwarkesh: So it's great to finally meet. and this was a lot of fun.
1:30:46
Ryan Sean Adams: Appreciate it. It was great. Thanks a lot.