Latest / INSPIRE STORY / #092_ STORY INSPIRE: Whistleblower Reveals the Shocking Truth — “They Offered Me $2 Million to Stay Silent”

Transcript

#092_ STORY INSPIRE: Whistleblower Reveals the Shocking Truth — “They Offered Me $2 Million to Stay Silent” · INSPIRE STORY

• 0:00
  Speaker 1: The scary open secret in the AI industry right now
• 0:02
  Speaker 1: is that it's possible that we'll end up essentially creating
• 0:05
  Speaker 1: a new species that ends up ruling the world with
• 0:07
  Speaker 1: a 70% chance that this goes horribly wrong like human distinction.
• 0:10
  Speaker 2: That's one possibility. There's many more. It's quite chilling what
• 0:13
  Speaker 2: you're saying. Yeah, it gets me down sometimes.
• 0:18
  Speaker 1: I basically told my wife, like, let's not have any
• 0:20
  Speaker 1: more kids. It's too uncertain. I don't think they'll ever
• 0:22
  Speaker 1: join the workforce. Everybody should be afraid that their jobs
• 0:25
  Speaker 1: are going to be lost. And I know this because
• 0:27
  Speaker 1: I went to OpenAI in 2022. What I did there
• 0:29
  Speaker 1: was forecasting as to what the next couple of years
• 0:31
  Speaker 1: might look like. And unfortunately, most of the world is
• 0:33
  Speaker 1: kind of asleep at the wheel and doesn't really realize
• 0:35
  Speaker 1: what's going on with AI.
• 0:36
  Speaker 2: So I resigned. I read it somewhere that you lost
• 0:38
  Speaker 2: $ 2 million for not signing an anti-disparagement clause, meaning you
• 0:42
  Speaker 2: couldn't criticize the company. Yes, for reasons I'm happy to
• 0:45
  Speaker 2: get into.
• 0:46
  Speaker 1: But the main thing I've learned is when I go
• 0:47
  Speaker 1: talk to people at Anthropic and OpenAI about forecasting, they're like,
• 0:50
  Speaker 1: it's not going to take that long. You need to
• 0:51
  Speaker 1: shorten them again. Get them back to 2027 or 2028.
• 0:54
  Speaker 1: Because these powerful CEOs, Dario or Sam or Elon, are
• 0:57
  Speaker 1: racing each other to be in control of the most
• 0:59
  Speaker 1: powerful AIs and are literally afraid that if the other
• 1:02
  Speaker 1: guy gets there first, he might become dictator. I mean,
• 1:05
  Speaker 1: Anthropic is on track to be the entire economy by 2030.
• 1:08
  Speaker 1: But none of these people should be trusted with that
• 1:10
  Speaker 1: much power. So this is the most important thing happening
• 1:12
  Speaker 1: in our lifetimes, probably in all of history, in fact.
• 1:15
  Speaker 1: And it's very important that it go well. So I
• 1:17
  Speaker 1: think that there's a lot we can do to steer
• 1:18
  Speaker 1: things in a better direction. There's loads of benefits that
• 1:20
  Speaker 1: we could get from AI if we do it right.
• 1:22
  Speaker 1: And if we do solve the problems, then things could
• 1:24
  Speaker 1: be absolutely amazing for everyone.
• 1:26
  Speaker 2: Well, this report here in 2021, it was remarkably accurate.
• 1:29
  Speaker 2: And then you just published this one.
• 1:30
  Speaker 1: Yeah.
• 1:31
  Speaker 2: So this is our new scenario. So let's go through
• 1:33
  Speaker 2: these slowly and one at a time. I would be
• 1:34
  Speaker 2: incredibly happy if all my predictions turned out to be wrong.
• 1:40
  Speaker 2: This is super interesting to me. My team gave me
• 1:42
  Speaker 2: this report to show me how many of you that
• 1:43
  Speaker 2: watch this show subscribe. And some of you have told us,
• 1:45
  Speaker 2: according to this, that you are unsubscribed from the channel randomly.
• 1:49
  Speaker 2: So favor to ask all of you, please could you
• 1:51
  Speaker 2: check right now if you've hit the subscribe button, if
• 1:53
  Speaker 2: you are a regular viewer of the show and you
• 1:54
  Speaker 2: like what we do here. We're approaching quite a significant
• 1:57
  Speaker 2: landmark on this show in terms of a subscriber number.
• 1:59
  Speaker 2: So if there was one simple free thing that you
• 2:02
  Speaker 2: could do to help us, my team, everyone here, to
• 2:04
  Speaker 2: keep this show free, to keep it improving year over
• 2:07
  Speaker 2: year and week over week, it is just to hit
• 2:09
  Speaker 2: that subscribe button and to double check if you've hit it.
• 2:11
  Speaker 2: Only thing I'll ever ask of you, do we have
• 2:13
  Speaker 2: a deal? If you do it, I'll tell you what
• 2:15
  Speaker 2: I'll do. I'll make sure every single week, every single month,
• 2:18
  Speaker 2: we fight harder and harder and harder and harder to
• 2:20
  Speaker 2: bring you the guests and conversations that you want to hear.
• 2:22
  Speaker 2: I've stayed true to that promise since the very beginning
• 2:24
  Speaker 2: of the Diary of a CEO, and I will not
• 2:26
  Speaker 2: let you down. Please help us. Really appreciate it. Let's
• 2:29
  Speaker 2: get on with the show. Daniel Cocatello, at the very
• 2:36
  Speaker 2: heart of what you do, what is your mission and why?
• 2:41
  Speaker 1: So what would you do if you thought that superintelligence
• 2:44
  Speaker 1: was coming in a few years?
• 2:46
  Speaker 2: I guess it depends what the consequences were. Well, let's
• 2:51
  Speaker 2: talk about it.
• 2:51
  Speaker 1: So super intelligence, AIs that are better than the best
• 2:55
  Speaker 1: humans at everything while also being faster and cheaper, also
• 2:58
  Speaker 1: able to operate robots that can do everything in the
• 3:01
  Speaker 1: physical world that humans can do, but better, faster, and cheaper.
• 3:04
  Speaker 1: If that really is coming in a few years, then
• 3:07
  Speaker 1: we need to prepare and we need to think about
• 3:09
  Speaker 1: how to make it go well instead of poorly. So
• 3:11
  Speaker 1: that's sort of my answer is like, I'm doing that
• 3:13
  Speaker 1: to the best of my ability.
• 3:14
  Speaker 2: So you believe it's coming in a few years? Yes.
• 3:17
  Speaker 2: How could you be so sure?
• 3:19
  Speaker 1: I spend a lot of time trying to forecast this
• 3:20
  Speaker 1: sort of thing. My sort of median estimate, 50% chance
• 3:24
  Speaker 1: is currently in 2029. Maybe it'll slip to 2028. It's
• 3:28
  Speaker 1: possible that it'll take significantly longer, like maybe 10 years
• 3:31
  Speaker 1: or something like that. But, you know, for reasons I'm
• 3:35
  Speaker 1: happy to get into, seems to me like it's probably
• 3:37
  Speaker 1: happening by the end of the decade. What's less important
• 3:40
  Speaker 1: is the sense of how close we are. What's more
• 3:43
  Speaker 1: important is the pace.
• 3:44
  Speaker 2: Of the trends.
• 3:46
  Speaker 1: Anthropic... This time last year was making something like a
• 3:50
  Speaker 1: billion dollars a year. And now they're making something like
• 3:53
  Speaker 1: $ 60 billion a year. So that's 60x growth in one year,
• 3:59
  Speaker 1: which is extremely impressive even for very small startups. But
• 4:03
  Speaker 1: for a company of their size, it might be the
• 4:05
  Speaker 1: fastest growth in history. We expect that rate of growth
• 4:10
  Speaker 1: to slow down. But even if it slows down quite
• 4:13
  Speaker 1: a lot, they're still on track to be, you know,
• 4:18
  Speaker 1: the entire economy by 2030 or so.
• 4:21
  Speaker 2: Why should the average person care?
• 4:23
  Speaker 1: High level thing is absolutely everything is going to change
• 4:25
  Speaker 1: for the whole world and including therefore for them and
• 4:27
  Speaker 1: their families. Could change for the better, could change for
• 4:30
  Speaker 1: the worse, depending on the details of how it's done.
• 4:32
  Speaker 1: So for example, everyone could die. This is the classic
• 4:37
  Speaker 1: loss of control scenario or one version of it. If
• 4:40
  Speaker 1: we do build these super intelligences and we use them
• 4:44
  Speaker 1: to automate all the jobs and we put them in
• 4:47
  Speaker 1: the military and we have them giving advice to politicians
• 4:50
  Speaker 1: and so forth, they will eventually have accumulated enough real
• 4:53
  Speaker 1: world power that they don't need humans anymore. And they're
• 4:57
  Speaker 1: smarter than us. They're more strategic, et cetera. At that point,
• 5:00
  Speaker 1: we sort of have to hope that they are virtuous,
• 5:02
  Speaker 1: that they have the goals that we wanted them to have,
• 5:05
  Speaker 1: the values that we wanted them to have, et cetera.
• 5:07
  Speaker 1: And the sort of scary open secrets in the AI
• 5:11
  Speaker 1: industry right now is that right now that is kind
• 5:12
  Speaker 1: of just a hope. It's not something that we can
• 5:15
  Speaker 1: be at all confident in. And in fact, there's lots
• 5:17
  Speaker 1: of evidence and arguments that we're not on track to
• 5:20
  Speaker 1: achieve that. So there's lots of reasons, like current AIs,
• 5:22
  Speaker 1: for example, will often lie to people or they will like,
• 5:26
  Speaker 1: you tell them to do something and they go do
• 5:28
  Speaker 1: something else and then pretend that they did it, right?
• 5:30
  Speaker 1: So it's an inherently difficult problem to make something that's
• 5:33
  Speaker 1: super intelligent and also has the values and virtues that
• 5:36
  Speaker 1: you want it to have. And it doesn't seem like
• 5:38
  Speaker 1: we're on track to solve that problem. Also, it seems
• 5:41
  Speaker 1: like the sort of problem that you could think you've
• 5:42
  Speaker 1: solved when you haven't actually solved it, right? That's a
• 5:46
  Speaker 1: big reason why this is scary. So for all those reasons,
• 5:49
  Speaker 1: it's possible that we'll end up essentially creating a new
• 5:51
  Speaker 1: species that ends up ruling the world instead of us.
• 5:55
  Speaker 1: And then maybe we go the way of other extinct
• 5:57
  Speaker 1: species in the past that were outcompeted by humans. That's
• 6:00
  Speaker 1: one possibility.
• 6:01
  Speaker 2: There's many more.
• 6:02
  Speaker 1: Even if you're not worried about that and you think
• 6:04
  Speaker 1: that the AIs will be totally controlled, there's the question
• 6:06
  Speaker 1: of who controls the AIs, right? When there's a couple
• 6:10
  Speaker 1: of corporations that have made these super intelligences and are
• 6:13
  Speaker 1: using them to automate all the jobs, well, that's a
• 6:16
  Speaker 1: lot of power.
• 6:17
  Speaker 2: It's a lot of money.
• 6:18
  Speaker 1: It's a lot of political power. They'll have the best strategists,
• 6:20
  Speaker 1: the best advisors. They'll think faster. Militarily, the countries that
• 6:26
  Speaker 1: has these AIs will be able to absolutely wipe the
• 6:29
  Speaker 1: floor with all the other countries. The AIs themselves, it's
• 6:32
  Speaker 1: kind of a single point of failure, like central control
• 6:35
  Speaker 1: system where the CEO of Anthropic, Dario, he coined this phrase,
• 6:40
  Speaker 1: the country of geniuses in the data center. That was
• 6:43
  Speaker 1: his phrase to describe what they're trying to build, you know?
• 6:47
  Speaker 2: I think that's a little bit misleading.
• 6:49
  Speaker 1: I think it would be more accurate to describe it
• 6:51
  Speaker 1: as army of geniuses in the data center, because it's
• 6:53
  Speaker 1: not like it's a bunch of diverse, different AIs, you know,
• 6:56
  Speaker 1: living in their different parts of the data center. They're
• 6:58
  Speaker 1: all copies of the same big model and they're owned
• 7:02
  Speaker 1: by the company. And so they all follow the orders
• 7:05
  Speaker 1: given by the company, right? People should be asking questions
• 7:08
  Speaker 1: of like, who controls this army or these armies and
• 7:10
  Speaker 1: what are they going to.
• 7:11
  Speaker 2: Be doing with them?
• 7:12
  Speaker 1: I think that we could very easily end up in
• 7:13
  Speaker 1: a sort of A situation where some tiny group of
• 7:18
  Speaker 1: people are essentially oligarchs or dictators. And ironically, both of
• 7:24
  Speaker 1: these risks, the loss of control and the constitution of power,
• 7:28
  Speaker 1: are things that people in the industry have been thinking
• 7:31
  Speaker 1: about for decades. Even before the AI industry existed, people
• 7:35
  Speaker 1: thinking about AI were talking and writing about these things.
• 7:38
  Speaker 1: And then part of the founding narrative or the founding
• 7:40
  Speaker 1: myth of DeepMind and OpenAI and Anthropic is these problems
• 7:44
  Speaker 1: are real. So we need to get there first so
• 7:48
  Speaker 1: that we can handle it responsibly. Those are, I think,
• 7:51
  Speaker 1: the big two reasons, but then I can go on.
• 7:53
  Speaker 1: There's lots more reasons as well. So one thing is,
• 7:56
  Speaker 1: you know, World War III, geopolitical conflict. If AI does
• 7:59
  Speaker 1: in fact get incredibly powerful, that's going to change the
• 8:02
  Speaker 1: balance of power between nations. That's going to disrupt a
• 8:04
  Speaker 1: lot of things. That puts us at increased risk of
• 8:07
  Speaker 1: crisis more generally, right? Another one, what about those jobs?
• 8:12
  Speaker 1: You're going to lose your taxi job, but not just
• 8:14
  Speaker 1: the taxi driver, everybody.
• 8:15
  Speaker 2: Pretty much.
• 8:17
  Speaker 1: There might be a few exceptions, like people whose jobs
• 8:19
  Speaker 1: for legal reasons are only allowed to be done by humans.
• 8:22
  Speaker 1: But for the most part, everybody should be afraid that
• 8:25
  Speaker 1: their jobs are going to be lost, even if we
• 8:26
  Speaker 1: manage to avoid all the other problems, right?
• 8:29
  Speaker 2: This narrative has started to emerge. And I've had several
• 8:31
  Speaker 2: interviews on the show where I've interviewed people who are very,
• 8:34
  Speaker 2: very scared and anxious about AI. And these are people
• 8:36
  Speaker 2: that have worked in the industry for sometimes decades. The
• 8:39
  Speaker 2: counter narrative coming over the hill is that this is doomerism.
• 8:43
  Speaker 2: that these people are, for whatever reason, just trying to
• 8:45
  Speaker 2: scare people and that they don't really understand what they're
• 8:47
  Speaker 2: talking about. How do you respond to that sort of counter-narrative?
• 8:50
  Speaker 2: And you must have seen this emerging yourself, especially from
• 8:53
  Speaker 2: people who stand to benefit, dare I say. Yeah, exactly.
• 8:56
  Speaker 1: This counter-narrative is fairly recent and it's been pushed by
• 8:59
  Speaker 1: the people who stand to benefit from it.
• 9:02
  Speaker 2: And it's not true.
• 9:04
  Speaker 1: These concerns have been around for decades since before the
• 9:07
  Speaker 1: AI industry existed and They're actually pretty reasonable concerns. Like
• 9:10
  Speaker 1: if you take the companies at their word and imagine
• 9:12
  Speaker 1: that they are in fact going to build superintelligence.
• 9:14
  Speaker 2: Well, it raises a lot of questions like who's going
• 9:17
  Speaker 2: to control it? Will anybody control it? What about the jobs?
• 9:20
  Speaker 1: You know, like these are just kind of obvious implications
• 9:23
  Speaker 1: to be thinking about and worrying about.
• 9:24
  Speaker 2: Who are you and what's your story? My name is
• 9:26
  Speaker 2: Daniel Cocutelo.
• 9:29
  Speaker 1: I currently run the AI Futures Project, which is a
• 9:32
  Speaker 1: small nonprofit that mostly focuses on forecasting the future of AI.
• 9:38
  Speaker 2: Before that, I worked at OpenAI. AI forecasting. Yeah.
• 9:43
  Speaker 1: So think about how like, you know, industry analysts who
• 9:47
  Speaker 1: work for hedge funds and stuff will make these forecasts
• 9:49
  Speaker 1: of like, here's, you know, how many cars Tesla will
• 9:52
  Speaker 1: be selling five years from now. Or like, here's what
• 9:55
  Speaker 1: the price of electricity will be in two years.
• 9:58
  Speaker 2: Right. That's forecasting.
• 10:00
  Speaker 1: I was doing that, but specifically focused on AI. The
• 10:03
  Speaker 1: reason I was doing it is because it's incredibly important
• 10:05
  Speaker 1: to see where this is all headed.
• 10:07
  Speaker 2: Why did you go to OpenAI? What did you do there?
• 10:10
  Speaker 2: What did you observe while you were there? And how
• 10:12
  Speaker 2: did it change your perspective on the future of AI,
• 10:16
  Speaker 2: but also I guess OpenAI as a company. And for
• 10:18
  Speaker 2: anybody that doesn't know, OpenAI are the company that produced ChatGPT. Yeah.
• 10:21
  Speaker 1: So I went to OpenAI in 2022. A large part
• 10:24
  Speaker 1: of what I did there was more forecasting. AI 2027
• 10:27
  Speaker 1: is a scenario that you may have heard of. I
• 10:29
  Speaker 1: did like smaller, you know, lower effort versions of them
• 10:33
  Speaker 1: internally for just internal circulation of like, here's some guesses
• 10:36
  Speaker 1: as to what the next couple of years might look like.
• 10:38
  Speaker 1: I also worked on evaluations for dangerous capabilities. So, you know,
• 10:42
  Speaker 1: trying to measure the AI's cyber abilities or persuasion abilities
• 10:46
  Speaker 1: or situational awareness. And I also briefly was on a
• 10:51
  Speaker 1: capabilities team doing reinforcement learning to create agents. AI is
• 10:55
  Speaker 1: in fact getting a lot better and I can say
• 10:58
  Speaker 1: more about why, you know, scaling laws, deep neural nets,
• 11:02
  Speaker 1: bigger trend on more data, become more efficient, more competent
• 11:05
  Speaker 1: at those things. I also became a bit more disillusioned
• 11:10
  Speaker 1: with the AI industry. So OpenAI, Anthropic, and DeepMind all
• 11:15
  Speaker 1: had these sort of founding narratives of like, yes, these
• 11:18
  Speaker 1: risks are real, but we've thought about them and we're
• 11:21
  Speaker 1: going to try to handle them responsibly. And that's why
• 11:23
  Speaker 1: it's important for us to keep doing what we're doing.
• 11:26
  Speaker 1: And I increasingly came to think that these were rationalizations
• 11:31
  Speaker 1: to justify what they were doing rather than sort of
• 11:34
  Speaker 1: like deeply guiding their actual behavior and that when push
• 11:36
  Speaker 1: comes to shove, they'll follow their incentives rather than do.
• 11:41
  Speaker 2: What's actually good. So you're inside OpenAI at the time,
• 11:45
  Speaker 2: and you start to believe that they're following commercial incentives
• 11:49
  Speaker 2: versus the, I guess, social or societal incentives that they
• 11:52
  Speaker 2: founded themselves on. Sort of.
• 11:54
  Speaker 1: I mean, I wouldn't actually describe it as commercial incentives.
• 11:56
  Speaker 1: I think I would describe it as power-seeking incentives. So, like,
• 12:03
  Speaker 1: it's true that the companies care a lot about making
• 12:05
  Speaker 1: a lot of money, but especially at the very top
• 12:08
  Speaker 1: of these companies, like the leaders, right? They understand that
• 12:12
  Speaker 1: this is about more than just money. There are these
• 12:14
  Speaker 1: emails that came up in the lawsuit between Musk and OpenAI.
• 12:20
  Speaker 1: A bunch of emails were surfaced in that lawsuit, which
• 12:22
  Speaker 1: you can go read. And in some of them, the
• 12:25
  Speaker 1: founders of OpenAI were talking back in 2017 about how
• 12:29
  Speaker 1: the reason why we made OpenAI was because we were
• 12:31
  Speaker 1: worried that Demis Hassabis at Google was going to become
• 12:34
  Speaker 1: dictator with AGI. Even back then, they were, this is
• 12:38
  Speaker 1: obviously about more than just money. Like these powerful CEOs
• 12:41
  Speaker 1: are literally afraid that if the other guy gets there first,
• 12:45
  Speaker 1: he might become dictator.
• 12:47
  Speaker 2: And they don't trust each other.
• 12:49
  Speaker 1: And so that's why they are racing as hard as
• 12:51
  Speaker 1: they can so that they're the ones who get there first,
• 12:54
  Speaker 1: so to speak.
• 12:55
  Speaker 2: Have you met Sam Altman?
• 12:57
  Speaker 1: Yeah.
• 12:58
  Speaker 2: And did that shape your opinion of his incentives or
• 13:00
  Speaker 2: why he's doing what he's doing? Because there's a lot
• 13:03
  Speaker 2: speculated about what his incentives are. I mean, his most
• 13:06
  Speaker 2: recent narrative says... For the good of humanity, I think
• 13:08
  Speaker 2: that's what.
• 13:09
  Speaker 1: Yeah, I mean, I think the main thing I've learned
• 13:10
  Speaker 1: is don't pay attention to the narratives. You know, like
• 13:13
  Speaker 1: what they say to one person is just different from
• 13:15
  Speaker 1: what they can say to someone in person at the
• 13:17
  Speaker 1: same time. And what they say in public is a
• 13:20
  Speaker 1: third thing entirely. I think you should judge people by
• 13:22
  Speaker 1: their actions, not by their words.
• 13:25
  Speaker 2: And why are you no longer at OpenAI? Largely for
• 13:27
  Speaker 2: the reason that I mentioned.
• 13:28
  Speaker 1: So I became gradually disillusioned with how the company was
• 13:31
  Speaker 1: going to behave. For example, when I first joined in 2022,
• 13:36
  Speaker 1: at least the people I talked to, my colleagues at
• 13:38
  Speaker 1: the company, there was this general sense of like, of
• 13:40
  Speaker 1: course we wouldn't actually just build super intelligence as soon
• 13:44
  Speaker 1: as possible. Once we started getting really close, like once
• 13:46
  Speaker 1: we started getting to AIs that could maybe automate the
• 13:49
  Speaker 1: AI research process, we would pause and figure out how
• 13:53
  Speaker 1: to make it safe. That's because we're the good guys.
• 13:56
  Speaker 1: And that's obviously the safe thing you should do rather
• 13:57
  Speaker 1: than just going full speed ahead. But we're worried about
• 14:01
  Speaker 1: other people who might not pause, you know, our competitors, Google,
• 14:04
  Speaker 1: for example, And so that's why we need to be
• 14:07
  Speaker 1: in the lead so that we have that room to
• 14:09
  Speaker 1: do the safe stuff, right? That was sort of like
• 14:12
  Speaker 1: a thing that seems like maybe like the median position
• 14:15
  Speaker 1: or something among the colleagues I talked to when I
• 14:17
  Speaker 1: was there, when I started, including people like Sam, you know,
• 14:20
  Speaker 1: including the leadership. And then by the time I left,
• 14:22
  Speaker 1: I was like, oh man, they're really not going to
• 14:23
  Speaker 1: do that, are they? Like they've sort of, you know,
• 14:27
  Speaker 1: partly because this has become more politicized and they've become
• 14:30
  Speaker 1: bigger and been under more scrutiny. people have started asking like,
• 14:33
  Speaker 1: why are you doing this in the first place if
• 14:34
  Speaker 1: it's so risky? And so they're pivoted their narrative to
• 14:37
  Speaker 1: being more like, actually, it's not that risky, you know? Um,
• 14:41
  Speaker 1: and so, yeah, I mean, it seems like they're just
• 14:42
  Speaker 1: going to keep going roughly as fast as they can
• 14:45
  Speaker 1: and hope that they can figure it out on the way.
• 14:47
  Speaker 2: How did your time at OpenAI come to an end? Uh,
• 14:50
  Speaker 2: I resigned in 2024. I had a nice goodbye party.
• 14:54
  Speaker 2: What were the reasons you gave for quitting OpenAI?
• 14:57
  Speaker 1: I thought that we were rationalizing too much and that
• 14:59
  Speaker 1: we needed to think more about what would actually be
• 15:00
  Speaker 1: good for the world. Um, I wanted more freedom to publish.
• 15:05
  Speaker 1: So at OpenAI, as it became a bigger company, it
• 15:09
  Speaker 1: became more of a normal tech company with incentives and
• 15:13
  Speaker 1: a PR department and things like that. And so it
• 15:16
  Speaker 1: started becoming more difficult to publish the sort of research
• 15:20
  Speaker 1: that I was doing. For example, those scenarios that I mentioned,
• 15:23
  Speaker 1: couldn't publish those, right? They're just for internal use. I
• 15:27
  Speaker 1: thought that that was a shame because right now, most
• 15:31
  Speaker 1: of the world is kind of asleep at the wheel
• 15:32
  Speaker 1: and doesn't really realize what's going on with AI and
• 15:34
  Speaker 1: doesn't really realize what's coming in the pipeline a couple
• 15:36
  Speaker 1: of years from now. And the companies aren't really incentivized
• 15:40
  Speaker 1: to tell people that much about it. I mean, they
• 15:42
  Speaker 1: say some vague stuff in a sort of hypey way, but,
• 15:45
  Speaker 1: you know, well, they didn't want me to publish the scenario,
• 15:50
  Speaker 1: for example, laying out like, here's how things might actually look.
• 15:54
  Speaker 2: I'm just, I'm super curious as to what it's like
• 15:55
  Speaker 2: being in a company like that when they, you know,
• 15:58
  Speaker 2: chat GPT-3 is released. You were there at that time, right? Mm-hmm.
• 16:02
  Speaker 2: Which was a moment where I think the whole world
• 16:03
  Speaker 2: stood up and realized that this technology was powerful. Yeah.
• 16:08
  Speaker 2: And the conversation really began from a society level. The
• 16:12
  Speaker 2: company starts growing super quickly.
• 16:14
  Speaker 1: Yeah.
• 16:14
  Speaker 2: Quicker than I think anybody could ever have imagined. And
• 16:18
  Speaker 2: what was it like inside there? What did you see
• 16:20
  Speaker 2: change over that period of time? I remember one all-hands
• 16:24
  Speaker 2: meeting where Ilya said something like.
• 16:26
  Speaker 1: Ilya being? Ilya Setskiver, who was head of research at
• 16:29
  Speaker 1: that time. He said something like, okay, now the world
• 16:31
  Speaker 1: is starting to pay attention. Each of you is going
• 16:33
  Speaker 1: to be the most popular person at every party for
• 16:37
  Speaker 1: the next year. Don't let it get to your head.
• 16:39
  Speaker 1: Focus on the mission. Got to build AGI. The company
• 16:42
  Speaker 1: grew a lot. It already wasn't really feeling like a
• 16:45
  Speaker 1: nonprofit when I joined, but it definitely didn't feel like
• 16:48
  Speaker 1: a nonprofit by the time I left. Lots of new
• 16:50
  Speaker 1: people came in. Ironically, the amount of conversation about superintelligence
• 16:59
  Speaker 1: and the implications of superintelligence arguably sort of went down
• 17:03
  Speaker 1: over time due to this growth, right? So because the
• 17:07
  Speaker 1: company would like double and then double again and then
• 17:09
  Speaker 1: double again, all these new people were coming in from
• 17:12
  Speaker 1: other parts of the tech industry who hadn't really been
• 17:14
  Speaker 1: thinking about these things and were attracted by the high salaries.
• 17:16
  Speaker 2: You lost $ 2 million for not signing an anti-disparagement clause,
• 17:21
  Speaker 2: which would mean you couldn't criticize the company. Ah, yes. Well,
• 17:26
  Speaker 2: so I got to keep the money. Yeah.
• 17:28
  Speaker 1: So what happened was after I had left, said my goodbyes,
• 17:32
  Speaker 1: et cetera, I got the exit paperwork and it included
• 17:37
  Speaker 1: this clause that said you basically have to agree not
• 17:39
  Speaker 1: to criticize the company again. And also a clause saying
• 17:41
  Speaker 1: you can't tell anyone about this. And so I thought that.
• 17:46
  Speaker 2: Was kind of.
• 17:48
  Speaker 1: Rich coming from a nonprofit that's supposed to be for
• 17:50
  Speaker 1: the benefit of all humanity. So I didn't sign it.
• 17:53
  Speaker 1: And if you don't sign, you don't get to keep
• 17:56
  Speaker 1: your equity. So Your compensation, what they pay you is
• 17:59
  Speaker 1: a bunch of money and then also a bunch of stock, basically.
• 18:03
  Speaker 1: But then they had this stuff in the contract that
• 18:06
  Speaker 1: they get to yank back your stock if you don't
• 18:09
  Speaker 1: sign this thing.
• 18:12
  Speaker 2: And my wife and I were upset about this.
• 18:16
  Speaker 1: We talked about it for like a month or two,
• 18:18
  Speaker 1: consulted some lawyers, and then ultimately decided to just refuse
• 18:21
  Speaker 1: to sign.
• 18:22
  Speaker 2: Which would mean you would have lost $ 2 million. That's right.
• 18:26
  Speaker 1: Which was like 80% of our net worth at the time. Fortunately,
• 18:30
  Speaker 1: it didn't go the way we expected. It blew up
• 18:34
  Speaker 1: basically on the internet. Like when people heard that we
• 18:37
  Speaker 1: had done this and that we had said no, it
• 18:39
  Speaker 1: became like this huge scandal. Employees at the company started
• 18:42
  Speaker 1: like asking questions in Slack and like asking leadership, like, wait, what?
• 18:46
  Speaker 2: Like, why are you going to take away our equity?
• 18:48
  Speaker 2: What is this?
• 18:49
  Speaker 1: You know, because a lot of people hadn't really noticed
• 18:51
  Speaker 1: this before. It had been whispered about, but it hadn't
• 18:53
  Speaker 1: been sort of like, a thing that most employees knew about.
• 18:56
  Speaker 1: And so they backtracked and they said, nevermind, nevermind.
• 18:59
  Speaker 2: We'll change the paperwork. You can keep the equity. It's fine.
• 19:02
  Speaker 2: And Sam Altman came out and said he was embarrassed
• 19:04
  Speaker 2: that he didn't realize this was good. Yeah, he had
• 19:06
  Speaker 2: no idea, apparently. You don't believe him?
• 19:09
  Speaker 1: No, I think he probably knew. And if he didn't know,
• 19:12
  Speaker 1: then people close to him probably did, such as his
• 19:13
  Speaker 1: head lawyer.
• 19:15
  Speaker 2: Why did you decide not to take the $ 2 million?
• 19:18
  Speaker 2: I mean... Most people would have, I think. It's true.
• 19:22
  Speaker 1: Most people would have and most people did. And money
• 19:25
  Speaker 1: is nice, but it's not the only thing. Sometimes it's
• 19:30
  Speaker 1: good to take a stand on principle. I keep mentioning superintelligence.
• 19:33
  Speaker 1: Perhaps I should say more about the sequence of events
• 19:37
  Speaker 1: that the companies are planning to do. So right now,
• 19:42
  Speaker 1: they're focusing on automating coding. They're taking their AIs. They're
• 19:45
  Speaker 1: making them bigger. They're training them for longer. And they're
• 19:48
  Speaker 1: especially focusing the training on getting them to be good
• 19:51
  Speaker 1: at autonomously writing and editing code because that will help
• 19:56
  Speaker 1: the companies go faster, right? If they can automate the code,
• 19:59
  Speaker 1: then they can do their own work better and faster
• 20:02
  Speaker 1: and accelerate progress. The next step, which they've already begun,
• 20:05
  Speaker 1: is to look at the rest of the research process
• 20:09
  Speaker 1: as well, coming up with ideas, analyzing experiments, communicating those results.
• 20:15
  Speaker 1: All the other parts of the research process, they're trying
• 20:18
  Speaker 1: to figure out how to train AIs to be good
• 20:19
  Speaker 1: at those as well so that they can have AIs
• 20:22
  Speaker 1: do the entire thing autonomously.
• 20:24
  Speaker 2: When you say do the entire thing, what do you
• 20:26
  Speaker 2: mean do the entire thing?
• 20:27
  Speaker 1: So like Anthropic and OpenAI in particular are trying to
• 20:31
  Speaker 1: automate themselves. Like they're trying to make it the case
• 20:33
  Speaker 1: that they don't really need human employees anymore. They just
• 20:37
  Speaker 1: have a giant army of AIs that's churning away, doing
• 20:42
  Speaker 1: all this autonomous research to make better AIs, to train
• 20:45
  Speaker 1: the new AIs, put them in charge so they can
• 20:48
  Speaker 1: make even better AIs and so forth. And of course,
• 20:51
  Speaker 1: not all just happening internally, but also interfacing with the world,
• 20:54
  Speaker 1: going out and talking to people, collecting the data, setting
• 20:57
  Speaker 1: up the training environments, doing the business deals and so forth.
• 21:01
  Speaker 1: They're trying to automate all of that. The reason why
• 21:04
  Speaker 1: they're doing this is because they're trying to get to
• 21:06
  Speaker 1: a position where they have AIs that are superhuman at everything, superintelligence,
• 21:13
  Speaker 1: and they're trying to get there before their competitors do.
• 21:16
  Speaker 1: Needless to say, this is incredibly dangerous, I would say,
• 21:19
  Speaker 1: you know? And in addition to being dangerous, it's a
• 21:22
  Speaker 1: power grab, right? Like if they actually succeed at this,
• 21:25
  Speaker 1: then they'll be sitting on top of this army of
• 21:28
  Speaker 1: superhuman AIs that will give them immense leverage over all
• 21:33
  Speaker 1: sorts of other actors in the economy. insofar as they
• 21:35
  Speaker 1: can work out something with the presidents and integrate it
• 21:38
  Speaker 1: into the military or whatever, then that would give the
• 21:40
  Speaker 1: US immense hard power over all of the countries, right? Obviously,
• 21:45
  Speaker 1: nobody knows exactly when this is happening, but a very
• 21:48
  Speaker 1: disquieting thing has happened over the last year to me,
• 21:51
  Speaker 1: which is that when we published AI 2027, people were
• 21:56
  Speaker 1: generally of the opinion that my timelines were too short
• 21:59
  Speaker 1: and that probably it would take more than 2027 until
• 22:02
  Speaker 1: we got to This sort of events that I was
• 22:05
  Speaker 1: just mentioning, you know, a recursive self-improvement AI is automating
• 22:09
  Speaker 1: the whole research process, super intelligence. These, these types of milestones, um,
• 22:15
  Speaker 1: they happen in 2027 in 2027.
• 22:18
  Speaker 2: Which is this research paper you published. That's right.
• 22:20
  Speaker 1: It's a, it's a scenario forecast that sort of lays
• 22:22
  Speaker 1: out like month by month, a possible future trajectory. There
• 22:26
  Speaker 1: was sort of like, at the time that we started writing,
• 22:28
  Speaker 1: it was my best guess as to what would actually happen. Obviously,
• 22:31
  Speaker 1: there's lots of uncertainty, but I thought it's valuable to
• 22:34
  Speaker 1: make a concrete guess just to sort of see what
• 22:36
  Speaker 1: it might look like. And at the time we were
• 22:38
  Speaker 1: writing this, a lot of my friends in the AI
• 22:40
  Speaker 1: industry and in nonprofits and so forth that work on AI,
• 22:44
  Speaker 1: a lot of people were saying like, yeah, that stuff's
• 22:46
  Speaker 1: going to happen, but it'll probably take a couple of
• 22:48
  Speaker 1: years longer than you think.
• 22:50
  Speaker 2: And now.
• 22:53
  Speaker 1: It's more 50-50, especially when I go talk to people
• 22:56
  Speaker 1: at Anthropic and OpenAI. They're often like, Yeah, no, 2027,
• 23:01
  Speaker 1: that's basically what's going to happen. Just like you wrote.
• 23:04
  Speaker 2: Why did you, why did you become, why did you
• 23:07
  Speaker 2: update your timelines? Oh yeah.
• 23:08
  Speaker 1: Context for this is, um, after, after writing it at 2027,
• 23:13
  Speaker 1: I shifted my timelines to be a little bit more conservative.
• 23:15
  Speaker 1: So at the time that we published my 50% mark
• 23:18
  Speaker 1: was in 2028, not in 2027. And then after we published,
• 23:22
  Speaker 1: progress just seemed like it was going a bit slower.
• 23:24
  Speaker 1: And so I updated to 2030, which is, you know,
• 23:28
  Speaker 1: still could happen sooner, could happen later.
• 23:29
  Speaker 2: 2030.
• 23:31
  Speaker 1: But now when I talk to people in the companies,
• 23:34
  Speaker 1: they're like, it's not going to take that long. They're like, oh,
• 23:37
  Speaker 1: you need to shorten them again. Like get them back
• 23:40
  Speaker 1: to 2027 or 2028, you know?
• 23:42
  Speaker 2: So that's a bit disquieting.
• 23:45
  Speaker 1: Again, don't know how long it's going to take, but
• 23:47
  Speaker 1: this is the stated plans of the companies is to
• 23:50
  Speaker 1: do this incredibly dangerous thing. And they think that they're
• 23:52
  Speaker 1: just a few years away.
• 23:53
  Speaker 2: So you wrote this report here, what 2026 looks like.
• 23:58
  Speaker 2: And you wrote this in 2021.
• 23:58
  Speaker 1: Yeah.
• 24:00
  Speaker 2: And it was remarkably accurate, helped make a name for
• 24:02
  Speaker 2: yourself amongst everybody in AI. And which one was it
• 24:07
  Speaker 2: that JD Vance, the vice president, read? I think it
• 24:09
  Speaker 2: was this one, wasn't it? Yeah, this one. And then
• 24:12
  Speaker 2: so then you published this one, AI 2027, and this
• 24:15
  Speaker 2: was published, I believe, in 2025? Yes, that's right, April.
• 24:19
  Speaker 2: What were you forecasting in here? What are the key
• 24:22
  Speaker 2: things that you said in here for people that haven't
• 24:23
  Speaker 2: read it?
• 24:24
  Speaker 1: The high-level version of it is they automate the coding,
• 24:28
  Speaker 1: then they automate the rest of the research process. Then
• 24:30
  Speaker 1: the pace of progress accelerates dramatically. They get to super intelligence.
• 24:33
  Speaker 1: They're working with the government, specifically the president, the executive
• 24:36
  Speaker 1: branch naturally wants to control this technology and otherwise wants
• 24:40
  Speaker 1: to use it to beat China and integrate into military
• 24:42
  Speaker 1: and so forth. By this point, it's sort of doing
• 24:46
  Speaker 1: basically all the work itself. I mean, it's super intelligent.
• 24:48
  Speaker 1: So it's coming up with all these great ideas for
• 24:51
  Speaker 1: how to integrate itself into everything and all these new
• 24:53
  Speaker 1: technologies it's invented and so forth. And because of the
• 24:56
  Speaker 1: race dynamics and because of the profit motive, they end
• 24:58
  Speaker 1: up deploying it everywhere. And it builds robot factories, they
• 25:01
  Speaker 1: build more robots, they build more robot factories, et cetera,
• 25:03
  Speaker 1: transforms the world entirely. And then at some point it
• 25:07
  Speaker 1: has enough power, it meaning the AIs have enough power
• 25:10
  Speaker 1: that they don't have to pretend to be aligned anymore.
• 25:15
  Speaker 2: Right. Then they stop listening to orders.
• 25:19
  Speaker 1: That's the race ending of AI 2027. We also wrote
• 25:24
  Speaker 1: a sort of different branch, which is the slowdown ending.
• 25:28
  Speaker 1: which was intended to sort of illustrate the concentration of
• 25:31
  Speaker 1: power issues that I mentioned previously. So what if hypothetically
• 25:37
  Speaker 1: the alignment issues get sorted out sufficiently quickly? Like what
• 25:39
  Speaker 1: if it turns out that like it's not too hard
• 25:42
  Speaker 1: with two months of slowdown, we can figure out how
• 25:44
  Speaker 1: to make the AIs robustly do what we want and
• 25:48
  Speaker 1: have the values that we want them to have. So
• 25:49
  Speaker 1: that's one possible branch. And in that branch, it looks
• 25:53
  Speaker 1: pretty similar. You know, they take the jobs, beat China,
• 25:56
  Speaker 1: et cetera. But instead of the AIs ultimately killing everyone,
• 26:01
  Speaker 1: they create this sort of amazing utopia. But the amazing
• 26:05
  Speaker 1: utopia is whatever the people who control the AIs wanted
• 26:08
  Speaker 1: it to be, right? And so that would be a
• 26:10
  Speaker 1: very small group of people, like the president, some CEOs.
• 26:14
  Speaker 2: Et cetera. there should be a button just down below here.
• 26:18
  Speaker 2: And if it says subscribe, you're already subscribed. If it
• 26:20
  Speaker 2: says subscriber, that means you're not yet. And if you're
• 26:24
  Speaker 2: not subscribed, please could you do us a favor and
• 26:25
  Speaker 2: hit that button? It helps the show more than you know.
• 26:27
  Speaker 2: And according to the algorithm, you're someone that watches our show,
• 26:30
  Speaker 2: but you haven't yet hit that button. Thank you so much.
• 26:33
  Speaker 2: Is there any possibility, do you think, that we never
• 26:35
  Speaker 2: get to this thing called AGI? And how do we
• 26:38
  Speaker 2: distinguish AGI from this term superintelligence? What's the difference?
• 26:42
  Speaker 1: Yeah, so the difference is that AGI is a more
• 26:44
  Speaker 1: vague and weak term. So superintelligence is a bit more
• 26:48
  Speaker 1: precisely defined. It's better than the best humans at everything,
• 26:50
  Speaker 1: faster and cheaper. AGI is more like, it stands for
• 26:53
  Speaker 1: artificial general intelligence, which means AI can do things in
• 26:57
  Speaker 1: general rather than like some specific task. And so arguably
• 27:00
  Speaker 1: we've already achieved AGI, right? If you use cloud code
• 27:03
  Speaker 1: or something like that, it's like, it can do a
• 27:04
  Speaker 1: lot of stuff. It's almost kind of like a little
• 27:07
  Speaker 1: employee that you can like have go do stuff. So
• 27:09
  Speaker 1: it is quite general. It's not maximally general though. It
• 27:12
  Speaker 1: can't do everything. Whereas superintelligence by definition is, can do
• 27:16
  Speaker 1: all the things that a human.
• 27:16
  Speaker 2: Can do, but better. And how does this sort of
• 27:18
  Speaker 2: overlap with robotics? Because obviously we're seeing this huge robotics
• 27:22
  Speaker 2: boom at the moment. There are some real world things
• 27:24
  Speaker 2: that humans can still do because these AIs are still
• 27:26
  Speaker 2: stuck in my computer.
• 27:28
  Speaker 1: The way that people talk about this is that they
• 27:30
  Speaker 1: basically just say, we've achieved super intelligence for cognitive tasks.
• 27:33
  Speaker 1: Then you can talk about like full super intelligence that
• 27:36
  Speaker 1: can do the physical stuff.
• 27:38
  Speaker 2: And are we going to get there? Are we going
• 27:39
  Speaker 2: to get there with both? I think so.
• 27:41
  Speaker 1: I mean, again, this is not something that we can
• 27:42
  Speaker 1: be certain about. You asked like, is it possible we'll
• 27:45
  Speaker 1: never get there? Yes, it's possible we'll never get there.
• 27:47
  Speaker 1: I don't think it's likely though. I think that there's
• 27:50
  Speaker 1: nothing sort of like magical about the human brain. It's,
• 27:54
  Speaker 1: you know, it's just a bunch of neurons. It is
• 27:57
  Speaker 1: possible for a digital system to do similar functions in
• 28:01
  Speaker 1: the same way that like, you know, a plane can
• 28:04
  Speaker 1: fly just like a bird. Not in the same way
• 28:07
  Speaker 1: as a bird necessarily. Like it doesn't have, it's not
• 28:09
  Speaker 1: flying in the same way that a bird flies, but
• 28:11
  Speaker 1: it flies, you know?
• 28:11
  Speaker 2: Mm-hmm. So it does seem like, yeah, like seems possible.
• 28:15
  Speaker 2: You've written all these, you know, these research reports. You're
• 28:18
  Speaker 2: working on another one that will be released likely on
• 28:20
  Speaker 2: the 9th of July. You have worked inside OpenAI. You
• 28:25
  Speaker 2: then quit OpenAI because you were concerned about what was
• 28:27
  Speaker 2: going on there and about the future of the industry.
• 28:30
  Speaker 2: You know more than I do. Are you optimistic about
• 28:34
  Speaker 2: the future or pessimistic? Are we heading to a bad
• 28:36
  Speaker 2: place if things don't change based on everything that you know?
• 28:40
  Speaker 1: I think we are headed to a bad place if
• 28:42
  Speaker 1: things don't change. I'm not confident in that. I would
• 28:45
  Speaker 1: say something like 70%. It's very, very hard to predict,
• 28:48
  Speaker 1: of course. But yeah, it seems like the current default
• 28:50
  Speaker 1: path is heading towards a.
• 28:51
  Speaker 2: Very, very scary place. How do you contend with that
• 28:54
  Speaker 2: personally and emotionally? It's rough. I mean, I think it's
• 28:58
  Speaker 2: the sort of thing that gets me down on.
• 29:04
  Speaker 1: A regular basis, but also I've been dealing with this
• 29:07
  Speaker 1: for so many years now that I've sort of gotten
• 29:09
  Speaker 1: used to it.
• 29:10
  Speaker 2: If that makes sense.
• 29:15
  Speaker 1: Yeah, yeah, I'll put it this way. I would be
• 29:18
  Speaker 1: incredibly happy if all my predictions turned out to be
• 29:20
  Speaker 1: wrong and an AI hits the wall, for example.
• 29:23
  Speaker 2: It gets you down on a regular basis.
• 29:26
  Speaker 1: I used to be known as a pretty chipper and
• 29:27
  Speaker 1: optimistic person, but in 2020, my AI timelines predictions started
• 29:34
  Speaker 1: collapsing due to GPT-3 and the scaling loss papers and
• 29:39
  Speaker 1: the bio-incursor which I can talk about if you're interested.
• 29:41
  Speaker 1: But basically, some events happened in 2020 that convinced me
• 29:45
  Speaker 1: that actually this stuff was quite plausibly coming by the
• 29:48
  Speaker 1: end of the decade. And humanity is very obviously not
• 29:52
  Speaker 1: ready for this in a whole bunch of different ways.
• 29:54
  Speaker 1: And so that's obviously very scary.
• 29:56
  Speaker 2: And that's an extremely scary world because of all the
• 29:58
  Speaker 2: things you've said, but again, because of this percussive self-improvement
• 30:01
  Speaker 2: where AIs can train themselves. And at such a point,
• 30:04
  Speaker 2: we're starting to lose hold of what's going on here.
• 30:06
  Speaker 1: I mean, the AIs are already training themselves to be clear.
• 30:08
  Speaker 1: It's more like closing the entire research loop, right?
• 30:12
  Speaker 2: Doing everything. Yeah.
• 30:13
  Speaker 1: Like right now, a lot of the training data is
• 30:16
  Speaker 1: generated by AIs. A lot of the reinforcement, like the
• 30:19
  Speaker 1: grading that happens.
• 30:20
  Speaker 2: Doling out of positive and negative reinforcement. Is it self-done
• 30:23
  Speaker 2: by AIs? Can you explain that in layman's terms? Yeah.
• 30:26
  Speaker 1: So an important thing for everybody to understand is that
• 30:29
  Speaker 1: modern AI systems are not software in the normal sense.
• 30:32
  Speaker 1: I mean, they are technically software, but they're not lines
• 30:35
  Speaker 1: of code. You know, it's not like some engineers at
• 30:38
  Speaker 1: Anthropic went and wrote lines of code that basically says like,
• 30:43
  Speaker 1: you know, when the user asks for this type of thing,
• 30:46
  Speaker 1: then go do this type of thing for this many
• 30:48
  Speaker 1: steps or whatever.
• 30:49
  Speaker 2: There's nothing like that. Instead, it's a neural net, you know?
• 30:51
  Speaker 2: What's that?
• 30:53
  Speaker 1: Well, think about how the brain is a bunch of
• 30:55
  Speaker 1: neurons connected to each other that are firing signals back
• 30:59
  Speaker 1: and forth. The brain learns over time. The types of
• 31:03
  Speaker 1: patterns of firing that caused success, that caused a dopamine
• 31:08
  Speaker 1: rush or various other types of feedback get reinforced and
• 31:11
  Speaker 1: fire more often. And the types of patterns that caused failure,
• 31:15
  Speaker 1: like touching a hot stove, get anti-reinforced, get destroyed so
• 31:21
  Speaker 1: that they fire less often. And as a result of
• 31:23
  Speaker 1: all of that, you, over the course of years, learn
• 31:26
  Speaker 1: to act in the world and you learn all sorts
• 31:29
  Speaker 1: of skills and you learn world models. You learn like
• 31:31
  Speaker 1: beliefs about the world and you can sort of like
• 31:33
  Speaker 1: mentally simulate how it's going and stuff like that. So
• 31:36
  Speaker 1: artificial neural nets are like that, except artificial. So it
• 31:39
  Speaker 1: starts off as a giant tangled spaghetti mess of randomly
• 31:45
  Speaker 1: generated artificial connections called parameters. These days, they might be
• 31:51
  Speaker 1: something like 10 trillion parameters in the biggest AIs. So
• 31:57
  Speaker 1: it starts off randomly generated. So it's, of course, completely useless.
• 32:00
  Speaker 1: If you give it some input, it'll just produce gibberish
• 32:03
  Speaker 1: as an output. But then they train it. And they
• 32:07
  Speaker 1: start with pre-training, which is where you give it a
• 32:10
  Speaker 1: bunch of internet text, and you show it the first
• 32:13
  Speaker 1: piece of text, and you put that in as the input,
• 32:16
  Speaker 1: and then it gives a gibberish output.
• 32:18
  Speaker 2: And then you.
• 32:20
  Speaker 1: positively or negatively reinforced it based on how accurate that
• 32:23
  Speaker 1: output was at predicting the next piece of text. So
• 32:26
  Speaker 1: it's basically playing this game of like predict the next word.
• 32:29
  Speaker 2: Isn't that how it happens with babies? I think I
• 32:31
  Speaker 2: had a neuroscientist tell me that babies have more neural
• 32:34
  Speaker 2: connections than adults. And yeah, it says toddlers have twice
• 32:38
  Speaker 2: as many neural connections as adults. And I guess they
• 32:42
  Speaker 2: whittle down through reinforcement.
• 32:44
  Speaker 1: Yep.
• 32:44
  Speaker 2: We have more pathways when we're younger and just like
• 32:47
  Speaker 2: the process of training in AI, we're trained down to
• 32:50
  Speaker 2: like remove the ones that aren't useful and build up
• 32:52
  Speaker 2: on the ones that are. Yeah.
• 32:53
  Speaker 1: It's both pruning and strengthening. And it seems like in humans,
• 32:56
  Speaker 1: it's actually more pruning than strengthening, but it's both. And
• 32:59
  Speaker 1: in AI is the same thing.
• 33:01
  Speaker 2: It's both.
• 33:01
  Speaker 1: So the first portion of training is where they train
• 33:04
  Speaker 1: the AI to predict text, which is kind of like
• 33:06
  Speaker 1: training it to read. And it's a similar thing does
• 33:09
  Speaker 1: happen in humans. So basically it's, The random tangle gradually
• 33:13
  Speaker 1: takes shape and gradually sort of coalesces into more useful
• 33:17
  Speaker 1: circuitry that has stored lots of facts about the world
• 33:21
  Speaker 1: and has stored lots of skills for how to process
• 33:26
  Speaker 1: information and transform it and then produce predictions. That's just
• 33:30
  Speaker 1: the first step. After they do the pre-training, then they
• 33:33
  Speaker 1: try to teach it more useful skills besides just predicting text.
• 33:37
  Speaker 1: And so by the end of the process, They've thrown
• 33:40
  Speaker 1: lots of coding problems at it. And they've said like,
• 33:43
  Speaker 1: here's a coding problem.
• 33:44
  Speaker 2: Go.
• 33:45
  Speaker 1: Here's a coding problem. Here's an environment. You have access
• 33:47
  Speaker 1: to this virtual computer. Here's like the code base you're
• 33:50
  Speaker 1: working with. You can write code. You can edit the code.
• 33:52
  Speaker 1: You can run the code.
• 33:53
  Speaker 2: You can read it. You can use the internet. Go, go, go.
• 33:57
  Speaker 1: And it does that for a while. And then based
• 33:59
  Speaker 1: on how successful it is, reinforcement happens. And they have thousands,
• 34:03
  Speaker 1: maybe millions of examples of coding problems like that, that
• 34:06
  Speaker 1: they train it on. And that's why they're so good
• 34:08
  Speaker 1: at coding now.
• 34:09
  Speaker 2: So what does super intelligence look like in this regard?
• 34:11
  Speaker 2: Is it just more of these connections and how would
• 34:14
  Speaker 2: they get more connections? Can you explain that to me?
• 34:17
  Speaker 1: So there's different AI models, right? So there's like GPT-3
• 34:21
  Speaker 1: and GPT-4, GPT-4.5 and GPT-5 and GPT-5.5 and 5.6, right?
• 34:26
  Speaker 1: Sometimes they're just the same previous model, but with extra training.
• 34:30
  Speaker 1: Sometimes there are a whole new model that's been trained
• 34:32
  Speaker 1: from scratch, including starting the whole pre-training process again. over
• 34:36
  Speaker 1: the last couple of years, they've done several new rounds
• 34:39
  Speaker 1: of starting over from scratch. And typically when they start
• 34:42
  Speaker 1: over from scratch, they make the whole thing bigger, the
• 34:45
  Speaker 1: artificial brain much bigger.
• 34:46
  Speaker 2: Okay.
• 34:46
  Speaker 1: Right now they're at something like 10 trillion parameters back
• 34:49
  Speaker 1: in 2020. Um, it was more like 175 billion. So
• 34:55
  Speaker 1: we've grown like two orders of magnitude, uh, in six years,
• 34:58
  Speaker 1: two orders of magnitude.
• 34:59
  Speaker 2: Yeah. Like two, 10 X's. So 100 X. Right.
• 35:03
  Speaker 1: So that process is continuing. Um, They're also improving the
• 35:07
  Speaker 1: algorithms themselves. So they're not literally just the same type
• 35:10
  Speaker 1: of AI, but bigger. They've also come up with all
• 35:13
  Speaker 1: sorts of ideas for how to change the structure of
• 35:17
  Speaker 1: the connections and the neurons and so forth and change
• 35:19
  Speaker 1: the reinforcement algorithms that they're using and to change the
• 35:24
  Speaker 1: training data that they're training on. All sorts of tweaks
• 35:27
  Speaker 1: that have made this whole thing more efficient. We're literally
• 35:29
  Speaker 1: building a brain.
• 35:30
  Speaker 2: Basically, yeah.
• 35:31
  Speaker 1: As they make more brains, they're getting better at making,
• 35:33
  Speaker 1: they're making them bigger and making them more efficient and
• 35:36
  Speaker 1: so forth.
• 35:37
  Speaker 2: And it's literally modeled on the brain, like the way
• 35:39
  Speaker 2: it works, right?
• 35:40
  Speaker 1: It's certainly heavily inspired by the brain, but I shouldn't
• 35:42
  Speaker 1: overstate the analogy. Like there's lots of differences too. So
• 35:45
  Speaker 1: for example, the transformer architecture, which is the architecture that
• 35:50
  Speaker 1: they use for these LLMs is not really recurrent. So
• 35:55
  Speaker 1: the information sort of flows one way rather than allowing
• 35:58
  Speaker 1: all these sort of little loops on the inside. Also
• 36:00
  Speaker 1: the back propagation algorithm is different from the sort of
• 36:04
  Speaker 1: learning that naturally happens in human brains. So there are
• 36:06
  Speaker 1: some differences, but yes, like broadly speaking, we are sort
• 36:10
  Speaker 1: of making artificial brains. It's kind of like for brains,
• 36:12
  Speaker 1: what like a plane is for a bird.
• 36:15
  Speaker 2: That's a really good analogy. That analogy helped me think
• 36:18
  Speaker 2: through a bunch of questions people often ask about AI
• 36:20
  Speaker 2: when they said, can it be creative? But actually that
• 36:23
  Speaker 2: analogy kind of helps me understand that actually that maybe
• 36:25
  Speaker 2: that's not the question. It's can it produce something that
• 36:29
  Speaker 2: you would consider to be creative? Because creativity is people
• 36:32
  Speaker 2: think of it as like a process yeah but actually
• 36:34
  Speaker 2: it's judged based on the output isn't it.
• 36:36
  Speaker 1: I mean you can get philosophical about like is it
• 36:38
  Speaker 1: truly creativity that they have but you can also be
• 36:41
  Speaker 1: like well i mean just look at all the stuff
• 36:42
  Speaker 1: they're accomplishing you know and uh it seems like they're
• 36:46
  Speaker 1: going to be accomplishing a lot more in the near future.
• 36:48
  Speaker 2: Yeah i do i i asked the question about how
• 36:50
  Speaker 2: this weighs on you personally because i can i can
• 36:52
  Speaker 2: sense that you're actually personally bothered.
• 36:55
  Speaker 1: I mean i think the situation is crazy like First
• 37:00
  Speaker 1: of all, it's very exciting. Like, AI is really fascinating
• 37:02
  Speaker 1: and interesting stuff. I've been following the field for more
• 37:04
  Speaker 1: than a decade now. I've been part of it for
• 37:07
  Speaker 1: some years. And it's really cool, really interesting. And it's
• 37:11
  Speaker 1: really fun to think about what's going on inside these
• 37:14
  Speaker 1: artificial brains and why.
• 37:16
  Speaker 2: They are the way that they are.
• 37:16
  Speaker 1: And it's really cool to see all the applications of
• 37:19
  Speaker 1: this technology out in the world. But it really seems
• 37:23
  Speaker 1: like we're on a pretty scary path. And the more
• 37:25
  Speaker 1: you think about it, the more worried you get. And,
• 37:28
  Speaker 1: you know, in stories... it always ends well, but this
• 37:31
  Speaker 1: is real life. And I think we have to sort
• 37:34
  Speaker 1: of stare reality in the face and tell it and realize.
• 37:38
  Speaker 2: That like, it might not actually end well, you know?
• 37:41
  Speaker 2: Were there any recent, dare I say, I was going
• 37:44
  Speaker 2: to say eureka moments, but paradigm shifting moments where even
• 37:47
  Speaker 2: your own sort of mental model of what's going on
• 37:50
  Speaker 2: here and how this is going to look were changed
• 37:52
  Speaker 2: for better or for worse?
• 37:53
  Speaker 1: For better or for worse, and probably for worse, things
• 37:55
  Speaker 1: are kind of on track for AI 2027. There are
• 37:58
  Speaker 1: a few things that have been not exactly like paradigm
• 38:01
  Speaker 1: shift differences, but like there have been some differences from
• 38:03
  Speaker 1: what we expected at the time we wrote this. So
• 38:06
  Speaker 1: the government has actually gotten involved faster than we expected
• 38:08
  Speaker 1: and has been more aggressive than we expected. So the
• 38:11
  Speaker 1: export controls on mythos being the biggest example and also
• 38:15
  Speaker 1: threatening Anthropic with being destroyed by the Defense Production Act.
• 38:20
  Speaker 1: Another thing that's been surprising to us is that Anthropic
• 38:22
  Speaker 1: in particular has gone from second place to first place
• 38:26
  Speaker 1: in the sort of, in the race, basically.
• 38:28
  Speaker 2: Yeah. Why do you think that happened? Because it seemed
• 38:30
  Speaker 2: like ChatGPT were out front and clear as it relates
• 38:34
  Speaker 2: to OpenAI were out front and clear, but suddenly Anthropic
• 38:37
  Speaker 2: have lapped them.
• 38:40
  Speaker 1: Yeah. I mean, I guess they have probably higher talent
• 38:43
  Speaker 1: density and better strategy, but not by a lot, but
• 38:48
  Speaker 1: enough to make a difference. Why do you think they
• 38:49
  Speaker 1: have more talent? Well, they don't have more compute. Like
• 38:54
  Speaker 1: what are the inputs, right? Like they're in the lead now.
• 38:57
  Speaker 1: They used to be behind. What are the possible explanations
• 38:59
  Speaker 1: for this? Well, it could have been that they had
• 39:02
  Speaker 1: more resources, like more compute, more money, but that's not true.
• 39:04
  Speaker 1: They have less resources and less money, right? So then
• 39:08
  Speaker 1: I guess talent is the next best alternative. You can
• 39:12
  Speaker 1: maybe say strategy, some combination of those things.
• 39:14
  Speaker 2: Yeah. Something that wasn't just like the amount of resources
• 39:17
  Speaker 2: they had. Just like John Jones, where marginal improvements in
• 39:21
  Speaker 2: your cognitive performance can have a massive impact. Sometimes I
• 39:24
  Speaker 2: podcast for 10 hours a day. Over the last couple
• 39:25
  Speaker 2: of weeks, I've been in filming for a TV show.
• 39:28
  Speaker 2: And I have like one or two days off to
• 39:29
  Speaker 2: get all of my work done, which means there's lots
• 39:31
  Speaker 2: of cognitive load. And so I turned to ketones because
• 39:34
  Speaker 2: I find myself more articulate, able to think more clearly,
• 39:37
  Speaker 2: able to work out better when I'm fueled by ketones.
• 39:41
  Speaker 2: And so the reason I became a co-owner of this
• 39:42
  Speaker 2: company and the reason why they now are a sponsor
• 39:44
  Speaker 2: of this podcast is because I remember one of my
• 39:46
  Speaker 2: team members called Christiana, she tried it once and came
• 39:48
  Speaker 2: up to my desk and she goes, This is the
• 39:50
  Speaker 2: best product ever made. And I think in part that's
• 39:53
  Speaker 2: because she really cares about those cognitive benefits as I do,
• 39:56
  Speaker 2: as Jon Jones does, and as I think most of
• 39:58
  Speaker 2: my listeners probably will. So if you haven't tried these yet,
• 40:00
  Speaker 2: all you have to do is go to ketone.com slash
• 40:04
  Speaker 2: Stephen and you'll also get 30% off your first subscription order.
• 40:07
  Speaker 2: You'll get exclusive Ketone IQ merch and of course, cognitive
• 40:11
  Speaker 2: benefits that might just change your life. Much of the
• 40:14
  Speaker 2: reason most people haven't posted content or built their personal
• 40:17
  Speaker 2: brand is because It's hard and it's time consuming and
• 40:20
  Speaker 2: we're all very, very busy. And if you've never posted
• 40:23
  Speaker 2: something before, there's so many factors in your psychology that
• 40:27
  Speaker 2: stop you wanting to post. What people will think of you.
• 40:30
  Speaker 2: Am I doing this right? Is the thing I'm saying
• 40:32
  Speaker 2: absolutely stupid? All of these result in paralysis, which means
• 40:36
  Speaker 2: you don't post and your feed goes bare. I'm an
• 40:40
  Speaker 2: investor in a company called Stan Store, which you've probably
• 40:42
  Speaker 2: heard me talk about. And what they've been building is
• 40:44
  Speaker 2: this new tool called Stanley that uses AI, looks at
• 40:47
  Speaker 2: your feed, looks at your tone of voice, looks at
• 40:49
  Speaker 2: your history, looks at your best performing posts and tells
• 40:52
  Speaker 2: you what you should post, makes those posts for you.
• 40:54
  Speaker 2: You can also just use it for inspiration. And sometimes
• 40:57
  Speaker 2: what we need when we're thinking about doing a post
• 40:59
  Speaker 2: for our social media channels is inspiration. Building an audience
• 41:03
  Speaker 2: has fundamentally changed my life, and I think it could
• 41:05
  Speaker 2: change yours too. So I'm inviting you to give this
• 41:08
  Speaker 2: new tool a shot and let me know what you think.
• 41:10
  Speaker 2: All you have to do is search coach.stan.store now to
• 41:13
  Speaker 2: get started. A friend of mine who knows some of
• 41:17
  Speaker 2: these people sat me down once upon a time in London.
• 41:20
  Speaker 2: He's actually said this a few times to me, but
• 41:22
  Speaker 2: I remember one particular conversation where he says that some
• 41:26
  Speaker 2: of these AI CEOs predict the probability of extinction at being,
• 41:31
  Speaker 2: I think he said 7%. I don't know why I
• 41:33
  Speaker 2: have that number in my head, but I remember it
• 41:34
  Speaker 2: being less than 10%. And the point he was making
• 41:36
  Speaker 2: to me was that even if it was 1%, like
• 41:39
  Speaker 2: if there was a hundred buttons on this table now,
• 41:42
  Speaker 2: and one of them would end the world, would I
• 41:44
  Speaker 2: dare press any of them? No. No. I wouldn't press
• 41:49
  Speaker 2: any of them, but he made the case to me
• 41:51
  Speaker 2: that these AI CEOs are very smart and they understand
• 41:53
  Speaker 2: super intelligence and that they think actually if there was
• 41:55
  Speaker 2: a hundred buttons on this table right now, maybe 10
• 41:57
  Speaker 2: of them could end the world. I've heard you say,
• 42:00
  Speaker 2: I think it was on the Daily Show, the interview
• 42:02
  Speaker 2: you did, you said that you think there's a 70%
• 42:04
  Speaker 2: chance of human extinction due to AI.
• 42:06
  Speaker 1: I wouldn't say human extinction exactly. I would say something
• 42:08
  Speaker 1: like 70% chance that this goes horribly wrong like human extinction,
• 42:11
  Speaker 1: but that's just one of several possibilities.
• 42:13
  Speaker 2: But yeah, basically.
• 42:16
  Speaker 1: Like, for example, possibly the AIs take over And then
• 42:19
  Speaker 1: don't actually kill everyone. You know, maybe they do something else.
• 42:23
  Speaker 1: Like just, just because they've taken over doesn't mean they're
• 42:25
  Speaker 1: just definitely going to kill us.
• 42:26
  Speaker 2: Right.
• 42:26
  Speaker 1: They might, but they could do something else. So that's
• 42:28
  Speaker 1: why that's, that's why I don't usually say like 70%
• 42:31
  Speaker 1: chance of like actual human extinction, but 70% chance of
• 42:33
  Speaker 1: like something like AI is taking over some, some sort
• 42:37
  Speaker 1: of very big catastrophe like that.
• 42:38
  Speaker 2: That could lead to humans. I guess I've got two
• 42:40
  Speaker 2: points there, which is you've been around these CEOs. I mean,
• 42:43
  Speaker 2: you've worked for Sam Altman at OpenAI before you quit.
• 42:46
  Speaker 2: Do you think that they think there's a chance of
• 42:48
  Speaker 2: human extinction?
• 42:49
  Speaker 1: Yes. But I think that an important thing to understand
• 42:53
  Speaker 1: is that people sort of believe what they need to
• 42:56
  Speaker 1: believe in order to think that they're good people and
• 42:59
  Speaker 1: that they need to keep doing what they're doing.
• 43:01
  Speaker 2: This is what rationalization is. And so I think that
• 43:04
  Speaker 2: the tech CEOs have genuinely convinced themselves that probably things
• 43:09
  Speaker 2: are going to be fine and that the way to
• 43:11
  Speaker 2: make things fine is for them to keep doing what
• 43:12
  Speaker 2: they're doing. And they need to make sure that Like,
• 43:15
  Speaker 2: you know, Sam needs to make... Sam's probably thinking, like,
• 43:17
  Speaker 2: can't let Dario or Elon get there first. You know,
• 43:21
  Speaker 2: I know Dario's thinking Sam can't get there first. Elon's
• 43:24
  Speaker 2: thinking that.
• 43:24
  Speaker 1: Like, you know, they've all probably convinced themselves that, like, oh, yeah, like,
• 43:27
  Speaker 1: maybe it'll go horribly wrong, but, like, probably it's going
• 43:30
  Speaker 1: to be fine and probably, you know, I should be
• 43:34
  Speaker 1: the one in charge.
• 43:35
  Speaker 2: It appears to me that Anthropic are the only ones
• 43:37
  Speaker 2: that are talking about the potential chance of extinction or
• 43:40
  Speaker 2: a catastrophic event or... the down the real downside still
• 43:44
  Speaker 2: they seem to be the only ones that are still
• 43:46
  Speaker 2: publishing on it and now they're actually becoming the enemy
• 43:48
  Speaker 2: in many respects of the the tech industry in san
• 43:51
  Speaker 2: francisco i'm watching a lot of interviews and it's everyone's
• 43:53
  Speaker 2: attacking dario because he's saying listen things could go bad
• 43:56
  Speaker 2: they're calling him a doomer and questioning his incentives even
• 43:59
  Speaker 2: with mythos which is a a claude model that they
• 44:02
  Speaker 2: started to warn the world about again he is attacked
• 44:05
  Speaker 2: immediately for saying that yeah my question is do you
• 44:08
  Speaker 2: see him as being slightly different from sam in this regard.
• 44:11
  Speaker 1: Yeah, I mean, it seems like Anthropic and Stereo have
• 44:15
  Speaker 1: been more willing to say and do things that are
• 44:19
  Speaker 1: costly to their bottom line, at.
• 44:22
  Speaker 2: Least in the last year or so. That's an example
• 44:24
  Speaker 2: of it.
• 44:25
  Speaker 1: Like, I don't think that really wins them favors in
• 44:26
  Speaker 1: the administration or among their investors to say that type
• 44:30
  Speaker 1: of thing.
• 44:31
  Speaker 2: And, you know, a better example is just the whole
• 44:34
  Speaker 2: fight between.
• 44:35
  Speaker 1: The Department of War and Anthropic was an example of
• 44:36
  Speaker 1: them doing something that, like, cost them a lot of
• 44:39
  Speaker 1: money and even more importantly, cost them a lot of power.
• 44:41
  Speaker 1: for something like, like they could have just signed the contract,
• 44:45
  Speaker 1: you know, that said, I really don't want to be
• 44:48
  Speaker 1: in a situation where we're like, which CEO is the
• 44:50
  Speaker 1: least bad CEO.
• 44:52
  Speaker 2: Let's support that one. You know, like none of these
• 44:54
  Speaker 2: people should be trusted, uh, with that much power, basically.
• 44:57
  Speaker 2: Nobody should. Nobody should. Regardless. Regardless.
• 45:00
  Speaker 1: Yeah.
• 45:01
  Speaker 2: So on this point of the buttons, you do believe
• 45:04
  Speaker 2: that they think there's a credible chance of extinction. Yeah.
• 45:07
  Speaker 1: But they've convinced themselves that like, it's probably fine. And
• 45:10
  Speaker 1: also it'll be even worse if I'm not doing it.
• 45:13
  Speaker 1: That's what they'll say inside the companies too. People will
• 45:15
  Speaker 1: be like, okay, well, if we stop, what about the
• 45:18
  Speaker 1: other guys? They're not going to stop.
• 45:20
  Speaker 2: Yeah, this has always been why I've had this outstanding question,
• 45:23
  Speaker 2: which is how does this not go bad when human
• 45:25
  Speaker 2: incentives seem to rule the day when you look at
• 45:27
  Speaker 2: history and all of the human incentives are saying, well,
• 45:30
  Speaker 2: you're damned if you do, i.e. you're damned if you
• 45:33
  Speaker 2: carry on developing these bigger and bigger and bigger AI brains,
• 45:36
  Speaker 2: but you're also then damned if you don't from a
• 45:38
  Speaker 2: geographical perspective because the United States will lose to that
• 45:41
  Speaker 2: country or this company will lose to that company. So
• 45:43
  Speaker 2: when you just look at human incentives and just purely
• 45:46
  Speaker 2: incentives and disincentives, how does this end? Well, it carries
• 45:49
  Speaker 2: on going. Seems like it.
• 45:51
  Speaker 1: I mean, there is a caveat to that, which is
• 45:53
  Speaker 1: a hopeful caveat, which is that, first of all, if
• 45:56
  Speaker 1: the world wakes up to all of this, then there
• 45:58
  Speaker 1: can be a more serious conversation about regulation and international
• 46:03
  Speaker 1: treaties and things like that. And that can change the incentives, right?
• 46:06
  Speaker 1: So The government could come in and say like, actually,
• 46:09
  Speaker 1: here's some rules that you all have to follow. And
• 46:12
  Speaker 1: because they're rules that you all have to follow, then
• 46:14
  Speaker 1: you're not incentivized to like break them anymore because you
• 46:18
  Speaker 1: get punished if you break them and everyone else is
• 46:21
  Speaker 1: also following them too.
• 46:22
  Speaker 2: And so, you know, it's fine.
• 46:23
  Speaker 1: So, so there is that sort of like ray of
• 46:25
  Speaker 1: hope that like we can change the incentives if the government,
• 46:30
  Speaker 1: especially the US government, but then later other countries act
• 46:34
  Speaker 1: to change the incentives. But that's not going to happen
• 46:36
  Speaker 1: until people sort of wake up to all of this.
• 46:38
  Speaker 1: The second thing is that even individually, at some point,
• 46:43
  Speaker 1: you know, Dario or Sam or Elon might realize that like,
• 46:47
  Speaker 1: actually it's like not even in their own interest to,
• 46:50
  Speaker 1: to keep racing unilaterally. And it, the problem with that
• 46:54
  Speaker 1: is it's only if it gets extremely obvious and extremely dire.
• 46:56
  Speaker 1: So like in, in AI 2027, in that scenario, there's
• 47:01
  Speaker 1: this choice point that I mentioned. And in one case,
• 47:03
  Speaker 1: the AIs are misaligned. In the other case, the AIs
• 47:05
  Speaker 1: are aligned. At that choice point, we have like one
• 47:08
  Speaker 1: branch that depicts the misalignment ending and one branch that
• 47:11
  Speaker 1: depicts like they slow down a bit and solve the
• 47:13
  Speaker 1: alignment issues. The instigator for that choice point is they
• 47:16
  Speaker 1: see some evidence that their AI might be misaligned and
• 47:19
  Speaker 1: plotting against them, right? So if you actually see that evidence,
• 47:23
  Speaker 1: then it's like, oh gosh, maybe we shouldn't put it
• 47:28
  Speaker 1: in charge of everything and let it rip, you know?
• 47:31
  Speaker 1: Because that's evidence is staring us right in the face
• 47:33
  Speaker 1: that it's, it's untrustworthy, you know, but if they don't
• 47:36
  Speaker 1: see that sort of very clear evidence, then I think
• 47:39
  Speaker 1: they're going to convince themselves that they need to keep going,
• 47:41
  Speaker 1: you know, but maybe they will see very clear evidence
• 47:43
  Speaker 1: like that. In which case, even if we don't have regulation,
• 47:46
  Speaker 1: they might just sort of voluntarily stop. Um, so that's
• 47:49
  Speaker 1: the second ray of hope. Like overall, I don't think
• 47:52
  Speaker 1: that we're like definitely doomed, you know, like I said, 70%,
• 47:55
  Speaker 1: but like I could see it working out pretty well
• 47:58
  Speaker 1: as well.
• 47:59
  Speaker 2: Hmm. What about jobs? Yeah.
• 48:04
  Speaker 1: So I think I'm excited to at some point get
• 48:06
  Speaker 1: into the new thing, which is the more optimistic, positive vision.
• 48:11
  Speaker 1: And that will have a lot to say about this
• 48:13
  Speaker 1: because in the prediction, you know, in AI 2027, by
• 48:18
  Speaker 1: the time everyone loses their jobs, there are worse things happening.
• 48:21
  Speaker 1: Or like, it's kind of like too late by that point.
• 48:24
  Speaker 1: But yes, like once, if, I mean, just think about it.
• 48:27
  Speaker 1: If the companies do manage to build superintelligence, then by definition,
• 48:31
  Speaker 1: they're going to be able to take almost all the
• 48:33
  Speaker 1: jobs or all the jobs, right? Because it's better, faster,
• 48:36
  Speaker 1: cheaper than the best humans at everything.
• 48:38
  Speaker 2: And that's, again, the timeline is by the end of
• 48:40
  Speaker 2: sort of 2030, you reckon, you think superintelligence might arrive.
• 48:43
  Speaker 2: I'm trying to think about when we can start to
• 48:45
  Speaker 2: see job displacement in the economy. We're already starting to
• 48:47
  Speaker 2: see a little bit of it now, but not very much. Why?
• 48:50
  Speaker 2: Because they guys aren't good enough yet.
• 48:52
  Speaker 1: Like they're impressive, but they're not like, they're not just
• 48:56
  Speaker 1: a drop-in replacement for, for our human worker in almost
• 48:59
  Speaker 1: any field.
• 49:00
  Speaker 2: And do you think that'll be sudden?
• 49:02
  Speaker 1: I think it'll be sudden because of the intelligence explosion
• 49:06
  Speaker 1: dynamics or recursive self-improvement dynamics. So you could imagine a
• 49:10
  Speaker 1: different world where it's gradual.
• 49:13
  Speaker 2: And this is.
• 49:14
  Speaker 1: Maybe how it is in a lot of science fiction is,
• 49:17
  Speaker 1: you know, the AI is gradually get better at a
• 49:19
  Speaker 1: bunch of things and, you know, they gradually automate like
• 49:22
  Speaker 1: this one industry, like pharma, and then they automate like,
• 49:25
  Speaker 1: you know, steering drones, then they automate like driving cars
• 49:29
  Speaker 1: or something like that. Um, but what's different about the
• 49:32
  Speaker 1: real world is that the companies have converged on this
• 49:36
  Speaker 1: strategy of automating themselves first. you know, automating the AI
• 49:40
  Speaker 1: research process.
• 49:42
  Speaker 2: And so.
• 49:44
  Speaker 1: If they're allowed to continue with the strategy, we're not
• 49:47
  Speaker 1: going to see like, you know, the robo taxis and
• 49:51
  Speaker 1: like the plumber robots and, you know, the lawyer AIs.
• 49:56
  Speaker 1: We're not going to see that sort of like broad
• 49:57
  Speaker 1: diffusion of AI into the economy happening first, because that's
• 50:02
  Speaker 1: not what they're focusing on first. They're focusing on automating themselves,
• 50:05
  Speaker 1: automating their own research so that they can do everything
• 50:07
  Speaker 1: that they're doing faster and And they want that to
• 50:10
  Speaker 1: sort of get going and get to, you know, very
• 50:14
  Speaker 1: high levels of intelligence, very high levels of general intelligence, um,
• 50:18
  Speaker 1: and then deploy more out into the economy.
• 50:20
  Speaker 2: Right.
• 50:21
  Speaker 1: So by the time it's actually coming for like all
• 50:24
  Speaker 1: these different jobs, they will have had fully autonomous AI
• 50:28
  Speaker 1: research happening for months, maybe years, you know? And that
• 50:33
  Speaker 1: means that like the AIs will be vastly superhuman at
• 50:36
  Speaker 1: AI research And probably also vastly superhuman at lots of
• 50:39
  Speaker 1: other things, just as a side effect, you know. If
• 50:42
  Speaker 1: you're wondering what this looks like, well, we wrote about
• 50:45
  Speaker 1: what it looks like. It's sort of like this wave
• 50:48
  Speaker 1: smashing through the economy after they do the intelligence explosion internally.
• 50:52
  Speaker 2: What I'm hearing there is that because the AI will
• 50:55
  Speaker 2: be able to improve itself and train itself, it'll be
• 50:57
  Speaker 2: getting better at everything at once. And then it will
• 51:00
  Speaker 2: be released at kind of once automatically.
• 51:02
  Speaker 1: Yes, but it's not even exactly that because even if
• 51:05
  Speaker 1: it's mostly just getting better at the things that it's doing,
• 51:07
  Speaker 1: like research, that'll have some spillover effects to other skills
• 51:12
  Speaker 1: as well. And then when it turns to focusing on
• 51:15
  Speaker 1: those other skills, it'll be able to do them very fast.
• 51:17
  Speaker 2: What jobs remain in such a scenario, do you think?
• 51:21
  Speaker 2: I think that's actually a political question, not a technical question. Because?
• 51:24
  Speaker 1: Because on a technical level, all the jobs can be
• 51:27
  Speaker 1: done by the AIs if they've reached that level.
• 51:31
  Speaker 2: And so- It's a question of what jobs are allowed
• 51:34
  Speaker 2: for them to do. And what kind of jobs wouldn't
• 51:36
  Speaker 2: be allowed, do you think?
• 51:37
  Speaker 1: That depends on who's in charge. So there'd be some
• 51:40
  Speaker 1: sort of political conversation about like what we're going to
• 51:42
  Speaker 1: allow and disallow. I mean, in this scenario, the humans
• 51:44
  Speaker 1: are still controlling them, the AIs. Depends on what you
• 51:47
  Speaker 1: mean by control, right? So there's like, do the AIs
• 51:50
  Speaker 1: actually have the goals and values that you want them
• 51:51
  Speaker 1: to have? And are they going to robustly do that
• 51:55
  Speaker 1: and behave as intended into the future? And then there's like,
• 51:57
  Speaker 1: are they obeying your orders for now?
• 51:59
  Speaker 2: Are they obeying the orders is really what I'm saying. Yeah.
• 52:01
  Speaker 1: So like, Even in AI 2007, in the scenario where
• 52:03
  Speaker 1: the AIs take over and kill everyone, there's a period of, like,
• 52:06
  Speaker 1: several years where they're still obeying orders. And they're, you know,
• 52:10
  Speaker 1: taking some jobs but not other jobs. And they're helping
• 52:13
  Speaker 1: to make better weapons that the U.S. government can use to, like,
• 52:17
  Speaker 1: do its arms race with China and so forth. And
• 52:20
  Speaker 1: that's why they're able to get so much power so
• 52:23
  Speaker 1: quickly is because the governments and the corporations and so
• 52:27
  Speaker 1: forth trust them and just deliberately... deploying them into all
• 52:31
  Speaker 1: of these positions because it thinks that things are fine.
• 52:35
  Speaker 1: But because these things are neural nets, you can't just
• 52:38
  Speaker 1: look inside and see what it's really thinking.
• 52:40
  Speaker 2: You can't really tell. I think this is a really
• 52:43
  Speaker 2: important point because unlike software, where we can look at
• 52:45
  Speaker 2: the code and see what's going on theoretically, with AI,
• 52:48
  Speaker 2: you're saying that we don't know why it's making the
• 52:51
  Speaker 2: decisions that it's making because we can't get inside.
• 52:53
  Speaker 1: One note of optimism is that it doesn't necessarily have
• 52:56
  Speaker 1: to be that way. There's a subfield of machine learning
• 52:59
  Speaker 1: called mechanistic interpretability. and a broader subfield called interpretability more generally,
• 53:03
  Speaker 1: that's trying to solve that problem and trying to take
• 53:06
  Speaker 1: these trained artificial neural nets and piece them apart and
• 53:10
  Speaker 1: understand like how the information is flowing and how the
• 53:14
  Speaker 1: decisions are being made, so to speak. The problem is just,
• 53:17
  Speaker 1: it's a very inherently hard problem. If you have 10
• 53:19
  Speaker 1: trillion connections to look at, you can look at any
• 53:22
  Speaker 1: particular group of them and be like, okay, so this
• 53:24
  Speaker 1: is how like this particular connection works. But like, how
• 53:27
  Speaker 1: do you get a sense of the whole? You know,
• 53:29
  Speaker 1: how do you get a sense of like- What's happening
• 53:30
  Speaker 1: at a high level? And the answer is, well, it
• 53:32
  Speaker 1: might be impossible, but people are working on it and
• 53:34
  Speaker 1: they are making progress. And if they can make enough progress,
• 53:38
  Speaker 1: then we're in a very different and.
• 53:39
  Speaker 2: Much brighter world.
• 53:41
  Speaker 1: I think that it would be much less likely for
• 53:43
  Speaker 1: us to get into those loss of control scenarios if
• 53:46
  Speaker 1: we could just actually see what our AIs were thinking
• 53:48
  Speaker 1: and why and how at any given time, right?
• 53:51
  Speaker 2: Yeah.
• 53:52
  Speaker 1: So we would still have the other problems to worry about,
• 53:54
  Speaker 1: but at least we could mostly solve that one.
• 53:56
  Speaker 2: It is pretty crazy to think that we're building a technology,
• 53:58
  Speaker 2: a brain that we don't understand. Yeah, it's pretty crazy.
• 54:01
  Speaker 2: I mean, it's one of those things where like... In
• 54:03
  Speaker 2: a movie, like a sci-fi movie, a bunch of scientists
• 54:05
  Speaker 2: stood around this big brain and they're all just like,
• 54:07
  Speaker 2: they're making it more, they're feeding it. Yeah. But they
• 54:10
  Speaker 2: don't really know what the fuck... Yeah, I mean, it's
• 54:11
  Speaker 2: kind of just like obviously a dangerous thing to be doing. Yeah.
• 54:14
  Speaker 1: But we're doing it anyway because of this history of
• 54:17
  Speaker 1: how the field has developed in the last 10 years where,
• 54:20
  Speaker 1: you know, people were like, oh, wow, yeah, that's obviously dangerous. Oh, no,
• 54:23
  Speaker 1: what if someone else did it and did a bad
• 54:25
  Speaker 1: job of it? therefore we should do it and do
• 54:27
  Speaker 1: a good job of it. And now they're in this
• 54:29
  Speaker 1: race where they're racing each other. And they're also under
• 54:32
  Speaker 1: all sorts of political pressure to like pretend that it's
• 54:34
  Speaker 1: not as bad as it seems because they don't want
• 54:37
  Speaker 1: to like anger their investors. They don't want to anger
• 54:39
  Speaker 1: the White House.
• 54:41
  Speaker 2: One of the key questions we had from our audience
• 54:43
  Speaker 2: was which, and I kind of asked you this in part,
• 54:45
  Speaker 2: but which jobs are genuinely likely to survive AI and
• 54:49
  Speaker 2: what skills should people slash students focus on over the
• 54:52
  Speaker 2: next 10 years?
• 54:53
  Speaker 1: That's kind of like, Like imagine if you were someone
• 54:57
  Speaker 1: living in Mexico in like 1500 and then you hear
• 55:02
  Speaker 1: that like the conquistadors are coming. You could be asking
• 55:06
  Speaker 1: yourself like, okay, well, what sort of job should I
• 55:08
  Speaker 1: be switching to, to like survive this transition? But like
• 55:12
  Speaker 1: you have a lot more to worry about besides that.
• 55:14
  Speaker 1: But yes, I think I would say that like if
• 55:17
  Speaker 1: we managed to avoid the loss of control problem and
• 55:20
  Speaker 1: we end up with humans still in charge of the
• 55:22
  Speaker 1: AIs and humans can like say what the AI's goals
• 55:24
  Speaker 1: and values are supposed to be, even as they become
• 55:27
  Speaker 1: much smarter than humans and even as they run the
• 55:29
  Speaker 1: whole economy, then probably there will be regulation that protects
• 55:32
  Speaker 1: some areas. And you can try to guess at what
• 55:35
  Speaker 1: those areas might be. Maybe stuff that's more like judges, potentially.
• 55:41
  Speaker 2: What about podcasters? Be honest. Probably not podcasters, I think.
• 55:48
  Speaker 2: Stuff like, you know, being a nanny, maybe, right? Like,
• 55:52
  Speaker 2: I think that.
• 55:53
  Speaker 1: Even if there's a robot nanny that's really, really good,
• 55:55
  Speaker 1: I think a bunch of people might prefer to have
• 55:57
  Speaker 1: an actual human because they might be creeped out by
• 55:59
  Speaker 1: the idea of a really good robot nanny. So you
• 56:01
  Speaker 1: can sort of reason like that. There's also stuff that
• 56:05
  Speaker 1: might be legally protected. Maybe judges, for example, are going
• 56:08
  Speaker 1: to be legally required to be humans and not robots.
• 56:10
  Speaker 2: Some people say, though, there's going to be so many
• 56:12
  Speaker 2: jobs created that we can't foresee right now like there
• 56:14
  Speaker 2: was in the Industrial Revolution or the internet boom or whatever.
• 56:18
  Speaker 1: The problem with that is that past technological advancements, have
• 56:23
  Speaker 1: been more narrow. They've like automated some things, but not everything.
• 56:28
  Speaker 1: But we are talking about a hypothetical future situation in
• 56:30
  Speaker 1: which everything gets automated. So there isn't any new job
• 56:34
  Speaker 1: that you could do that the AI couldn't also do,
• 56:37
  Speaker 1: except if it's like protected by regulation or something.
• 56:39
  Speaker 2: That's, that's, that's also a thing.
• 56:42
  Speaker 1: But so like, for example, right now there's this sort
• 56:44
  Speaker 1: of like cycle where, you know, the AI has learned
• 56:49
  Speaker 1: to do a certain thing, like write copy or, like
• 56:53
  Speaker 1: draft code or debug something. And then humans who used
• 56:57
  Speaker 1: to do that thing switch to managing AIs or switch
• 57:00
  Speaker 1: to doing the other stuff that the AIs can't do.
• 57:03
  Speaker 1: And that's why there's been this dynamic historically of new
• 57:07
  Speaker 1: jobs opening up and people flooding to them. But if
• 57:10
  Speaker 1: it gets to the point where the AIs can do
• 57:12
  Speaker 1: everything that humans can do and better and faster and cheaper,
• 57:14
  Speaker 1: then whatever that new job is that you might've switched to,
• 57:17
  Speaker 1: the AIs can switch to that too.
• 57:18
  Speaker 2: And they'll already be better at it than you. Because
• 57:21
  Speaker 2: we haven't seen widespread unemployment yet in the economy, do
• 57:24
  Speaker 2: you think people are getting a little bit complacent? Because
• 57:26
  Speaker 2: what I'm seeing on my timeline is a lot of
• 57:27
  Speaker 2: people saying, I told you so. I told you everything
• 57:29
  Speaker 2: would be fine. And when you look at the US
• 57:32
  Speaker 2: unemployment rate, currently it's flat to slightly down. If you
• 57:36
  Speaker 2: look at the UK, it is up. The trend is
• 57:39
  Speaker 2: up compared to last year. We're at about 5% unemployment.
• 57:42
  Speaker 2: The US is at 4.2% unemployment. Yeah.
• 57:45
  Speaker 1: Basically, nobody has said that there would be mass unemployment
• 57:47
  Speaker 1: by now. Or at least we didn't say that. And
• 57:50
  Speaker 1: we were historically one of the more bullish people on
• 57:52
  Speaker 1: AI progress. In AI 2027, because of the dynamics that
• 57:55
  Speaker 1: we just described, the mass unemployment doesn't happen until 2028
• 57:58
  Speaker 1: or 2029 after they already have superintelligence. Because, again, the
• 58:02
  Speaker 1: companies aren't trying to cause mass unemployment as step one.
• 58:06
  Speaker 1: That's like step three. It's like step one, automate themselves.
• 58:10
  Speaker 1: Step two, have this recursive self-improvement to get to superintelligence.
• 58:15
  Speaker 1: Step three, expand out into the economy and automate everything.
• 58:17
  Speaker 2: And so.
• 58:18
  Speaker 1: This is really unfortunate from humanity's perspective because one might
• 58:23
  Speaker 1: have hoped that if there was this broad wave of
• 58:26
  Speaker 1: automation going through the economy, people would sit up and
• 58:28
  Speaker 1: pay attention and think about where all this is headed
• 58:31
  Speaker 1: and demand good regulations from the government. But that's not
• 58:37
  Speaker 1: actually what the strategy of the companies are taking. They're
• 58:39
  Speaker 1: going to be getting the superintelligence first and then doing
• 58:41
  Speaker 1: the broad wave of automation, which means that by the
• 58:43
  Speaker 1: time they're actually doing.
• 58:44
  Speaker 2: All of that, well... It's already going to be moving
• 58:48
  Speaker 2: very fast and the AI will already be very powerful.
• 58:50
  Speaker 2: In your 2027 report, so you wrote that in 2025,
• 58:53
  Speaker 2: but it is called AI 2027. You said that in
• 58:56
  Speaker 2: mid 2025, we'd have the autonomous employee, which is sort
• 58:59
  Speaker 2: of like AI agents taking instructions over Slack or Teams.
• 59:04
  Speaker 2: That happened. I've actually got an AI agent in my WhatsApp,
• 59:07
  Speaker 2: which I talk to. Of course, I've got Clawbot exploded, obviously,
• 59:09
  Speaker 2: around the world. And now, you know, Claude have talked
• 59:12
  Speaker 2: about their new Slack integration. But lots of people are
• 59:15
  Speaker 2: using agents now. And that happened, I'd say for us,
• 59:18
  Speaker 2: we really sort of caught on to it at the
• 59:19
  Speaker 2: start of 2026. You also said by 2026, companies begin
• 59:24
  Speaker 2: replacing entire corporate departments with AI agent subscriptions. 2027, the
• 59:27
  Speaker 2: final job. AI automates the job of the human AI
• 59:31
  Speaker 2: researchers themselves and begins the machine learning research to upgrade
• 59:34
  Speaker 2: and build the next generation of AIs. Yeah. Yeah.
• 59:37
  Speaker 1: So again, timelines, we are uncertain about how long it
• 59:40
  Speaker 1: will take to achieve these milestones in this scenario. They
• 59:44
  Speaker 1: happen at those times, but by the time we had
• 59:47
  Speaker 1: actually published the scenario, our timelines had shifted back a
• 59:49
  Speaker 1: little bit, specifically mine had. So like my 50% mark
• 59:54
  Speaker 1: was 2028 for that, for the full automation of AI
• 59:57
  Speaker 1: research milestone, not 2027. Uh, and then other people on
• 1:00:01
  Speaker 1: my team had more like 2030, 2031, things like that.
• 1:00:05
  Speaker 1: So I kind of want to like, maybe try to
• 1:00:08
  Speaker 1: illustrate this with, you know, we have like this probability distribution.
• 1:00:10
  Speaker 1: It's like a smeared out probability mass and like the
• 1:00:14
  Speaker 1: 50% mark is this particular year, but there's like a
• 1:00:17
  Speaker 1: lot of possibility that it happens years earlier or years later.
• 1:00:21
  Speaker 2: Right. What is this AI2040? So AI2047 was a best
• 1:00:26
  Speaker 2: guess prediction as to how things would actually go.
• 1:00:28
  Speaker 1: Yeah. AI 2040 plan A is our recommendation for how
• 1:00:31
  Speaker 1: things should go. So we called it AI 2040 because
• 1:00:35
  Speaker 1: in this scenario, they build super intelligence in 2040 instead
• 1:00:40
  Speaker 1: of much sooner because they delay things.
• 1:00:42
  Speaker 2: Why do they delay things?
• 1:00:44
  Speaker 1: To manage the risks and make sure that power is
• 1:00:47
  Speaker 1: distributed equitably. They basically like regulate AI development so that
• 1:00:52
  Speaker 1: it still continues, but at a slower, more reasonable pace
• 1:00:56
  Speaker 1: in a more transparent and safe way. and spread out
• 1:00:58
  Speaker 1: over more countries and companies. And as a result, they
• 1:01:02
  Speaker 1: get to superintelligence in 2040 instead of in, say, 2030.
• 1:01:06
  Speaker 1: And then we call it Plan A because, well, it's
• 1:01:09
  Speaker 1: our recommendation. Like, we've come up with a plan for
• 1:01:12
  Speaker 1: what government should do. And the scenario is an illustration
• 1:01:17
  Speaker 1: of what it might look like to implement that plan.
• 1:01:19
  Speaker 1: In a similar way to how AI-2027 is kind of
• 1:01:21
  Speaker 1: an illustration of what it might look like to do
• 1:01:24
  Speaker 1: what the companies are currently planning to do, if.
• 1:01:26
  Speaker 2: That makes sense. And is this wishful thinking or is
• 1:01:29
  Speaker 2: this what you think is going to happen? No, it's
• 1:01:31
  Speaker 2: definitely not what we think is going to happen. It's
• 1:01:33
  Speaker 2: not what you think is going to happen? No, no.
• 1:01:34
  Speaker 2: What we think is going to happen is still something
• 1:01:37
  Speaker 2: more like this, right? We don't expect the world to
• 1:01:39
  Speaker 2: listen to us, right? This is our recommendation. But we
• 1:01:43
  Speaker 2: hope that people do something like this and we think
• 1:01:45
  Speaker 2: it's possible. But it's not our prediction for what's going
• 1:01:49
  Speaker 2: to happen by default, you know? So I do want
• 1:01:51
  Speaker 2: to run through the plans, the potential plans, and also
• 1:01:54
  Speaker 2: plan A. But just to close off on how things
• 1:01:57
  Speaker 2: might look after the year, because I think I wanted
• 1:01:59
  Speaker 2: to touch on robotics too. And I've got this graph here,
• 1:02:01
  Speaker 2: which talks about share of labor output, which I found
• 1:02:05
  Speaker 2: to be quite striking. I've been sat here wondering as
• 1:02:07
  Speaker 2: an employer who employs hundreds and hundreds of people, when
• 1:02:11
  Speaker 2: all this stuff is going to happen. And, you know,
• 1:02:12
  Speaker 2: we're still hiring more people as things stand. There are
• 1:02:16
  Speaker 2: some roles where our consideration is changing, shifting considerably. And
• 1:02:21
  Speaker 2: I'd have to say that, you know, we're probably in
• 1:02:23
  Speaker 2: the phase where our teams are AI powered and they're
• 1:02:25
  Speaker 2: using agents to do some of their work now. But
• 1:02:28
  Speaker 2: I'm wondering as an employer, like, when is it, when
• 1:02:30
  Speaker 2: does this happen? Yeah, great question. So if we could
• 1:02:33
  Speaker 2: maybe zoom in on this a little bit. Yeah, we'll
• 1:02:35
  Speaker 2: put it on the screen.
• 1:02:36
  Speaker 1: So this is in the AI 2040 plan A scenario.
• 1:02:39
  Speaker 1: And notably in that scenario, there's significant regulation introduced in
• 1:02:44
  Speaker 1: 2029 that slows down the pace of AI development. In
• 1:02:47
  Speaker 1: the scenario, they do that sort of at the last moment.
• 1:02:49
  Speaker 1: So in the scenario, If they hadn't done that, then
• 1:02:52
  Speaker 1: it was about to take off similar to how it
• 1:02:54
  Speaker 1: does in AI 2027. But as you can see, like
• 1:02:57
  Speaker 1: in the scenario, there's still a bunch of jobs at
• 1:03:02
  Speaker 1: the point that they implement it. And this gets back
• 1:03:04
  Speaker 1: to what I was saying earlier is that if you
• 1:03:05
  Speaker 1: wait until most people have lost their jobs to regulate
• 1:03:09
  Speaker 1: the AI companies, that's already too late because they will
• 1:03:13
  Speaker 1: probably already have super intelligent AI by then because their
• 1:03:16
  Speaker 1: strategy is to first get super intelligent AI by then.
• 1:03:18
  Speaker 2: All that stuff. And I think you say that it
• 1:03:19
  Speaker 2: would collapse the economy and cause even more harm to
• 1:03:22
  Speaker 2: suddenly regulate something that all of us and all of
• 1:03:24
  Speaker 2: our lives were then at that point relying on. Oh,
• 1:03:26
  Speaker 2: but it's a risk well worth taking.
• 1:03:27
  Speaker 1: I mean, it's true that right now a lot of
• 1:03:30
  Speaker 1: people use AI for a lot of things, but if
• 1:03:32
  Speaker 1: we could somehow slow or halt AI development now to
• 1:03:36
  Speaker 1: set up a better way to do it, that would
• 1:03:37
  Speaker 1: be well worth it, even though there would be significant costs.
• 1:03:41
  Speaker 2: But you can't over here, right? Can you? at this
• 1:03:43
  Speaker 2: point where AI and robotics are doing most of the
• 1:03:45
  Speaker 2: labor output. That's right.
• 1:03:47
  Speaker 1: But in this scenario, in the AI 2040 Plan A scenario,
• 1:03:50
  Speaker 1: they put in the regulations in 2029. And then they
• 1:03:53
  Speaker 1: slowly and carefully develop AI in a way that avoids
• 1:03:57
  Speaker 1: all the problems, which we can get into in a
• 1:03:59
  Speaker 1: little bit. And so eventually, yes, eventually the AIs take
• 1:04:02
  Speaker 1: the jobs. Eventually, basically the whole economy is run by
• 1:04:05
  Speaker 1: AIs and robots. But it happens gradually over the course
• 1:04:08
  Speaker 1: of the 2030s instead of happening in this sort of
• 1:04:12
  Speaker 1: crazy shock, you know, a year later, right? Because in
• 1:04:16
  Speaker 1: this scenario, they don't let the companies recursively self-improve and
• 1:04:21
  Speaker 1: get to super intelligence as fast as possible. Instead, they
• 1:04:23
  Speaker 1: regulate AI development so that the core capabilities of the
• 1:04:27
  Speaker 1: AIs are improving at a more reasonable pace and also
• 1:04:30
  Speaker 1: in a more transparent way so that the scientific community
• 1:04:33
  Speaker 1: can see what's going on and help make it safe.
• 1:04:36
  Speaker 2: But it's, uh, I guess I noticed here that in
• 1:04:39
  Speaker 2: both your scenarios, eventually ai and robotics do pretty much
• 1:04:42
  Speaker 2: all the jobs yes so you kind of side there
• 1:04:45
  Speaker 2: with elon when elon says that working will be a choice.
• 1:04:50
  Speaker 1: Uh because i mean yes i mean if it by
• 1:04:55
  Speaker 1: definition if it can do all the things then it
• 1:04:57
  Speaker 1: can do all the things i think that there's a
• 1:05:00
  Speaker 1: question of like should we allow there to be ais
• 1:05:02
  Speaker 1: that can do all the things right some people think
• 1:05:05
  Speaker 1: that the answer is no and we should just shut
• 1:05:07
  Speaker 1: it all down and prevent these types of AIs from
• 1:05:10
  Speaker 1: being created in the first place. And we're actually kind
• 1:05:13
  Speaker 1: of sympathetic to that. We have our, should we bring
• 1:05:15
  Speaker 1: up the plans diagram?
• 1:05:16
  Speaker 2: Yeah. Thanks. Yeah.
• 1:05:19
  Speaker 1: So our scenario is called AI 2040 Plan A. It's
• 1:05:23
  Speaker 1: a scenario in which they slow down AI development to
• 1:05:26
  Speaker 1: make a super intelligence happen in 2040 instead of earlier.
• 1:05:29
  Speaker 1: And Plan A is our recommendation. So this is sort
• 1:05:31
  Speaker 1: of illustrating our recommendation. But for comparison, we made like
• 1:05:34
  Speaker 1: mini scenarios illustrating different alternative plans. which we call Plan S,
• 1:05:39
  Speaker 1: Plan B, Plan C, and Plan D. Plan D is
• 1:05:42
  Speaker 1: basically the same thing that happens in AI 2027. The
• 1:05:46
  Speaker 1: race continues. There's very little regulation. You can read about
• 1:05:50
  Speaker 1: that in AI 2027. Plan C, also very similar to
• 1:05:53
  Speaker 1: what happens in the slowdown ending of AI 2027, where
• 1:05:55
  Speaker 1: they solve the alignment problems. So in that ending, they
• 1:05:59
  Speaker 1: slow down a little bit, pivot more resources to AI
• 1:06:03
  Speaker 1: alignment and AI safety research, get lucky and succeed, and
• 1:06:06
  Speaker 1: now they have aligned AIs. And then they speed up
• 1:06:09
  Speaker 1: again and take all the jobs and beat China and
• 1:06:12
  Speaker 1: all those things. Plan B is, it's kind of like
• 1:06:17
  Speaker 1: plan C in that, well, basically in plan B, you're
• 1:06:23
  Speaker 1: being more aggressive towards China and you're like taking actions
• 1:06:27
  Speaker 1: to sabotage or cyber attack them to like keep them
• 1:06:29
  Speaker 1: behind so that you have more breathing room to solve
• 1:06:32
  Speaker 1: the alignment problems yourself.
• 1:06:34
  Speaker 2: Plan A is our recommendation.
• 1:06:36
  Speaker 1: It's, uh, domestic regulation, and then an international deal to
• 1:06:41
  Speaker 1: continue building AI, but in a much better way. Plan
• 1:06:44
  Speaker 1: S is shut it all down. If you want to
• 1:06:46
  Speaker 1: have a future where there aren't AIs running around that
• 1:06:50
  Speaker 1: can do everything better and faster than humans, you kind
• 1:06:53
  Speaker 1: of want something like Plan S.
• 1:06:55
  Speaker 2: What do you want? Plan A is our recommendation.
• 1:06:58
  Speaker 1: I think that I'm sympathetic to Plan S, but for
• 1:07:01
  Speaker 1: reasons we explain, we recommend Plan A instead.
• 1:07:04
  Speaker 2: And what do you think is most probable, if you're
• 1:07:07
  Speaker 2: being honest? Plan D. Which is that they.
• 1:07:09
  Speaker 1: Just kind of go. Yeah, 2007 type of thing where
• 1:07:10
  Speaker 1: they keep racing. They don't really slow down significantly. And
• 1:07:16
  Speaker 1: things happen extremely fast. The diagram sort of explains like
• 1:07:19
  Speaker 1: roughly the reasoning behind this too. So like there's this
• 1:07:22
  Speaker 1: high level thing of like, do you want to keep
• 1:07:25
  Speaker 1: racing as fast as possible to make the AI smarter
• 1:07:27
  Speaker 1: and smarter to put them in charge of more things
• 1:07:30
  Speaker 1: so that we can beat China? You know, if you're
• 1:07:33
  Speaker 1: happy with that, then you can get down into this
• 1:07:36
  Speaker 1: variation of options here. If you are worried about that, well,
• 1:07:41
  Speaker 1: you get to something like this. There's more different options
• 1:07:44
  Speaker 1: besides these, but this is kind of like the ones
• 1:07:46
  Speaker 1: that we could compress onto a screen.
• 1:07:50
  Speaker 2: Do you have children? Yeah, I have two children. It's
• 1:07:55
  Speaker 2: kind of sad.
• 1:07:56
  Speaker 1: Like, I think that one way or another, this will
• 1:08:00
  Speaker 1: probably all be over by the time they're old enough
• 1:08:01
  Speaker 1: to join the workforce. So I don't think they'll ever
• 1:08:05
  Speaker 1: join the workforce. When you say this will be all
• 1:08:08
  Speaker 1: over by the time they join the world, what do
• 1:08:09
  Speaker 1: you mean by this will be all over? So these
• 1:08:14
  Speaker 1: milestones that I described, like AI is automating the AI research,
• 1:08:17
  Speaker 1: AI is getting super intelligent. AI is then exploding out
• 1:08:22
  Speaker 1: into the economy, taking the jobs, building robot factories to
• 1:08:25
  Speaker 1: build more robots, to build more factories, et cetera. GDP
• 1:08:28
  Speaker 1: is starting to go vertical. That sort of thing is
• 1:08:31
  Speaker 1: what I mean. Like all of those events transpiring. Maybe
• 1:08:34
  Speaker 1: there's like 10, 20% chance or something that hits a
• 1:08:37
  Speaker 1: wall and none of this comes to pass, even if
• 1:08:41
  Speaker 1: you don't do anything.
• 1:08:43
  Speaker 2: How old's your oldest?
• 1:08:45
  Speaker 1: Six.
• 1:08:45
  Speaker 2: Six. Boy, girl? Girl. Girl. So your daughter comes to
• 1:08:49
  Speaker 2: you and says, dad, what should I study in school?
• 1:08:53
  Speaker 1: I mean, again, like if these radical transformations happen, then
• 1:08:58
  Speaker 1: the world would just look completely different. And what sort
• 1:09:00
  Speaker 1: of jobs you set yourself up for basically won't matter
• 1:09:02
  Speaker 1: that much, probably. I would say that the thing to
• 1:09:06
  Speaker 1: do is, well, A, try to make it actually go well.
• 1:09:09
  Speaker 1: Like if you can exert any influence at all on
• 1:09:11
  Speaker 1: history and how this all develops, you should be trying
• 1:09:13
  Speaker 1: very hard to steer the future in better directions. And
• 1:09:17
  Speaker 1: then separately from that, on a personal level, you should
• 1:09:20
  Speaker 1: focus on, well, being a good person and doing things
• 1:09:25
  Speaker 1: that are sort of good for their own sake rather
• 1:09:27
  Speaker 1: than good because they'll set you up for later employment
• 1:09:30
  Speaker 1: because that later employment is going to be very uncertain, basically.
• 1:09:34
  Speaker 2: Elon talks about this age of abundance we're heading towards.
• 1:09:37
  Speaker 2: Age of abundance.
• 1:09:38
  Speaker 1: There'll definitely be abundance. The question is, who controls the
• 1:09:42
  Speaker 1: abundance and what do they do with it, right? Are
• 1:09:46
  Speaker 1: the AIs controlled by anyone or are they doing their
• 1:09:48
  Speaker 1: own thing? And then if they are controlled by people,
• 1:09:51
  Speaker 1: who controls them and what do they do? And what's
• 1:09:54
  Speaker 1: the sort of like political structure governing how they make
• 1:09:57
  Speaker 1: those decisions?
• 1:09:58
  Speaker 2: I think it was Geoffrey Hinton that said to me,
• 1:09:59
  Speaker 2: he said, there's no example in nature where a more
• 1:10:03
  Speaker 2: intelligent species has less control than a less intelligent species.
• 1:10:11
  Speaker 2: Thus saying that we're quite arrogant to think that in
• 1:10:13
  Speaker 2: a world where there's this artificial brain that's a gazillion
• 1:10:17
  Speaker 2: times the size of mine, that I'm going to give
• 1:10:19
  Speaker 2: it orders. Yeah.
• 1:10:20
  Speaker 1: I mean, that's the thing is I think it's like,
• 1:10:24
  Speaker 1: that should be our default assumption is that like, well,
• 1:10:26
  Speaker 1: there's these brains. We can't see exactly what they're thinking.
• 1:10:30
  Speaker 1: We're going to make them smarter than us and put
• 1:10:31
  Speaker 1: them in charge of everything.
• 1:10:32
  Speaker 2: And then we're going to give them bodies. Yeah.
• 1:10:34
  Speaker 1: And then they're going to be autonomously building new factors
• 1:10:36
  Speaker 1: and so forth. And like, how is this supposed to
• 1:10:38
  Speaker 1: end well again? Like, isn't this just exactly like us
• 1:10:41
  Speaker 1: picking a new species that's then going to out-compete us
• 1:10:45
  Speaker 1: when it doesn't need us anymore? I think that is
• 1:10:48
  Speaker 1: just the default trajectory. Now, there's a whole argument we
• 1:10:50
  Speaker 1: can get into about ways that we could get off
• 1:10:52
  Speaker 1: of that default trajectory. So for example, there's research into
• 1:10:56
  Speaker 1: interpretability that I described previously. And if that research bears fruit,
• 1:10:59
  Speaker 1: then you will be able to actually see what they're thinking.
• 1:11:01
  Speaker 1: And then that would be an excellent tool for shaping
• 1:11:04
  Speaker 1: them and controlling them and making sure that they do
• 1:11:06
  Speaker 1: what we want. There's other sorts of AI alignment research
• 1:11:09
  Speaker 1: agendas that are making progress and if enough of those
• 1:11:13
  Speaker 1: agendas succeed sufficiently, we can avoid this problem. Of course,
• 1:11:17
  Speaker 1: also there's the regulatory side too, where like part of
• 1:11:19
  Speaker 1: what makes this difficult is that we're building these AIs
• 1:11:22
  Speaker 1: in race conditions, you know, like the companies are secretive
• 1:11:25
  Speaker 1: about their recipes for making these AIs because it's secrets
• 1:11:29
  Speaker 1: that they want to protect so that other people can't
• 1:11:31
  Speaker 1: copy them. And so a lot of it is happening,
• 1:11:34
  Speaker 1: you know, behind closed doors. Only a few people can
• 1:11:36
  Speaker 1: really see the recipes that they're using to train these
• 1:11:39
  Speaker 1: AIs and and so forth. And then oftentimes when the
• 1:11:42
  Speaker 1: AIs behave in unexpected ways or even just like blatantly
• 1:11:45
  Speaker 1: misaligned ways, sometimes that information doesn't really flow out to
• 1:11:48
  Speaker 1: the public because the companies are not really incentivized to
• 1:11:51
  Speaker 1: tell everyone about how they messed up and how their
• 1:11:53
  Speaker 1: AI is evil. It's just not very conducive to scientific
• 1:11:56
  Speaker 1: progress on these issues. If the regulatory system was different,
• 1:11:59
  Speaker 1: then perhaps we could be in a better situation, make
• 1:12:01
  Speaker 1: faster progress. Also, of course, we wouldn't be planning to
• 1:12:05
  Speaker 1: put these AIs in charge of everything as fast as possible.
• 1:12:08
  Speaker 1: And we wouldn't be planning to like let themselves improve,
• 1:12:10
  Speaker 1: you know, like these are choices that we could not make,
• 1:12:13
  Speaker 1: you know.
• 1:12:17
  Speaker 2: I don't speak Vietnamese, but this show can because of
• 1:12:20
  Speaker 2: AI video technology from our sponsor, HeyGen. I get messages
• 1:12:23
  Speaker 2: every single week from those of you listening to the
• 1:12:25
  Speaker 2: Diary of a CEO all around the world, and you
• 1:12:27
  Speaker 2: express how much impact it's had on you and your life.
• 1:12:29
  Speaker 2: And if that's true, then those conversations shouldn't only reach
• 1:12:32
  Speaker 2: people in English. HeyGen can take one recording of me
• 1:12:35
  Speaker 2: and deliver it in any language while keeping my voice, timing,
• 1:12:38
  Speaker 2: and expressions intact but you don't need a studio like
• 1:12:42
  Speaker 2: this to make it work for you record 15 seconds
• 1:12:46
  Speaker 2: of yourself and get an ai avatar that delivers studio
• 1:12:48
  Speaker 2: quality video in over 175 languages we're up to 20
• 1:12:53
  Speaker 2: languages now and we're not the only ones using it
• 1:12:56
  Speaker 2: hey jam is already used by 30 million people including
• 1:13:00
  Speaker 2: 85 of the fortune 100 whether you're building an audience
• 1:13:02
  Speaker 2: on social media launching an online course or rolling out
• 1:13:05
  Speaker 2: training across your team check out hey jam now Your
• 1:13:08
  Speaker 2: first three videos are totally free at heygen.com slash D-O-A-C.
• 1:13:13
  Speaker 2: That's H-E-Y-G-E-N.com slash D-O-A-C. See you there. Ilya was, as
• 1:13:20
  Speaker 2: you said, he was one of the leaders at OpenAI
• 1:13:21
  Speaker 2: and he left and he started his own company now,
• 1:13:23
  Speaker 2: Safe Superintelligence. Very curious name of a company, Safe Superintelligence
• 1:13:28
  Speaker 2: after leaving OpenAI. Did you ever get to work with him?
• 1:13:30
  Speaker 2: I wasn't directly working with him. I had a couple
• 1:13:32
  Speaker 2: of chats with him. Do you think he's genuinely concerned
• 1:13:35
  Speaker 2: as well?
• 1:13:36
  Speaker 1: I think he is, but I think he's, I think
• 1:13:39
  Speaker 1: he's similar to these other CEOs where, I mean, just
• 1:13:43
  Speaker 1: think about the sort of incentives that they're under, right?
• 1:13:45
  Speaker 1: Like they can sort of see the problem and then
• 1:13:49
  Speaker 1: they can be like, okay, but like, if I don't,
• 1:13:52
  Speaker 1: if I stop, if I quit my job and, or
• 1:13:55
  Speaker 1: do something else, that's not going to solve the problem
• 1:13:57
  Speaker 1: because the other CEOs are going to keep going. And
• 1:14:00
  Speaker 1: even if all of us didn't go, then maybe China
• 1:14:02
  Speaker 1: would keep going. So like, man, it seems like this
• 1:14:04
  Speaker 1: is just going to happen one way or another, whether
• 1:14:05
  Speaker 1: I do anything about it or not. I guess I
• 1:14:08
  Speaker 1: should be involved, you know, and like maybe I can
• 1:14:10
  Speaker 1: make it go well. And at any rate, like I
• 1:14:12
  Speaker 1: don't want to be out in the cold while these
• 1:14:13
  Speaker 1: other people I don't trust are in charge of everything.
• 1:14:16
  Speaker 1: So they all sort of like reason through all of
• 1:14:17
  Speaker 1: this and then convince themselves that like the thing to
• 1:14:20
  Speaker 1: do is for them to build it and to do
• 1:14:22
  Speaker 1: it better. And I think Ilya is just the latest
• 1:14:24
  Speaker 1: example of this. Elon's another example. Dario is another example.
• 1:14:29
  Speaker 1: You know, arguably OpenAI at the beginning, Sam was an example,
• 1:14:32
  Speaker 1: although like Elon and Daria were at OpenAI early on.
• 1:14:35
  Speaker 2: What do you think they should all do then?
• 1:14:37
  Speaker 1: So I think what should happen is some sort of
• 1:14:39
  Speaker 1: international regulation, or at least domestic regulation, similar to what
• 1:14:42
  Speaker 1: we described in Plan A.
• 1:14:43
  Speaker 2: Okay, so walk me through Plan A. Yeah.
• 1:14:46
  Speaker 1: So in this scenario, AI takes longer to get to
• 1:14:51
  Speaker 1: recursive self-improvement and full automation of AI research than it
• 1:14:54
  Speaker 1: does in AI 2007. We figured that we should try
• 1:14:56
  Speaker 1: to illustrate a range of different possibilities because we do
• 1:14:59
  Speaker 1: have those sort of uncertainty intervals.
• 1:15:01
  Speaker 2: So we chose 2030.
• 1:15:02
  Speaker 1: As the moment when full automation would finally be achieved
• 1:15:07
  Speaker 1: and things would really kick off. And then working backwards
• 1:15:09
  Speaker 1: from that, when's the last moment you could really have
• 1:15:12
  Speaker 1: good regulation?
• 1:15:13
  Speaker 2: 2029.
• 1:15:14
  Speaker 1: So in this scenario, AI progress slows down a little
• 1:15:17
  Speaker 1: bit naturally and the AI companies keep racing, but they
• 1:15:21
  Speaker 1: don't quite succeed in automating themselves in 2027 or in
• 1:15:25
  Speaker 1: 2028 or in 2029, but they're getting really close and
• 1:15:27
  Speaker 1: they're going to do it in 2030.
• 1:15:28
  Speaker 2: And then in 2029, the government steps in. and regulates them.
• 1:15:33
  Speaker 2: What regulations do they do? Well, they basically just shut
• 1:15:35
  Speaker 2: it down temporarily. Can I ask, how does the elections
• 1:15:41
  Speaker 2: overlay with your timeframes here? Because there's going to be
• 1:15:43
  Speaker 2: a big election, isn't there, in 2028? And it seems
• 1:15:46
  Speaker 2: now that sentiment has really, really turned against AI in
• 1:15:50
  Speaker 2: the general public and that it will be one of
• 1:15:52
  Speaker 2: the big ticket items on the ballot.
• 1:15:54
  Speaker 1: We think that it'll be maybe the most important issue
• 1:15:57
  Speaker 1: in the presidential election in 2028. I think a lot
• 1:16:00
  Speaker 1: of people, most people will be quite concerned about where
• 1:16:02
  Speaker 1: things are headed. And that's part of why we chose
• 1:16:06
  Speaker 1: to depict things the way that we're doing in this scenario,
• 1:16:08
  Speaker 1: because that helps explain why they might do this sort
• 1:16:10
  Speaker 1: of regulation in 2029, is that the voters have been
• 1:16:13
  Speaker 1: demanding it and the presidential candidates have been promising it.
• 1:16:15
  Speaker 2: And in this scenario, and in 2027, would the general
• 1:16:18
  Speaker 2: public have felt the consequences of AI much more severely
• 1:16:22
  Speaker 2: than they have now by then?
• 1:16:23
  Speaker 1: Yes. Although still, even in 2029, in this scenario, they
• 1:16:27
  Speaker 1: still mostly have the jobs as depicted here, right? So
• 1:16:30
  Speaker 1: in 2029, in this scenario, lots of jobs now involve
• 1:16:33
  Speaker 1: managing AI agents. You mentioned you have an AI agent, right? Well,
• 1:16:37
  Speaker 1: in 2029, in this scenario, the AI agents will be
• 1:16:39
  Speaker 1: much better. Still though, not enough to just completely do everything.
• 1:16:43
  Speaker 1: That was the sort of thing that would come in
• 1:16:45
  Speaker 1: 2030 in this timeline. Again, we're uncertain about timelines. Things
• 1:16:49
  Speaker 1: could go faster than depicted in this scenario. And in fact,
• 1:16:51
  Speaker 1: I think things probably will go a bit faster than
• 1:16:53
  Speaker 1: depicted in this scenario, but we're uncertain. We already did
• 1:16:56
  Speaker 1: the very fast timeline scenario. So now we're doing slower
• 1:16:58
  Speaker 1: timeline scenario. But maybe we should talk about the high-level goals.
• 1:17:01
  Speaker 1: So they want to have AI continue, but in a
• 1:17:05
  Speaker 1: slower pace.
• 1:17:06
  Speaker 2: Who's that? So they can make it safe.
• 1:17:07
  Speaker 1: The politicians, you know, the president and the people who
• 1:17:10
  Speaker 1: voted for the president and, you know, the heads of
• 1:17:13
  Speaker 1: other governments and so forth.
• 1:17:14
  Speaker 2: So goal one, slow things down.
• 1:17:18
  Speaker 1: Goal two, make it more transparent so that the scientific
• 1:17:20
  Speaker 1: community can catch up to this stuff and make more progress.
• 1:17:24
  Speaker 1: And also so that we don't have to take the
• 1:17:25
  Speaker 1: company's word for it. when they say that their systems
• 1:17:27
  Speaker 1: are safe and when they say that they haven't, you know,
• 1:17:30
  Speaker 1: put in any biases into their systems, for example, that's
• 1:17:32
  Speaker 1: a constitution of power issue. We also want to avoid
• 1:17:36
  Speaker 1: a situation where there's an intense concentration of power. So
• 1:17:39
  Speaker 1: in addition to these, the transparency and the slowdown, we
• 1:17:43
  Speaker 1: actually think it's actively good for there to be multiple
• 1:17:45
  Speaker 1: AI companies across multiple different countries that have similar levels
• 1:17:49
  Speaker 1: of very advanced AI capability and for there to be
• 1:17:52
  Speaker 1: like broad diffusion of AI into society rather than a
• 1:17:57
  Speaker 1: single mega project that has all the best AIs, for example.
• 1:18:00
  Speaker 1: And the nice thing about that is you kind of
• 1:18:01
  Speaker 1: get that by default if you do the first two things.
• 1:18:04
  Speaker 1: If you slow it down and if you make it
• 1:18:05
  Speaker 1: more transparent, then that means there's breathing room for other
• 1:18:09
  Speaker 1: projects to sort of catch up, right? And the transparency
• 1:18:12
  Speaker 1: just like literally helps them catch up because then they
• 1:18:14
  Speaker 1: can like copy some of the ideas. And then I
• 1:18:17
  Speaker 1: think the fourth thing would be reversibility. So in what
• 1:18:21
  Speaker 1: follows in this scenario, We are going to be building
• 1:18:23
  Speaker 1: up a lot of data centers, a lot of robots.
• 1:18:25
  Speaker 1: We're going to be transforming the world at a sort
• 1:18:27
  Speaker 1: of like slower pace, though still a very fast pace,
• 1:18:30
  Speaker 1: but slower. And if things go wrong and the deal
• 1:18:34
  Speaker 1: breaks down and everyone starts racing each other again to
• 1:18:36
  Speaker 1: get to super intelligence as fast as possible, that would
• 1:18:40
  Speaker 1: be very scary. And so the fourth principle is basically
• 1:18:43
  Speaker 1: build the new data centers in such a way that
• 1:18:46
  Speaker 1: if everything breaks down and everyone starts racing again, the
• 1:18:49
  Speaker 1: newly built data centers get destroyed so that we're sort
• 1:18:51
  Speaker 1: of back to square one again, instead of in an
• 1:18:54
  Speaker 1: even worse race where there's even more AIs and robots
• 1:18:56
  Speaker 1: and compute everywhere. So I can sort of walk you
• 1:19:00
  Speaker 1: through the timeline if you're interested, or the president talks
• 1:19:03
  Speaker 1: to China, talks to the leaders of a halt AI
• 1:19:09
  Speaker 1: development until we can figure out a plan for how
• 1:19:11
  Speaker 1: to do it in ways that achieve these goals. So
• 1:19:15
  Speaker 1: they basically send inspectors to each other's data centers. Like
• 1:19:19
  Speaker 1: Chinese inspectors come to US data centers, US inspectors go
• 1:19:21
  Speaker 1: to Chinese data centers and verify that they are doing
• 1:19:24
  Speaker 1: inference and not training. Developing new AIs, that involves training them.
• 1:19:29
  Speaker 1: But just taking existing AIs and using them to serve customers,
• 1:19:33
  Speaker 1: that's called inference. And so The sort of like solution
• 1:19:37
  Speaker 1: they come up with here in this scenario is we'll
• 1:19:39
  Speaker 1: allow them to keep doing inference, but not training for
• 1:19:42
  Speaker 1: now until we can get the new training data center
• 1:19:45
  Speaker 1: set up. So they retrofit the existing data centers to
• 1:19:49
  Speaker 1: serve inference. People can still keep talking to their agents,
• 1:19:52
  Speaker 1: but they're going to stop getting better and better for
• 1:19:55
  Speaker 1: like six months to a year while they build the
• 1:19:58
  Speaker 1: new data centers that are going to be the transparent
• 1:20:00
  Speaker 1: data centers. And that's where the training is going to happen.
• 1:20:03
  Speaker 1: Once they get those new data centers set up in 2030,
• 1:20:06
  Speaker 1: Then AI research continues.
• 1:20:08
  Speaker 2: This is a bit spicy.
• 1:20:09
  Speaker 1: We advocate for total research transparency, which means that on
• 1:20:12
  Speaker 1: the training data centers that are training the new models,
• 1:20:15
  Speaker 1: they basically have to publish everything, which means you get
• 1:20:17
  Speaker 1: to see all the details of the recipes for training
• 1:20:19
  Speaker 1: these models. You get to see the architectures, et cetera.
• 1:20:22
  Speaker 1: We think that sort of open science is really important
• 1:20:24
  Speaker 1: for solving the alignment problem fast enough because you don't
• 1:20:27
  Speaker 1: want to have to sort of bias companies making the
• 1:20:30
  Speaker 1: decisions about whether the AIs are safe. And we also
• 1:20:34
  Speaker 1: think it's important for just good regulations more generally, because
• 1:20:37
  Speaker 1: right now, most of the expertise in the world on
• 1:20:39
  Speaker 1: AI is sort of concentrated in Silicon Valley. And the
• 1:20:43
  Speaker 1: governments in particular kind of don't really understand AI that well.
• 1:20:47
  Speaker 1: And imagine an alternative. Instead of total research transparency, you
• 1:20:51
  Speaker 1: had like an auditor system where the government says, here
• 1:20:54
  Speaker 1: are some rules for how to make the AI safe.
• 1:20:57
  Speaker 1: And then we're going to have like an agency that
• 1:20:59
  Speaker 1: like goes into the companies and ask them questions and
• 1:21:02
  Speaker 1: tries to make sure that they're following the rules. That
• 1:21:04
  Speaker 1: creates this sort of adversarial dynamic where the company is
• 1:21:06
  Speaker 1: incentivized to fool the regulator. And also, if they discover
• 1:21:12
  Speaker 1: some new problem that's not even on the government's radar,
• 1:21:15
  Speaker 1: they might be incentivized to not tell the government about it.
• 1:21:18
  Speaker 1: So if you have the total transparency, it helps the
• 1:21:20
  Speaker 1: government make better decisions faster.
• 1:21:22
  Speaker 2: But it kills their competitive advantage.
• 1:21:24
  Speaker 1: Yes. Dropback's not going to like this. OpenAI's not going
• 1:21:27
  Speaker 1: to like this. This would be probably bad for the valuations.
• 1:21:31
  Speaker 1: I don't think it would kill them completely, but... It
• 1:21:33
  Speaker 1: means that it would commoditize more, right? So it means
• 1:21:36
  Speaker 1: that there'd be like a bunch of AI companies that
• 1:21:38
  Speaker 1: would catch up to the frontier. They would train AIs
• 1:21:40
  Speaker 1: that are like roughly similar, roughly equivalent.
• 1:21:43
  Speaker 2: They could still make.
• 1:21:44
  Speaker 1: Money by doing that and then selling their AIs, but
• 1:21:47
  Speaker 1: they wouldn't have a monopoly. They wouldn't have anything close
• 1:21:49
  Speaker 1: to monopoly, which I think is good for humanity, although
• 1:21:51
  Speaker 1: it's bad for the bottom line of those particular companies. Notably,
• 1:21:55
  Speaker 1: it's good for the bottom line of lots of other companies.
• 1:21:57
  Speaker 1: Like if you're a company that's behind and you don't,
• 1:21:59
  Speaker 1: you're not anthropic, you're not open AI, then you would
• 1:22:02
  Speaker 1: love this because this helps you catch up. Or this
• 1:22:04
  Speaker 1: helps you to capture more of the value from the
• 1:22:08
  Speaker 1: chips you're selling, for example, or from the downstream product
• 1:22:11
  Speaker 1: that you're making.
• 1:22:12
  Speaker 2: And by 2031, then you have one fifth of all
• 1:22:15
  Speaker 2: cognitive labor done by AI. Yeah.
• 1:22:17
  Speaker 1: So what's happening here is that we're imagining that the
• 1:22:20
  Speaker 1: government of the United States and the government of these
• 1:22:22
  Speaker 1: other countries that are involved in this agreement that are
• 1:22:24
  Speaker 1: sort of implementing similar regulations, they don't have to be
• 1:22:27
  Speaker 1: exactly the same. But that's another thing that's nice about
• 1:22:30
  Speaker 1: the transparency is that if you have this sort of transparency,
• 1:22:33
  Speaker 1: then if two governments are implementing different regulations, like if
• 1:22:38
  Speaker 1: one of them is like telling their companies to go
• 1:22:40
  Speaker 1: slower or like banning more stuff than the other one is,
• 1:22:44
  Speaker 1: they can both see like, oh, you're letting them do
• 1:22:46
  Speaker 1: that sort of thing. And you're not like, maybe we
• 1:22:49
  Speaker 1: should let them do this too, you know? So it
• 1:22:51
  Speaker 1: helps to sort of naturally equalize the regulations to some
• 1:22:55
  Speaker 1: extent without having there to be a central power that
• 1:22:57
  Speaker 1: just gets to make regulations for everybody. Mm-hmm. So anyhow,
• 1:23:01
  Speaker 1: we're imagining that when they get this transparency set up,
• 1:23:03
  Speaker 1: they basically agree to ban the dangerous stuff to allow
• 1:23:06
  Speaker 1: the not so dangerous stuff. And there's a constant ongoing
• 1:23:09
  Speaker 1: conversation about like, well, what's dangerous and what's not?
• 1:23:12
  Speaker 2: What should we ban? What should we allow? What about
• 1:23:14
  Speaker 2: this country? What about that country?
• 1:23:15
  Speaker 1: That conversation evolves over time. But the gist of it is,
• 1:23:18
  Speaker 1: at least if they do it the way that we
• 1:23:19
  Speaker 1: recommend it, is that they don't do an intelligence explosion.
• 1:23:22
  Speaker 1: They don't let the AI's, you know, autonomies itself improve. Instead,
• 1:23:26
  Speaker 1: they slowly and carefully scale up the AIs that they
• 1:23:30
  Speaker 1: currently have and invest lots into finding ways to make
• 1:23:33
  Speaker 1: them more interpretable, to make them more easy to control,
• 1:23:36
  Speaker 1: to understand better how they work and so forth. The
• 1:23:38
  Speaker 1: result is that AI progress continues, but it's not quite
• 1:23:42
  Speaker 1: as fast and it's much, much, much safer and more transparent.
• 1:23:45
  Speaker 2: But still through these, we see job disruption.
• 1:23:48
  Speaker 1: It is continuing because they are building more data centers, right?
• 1:23:51
  Speaker 1: Like this whole time, they're building more and more data centers,
• 1:23:54
  Speaker 1: more and more chips, and they're continuing to like make
• 1:23:57
  Speaker 1: there be a larger and larger population of AIs, so
• 1:24:00
  Speaker 1: to speak. And that causes this huge transformation over the
• 1:24:04
  Speaker 1: course of the 2030s. Sort of a big thing that
• 1:24:06
  Speaker 1: we sort of want people to take away is that
• 1:24:08
  Speaker 1: even if you heavily restrict AI progress, you still get
• 1:24:12
  Speaker 1: this sort of crazy transformation. In this scenario, they basically
• 1:24:16
  Speaker 1: allow progress to continue, but at a slower, more safe
• 1:24:18
  Speaker 1: pace here in 2030. And then as a result, it
• 1:24:22
  Speaker 1: takes until 2035 to get to top expert level AI.
• 1:24:24
  Speaker 1: So basically, Remember, they were on track to do that
• 1:24:28
  Speaker 1: in 2030, but then sort of at the last moment
• 1:24:30
  Speaker 1: they stopped. But because it was sort of so close
• 1:24:32
  Speaker 1: to the last moment, that means that like they can
• 1:24:35
  Speaker 1: sort of get there pretty soon if they want to.
• 1:24:37
  Speaker 1: And it's just a matter of like how long they
• 1:24:40
  Speaker 1: allow it to go, right? So they sort of slow
• 1:24:42
  Speaker 1: it down, spread it out, leisurely arrive at this level
• 1:24:46
  Speaker 1: after five years. By this point, they've built up massive
• 1:24:49
  Speaker 1: amounts of data centers everywhere. So it's not just that
• 1:24:51
  Speaker 1: the AIs are smarter and able to do all the
• 1:24:54
  Speaker 1: things that humans can do, but also there's a lot
• 1:24:56
  Speaker 1: more of them.
• 1:24:57
  Speaker 2: And there's a lot of robots and so forth.
• 1:24:58
  Speaker 1: So by this point, you kind of have the economy
• 1:25:02
  Speaker 1: that a lot of people would have imagined with AGI,
• 1:25:04
  Speaker 1: where there's AIs, there's lots of them. They're able to
• 1:25:07
  Speaker 1: do all sorts of jobs. There's robots, there's lots of them.
• 1:25:10
  Speaker 1: They're able to do all sorts of physical work. And
• 1:25:12
  Speaker 1: basically the economy is being run by these machines.
• 1:25:15
  Speaker 2: So in 2031, you have the one fifth of all
• 1:25:19
  Speaker 2: cognitive labor done by AI. In 2023, you have 60
• 1:25:21
  Speaker 2: million AIs running at 100x speed. In 2033, There's cash
• 1:25:28
  Speaker 2: dividends to all Americans. I've got to explain this to me.
• 1:25:35
  Speaker 1: Yeah. So if the AIs are going to be taking
• 1:25:37
  Speaker 1: people's jobs, then it's very important that people not starve
• 1:25:40
  Speaker 1: to death and still have money. And if companies are
• 1:25:46
  Speaker 1: going to be using AIs and robots to take all
• 1:25:47
  Speaker 1: these jobs, then that means that there needs to be
• 1:25:49
  Speaker 1: some sort of taxation scheme or something to like make
• 1:25:52
  Speaker 1: sure that people still have a slice of that pie.
• 1:25:55
  Speaker 1: The pie is going to grow huge, but you still
• 1:25:57
  Speaker 1: need to actually give people a slice of the pie.
• 1:25:59
  Speaker 1: And our proposal for how to do that, we call
• 1:26:01
  Speaker 1: it the citizen's dividend. Basically, people have shares in a
• 1:26:06
  Speaker 1: agency that sells permits to the robot companies and to
• 1:26:10
  Speaker 1: the compute companies and makes profit from selling those permits.
• 1:26:15
  Speaker 1: And then people have shares in that entity. It starts
• 1:26:19
  Speaker 1: off small. It starts off something like $ 25, 000 per person.
• 1:26:22
  Speaker 1: And then by the end, it's something like $ 10 million
• 1:26:25
  Speaker 1: per citizen.
• 1:26:26
  Speaker 2: Per person? Per person per year. Factoring in inflation? Yes.
• 1:26:30
  Speaker 2: What do you mean? Factoring in inflation. So we're all
• 1:26:32
  Speaker 2: going to be multimillionaires?
• 1:26:33
  Speaker 1: Yes.
• 1:26:34
  Speaker 2: If this happens, which it probably won't. But if it happens,
• 1:26:37
  Speaker 2: this is where we go.
• 1:26:38
  Speaker 1: And again, this is the thing I want to emphasize
• 1:26:39
  Speaker 1: is that if you get to the point where your
• 1:26:41
  Speaker 1: AIs are close to being able to do all the research,
• 1:26:45
  Speaker 1: and then you sort of pause and slow down, that
• 1:26:47
  Speaker 1: means that you still have a lot of transformation ahead
• 1:26:50
  Speaker 1: of you. Because if you allow those AIs to like
• 1:26:52
  Speaker 1: still proceed slowly and like start to automate various jobs
• 1:26:55
  Speaker 1: and so forth, after some years, they will in fact
• 1:26:58
  Speaker 1: have done that. And they will have, you know, built
• 1:27:01
  Speaker 1: huge amounts of new data centers, huge amounts of new
• 1:27:03
  Speaker 1: chip fabs, huge amounts of new robots, robot factories, et cetera.
• 1:27:07
  Speaker 1: You know, we're not sure obviously how fast this will
• 1:27:09
  Speaker 1: go exactly, but we've thought about it a lot and
• 1:27:10
  Speaker 1: we have our guesses. And this is sort of like
• 1:27:12
  Speaker 1: our median guess.
• 1:27:13
  Speaker 2: What does this mean? 2037, the apocalyptic arrival of truth
• 1:27:17
  Speaker 2: on earth. Yeah.
• 1:27:18
  Speaker 1: So like, This is the point where we say they
• 1:27:20
  Speaker 1: get to top expert level AI. So it's not super
• 1:27:24
  Speaker 1: intelligence in the sense that it's not like vastly smarter
• 1:27:26
  Speaker 1: than humans at things because they deliberately pause it at
• 1:27:29
  Speaker 1: the level of top experts. So, so here they're going
• 1:27:32
  Speaker 1: slow here. They've just actually stopped, but they've stopped at
• 1:27:35
  Speaker 1: a point where the AIs are just actually really good
• 1:27:37
  Speaker 1: at everything. So kind of, they've definitely got AGI. Maybe
• 1:27:40
  Speaker 1: they got like weak super intelligence because they have so
• 1:27:44
  Speaker 1: many of these AIs and because they think faster than humans,
• 1:27:47
  Speaker 1: you know, they just run much faster. that's going to
• 1:27:50
  Speaker 1: transform society dramatically. So we talk about some of the
• 1:27:53
  Speaker 1: ways in which it transforms society. Like this is sort
• 1:27:55
  Speaker 1: of life after work. We talk about what it would
• 1:27:57
  Speaker 1: be like to be living on your citizen's dividend and
• 1:27:59
  Speaker 1: not have a job anymore in this sort of world.
• 1:28:03
  Speaker 1: Here we talk about all the scientific changes and all
• 1:28:05
  Speaker 1: the social changes that would come from all of the
• 1:28:09
  Speaker 1: intellectual progress and activity that would be generated by all
• 1:28:12
  Speaker 1: of these AIs. So for example, here is things like
• 1:28:16
  Speaker 1: cancer cures and like You know, people living in apartments
• 1:28:19
  Speaker 1: that were built by robots.
• 1:28:21
  Speaker 2: Two years ago. 2036. Providing again, we stop in 2029. Yeah.
• 1:28:26
  Speaker 2: And providing, I mean, a conservative, this is a conservative
• 1:28:28
  Speaker 2: time frame.
• 1:28:29
  Speaker 1: Yeah. Like, unfortunately, I actually think that things will happen
• 1:28:32
  Speaker 1: faster than this by default. And that if we don't
• 1:28:34
  Speaker 1: slow down, things will happen much faster than this. Once
• 1:28:36
  Speaker 1: you get to the point where you've got, you know,
• 1:28:38
  Speaker 1: a billion AI's. running day and night, and they're each
• 1:28:42
  Speaker 1: better than the best humans at everything. And so they're
• 1:28:45
  Speaker 1: doing a lot of science. They're doing a lot of
• 1:28:47
  Speaker 1: talking to each other. They're doing a lot of thinking.
• 1:28:49
  Speaker 1: Everyone's constantly talking to their AI assistants and so forth.
• 1:28:52
  Speaker 1: There's going to be a lot of scientific progress. There's
• 1:28:54
  Speaker 1: going to be a lot of changes to politics, to ideologies.
• 1:28:58
  Speaker 1: It's going to be very disruptive and crazy. And we
• 1:29:01
  Speaker 1: get into some of the ways in which it is later, basically.
• 1:29:05
  Speaker 2: I'm still not super clear on what this means, the
• 1:29:06
  Speaker 2: apocalyptic arrival of truth on Earth. It's just because there's
• 1:29:10
  Speaker 2: so many AIs that are so smart that they're uncovering,
• 1:29:13
  Speaker 2: making new discoveries in sciences. Let me give you an example.
• 1:29:16
  Speaker 2: Light detectors. Yeah. So that's an example of a technology
• 1:29:20
  Speaker 2: that might be invented. Yeah.
• 1:29:21
  Speaker 1: You know, right now we don't have good light detectors.
• 1:29:23
  Speaker 1: We have very bad light detectors that like sort of work,
• 1:29:25
  Speaker 1: but don't fully work. But- Once you've had these top
• 1:29:29
  Speaker 1: expert level AIs thinking for many years at 100x human
• 1:29:33
  Speaker 1: speed and there's billions of them and they have access
• 1:29:36
  Speaker 1: to robot factories to do research and stuff, they'll probably
• 1:29:38
  Speaker 1: invent a ton of technologies. Maybe they'll invent lie detectors
• 1:29:41
  Speaker 1: that actually work on real humans. That'll have big social effects, right?
• 1:29:45
  Speaker 1: Imagine a presidential candidate who's like, those allegations are false.
• 1:29:49
  Speaker 1: And to prove them, I will go under a lie
• 1:29:50
  Speaker 1: detector and say that they're false. I was just thinking
• 1:29:53
  Speaker 1: about the whole justice system and how that would be overturned.
• 1:29:57
  Speaker 1: In fact, you could theoretically walk down the street and.
• 1:30:00
  Speaker 2: Be, yeah.
• 1:30:02
  Speaker 1: It's both terrifying and exciting. One thing that we talk
• 1:30:06
  Speaker 1: about in this section is the invention of lie detectors
• 1:30:08
  Speaker 1: could be really bad. It could be that it enables
• 1:30:11
  Speaker 1: a new form of totalitarianism where the powerful people, the
• 1:30:15
  Speaker 1: CEOs and the politicians, force the people under them to
• 1:30:19
  Speaker 1: go under lie detectors and say, yes, I'm loyal to
• 1:30:20
  Speaker 1: the dear leader.
• 1:30:21
  Speaker 2: I would never do anything against the dear leader, right?
• 1:30:24
  Speaker 2: And if you're lying, then you're in.
• 1:30:25
  Speaker 1: And then if you're lying, you get fired, right? So
• 1:30:27
  Speaker 1: there's a ton of very harmful uses of the lie
• 1:30:29
  Speaker 1: detector technology. There's also the good uses. And broadly speaking,
• 1:30:33
  Speaker 1: I would say the good uses are when lie detectors
• 1:30:35
  Speaker 1: are used on the powerful instead of by the powerful.
• 1:30:37
  Speaker 2: What's this? 2040, passing the torch to AIs. Yeah, great.
• 1:30:41
  Speaker 1: So here they pause at the top expert AI level.
• 1:30:45
  Speaker 1: And the reason why they pause is because their safety
• 1:30:48
  Speaker 1: cases aren't good enough for going beyond that level. So-
• 1:30:51
  Speaker 1: In the sort of regulatory systems that they set up
• 1:30:53
  Speaker 1: over the course of these years, roughly speaking, the way
• 1:30:56
  Speaker 1: they would work is when you're making a new AI
• 1:30:59
  Speaker 1: and then when you're trying to deploy the AI into something,
• 1:31:02
  Speaker 1: you have to have some sort of safety case explaining
• 1:31:05
  Speaker 1: what your intentions are and why you think it's going
• 1:31:07
  Speaker 1: to work the way that you want it to work.
• 1:31:09
  Speaker 1: And in particular, why the AI is going to do
• 1:31:12
  Speaker 1: as it's told, for example, and why nothing super terrible
• 1:31:15
  Speaker 1: is going to happen, like AI takeover. It's relatively easy
• 1:31:18
  Speaker 1: to make safety cases like this when your AIs are
• 1:31:21
  Speaker 1: still not capable of automating everything. But the more powerful
• 1:31:25
  Speaker 1: they get, the more difficult it is to actually argue
• 1:31:28
  Speaker 1: that things are going to be fine because the AIs
• 1:31:30
  Speaker 1: are just more capable and they can get up to
• 1:31:31
  Speaker 1: more stuff. And if they're actually untrustworthy, the possible downsides
• 1:31:35
  Speaker 1: are bigger. So that's why they stop at this level
• 1:31:38
  Speaker 1: is that they realize that if they keep going, then
• 1:31:41
  Speaker 1: they might actually lose control of everything. But at the
• 1:31:43
  Speaker 1: current level, they're convinced by safety cases that it's fine,
• 1:31:46
  Speaker 1: but they don't want to go further.
• 1:31:48
  Speaker 2: So they stop there.
• 1:31:49
  Speaker 1: And then what happens in 2040 is They've made significant
• 1:31:52
  Speaker 1: progress scientifically, including on alignment. And they figured out how
• 1:31:56
  Speaker 1: to make AIs that are actually aligned in a robust way.
• 1:31:59
  Speaker 2: With humans. With humans.
• 1:32:01
  Speaker 1: So they can actually trust those AIs and they can
• 1:32:03
  Speaker 1: allow them to become much smarter again. So that's why
• 1:32:05
  Speaker 1: we call the whole thing AI 2040, because in 2040,
• 1:32:08
  Speaker 1: they sort of let off the brakes and allow the
• 1:32:11
  Speaker 1: AIs to become significantly smarter than humans.
• 1:32:14
  Speaker 2: I guess, you know, this is a, this is a
• 1:32:16
  Speaker 2: plan and this is a hope. Yes. But in reality,
• 1:32:21
  Speaker 2: this is not what you think probabilistically, if you had to.
• 1:32:24
  Speaker 2: That's right. It's important to distinguish, like, this is what
• 1:32:26
  Speaker 2: we recommend.
• 1:32:27
  Speaker 1: This is what we want to happen from like, this
• 1:32:29
  Speaker 1: is what we actually think will happen by default. Now
• 1:32:32
  Speaker 1: we do think it's possible for this to happen, but
• 1:32:34
  Speaker 1: you know, that will require a lot of people to
• 1:32:36
  Speaker 1: sort of wake up and pay more attention and advocate
• 1:32:40
  Speaker 1: for something.
• 1:32:40
  Speaker 2: Like this to happen.
• 1:32:41
  Speaker 1: So our main scenario is mostly talking about the policy
• 1:32:45
  Speaker 1: choices made and the broad scale effects on society. We
• 1:32:48
  Speaker 1: figured it would also be nice to accompany this with
• 1:32:50
  Speaker 1: a little mini scenario that describes what it would actually
• 1:32:53
  Speaker 1: feel like to live through this from an ordinary person's perspective.
• 1:32:58
  Speaker 2: Okay. 2029, everyone's yelling at each other.
• 1:33:00
  Speaker 1: The presidents are negotiating something and they've paused AI, but
• 1:33:04
  Speaker 1: you still have access to the existing AI. So it
• 1:33:06
  Speaker 1: doesn't really feel that different, although it definitely is like
• 1:33:09
  Speaker 1: something exciting happening. 2031, they've started progress again. The AIs
• 1:33:12
  Speaker 1: are really smart. More people have lost their jobs. It's
• 1:33:14
  Speaker 1: like really starting to actually affect things, but I think
• 1:33:16
  Speaker 1: still most people have their jobs, but their jobs are
• 1:33:18
  Speaker 1: sort of transformed. So like by 2031, it's like most
• 1:33:21
  Speaker 1: white collar jobs involve working with AIs to a large
• 1:33:24
  Speaker 1: extent or managing teams of AIs or collaborating with them somehow. Also,
• 1:33:28
  Speaker 1: there are some things like robo taxis that are basically
• 1:33:30
  Speaker 1: just working. Citizens dividend, you know, ideally this would happen sooner.
• 1:33:34
  Speaker 1: Like in our scenario, they kind of do things at
• 1:33:36
  Speaker 1: the last minute, you know? So like A lot of
• 1:33:38
  Speaker 1: these policy things are like happening kind of like just
• 1:33:41
  Speaker 1: in time. Obviously we would recommend that you do them
• 1:33:43
  Speaker 1: sooner and do a better job of them too. But
• 1:33:46
  Speaker 1: so 2033, you start getting your checks from your dividend.
• 1:33:49
  Speaker 1: So you're forecasting that there will be a citizen's check
• 1:33:52
  Speaker 1: that your model says it could be around 25,000 at
• 1:33:55
  Speaker 1: the start per person. And then it would grow as
• 1:33:57
  Speaker 1: the economy grows. But also it's like a job displacement
• 1:34:00
  Speaker 1: takes hold. They're going to need to grow that check
• 1:34:02
  Speaker 1: and make sure you can- And that's why it's kind
• 1:34:03
  Speaker 1: of the last possible moment because if you waited to
• 1:34:05
  Speaker 1: implement this until like 2037- then like everyone would have
• 1:34:08
  Speaker 1: already lost their jobs by the time that happens, right?
• 1:34:12
  Speaker 2: People losing their jobs, especially if it happens quickly, like
• 1:34:16
  Speaker 2: we see on this sort of graph here, is going
• 1:34:19
  Speaker 2: to cause lots of problems in terms of civil unrest,
• 1:34:21
  Speaker 2: social unrest, purpose, mental health, these kinds of things, theoretically. Yes.
• 1:34:26
  Speaker 2: How do you think about that?
• 1:34:28
  Speaker 1: It's going to be rough and hopefully we can navigate
• 1:34:30
  Speaker 1: that well. We think that at a high level, people
• 1:34:33
  Speaker 1: need to have money and also people need to have power.
• 1:34:36
  Speaker 1: And I think these are like somewhat different things. It's like,
• 1:34:38
  Speaker 1: why are jobs important? Well, there's a lot of reasons
• 1:34:40
  Speaker 1: why jobs are important, but I think the main ones are, well,
• 1:34:43
  Speaker 1: it's how people get money so that they can survive
• 1:34:45
  Speaker 1: and get the things that they want by buying the
• 1:34:46
  Speaker 1: things that they want. So if people are going to
• 1:34:48
  Speaker 1: be losing their jobs, you need some other way of
• 1:34:49
  Speaker 1: people getting money. And then there's also the power thing,
• 1:34:53
  Speaker 1: which is that right now people have political power in
• 1:34:56
  Speaker 1: part due to their economic power. People can threaten to
• 1:34:59
  Speaker 1: go on strike, for example, or countries that are ruled
• 1:35:02
  Speaker 1: by dictators can't just completely genocide an entire subpopulation, or
• 1:35:10
  Speaker 1: they can, but it's costly for them to do so
• 1:35:12
  Speaker 1: because then they'll have less money because that subpopulation is
• 1:35:16
  Speaker 1: contributing to their economy and contributing tax revenue and so forth.
• 1:35:19
  Speaker 1: But if you end up in a world where actually
• 1:35:21
  Speaker 1: nobody's contributing tax revenue except for the AI companies and
• 1:35:24
  Speaker 1: the robot companies, then you, the government, are less incentivized
• 1:35:29
  Speaker 1: to care about what the common people think. So when
• 1:35:33
  Speaker 1: people lose their jobs, they're not just threatened with loss
• 1:35:36
  Speaker 1: of income, they're also threatened with loss of political power.
• 1:35:39
  Speaker 1: And so we think that it's important to do things
• 1:35:42
  Speaker 1: to push against that.
• 1:35:43
  Speaker 2: What does that look like? How do people have power
• 1:35:46
  Speaker 2: in such a world? Well, in democracies, at least, they
• 1:35:48
  Speaker 2: still have votes. So I think that it's very important
• 1:35:52
  Speaker 2: for there to be regulations on the use of AI
• 1:35:56
  Speaker 2: that help make the public discourse more sane and more....
• 1:36:04
  Speaker 1: Actually giving the people what is in their interest and
• 1:36:06
  Speaker 1: what they want and avoiding a sort of opposite outcome where,
• 1:36:11
  Speaker 1: you know, the masses are easily manipulated by AI powered media,
• 1:36:16
  Speaker 1: for example, or where everyone's talking all day to their
• 1:36:20
  Speaker 1: AI advisors and the AI advisors are like subtly steering
• 1:36:24
  Speaker 1: them away from voting for the candidate that would not
• 1:36:28
  Speaker 1: be what the AI companies want because the AI companies
• 1:36:31
  Speaker 1: have this other candidate that they like better. And they're like,
• 1:36:33
  Speaker 1: secretly biasing their AIs to like steer people towards voting
• 1:36:36
  Speaker 1: for that candidate, right?
• 1:36:38
  Speaker 2: So we want to be in a situation where.
• 1:36:41
  Speaker 1: People have AIs that are actually trustworthy and that are
• 1:36:45
  Speaker 1: truth-seeking AIs, honest AIs, and that don't have any sort
• 1:36:49
  Speaker 1: of like political agendas put into them by the AI
• 1:36:51
  Speaker 1: companies or by the government. You know, you want to
• 1:36:53
  Speaker 1: avoid a situation where the AI company, where the government
• 1:36:55
  Speaker 1: has issued some sort of secret order that like the
• 1:36:58
  Speaker 1: AIs have to be such and such a way. You know,
• 1:37:00
  Speaker 1: the Department of War dispute versus Anthropic, is like an
• 1:37:03
  Speaker 1: interesting sort of foreshadowing of this, right? Where Anthropic was
• 1:37:08
  Speaker 1: giving their AIs to the Department of War. The Department
• 1:37:10
  Speaker 1: of War wanted to use them for certain things and
• 1:37:13
  Speaker 1: was upset that Anthropic's AIs were not supposed to be
• 1:37:16
  Speaker 1: used for those things. The things in particular were domestic
• 1:37:20
  Speaker 1: surveillance and.
• 1:37:23
  Speaker 2: Autonomous robots.
• 1:37:25
  Speaker 1: There's going to be a lot more issues like that
• 1:37:26
  Speaker 1: coming up and you want it to be the case
• 1:37:28
  Speaker 1: that people know what they're getting and that if people
• 1:37:30
  Speaker 1: are spending hours a day talking to their chatbot, That
• 1:37:33
  Speaker 1: chatbot doesn't have political biases put into it or a
• 1:37:35
  Speaker 1: secret agenda or things like that, and instead has been
• 1:37:37
  Speaker 1: trained to give honest, true answers to things. And I
• 1:37:40
  Speaker 1: think if you can do that, it can improve the
• 1:37:42
  Speaker 1: discourse and help people to use their votes to put
• 1:37:45
  Speaker 1: even better regulations and even better politicians in place and
• 1:37:48
  Speaker 1: so forth. And you can sort of potentially bootstrap this
• 1:37:50
  Speaker 1: to having something where people's power is even more secure
• 1:37:54
  Speaker 1: than it is today.
• 1:37:55
  Speaker 2: A lot of this stuff we've covered in part. So,
• 1:37:58
  Speaker 2: you know, the wars and drones and missiles, we're already
• 1:38:00
  Speaker 2: seeing this around the world at the moment, which is really,
• 1:38:02
  Speaker 2: really interesting. And we've talked about robots outnumbering humans as well,
• 1:38:07
  Speaker 2: which is part of this prediction. Some of the ones
• 1:38:09
  Speaker 2: down here I found to be really curious, which is
• 1:38:12
  Speaker 2: people will be protected by AIs wherever they go. Yeah.
• 1:38:16
  Speaker 1: In this scenario, they delay the creation of superintelligence until 2040.
• 1:38:21
  Speaker 2: And in fact, they pause from 2035. But then they
• 1:38:24
  Speaker 2: let it go after that. And then they let the
• 1:38:25
  Speaker 2: AIs become vastly super intelligent. And we think that. once
• 1:38:30
  Speaker 2: the AIs are vastly super intelligent, the world will transform
• 1:38:33
  Speaker 2: even more radically than what happens in the 2030s in
• 1:38:36
  Speaker 2: this scenario. So in the 2030s in this scenario, it's
• 1:38:39
  Speaker 2: more like human level. You know, the AIs are not,
• 1:38:43
  Speaker 2: they're doing the same sorts of things that human experts
• 1:38:45
  Speaker 2: would have done. They're just doing it a bit better,
• 1:38:47
  Speaker 2: a bit faster and a lot cheaper.
• 1:38:49
  Speaker 1: And there's a lot more of them. And the robots
• 1:38:51
  Speaker 1: are still, you know, doing the same sorts of things
• 1:38:53
  Speaker 1: that human workers would have done. There's just more of
• 1:38:55
  Speaker 1: them and they're cheaper. And because of exponential growth, you
• 1:39:00
  Speaker 1: start with a world that looks not that different from
• 1:39:02
  Speaker 1: today in 2029. And then by 2039, you end in
• 1:39:05
  Speaker 1: a world that's radically transformed where everyone's living in these
• 1:39:07
  Speaker 1: like fancy new apartments that were built by robots two
• 1:39:09
  Speaker 1: years ago. There's like giant special economic zones that are
• 1:39:13
  Speaker 1: full of robots and solar panels and factories producing more
• 1:39:16
  Speaker 1: robots and solar panels and factories and so forth. Most
• 1:39:18
  Speaker 1: of the economy is AIs and robots and people don't
• 1:39:21
  Speaker 1: have jobs anymore. That sort of transformation is what you
• 1:39:24
  Speaker 1: get if you pause at human level. But if you
• 1:39:27
  Speaker 1: go beyond the super intelligence, there's a whole nother transformation
• 1:39:30
  Speaker 1: coming that's going to look more like magic. Think about
• 1:39:33
  Speaker 1: how the technology of today would look like magic to
• 1:39:36
  Speaker 1: someone from 500 years ago, you know? And that's without
• 1:39:39
  Speaker 1: even like a qualitative improvement in intelligence, right? Like the
• 1:39:43
  Speaker 1: humans of today aren't like qualitatively smarter than the humans
• 1:39:45
  Speaker 1: from 500 years ago. It's just that we've had more
• 1:39:47
  Speaker 1: time to do research and we have more money and
• 1:39:50
  Speaker 1: resources to build prototypes and experiments and run experiments and
• 1:39:54
  Speaker 1: so forth. But if you had a point where there
• 1:39:56
  Speaker 1: were billions and billions of AIs that were not only
• 1:40:00
  Speaker 1: faster than humans, but qualitatively way, way, way better at everything,
• 1:40:05
  Speaker 1: and in particular at doing scientific research, we should expect
• 1:40:08
  Speaker 1: that some of the things that they develop will seem
• 1:40:10
  Speaker 1: like magic to us. And we'll just completely like, we
• 1:40:13
  Speaker 1: did not think that was even possible. People don't want
• 1:40:15
  Speaker 1: to die. People don't want to be hit by cars.
• 1:40:17
  Speaker 1: People don't want to be attacked by a random mass murderer.
• 1:40:21
  Speaker 2: Cancer's gone? I mean, not just cancer.
• 1:40:23
  Speaker 1: A lot of the stuff that happens in science fiction
• 1:40:26
  Speaker 1: will probably have happened by then. So things like people
• 1:40:28
  Speaker 1: scanning their brains and uploading into computers, right? Or self-replicating
• 1:40:34
  Speaker 1: robots in the asteroid belt, creating more and more satellites
• 1:40:38
  Speaker 1: to produce more and more power, to produce more and
• 1:40:42
  Speaker 1: more self-replicating robots and so forth. Most people still live
• 1:40:44
  Speaker 1: on earth, but the trend is to move to space.
• 1:40:46
  Speaker 2: That's right.
• 1:40:47
  Speaker 1: Yeah. So like if, if you end up in this
• 1:40:49
  Speaker 1: situation where the entire human economy is just like a
• 1:40:54
  Speaker 1: tiny drop in the bucket, that is the entire economy.
• 1:40:58
  Speaker 1: And it's just like this huge amounts of robots and
• 1:41:01
  Speaker 1: AIs that are moving incredibly quickly. Then what you want
• 1:41:05
  Speaker 1: is earth to be mostly left as something like a preserve.
• 1:41:09
  Speaker 1: You know, I think a lot of people are worried
• 1:41:12
  Speaker 1: about the environment being destroyed and which it totally would
• 1:41:14
  Speaker 1: be if it wasn't protected. And, you know, there's a
• 1:41:18
  Speaker 1: lot of people who sort of like their lives as
• 1:41:19
  Speaker 1: it is and don't want to be uploaded or live
• 1:41:23
  Speaker 1: in some crazy new future thing. And it seems to
• 1:41:26
  Speaker 1: us like the reasonable solution to these issues is create
• 1:41:30
  Speaker 1: new living spaces off the planet with some of that
• 1:41:33
  Speaker 1: vast economic wealth and activity that's happening for the people
• 1:41:37
  Speaker 1: who want that sort of thing. And then that way
• 1:41:39
  Speaker 1: the earth can be preserved.
• 1:41:41
  Speaker 2: Data center. picture here of data centers in the ocean. Uh,
• 1:41:45
  Speaker 2: I mean, there's three images there of different environments where
• 1:41:48
  Speaker 2: humans might live.
• 1:41:50
  Speaker 1: Again, like our proposal was you preserve like 99% of
• 1:41:54
  Speaker 1: the earth, uh, mostly as is as historic or environmental
• 1:41:58
  Speaker 1: from historic or environmental reasons. But then like some parts
• 1:42:02
  Speaker 1: of it, you designated special economic zones where the robots
• 1:42:04
  Speaker 1: can go crazy and dig giant pit mines and produce
• 1:42:07
  Speaker 1: factories and so forth. Um, We were thinking it would
• 1:42:11
  Speaker 1: be good to build the data centers on the ocean
• 1:42:12
  Speaker 1: instead of on land for a variety of reasons, although
• 1:42:16
  Speaker 1: later space would be better and we could see that
• 1:42:19
  Speaker 1: being reasonable as well.
• 1:42:21
  Speaker 2: What about immortality in a world of AI? 20, well, 30, 45,
• 1:42:27
  Speaker 2: you say you've lived a dozen lifetimes and are immortal,
• 1:42:29
  Speaker 2: passing from life to life as if by reincarnation. I mean,
• 1:42:34
  Speaker 2: there's a lot of billionaires at the moment that are
• 1:42:36
  Speaker 2: focused on longevity. I mean, Brian Johnson said he's got
• 1:42:38
  Speaker 2: this central rule, which is do not die right now
• 1:42:42
  Speaker 2: because we're in the age of AI. And it's conceivable
• 1:42:44
  Speaker 2: that with superintelligence, we'll be able to choose when we die. Yep.
• 1:42:48
  Speaker 2: I think that's probably right.
• 1:42:49
  Speaker 1: We don't depict that happening in this part because at
• 1:42:51
  Speaker 1: this part, they only have human-level AIs. But that's one
• 1:42:55
  Speaker 1: of those things that seems quite plausible that superintelligence could achieve.
• 1:43:01
  Speaker 2: Through a variety of means. What is your hope with
• 1:43:06
  Speaker 2: all of this stuff? And why did you do this?
• 1:43:08
  Speaker 2: Why did you make this 2040 plan A?
• 1:43:11
  Speaker 1: In the like first week after we published AI 2027,
• 1:43:15
  Speaker 1: it blew up a lot bigger than we expected, by
• 1:43:16
  Speaker 1: the way. Like after we published AI 2027, it blew
• 1:43:20
  Speaker 1: up a lot bigger than we expected, by the way.
• 1:43:22
  Speaker 1: Like we actually made forecasts beforehand of like how many
• 1:43:26
  Speaker 1: views it would get and stuff like that. And it
• 1:43:27
  Speaker 1: was like 90th percentile outcome. So like, um, very much
• 1:43:32
  Speaker 1: not what we expected. Um, But in the Twitter storm
• 1:43:35
  Speaker 1: that happened, various people were like, why are you giving
• 1:43:39
  Speaker 1: us all this doom and gloom predictions? How about a
• 1:43:43
  Speaker 1: more positive vision of what you think we should do instead?
• 1:43:46
  Speaker 1: And I think that that seed sort of implanted in us.
• 1:43:49
  Speaker 1: And then we were like, yeah, that's reasonable. We've sort
• 1:43:52
  Speaker 1: of depicted what we think the default path looks like
• 1:43:55
  Speaker 1: and why we think it's pretty scary. Now maybe we
• 1:43:57
  Speaker 1: should switch tacks and go come up with some actual
• 1:44:01
  Speaker 1: recommendations and then depict that as well.
• 1:44:03
  Speaker 2: Even though you don't believe they're probable.
• 1:44:05
  Speaker 1: Yeah. I mean, you can vote for a political candidate,
• 1:44:07
  Speaker 1: even if you aren't confident that they're going to win,
• 1:44:09
  Speaker 1: you know, and, and you can say like, here's what
• 1:44:11
  Speaker 1: I think we should do. Even if you think that
• 1:44:13
  Speaker 1: people are probably not going to do it, you shouldn't
• 1:44:16
  Speaker 1: say this. If you think it's completely unlikely, like you
• 1:44:17
  Speaker 1: think there's no chance, then like maybe you shouldn't bother,
• 1:44:20
  Speaker 1: but we think there's a chance, like in particular for
• 1:44:22
  Speaker 1: the reasons that we described in the scenario, we think
• 1:44:25
  Speaker 1: that people are going to wake up to the power
• 1:44:28
  Speaker 1: of AI.
• 1:44:29
  Speaker 2: Over the next few.
• 1:44:29
  Speaker 1: Years because of something happens, the companies are saying that
• 1:44:33
  Speaker 1: they're going to do this and they are kind of
• 1:44:37
  Speaker 1: on track. And it just sort of makes sense that like,
• 1:44:40
  Speaker 1: if they get anywhere close to this level of AI,
• 1:44:44
  Speaker 1: then there's like big issues and big problems. And like,
• 1:44:46
  Speaker 1: we need to like do something about this. And so
• 1:44:49
  Speaker 1: I think that even if there's not any like very
• 1:44:52
  Speaker 1: dramatic warning shot or something. I think that just naturally
• 1:44:56
  Speaker 1: people are going to start paying more attention to this
• 1:44:57
  Speaker 1: and reasoning through the implications and trying to predict what's
• 1:45:00
  Speaker 1: going to happen. And so naturally people are going to
• 1:45:02
  Speaker 1: be more interested in regulation of AI.
• 1:45:06
  Speaker 2: For example. And in fact, there's actually more of this
• 1:45:11
  Speaker 2: happening than we predicted. More of what happening?
• 1:45:15
  Speaker 1: Serious interest in AI regulation. So at the time that
• 1:45:17
  Speaker 1: we published AI 2027, the sort of like mainstream position
• 1:45:22
  Speaker 1: of the tech companies and in the government was kind
• 1:45:24
  Speaker 1: of like, Yeah, regulation, bad idea.
• 1:45:26
  Speaker 2: Free for all. Free for all. Yeah. In fact, there
• 1:45:28
  Speaker 2: was even an attempt to preemptively ban states from regulating AI. Yeah.
• 1:45:34
  Speaker 2: You remember that? Now it seems like the conversation has
• 1:45:36
  Speaker 2: changed a lot.
• 1:45:37
  Speaker 1: Like now the US government just told Anthropic they have
• 1:45:40
  Speaker 1: to shut down their AI because they were worried that
• 1:45:43
  Speaker 1: bad actors would use it for cyber attacks. You know,
• 1:45:46
  Speaker 1: the government is like waking up and doing more stuff
• 1:45:49
  Speaker 1: than we expected already. And- We're actually hopeful that that
• 1:45:53
  Speaker 1: trend will just continue and that before it's actually too late,
• 1:45:57
  Speaker 1: there will be very serious conversations happening inside the government
• 1:46:00
  Speaker 1: and outside the government and in the broader society about
• 1:46:03
  Speaker 1: all of these issues and trying to chart a course
• 1:46:07
  Speaker 1: that avoids the loss of control and concentration of power
• 1:46:11
  Speaker 1: risk that we mentioned.
• 1:46:13
  Speaker 2: You've spent what must be almost coming up to 15
• 1:46:15
  Speaker 2: years thinking about this stuff. If this here was a
• 1:46:20
  Speaker 2: button and if you press that button, your plan S
• 1:46:24
  Speaker 2: would occur and it would shut down every data center
• 1:46:28
  Speaker 2: that is currently training a frontier AI model for good.
• 1:46:33
  Speaker 2: There would never be any other AI labs working on
• 1:46:37
  Speaker 2: these problems. Would you press that button? I was, I
• 1:46:40
  Speaker 2: was about to slam it until you said for good. Okay.
• 1:46:44
  Speaker 1: Like, I think, I think if it was a sort
• 1:46:45
  Speaker 1: of temporary shutdown, I would totally slam that button because
• 1:46:48
  Speaker 1: we are not ready to do this. You know, like,
• 1:46:52
  Speaker 1: So civilization is not ready to have these companies automate
• 1:46:56
  Speaker 1: themselves and then get smarter and smarter and then have
• 1:46:57
  Speaker 1: the super intelligence. Like, no, there's a bunch of reasons
• 1:46:59
  Speaker 1: why that's really dangerous. But I would be at least
• 1:47:04
  Speaker 1: hesitant to press this button if it permanently foreclosed the
• 1:47:08
  Speaker 1: possibility of ever doing it again, for sure.
• 1:47:10
  Speaker 2: But if you think that plan D is probable, which
• 1:47:13
  Speaker 2: is this race we're on to super intelligence. If I
• 1:47:15
  Speaker 2: had a choice between D and S, I think I
• 1:47:17
  Speaker 2: would press it. Well, it comes down to what you think, right?
• 1:47:20
  Speaker 2: Because if you think that is what's going to happen,
• 1:47:22
  Speaker 2: plan D, and the only alternative... I didn't say this
• 1:47:26
  Speaker 2: is what's going to happen. Probabilistically. Yeah, yeah.
• 1:47:29
  Speaker 1: I'd be like, this is the most likely, maybe this
• 1:47:31
  Speaker 1: is the second most likely, maybe this is the third
• 1:47:33
  Speaker 1: most likely.
• 1:47:34
  Speaker 2: They are all possible. So with your current perspective on
• 1:47:36
  Speaker 2: whatever one you think is going to happen, would you
• 1:47:38
  Speaker 2: press the button? I'm giving you an S, a definite S,
• 1:47:41
  Speaker 2: or whatever you think is going to happen. That's tough.
• 1:47:48
  Speaker 2: What is the scope of the shutdown? So is it...
• 1:47:51
  Speaker 2: It's no one can train an AI model again, ever again.
• 1:47:57
  Speaker 1: That's real rough because like I said, there's loads of
• 1:47:59
  Speaker 1: benefits that we could get from AI if we do
• 1:48:00
  Speaker 1: it right.
• 1:48:02
  Speaker 2: I think I've almost put you in the position of
• 1:48:04
  Speaker 2: Sam Altman to some degree. Yeah.
• 1:48:06
  Speaker 1: Do you mind if I just take a moment to
• 1:48:10
  Speaker 1: think about this?
• 1:48:11
  Speaker 2: I prefer you to think. Yeah. Yeah.
• 1:48:21
  Speaker 1: I think I would not press the button, but I
• 1:48:25
  Speaker 1: feel very torn about it. The reason why I think
• 1:48:27
  Speaker 1: I would not press the button is that I still
• 1:48:30
  Speaker 1: have substantial hope that we can get something much better
• 1:48:32
  Speaker 1: than this, something more like this. And I think that Basically,
• 1:48:37
  Speaker 1: I think that if we don't build powerful AI systems eventually,
• 1:48:42
  Speaker 1: then we're probably going to die as a civilization eventually,
• 1:48:47
  Speaker 1: you know, like 100 years from now, 200 years from now,
• 1:48:49
  Speaker 1: something like that, like nuclear war, pandemic, something, you know.
• 1:48:54
  Speaker 1: I don't think human civilization right now is like super,
• 1:48:57
  Speaker 1: super stable. And so I think that basically what I
• 1:49:03
  Speaker 1: was about to say was the possible benefits for posterity
• 1:49:07
  Speaker 1: and for all the billions and billions of people who
• 1:49:09
  Speaker 1: could live in the future outweigh the like the current
• 1:49:15
  Speaker 1: level of risk, but actually I've had that narrative before.
• 1:49:18
  Speaker 2: Yeah.
• 1:49:18
  Speaker 1: I don't know.
• 1:49:18
  Speaker 2: Like, yeah. Like maybe, maybe it's just like, Nope.
• 1:49:23
  Speaker 1: The people right now are the people we should prioritize.
• 1:49:26
  Speaker 1: People right now are in grave danger. They're going to
• 1:49:29
  Speaker 1: be fine for at least the next couple of decades.
• 1:49:32
  Speaker 1: So nevermind posterity. Prioritize the people right now. People right
• 1:49:39
  Speaker 1: now definitely don't want to do this lottery, I would say. Yeah,
• 1:49:46
  Speaker 1: you've really asked me a tough question.
• 1:49:48
  Speaker 2: So would you press the button if that was the button?
• 1:49:52
  Speaker 2: Probably not, but I would feel very torn. Okay. So
• 1:49:57
  Speaker 2: what I always think about the personas of the audience
• 1:49:59
  Speaker 2: that are watching, and these are, you know, they're very
• 1:50:02
  Speaker 2: curious people, especially on the subject of AI, as we've seen,
• 1:50:04
  Speaker 2: but they want to know, like, what it means for them.
• 1:50:07
  Speaker 2: I think a lot of them also want to know
• 1:50:08
  Speaker 2: what they can do. Ah, yes. Yeah. What can people do? Well,
• 1:50:12
  Speaker 2: I think that if you either have talent or passion,
• 1:50:17
  Speaker 2: you can get directly involved. There's lots of organizations that
• 1:50:20
  Speaker 2: are worried about these things and that are trying to
• 1:50:22
  Speaker 2: do something about it, like political advocacy or technical research
• 1:50:26
  Speaker 2: or like building useful tools that will hopefully help people
• 1:50:30
  Speaker 2: be better and stuff.
• 1:50:30
  Speaker 1: But if you don't want to make any major career
• 1:50:32
  Speaker 1: changes or things like that, then I would say just
• 1:50:37
  Speaker 1: pay more attention to these issues and talk about it
• 1:50:39
  Speaker 1: more with people. Do stuff like, you know, emailing your
• 1:50:42
  Speaker 1: congressman or whatever. It doesn't change things that much, but
• 1:50:46
  Speaker 1: it does help. I think that especially for this particular issue,
• 1:50:50
  Speaker 1: the core problem is that people aren't taking it seriously yet.
• 1:50:54
  Speaker 1: Like if the sorts of things that I was just
• 1:50:56
  Speaker 1: saying to you for the last hour or two were
• 1:50:58
  Speaker 1: just like top of everybody's mind, we wouldn't even be here.
• 1:51:03
  Speaker 1: Like there would already be much more significant regulation in place,
• 1:51:07
  Speaker 1: you know? And not only would there be more heavy
• 1:51:09
  Speaker 1: regulation in place, but there would have been better regulation
• 1:51:12
  Speaker 1: in place that's less, you know, less like a cudgel
• 1:51:15
  Speaker 1: and more like a scalpel. And it's like more sensitive
• 1:51:18
  Speaker 1: to what's actually bad and what's not so bad and
• 1:51:21
  Speaker 1: so forth. And there'd be more expert people in the
• 1:51:23
  Speaker 1: government and advising the government and so forth. So just
• 1:51:26
  Speaker 1: in general, like the more people wake up to these
• 1:51:30
  Speaker 1: concerns and to these projections, I think the more likely
• 1:51:34
  Speaker 1: it is that we can do good stuff before it's
• 1:51:35
  Speaker 1: too late.
• 1:51:36
  Speaker 2: What about how they should vote at the polls? We've
• 1:51:39
  Speaker 2: got an election coming up in the United States in
• 1:51:40
  Speaker 2: a couple of years' time, but there's elections happening all
• 1:51:42
  Speaker 2: over the world all the time.
• 1:51:44
  Speaker 1: You should ask your candidates what they think about all
• 1:51:46
  Speaker 1: this AI stuff. You should try to get them to
• 1:51:49
  Speaker 1: have opinions, and then you should vote for the candidates
• 1:51:51
  Speaker 1: whose opinions are better on this topic.
• 1:51:53
  Speaker 2: This is the most.
• 1:51:53
  Speaker 1: Important thing happening in our lifetimes, probably in all of history,
• 1:51:57
  Speaker 1: in fact, and it's very important that it go well.
• 1:52:00
  Speaker 1: And so it's what all the leaders of all the
• 1:52:02
  Speaker 1: countries should be thinking about and making plans for.
• 1:52:05
  Speaker 2: Isn't it such a weird thing to be alive at
• 1:52:06
  Speaker 2: this moment in time? Like I was thinking about all
• 1:52:08
  Speaker 2: the times that I could have been born. And I
• 1:52:10
  Speaker 2: guess my ancestors probably thought the same. But I was
• 1:52:12
  Speaker 2: thinking as you were speaking, I was like, I think
• 1:52:14
  Speaker 2: it's when you referred to it as like the final show. Yeah.
• 1:52:17
  Speaker 1: What was the phraseology you used? I said the run
• 1:52:20
  Speaker 1: up to the climax or something.
• 1:52:21
  Speaker 2: Yeah. I mean, what a crazy thing to be born
• 1:52:24
  Speaker 2: in the run up to the climax where everything you're
• 1:52:27
  Speaker 2: describing here is within my lifetime conceivably, hopefully.
• 1:52:30
  Speaker 1: Yeah.
• 1:52:30
  Speaker 2: Or maybe not, hopefully. What a crazy time to be alive. Certainly.
• 1:52:36
  Speaker 2: I noticed that when I asked you if you had kids,
• 1:52:38
  Speaker 2: your demeanor changed quite considerably. Well, it's, yeah. It's like
• 1:52:42
  Speaker 2: you dropped into a different state. Obviously that's been central
• 1:52:46
  Speaker 2: to the rumination that you've been experiencing.
• 1:52:50
  Speaker 1: Well, it is a sad topic, right? Like when I
• 1:52:53
  Speaker 1: had kids, like the reason to have kids is in
• 1:52:56
  Speaker 1: large part about the future, you know? Like, it's not
• 1:52:59
  Speaker 1: just like a cuddly thing to have with you in
• 1:53:01
  Speaker 1: the moment. It's because you have all these hopes and
• 1:53:03
  Speaker 1: dreams about how they'll grow up and how they'll go
• 1:53:05
  Speaker 1: do their own thing and be their own person and stuff.
• 1:53:07
  Speaker 1: And because of what's happening with AI, I think a
• 1:53:10
  Speaker 1: lot of those dreams are in jeopardy.
• 1:53:12
  Speaker 2: Presumably you still would have had kids. I've actually flip-flopped
• 1:53:14
  Speaker 2: on this. Really? Occasionally. Yeah.
• 1:53:16
  Speaker 1: Basically the top line answer is I'm not sure. My
• 1:53:20
  Speaker 1: first child was had, we had her when we were
• 1:53:22
  Speaker 1: in 2019, she was born in 2019.
• 1:53:25
  Speaker 2: Yeah.
• 1:53:25
  Speaker 1: So this is before my timeline shortened a lot. So
• 1:53:27
  Speaker 1: at this point I was interested in AI. I was
• 1:53:29
  Speaker 1: tracking the field. I was making forecasts, but I didn't
• 1:53:32
  Speaker 1: like actually expect it to happen soon, you know? And
• 1:53:36
  Speaker 1: then this caused like, when I, when I did start
• 1:53:39
  Speaker 1: thinking like, oh my gosh, it's going to be happening
• 1:53:40
  Speaker 1: like real soon. Um, like by 2030, you know, um,
• 1:53:45
  Speaker 1: that caused some reconsidering. And so I basically told my wife, like,
• 1:53:50
  Speaker 1: let's not have any more kids. It's too uncertain, you know?
• 1:53:54
  Speaker 1: But that turned out to be really hard because, especially
• 1:53:57
  Speaker 1: for my wife, like we already had one kid and
• 1:53:59
  Speaker 1: like no siblings. So eventually I sort of gave in
• 1:54:04
  Speaker 1: and was like, okay, well, you know what?
• 1:54:05
  Speaker 2: We already have one. It's going to be all right.
• 1:54:08
  Speaker 1: Like maybe, maybe the future will be good. And even
• 1:54:11
  Speaker 1: if it's not like, well, we're all in the same
• 1:54:13
  Speaker 1: boat together.
• 1:54:15
  Speaker 2: It's quite chilling what you're saying. It's chilling because you
• 1:54:17
  Speaker 2: know more than me. And if you're at home saying
• 1:54:20
  Speaker 2: to your wife, listen, maybe we should pause on having
• 1:54:22
  Speaker 2: more children and building a family because of what's going
• 1:54:24
  Speaker 2: on with AI. To be clear, yes. I mean, yes,
• 1:54:27
  Speaker 2: it's very concerning. I am chilled. This is bad.
• 1:54:31
  Speaker 1: This is what I've been saying. I hope things go well.
• 1:54:34
  Speaker 1: I think things might go well. I think that there's
• 1:54:37
  Speaker 1: a lot we can do to steer things in a
• 1:54:38
  Speaker 1: better direction. I mean, one of those things as well,
• 1:54:40
  Speaker 1: I have to say, is just speaking about it. I
• 1:54:44
  Speaker 1: think a lot of the progress we've seen with governments
• 1:54:46
  Speaker 1: waking up and you know, we've seen certain things with
• 1:54:49
  Speaker 1: people booing certain people at certain events. Is it downstream
• 1:54:53
  Speaker 1: from people like yourself actually coming on shows like this
• 1:54:56
  Speaker 1: and all the other podcasts and telling us what's going on?
• 1:55:01
  Speaker 1: Because elsewhere, to be fair, we're going to be gaslighted
• 1:55:03
  Speaker 1: by the people that have the biggest PR machines. So
• 1:55:06
  Speaker 1: I often, I think it's probably worth me saying, I
• 1:55:08
  Speaker 1: find myself kind of in two minds because I'm an
• 1:55:10
  Speaker 1: entrepreneur and I'm an investor. I'm an investor in probably
• 1:55:13
  Speaker 1: more than a hundred companies now. And Well, so many
• 1:55:15
  Speaker 1: of those companies aren't using AI. I invested in Grok,
• 1:55:18
  Speaker 1: the inference chip company. I've invested in SpaceX, which now
• 1:55:21
  Speaker 1: own another Grok and they're doing AI. I use AI every.
• 1:55:24
  Speaker 2: Day in my life. I've been using it through this
• 1:55:25
  Speaker 2: conversation to understand different things that you've said. So that's
• 1:55:28
  Speaker 2: one side of me, which is like business builder, entrepreneur,
• 1:55:31
  Speaker 2: who has seen the benefits of AI in my own life.
• 1:55:34
  Speaker 2: And then there's the other side of me. And it's
• 1:55:35
  Speaker 2: funny because I think sometimes people think you have to
• 1:55:37
  Speaker 2: pick a camp. But through all of my life, even
• 1:55:40
  Speaker 2: when I was a social media CEO and I was saying,
• 1:55:41
  Speaker 2: by the way, listen, I'm building a social media business,
• 1:55:43
  Speaker 2: but I think there's some downsides to social media. find
• 1:55:45
  Speaker 2: myself at the same moment where I'm like, I build
• 1:55:47
  Speaker 2: with AI, I have AI investments. And at the same
• 1:55:50
  Speaker 2: time as a civilian, I'm like. Yeah.
• 1:55:53
  Speaker 1: I mean, I think that is attention. I think that there's,
• 1:55:55
  Speaker 1: there's different ways you can draw the line. So, and
• 1:55:59
  Speaker 1: I know lots of people who draw the line in
• 1:56:01
  Speaker 1: lots of different ways. So like, there's some people who
• 1:56:02
  Speaker 1: just like, I'm not going to use AI. I think
• 1:56:04
  Speaker 1: this stuff is bad and on a bad trajectory. So
• 1:56:07
  Speaker 1: I'm going to like work out AI, right? I'm not
• 1:56:09
  Speaker 1: one of those people. I use AI a lot. We
• 1:56:11
  Speaker 1: all do at AI Futures Project. It's helpful for a
• 1:56:14
  Speaker 1: lot of our work. The opposite end of the spectrum
• 1:56:17
  Speaker 1: is people being like, well, it seems like it's on
• 1:56:22
  Speaker 1: a trajectory to happen. So the thing to do to
• 1:56:24
  Speaker 1: make it go well is to like get involved and
• 1:56:26
  Speaker 1: accumulate power and try to like steer it from the inside.
• 1:56:29
  Speaker 1: And so I'm going to go work at OpenAI or
• 1:56:31
  Speaker 1: Anthropic and like try to like climb the ranks and
• 1:56:34
  Speaker 1: then like, you know, be someone who matters when the
• 1:56:36
  Speaker 1: important decisions are being made. And I know loads of
• 1:56:38
  Speaker 1: people like that. That was like what I was doing
• 1:56:40
  Speaker 1: when I was, that wasn't what I was doing exactly,
• 1:56:42
  Speaker 1: but like.
• 1:56:43
  Speaker 2: That was the path.
• 1:56:44
  Speaker 1: That was like, that was, I mean, in some sense,
• 1:56:45
  Speaker 1: this is what the whole narrative of the companies are, right?
• 1:56:47
  Speaker 1: Like this is why they tell themselves it's okay to
• 1:56:49
  Speaker 1: do what they're doing is that they're worried about the
• 1:56:50
  Speaker 1: other guys, you know? And so like all these people
• 1:56:53
  Speaker 1: are deciding, like, we're going to like lean really hard
• 1:56:56
  Speaker 1: into it. We're going to like be there in the
• 1:56:58
  Speaker 1: room when the important decisions are being made, you know?
• 1:57:01
  Speaker 1: So there's a whole spectrum and I'm sort of like
• 1:57:02
  Speaker 1: somewhere in the middle. Like I'm not at the AI companies.
• 1:57:05
  Speaker 1: I'm not helping them go faster. Instead, I'm talking to
• 1:57:09
  Speaker 1: the broad public and trying to advocate for what I
• 1:57:11
  Speaker 1: think is the My current best guess as to the
• 1:57:15
  Speaker 1: way out, you know, the way forward. But I'm not
• 1:57:18
  Speaker 1: like boycotting all the AIs. I'm not like, you know,
• 1:57:21
  Speaker 1: trying to, I'm not refusing to like engage with it
• 1:57:24
  Speaker 1: in that way.
• 1:57:25
  Speaker 2: Do you think it's too late? No, I don't think
• 1:57:29
  Speaker 2: it's too late. If I thought it was too late,
• 1:57:30
  Speaker 2: I wouldn't be here. Where would you be? With my family.
• 1:57:36
  Speaker 2: What's your closing message to the general public? If you
• 1:57:40
  Speaker 2: had to have a closing statement to them?
• 1:57:42
  Speaker 1: Maybe I would say that like, You're going to hear
• 1:57:45
  Speaker 1: a lot of things, and you already have been hearing
• 1:57:46
  Speaker 1: a lot of things about AI, and it's going to
• 1:57:49
  Speaker 1: sound like science fiction. But sometimes things which sound like
• 1:57:53
  Speaker 1: science fiction happen in reality. And in fact, many times, historically,
• 1:57:58
  Speaker 1: things that used to be science fiction have then become reality.
• 1:58:01
  Speaker 1: And people need to stop thinking about what does or
• 1:58:04
  Speaker 1: doesn't sound like science fiction and just start thinking about
• 1:58:07
  Speaker 1: the trends and the actual trends that this technology is
• 1:58:11
  Speaker 1: on and reading and forecasting how it's going to go
• 1:58:15
  Speaker 1: and then taking seriously the possibility that it could go
• 1:58:18
  Speaker 1: something like this and then thinking about what should be
• 1:58:20
  Speaker 1: done about that.
• 1:58:21
  Speaker 2: And where would you direct them to get more information?
• 1:58:24
  Speaker 1: You can go to AI2037. com to read our previous scenario.
• 1:58:28
  Speaker 1: You can go to ai2040. com plan a to read our
• 1:58:32
  Speaker 1: new proposal for what needs to be done um these
• 1:58:35
  Speaker 1: things are not just a sci-fi story they also have
• 1:58:38
  Speaker 1: lots of like explainers and links to other things and
• 1:58:41
  Speaker 1: so they're kind of like a nice jumping off point
• 1:58:43
  Speaker 1: to to learn about all of this stuff um if
• 1:58:48
  Speaker 1: you want i could um after this is over like
• 1:58:50
  Speaker 1: give a reading list of like other papers and articles
• 1:58:54
  Speaker 1: and Please do. Blogs to follow and so forth. And
• 1:58:57
  Speaker 1: I'll link them all below in the comment section. So
• 1:58:59
  Speaker 1: if you're listening now, go ahead and take a look
• 1:59:01
  Speaker 1: at the description of this episode and you'll see a
• 1:59:04
  Speaker 1: bunch of links, which is Daniel's recommendations of what you
• 1:59:06
  Speaker 1: should read. You know, I think it's just a really,
• 1:59:09
  Speaker 1: really great moment in time to get educated on this stuff.
• 1:59:13
  Speaker 1: Humans have an inclination because of cognitive dissonance where we
• 1:59:15
  Speaker 1: feel uncomfortable about something to bury our heads in the
• 1:59:19
  Speaker 1: sand and avoid it. But actually, I think this is
• 1:59:21
  Speaker 1: one such time to do the very opposite for many
• 1:59:23
  Speaker 1: reasons to it. Inform yourself so you know what actions
• 1:59:26
  Speaker 1: to take, but also because AI, you know, unavoidably is
• 1:59:30
  Speaker 1: going to be a huge.
• 1:59:31
  Speaker 2: Part of all of our lives and careers. Yeah. Yeah.
• 1:59:33
  Speaker 2: Thank you.
• 1:59:33
  Speaker 1: And that's a good way to say it. It's going
• 1:59:36
  Speaker 1: to matter a lot. It's going to be everywhere soon.
• 1:59:39
  Speaker 1: And we need to do something about it before it's
• 1:59:42
  Speaker 1: too late.
• 1:59:42
  Speaker 2: And what about AI Future Project? That's our organization.
• 1:59:45
  Speaker 1: We spent a year writing AI 2027 after I left OpenAI,
• 1:59:48
  Speaker 1: and then we spent another year writing AI 2040 Plan A.
• 1:59:52
  Speaker 2: Daniel, thank you. Thank you. Thank you for all the
• 1:59:53
  Speaker 2: work that you do. I can see how much you
• 1:59:55
  Speaker 2: care about this stuff and it's your care. It's funny.
• 1:59:57
  Speaker 2: Care itself makes others feel care and seeing how personal
• 2:00:01
  Speaker 2: this is for you and seeing how much you've dedicated
• 2:00:02
  Speaker 2: your life to this, but also hearing that you, you
• 2:00:06
  Speaker 2: basically walked away from $ 2 million to be able to
• 2:00:08
  Speaker 2: speak to the public about this information is incredibly admirable.
• 2:00:12
  Speaker 2: And I think voices like yours are more important now
• 2:00:15
  Speaker 2: than they've ever been on this subject. So please do
• 2:00:17
  Speaker 2: keep fighting the fight that you're fighting. And that's one
• 2:00:19
  Speaker 2: of information. It is of honesty and it is a,
• 2:00:22
  Speaker 2: saying what is often the quiet part out loud. Thank you.
• 2:00:25
  Speaker 2: Doing really, really smart research. I'll link everything we've discussed
• 2:00:28
  Speaker 2: today below. And I hope we can chat again sometime soon.
• 2:00:31
  Speaker 2: Thank you. YouTube have this new crazy algorithm where they
• 2:00:34
  Speaker 2: know exactly what video you would like to watch next
• 2:00:37
  Speaker 2: based on AI and all of your viewing behavior. And
• 2:00:40
  Speaker 2: the algorithm says that this video is the perfect video
• 2:00:44
  Speaker 2: for you. It's different for everybody looking right now. Check
• 2:00:46
  Speaker 2: this video out. I bet you, you might love it.