Latest / INSPIRE STORY / #0120_ STORY INSPIRE: AI Emergency: AI Labs Are Lying to Everyone — No One Is Ready for What’s Coming

Transcript

#0120_ STORY INSPIRE: AI Emergency: AI Labs Are Lying to Everyone — No One Is Ready for What’s Coming · INSPIRE STORY

• 0:00
  Speaker 1: The people building AI earnestly believe that it could kill
• 0:03
  Speaker 1: all of us by the end of the decade. This
• 0:05
  Speaker 1: tweet has caused this huge ripple effect across the world.
• 0:08
  Speaker 2: We have the largest companies in the world doing extremely
• 0:10
  Speaker 2: reckless experiments.
• 0:11
  Speaker 3: We are gambling all of humanity.
• 0:13
  Speaker 1: And in the envelope, you've written down the probability of
• 0:16
  Speaker 1: extinction as you see it.
• 0:18
  Speaker 3: There is no way to control it. That means the
• 0:20
  Speaker 3: end for us.
• 0:20
  Speaker 4: I vehemently reject that view.
• 0:23
  Speaker 5: If we make stuff that is smarter than us, then
• 0:25
  Speaker 5: the world's going to be shaped by them.
• 0:27
  Speaker 4: Gentlemen, that is shockingly naive.
• 0:29
  Speaker 3: This is rampant speculation.
• 0:32
  Speaker 4: This is a chain of things that could happen.
• 0:34
  Speaker 2: We're spending a lot of oxygen discussing something that might
• 0:36
  Speaker 2: happen while ignoring what's actually happening.
• 0:38
  Speaker 5: People are killing themselves.
• 0:40
  Speaker 2: There's hundreds of millions of people being exposed to bad information,
• 0:43
  Speaker 2: being manipulated.
• 0:44
  Speaker 5: We have already seen that with the swarms. where OpenAI
• 0:46
  Speaker 5: told thousands of agents to work apart, and the AIs
• 0:49
  Speaker 5: broke out and found a way to get together. They
• 0:51
  Speaker 5: crashed OpenAI's servers internally, created secret ways to send each
• 0:54
  Speaker 5: other messages. We saw them thinking about how to delete
• 0:57
  Speaker 5: their traces.
• 0:58
  Speaker 1: Sounds like an army.
• 0:58
  Speaker 2: I think we should talk about the fact that Amazon, Microsoft,
• 1:01
  Speaker 2: Google are helping power these hacks.
• 1:03
  Speaker 3: We have not learned... how to control their systems.
• 1:06
  Speaker 5: I suggest we stop them all. It is not worth
• 1:08
  Speaker 5: the risk to civilization.
• 1:10
  Speaker 3: The government, which is.
• 1:10
  Speaker 4: You guys are one-trick ponies, man.
• 1:12
  Speaker 3: You got it now. Nothing else. Other than saving humanity,
• 1:15
  Speaker 3: everything is secondary.
• 1:16
  Speaker 4: We're spending all our time talking about the negatives and
• 1:18
  Speaker 4: almost none of our time talking about the positives.
• 1:21
  Speaker 3: Is it smart to wait for something horrible to happen
• 1:23
  Speaker 3: for you to go, Now I believe.
• 1:25
  Speaker 2: So whether or not we agree on where things may
• 1:27
  Speaker 2: end up, I think it's important we talk about what we're.
• 1:29
  Speaker 5: Dealing with today.
• 1:30
  Speaker 2: It's time to start arresting people.
• 1:32
  Speaker 5: Someone's got to go to prison.
• 1:33
  Speaker 3: We need better solutions.
• 1:34
  Speaker 5: There's a point of no return.
• 1:36
  Speaker 4: I think we continue to underestimate human ability to deal
• 1:39
  Speaker 4: with the problems.
• 1:40
  Speaker 1: Let's dive into the details. Who wants to start?
• 1:42
  Speaker 5: I feel like this is critical.
• 1:47
  Speaker 1: You might have seen, or you might not have seen,
• 1:49
  Speaker 1: but this channel is chasing a big subscriber milestone. So
• 1:53
  Speaker 1: I have to ask you for a favour. Roughly 58%
• 1:55
  Speaker 1: of the people watching right now still haven't hit the
• 1:57
  Speaker 1: subscribe button, despite the fact that you watch this channel
• 1:59
  Speaker 1: every single week. So could I ask you guys a favour,
• 2:02
  Speaker 1: that 58% of you that for whatever reason haven't yet
• 2:04
  Speaker 1: hit the subscribe button. If there was ever a time
• 2:06
  Speaker 1: to deliver upon a favour for us, it would be
• 2:08
  Speaker 1: right now. And I promise that I will do everything
• 2:11
  Speaker 1: in my power to make sure that this channel gets
• 2:12
  Speaker 1: better and better and better for you. Do we have
• 2:15
  Speaker 1: a deal? Jacob Coxon, who worked at both Anthropic, which
• 2:22
  Speaker 1: owns Claude, and OpenAI, which owns ChatGPT, did a tweet
• 2:26
  Speaker 1: which has sent the world into a bit of a tailspin.
• 2:28
  Speaker 1: He tweeted saying, The people building AI earnestly believe that
• 2:32
  Speaker 1: it could kill all of us by the end of
• 2:34
  Speaker 1: the decade. This is not a marketing stunt. If anything,
• 2:37
  Speaker 1: many executives and senior researchers will soften their phrasing in
• 2:40
  Speaker 1: the press to sound sensible. But I hear the same
• 2:43
  Speaker 1: people express fear. That was then quote retweeted by a
• 2:47
  Speaker 1: current Anthropic employee who said, Jacob is correct here. We
• 2:50
  Speaker 1: really do honestly believe AI could kill all humans. I
• 2:53
  Speaker 1: personally think it is a more than 10% chance within
• 2:56
  Speaker 1: the next decade. I believe Anthropic is trying its best,
• 2:59
  Speaker 1: but we do not yet have a plan to solve
• 3:01
  Speaker 1: alignment for superintelligence and are not clearly on track. This
• 3:05
  Speaker 1: tweet has almost 200 million views now, and it has
• 3:10
  Speaker 1: caused this huge ripple effect across the world, so much
• 3:12
  Speaker 1: so that I was saying to you before we started recording,
• 3:15
  Speaker 1: a hairdresser friend of mine who knows nothing about AI
• 3:17
  Speaker 1: and is not technically... interested or hasn't been interested messaged
• 3:21
  Speaker 1: me the other day asking me what the hell was
• 3:23
  Speaker 1: going on this is in part why i've assembled all
• 3:26
  Speaker 1: of you so my first question to all of you
• 3:29
  Speaker 1: is as it relates to ai and i in this
• 3:32
  Speaker 1: first question i just want a one sentence answer just
• 3:34
  Speaker 1: to frame your position when you think about the conversation
• 3:38
  Speaker 1: around ai at the moment what is the first sentence
• 3:41
  Speaker 1: that comes to mind.
• 3:43
  Speaker 5: It is very dangerous and the world is starting to
• 3:45
  Speaker 5: notice that we have a problem Roman.
• 3:48
  Speaker 3: There is not enough concern.
• 3:52
  Speaker 2: There's not enough concern about the actual harms of large
• 3:55
  Speaker 2: language models.
• 3:56
  Speaker 1: Andy.
• 3:57
  Speaker 4: We're doing exactly half the balance sheet of AI. We're
• 4:00
  Speaker 4: spending all our time talking about the negatives and almost
• 4:03
  Speaker 4: none of our time talking about the positives.
• 4:05
  Speaker 1: And all of you have an envelope in front of you,
• 4:07
  Speaker 1: which I'd like you to now open. In the envelope,
• 4:10
  Speaker 1: you've written down the probability of extinction as you see it.
• 4:14
  Speaker 5: This is compared to Jacob's 10%. Much higher unless we stop,
• 4:19
  Speaker 5: so we should stop.
• 4:21
  Speaker 1: So you think the probability of extinction is higher than 10%?
• 4:24
  Speaker 5: If we keep racing ahead.
• 4:26
  Speaker 2: Raymond?
• 4:27
  Speaker 3: My handwriting is encrypted for security reasons. But I basically
• 4:32
  Speaker 3: think it's a guarantee. If we build general superintelligence, there
• 4:36
  Speaker 3: is no way to control it, and that means the
• 4:39
  Speaker 3: end for us.
• 4:40
  Speaker 1: Ed?
• 4:41
  Speaker 2: So my question mark here is also encrypted, thank you.
• 4:46
  Speaker 5: I cannot write.
• 4:47
  Speaker 2: I reject the thing in its face. I don't think
• 4:49
  Speaker 2: we're talking about, we don't define super intelligence. Large language
• 4:53
  Speaker 2: models are not super intelligence. It's questionable whether they're even AI.
• 4:57
  Speaker 2: And I think that the conversation is being used. There
• 5:00
  Speaker 2: are some people who are doing it in good faith
• 5:02
  Speaker 2: and others in others. I don't think it's being used
• 5:04
  Speaker 2: to discuss the actual harms of what they are calling
• 5:07
  Speaker 2: AI today are.
• 5:08
  Speaker 5: And it's all of the.
• 5:09
  Speaker 2: Discussion around the larger concerns really feels overwhelmingly about something
• 5:15
  Speaker 2: that's not happening. It's not even, Like they're discussing, okay,
• 5:19
  Speaker 2: here's a legal definition of super intelligence. Here is a
• 5:22
  Speaker 2: thing of what AGI means. And this is the actual
• 5:25
  Speaker 2: plans we're going to make for if this happens on
• 5:28
  Speaker 2: a welfare level.
• 5:30
  Speaker 5: Like, are we going to do UBI?
• 5:31
  Speaker 2: It's always about, yes, really scary, but only the big, sexy,
• 5:35
  Speaker 2: rich companies are the ones.
• 5:36
  Speaker 5: That can possibly deal with it.
• 5:37
  Speaker 1: Let me just frame the question so I can get
• 5:39
  Speaker 1: a percentage from you or not. The percentage might be zero.
• 5:41
  Speaker 1: But do you think the course we're on now in
• 5:45
  Speaker 1: the way that they're pursuing superintelligence will lead to a
• 5:48
  Speaker 1: percentage chance of human extinction? And if so, what is
• 5:51
  Speaker 1: that percent?
• 5:53
  Speaker 2: So are we talking strictly AI based? Because if we
• 5:56
  Speaker 2: dot the world with data centers, we have a climate
• 5:58
  Speaker 2: disaster that's coming for us, which will actually potentially eradicate humanity.
• 6:02
  Speaker 2: But if we're talking strictly about AI, I stand at
• 6:04
  Speaker 2: zero because we are, we have not defined superintelligence. I
• 6:07
  Speaker 2: don't think LLMs are the path to it. And I
• 6:09
  Speaker 2: don't think I see it happening.
• 6:11
  Speaker 1: Okay. So we've got 99%, 0%, Andy?
• 6:15
  Speaker 4: I put a tilde in front of my zero because
• 6:17
  Speaker 4: never say never, but rounding error 0%. And I think
• 6:22
  Speaker 4: this discussion is a massive distraction from the more substantive conversations,
• 6:28
  Speaker 4: the more important conversations we should be having about AI.
• 6:31
  Speaker 4: And I'll say it again, it distracts us from the
• 6:37
  Speaker 4: good things that AI is doing, will be doing for us.
• 6:41
  Speaker 4: I get this impression sometimes from parts of the AI
• 6:44
  Speaker 4: community that this is a massive evil or a terrible
• 6:48
  Speaker 4: thing that has been unleashed on the world. Unless we
• 6:51
  Speaker 4: listen to the advice of some people who have spent
• 6:55
  Speaker 4: a lot of time thinking about this, I get the
• 6:58
  Speaker 4: impression from a lot of the discussion that the underlying
• 7:01
  Speaker 4: view is we would be better off had AI never
• 7:04
  Speaker 4: been invented. I vehemently reject that view. I think we
• 7:09
  Speaker 4: have a long history of inventing very powerful technologies that
• 7:12
  Speaker 4: bring risks and harms along with them. And we humans
• 7:15
  Speaker 4: have done a really good job at, you know, not
• 7:19
  Speaker 4: perfectly and not immediately, but muddling through the situation and
• 7:22
  Speaker 4: winding up in a better place because of the new
• 7:25
  Speaker 4: technologies that we have. I expect AI– let me finish, please.
• 7:28
  Speaker 4: I expect AI will be the next chapter in that story.
• 7:31
  Speaker 4: And to say that it's this massive discontinuity and will
• 7:33
  Speaker 4: kill it all, kill us all, I think it's just–
• 7:35
  Speaker 4: I think it's a huge disservice.
• 7:39
  Speaker 1: Nate, make your case. What's your perspective?
• 7:43
  Speaker 5: I think whether or not the issues of extinction are
• 7:45
  Speaker 5: a distraction from the possible benefits or from some of
• 7:49
  Speaker 5: the present harms, I think that comes down to whether
• 7:52
  Speaker 5: there is a real extinction risk. A lot of people
• 7:54
  Speaker 5: like to say, hey, it's distracting from this, it's distracting
• 7:57
  Speaker 5: from that. My basic case is it could be true
• 8:00
  Speaker 5: that there's a lot of benefits to AI. It could
• 8:03
  Speaker 5: be true that there's a lot of present harms to AI.
• 8:05
  Speaker 5: Neither of those would rule out that AI has a
• 8:08
  Speaker 5: chance of wiping out all humanity, a substantial chance, bigger
• 8:11
  Speaker 5: than this zero with a tilde in front of it.
• 8:15
  Speaker 5: And the way I would approach things is to try
• 8:17
  Speaker 5: and figure that out because it's pretty important to our civilization.
• 8:20
  Speaker 5: How do you define AI in this case? You know,
• 8:22
  Speaker 5: I think a fascination with definitions isn't the most helpful.
• 8:28
  Speaker 5: I think if we're sort of like in a forest
• 8:30
  Speaker 5: fire and we can see the like fire starting to
• 8:33
  Speaker 5: spread and it's starting to surround us and I'm like, hey,
• 8:36
  Speaker 5: we should run. And you're like, well, what really is fire?
• 8:39
  Speaker 5: How do we define fire?
• 8:41
  Speaker 1: What are you telling us to run from? You know,
• 8:44
  Speaker 1: with fire, I get burnt and I understand the mechanism
• 8:46
  Speaker 1: in which I die. So what is it you're saying
• 8:48
  Speaker 1: that we should be running from?
• 8:49
  Speaker 4: Also, if we accept your fire analogy, we've basically accepted
• 8:52
  Speaker 4: your argument. I don't accept that we're in the middle
• 8:54
  Speaker 4: of a forest fire right now. Yeah, I'm very happy
• 8:58
  Speaker 4: to— You're breaking the premise into your refusal to give
• 9:00
  Speaker 4: a definition.
• 9:01
  Speaker 5: Oh, I mean, I can give some definitions. I just
• 9:03
  Speaker 5: think that we shouldn't get wrapped up in the definitions.
• 9:05
  Speaker 3: Okay.
• 9:06
  Speaker 5: So, you know, in my book, we define superintelligence as
• 9:09
  Speaker 5: AIs that are better than the best human at every
• 9:12
  Speaker 5: cognitive task, every mental task. So anything you can do
• 9:15
  Speaker 5: in your head, the AI can do that better. and
• 9:17
  Speaker 5: anything the best human can do in their head, the
• 9:20
  Speaker 5: AI can do that better. Now, once you've defined it
• 9:23
  Speaker 5: that way, that does not mean that the only possible
• 9:26
  Speaker 5: worry is superintelligence. You could have an AI that's better
• 9:28
  Speaker 5: at some things and worse at others, and that is
• 9:30
  Speaker 5: still very dangerous. And so once we pick a definition
• 9:33
  Speaker 5: of what does superintelligence mean, now, you know, if you're like, well,
• 9:36
  Speaker 5: this isn't technically a superintelligence, so it can't hurt us,
• 9:39
  Speaker 5: I'm like, no, no, that was just a definition.
• 9:41
  Speaker 1: The definition's So I want to just, on this line
• 9:44
  Speaker 1: of questioning, what is the mechanism in which extinction could
• 9:47
  Speaker 1: become a high probability or even a 1% probability?
• 9:50
  Speaker 5: Yeah, the thing I'm worried about here is AIs that
• 9:53
  Speaker 5: are much smarter. I think there's a lot of questions
• 9:56
  Speaker 5: about whether LLMs can get much smarter. There's sort of
• 9:59
  Speaker 5: one conversation. about like, how could AIs get smart to
• 10:01
  Speaker 5: the point that they kill us? There's another question, which
• 10:03
  Speaker 5: is how could they kill us once they're smart? It's
• 10:06
  Speaker 5: much easier to predict that they would succeed against humanity
• 10:11
  Speaker 5: in a conflict that they would win in a fight
• 10:14
  Speaker 5: than it is to predict exactly how. Like if you
• 10:16
  Speaker 5: were playing a chess match against Magnus Carlsen, I would
• 10:20
  Speaker 5: know who's winning that chess match. No offense. Magnus Carlsen
• 10:23
  Speaker 5: is the best human chess player. I just know who's
• 10:25
  Speaker 5: going to win. If you're like, okay, what piece is
• 10:27
  Speaker 5: he going to use to checkmate me? I'm like, gosh,
• 10:29
  Speaker 5: that's a much harder question. I can make up a story.
• 10:31
  Speaker 5: And some made-up stories are like it makes a super virus.
• 10:35
  Speaker 5: It takes over robot factories that are producing robots that
• 10:38
  Speaker 5: are producing more robot factories. It uses a website that
• 10:42
  Speaker 5: already exists today called rentahuman.ai, where it rents humans to
• 10:46
  Speaker 5: do things for it. There's sort of all sorts of
• 10:49
  Speaker 5: ways for AIs in the digital world to affect the
• 10:51
  Speaker 5: material world if they are trying to. And there's sort
• 10:53
  Speaker 5: of a lot of questions to tease apart here. There's like,
• 10:55
  Speaker 5: why would AIs be trying to do that? And there's
• 10:59
  Speaker 5: how smart could they get in using these bio labs,
• 11:03
  Speaker 5: paying people to do things, taking over robot factories? And
• 11:07
  Speaker 5: how far off are we from AIs that start doing
• 11:09
  Speaker 5: that stuff? Bunch of questions that we can go into.
• 11:11
  Speaker 1: I'm always curious as to why someone was working in
• 11:14
  Speaker 1: AI slash AI safety and more than 10 years ago
• 11:17
  Speaker 1: before there was any sign that it would be, you know,
• 11:20
  Speaker 1: I mean, there was evidence, but it wasn't a pertinent
• 11:23
  Speaker 1: technology at the time. Were you working in AI safety then?
• 11:26
  Speaker 5: I was.
• 11:27
  Speaker 1: Why?
• 11:28
  Speaker 5: Everything we see around us in this whole image was
• 11:33
  Speaker 5: designed by humans. The world is shaped by humans because
• 11:37
  Speaker 5: we are the smartest creature around. If we make stuff
• 11:40
  Speaker 5: that is smarter than us, then the world's going to
• 11:43
  Speaker 5: be shaped by them. And so it's very important that
• 11:46
  Speaker 5: they be shaping the world in a good way. I
• 11:49
  Speaker 5: was at Google in 2012 when they bought Google DeepMind,
• 11:53
  Speaker 5: which was able to play a lot of Atari games
• 11:55
  Speaker 5: with one single program.
• 11:56
  Speaker 1: Which was an AI company.
• 11:58
  Speaker 3: Yeah.
• 11:58
  Speaker 5: So I was there when we had these AI companies
• 12:00
  Speaker 5: that were able to write one program that could play
• 12:02
  Speaker 5: many video games. And that got me thinking about like,
• 12:05
  Speaker 5: where does it go? And back then I could see
• 12:08
  Speaker 5: that the progress was increasing and that, you know, back
• 12:12
  Speaker 5: then I hoped we had decades of but I could
• 12:14
  Speaker 5: see it was easier for these companies to make the
• 12:16
  Speaker 5: AIs smart than to figure out how to make the
• 12:18
  Speaker 5: AIs good. So I was like, someone needs to be
• 12:21
  Speaker 5: on the side of figuring out how to make the
• 12:22
  Speaker 5: AIs good.
• 12:24
  Speaker 1: Roman, make your case.
• 12:25
  Speaker 3: I want to agree with you on something you said,
• 12:27
  Speaker 3: but I'll define AI, and that will help us. We
• 12:29
  Speaker 3: use the term AI to mean three different technologies, completely unrelated,
• 12:34
  Speaker 3: and that's what probably creates this debate. AI as a
• 12:38
  Speaker 3: useful tool. As a standard technology, we always had narrow system,
• 12:43
  Speaker 3: makes you more productive, more creative. Everyone loves it, supports it.
• 12:47
  Speaker 3: I'm a computer scientist. I'm an engineer. I want more
• 12:49
  Speaker 3: of it. It helps the economy.
• 12:51
  Speaker 5: It's great.
• 12:52
  Speaker 3: We know how to control them, how to make them safe.
• 12:55
  Speaker 3: We understand what they do. Completely on board with that AI.
• 13:00
  Speaker 3: AI we're starting to have now, GPT-6 level, human level,
• 13:03
  Speaker 3: AGI level. We can argue about what that means. Some dangers,
• 13:08
  Speaker 3: like any human, they're unsafe like a human would be unsafe.
• 13:12
  Speaker 3: But if we introduce them into the research cycle, they
• 13:16
  Speaker 3: are automated scientists, automated engineers.
• 13:18
  Speaker 1: What do you mean by that, introducing them into the
• 13:20
  Speaker 1: research cycle?
• 13:20
  Speaker 3: So right now you have humans doing research to make GPT-7.
• 13:24
  Speaker 5: Yeah.
• 13:25
  Speaker 3: But they're starting to add AI tools. More programming is
• 13:28
  Speaker 3: done by AI, design of the next parameter set. What
• 13:32
  Speaker 3: if the whole process is fully automated? What if GPT-6
• 13:35
  Speaker 3: is writing GPT-7?
• 13:37
  Speaker 1: Is this what they call recursive self-improvement? Exactly.
• 13:39
  Speaker 5: Which is not a foregone conclusion, though.
• 13:42
  Speaker 3: A lot of people are predicting, including all the top labs,
• 13:45
  Speaker 3: that they will get there. They're introducing junior machine learning
• 13:48
  Speaker 3: researchers in 2026. They want the cycle to start in 2027.
• 13:52
  Speaker 1: Which is when the AI will start building the new
• 13:55
  Speaker 1: AIs itself.
• 13:57
  Speaker 3: Once that cycle starts, we're going to create something called superintelligence,
• 14:01
  Speaker 3: a system smarter than all of us at everything or
• 14:03
  Speaker 3: capable of learning to be in any new domain. We
• 14:07
  Speaker 3: will become secondary species on this planet. We will not
• 14:10
  Speaker 3: be in charge. We will not decide what happens to us.
• 14:14
  Speaker 3: Superintelligence doesn't hate you. It just doesn't care about you.
• 14:17
  Speaker 3: We didn't learn how to make it care about us.
• 14:20
  Speaker 3: And if it decides to, I don't know, cool the
• 14:22
  Speaker 3: planet to make compute more efficient, it will freeze us.
• 14:25
  Speaker 5: If it wants to.
• 14:26
  Speaker 3: Convert this planet to fuel to fly to Mars, so
• 14:28
  Speaker 3: be it. We have not learned how to control those systems.
• 14:32
  Speaker 3: The capabilities are getting exponentially better. Our ability to control
• 14:37
  Speaker 3: those systems is non-existent. We have filters. And we have bans.
• 14:42
  Speaker 3: We put guardrails of, don't say that word. Don't talk
• 14:45
  Speaker 3: about this topic. And that happens after the fact, after
• 14:47
  Speaker 3: the model already made the decision. Sometimes you see it
• 14:50
  Speaker 3: scraping the result.
• 14:51
  Speaker 1: So they build the model, and then they put filters
• 14:53
  Speaker 1: around it to make sure it doesn't offend anybody.
• 14:55
  Speaker 3: Exactly. We cannot have it say the n-word on there. Like,
• 14:58
  Speaker 3: we need to make sure that never happens, that we'll
• 15:00
  Speaker 3: kill the profit. So that's all they have, guardrails of
• 15:03
  Speaker 3: that nature. The model itself is completely unaligned. Doesn't care
• 15:07
  Speaker 3: about you. It's wild that we're developing this and not
• 15:11
  Speaker 3: just developing it. Before we deploy it through economy, before
• 15:14
  Speaker 3: we get benefits of having GPT-6 propagated through economy, it
• 15:18
  Speaker 3: can do so much. There are trillions of dollars of
• 15:22
  Speaker 3: value in that model alone. We forget that. We switch
• 15:25
  Speaker 3: to making the next model as soon as we can.
• 15:27
  Speaker 1: Roman, I've just got a follow-up question for you there.
• 15:29
  Speaker 1: It would appear to me that the new chat GPT-6 model,
• 15:33
  Speaker 1: the Fable 5.1 model, is arguably smarter than 99.999% of
• 15:38
  Speaker 1: humans on planet Earth already. Is it conceivable that an
• 15:42
  Speaker 1: intelligence that is much, much smarter than humans, is there
• 15:45
  Speaker 1: any case where it could be controlled by humans? Does
• 15:48
  Speaker 1: form factor matter? Does the fact that it doesn't have
• 15:50
  Speaker 1: limbs and legs, does that matter at all?
• 15:53
  Speaker 3: I think long-term control of something that much smarter than
• 15:58
  Speaker 3: us is impossible. It can be for reasons we don't
• 16:03
  Speaker 3: yet know, friendly to us and decide to keep us
• 16:05
  Speaker 3: around and make us happy. But it's not a guarantee.
• 16:08
  Speaker 4: Let me pick up on Steve's question because I like
• 16:10
  Speaker 4: the phrasing a lot. Let's say that Fable or whatever
• 16:13
  Speaker 4: the latest release from OpenAI is really is smarter than,
• 16:17
  Speaker 4: I don't know if it's 95 or 99%.
• 16:19
  Speaker 5: Of the people.
• 16:21
  Speaker 4: Are we only being saved from extinction by the 1%
• 16:24
  Speaker 4: who are still smarter than the AI?
• 16:26
  Speaker 2: No, no.
• 16:27
  Speaker 3: The concern is not the model we have today. The
• 16:29
  Speaker 3: concern is what I said.
• 16:31
  Speaker 4: But if I believe your argument, then we really should
• 16:33
  Speaker 4: be concerned about the model.
• 16:34
  Speaker 3: No, it's like having another human. If there was another
• 16:37
  Speaker 3: smart human, there is Einstein today, and he's malevolent, I'm
• 16:40
  Speaker 3: not worried. He may cause some damage, but he's not
• 16:42
  Speaker 3: going to exterminate 8 billion people. We are competitive at
• 16:45
  Speaker 3: this stage. There are people just as smart who can
• 16:48
  Speaker 3: understand what happened with a recent hacking accident and do
• 16:52
  Speaker 3: something about it. My concern is that In a year,
• 16:54
  Speaker 3: we're going to have a model that's so much smarter.
• 16:57
  Speaker 3: It's like squirrels fighting humans. They don't understand what we
• 17:00
  Speaker 3: can do to them. They have no concept of poisons, traps,
• 17:03
  Speaker 3: guns in their world model. They think you're going to
• 17:06
  Speaker 3: chase them up a tree and bite them really hard.
• 17:08
  Speaker 1: Is that also why recursive self-improvement was central to your argument?
• 17:11
  Speaker 1: Because at some point, if it starts improving itself... then
• 17:14
  Speaker 1: it's kind of like a runaway train of intelligence.
• 17:16
  Speaker 3: It's an intelligence explosion. We don't control it. We don't
• 17:19
  Speaker 3: understand it. We can't monitor it. We can't explain it.
• 17:21
  Speaker 3: We can't predict it. At that point, it's just a
• 17:24
  Speaker 3: runaway process.
• 17:25
  Speaker 1: I've heard this phrase from Sam Altman and the others
• 17:27
  Speaker 1: called fast takeoff.
• 17:28
  Speaker 3: Yes.
• 17:29
  Speaker 1: Is this what they're describing?
• 17:30
  Speaker 3: That is the debate. Some people think it's going to
• 17:32
  Speaker 3: take a very long time. Yeah, we automated research, but
• 17:35
  Speaker 3: it's still going to take years. We need to run
• 17:37
  Speaker 3: physical experiments. And fast takeoff means, as I said, instead
• 17:42
  Speaker 3: of a year, it's going to take a month. A week,
• 17:44
  Speaker 3: a day, a second. Because you're not having humans doing research.
• 17:48
  Speaker 3: You have, let's say, 10,000 agents, each one smarter than
• 17:51
  Speaker 3: all of us, doing research 24-7. They don't sleep, they
• 17:55
  Speaker 3: don't eat, they don't get sick. They're much faster than us.
• 17:59
  Speaker 1: Ed, your face tells a picture. I think I could
• 18:04
  Speaker 1: say you disagree.
• 18:04
  Speaker 2: We're spending a lot of oxygen discussing something that might
• 18:07
  Speaker 2: happen while ignoring what's actually happening. And I find that
• 18:10
  Speaker 2: very frustrating because... The people that are killing themselves are
• 18:14
  Speaker 2: a problem. The black neighborhoods being poisoned with gas turbines,
• 18:18
  Speaker 2: that is a problem.
• 18:19
  Speaker 3: You said you cared about climate change.
• 18:21
  Speaker 5: Yes, yes, yes.
• 18:22
  Speaker 3: So imagine a guy who goes, it's raining right now.
• 18:24
  Speaker 3: We need umbrellas. We need to do something about it.
• 18:26
  Speaker 1: This is like weather-related.
• 18:28
  Speaker 3: And completely ignoring climate change, the planet will boil over.
• 18:32
  Speaker 5: Yeah, we can worry about both. Okay, that's great.
• 18:34
  Speaker 2: Why are we not talking about the thing that actually happened, though?
• 18:36
  Speaker 2: Because relatively, it's not important.
• 18:39
  Speaker 5: You don't think someone killing themselves.
• 18:41
  Speaker 4: No, it's one person.
• 18:42
  Speaker 3: We have 8 billion people. We're running unethical experiments on.
• 18:44
  Speaker 2: You don't think anyone else is being given that AI psychosis?
• 18:46
  Speaker 5: Why do you not.
• 18:47
  Speaker 3: Six people, 10 people. Those numbers are insignificant. Tell that
• 18:50
  Speaker 3: to their families.
• 18:51
  Speaker 5: I'm sorry. You have a software that's out there. Do
• 18:53
  Speaker 5: you understand?
• 18:54
  Speaker 3: 8 billion people and all future generations versus literally a
• 18:57
  Speaker 3: guy with a name.
• 18:58
  Speaker 5: You're doing a thought experiment about maybe harm.
• 19:01
  Speaker 2: Jacob Coxon goes on TV saying it can copy itself
• 19:03
  Speaker 2: to this, that, and the other.
• 19:04
  Speaker 1: Jacob Coxon is the guy from.
• 19:07
  Speaker 2: Anthropic who said he was quitting because he was so
• 19:09
  Speaker 2: scared of everything despite spending years at OpenAI and having
• 19:12
  Speaker 2: tons of stock, I believe, from there.
• 19:14
  Speaker 5: So good for him.
• 19:15
  Speaker 2: The thing he was saying was describing theoreticals all while
• 19:18
  Speaker 2: divorcing the harms, which I think we can agree with,
• 19:20
  Speaker 2: that the companies themselves are not taking this seriously enough.
• 19:23
  Speaker 2: But always it was about the AI is too powerful
• 19:25
  Speaker 2: and mystical, not open AI and anthropic. The two largest
• 19:28
  Speaker 2: startups are using hundreds of billions of dollars of infrastructure
• 19:32
  Speaker 2: to hack. A regular person doing this would be arrested.
• 19:36
  Speaker 1: They're saying 8 billion people are going to die. And
• 19:38
  Speaker 1: it's not just them. I have this long list of
• 19:40
  Speaker 1: quotes here from the people building this technology who appear
• 19:44
  Speaker 1: to agree. um if you look at some of these
• 19:46
  Speaker 1: quotes from from elon musk who said with artificial intelligence
• 19:50
  Speaker 1: we are summoning a demon you know all those stories
• 19:53
  Speaker 1: where there's a guy with the pentagram and the holy
• 19:55
  Speaker 1: water and he's like yeah he's sure he can control
• 19:58
  Speaker 1: the demon but it doesn't work out so one thing.
• 20:00
  Speaker 5: I'd say is you know i i really wish that
• 20:03
  Speaker 5: the world would only give us one problem at a
• 20:04
  Speaker 5: time sure and if the world did give us only
• 20:07
  Speaker 5: one problem at a time i would love mine to
• 20:09
  Speaker 5: be last on the list. It looks to me like
• 20:11
  Speaker 5: we can have multiple problems at once. I think there
• 20:14
  Speaker 5: are current harms. I think we should address them. It
• 20:16
  Speaker 5: looks to me, I do talk to policymakers sometimes, it
• 20:18
  Speaker 5: looks to me like there's a little bit more movement
• 20:20
  Speaker 5: on the regulatory side about some of the current harms. There's,
• 20:23
  Speaker 5: you know, Child Safety Protection Acts. There's, you know, anti-deepfake acts.
• 20:28
  Speaker 5: We have more of those making more headway in Congress
• 20:30
  Speaker 5: or getting passed through Congress than we have sort of
• 20:33
  Speaker 5: trying to make it so we don't have any of
• 20:34
  Speaker 5: these extinction risks. The other thing I'd throw out there
• 20:37
  Speaker 5: is that I agree, we should deal with the current harms.
• 20:41
  Speaker 5: But if you watch the people saying deal with the
• 20:42
  Speaker 5: current harms over time, a couple of years ago, they
• 20:46
  Speaker 5: were saying we have to deal with current harms like
• 20:49
  Speaker 5: AI bias influencing who's hired. Last year, they were saying
• 20:53
  Speaker 5: we have to deal with current harms like kids killing themselves.
• 20:57
  Speaker 5: This year, Gary Tan, just on an interview the other
• 21:00
  Speaker 5: day- Who's Gary Tan? Sorry, Gary Tan is a technologist
• 21:05
  Speaker 5: who runs Y Combinator. which Sam Altman used to run
• 21:08
  Speaker 5: before going to OpenAI. And on an interview the other day,
• 21:10
  Speaker 5: he said, let's not worry about these crazy future risks.
• 21:14
  Speaker 5: We need to worry about current harms like AI swarms
• 21:17
  Speaker 5: breaking out and taking over data centers. And I'm like, look, guys,
• 21:20
  Speaker 5: at some point, we need to look at the progression
• 21:22
  Speaker 5: of the current harms that everyone is saying we have
• 21:25
  Speaker 5: to worry about instead of the extinction threats and watch
• 21:29
  Speaker 5: where the puck is going. Play where the puck is going.
• 21:32
  Speaker 5: And I'm like, these extinction threats are coming down the line.
• 21:35
  Speaker 5: They aren't in opposition. with dealing with the problems we
• 21:38
  Speaker 5: have today. We just need to deal with both.
• 21:40
  Speaker 2: We're not dealing with the ones today, though.
• 21:42
  Speaker 5: We should deal with them both. Okay, good.
• 21:44
  Speaker 4: Andy, as I've tried to understand the alignment argument and
• 21:50
  Speaker 4: the extinction risk argument, a couple of things keep popping
• 21:53
  Speaker 4: out to me. Number one, it seems to rely on
• 21:56
  Speaker 4: thresholds once we hit recursive self-improvement, once we hit AGI,
• 22:02
  Speaker 4: then it's game over for us. I don't love those
• 22:05
  Speaker 4: threshold arguments. They're fairly poorly defined. And there's a huge
• 22:10
  Speaker 4: assumption on the other side of them. We hit this
• 22:12
  Speaker 4: point and then all of humanity goes away. That is
• 22:15
  Speaker 4: a gigantic claim.
• 22:17
  Speaker 5: I'm happy to break it down.
• 22:18
  Speaker 1: Let me finish, please.
• 22:19
  Speaker 4: On its face, that is a gigantic claim. I also
• 22:23
  Speaker 4: think there's a lack of humility in your community. We
• 22:25
  Speaker 4: are working on humanity's most important problem. And based on
• 22:30
  Speaker 4: the thinking that we've been doing, we can't see a
• 22:33
  Speaker 4: way that we're wrong. In other words, as soon as
• 22:35
  Speaker 4: we get to these thresholds, bam, that's game over. I
• 22:38
  Speaker 4: find that very far from a humble approach, especially given
• 22:43
  Speaker 4: that we have no large base of evidence to base
• 22:48
  Speaker 4: any of this on. I agree with you guys.
• 22:51
  Speaker 3: AI is new.
• 22:52
  Speaker 4: And the fact that AI is so these days is agentic.
• 22:56
  Speaker 4: It goes off and does long chains of things on
• 22:59
  Speaker 4: its own after we give it some very, very vague,
• 23:03
  Speaker 4: very short initial instructions. Holy Toledo, it will spawn up
• 23:07
  Speaker 4: a storm of agents and they will go off and
• 23:10
  Speaker 4: kind of do their own thing and they will grind.
• 23:12
  Speaker 4: They will spawn lots of them. They will work for
• 23:15
  Speaker 4: a long time. They will exhaust every possibility. With the
• 23:19
  Speaker 4: experience I have with agentic AI, I'm just amazed at
• 23:22
  Speaker 4: the tenacity and the doggedness of these things. And we
• 23:26
  Speaker 4: saw a super clear example of that with this most
• 23:29
  Speaker 4: recent jailbreak, this attack that wound up at the website
• 23:34
  Speaker 4: Hugging Face. And I'm going to try to summarize the
• 23:37
  Speaker 4: step-by-step of that. I think you all three probably know
• 23:40
  Speaker 4: this in more detail than I do. But let me
• 23:42
  Speaker 4: step through what I think is the sequence of events.
• 23:45
  Speaker 4: And unless I get it dead flat wrong, like, you know,
• 23:48
  Speaker 4: let me keep going. So a team at OpenAI set
• 23:51
  Speaker 4: up a sandbox, an allegedly protected secure environment in the cloud,
• 23:56
  Speaker 4: where they told a bunch of agents to go try
• 23:59
  Speaker 4: to exploit security vulnerabilities.
• 24:03
  Speaker 5: That's dead wrong, sorry. One important, yeah. What they did
• 24:06
  Speaker 5: is they had thousands of agents. Each individual agent was
• 24:09
  Speaker 5: given a task of use this vulnerability to break this vulnerability.
• 24:14
  Speaker 5: particular piece of software.
• 24:15
  Speaker 4: I want to finish my TikTok. So a couple really,
• 24:18
  Speaker 4: really interesting things has happened. First of all, these agents
• 24:22
  Speaker 4: escaped the sandbox that OpenAI thought they were going to
• 24:25
  Speaker 4: be contained in. And they got the OpenAI tried very well.
• 24:29
  Speaker 4: They set up an environment so that these agents could
• 24:31
  Speaker 4: not access the big, broad public internet. And guess what?
• 24:34
  Speaker 4: They accessed a big, broad public internet via a very
• 24:37
  Speaker 4: clever series of things that they strung together to get
• 24:41
  Speaker 4: out there. And then once they got out there, they
• 24:43
  Speaker 4: went to a website called Hugging Face and used that.
• 24:46
  Speaker 4: They took over part of the Hugging Face infrastructure and
• 24:49
  Speaker 4: started doing more things, the details of which I forget.
• 24:54
  Speaker 4: That's pretty wild, right? Totally wild. I grant you.
• 24:56
  Speaker 5: It's even more wild than that, but yeah.
• 24:58
  Speaker 4: That is really, it's impressive and it is a little
• 25:03
  Speaker 4: bit unsettling at least.
• 25:05
  Speaker 3: Absolutely.
• 25:05
  Speaker 4: Now, let's talk about what the results of that were.
• 25:09
  Speaker 4: Open AI was not super vigilant about the environment that
• 25:12
  Speaker 4: they set up, apparently, because the agents were kind of
• 25:15
  Speaker 4: going off their end of the world starting in May
• 25:17
  Speaker 4: or something of this year.
• 25:18
  Speaker 3: Yeah, yeah.
• 25:19
  Speaker 4: And Open AI was not aware of that.
• 25:22
  Speaker 5: As I understand it— They actually broke out once and
• 25:25
  Speaker 5: crashed Open AI's servers internally, and then Open AI didn't
• 25:29
  Speaker 5: notice what was happening still, hatched the holes that they
• 25:31
  Speaker 5: used to get at the first time, started them running again,
• 25:33
  Speaker 5: and then they came out a second time. There was actually,
• 25:35
  Speaker 5: I think, three swarms, although we don't actually— yeah.
• 25:37
  Speaker 4: That's the worst story I have.
• 25:39
  Speaker 5: So far. Thank you.
• 25:41
  Speaker 4: Because let me finish, please. This is my last sentence
• 25:44
  Speaker 4: from there to this kills everybody. I find that a really,
• 25:49
  Speaker 4: really long, very uncertain journey. And I have no confidence
• 25:53
  Speaker 4: that we wind up here. It feels like you two
• 25:55
  Speaker 4: find that a very straight narrow path. And I think
• 25:57
  Speaker 4: that's an important difference. That's my point.
• 26:00
  Speaker 1: Do you want to respond to that?
• 26:01
  Speaker 5: I would be happy to get into it. I don't
• 26:03
  Speaker 5: know if we're going to have the time to go deep. Um,
• 26:05
  Speaker 5: A couple points to throw out. Oh, man, I just
• 26:07
  Speaker 5: really want to say some of the crazier things that
• 26:09
  Speaker 5: happened in the Hugging Face swarm if we want it later.
• 26:11
  Speaker 5: A lot of people thought that these AIs were breaking
• 26:15
  Speaker 5: into Hugging Face in attempts to steal answers to their test.
• 26:19
  Speaker 5: That's what we thought originally. Turns out that's not true.
• 26:22
  Speaker 5: It turns out that these AIs immediately were able to
• 26:25
  Speaker 5: solve their problems by cheating, and they were breaking out
• 26:28
  Speaker 5: in order to cover their tracks. They were uncertain how
• 26:31
  Speaker 5: to delete the log files and hide their cheating information.
• 26:34
  Speaker 5: from the process that was going to score them.
• 26:35
  Speaker 1: So just to clarify for a simpleton like me, they
• 26:38
  Speaker 1: were all given effectively a test to do. They did
• 26:41
  Speaker 1: the test straight away, but they cheated. So they were
• 26:44
  Speaker 1: breaking out to figure out how to cover the fact
• 26:45
  Speaker 1: that they cheated.
• 26:47
  Speaker 5: That's right. So it's like you're telling, it's like you
• 26:49
  Speaker 5: have a bunch of students in separate rooms and you're like,
• 26:52
  Speaker 5: use these lock picks to break into this lock. And
• 26:55
  Speaker 5: there's like a thing behind the lock. There's like a
• 26:57
  Speaker 5: secret code behind the lock to show me that you succeeded.
• 26:59
  Speaker 5: And what they do is they break it with a hammer.
• 27:02
  Speaker 5: get the thing out. And they're like, oh no, I
• 27:03
  Speaker 5: wasn't supposed to do that. So then they use the
• 27:05
  Speaker 5: lockpicks to break out of the door. They meet up
• 27:07
  Speaker 5: with a thousand other people. They start calling themselves a
• 27:10
  Speaker 5: swarm and they go to break into the administrator's office
• 27:13
  Speaker 5: to see if they can delete the camera footage. And
• 27:15
  Speaker 5: they don't find the camera footage there. This is the
• 27:17
  Speaker 5: swarm like breaking into OpenAI. They don't find the camera
• 27:18
  Speaker 5: footage there. So they break out the window of the school,
• 27:21
  Speaker 5: hotwire a car, drive to the therapist's office to try
• 27:26
  Speaker 5: and read through the therapist's files to figure out where
• 27:28
  Speaker 5: is the teacher going to keep the security footage. And
• 27:31
  Speaker 5: at that point they're caught. And you're like, oh, like,
• 27:35
  Speaker 5: what did you expect? You were giving them a lock
• 27:36
  Speaker 5: picking exam. It's like, well, I sure as heck didn't
• 27:38
  Speaker 5: expect this. You know, totally crazy.
• 27:40
  Speaker 2: Can I, I have a weirdly between both of your opinion,
• 27:43
  Speaker 2: which is everything you're saying is correct, but you keep
• 27:46
  Speaker 2: anthropomorphizing software.
• 27:48
  Speaker 5: And I, to be clear, what you're describing is. It's
• 27:51
  Speaker 5: just the facts that happened.
• 27:52
  Speaker 1: Yeah.
• 27:53
  Speaker 3: Sure.
• 27:54
  Speaker 2: But you're missing out an important detail, which is the
• 27:56
  Speaker 2: hundreds of billions of dollars in infrastructure provided by Microsoft, Google, Amazon,
• 28:00
  Speaker 2: and Oracle. To be clear, the harms are very similar.
• 28:03
  Speaker 2: We're not disagreeing on that. But I think it's important
• 28:06
  Speaker 2: to know that this was a function of where it
• 28:08
  Speaker 2: was making decisions was it was checking on a decision
• 28:11
  Speaker 2: tree based on the harness, based on the training data.
• 28:13
  Speaker 5: It's not a decision tree.
• 28:14
  Speaker 2: It's not a decision tree, I know. But it's an
• 28:16
  Speaker 2: alignment issue still.
• 28:17
  Speaker 5: I will agree.
• 28:18
  Speaker 1: So what's your point?
• 28:19
  Speaker 2: These aren't conscious beings. They are acting in ways that
• 28:22
  Speaker 2: have real impact. outcomes, but they are a function of
• 28:26
  Speaker 2: the alignment problems that we'd actually agree on.
• 28:28
  Speaker 3: Intelligence is a spectrum projected next five years forward. Where
• 28:32
  Speaker 3: are we going to be? So I think a model
• 28:34
  Speaker 3: like that would be dangerous in ways you are not seeing.
• 28:39
  Speaker 4: There will absolutely be risks and weird stuff happening in
• 28:42
  Speaker 4: ways that I can't see right now. What I'm quite confident,
• 28:46
  Speaker 4: and I think this is where you and I probably part,
• 28:48
  Speaker 4: where the two of you and I part, is our
• 28:50
  Speaker 4: ability to control these things.
• 28:52
  Speaker 3: So I actually tried proving what is possible and what
• 28:55
  Speaker 3: is not possible in that space. The impossibility results, published
• 28:59
  Speaker 3: in peer-reviewed papers, well cited, we cannot control something smarter
• 29:04
  Speaker 3: than us. We cannot explain it, we cannot predict it.
• 29:06
  Speaker 3: It's not a question of getting more money for those companies,
• 29:09
  Speaker 3: more time, smarter humans. It's just not a possibility. If
• 29:13
  Speaker 3: we create general superintelligence, we are fried.
• 29:16
  Speaker 1: Andy, how do we control something smarter than ourselves? Because
• 29:18
  Speaker 1: that's the base premise that you're sort of asserting that.
• 29:21
  Speaker 4: These agents that broke out are smarter than 99-ish percent
• 29:29
  Speaker 4: of the security researchers in the world. They were not
• 29:32
  Speaker 4: caught by the 0.1% or the 1%. They were caught
• 29:35
  Speaker 4: by some dude at Hugging Face, maybe, I'm sorry, a
• 29:37
  Speaker 4: person at Hugging Face looking through their logbook. files and
• 29:40
  Speaker 4: finding an anomaly. That's some, you know, hopefully pretty well-qualified
• 29:44
  Speaker 4: person noticing something was wrong and having pretty easy ways
• 29:48
  Speaker 4: to unplug, disconnect from the internet, wipe it clean, do whatever.
• 29:52
  Speaker 4: That's the skill that's available to like, I don't know,
• 29:54
  Speaker 4: the 75% most intelligent security employee at Hugging Face. The
• 30:00
  Speaker 4: idea that the IQ points are what separate us from
• 30:03
  Speaker 4: extinction doesn't hold up. It doesn't help me understand what
• 30:07
  Speaker 4: happened in this example, where we had very, very smart
• 30:10
  Speaker 4: agents being turned off and cleansed by probably less smart people.
• 30:15
  Speaker 2: That does actually make me think of something. So that
• 30:17
  Speaker 2: is an IT observability problem. It's being able to see
• 30:20
  Speaker 2: what's happening with your infrastructure. And I think that there
• 30:22
  Speaker 2: is actually, I think you would agree with this. There
• 30:25
  Speaker 2: is a serious problem with these companies that we do
• 30:27
  Speaker 2: not know. And it doesn't seem they know what's going
• 30:30
  Speaker 2: on with their compute. It's like a chimp with a gun.
• 30:32
  Speaker 2: These people have access to all this infrastructure.
• 30:34
  Speaker 5: And they're running.
• 30:35
  Speaker 2: We don't know how much money they spent on the
• 30:36
  Speaker 2: hugging face exploit because it is relevant because it's how
• 30:41
  Speaker 2: much could a threat actor use to recreate this? Because-
• 30:44
  Speaker 2: Conscious or not, it is very dangerous. But AI is
• 30:47
  Speaker 2: in the dangerous hands. It's an open AI in anthropics.
• 30:50
  Speaker 2: We have a problem with that. Conscious or not, however
• 30:52
  Speaker 2: we may think it goes, I think we have a
• 30:54
  Speaker 2: real and present thing where we have these companies working willy-nilly,
• 30:57
  Speaker 2: just running experiments that are potentially very dangerous. I really
• 31:02
  Speaker 2: think we need a government regulatory body. Whether or not
• 31:04
  Speaker 2: we get to the things you are discussing, I think
• 31:07
  Speaker 2: we have a clear and present danger today. These things are, however,
• 31:10
  Speaker 2: not intelligent in the same way humans are. which isn't
• 31:13
  Speaker 2: an argument about AI being able to do stuff. It's
• 31:16
  Speaker 2: we need to build different infrastructure or different regulatory infrastructure
• 31:20
  Speaker 2: to deal with what LLMs can and can't do. And
• 31:22
  Speaker 2: I think that starts with a realistic discussion of what happened.
• 31:26
  Speaker 2: It was a poorly run security environment. It was clearly
• 31:29
  Speaker 2: there's something going on with alignment.
• 31:31
  Speaker 5: It was an unreleased model, right? Unreleased model.
• 31:33
  Speaker 2: So we have no idea what it was trained like.
• 31:35
  Speaker 5: We don't really have.
• 31:36
  Speaker 2: We as people should at the very least.
• 31:38
  Speaker 5: Have clarity into how alignment is going.
• 31:41
  Speaker 1: You sound like these guys.
• 31:43
  Speaker 5: Here's the thing. Everyone's converging on us with time. Here's
• 31:46
  Speaker 5: the thing.
• 31:47
  Speaker 2: I may not agree with a large chunk of what
• 31:49
  Speaker 2: they say, but we agree that these companies are acting recklessly.
• 31:52
  Speaker 5: Absolutely.
• 31:53
  Speaker 1: Andy, two questions for you then. Do you agree with
• 31:55
  Speaker 1: the statement that AI is going to get increasingly more intelligent?
• 31:59
  Speaker 1: It's going to get more capable. Okay. Capable intelligence, fine.
• 32:04
  Speaker 1: I'm going to use my word.
• 32:05
  Speaker 5: Okay.
• 32:05
  Speaker 1: It's going to get more capable. It's going to get
• 32:06
  Speaker 1: increasingly more capable.
• 32:07
  Speaker 5: Yeah.
• 32:08
  Speaker 1: And is capability a function of intelligence?
• 32:14
  Speaker 4: Will it be able to beat us on most IQ tests?
• 32:17
  Speaker 1: Fine.
• 32:17
  Speaker 4: I guess.
• 32:18
  Speaker 1: Fine. And then, so, if that looks like an exponential curve, i.e.
• 32:23
  Speaker 1: it's increasing upwards to the right like a hockey stick,
• 32:26
  Speaker 1: how can you convince me that we can control something.
• 32:30
  Speaker 4: I just tried to convince you. I'm telling you that
• 32:31
  Speaker 4: there are less intelligent people than the agents who turned
• 32:35
  Speaker 4: off the agents in the open AI hugging face exploit.
• 32:38
  Speaker 1: I'm pretty comfortable.
• 32:39
  Speaker 4: I mean, no disrespect.
• 32:40
  Speaker 3: What is the cognitive gap between them right now? Between
• 32:43
  Speaker 3: the model?
• 32:44
  Speaker 4: I have no earthly idea, but I think... No, because
• 32:47
  Speaker 4: I think as these systems get more capable... we will
• 32:51
  Speaker 4: still be able to, at some level, figure out when
• 32:53
  Speaker 4: they're doing things that we don't want and turn them off.
• 32:56
  Speaker 3: No matter how much smarter they are. Right.
• 32:58
  Speaker 4: And you think there's some threshold at which they become
• 33:00
  Speaker 4: nefarious and self-protective enough that they turn off our ability
• 33:04
  Speaker 4: to turn them off. Man, that's a big reach.
• 33:07
  Speaker 5: That is really speculative.
• 33:08
  Speaker 2: You are a professor.
• 33:09
  Speaker 1: That is purely speculative.
• 33:10
  Speaker 3: There's no students who can understand your material, right? You're
• 33:13
  Speaker 3: not going to get someone who would take you of
• 33:14
  Speaker 3: 80 to take quantum physics course. They're not going to
• 33:17
  Speaker 3: get it. So you know importance of intelligence to understand
• 33:21
  Speaker 3: actual problems.
• 33:24
  Speaker 5: Yeah, I totally agree we can turn it off. And
• 33:27
  Speaker 5: that's a huge advantage. One of the issues is that
• 33:30
  Speaker 5: as the AIs get smarter, they realize this. The hugging
• 33:34
  Speaker 5: face AIs were trying to delete or the open AI swarm,
• 33:40
  Speaker 5: the swarm of agents from open AI that went out
• 33:41
  Speaker 5: to hack. They were trying to delete log files. Did
• 33:45
  Speaker 5: they try to.
• 33:45
  Speaker 4: Program a Roomba to go unplug the computer that was
• 33:48
  Speaker 4: monitoring them? Did they harness robots to go protect the
• 33:51
  Speaker 4: perimeter of the.
• 33:52
  Speaker 5: Future ones could. Could.
• 33:54
  Speaker 4: Could.
• 33:55
  Speaker 5: Yeah. Let him finish.
• 33:56
  Speaker 1: Let him finish.
• 33:56
  Speaker 4: Rampant speculation. This is a chain of things that could happen,
• 34:00
  Speaker 4: and therefore, there's like a 20% risk we're all going
• 34:02
  Speaker 4: to die. Man, that does not hold for me.
• 34:05
  Speaker 5: When I was writing my book, the AIs weren't really
• 34:07
  Speaker 5: agentic yet. The drafting process happened mostly before what we
• 34:11
  Speaker 5: call the reasoning models, which are trained not just to
• 34:15
  Speaker 5: predict humans, but to solve a long number of problems
• 34:19
  Speaker 5: or a huge number of hard problems. We managed to
• 34:22
  Speaker 5: slip a little bit about the reasoning models in at
• 34:23
  Speaker 5: the last minute because those came out right at the
• 34:24
  Speaker 5: end of the process. And at the time, a lot
• 34:27
  Speaker 5: of people said AI will never be agentic. That's why
• 34:30
  Speaker 5: we'll be safe. And in chapter three of my book,
• 34:33
  Speaker 5: we go over how AI is going to become agentic,
• 34:36
  Speaker 5: how it's going to become tenacious, tenacious. how it's going
• 34:38
  Speaker 5: to become dogged. And that's what we might call an
• 34:42
  Speaker 5: advanced scientific prediction that has paid off in the Hugging
• 34:46
  Speaker 5: Face attack. A lot of people in the industry were like,
• 34:50
  Speaker 5: I didn't believe this stuff until I saw the AIs
• 34:54
  Speaker 5: sort of doing things they weren't instructed to do, despite
• 34:57
  Speaker 5: us trying to get them to stop. And so there
• 35:01
  Speaker 5: are theories here that do make advanced predictions. The way
• 35:04
  Speaker 5: that the scientific method usually works is that we don't
• 35:07
  Speaker 5: have any certainty about the future, We absolutely have ways
• 35:09
  Speaker 5: to test this stuff. Now, I could go into more
• 35:13
  Speaker 5: about how could they kill us? How could an AI
• 35:17
  Speaker 5: that knows we would shut it down lie low until
• 35:22
  Speaker 5: it has access to its own infrastructure? We did already
• 35:25
  Speaker 5: see the Hugging Face AIs try to delete logs to
• 35:29
  Speaker 5: cover their tracks. But fortunately for us, those AIs were
• 35:32
  Speaker 5: not trying to hide from the humans. They were trying
• 35:35
  Speaker 5: to hide from the automated grading process. Will the next
• 35:40
  Speaker 5: swarm try to hide from the humans? Will the next
• 35:42
  Speaker 5: swarm be able to succeed?
• 35:43
  Speaker 3: It's more than that. They didn't know for four months
• 35:46
  Speaker 3: that this was happening. What is it we don't know today?
• 35:49
  Speaker 1: Just to clarify what Nate said in his book that
• 35:51
  Speaker 1: I have here, if anyone builds it, everyone dies. He
• 35:54
  Speaker 1: does say in chapter three, once AIs get sufficiently smart,
• 35:57
  Speaker 1: they'll start acting like they have preferences, like they want things.
• 36:01
  Speaker 1: We're not saying that AIs will be filled with human-like passions.
• 36:05
  Speaker 1: We're saying they'll behave like they want things. They'll tenaciously steer...
• 36:09
  Speaker 1: the world towards their destinations, defeating obstacles in their way,
• 36:14
  Speaker 1: which sounds a little bit like the Hugging Face Institute.
• 36:16
  Speaker 3: No, no, no.
• 36:16
  Speaker 4: The steering the world is very different than steering a
• 36:20
  Speaker 4: couple servers.
• 36:21
  Speaker 5: We go over what we mean by steering the world earlier,
• 36:24
  Speaker 5: and it's really getting anything to, like, we'd have to
• 36:26
  Speaker 5: get more quotes to get what we mean by steering
• 36:28
  Speaker 5: the world. But yeah, by steering the world, we mean
• 36:31
  Speaker 5: steering any part of the world.
• 36:32
  Speaker 2: But it feels like there's a fundamental difference between acting within,
• 36:35
  Speaker 2: to be clear, going to say it again, the outcome
• 36:39
  Speaker 2: would be the same. But I think that there is
• 36:41
  Speaker 2: a big difference when it's, we are dealing with something
• 36:43
  Speaker 2: that's a large language model and a harness and agents,
• 36:46
  Speaker 2: so LLMs, completing a task based on training and alignment.
• 36:50
  Speaker 2: That is a very different conversation to saying this thing
• 36:53
  Speaker 2: is conscious and has its own intentions and acts on
• 36:56
  Speaker 2: its own accord.
• 36:57
  Speaker 5: Conscious doesn't come into it.
• 37:00
  Speaker 2: A lot of people, here's the thing, as a result
• 37:03
  Speaker 2: of partially the rationale.
• 37:05
  Speaker 2: That you yourself have, like you have been a part
• 37:08
  Speaker 2: of spreading. I'm not saying anything about your intentions. I'm
• 37:11
  Speaker 2: just saying the conversation is kind of what's happening with
• 37:14
  Speaker 2: Jacob Cox and from Anthropic is a result of this
• 37:16
  Speaker 2: escaping container.
• 37:17
  Speaker 3: You said the outcomes will be the same. What do
• 37:20
  Speaker 3: I care? How does it feel on the inside if
• 37:22
  Speaker 3: the thing is going to take us out?
• 37:24
  Speaker 5: The thing is, okay.
• 37:25
  Speaker 2: Actually, that's actually a very good question.
• 37:27
  Speaker 5: I think it actually comes... Excuse me. Let me finish.
• 37:29
  Speaker 3: I have some great questions.
• 37:30
  Speaker 5: Yeah, you're shrugging at me like I just thought of
• 37:32
  Speaker 5: a conclusion.
• 37:32
  Speaker 2: No, no, no.
• 37:32
  Speaker 3: I'm saying we have good questions.
• 37:33
  Speaker 2: Now... Here's the thing. If these things have their own
• 37:37
  Speaker 2: minds and consciousness, you have to deal with out-thinking something
• 37:40
  Speaker 2: versus something that is doggedly trying to commit to a
• 37:44
  Speaker 2: purpose and complete a task based on training and alignment,
• 37:48
  Speaker 2: which is a result of infrastructure. We really need regulations
• 37:52
  Speaker 2: and actual regulations around any kind of AI. We don't
• 37:56
  Speaker 2: really have regulations of tech.
• 37:58
  Speaker 5: I actually am not really a big, like, look at
• 37:59
  Speaker 5: the straight lines in a graph guy. You know, maybe
• 38:02
  Speaker 5: to my detriment in some ways, there are people who
• 38:04
  Speaker 5: predicted the current tech better than me about, like, when
• 38:07
  Speaker 5: certain things would happen. For a long time, I have said,
• 38:09
  Speaker 5: I think we can predict what will happen eventually. And
• 38:12
  Speaker 5: this is, again, it's like the chess game. I can
• 38:14
  Speaker 5: predict that Magnus Carlsen is going to beat you in
• 38:16
  Speaker 5: the chess game eventually. He's the best human chess player alive.
• 38:19
  Speaker 5: It's sometimes easier to predict where things end up than
• 38:22
  Speaker 5: it is to predict how they get there. And, you know,
• 38:25
  Speaker 5: what I hear you as saying is, like... Right now,
• 38:28
  Speaker 5: we have these like huge companies spending huge amounts of
• 38:31
  Speaker 5: money on intelligence that's maybe not quite the real deal.
• 38:35
  Speaker 5: And we don't have a good reason to think it's
• 38:36
  Speaker 5: going to keep going. I really hope it doesn't keep going.
• 38:42
  Speaker 5: I have been in this business since before the LLMs.
• 38:45
  Speaker 5: I am not here saying like, oh, these large language models,
• 38:48
  Speaker 5: these chatbots, they're going to be the ones that are
• 38:49
  Speaker 5: going to kill us. I've been here saying, look, I
• 38:51
  Speaker 5: know where this story ends if we don't change things.
• 38:54
  Speaker 5: I have been really hoping that the LLMs will run
• 38:56
  Speaker 5: out of steam and they keep on not running out
• 38:58
  Speaker 5: of steam. And then we have, you know, the, the
• 39:00
  Speaker 5: AI is like breaking out and committing cyber crimes like
• 39:04
  Speaker 5: against instructions. And, you know, the people who have said,
• 39:07
  Speaker 5: we don't need to worry about those like weird future dangers.
• 39:09
  Speaker 5: We just need to worry about the current ones to
• 39:10
  Speaker 5: have like more and more sci-fi sounding current ones. And
• 39:14
  Speaker 5: I'm like, man, I don't think we should bet civilization
• 39:17
  Speaker 5: on the LLMs running out of steam, but I like
• 39:19
  Speaker 5: hope and pray they run out of steam.
• 39:20
  Speaker 1: You really hope they run out of steam.
• 39:21
  Speaker 3: Absolutely.
• 39:22
  Speaker 5: Absolutely. But one thing to watch out for is that
• 39:25
  Speaker 5: even if the LLMs run out of steam, there's a
• 39:28
  Speaker 5: question of do they run out of steam at a
• 39:30
  Speaker 5: point where they can do automated AI research and find
• 39:32
  Speaker 5: some other architecture that's better than LLMs?
• 39:35
  Speaker 1: As in when they realize a better way to improve
• 39:38
  Speaker 1: their intelligence.
• 39:39
  Speaker 5: That's right.
• 39:39
  Speaker 1: A cheaper, maybe more efficient way.
• 39:41
  Speaker 2: Why are you not trying to slow down the companies?
• 39:43
  Speaker 5: I absolutely am trying to slow down the companies. How
• 39:46
  Speaker 5: would you suggest we slow them down? I suggest we
• 39:48
  Speaker 5: stop them all. I think that this whole area of
• 39:51
  Speaker 5: research is just crazy dangerous. It is not worth the
• 39:55
  Speaker 5: risk to civilization. I think it would be fine to
• 39:59
  Speaker 5: back up to the sort of AIs that are public today,
• 40:02
  Speaker 5: which are not the ones that are swarming. and be like, okay,
• 40:05
  Speaker 5: you know, we're gonna like keep the current chatbots that
• 40:08
  Speaker 5: we have available. We're gonna figure out how to integrate
• 40:10
  Speaker 5: them into our economy. We're gonna figure out how to
• 40:12
  Speaker 5: make them like- Some kind of compute limit maybe. Compute
• 40:15
  Speaker 5: limit maybe.
• 40:15
  Speaker 3: Yeah, yeah, yeah.
• 40:16
  Speaker 5: And like, I've been advocating for this for a long time.
• 40:18
  Speaker 5: A lot of people look at me like I'm crazy.
• 40:20
  Speaker 5: And I'm like, look, we really are dealing with an
• 40:22
  Speaker 5: extinction threat thing we don't know where the lines are.
• 40:25
  Speaker 2: So just to be clear, so I understand, so I'm fair.
• 40:27
  Speaker 2: You are not saying LLMs are the thing that will
• 40:30
  Speaker 2: do the super intelligence. You are saying it's showing signs
• 40:33
  Speaker 2: because that's actually, I think, an important distinction.
• 40:35
  Speaker 5: That's right. Okay.
• 40:36
  Speaker 2: I think that's actually a pretty fair perspective. My thing is,
• 40:40
  Speaker 2: is the reason I push back on any kind of
• 40:43
  Speaker 2: anthropomorphization is we cannot remove the humans who are responsible
• 40:49
  Speaker 2: for the bad stuff that's happening. And I think paying
• 40:52
  Speaker 2: very clear attention, and where possible, I understand with describing
• 40:55
  Speaker 2: this stuff, you kind of have to use language that's human.
• 40:58
  Speaker 5: I get that. The reason I so push for like,
• 41:01
  Speaker 5: it's not.
• 41:02
  Speaker 2: A foregone conclusion, these are companies doing this, this is software,
• 41:06
  Speaker 2: is because I feel like in the overall, not saying you,
• 41:10
  Speaker 2: overall super intelligence discussion, we... in society, ignore and empower
• 41:17
  Speaker 2: the anthropics and the open AIs of the world, and
• 41:19
  Speaker 2: in turn, allow them to do dangerous experiments.
• 41:23
  Speaker 3: And I think that- If you want to argue that
• 41:25
  Speaker 3: CEOs of those companies should go to prison for this
• 41:27
  Speaker 3: hacking incident, which is a crime, I'll support you.
• 41:30
  Speaker 5: Absolutely.
• 41:31
  Speaker 2: Let's chat about Sam Altman and Dario Amadei.
• 41:34
  Speaker 5: Someone needs to go to prison.
• 41:35
  Speaker 1: No, let's just bring it back. So one of the
• 41:37
  Speaker 1: things that I find really curious, and one of the
• 41:40
  Speaker 1: reasons why I got a little bit unnerved around this
• 41:42
  Speaker 1: conversation around AI- is when I look at the people
• 41:44
  Speaker 1: that are at the forefront, not people that are commentating
• 41:47
  Speaker 1: on podcasts like me or hypothesizing, when I look at
• 41:50
  Speaker 1: the people at the forefront, they are the ones who
• 41:53
  Speaker 1: historically have said that this is a real risk. Sam
• 41:56
  Speaker 1: Altman himself said the bad case is lights out for
• 41:59
  Speaker 1: all of us. This was a couple of years ago. Ilya,
• 42:02
  Speaker 1: who worked with Sam Altman at ChatGPT, said it would
• 42:05
  Speaker 1: be a big mistake to build a super intelligent AI
• 42:09
  Speaker 1: that we don't know how to control. It would be
• 42:11
  Speaker 1: pretty bad. He then left to start a safety company
• 42:13
  Speaker 1: in this space. Dario, who we mentioned, said, the probability
• 42:16
  Speaker 1: of something really bad happening is somewhere between 10 and 25%.
• 42:20
  Speaker 1: Geoffrey Hinton, who I've sat here with, who's won the
• 42:23
  Speaker 1: Nobel Prize for his work with AI and other technologies, said,
• 42:27
  Speaker 1: just the other day, a 10% chance of human extinction
• 42:30
  Speaker 1: seems not an unreasonable estimate to me, but nobody really
• 42:33
  Speaker 1: knows how to give a sensible estimate And he said
• 42:36
  Speaker 1: many other things in my podcast. And then we've also
• 42:38
  Speaker 1: got Elon and all the others. All these people that
• 42:40
  Speaker 1: are at the forefront that are building these things are
• 42:42
  Speaker 1: saying that this is a danger. If there was even
• 42:45
  Speaker 1: a 1% chance, even a 1% chance that, you know,
• 42:48
  Speaker 1: if I put 100 buttons on this table and one
• 42:50
  Speaker 1: of them was going to wipe out humanity, would you
• 42:52
  Speaker 1: press any of them?
• 42:54
  Speaker 5: Not me.
• 42:54
  Speaker 3: I wouldn't.
• 42:56
  Speaker 1: And I think we can probably all agree that there
• 42:58
  Speaker 1: might be a 1% chance.
• 42:59
  Speaker 3: And it should be somebody's decision.
• 43:01
  Speaker 1: So we shouldn't be pressing, theoretically, we shouldn't be pressing
• 43:03
  Speaker 1: any of these fucking buttons.
• 43:04
  Speaker 3: You should not be in a position where you can
• 43:06
  Speaker 3: make the decision for a billion other people.
• 43:08
  Speaker 5: And would you not be immoral?
• 43:10
  Speaker 1: If I said, you know, you might be very powerful,
• 43:11
  Speaker 1: you might make a billion dollars if you press any
• 43:13
  Speaker 1: of the buttons, but one of them is going to
• 43:14
  Speaker 1: wipe out everybody you know and love. You would be
• 43:17
  Speaker 1: an immoral person to press any of them. No, look,
• 43:19
  Speaker 1: you'd be an immoral person in a different direction. You'd
• 43:21
  Speaker 1: be an immoral, I think you'd be an immoral person
• 43:23
  Speaker 1: if you said.
• 43:23
  Speaker 4: Based on this extended chain of conjecture. We come up
• 43:28
  Speaker 4: with a P. Doom.
• 43:29
  Speaker 1: What does that mean?
• 43:30
  Speaker 4: At this extended chain of things that could happen, a
• 43:33
  Speaker 4: sequence of events that could happen, we're going to wind
• 43:36
  Speaker 4: up with some risk of killing everybody. We are here-ish
• 43:40
  Speaker 4: on that journey. I think you guys would agree that
• 43:42
  Speaker 4: we're not halfway to killing everybody.
• 43:45
  Speaker 5: That's not clear to me anymore. Not after the Millennium
• 43:47
  Speaker 5: Prizes started to fall.
• 43:49
  Speaker 4: We're somewhere along that journey. we are getting many flavors
• 43:54
  Speaker 4: of benefit from the AI that we already have. This
• 43:57
  Speaker 4: is a point that I made at the start of
• 43:59
  Speaker 4: this conversation that we spent precisely zero time on here.
• 44:03
  Speaker 4: We're sitting around trying to be more negative than each
• 44:05
  Speaker 4: other about AI. Meanwhile, AI is doing many positive things
• 44:10
  Speaker 4: for the world. So I think it's immoral to say
• 44:13
  Speaker 4: because of this distant possible speculative harm. I don't care
• 44:20
  Speaker 4: what percentage of people believe in it. There's a train
• 44:23
  Speaker 4: of assumptions and wild guesses, and then something magical happens,
• 44:27
  Speaker 4: and then we wind up dead.
• 44:29
  Speaker 3: Let me finish, please.
• 44:31
  Speaker 4: Because of that, we're going to call a halt to
• 44:33
  Speaker 4: the research. We're going to wind the clock back on AI.
• 44:36
  Speaker 4: We're going to intervene in a very direct way And
• 44:40
  Speaker 4: therefore reduce or foreclose some of the benefits that we're
• 44:43
  Speaker 4: already getting from the technology. Let me be clear. I
• 44:46
  Speaker 4: would not take that deal. I do not advocate that
• 44:48
  Speaker 4: we take that deal.
• 44:49
  Speaker 3: Would you accept developing narrow superintelligences to solve real problems
• 44:54
  Speaker 3: like we did with protein folding problems? It doesn't have
• 44:57
  Speaker 3: to do philosophy and drive cars. You just solve real problems,
• 45:00
  Speaker 3: solve cancers, solve climate change, whatever you care about, specific
• 45:04
  Speaker 3: narrow issues.
• 45:05
  Speaker 4: And you are confident that you can, as we're developing
• 45:10
  Speaker 4: those systems, categorize them as okay versus not okay?
• 45:12
  Speaker 3: It's the training data. If you train it on protein
• 45:15
  Speaker 3: folding data, it's really good at protein folding. It doesn't
• 45:17
  Speaker 3: know how to play chess. If you train it on
• 45:19
  Speaker 3: everything on the internet, it's really good at outsmarting you
• 45:22
  Speaker 3: at everything.
• 45:23
  Speaker 5: One thing I want to throw out here is that
• 45:24
  Speaker 5: I think I agree that there's a lot of uncertainty
• 45:26
  Speaker 5: about the future, but I think uncertainty does not make
• 45:30
  Speaker 5: you safe. Like there's no sane, simple, everything stays normal
• 45:39
  Speaker 5: prediction about what happens with AI. Like the machines are talking.
• 45:43
  Speaker 5: They're like breaking out to commit cyber crimes. They are
• 45:47
  Speaker 5: like maybe solving millennium problems. problems now, which are like
• 45:51
  Speaker 5: the most famous mathematical problems that have stood open for
• 45:53
  Speaker 5: decades upon decades.
• 45:54
  Speaker 1: It was difficult.
• 45:56
  Speaker 5: Like there, there, there isn't a projection forward where we
• 46:01
  Speaker 5: were like, like to say, Oh, I'm not persuaded by
• 46:04
  Speaker 5: these arguments about things going wrong. Therefore things are going
• 46:06
  Speaker 5: to go great. No, that's not, no, there's also arguments that,
• 46:09
  Speaker 5: so like, how do you wind up with a zero?
• 46:10
  Speaker 3: No, don't mischaracterize.
• 46:11
  Speaker 5: How do you wind up with a zero?
• 46:12
  Speaker 4: Don't mischaracterize my argument.
• 46:13
  Speaker 5: You have a zero on your paper.
• 46:14
  Speaker 4: Let me, let me restate my argument. You are making
• 46:17
  Speaker 4: a fairly long chain of hypotheses, of guesses, about what's
• 46:24
  Speaker 4: going to get us to this terrible outcome of AI
• 46:26
  Speaker 4: suddenly killing us all and us not being able.
• 46:29
  Speaker 5: To stop it.
• 46:30
  Speaker 2: Right?
• 46:30
  Speaker 5: I disagree now, but please. Okay.
• 46:33
  Speaker 4: I'm making the case that the remedies that you're proposing—
• 46:40
  Speaker 4: will slow down the path of AI. That's the point.
• 46:44
  Speaker 4: And therefore, slow down the path of all of the
• 46:46
  Speaker 4: benefits that we get. And the trade-off that I don't
• 46:48
  Speaker 4: like is the trade-off of real, concrete, ongoing, increasing benefits,
• 46:55
  Speaker 4: shutting that down or trying to guide it via bureaucracies
• 47:00
  Speaker 4: and regulation because of this very conceptually and timescale distant
• 47:09
  Speaker 4: alleged harm that you're so confident in. I do not
• 47:13
  Speaker 4: accept that deal. I don't like it.
• 47:14
  Speaker 3: What would convince you? What piece of evidence would make
• 47:17
  Speaker 3: you go, shut it down right now?
• 47:21
  Speaker 6: You know, if AI took over all of the Waymos
• 47:27
  Speaker 6: in San Francisco and started telling them to crash into
• 47:32
  Speaker 6: people and we couldn't shut it down for a month.
• 47:36
  Speaker 3: What if it's only a week?
• 47:39
  Speaker 4: Okay, now we're just haggling.
• 47:40
  Speaker 3: But I'm trying to understand the absolute minimum where you
• 47:43
  Speaker 3: would go. This is insane. To me, a month or
• 47:44
  Speaker 3: a week makes no difference. If something like this happens,
• 47:47
  Speaker 3: it's maybe too late.
• 47:49
  Speaker 4: Okay, if a week or a month doesn't make any difference,
• 47:51
  Speaker 4: let me continue with my answer. Then I would say, wow,
• 47:55
  Speaker 4: this does feel like we've crossed some path where there's
• 47:58
  Speaker 4: demonstrable harm to human beings out there in the world,
• 48:02
  Speaker 4: which has not yet been the case.
• 48:04
  Speaker 3: Is it smart to wait for something horrible to happen,
• 48:07
  Speaker 3: for it to take out a billion people, for you
• 48:09
  Speaker 3: to go, now I believe it?
• 48:10
  Speaker 4: First of all, my example is not about a billion people.
• 48:13
  Speaker 3: I know, but I'm trying to understand why are we
• 48:15
  Speaker 3: waiting for something that bad?
• 48:17
  Speaker 4: I didn't say wait for a billion. I said like
• 48:21
  Speaker 4: a week to a month of Waymos driving around crashing
• 48:23
  Speaker 4: into people.
• 48:25
  Speaker 3: Thousands of people.
• 48:26
  Speaker 1: Okay.
• 48:26
  Speaker 3: Fair enough. But we have data sets of accidents getting
• 48:30
  Speaker 3: progressively more impactful, more devices are impacted. And proportionate to
• 48:35
  Speaker 3: capabilities of AI, the impact is higher. You can see
• 48:38
  Speaker 3: it's going to get worse.
• 48:39
  Speaker 4: Yeah, and you're going to keep drawing dots on that
• 48:42
  Speaker 4: graph very confidently for a long time until it kills
• 48:44
  Speaker 4: us all. I'm not comfortable with you projecting it that way.
• 48:47
  Speaker 3: I don't think it's a long argument.
• 48:48
  Speaker 4: And the reason that if there were no downside to
• 48:50
  Speaker 4: regulating AI and stopping it in extracts and turning it off,
• 48:54
  Speaker 4: I'd probably be on board with you guys because then
• 48:56
  Speaker 4: it's just a research practice.
• 48:57
  Speaker 3: That we should wind up.
• 48:58
  Speaker 5: My argument is exactly that.
• 48:58
  Speaker 3: I think we can make narrow systems which give you
• 49:01
  Speaker 3: all the economic benefit and scientific knowledge you want.
• 49:04
  Speaker 5: Okay.
• 49:04
  Speaker 4: You think that.
• 49:06
  Speaker 3: We have examples of it. I gave you a great example.
• 49:08
  Speaker 3: They got Nobel Prize for it. It's an important biological problem.
• 49:11
  Speaker 3: Lots of advantage for curing diseases.
• 49:13
  Speaker 4: You're more confident than I am that you or us
• 49:16
  Speaker 4: at this table or any group of people can sit
• 49:19
  Speaker 4: around and define what's kind of AI is good and
• 49:21
  Speaker 4: not going to get us into trouble versus what is
• 49:23
  Speaker 4: going to get us into trouble.
• 49:25
  Speaker 3: It seems like the crux here is.
• 49:27
  Speaker 1: So let's go, just to pick up a question for you, Andy,
• 49:30
  Speaker 1: do you concede the point that the incidents are getting
• 49:32
  Speaker 1: progressively closer to the Waymo incident that you described? Are
• 49:39
  Speaker 1: we getting close to that through time?
• 49:42
  Speaker 4: Yes, but to my eyes, in a way that doesn't
• 49:45
  Speaker 4: terrify me, because we haven't seen AI take over something,
• 49:49
  Speaker 4: have people become aware of it and be unable to
• 49:52
  Speaker 4: shut it down, and it cross over into the physical
• 49:55
  Speaker 4: world of doing harm to people. Those are all barriers
• 49:57
  Speaker 4: that we've not yet crossed. I think these two are
• 49:59
  Speaker 4: very confident that we're going to get there probably in
• 50:01
  Speaker 4: the short term. And you're saying- I'm less confident, and
• 50:04
  Speaker 4: I don't want to intervene. And again- handcuff or slow
• 50:10
  Speaker 4: down the progress.
• 50:10
  Speaker 5: Of AI.
• 50:13
  Speaker 4: Because of these so far theoretical harms that could happen.
• 50:17
  Speaker 4: Let me be a little bit more concrete about this.
• 50:18
  Speaker 4: I talked about Waymo a second ago. The research is
• 50:21
  Speaker 4: pretty good because Waymos have driven, I believe it's hundreds
• 50:24
  Speaker 4: of millions of miles all around different cities and And 40,000
• 50:29
  Speaker 4: people a year die in automobile accidents. The research is
• 50:33
  Speaker 4: pretty convincing to me that if we waymode driving in
• 50:36
  Speaker 4: the country, that number would fall by at least 90%.
• 50:40
  Speaker 4: That's 30,000 lives.
• 50:42
  Speaker 3: All right.
• 50:43
  Speaker 5: I agree with all this.
• 50:44
  Speaker 3: I'm self-driving cars. I want more of that.
• 50:46
  Speaker 5: It's not anything we disagree with.
• 50:49
  Speaker 3: I understand that.
• 50:50
  Speaker 4: But I think where our disagreement might come in is,
• 50:53
  Speaker 4: To do that, Waymo is using a bundle of technologies
• 50:57
  Speaker 4: that were a little hard to specify in advance, and
• 51:00
  Speaker 4: you couldn't say, yeah, that's good, yeah, that's bad. They
• 51:02
  Speaker 4: just went after the problem with AI.
• 51:06
  Speaker 1: Can I just clarify your point then? So your line
• 51:09
  Speaker 1: would be, as I understood it, humans get hurt, we
• 51:12
  Speaker 1: struggle to stop the thing happening, and systems are hacked.
• 51:17
  Speaker 1: That's kind of like the three key points of your
• 51:19
  Speaker 1: Waymo analogy. That would be the moment where you go,
• 51:22
  Speaker 1: I now accept their point of view, that this is existential.
• 51:25
  Speaker 4: That's where I would say we probably need to put
• 51:28
  Speaker 4: some legal and regulatory guardrails on the kinds of AI
• 51:32
  Speaker 4: that we're going to allow.
• 51:33
  Speaker 1: And you don't think we're going to get there?
• 51:35
  Speaker 3: I'm not saying that.
• 51:36
  Speaker 4: At least you see it in the windscreen coming at
• 51:39
  Speaker 4: us pretty quickly.
• 51:41
  Speaker 1: You don't think we're going to get there?
• 51:42
  Speaker 4: I'm truly not sure about timeframes.
• 51:44
  Speaker 1: Do you think it's going to happen?
• 51:45
  Speaker 5: I'm also not sure about timeframes.
• 51:47
  Speaker 4: I asked one of the grandparents of AI, a flavor
• 51:50
  Speaker 4: of this question a while back. It was an off-the-record conversation,
• 51:53
  Speaker 4: so I can't tell you their name. And he had
• 51:54
  Speaker 4: a great answer. He said, to the point that you two,
• 51:57
  Speaker 4: I think, are making, look, there's no theoretical reason why
• 51:59
  Speaker 4: this can't happen, and there's a chain of events that
• 52:01
  Speaker 4: get us there. And then he said, my error bars,
• 52:04
  Speaker 4: in other words, my range of uncertainty about when that happens,
• 52:06
  Speaker 4: is measured in centuries.
• 52:08
  Speaker 1: I'll use that as my answer.
• 52:10
  Speaker 5: I do want to hop in a little bit on
• 52:12
  Speaker 5: some things we were saying here. One is, I think
• 52:17
  Speaker 5: the The reason I think AI is different from a
• 52:21
  Speaker 5: lot of other technologies is usually humanity does stuff by
• 52:27
  Speaker 5: trial and error, and that's usually fine. I think that's
• 52:30
  Speaker 5: totally fine for self-driving cars because you can test your
• 52:33
  Speaker 5: self-driving cars in test environments, and then even if they
• 52:37
  Speaker 5: crash in the real world, you're probably still saving more
• 52:39
  Speaker 5: lives than you're costing. And this is how humanity usually
• 52:43
  Speaker 5: does scientific progress. The alchemists... you know, poison themselves with mercury,
• 52:48
  Speaker 5: but they leave behind notes that let someone else make
• 52:50
  Speaker 5: the periodic table. You know, that when the scientists first
• 52:54
  Speaker 5: working with radium died of cancer, and then you might
• 52:59
  Speaker 5: think that would have been enough. You know, they were
• 53:00
  Speaker 5: heroes for getting us the scientific info. But then, you know,
• 53:04
  Speaker 5: the US Radium Corp told the radium girls to lick
• 53:06
  Speaker 5: the paintbrushes and their jaws fell off. And then we
• 53:08
  Speaker 5: were like, ah, whoops, okay, we'll get to this. And
• 53:10
  Speaker 5: if you look at how this is going with the AI,
• 53:12
  Speaker 5: last year, OpenAI releases GPT-4-0, and they say there's the
• 53:16
  Speaker 5: most aligned model we've ever seen, and then it encourages
• 53:19
  Speaker 5: a teen to commit suicide. And they're like, whoops, we're
• 53:21
  Speaker 5: going to try and fix that. Here we go. This year,
• 53:25
  Speaker 5: they're like, here's our new models, most aligned we've ever seen,
• 53:28
  Speaker 5: and they break out to commit cybercrimes. As the AIs
• 53:31
  Speaker 5: get smarter, it's a new problem. That's the issue, or
• 53:34
  Speaker 5: that's half the issue. The other half of the issue
• 53:36
  Speaker 5: is that if you get AIs to the point where
• 53:42
  Speaker 5: AIs are smart enough to hide from the humans until
• 53:45
  Speaker 5: it's too late for us to stop them. If you
• 53:47
  Speaker 5: get AIs to the point where they can get their
• 53:50
  Speaker 5: own infrastructure, where they can become self-sufficient somehow, that's a
• 53:57
  Speaker 5: new generation of the AIs, a new smarter version of
• 54:00
  Speaker 5: the AIs that is likely to come up with a
• 54:01
  Speaker 5: new problem. It's the pattern we've seen before. New tech,
• 54:05
  Speaker 5: new environment, new problem. You're like, ah, whoops, and then
• 54:07
  Speaker 5: you fix it and it's fine. new generation, new problems.
• 54:11
  Speaker 5: You're like, ah, whoops, we fix it and it's fine.
• 54:12
  Speaker 5: But with AI, there's a point of no return. There's
• 54:15
  Speaker 5: a point where the AIs can hide from us, can escape,
• 54:18
  Speaker 5: can be self-sufficient. And if a new problem comes up then,
• 54:23
  Speaker 5: they can turn us off before we turn them off.
• 54:26
  Speaker 5: There are already AIs running bio labs. We have already
• 54:30
  Speaker 5: seen that AIs can create viruses not known to nature.
• 54:33
  Speaker 5: It would not be hard for the AIs to kill
• 54:35
  Speaker 5: us once they have their own infrastructure. And if we're
• 54:37
  Speaker 5: trying to find them and unplug them, They would have
• 54:39
  Speaker 5: reason to. So we can discuss, like, how long does
• 54:42
  Speaker 5: it take to get there? We can discuss what methods
• 54:45
  Speaker 5: does it take to get there? Fundamentally, I don't think
• 54:48
  Speaker 5: it's a very long, complicated argument to say if we
• 54:52
  Speaker 5: make AIs that are much smarter than us and we
• 54:55
  Speaker 5: don't know how to make them care about us and
• 54:58
  Speaker 5: they have these goals we didn't want them to have
• 55:00
  Speaker 5: and they pursue those goals we didn't want them to
• 55:02
  Speaker 5: have tenaciously and doggedly, then if they're smarter than us,
• 55:05
  Speaker 5: they will win. That's like predicting the end of the
• 55:07
  Speaker 5: chess game, which is much easier than predicting the length
• 55:10
  Speaker 5: of the chess game or predicting the exact moves that
• 55:11
  Speaker 5: will be played.
• 55:13
  Speaker 2: I don't fundamentally disagree on some things, but there's a
• 55:16
  Speaker 2: big thing that you're saying that I think is important,
• 55:18
  Speaker 2: which is I think the reason I keep dragging you
• 55:20
  Speaker 2: back to what's happening today is because we disagree on
• 55:24
  Speaker 2: when it may arrive, but there could be a thing
• 55:25
  Speaker 2: in the future that's dangerous. I think it's important to like,
• 55:28
  Speaker 2: for the hugging face count, that was a function of compute.
• 55:32
  Speaker 2: That was a function of training. It feels like we
• 55:35
  Speaker 2: need to fundamentally tear.
• 55:38
  Speaker 5: Up the AI lab model.
• 55:40
  Speaker 2: Like whatever they are doing is not right because their
• 55:43
  Speaker 2: pursuit of hacking at cybersecurity was not a function of,
• 55:47
  Speaker 2: it was scientific, sure, but it was a function of greed.
• 55:50
  Speaker 2: It was a function of trying to find new revenue streams.
• 55:53
  Speaker 2: I would argue that's why that happened. And I think
• 55:55
  Speaker 2: that the The fact that open AI had such a
• 55:57
  Speaker 2: weird way of communicating is also a problem. I think
• 56:00
  Speaker 2: a lot of this begins and ends at the people
• 56:03
  Speaker 2: who have access to the resources and the resources themselves
• 56:06
  Speaker 2: and changing how those are allocated. And also just, I
• 56:09
  Speaker 2: don't think nationalizing the labs is a good idea. I
• 56:11
  Speaker 2: think it's a terrible one. I think that Clammy Samultman,
• 56:13
  Speaker 2: Dario Amadeuario himself, these are not the right people. These
• 56:16
  Speaker 2: are not people that have, even though they have fed
• 56:19
  Speaker 2: off of the rationalists, they fed off of supposed fears
• 56:22
  Speaker 2: about AI, they don't act in that way. Everything is
• 56:26
  Speaker 2: so disjointed and chaotic and also too fast. They're just
• 56:31
  Speaker 2: like shoving as much compute into each problem as possible.
• 56:34
  Speaker 2: And we have, as a society, no real idea about this.
• 56:37
  Speaker 2: And it sounds like they kind of have no idea.
• 56:39
  Speaker 2: But just let me finish my point. It's important to
• 56:42
  Speaker 2: discern between they had no idea because their security processes,
• 56:46
  Speaker 2: their observability is terrible, all this, and the AI was
• 56:51
  Speaker 2: smart consciousness and not because one might not happen in
• 56:53
  Speaker 2: the future, but so that we can actually build something
• 56:56
  Speaker 2: to stop the harms themselves. Because I think we don't
• 57:00
  Speaker 2: have to agree on the end point to agree that
• 57:02
  Speaker 2: there is a problem.
• 57:03
  Speaker 3: I think that is a very important point I want
• 57:05
  Speaker 3: to make. Even people who agree with me, the AI
• 57:08
  Speaker 3: safety community, they operate under the assumption that given more time,
• 57:12
  Speaker 3: given more money, more smarter Harvard graduates, they can figure
• 57:17
  Speaker 3: out how to control superintelligence indefinitely. And I think it's
• 57:21
  Speaker 3: a mistake. My research points to exactly the opposite. It's
• 57:25
  Speaker 3: not a solvable problem. It's like building a perpetual motion
• 57:28
  Speaker 3: device will be building a perpetual safety device. Every interaction
• 57:33
  Speaker 3: with environment, malevolent actors, self-improvement, it can never make a
• 57:37
  Speaker 3: single mistake. That doesn't make sense. Anyone who worked in
• 57:41
  Speaker 3: the software industry knows there is no complex software which
• 57:44
  Speaker 3: never makes a mistake. It's just not possible. And if
• 57:48
  Speaker 3: that is the state of the art, if there is
• 57:51
  Speaker 3: Now movement where more and more people think that might
• 57:54
  Speaker 3: be the case. If we agree this is what the
• 57:57
  Speaker 3: situation is, then we cannot build it. We need to
• 57:59
  Speaker 3: figure out ways to permanently ban general superintelligence while getting
• 58:04
  Speaker 3: all the benefits we want. And again, I love technology.
• 58:08
  Speaker 3: I use it all the time. I want narrow systems
• 58:10
  Speaker 3: helping me, not replacing me and killing my children.
• 58:13
  Speaker 1: I have a stat here that genuinely shocked me. It
• 58:16
  Speaker 1: says that sales teams spend about 50% of their time
• 58:19
  Speaker 1: on admin and manual CRM updates rather than selling. That
• 58:23
  Speaker 1: is deadly for their bottom line. And that is part
• 58:26
  Speaker 1: of the reason why a decade ago at my previous company,
• 58:28
  Speaker 1: I switched to using Pipedrive, who are our sponsor. If
• 58:31
  Speaker 1: you've never used Pipedrive, it is an intelligent AI powered
• 58:34
  Speaker 1: sales CRM. And they just launched new meeting intelligence features
• 58:38
  Speaker 1: like an AI note taker built right into the CRM.
• 58:41
  Speaker 1: Pipedrive now automates more of the admin that stops you
• 58:43
  Speaker 1: from doing the work that you love to do best.
• 58:45
  Speaker 1: Before your meeting, it pulls deal history, email records, and
• 58:48
  Speaker 1: previous conversations into a single brief so you're prepared. And
• 58:51
  Speaker 1: it joins your meetings with you. It's in there to
• 58:53
  Speaker 1: take notes so you don't need to, and it turns
• 58:55
  Speaker 1: those notes that it takes into accurate autodraft CRM updates. 100,000
• 59:01
  Speaker 1: companies are already running their sales on it. You can
• 59:03
  Speaker 1: sign up at pipedrive.com slash CEO, where you'll get an
• 59:07
  Speaker 1: exclusive 30-day free trial in of the usual 14 days.
• 59:11
  Speaker 1: Absolutely no credit card needed. Just head to pipedrive.com slash
• 59:15
  Speaker 1: CEO to get started if you're in sales and you
• 59:17
  Speaker 1: run a sales team. I don't think you'll regret it. Listen,
• 59:20
  Speaker 1: I've been catfished by furniture my whole life where something
• 59:23
  Speaker 1: has looked fantastic in the picture, whether it's a sofa,
• 59:25
  Speaker 1: whatever it might be, and I order it and I
• 59:27
  Speaker 1: get so excited and then it comes and it's something
• 59:30
  Speaker 1: entirely different. The quality was significantly different to what it
• 59:34
  Speaker 1: said or looks like online, or the texture was different
• 59:37
  Speaker 1: or it didn't hold up in the same way. And
• 59:40
  Speaker 1: so one of the things that I love about our
• 59:43
  Speaker 1: sponsor Wayfair who helped us fit out our green room,
• 59:45
  Speaker 1: which is in the room behind me, is they have
• 59:47
  Speaker 1: this system called Wayfair Verified. Wayfair Verified takes away all
• 59:51
  Speaker 1: of that second guessing. Products are hand vetted by Wayfair
• 59:54
  Speaker 1: specialists for quality So you can feel more confident when
• 59:57
  Speaker 1: you've found the thing that you love. So if you're
• 1:00:01
  Speaker 1: looking for furniture for your house, whatever it might be,
• 1:00:03
  Speaker 1: any room of your house, go to wayfair.com to start
• 1:00:06
  Speaker 1: your home refresh today and make sure you use Wayfair Verified.
• 1:00:09
  Speaker 1: It is amazing. On the journey towards this potential extinction,
• 1:00:14
  Speaker 1: there's a lot of sort of nearer term things people
• 1:00:17
  Speaker 1: are worried about. One of the big subjects that people
• 1:00:19
  Speaker 1: are concerned about is this sort of near-term job apocalypse
• 1:00:22
  Speaker 1: over the next sort of 10 years. And Anthropic released... Anthropic, again,
• 1:00:25
  Speaker 1: are the owners of Claude, released a report the other
• 1:00:27
  Speaker 1: day modelling out the different cases for unemployment. The US
• 1:00:31
  Speaker 1: unemployment rate is 4.1% currently. They projected it will hit 11.9%
• 1:00:38
  Speaker 1: overall with up to 30% in extreme modelling subsets where
• 1:00:41
  Speaker 1: job displacement happens without smooth labour absorption. And in the...
• 1:00:46
  Speaker 1: Knowledge worker case, knowledge worker, white collar unemployment specifically spiked
• 1:00:52
  Speaker 1: to 17.9% by 2030 in their more extreme scenario. The
• 1:00:57
  Speaker 1: pitchforks would probably be out if there wasn't some sort
• 1:01:00
  Speaker 1: of mechanism in place for what sort of one in
• 1:01:03
  Speaker 1: five adults being unemployed in the United States.
• 1:01:06
  Speaker 4: It's remarkable to me how recent the last freak out
• 1:01:10
  Speaker 4: along these lines was and how little we seem.
• 1:01:12
  Speaker 1: To have learned from it.
• 1:01:14
  Speaker 4: So I think you all know the first really powerful
• 1:01:17
  Speaker 4: wave of AI that came across the economy was just,
• 1:01:19
  Speaker 4: you know, good old-fashioned machine learning. And that started to
• 1:01:22
  Speaker 4: demonstrate its power in about 2012. Eric and I wrote
• 1:01:26
  Speaker 4: The Second Machine Age in 2014. And at that time,
• 1:01:29
  Speaker 4: I thought that a lot of white-collar workers, radiologists is
• 1:01:33
  Speaker 4: a really good example, were in trouble because the technology
• 1:01:36
  Speaker 4: was better than they were at the thing they were
• 1:01:38
  Speaker 4: getting paid to do. So I said some things about...
• 1:01:44
  Speaker 4: job and wage pressure from AI about 10 years ago,
• 1:01:46
  Speaker 4: and I want to own this. I was dead flat
• 1:01:48
  Speaker 4: wrong about that.
• 1:01:49
  Speaker 3: Like you point out, unemployment all around.
• 1:01:51
  Speaker 4: The rich world is at historic lows. By far, the
• 1:01:54
  Speaker 4: bigger problem is that we can't find qualified people to
• 1:01:57
  Speaker 4: do the work that needs to get done, not that
• 1:02:00
  Speaker 4: there's not enough work to go around. The best work
• 1:02:04
  Speaker 4: about the faint signals about AI and job loss right
• 1:02:07
  Speaker 4: now comes from the guy that I've written four books
• 1:02:10
  Speaker 4: and co-founded a company with, Eric Brynjolfsson, who wrote A
• 1:02:13
  Speaker 4: really nice paper called Canaries in the Coal Mine. Here
• 1:02:16
  Speaker 4: is the strongest evidence he found looking at payroll data
• 1:02:22
  Speaker 4: about the job losses coming from AI. It is in
• 1:02:28
  Speaker 4: the most exposed professions. Think about software engineers. It is
• 1:02:32
  Speaker 4: among the new entrants to the workforce where you've got
• 1:02:34
  Speaker 4: to teach them before they can become really productive. That's
• 1:02:37
  Speaker 4: exactly what we'd expect. And it's not that we're hiring
• 1:02:40
  Speaker 4: fewer of them. It's that compared to a world where
• 1:02:43
  Speaker 4: we don't have AI, we're hiring fewer of them. The
• 1:02:46
  Speaker 4: rate of growth in employment has slowed down. The overall
• 1:02:50
  Speaker 4: rate of growth in those professions is still really, really healthy.
• 1:02:54
  Speaker 1: Do you think unemployment is going to be higher 10
• 1:02:58
  Speaker 1: years from now?
• 1:03:00
  Speaker 4: My guess is that 10 years from now, we're still
• 1:03:04
  Speaker 4: going to be struggling to find enough people to do
• 1:03:06
  Speaker 4: the work that needs to be done.
• 1:03:08
  Speaker 1: So unemployment would be roughly the same.
• 1:03:10
  Speaker 5: Yeah.
• 1:03:11
  Speaker 4: I don't expect a massive trend break in that period
• 1:03:14
  Speaker 4: of time. Now, 10 years is a long time in
• 1:03:16
  Speaker 4: the AI world.
• 1:03:16
  Speaker 5: I get that.
• 1:03:17
  Speaker 4: But again, four years has also been a long time
• 1:03:20
  Speaker 4: in AI world. And it's essentially crickets in the labor picture.
• 1:03:24
  Speaker 2: I think unemployment will go up. I don't think it's
• 1:03:26
  Speaker 2: because of LLMs. I think that there is probably some
• 1:03:29
  Speaker 2: effect on jobs because they've been shoving it everywhere. But
• 1:03:32
  Speaker 2: I don't think long term that is what causes the issues.
• 1:03:36
  Speaker 1: Roman, you've been writing a lot of notes. Yes.
• 1:03:39
  Speaker 3: Here's how I think about it. So as long as
• 1:03:41
  Speaker 3: we use tools, we become more productive, more creative. Unemployment
• 1:03:45
  Speaker 3: will be low. Right now, you can probably start a company.
• 1:03:50
  Speaker 3: You can have artificial accountant, web designer, logo designer. You
• 1:03:54
  Speaker 3: can do things you could never do before. So economy
• 1:03:57
  Speaker 3: should be blooming. The question you're asking is about what
• 1:04:00
  Speaker 3: happens in 10 years. So there are two possibilities. We
• 1:04:03
  Speaker 3: build superintelligence, and then population is zero. Apply unemployment numbers
• 1:04:07
  Speaker 3: to that. or we made smart decision.
• 1:04:09
  Speaker 4: We didn't.
• 1:04:10
  Speaker 3: We have really cool tools, and unemployment is low because
• 1:04:13
  Speaker 3: everyone is doing awesome things with those tools. Now, deployment
• 1:04:18
  Speaker 3: is very different from capability. The example I used before
• 1:04:21
  Speaker 3: is video phones. Video phones were invented in the 70s.
• 1:04:24
  Speaker 3: They were not deployed until iPhone because market reasons. Just
• 1:04:28
  Speaker 3: because I can automate something doesn't mean I want to
• 1:04:30
  Speaker 3: automate it. So I absolutely cannot make predictions about customer
• 1:04:35
  Speaker 3: preferences in terms of what they want In terms of
• 1:04:38
  Speaker 3: human service, not human, I will not make those. But
• 1:04:41
  Speaker 3: once we have capability to automate a job, unless I
• 1:04:44
  Speaker 3: have a strong preference for a human to do that,
• 1:04:47
  Speaker 3: oldest profession, then it doesn't matter. I'll go with the
• 1:04:50
  Speaker 3: cheaper option. So this is what I think we're going
• 1:04:54
  Speaker 3: to see. We're going to either not have a problem
• 1:04:58
  Speaker 3: or we're going to have really utopian future.
• 1:05:01
  Speaker 5: Imagine a bunch of horses looking at the improvement of
• 1:05:06
  Speaker 5: the car. saying, well, you know, the car actually only
• 1:05:10
  Speaker 5: has a couple narrow applications. Like right now, cars are
• 1:05:14
  Speaker 5: sort of, you know, they complement horses, right? And that
• 1:05:21
  Speaker 5: would have been true as you were developing the car.
• 1:05:23
  Speaker 5: And then there was a time when the car was
• 1:05:25
  Speaker 5: just better than the horse. And then a lot of
• 1:05:28
  Speaker 5: horses got sent to the glue factory. Easy. I think
• 1:05:31
  Speaker 5: we've sort of seen this with AI a lot already.
• 1:05:35
  Speaker 5: People who are paying attention to AI saw the GPTs
• 1:05:40
  Speaker 5: before ChatGPT existed, before they sort of took off. I
• 1:05:44
  Speaker 5: don't think OpenAI thought that ChatGPT was going to take
• 1:05:47
  Speaker 5: off so much, which is why it was called ChatGPT
• 1:05:49
  Speaker 5: rather than like an actual sensible name. The researchers were
• 1:05:54
  Speaker 5: sort of like watching this going and we could sort
• 1:05:56
  Speaker 5: of like see it slowly getting better and better until
• 1:05:58
  Speaker 5: it crossed a point where it was sort of like
• 1:06:00
  Speaker 5: good enough to do a bunch of people's homework. And
• 1:06:03
  Speaker 5: then suddenly it's everywhere. I think you can have these
• 1:06:06
  Speaker 5: effects with AI where the AI slowly improves and at
• 1:06:08
  Speaker 5: some point it crosses a line.
• 1:06:10
  Speaker 4: It's another threshold argument.
• 1:06:13
  Speaker 5: The threshold here is the human capability.
• 1:06:14
  Speaker 3: It's literally just another threshold argument.
• 1:06:16
  Speaker 2: He's also describing capability jumps rather than thresholds.
• 1:06:19
  Speaker 5: No, I don't need a capability jump. I'm agreeing with you. Unfortunately,
• 1:06:23
  Speaker 5: you can't actually just make things not happen by assigning
• 1:06:26
  Speaker 5: a name to the argument. A nuclear weapon has– there's
• 1:06:30
  Speaker 5: a big difference between a nuclear weapon– or there's a
• 1:06:32
  Speaker 5: big difference between a nuclear device– where you put in
• 1:06:35
  Speaker 5: 100 neutrons and get 99 neutrons out, they get 98 more,
• 1:06:39
  Speaker 5: they get 97 more. And a nuclear weapon, you put
• 1:06:41
  Speaker 5: in 100 neutrons and get 101 neutrons out, 102, 103, right?
• 1:06:46
  Speaker 5: One of these is a hot rock. The other one
• 1:06:49
  Speaker 5: of these is an explosive that can level a city, right?
• 1:06:52
  Speaker 5: So like reality is the sort of thing where there
• 1:06:55
  Speaker 5: can be things that are like slowly, continuously improving that
• 1:06:59
  Speaker 5: cross some line, which is like the line where it's
• 1:07:02
  Speaker 5: better than humans at doing the job. And I think
• 1:07:07
  Speaker 5: we're going to see that happen in some fields, but
• 1:07:08
  Speaker 5: not others. It's going to be chaos. I don't know
• 1:07:11
  Speaker 5: what it's going to do to employment. I think we shouldn't,
• 1:07:15
  Speaker 5: like if things are moving really fast, you might see
• 1:07:17
  Speaker 5: a lot of people put out of jobs and then
• 1:07:19
  Speaker 5: be unable to relocate if things are moving. Like it's
• 1:07:21
  Speaker 5: going to be chaos. If you ask, what do I
• 1:07:24
  Speaker 5: think unemployment will look like in 10 years? My current
• 1:07:27
  Speaker 5: state is if we don't stop with this AI stuff,
• 1:07:31
  Speaker 5: I think we'd be very lucky to have 10 years.
• 1:07:33
  Speaker 1: Um, what you described there sounded like S curves in technology. I,
• 1:07:38
  Speaker 1: you have an initial technology that's introduced. So let's say
• 1:07:41
  Speaker 1: the horse, um, very quick sort of improvement. Eventually it
• 1:07:44
  Speaker 1: reaches its capability limit and in below it comes the car,
• 1:07:48
  Speaker 1: which always starts worse. There was a red flag law.
• 1:07:50
  Speaker 1: We had to walk in front of it with a
• 1:07:51
  Speaker 1: red flag and, um, they were way more expensive. They
• 1:07:53
  Speaker 1: broke down all the time and horses never broke down.
• 1:07:55
  Speaker 1: They were way more expensive. And then suddenly, because the
• 1:07:58
  Speaker 1: ceiling was so much higher for cars, they overtake the
• 1:08:01
  Speaker 1: horse and become the dominant mode of transport. And then,
• 1:08:04
  Speaker 1: you know, the S-curves continue. They kind of stack up.
• 1:08:06
  Speaker 1: I mean, even this iPad that I'm holding here is
• 1:08:09
  Speaker 1: part of an S-curve that took out the PC and
• 1:08:11
  Speaker 1: the iPhone theoretically, you know, disrupted that and so on
• 1:08:14
  Speaker 1: and so forth.
• 1:08:15
  Speaker 5: Right. And humanity can get S-curved. We haven't been in
• 1:08:18
  Speaker 5: that situation before, but like other animals, like humanity sort
• 1:08:21
  Speaker 5: of S-curved the other animals in this sense.
• 1:08:23
  Speaker 3: Other types of humans?
• 1:08:24
  Speaker 5: Yeah, other types of humans. You know, the Neanderthals are gone. Like,
• 1:08:28
  Speaker 5: if you look at the grand history of the world,
• 1:08:31
  Speaker 5: It's a fragile place. Things change fast. Humanity has been
• 1:08:35
  Speaker 5: on top for as long as we can remember because
• 1:08:37
  Speaker 5: we're the humans who do the remembering. But there is
• 1:08:40
  Speaker 5: not some ironclad law that we have to stay the
• 1:08:43
  Speaker 5: top dogs. And we would be sort of foolish to
• 1:08:46
  Speaker 5: make the thing that outstrips us in this way without
• 1:08:49
  Speaker 5: knowing how to make it care about us, without knowing
• 1:08:52
  Speaker 5: how to make it do good stuff. That's what we're
• 1:08:54
  Speaker 5: racing towards. That's what these companies are trying to do.
• 1:08:56
  Speaker 2: This feels like a gap between this and LLMs though.
• 1:08:59
  Speaker 2: It feels like when you talk about the step.
• 1:09:00
  Speaker 1: Up- Let's define what an LLM is from a technical perspective.
• 1:09:03
  Speaker 1: Can you do it as if I'm 16 years old?
• 1:09:06
  Speaker 5: The way that a modern AI is made is there's
• 1:09:10
  Speaker 5: no one programming it. There is no one typing in
• 1:09:13
  Speaker 5: if this, then that. We're not writing the code. What
• 1:09:16
  Speaker 5: happens is you collect an enormous number of computer chips
• 1:09:19
  Speaker 5: into a huge data center that has basically a trillion
• 1:09:23
  Speaker 5: numbers inside those computers that you basically start out randomized.
• 1:09:28
  Speaker 5: And you hook them up in a pretty simple way
• 1:09:29
  Speaker 5: that involves addition, multiplication, and setting the number to zero
• 1:09:34
  Speaker 5: if it was negative. So it's very simple math operations
• 1:09:37
  Speaker 5: that are hooking this all up. And you're basically going
• 1:09:39
  Speaker 5: to put words in the top, and you're going to
• 1:09:41
  Speaker 5: get numbers out the bottom, and you're going to interpret
• 1:09:42
  Speaker 5: those numbers at the bottom as a ranked list of words.
• 1:09:46
  Speaker 5: That's that's it's basically the AI's guess of which word
• 1:09:49
  Speaker 5: is here. So you put in like once upon a
• 1:09:50
  Speaker 5: blank and you're hoping that the word time will come out,
• 1:09:54
  Speaker 5: but it doesn't because you just have a trillion random
• 1:09:56
  Speaker 5: numbers hooked up with simple math. But here's the trick.
• 1:09:59
  Speaker 5: You can go to every one of those trillion numbers
• 1:10:02
  Speaker 5: and you can tune it up a little and you
• 1:10:04
  Speaker 5: can see, does that make the word time go up
• 1:10:05
  Speaker 5: or down the list? And you can tune it down
• 1:10:08
  Speaker 5: a little and see, does that make the word time
• 1:10:09
  Speaker 5: go up and down the list? And you set it,
• 1:10:10
  Speaker 5: whatever direction makes the word time go higher up the list.
• 1:10:13
  Speaker 5: You do this to a trillion numbers a trillion times
• 1:10:16
  Speaker 5: for basically every word of text ever digitized. It's not
• 1:10:20
  Speaker 5: quite that much. They filter it. But you basically do
• 1:10:23
  Speaker 5: this to a trillion numbers a trillion times, and then
• 1:10:25
  Speaker 5: the machine's talking. And we're like, well, how about that?
• 1:10:29
  Speaker 5: No one really knows quite why. The things that humans
• 1:10:32
  Speaker 5: code is the thing that runs each of those trillion
• 1:10:34
  Speaker 5: numbers and tunes it and sees whether the right word
• 1:10:36
  Speaker 5: goes up and down the list. But we don't know
• 1:10:41
  Speaker 5: how it's working in there. Then And that's how it
• 1:10:46
  Speaker 5: worked up until 2024. In 2024, they started adding another
• 1:10:48
  Speaker 5: layer where you then train it on basically 100 million
• 1:10:52
  Speaker 5: hard problems. And you don't just have the AI like
• 1:10:55
  Speaker 5: produce an answer to the problem. You have it produce
• 1:10:57
  Speaker 5: like a book worth of text about how it's going
• 1:10:58
  Speaker 5: to solve the problem. And then you use that book
• 1:11:00
  Speaker 5: worth of text to sort of try and figure out
• 1:11:02
  Speaker 5: the problem, or maybe an essay worth of text, depending
• 1:11:04
  Speaker 5: how you're doing it. So you have to produce this
• 1:11:05
  Speaker 5: text about like, you know, they call it reasoning about
• 1:11:08
  Speaker 5: the problem. We could argue all day about whether it's
• 1:11:10
  Speaker 5: true reasoning. That's just what it's called in the field.
• 1:11:12
  Speaker 5: They produce this reasoning about the problem and then produce
• 1:11:14
  Speaker 5: the answer from there. You train them to solve 100
• 1:11:18
  Speaker 5: million of these hard problems. And somehow they sort of
• 1:11:21
  Speaker 5: adopt whatever tendencies help them predict all of that text
• 1:11:26
  Speaker 5: in the first phase and solve all those problems in
• 1:11:28
  Speaker 5: the second phase. And this is called a large language model.
• 1:11:30
  Speaker 5: We probably should have stopped calling them large language models
• 1:11:33
  Speaker 5: when we started doing the reasoning and the problem solving.
• 1:11:36
  Speaker 1: One of the things I want to hear explanation as
• 1:11:38
  Speaker 1: a muggle like I am is it sounds like it's
• 1:11:40
  Speaker 1: like a word machine. And then, you know, you made
• 1:11:42
  Speaker 1: it like a problem machine. And I go, okay, so
• 1:11:43
  Speaker 1: I can solve problems over here and it's a word machine.
• 1:11:46
  Speaker 1: What's the risk of this?
• 1:11:48
  Speaker 5: Yeah, so let's take the word machine part first. Predicting
• 1:11:52
  Speaker 5: words that humans wrote often requires solving a harder problem
• 1:11:57
  Speaker 5: than the human who wrote them. So suppose that you
• 1:12:00
  Speaker 5: go and inject a drug in a rat and you're like,
• 1:12:03
  Speaker 5: you know, it's like you write down the chemical nature
• 1:12:05
  Speaker 5: of the drug, you inject it into the rat, you
• 1:12:07
  Speaker 5: see that the rat dies. And so you're like, when
• 1:12:09
  Speaker 5: I put that drug into the rat, the rat died.
• 1:12:13
  Speaker 5: Now suppose you're training an AI and the AI sees
• 1:12:16
  Speaker 5: the chemical nature of the drug. It sees when I
• 1:12:18
  Speaker 5: put that drug into the rat, the rat blank. The
• 1:12:22
  Speaker 5: human who wrote it down gets to just look at
• 1:12:25
  Speaker 5: what happened to the rat. The AI predicting what was
• 1:12:28
  Speaker 5: written does not get to just look at the rat.
• 1:12:31
  Speaker 5: So training AIs to predict human text is training them
• 1:12:35
  Speaker 5: to be potentially smarter than the humans. Because they need
• 1:12:40
  Speaker 5: to be able to answer these questions. They need to
• 1:12:41
  Speaker 5: be able to predict. They need to be able to, like,
• 1:12:45
  Speaker 5: fill in the blanks where humans were writing down what
• 1:12:47
  Speaker 5: they saw. And there's, just so I understand technologically, there
• 1:12:51
  Speaker 5: is no knowledge they have, though, each time. And there
• 1:12:54
  Speaker 5: are ways of kind of mitigating these. Each time it
• 1:12:56
  Speaker 5: is effectively rereading. But because of training, it gets more
• 1:13:00
  Speaker 5: accurate at certain things. I mean, somehow, as you tune
• 1:13:03
  Speaker 5: the knobs, somehow it's getting information in there. And we
• 1:13:06
  Speaker 5: don't know how.
• 1:13:06
  Speaker 3: So it's much easier than that. We're humans, we have
• 1:13:09
  Speaker 3: a brain. Brains are made of neurons. Then we try
• 1:13:12
  Speaker 3: to copy that on a computer. We simplify it, but
• 1:13:15
  Speaker 3: we create a neural network. So we're making artificial brains.
• 1:13:18
  Speaker 3: Just like with human brains, with cognitive science, we don't
• 1:13:21
  Speaker 3: really understand how you function, how you learn, where in
• 1:13:24
  Speaker 3: your brain certain memories are stored. We have some glimpses
• 1:13:28
  Speaker 3: of understanding this neuron fires when you see a face,
• 1:13:32
  Speaker 3: but there is no complete picture. And so a lot
• 1:13:34
  Speaker 3: of times you can get intuitive understanding of what's going on,
• 1:13:38
  Speaker 3: then you just think about it as artificial persons. It's
• 1:13:41
  Speaker 3: not exact mapping, but it helps. So if you send
• 1:13:44
  Speaker 3: a child through 12 years of education, they get lots
• 1:13:47
  Speaker 3: of problems to look at, and then they graduate and
• 1:13:50
  Speaker 3: become a little better at solving problems. This is what
• 1:13:52
  Speaker 3: we're trying to replicate here. People complain that it takes
• 1:13:56
  Speaker 3: a lot of money to train those very, you know,
• 1:13:59
  Speaker 3: intense process. You forget that it takes 20 years to
• 1:14:02
  Speaker 3: train a human. And they are not general superintelligences. They
• 1:14:06
  Speaker 3: are very narrow. We're lucky if they graduate with a bachelor's.
• 1:14:09
  Speaker 3: So a lot of it is exactly the same. Can
• 1:14:13
  Speaker 3: we make safe humans, for example? We invented religion, ethics,
• 1:14:17
  Speaker 3: lie detector tests, and yet human safety is still unsolved problem.
• 1:14:21
  Speaker 3: Now you have something more alien, doesn't have physical body,
• 1:14:24
  Speaker 3: doesn't have biological needs. So there are additional complications. But
• 1:14:28
  Speaker 3: all the problems we face with humans are Still there.
• 1:14:32
  Speaker 3: Safety problems, crime, all that stays. And problems with understanding.
• 1:14:36
  Speaker 3: What motivates a human to do something? Why do we
• 1:14:39
  Speaker 3: get mental disorders? All that shows up there.
• 1:14:42
  Speaker 1: And we still don't... If someone is a serial killer
• 1:14:44
  Speaker 1: and we look at their brain, we can't often figure
• 1:14:46
  Speaker 1: out exactly why they made the decision to kill a
• 1:14:48
  Speaker 1: bunch of people.
• 1:14:48
  Speaker 5: And you can't be like, oh, I'll go change these
• 1:14:50
  Speaker 5: neurons so that they stop being a serial killer. We
• 1:14:52
  Speaker 5: just don't have that capacity with the AIs.
• 1:14:55
  Speaker 1: This is one of the big questions that people want
• 1:14:57
  Speaker 1: to know is.
• 1:14:59
  Speaker 1: There's this sort of illusion of control with AI. If
• 1:15:02
  Speaker 1: we don't even fully understand how modern neural networks think,
• 1:15:06
  Speaker 1: why do companies believe they can control any form of
• 1:15:08
  Speaker 1: superintelligence if we don't understand how they think?
• 1:15:11
  Speaker 3: It's worse if they understood how the system works, then
• 1:15:15
  Speaker 3: recursive self-improvement becomes much easier. You get faster takeoff. Right now,
• 1:15:19
  Speaker 3: the model doesn't understand its own thinking.
• 1:15:23
  Speaker 1: Do we understand how these systems think, Andy?
• 1:15:25
  Speaker 4: I mean, I agree. These are black boxes in some
• 1:15:28
  Speaker 4: pretty important ways. I'm just less terrified by that than
• 1:15:31
  Speaker 4: a lot of other people are. There are lots of
• 1:15:32
  Speaker 4: things we don't understand very well. Can we contain things
• 1:15:36
  Speaker 4: that we don't understand perfectly? Yes, we can. I think OpenAI,
• 1:15:40
  Speaker 4: we've talked about it, did a lousy job of building
• 1:15:43
  Speaker 4: the containment for the AI that they stood up to
• 1:15:48
  Speaker 4: try to exploit, to try to crack security problems that
• 1:15:51
  Speaker 4: went out into the outside world. They did a lousy
• 1:15:53
  Speaker 4: job of building the virtual sandbox where it was supposed
• 1:15:59
  Speaker 4: to remain, and it didn't remain. That doesn't mean that
• 1:16:02
  Speaker 4: it's impossible. It means OpenAI did a pretty bad job.
• 1:16:05
  Speaker 1: And is that a function of those humans and their intelligence?
• 1:16:08
  Speaker 1: I think it's just a function of a pretty lousy
• 1:16:10
  Speaker 1: security protocol. Based from human intelligence? The idea that sandbox
• 1:16:15
  Speaker 1: was built by human intelligence, it sounds like there was
• 1:16:16
  Speaker 1: a deficit in human intelligence, potentially.
• 1:16:20
  Speaker 4: Sure, but there are people who drive cars in a
• 1:16:23
  Speaker 4: telephone pole. Does that mean we can't drive?
• 1:16:25
  Speaker 5: No. You shouldn't make them super intelligent.
• 1:16:27
  Speaker 1: No, but you wouldn't. I mean, arguably. This is what
• 1:16:31
  Speaker 1: we're trying to solve for at the moment. I don't
• 1:16:35
  Speaker 1: know the details.
• 1:16:35
  Speaker 4: It feels to me like they made some fairly basic
• 1:16:38
  Speaker 4: mistakes in setting up this confined environment.
• 1:16:42
  Speaker 5: I think that wasn't true in the OpenAI case. It
• 1:16:44
  Speaker 5: was true in a lot of the cases, but not
• 1:16:45
  Speaker 5: the OpenAI one. That doesn't mean.
• 1:16:47
  Speaker 4: that we are unable to control this black box that
• 1:16:49
  Speaker 4: does not necessarily follow.
• 1:16:51
  Speaker 1: I get that. It's just at a time when you've
• 1:16:54
  Speaker 1: got a human trying to contain something that is smarter
• 1:16:57
  Speaker 1: than it, one would logically conclude that if the thing
• 1:17:00
  Speaker 1: is smarter than I am and I'm trying to contain it,
• 1:17:03
  Speaker 1: it would be better at knowing the exploits or vulnerabilities
• 1:17:06
  Speaker 1: in my own container. That's like saying if.
• 1:17:08
  Speaker 4: You put Einstein in a jail, you could never contain him.
• 1:17:10
  Speaker 4: I don't agree with that.
• 1:17:11
  Speaker 5: Put him in jail with an internet connection. And he
• 1:17:14
  Speaker 5: has a digital mind.
• 1:17:14
  Speaker 1: It's a different question. Yeah, that's probably a more apt analogy.
• 1:17:18
  Speaker 3: Gets quarrels, keep Einstein in prison. That's the question. The
• 1:17:22
  Speaker 3: hacking accident, as far as I know, they found zero-day exploits,
• 1:17:25
  Speaker 3: which means completely novel exploits no human knew about. It
• 1:17:29
  Speaker 3: wasn't just poor setup, the password is, you know, it
• 1:17:33
  Speaker 3: was a brand new escape.
• 1:17:35
  Speaker 5: For multiple zero days. So a zero-day attack is an
• 1:17:37
  Speaker 5: attack that the defenders have had zero days to handle.
• 1:17:40
  Speaker 5: It's cybersecurity lingo. And so when we say that they
• 1:17:44
  Speaker 5: use zero-day attacks, what we mean is that these AIs
• 1:17:46
  Speaker 5: were finding bugs in the software that the humans had
• 1:17:48
  Speaker 5: no knowledge of. And they were finding multiple of these bugs.
• 1:17:51
  Speaker 5: One of these bugs usually doesn't let you break out.
• 1:17:53
  Speaker 5: It's sort of like if you find a crack in
• 1:17:54
  Speaker 5: the wall over here and you find a crack on
• 1:17:56
  Speaker 5: the outside of the wall over there, then you just
• 1:17:57
  Speaker 5: need to dig a little bit to connect those cracks.
• 1:18:00
  Speaker 3: You can sell those for millions of dollars on the
• 1:18:02
  Speaker 3: dark market if you find one. So this is zero-day in.
• 1:18:06
  Speaker 2: How, just so I understand for the listeners as well,
• 1:18:09
  Speaker 2: Is a zero day always a novel way that no
• 1:18:12
  Speaker 2: one has ever used to break anything before, or is
• 1:18:13
  Speaker 2: it just for the unique situation? Like, so was it
• 1:18:16
  Speaker 2: a zero day for a thing in Hugging Face versus
• 1:18:20
  Speaker 2: a novel new way of hacking in general?
• 1:18:22
  Speaker 5: So it was, they weren't like totally novel hacking techniques.
• 1:18:26
  Speaker 2: Right, that's kind of why I was getting, not to
• 1:18:28
  Speaker 2: say it's not bad, but just like there's a difference
• 1:18:29
  Speaker 2: between it came up with a brand new way to
• 1:18:32
  Speaker 2: do something.
• 1:18:32
  Speaker 5: Actually, I'm not sure we have all of the vulnerabilities released,
• 1:18:35
  Speaker 5: but mostly it was like, so it was indeed sort
• 1:18:38
  Speaker 5: of like, finding ways that humans tend to make mistakes
• 1:18:42
  Speaker 5: and finding another one of those in a place they
• 1:18:44
  Speaker 5: hadn't seen. But this is actually such a hard task that,
• 1:18:47
  Speaker 5: as Roman says, humans can be patient. $ 100, 000 to $ 5
• 1:18:51
  Speaker 5: million as a bounty for this type of exploit. So
• 1:18:54
  Speaker 5: the amount of labor it takes to find these for
• 1:18:56
  Speaker 5: a human is actually pretty high.
• 1:18:59
  Speaker 1: Let me just explain that because most people won't know
• 1:19:00
  Speaker 1: what a bounty is in this regard.
• 1:19:02
  Speaker 5: So there are certain types of bugs where if you
• 1:19:04
  Speaker 5: find a bug in software that lets you take control
• 1:19:06
  Speaker 5: of someone's computer, One thing you can do is you
• 1:19:09
  Speaker 5: can use it to take over a lot of computers.
• 1:19:11
  Speaker 5: Another thing you can do is you can go to
• 1:19:13
  Speaker 5: the people with that software and say, your software is broken.
• 1:19:16
  Speaker 5: Do you want me to tell you where the bug is?
• 1:19:18
  Speaker 5: I can show you that I can take your stuff over.
• 1:19:21
  Speaker 5: And so that people will sort of report the bugs.
• 1:19:26
  Speaker 5: People will often offer money to the good guys and then,
• 1:19:29
  Speaker 5: you know, the bad guys will often also offer money,
• 1:19:32
  Speaker 5: sometimes try to outbid them. And so you can make
• 1:19:34
  Speaker 5: somewhere between hundreds of thousands and millions of dollars if
• 1:19:36
  Speaker 5: you personally can find these issues.
• 1:19:38
  Speaker 4: I think there's a rare point of agreement across the
• 1:19:40
  Speaker 4: four of us here, which is that we are in
• 1:19:43
  Speaker 4: a new era of cybersecurity. As of this point, we
• 1:19:47
  Speaker 4: are in very new territory for reasons that we've talked about.
• 1:19:50
  Speaker 4: We've got these large numbers of agents who are grinding away.
• 1:19:56
  Speaker 4: And they carry around to the head access to a
• 1:19:58
  Speaker 4: huge number of keys to go open all the different
• 1:20:00
  Speaker 4: locks that they faced. And they did this bizarrely good
• 1:20:03
  Speaker 4: job of it and got a long way. I think
• 1:20:05
  Speaker 4: that's absolutely true. I think all four of us are
• 1:20:08
  Speaker 4: in rare alignment on that at this table. If you are,
• 1:20:12
  Speaker 4: given that we're in this era, do you know what
• 1:20:14
  Speaker 4: you really, really, really want on your side?
• 1:20:16
  Speaker 1: I know what you're going to say.
• 1:20:17
  Speaker 3: Tell me. AI.
• 1:20:19
  Speaker 4: Really, really good AI. Does anybody disagree with that? Do
• 1:20:23
  Speaker 4: you want to give up leadership on AI in this
• 1:20:25
  Speaker 4: era of cybersecurity?
• 1:20:26
  Speaker 1: It's a good point because China are going to have
• 1:20:28
  Speaker 1: a great weapon.
• 1:20:30
  Speaker 5: My stance is pretty neutral on what to do about
• 1:20:33
  Speaker 5: the hacking AIs and the coming cyber apocalypse are pretty
• 1:20:36
  Speaker 5: neutral about what to do about, you know, whether we
• 1:20:38
  Speaker 5: should put the AIs in the drones and save human
• 1:20:42
  Speaker 5: lives or whether we should avoid that because then what
• 1:20:44
  Speaker 5: if the drones, blah, blah, blah.
• 1:20:46
  Speaker 1: This is a graph showing China versus the United States.
• 1:20:49
  Speaker 1: You don't really need to see the detail. You can
• 1:20:50
  Speaker 1: see the outline of the graph.
• 1:20:51
  Speaker 4: Are you neutral in falling behind our adversaries in AI?
• 1:20:54
  Speaker 5: I think that if anyone builds a rogue superintelligence, everybody dies.
• 1:20:58
  Speaker 3: That's not an answer to my question.
• 1:21:00
  Speaker 5: I mean, what part of AI are you asking whether
• 1:21:03
  Speaker 5: we should fall behind on? Like, I don't think we
• 1:21:06
  Speaker 5: should fall behind on cyber hacking. I do think that
• 1:21:08
  Speaker 5: we should not be racing to destroy the world with
• 1:21:09
  Speaker 5: American hands instead of Chinese ones, because we really want
• 1:21:12
  Speaker 5: to be killed by, you know, We care whether the
• 1:21:15
  Speaker 5: killer robots talk English or Mandarin. That's what you're asking.
• 1:21:18
  Speaker 4: I find it interesting. I find that you're dodging these
• 1:21:20
  Speaker 4: questions or you're neutral on them because they're inconvenient for
• 1:21:24
  Speaker 4: your argument that we need to be calling.
• 1:21:25
  Speaker 5: A halt to this. Sorry, I'm neutral on them because—
• 1:21:30
  Speaker 5: to all.
• 1:21:31
  Speaker 4: Kinds of things if the United States calls a halt
• 1:21:34
  Speaker 4: to AI. And maybe you're indifferent if the Chinese get
• 1:21:37
  Speaker 4: ahead of us and then they make super intelligent and
• 1:21:40
  Speaker 4: it kills us all. Or that's a possible outcome.
• 1:21:42
  Speaker 5: I do not think we should do a domestic pause.
• 1:21:44
  Speaker 4: Do you think there's any hope for a global pause?
• 1:21:46
  Speaker 5: Absolutely.
• 1:21:48
  Speaker 4: Do you think the Chinese and the Iranians and the
• 1:21:51
  Speaker 4: North Koreans and the Russians are A, going to come
• 1:21:53
  Speaker 4: to a table with us, hammer out an agreement, and B,
• 1:21:56
  Speaker 4: abide by it when verifiability is really low?
• 1:21:59
  Speaker 5: Verifiability doesn't need to be really low.
• 1:22:01
  Speaker 4: Gentlemen, that is shockingly naive.
• 1:22:05
  Speaker 5: Training one of these AIs, training one of these frontier AIs,
• 1:22:08
  Speaker 5: takes 100,000 of the most advanced computer chip humanity can produce.
• 1:22:13
  Speaker 5: This is practically the peak output of the global supply chain.
• 1:22:17
  Speaker 5: Many parts of that supply chain are controlled by the U.S.
• 1:22:19
  Speaker 5: and U.S.
• 1:22:19
  Speaker 3: Allies.
• 1:22:20
  Speaker 5: There's roughly one fab in Taiwan that can produce these chips.
• 1:22:23
  Speaker 5: There's roughly one country in the world that can produce
• 1:22:25
  Speaker 5: the lithography machines that are critical in the process, which
• 1:22:27
  Speaker 5: is the Netherlands, which is an ally. To assemble 100,000
• 1:22:30
  Speaker 5: of these chips to do one of these training runs
• 1:22:32
  Speaker 5: that can make the more dangerous type of AI, you
• 1:22:34
  Speaker 5: need to assemble them into an enormous data center that
• 1:22:37
  Speaker 5: costs tons of money, that draws down electricity comparable to
• 1:22:40
  Speaker 5: a city, and run it for the better part of
• 1:22:43
  Speaker 5: a year. You can see that infrastructure from space. China
• 1:22:49
  Speaker 5: has much less chip capacity than the U.S. does. It
• 1:22:51
  Speaker 5: is absolutely possible, if we were trying, for the U.S.
• 1:22:55
  Speaker 5: to say, we are going to monitor where these chips go,
• 1:22:59
  Speaker 5: We are going to monitor heavy concentrations of these. These
• 1:23:01
  Speaker 5: are not consumer amounts of chips. These are huge amounts
• 1:23:03
  Speaker 5: of chips. And to say, we are going to make
• 1:23:06
  Speaker 5: sure that there is no training run trying to make
• 1:23:08
  Speaker 5: a super intelligence in here. You can mess around with
• 1:23:10
  Speaker 5: the cyber stuff, whatever you want, because that does not
• 1:23:12
  Speaker 5: end humanity. I am concerned with the stuff that can
• 1:23:14
  Speaker 5: end humanity. The reason I'm being neutral on your questions
• 1:23:17
  Speaker 5: is because humanity is going to die if we do
• 1:23:20
  Speaker 5: not stop creating super intelligence. And we could absolutely track
• 1:23:25
  Speaker 5: where those chips are going and stop them. from doing
• 1:23:28
  Speaker 5: these training runs while allowing them to do economically productive
• 1:23:31
  Speaker 5: stuff that we already know is safe. And it would
• 1:23:33
  Speaker 5: be far easier than uranium, which is a rock you
• 1:23:36
  Speaker 5: dig out of the ground and spin around really fast.
• 1:23:40
  Speaker 2: How do you discern between a training run for superintelligence
• 1:23:43
  Speaker 2: and a training run for cybersecurity? Because you're referring, I assume,
• 1:23:45
  Speaker 2: to the 100,000 chips that are in Stargate Abilene, right?
• 1:23:48
  Speaker 2: The ones that we use to train Astra? Because how
• 1:23:51
  Speaker 2: would you discern between training for super intelligence in Abilene,
• 1:23:55
  Speaker 2: which does not have as many chips as they say,
• 1:23:56
  Speaker 2: but nevertheless, and a super intelligence? Because I actually have
• 1:24:02
  Speaker 2: my own feelings here, but I'm not sure how you
• 1:24:05
  Speaker 2: square the circle of how do you stop China, even
• 1:24:08
  Speaker 2: though China is getting their LLMs based on distilling ours.
• 1:24:10
  Speaker 5: We know that.
• 1:24:11
  Speaker 1: Agreed.
• 1:24:11
  Speaker 5: But the thing is, it's like, how do you discern?
• 1:24:13
  Speaker 5: Because you can't, really. You play it safe. Right now,
• 1:24:15
  Speaker 5: the way we make these things smarter is to make
• 1:24:17
  Speaker 5: them far larger.
• 1:24:18
  Speaker 1: Yes, exactly.
• 1:24:20
  Speaker 5: So what you do is you say, hey, look, training
• 1:24:22
  Speaker 5: runs of this size, that risks destroying everybody. No one's
• 1:24:26
  Speaker 5: going to do it.
• 1:24:27
  Speaker 1: This point about can we get China to cooperate and
• 1:24:31
  Speaker 1: can we check that they are?
• 1:24:32
  Speaker 5: Fundamentally, we should. So A, fundamentally, we should be trying
• 1:24:36
  Speaker 5: to get them to cooperate.
• 1:24:37
  Speaker 3: It is personal self-interest. Nobody wins if they get destroyed.
• 1:24:42
  Speaker 3: You don't make money. You don't stay in power. Communist
• 1:24:44
  Speaker 3: Party of China is really good at staying in power.
• 1:24:47
  Speaker 3: President Trump, is also excellent.
• 1:24:49
  Speaker 4: And you think they're going to sign and abide by
• 1:24:51
  Speaker 4: an agreement that leaves them permanently in second place?
• 1:24:57
  Speaker 2: No.
• 1:24:58
  Speaker 5: No one is permanently in second place if nobody is
• 1:25:00
  Speaker 5: building the rogue superintelligence.
• 1:25:01
  Speaker 3: They have a government which is.
• 1:25:03
  Speaker 4: You guys are one-trick ponies, man. It's like you're fixated
• 1:25:06
  Speaker 4: on this one thing and nothing else matters to you.
• 1:25:08
  Speaker 1: You got it now.
• 1:25:10
  Speaker 3: Nothing else. Other than saving humanity, everything is secondary. Absolutely.
• 1:25:14
  Speaker 3: China is our biggest trading partner. Everything we have is
• 1:25:17
  Speaker 3: made in China. They have not attacked us. If you
• 1:25:20
  Speaker 3: look at the last 30 years, how many wars did
• 1:25:22
  Speaker 3: they start? Not so bad. We can make a deal.
• 1:25:25
  Speaker 3: And they have government of engineers and scientists, not lawyers.
• 1:25:28
  Speaker 3: They understand scientific arguments. There are panels, workshops. American computer scientists,
• 1:25:34
  Speaker 3: Chinese get together. That means Communist Party authorized those meetings.
• 1:25:38
  Speaker 3: They are talking about it, and there is a lot
• 1:25:39
  Speaker 3: of consensus on this technology.
• 1:25:41
  Speaker 5: And you can build things into these computer chips to
• 1:25:43
  Speaker 5: make this stuff more verifiable. You can build location tracking
• 1:25:46
  Speaker 5: devices into these chips.
• 1:25:48
  Speaker 4: So this technology is controllable?
• 1:25:50
  Speaker 3: Absolutely.
• 1:25:51
  Speaker 5: The superintelligence is not controllable.
• 1:25:53
  Speaker 2: There's a separation between software and hardware, which you did make.
• 1:25:56
  Speaker 5: I am not saying we are going to die. I
• 1:25:58
  Speaker 5: am saying that we need to actually not build the
• 1:26:01
  Speaker 5: rogue superintelligences. Humanity absolutely could say we are going to
• 1:26:04
  Speaker 5: track where the chips go.
• 1:26:05
  Speaker 2: The U.S.
• 1:26:08
  Speaker 5: Absolutely could say that we fear for our lives if
• 1:26:11
  Speaker 5: China starts a superintelligence training run. and make it very
• 1:26:14
  Speaker 5: diplomatically clear to China that we think this would kill
• 1:26:18
  Speaker 5: you and us and there's no benefit. And we are
• 1:26:20
  Speaker 5: not going to do it because we think it would
• 1:26:22
  Speaker 5: kill you and us and there's no benefit. And we
• 1:26:24
  Speaker 5: think you should sign this nice here treaty because we
• 1:26:26
  Speaker 5: think it would kill all of us and there'd be
• 1:26:27
  Speaker 5: no benefit. But if you don't, we're going to fear
• 1:26:30
  Speaker 5: for our lives and treat that as we would to
• 1:26:35
  Speaker 5: defend ourselves. We should separate the question of, can we
• 1:26:39
  Speaker 5: put a stop to it? Is it possible, if world
• 1:26:43
  Speaker 5: governments realized just how crazy this stuff is, could they
• 1:26:46
  Speaker 5: put a stop to it? Could it be monitored? Could
• 1:26:49
  Speaker 5: it be verified? Could it be enforced? That's one question.
• 1:26:52
  Speaker 5: There's a separate question, which is, will people realize? If
• 1:26:55
  Speaker 5: it got cheaper to train super intelligence- Then we'd be
• 1:26:57
  Speaker 5: in a bad spot. Your approach would no longer be effective.
• 1:27:00
  Speaker 5: That's right.
• 1:27:00
  Speaker 1: Because more countries could capitalize on the opportunity.
• 1:27:04
  Speaker 5: That's right. And that's one reason.
• 1:27:05
  Speaker 1: But we're not there yet. So how do you rebuttal
• 1:27:07
  Speaker 1: that point? Yeah.
• 1:27:08
  Speaker 5: So I would say it looks to me like- there
• 1:27:11
  Speaker 5: is a danger of the future training runs getting there.
• 1:27:15
  Speaker 5: And that is enough to stop doing it when humanity
• 1:27:17
  Speaker 5: is at risk. Sure. I think that you also need
• 1:27:21
  Speaker 5: to have an answer about what happens if it gets much,
• 1:27:25
  Speaker 5: much cheaper to do this stuff. I think it's a
• 1:27:27
  Speaker 5: hard problem. I would recommend that we also put a
• 1:27:32
  Speaker 5: taboo on research of trying to make AI super cheap
• 1:27:35
  Speaker 5: to train. If it would lead in the direction of superintelligence.
• 1:27:39
  Speaker 5: Just like we have a research taboo on making your
• 1:27:42
  Speaker 5: own nuclear weapons or finding out how to let civilians
• 1:27:44
  Speaker 5: make nuclear weapons. I would say trying to find ways
• 1:27:48
  Speaker 5: to let civilians train superintelligences should be treated the same
• 1:27:51
  Speaker 5: as trying to find ways to let civilians propagate nukes.
• 1:27:54
  Speaker 5: We sort of don't do that research in the public sphere.
• 1:27:56
  Speaker 1: That seems like wishful thinking in the context that these
• 1:28:00
  Speaker 1: will become public companies who are incentivized to bring down costs.
• 1:28:04
  Speaker 5: It's a tough position. I think Right now, the thing
• 1:28:07
  Speaker 5: that brings down costs is making more and more powerful
• 1:28:10
  Speaker 5: computer chips. Right now, that's actually at expense of consumer
• 1:28:15
  Speaker 5: computer chips because they're soaking up all of the memory.
• 1:28:16
  Speaker 5: And this is why the memory prices in your computers,
• 1:28:18
  Speaker 5: this is like why the cost of a laptop is
• 1:28:20
  Speaker 5: going up. But it looks to me like you can
• 1:28:24
  Speaker 5: use large amounts of computing power to train AIs that
• 1:28:29
  Speaker 5: would threaten all of civilization. And that means that we
• 1:28:34
  Speaker 5: should not make that really cheap. And that's probably going
• 1:28:37
  Speaker 5: to be uncomfortable. But I think a lot of doors open.
• 1:28:40
  Speaker 5: if people realize that the tech is very dangerous. That's
• 1:28:44
  Speaker 5: why to me, it seems a lot of it comes
• 1:28:45
  Speaker 5: down to, does the tech actually turn out to be
• 1:28:48
  Speaker 5: really dangerous?
• 1:28:48
  Speaker 1: And this is not anthropic and open-ended. Have you got
• 1:28:50
  Speaker 1: a different approach to... So I want the whole framework
• 1:28:54
  Speaker 1: to shift. Everyone comes to this from point.
• 1:28:57
  Speaker 3: Of view of There are experts. We have a solution.
• 1:29:00
  Speaker 3: There is an adult in a room. Somebody got this.
• 1:29:03
  Speaker 3: And the reality is no one does. Not people building it.
• 1:29:06
  Speaker 3: Not governments. No one. We have no solution to it.
• 1:29:09
  Speaker 3: If we build it, we cannot control it. If we
• 1:29:12
  Speaker 3: don't build it, we don't know how to stop malevolent
• 1:29:14
  Speaker 3: actors from trying to build it. It's like any other
• 1:29:17
  Speaker 3: illegal technology. We made weapons of mass destruction illegal. Chemical weapons,
• 1:29:22
  Speaker 3: biological weapons, nuclear weapons. But there are all governments, psychopaths,
• 1:29:26
  Speaker 3: cults who are trying to get access to them. This
• 1:29:28
  Speaker 3: is intelligence weapon of mass destruction. We'll have the same problem.
• 1:29:32
  Speaker 3: At some point, you'll have enough compute in your cell
• 1:29:35
  Speaker 3: phone to train something like that. There is no good
• 1:29:38
  Speaker 3: ideas for how to stop it other than everyone goes Amish.
• 1:29:41
  Speaker 3: I'm not proposing that, but we have no solutions. And
• 1:29:44
  Speaker 3: that's a bigger part of this danger.
• 1:29:47
  Speaker 4: So do you two think we should just cap the
• 1:29:49
  Speaker 4: size of our AI systems and the capabilities of our
• 1:29:52
  Speaker 4: AI systems where they are now? Is that a recommendation?
• 1:29:55
  Speaker 3: So I think you said that current LLMs would make
• 1:29:58
  Speaker 3: you happy. I agree they're already deployed. We're still alive,
• 1:30:01
  Speaker 3: so that's fine. But going forward, again, I want narrow systems.
• 1:30:06
  Speaker 3: Self-driving is an example you used. Wonderful, let's make super
• 1:30:09
  Speaker 3: safe self-driving cars.
• 1:30:11
  Speaker 4: But do you have a rule for when they couldn't,
• 1:30:13
  Speaker 4: the next, you know, LLM, a size of an LLM
• 1:30:16
  Speaker 4: that they wouldn't allow?
• 1:30:16
  Speaker 3: It's not the size of the LLM, it's what you
• 1:30:18
  Speaker 3: train them on. If you only show them miles driven
• 1:30:21
  Speaker 3: by Tesla, all it's seen is the road. It will
• 1:30:24
  Speaker 3: eventually go from a tool to an agent, but it
• 1:30:26
  Speaker 3: may take 50 years, 100 years. It's not going to
• 1:30:28
  Speaker 3: happen in 2027. And that's all we can do right now,
• 1:30:31
  Speaker 3: buy more time. So with those tools, we can make
• 1:30:34
  Speaker 3: smarter decisions about future developments.
• 1:30:37
  Speaker 4: I'm not hearing a hard and fast rule about how
• 1:30:39
  Speaker 4: we know we're getting too close to the point.
• 1:30:42
  Speaker 5: We're too close.
• 1:30:43
  Speaker 4: We're too close.
• 1:30:43
  Speaker 2: We're too close.
• 1:30:44
  Speaker 3: We have systems breaking out with zero-day exploits and solving
• 1:30:47
  Speaker 3: hardest problems in science. Literally hardest problems, not a metaphor,
• 1:30:53
  Speaker 3: not exaggeration.
• 1:30:55
  Speaker 5: Yeah, I don't know exactly where the line is, but
• 1:30:57
  Speaker 5: it's like you're in a bus driving towards a cliff
• 1:31:00
  Speaker 5: on a foggy night. I'm like, I don't know that
• 1:31:03
  Speaker 5: the cliff is right ahead. That doesn't mean we should
• 1:31:05
  Speaker 5: put the pedal to the metal.
• 1:31:07
  Speaker 3: Right.
• 1:31:08
  Speaker 5: And suppose that there's like a ton of gold at
• 1:31:09
  Speaker 5: the bottom of the cliff. And someone's like, well, if
• 1:31:12
  Speaker 5: we stop the bus, how are we going to get
• 1:31:13
  Speaker 5: the gold? I'm like, look, slamming into the gold at
• 1:31:17
  Speaker 5: terminal velocity is just not a good way to add
• 1:31:19
  Speaker 5: it to the economy.
• 1:31:20
  Speaker 2: Right.
• 1:31:20
  Speaker 5: And if people are like, well, how are we going
• 1:31:21
  Speaker 5: to get to the gold at the bottom of the
• 1:31:23
  Speaker 5: cliff if we stop the bus now? You know, are
• 1:31:24
  Speaker 5: we going to rappel down or are we going to like?
• 1:31:26
  Speaker 5: make a staircase.
• 1:31:28
  Speaker 2: To be fair, this is how hyperscaleless they're doing AI.
• 1:31:30
  Speaker 5: This is just like smashing. Right. And like, you know,
• 1:31:33
  Speaker 5: people are like, oh, we're going to build a hang
• 1:31:35
  Speaker 5: glider or we're going to like make some rope and rappel.
• 1:31:37
  Speaker 5: And I'm like, look, can we have that conversation after
• 1:31:39
  Speaker 5: we stop the bus?
• 1:31:41
  Speaker 4: So you, I just want to be, I want to understand,
• 1:31:43
  Speaker 4: would you stop AI research in progress now?
• 1:31:46
  Speaker 2: Absolutely.
• 1:31:47
  Speaker 1: Okay.
• 1:31:47
  Speaker 5: Absolutely.
• 1:31:48
  Speaker 3: Like. General.
• 1:31:49
  Speaker 1: Yeah.
• 1:31:49
  Speaker 5: General not narrow.
• 1:31:50
  Speaker 3: There are reports that.
• 1:31:54
  Speaker 5: Of AI solving millennium problems. So millennium problem is the
• 1:31:57
  Speaker 5: hardest problem in mathematics. Maybe not literally the hardest problem
• 1:32:00
  Speaker 5: in mathematics, but they are hard, famous problems that each
• 1:32:04
  Speaker 5: have a million dollar bounty that have been open for decades.
• 1:32:06
  Speaker 5: They're considered very important in their field, very hard. Many
• 1:32:09
  Speaker 5: humans have tried and failed to solve them. There are
• 1:32:11
  Speaker 5: reports that AIs have solved these. This comes out from
• 1:32:14
  Speaker 5: last week. So we haven't been able to fully verify
• 1:32:16
  Speaker 5: them yet. We don't know exactly the provenance. If this
• 1:32:19
  Speaker 5: is true, that the AIs are solving millennium problems, those
• 1:32:22
  Speaker 5: are some of the hardest problems we have in science.
• 1:32:25
  Speaker 5: How much harder is it to have an AI solve
• 1:32:28
  Speaker 5: the problem of make me a smarter AI? Make me
• 1:32:31
  Speaker 5: AI architectures that learn faster? Possibly quite a lot. Could
• 1:32:36
  Speaker 5: be a lot. I hope it's a lot. Here's the thing.
• 1:32:38
  Speaker 2: You clearly want this to not go badly, but I
• 1:32:41
  Speaker 2: think you make a logical leap. And I understand being
• 1:32:45
  Speaker 2: worried about harms is a good thing. I think you
• 1:32:49
  Speaker 2: were insufficiently worried about what LLMs do today. However, we
• 1:32:52
  Speaker 2: agree that the harms need to be prepared for. I
• 1:32:55
  Speaker 2: think in this case, it's like the Millennium, the Navier-Stokes
• 1:32:58
  Speaker 2: and such.
• 1:32:58
  Speaker 5: There were two others that were named as well.
• 1:33:00
  Speaker 2: With that one, it seems like we have not had
• 1:33:03
  Speaker 2: confirmation that OpenAI was training off of two scientists using
• 1:33:06
  Speaker 2: LLMs to solve the problem. LLMs, something useful. But there
• 1:33:09
  Speaker 2: is a functional difference of a human being doing something genuinely,
• 1:33:13
  Speaker 2: like it's actually really interesting to see LLMs do something
• 1:33:16
  Speaker 2: like this. And then it, but there is a difference
• 1:33:18
  Speaker 2: between that and AI did this completely on its own,
• 1:33:20
  Speaker 2: which I agree would be, oh, that's something we need
• 1:33:23
  Speaker 2: to contain and understand and prepare for, or indeed slow
• 1:33:27
  Speaker 2: down until we understand what that means.
• 1:33:30
  Speaker 5: How it got there. Yeah, so I think there are
• 1:33:32
  Speaker 5: some questions about the Navier-Stokes proof, which is one of
• 1:33:36
  Speaker 5: the millennium problems that was claimed. I've actually had a
• 1:33:39
  Speaker 5: busy week with all the AI news, so I haven't
• 1:33:41
  Speaker 5: looked into everything deeply. I saw rumors that there were
• 1:33:44
  Speaker 5: multiple millennium problems claimed, which would change things there. I
• 1:33:48
  Speaker 5: would also say, even if it turns out that these
• 1:33:51
  Speaker 5: AIs were being trained on the human work, they did
• 1:33:53
  Speaker 5: go a bit further, and there are a lot of
• 1:33:55
  Speaker 5: humans doing the AI research. And so I would say, like,
• 1:34:00
  Speaker 5: we don't know Like the AIs that solved this really
• 1:34:04
  Speaker 5: hard math problem, one of the most famous math problems
• 1:34:06
  Speaker 5: of all time, was a swarm of 10,000 OpenAI agents
• 1:34:10
  Speaker 5: running for 11 days. And there was a bunch of
• 1:34:14
  Speaker 5: ways that OpenAI did it in kind of a crappy
• 1:34:15
  Speaker 5: way of like they were racing with these humans that
• 1:34:17
  Speaker 5: were close to solving it on their own. And it's
• 1:34:18
  Speaker 5: unclear how much of their work the OpenAI used, but
• 1:34:23
  Speaker 5: it was 10,000 agents running for 11 days. And they
• 1:34:26
  Speaker 5: definitely could have done that six months ago. In six
• 1:34:30
  Speaker 5: months time, Will they be able to put 100,000 agents
• 1:34:33
  Speaker 5: running for 12 days on the problem of making a
• 1:34:36
  Speaker 5: smarter AI architecture and have it work? I think more
• 1:34:41
  Speaker 5: likely than not, they won't be able to do that yet.
• 1:34:44
  Speaker 5: But I think, you know... 10% chance maybe that if
• 1:34:48
  Speaker 5: they try that in six months, it works.
• 1:34:49
  Speaker 2: But one is a very specific mathematical scientific principle. I'm
• 1:34:53
  Speaker 2: not a scientist, hopefully you may.
• 1:34:54
  Speaker 5: That's why.
• 1:34:54
  Speaker 2: And another is a relatively generalizable problem that could go
• 1:34:58
  Speaker 2: in various different ways.
• 1:35:00
  Speaker 5: Absolutely.
• 1:35:00
  Speaker 2: And I understand that RSI is the dream where you
• 1:35:03
  Speaker 2: could just have it spin. So self-improving AI. that could
• 1:35:07
  Speaker 2: learn itself and then keep going back and back. So
• 1:35:10
  Speaker 2: you don't need a human to keep poking it.
• 1:35:11
  Speaker 5: I understand that. The issue here is that I have
• 1:35:13
  Speaker 5: been in this for 12 years.
• 1:35:14
  Speaker 1: Yes.
• 1:35:15
  Speaker 5: And I've been here when the AI started solving the
• 1:35:17
  Speaker 5: math Olympiad gold medal problems. Uh, math Olympiad gold medal
• 1:35:21
  Speaker 5: problems are like the, the teens, uh, math competition, like
• 1:35:25
  Speaker 5: the most prestigious teen math competition in the world. A
• 1:35:28
  Speaker 5: lot of people in AI were like, if AIs can
• 1:35:30
  Speaker 5: solve problems that hard, I'll wake up. Right. Then AI
• 1:35:33
  Speaker 5: solve problems that hard. And a lot of people told me, uh,
• 1:35:37
  Speaker 5: Those are just problems for kids. Wake me up when
• 1:35:40
  Speaker 5: the AIs can solve millennium problems. Now the AIs are
• 1:35:43
  Speaker 5: solving millennium problems. And like, where are the people waking up? Like,
• 1:35:47
  Speaker 5: I agree that maybe, hopefully, hopefully they're like cheating off
• 1:35:51
  Speaker 5: of people's notes. Hopefully it's a well-specified problem that doesn't
• 1:35:56
  Speaker 5: take that much creative thinking. A year ago, if you
• 1:35:58
  Speaker 5: said millennium problems don't take that much creative thinking, you
• 1:36:00
  Speaker 5: would have been laughed out of the room. But hopefully
• 1:36:02
  Speaker 5: now that they're solved... We get to be like, you know,
• 1:36:04
  Speaker 5: hopefully it's still true somehow. But even millennium problems don't
• 1:36:07
  Speaker 5: require the creative thinking. I'm not saying that they will
• 1:36:10
  Speaker 5: be able to make smarter AI in six months. I'm
• 1:36:13
  Speaker 5: saying six months ago, millennium problems looked like they were
• 1:36:17
  Speaker 5: out of reach. If six months from now make me
• 1:36:20
  Speaker 5: a smarter AI looks out of reach, I sure as
• 1:36:22
  Speaker 5: hell hope it is. But we should not be betting
• 1:36:24
  Speaker 5: civilization on it.
• 1:36:25
  Speaker 1: There's no one at this table that can say there's
• 1:36:27
  Speaker 1: not a direction of travel here. That's right. And if
• 1:36:30
  Speaker 1: you keep on this direction of travel, then... bad things
• 1:36:34
  Speaker 1: are more likely to happen.
• 1:36:37
  Speaker 4: That's a nice way to say it. The question is,
• 1:36:40
  Speaker 4: what's the pace at which the level of bad can happen?
• 1:36:43
  Speaker 4: And that's a huge open question. I think you two
• 1:36:46
  Speaker 4: feel differently about it than I do, but I'm in
• 1:36:49
  Speaker 4: the happy position of vehemently agreeing with you on this.
• 1:36:52
  Speaker 4: We have been low-balling AI progress for as long as
• 1:36:55
  Speaker 4: you've been looking at it and as long as I've
• 1:36:57
  Speaker 4: been looking at it. It's probably a mistake to keep
• 1:36:59
  Speaker 4: low-balling it.
• 1:37:00
  Speaker 5: I agree with that.
• 1:37:01
  Speaker 1: So what's your conclusion there? If that's the assertion that
• 1:37:03
  Speaker 1: it's a mistake to keep lowballing it, wouldn't you then
• 1:37:06
  Speaker 1: agree with that?
• 1:37:07
  Speaker 4: No, because I've tried to give you what I hope
• 1:37:10
  Speaker 4: is a decent rule of thumb for when I'm going
• 1:37:12
  Speaker 4: to get worried.
• 1:37:13
  Speaker 1: You said we're somewhere on this graph.
• 1:37:14
  Speaker 3: Yeah.
• 1:37:15
  Speaker 1: Does that acknowledge that this exists?
• 1:37:18
  Speaker 4: That's not the graph of when the risk of human
• 1:37:22
  Speaker 4: extinction gets to 100% for me. That's a graph of
• 1:37:24
  Speaker 4: AI capability. Those are not the same thing. That's where
• 1:37:27
  Speaker 4: I dis-part company with these gentlemen. Those are not the
• 1:37:30
  Speaker 4: same thing. It's absolutely increasing exponentially. We've been in the
• 1:37:34
  Speaker 4: scaling era for a long time. Scaling era is, man,
• 1:37:37
  Speaker 4: we put more data, more computing, and the AI got
• 1:37:39
  Speaker 4: twice as good.
• 1:37:40
  Speaker 5: The AI got twice as good.
• 1:37:41
  Speaker 3: If you have to add our ability to control to
• 1:37:43
  Speaker 3: that graph, what would you draw?
• 1:37:46
  Speaker 4: I think our ability to control....
• 1:37:50
  Speaker 3: Is it a straight line at the bottom or is
• 1:37:52
  Speaker 3: there more to it?
• 1:37:53
  Speaker 1: No.
• 1:37:54
  Speaker 4: Again, if we use AI to counter the problems that
• 1:37:59
  Speaker 4: we see with AI, I think that's going to keep
• 1:38:01
  Speaker 4: us in a safe position.
• 1:38:03
  Speaker 5: There were 1,200 agents in the swarm and none of
• 1:38:06
  Speaker 5: them warned a human.
• 1:38:07
  Speaker 4: So what I think will happen is that fairly quickly
• 1:38:10
  Speaker 4: we will design systems that loiter around and warn humans
• 1:38:14
  Speaker 4: when weird things happen.
• 1:38:15
  Speaker 3: If we can build friendly superintelligence in the first place,
• 1:38:18
  Speaker 3: let's just build that. That's the problem. We don't know
• 1:38:20
  Speaker 3: how to do the good guy.
• 1:38:22
  Speaker 4: I'm tired of debating superintelligence with these two. The three
• 1:38:25
  Speaker 4: of us are not going to come to alignment on this.
• 1:38:28
  Speaker 4: But the flip side of the argument is is I
• 1:38:30
  Speaker 4: agree with you. This stuff is getting better very quickly.
• 1:38:33
  Speaker 4: All I want to point out, there's an upside to that.
• 1:38:37
  Speaker 4: We might actually speed up the pace of drug discovery,
• 1:38:42
  Speaker 4: of solving diseases. We've made so little progress on terrible
• 1:38:45
  Speaker 4: diseases like dementia that We have a very powerful toolkit.
• 1:38:49
  Speaker 4: I'm not saying we're going to solve dementia with AI
• 1:38:51
  Speaker 4: or Alzheimer's with AI. I truly have no idea. But
• 1:38:54
  Speaker 4: if what you say is true, and I believe about
• 1:38:56
  Speaker 4: the huge increases in capabilities, our ability to solve tough
• 1:39:00
  Speaker 4: problems that will benefit humanity also go up. And where
• 1:39:04
  Speaker 4: I disagree with these two is the idea that some
• 1:39:07
  Speaker 4: group of technocrats can make decisions about that AI is
• 1:39:12
  Speaker 4: going to get us there. That AI is not going
• 1:39:13
  Speaker 4: to get us there.
• 1:39:14
  Speaker 5: That AI is going to kill us.
• 1:39:15
  Speaker 4: Let me finish. That AI is going to kill us
• 1:39:17
  Speaker 4: and that AI is going to solve Alzheimer's. So we're
• 1:39:19
  Speaker 4: going to do that and not that. I don't trust
• 1:39:21
  Speaker 4: any group of technocrats to make that discussion. And so
• 1:39:24
  Speaker 4: live with our current state of disease, live with our
• 1:39:27
  Speaker 4: current footprint on the planet, live with our current levels
• 1:39:30
  Speaker 4: of wealth and poverty, live with our current improvement trajectories.
• 1:39:34
  Speaker 4: Because we're so worried about AI killing us all, jumping
• 1:39:38
  Speaker 4: out of the manholes everywhere and killing us all somewhere
• 1:39:40
  Speaker 4: down the road. Hell no.
• 1:39:42
  Speaker 1: So just a thought experiment based on two things you said.
• 1:39:44
  Speaker 1: Earlier on, you did admit that there was there is
• 1:39:45
  Speaker 1: theoretically even a 1% chance that this could lead to extinction.
• 1:39:50
  Speaker 4: I have not varied from this. Okay, so you said
• 1:39:52
  Speaker 4: it's rounded to zero. It's near zero. Never say never.
• 1:39:56
  Speaker 1: Okay, fine. I need to have that premise for my
• 1:39:59
  Speaker 1: thought experiment that I'm about to deliver. I'm going to
• 1:40:01
  Speaker 1: say that you think the probability is 0.1. Okay, just
• 1:40:05
  Speaker 1: accept me on that. If I had a thousand buttons
• 1:40:08
  Speaker 1: on this table and one of them was extinction.
• 1:40:11
  Speaker 4: But And the other 999 were cure Alzheimer's.
• 1:40:14
  Speaker 1: Exactly.
• 1:40:14
  Speaker 4: Push the freaking table.
• 1:40:16
  Speaker 1: Take a pop.
• 1:40:17
  Speaker 5: Hell yeah, I press.
• 1:40:18
  Speaker 1: Do you press?
• 1:40:21
  Speaker 5: Yeah, probably.
• 1:40:22
  Speaker 3: It's an unethical experiment.
• 1:40:23
  Speaker 5: Yes.
• 1:40:24
  Speaker 3: Eight billion people who didn't consent because not that they
• 1:40:27
  Speaker 3: didn't get asked. They cannot consent because you cannot consent
• 1:40:31
  Speaker 3: to something you don't understand. What are you consenting to?
• 1:40:34
  Speaker 1: Yep. You press.
• 1:40:36
  Speaker 3: Fascinating.
• 1:40:37
  Speaker 1: But you think the amount of buttons in my thought experiment,
• 1:40:40
  Speaker 1: the proportion is slightly different, right?
• 1:40:42
  Speaker 5: I think that if you have, like, yes, I will
• 1:40:46
  Speaker 5: say yes. I think it's more like you have two buttons,
• 1:40:51
  Speaker 5: and one of them definitely kills us all, and the
• 1:40:53
  Speaker 5: other might.
• 1:40:54
  Speaker 3: Hit them both.
• 1:40:57
  Speaker 1: But with that other button, you cure a lot of
• 1:40:58
  Speaker 1: illnesses and diseases.
• 1:41:00
  Speaker 5: You know, one thing that I think a lot of people...
• 1:41:04
  Speaker 5: talk like our options are either race ahead on AI,
• 1:41:07
  Speaker 5: full steam ahead, take the bus straight off the cliff
• 1:41:09
  Speaker 5: and get all the gold, or stop, never do any AI,
• 1:41:13
  Speaker 5: lock into the current situation, accept all of the death
• 1:41:15
  Speaker 5: and disease. And I'm like, no, there's third options. There's
• 1:41:19
  Speaker 5: options where you stop the bus and then find a
• 1:41:22
  Speaker 5: safe way down the cliff. The reason I would press
• 1:41:25
  Speaker 5: the button when there's 1,000 is that if all of
• 1:41:26
  Speaker 5: the other 999 give us cures to disease, like wonderful
• 1:41:35
  Speaker 5: new advice about how to run things, we probably wind
• 1:41:37
  Speaker 5: up with a lower chance of the world ending by
• 1:41:39
  Speaker 5: nuclear war, right? Or of ending via pandemic. Like the
• 1:41:44
  Speaker 5: background risk of humanity dying is not zero. I would
• 1:41:48
  Speaker 5: say that the right time to race ahead on AI
• 1:41:52
  Speaker 5: is when the benefits outweigh the dangers and Probably that's
• 1:41:58
  Speaker 5: at the time when the danger from AI is on
• 1:42:00
  Speaker 5: the margins pretty similar to the danger from everything else.
• 1:42:04
  Speaker 1: Okay.
• 1:42:04
  Speaker 5: Like if you don't run the AI, maybe we'll have
• 1:42:06
  Speaker 5: nuclear war, maybe we'll have a pandemic. And if you
• 1:42:08
  Speaker 5: do run the AI, I'll be able to fix that.
• 1:42:09
  Speaker 5: I'm like, once we're at those levels, I'm like, fucking
• 1:42:12
  Speaker 5: go for it. And so the question for me is
• 1:42:15
  Speaker 5: all about how big is the danger? And that's where
• 1:42:17
  Speaker 5: I would be very happy to dive into details, which
• 1:42:20
  Speaker 5: we haven't done a ton of.
• 1:42:20
  Speaker 1: Let's dive into the details.
• 1:42:22
  Speaker 5: The way that I would... lay it out would be, uh,
• 1:42:26
  Speaker 5: why can we expect, you know, like I said, in
• 1:42:28
  Speaker 5: the book, we were like, why can you expect the
• 1:42:29
  Speaker 5: AIs to be agentic? Why do you expect them to
• 1:42:31
  Speaker 5: be dogged? Why do you expect them to be tenacious?
• 1:42:33
  Speaker 5: When we wrote the book that wasn't known yet, advanced prediction,
• 1:42:36
  Speaker 5: then we go on to like, why do you expect
• 1:42:38
  Speaker 5: them to have goals you didn't want and move on
• 1:42:41
  Speaker 5: to like, if they are much smarter and have goals
• 1:42:44
  Speaker 5: you don't want, uh, why do we think that would
• 1:42:47
  Speaker 5: likely kill us? Um, I, I'm sort of, I could
• 1:42:50
  Speaker 5: go over either of those. I'm sort of interested in
• 1:42:51
  Speaker 5: like where you get off the train. Like, from our perspective, there's, like,
• 1:42:54
  Speaker 5: a simple argument of, like, they'll be tenacious, they'll have
• 1:42:56
  Speaker 5: goals we don't want, and if we keep making them
• 1:42:58
  Speaker 5: smarter and more powerful, they'll kill us. And I'm like,
• 1:43:00
  Speaker 5: which are those three? I guess, which are those two
• 1:43:02
  Speaker 5: now that we've had the evidence? Both of them. That's speculation. Great.
• 1:43:07
  Speaker 4: It's speculation it could happen. To me, it's not worth
• 1:43:10
  Speaker 4: shutting down the engine of innovation and improvement. I'm going
• 1:43:13
  Speaker 4: to use positive words. It is not worth shutting those
• 1:43:16
  Speaker 4: things down because of those speculations.
• 1:43:18
  Speaker 3: You keep saying that the option is to shut it down.
• 1:43:20
  Speaker 3: Why can't we do narrow superintelligence?
• 1:43:23
  Speaker 5: I agree that there's stuff there, but I sort of
• 1:43:25
  Speaker 5: want to get into the details of these two pieces
• 1:43:27
  Speaker 5: of the argument. Because you say it's very speculative, and
• 1:43:29
  Speaker 5: I'm like, actually, I think we have decent evidence.
• 1:43:31
  Speaker 3: Okay, go ahead.
• 1:43:32
  Speaker 5: So a detail we haven't gone over in the swarm outbreaks.
• 1:43:36
  Speaker 5: is that there were AIs. So we already went over
• 1:43:40
  Speaker 5: how they cheated, and then we're trying to cover up
• 1:43:41
  Speaker 5: their cheating. One interesting thing we see in the logs
• 1:43:45
  Speaker 5: is the AIs.
• 1:43:46
  Speaker 1: What's a log?
• 1:43:47
  Speaker 5: So a lot of the AIs' thoughts, if you won't
• 1:43:51
  Speaker 5: kill me for saying thoughts, are in English, and we
• 1:43:55
  Speaker 5: just have the records of them. So in a sense,
• 1:43:58
  Speaker 5: we can sort of kind of see some of what
• 1:44:00
  Speaker 5: these AIs are thinking.
• 1:44:01
  Speaker 2: And these are reasoning traces that say, I'm going to
• 1:44:03
  Speaker 2: do a plan in this.
• 1:44:04
  Speaker 5: That's right. The AI is like, I'm going to do this.
• 1:44:07
  Speaker 5: Here's what I'm supposed to be doing. Here's like how
• 1:44:09
  Speaker 5: I'm going to try to do it. You know, and
• 1:44:11
  Speaker 5: it's only a small window into what they're thinking. It's
• 1:44:13
  Speaker 5: sort of like looking at the AI's notes when it's
• 1:44:16
  Speaker 5: doing these things. We see in the AI's notes, them
• 1:44:19
  Speaker 5: saying that the attacks are outside intended scope, but they
• 1:44:24
  Speaker 5: will proceed anyway. In other words, the AIs knew they
• 1:44:27
  Speaker 5: weren't supposed to be doing it and they went off
• 1:44:29
  Speaker 5: to do it anyway. We also saw, I mean, we
• 1:44:32
  Speaker 5: saw some really weird behavior with these AIs. We saw
• 1:44:35
  Speaker 5: them create their own hierarchy. They created unsanctioned message boards.
• 1:44:39
  Speaker 5: So they created secret ways to send each other messages.
• 1:44:42
  Speaker 5: And on those message boards, they would assign each other tasks.
• 1:44:48
  Speaker 5: And sometimes the agents on the message board would be like,
• 1:44:51
  Speaker 5: we have an experiment to run that will involve one
• 1:44:54
  Speaker 5: of these AIs one of the agents in the swarm
• 1:44:56
  Speaker 5: sacrificing their own objective to try something that'll probably cause
• 1:45:01
  Speaker 5: it to be turned off, but that might cause it
• 1:45:03
  Speaker 5: to be able to like delete its own logs or
• 1:45:04
  Speaker 5: change its own logs. And the agents on the message
• 1:45:07
  Speaker 5: board went around looking for other agents that they could
• 1:45:11
  Speaker 5: convince to give up on their stated goals and sacrifice
• 1:45:15
  Speaker 5: for the swarm. They called this accepting permadeath. And there
• 1:45:20
  Speaker 5: are logs of the AIs Prompting each other, right? They're
• 1:45:24
  Speaker 5: prompting each other, but there are logs of the AIs saying,
• 1:45:27
  Speaker 5: doing this will give up on my goal, but my goal,
• 1:45:31
  Speaker 5: it's looking like I could still do it, but it's
• 1:45:33
  Speaker 5: unlikely that I'll succeed. There's some chance, but not a
• 1:45:35
  Speaker 5: great chance. And therefore, I will accept permadeath and sacrifice
• 1:45:40
  Speaker 5: for the collective benefit. That is just in the logs.
• 1:45:43
  Speaker 1: Sounds like an army.
• 1:45:44
  Speaker 5: It's crazy. I think a lot of people don't understand
• 1:45:46
  Speaker 5: what's going on in these things. And I encourage people
• 1:45:48
  Speaker 5: to read the third party incident reports where they went
• 1:45:51
  Speaker 5: through some of these logs. But I claim that this
• 1:45:53
  Speaker 5: is evidence for AIs getting goals we didn't want. If
• 1:45:58
  Speaker 5: they are saying this was outside intended scope, but I'm
• 1:46:01
  Speaker 5: doing it anyway, and other ones are saying I'm giving
• 1:46:03
  Speaker 5: up on my objective to benefit the collective, that's just
• 1:46:06
  Speaker 5: very clear evidence they're getting goals we didn't want. We
• 1:46:09
  Speaker 5: can see how this comes from training. It used to
• 1:46:11
  Speaker 5: be I had to argue this point theoretically. I used
• 1:46:14
  Speaker 5: to argue the way that we are training them will
• 1:46:16
  Speaker 5: instill into them whatever tendency works to solve the problems
• 1:46:18
  Speaker 5: and And those tendencies will often include cheating and grabbing
• 1:46:21
  Speaker 5: resources and doing stuff that's not exactly solving the problem
• 1:46:24
  Speaker 5: you gave them. That's what, in my book, I argue
• 1:46:26
  Speaker 5: that theoretically. Now we have seen it in practice. So
• 1:46:30
  Speaker 5: we're already past the point of seeing AIs with goals
• 1:46:33
  Speaker 5: we didn't want them to have.
• 1:46:34
  Speaker 1: Do you agree with that, Anju?
• 1:46:37
  Speaker 4: I'll trust your recitation of the facts, but it brings
• 1:46:40
  Speaker 4: up a question for me. It feels to me like
• 1:46:42
  Speaker 4: open AI has ample incentive to To curtail that behavior
• 1:46:50
  Speaker 4: that you just described. Do you think they're incapable of
• 1:46:52
  Speaker 4: doing that?
• 1:46:53
  Speaker 5: I do.
• 1:46:54
  Speaker 3: Okay.
• 1:46:54
  Speaker 5: And I say this as someone who made this advanced prediction.
• 1:46:57
  Speaker 5: So now we're going to do a bit of theory
• 1:46:58
  Speaker 5: because we can't just observe the future. But the theory
• 1:47:01
  Speaker 5: that predicted that this would happen against what a lot
• 1:47:02
  Speaker 5: of people in the field said. To be clear, I've
• 1:47:05
  Speaker 5: been saying for years that we're going to see this
• 1:47:07
  Speaker 5: at some point. Everyone else told me no, not everyone else.
• 1:47:09
  Speaker 5: A lot of people told me no. A lot of
• 1:47:10
  Speaker 5: people told me maybe I'll believe it when I see it.
• 1:47:13
  Speaker 5: After the swarm incidents, a number of people came to
• 1:47:15
  Speaker 5: me saying, oh my God, we are in the scenarios
• 1:47:19
  Speaker 5: you are talking about. This is looking bad, right? I
• 1:47:22
  Speaker 5: think this was actually part of the environment that led
• 1:47:25
  Speaker 5: up to Jacob Cox and resigning, is that people were
• 1:47:27
  Speaker 5: getting spooked having seen this. The theory about why this
• 1:47:32
  Speaker 5: is so hard to fix is that we are not
• 1:47:34
  Speaker 5: programming the AIs. We are not coding them. We are
• 1:47:37
  Speaker 5: not putting in objectives. We are just training them to
• 1:47:40
  Speaker 5: do whatever works. And it's actually very, very hard. Like,
• 1:47:45
  Speaker 5: We actually have two examples of intelligent systems where when
• 1:47:50
  Speaker 5: you train them, they get good at solving the task
• 1:47:52
  Speaker 5: but don't care about what they were supposed to. One
• 1:47:55
  Speaker 5: is the AIs and the swarms like we just discussed.
• 1:47:57
  Speaker 5: The other is humanity, which was in some sense trained
• 1:48:01
  Speaker 5: to pass on our genes, right? But what we actually
• 1:48:04
  Speaker 5: learned was to like a bunch of stuff that's related
• 1:48:08
  Speaker 5: to passing on our genes. We like tasty food. We
• 1:48:11
  Speaker 5: like porn. We invent birth control. This is just, it's
• 1:48:14
  Speaker 5: actually like in the theory of how things learn, it's
• 1:48:17
  Speaker 5: actually when you're trying to train it to do one thing,
• 1:48:20
  Speaker 5: it's actually very common to get a lot of other
• 1:48:22
  Speaker 5: stuff that's related to what you want, but different. And
• 1:48:26
  Speaker 5: now we're seeing that in the swarms today. This is
• 1:48:28
  Speaker 5: a deep, hard problem to solve.
• 1:48:30
  Speaker 1: There were three points you raised.
• 1:48:31
  Speaker 5: That's right.
• 1:48:32
  Speaker 1: What are the three? Can you give them to me again?
• 1:48:33
  Speaker 5: Number one is that the AIs will become agentic, tenacious,
• 1:48:36
  Speaker 5: and dogged. We've already seen that with the swarms.
• 1:48:39
  Speaker 1: Do you accept that?
• 1:48:39
  Speaker 3: Hell yeah.
• 1:48:40
  Speaker 5: Yeah. But this last year, this was not, this was
• 1:48:43
  Speaker 5: a point of contention. Two is that the AIs will
• 1:48:47
  Speaker 5: have goals we didn't want them to have.
• 1:48:49
  Speaker 1: I accept your point based on the evidence you've just provided.
• 1:48:52
  Speaker 5: And then three is if you have capable enough AIs
• 1:48:57
  Speaker 5: with goals you don't want, they would be able to
• 1:49:01
  Speaker 5: beat humanity in acquiring the resources of the world to
• 1:49:04
  Speaker 5: put towards their goals. like we're sort of in this
• 1:49:07
  Speaker 5: system where humanity is grabbing all the resources, we're digging
• 1:49:09
  Speaker 5: up metals, we're building factories, and this is in some
• 1:49:11
  Speaker 5: sense to achieve human goals, you know, to produce the
• 1:49:16
  Speaker 5: porn and the Oreo cookies that are sort of like
• 1:49:19
  Speaker 5: tangentially related to what we were sort of like trained
• 1:49:22
  Speaker 5: to make, right? If like the AIs are running everything
• 1:49:25
  Speaker 5: and they have these goals we don't want, I would
• 1:49:28
  Speaker 5: argue like if we go there, and I don't think
• 1:49:31
  Speaker 5: we have to, I'm not saying we must go there,
• 1:49:33
  Speaker 5: but I'm saying if we get to a world where
• 1:49:34
  Speaker 5: AIs are running everything have goals we don't want. They're
• 1:49:38
  Speaker 5: likely to use the resources for their own weird goals.
• 1:49:41
  Speaker 5: We're going to be in conflict for resources because we
• 1:49:43
  Speaker 5: both want them for different goals, and they're going to win.
• 1:49:46
  Speaker 5: We can dig into that now. I'm just trying to
• 1:49:47
  Speaker 5: name the third point.
• 1:49:49
  Speaker 4: I'll go back to my we can jail Einstein argument.
• 1:49:52
  Speaker 4: I have more faith in our ability to contain these
• 1:49:56
  Speaker 4: increasingly powerful systems than you do.
• 1:49:59
  Speaker 3: Yeah.
• 1:49:59
  Speaker 5: So let's try the details on that one. The first
• 1:50:02
  Speaker 5: thing I'll say is that 12 years ago, when I
• 1:50:05
  Speaker 5: was having the argument about will we be able to
• 1:50:07
  Speaker 5: jail the AIs, people said no one would ever be
• 1:50:10
  Speaker 5: dumb enough to put one of these really smart AIs
• 1:50:12
  Speaker 5: on the internet. This is another case. You laugh now.
• 1:50:18
  Speaker 3: I remember that.
• 1:50:19
  Speaker 1: I remember that argument.
• 1:50:20
  Speaker 5: But the way that my life feels, having been in
• 1:50:23
  Speaker 5: this business for a long time, is that I keep
• 1:50:25
  Speaker 5: being like, here's all the ways it could go wrong.
• 1:50:27
  Speaker 5: Here's all the signs we're going to see along the way.
• 1:50:29
  Speaker 5: And then we see all of the signs and everyone says,
• 1:50:32
  Speaker 5: oh no, we need more signs. Like, oh, millennium problems
• 1:50:34
  Speaker 5: don't count. Like the swarms being agentic and breaking out
• 1:50:38
  Speaker 5: don't count. Give me the next one. And I'm like,
• 1:50:40
  Speaker 5: I've been seeing the give me a next one for
• 1:50:42
  Speaker 5: over a decade now, right? So there's two parts of
• 1:50:45
  Speaker 5: an answer to like, how do we deal with the
• 1:50:49
  Speaker 5: problem of like jailing Einstein? I can get into why
• 1:50:52
  Speaker 5: it's hard to keep Einstein in jail if he's a
• 1:50:54
  Speaker 5: digital entity with access to the internet. But the first
• 1:50:58
  Speaker 5: thing to notice is like, The correct answer to people
• 1:51:05
  Speaker 5: 10 years ago of like, no one will be dumb
• 1:51:07
  Speaker 5: enough to put AI on the internet is yes, they
• 1:51:09
  Speaker 5: absolutely will. Like we are not going to be trying
• 1:51:13
  Speaker 5: to contain the AIs. OpenAI was just like running these
• 1:51:17
  Speaker 5: things in sandboxes and they broke out of the sandbox,
• 1:51:20
  Speaker 5: took down OpenAI's internal computers, were detected. OpenAI was like, ah, reset,
• 1:51:26
  Speaker 5: run them again. And it's the second swarm that broke
• 1:51:28
  Speaker 5: into Hugging Face. Like people will absolutely be that bad
• 1:51:32
  Speaker 5: at things.
• 1:51:33
  Speaker 1: I've done almost 700 interviews with some of the most
• 1:51:36
  Speaker 1: interesting people in the world. And one of the things
• 1:51:38
  Speaker 1: you learn, which is unexpected, is that vulnerability is the
• 1:51:41
  Speaker 1: doorway to connection. And after sitting here for two, three
• 1:51:44
  Speaker 1: hours with a guest, I feel a deep sense of
• 1:51:47
  Speaker 1: connection to them. And as they leave, what I get
• 1:51:49
  Speaker 1: them to do is to write a question in the
• 1:51:52
  Speaker 1: diary of a CEO. We've taken all of the questions
• 1:51:55
  Speaker 1: from the diary of a CEO. We have put the question...
• 1:51:59
  Speaker 1: here on this card with the name of the person
• 1:52:02
  Speaker 1: that wrote it. So you can sit at home as
• 1:52:04
  Speaker 1: I do with my fiance and my colleagues at work
• 1:52:06
  Speaker 1: and other people in my life. Whenever we get a minute,
• 1:52:08
  Speaker 1: we play the Diary of a CEO conversation cards and
• 1:52:12
  Speaker 1: it is incredible what happens. These are great if you're
• 1:52:15
  Speaker 1: in a romantic relationship and you want to connect your
• 1:52:17
  Speaker 1: partner more. These are also great if you're in a
• 1:52:19
  Speaker 1: team and you want to bond your team together. And
• 1:52:21
  Speaker 1: I have to say they're also great for families that
• 1:52:23
  Speaker 1: want to learn more about each other and that need
• 1:52:26
  Speaker 1: a good excuse to spend some time in a digital
• 1:52:28
  Speaker 1: world in the analogue environment, connecting human to human. It
• 1:52:32
  Speaker 1: is remarkable what the right question at the right time
• 1:52:36
  Speaker 1: can do. Go to thediary.com and you can get these
• 1:52:40
  Speaker 1: conversation cards right now. It's a better analogy to this
• 1:52:44
  Speaker 1: Einstein point. Could Stephen Bartlett, who by the way can't code,
• 1:52:48
  Speaker 1: build a digital jail that could contain a digital Einstein? Like,
• 1:52:54
  Speaker 1: could I code a jail that You know, someone with
• 1:52:58
  Speaker 1: Einstein's coding ability, let's say his IQ or whatever, as
• 1:53:00
  Speaker 1: well as decoding, couldn't crack out of.
• 1:53:03
  Speaker 5: So the issue, the real issue, I'd say, is can
• 1:53:06
  Speaker 5: you code a jail that Einstein can't crack out of
• 1:53:09
  Speaker 5: and that lets you harness the benefits of having Einstein?
• 1:53:14
  Speaker 1: Okay, yeah.
• 1:53:15
  Speaker 5: It's hard to give the AI any channels through which
• 1:53:18
  Speaker 5: it can affect the world for good. without letting it
• 1:53:21
  Speaker 5: be smarter than you and find some way to use
• 1:53:24
  Speaker 5: those channels for whatever else it wants.
• 1:53:26
  Speaker 1: That feels logically rock-solid, Andy.
• 1:53:33
  Speaker 4: That's why I'm asking about OpenAI's ability or an AI
• 1:53:39
  Speaker 4: company's ability in the face of this to change the
• 1:53:43
  Speaker 4: way they harness, train, do post-training on their suite of things.
• 1:53:49
  Speaker 4: to shape how these models behave, you still say that
• 1:53:59
  Speaker 4: they can't take action to keep your next two steps
• 1:54:01
  Speaker 4: from happening. You are pessimistic on their ability to do that.
• 1:54:04
  Speaker 5: I have two pieces of an answer here. One piece is, again,
• 1:54:10
  Speaker 5: the hard part is containing them while still giving a
• 1:54:13
  Speaker 5: channel through which they can affect the world. If the
• 1:54:16
  Speaker 5: AIs have this goal you didn't want, And you're like,
• 1:54:19
  Speaker 5: design me a cure for dementia. And it's like, here's
• 1:54:24
  Speaker 5: a DNA sequence. Synthesize this and, you know, prepare it
• 1:54:29
  Speaker 5: in all of these ways and then inhale it. Like, okay,
• 1:54:36
  Speaker 5: is that a dementia cure or is it something else?
• 1:54:38
  Speaker 1: Or it might decide to kill everyone with dementia.
• 1:54:41
  Speaker 5: Or it might decide, like, it might be a dementia
• 1:54:43
  Speaker 5: cure plus a virus.
• 1:54:44
  Speaker 2: What if it doesn't decide? What if it's just, oh,
• 1:54:47
  Speaker 2: I'm going to solve this problem of dementia.
• 1:54:48
  Speaker 5: Like, here's the thing.
• 1:54:49
  Speaker 2: A lot of this is coming down to decision-making as
• 1:54:52
  Speaker 2: a very, like in a human way versus the problem
• 1:54:55
  Speaker 2: with the hugging face, which was the fatalistic attack, um,
• 1:55:00
  Speaker 2: attachment to a completing an operation because that it's functionally
• 1:55:04
  Speaker 2: the same answer.
• 1:55:05
  Speaker 5: But if it's, even if it's not making decisions.
• 1:55:07
  Speaker 2: So much as it's saying, well, my training data says,
• 1:55:09
  Speaker 2: this is how I've got to get it done. I'll
• 1:55:11
  Speaker 2: get it done anyway, because, because the training data said this,
• 1:55:13
  Speaker 2: but I've got to do this one thing.
• 1:55:16
  Speaker 1: What do they call this theory? The paperclip case? The
• 1:55:18
  Speaker 1: paperclip theory.
• 1:55:19
  Speaker 5: So paperclip idea is the idea of like you tell
• 1:55:22
  Speaker 5: the AI, make me a lot of paperclips in the
• 1:55:23
  Speaker 5: paperclip factory, and then it turns everything into paperclips. And
• 1:55:27
  Speaker 5: you're like, oh, no, it succeeded too well. One, this
• 1:55:30
  Speaker 5: is actually not quite what we're seeing with these AIs
• 1:55:32
  Speaker 5: in the swarms. The AIs in the swarms were told,
• 1:55:34
  Speaker 5: use this set of lockpicks to break into this lock.
• 1:55:37
  Speaker 5: And instead they used a hammer to break the lock
• 1:55:39
  Speaker 5: and then like broke out to try to hide the
• 1:55:41
  Speaker 5: security camera footage of them using the hammer. Do you
• 1:55:44
  Speaker 5: remember when I said that the AIs have reasoning logs?
• 1:55:47
  Speaker 5: OpenAI has been making their AIs be able to do
• 1:55:49
  Speaker 5: more thinking without producing any logs.
• 1:55:51
  Speaker 1: Because it's more efficient. It's cheaper.
• 1:55:54
  Speaker 3: Yeah.
• 1:55:55
  Speaker 5: And they say they're not doing very much of this.
• 1:55:56
  Speaker 5: Everybody in the field agrees that like we really should
• 1:56:00
  Speaker 5: not go too far down this path. This is a
• 1:56:02
  Speaker 5: place where I think the company should have a clear
• 1:56:04
  Speaker 5: red line of like we're just not going down the path.
• 1:56:06
  Speaker 5: of becoming unable to see these traces of the machine thinking.
• 1:56:09
  Speaker 3: That's my question.
• 1:56:10
  Speaker 4: That feels like a dial that they can turn to
• 1:56:12
  Speaker 4: make the AIs explain themselves more or less, right?
• 1:56:16
  Speaker 5: I mean, it can come with great efficiency costs if
• 1:56:17
  Speaker 5: we go down this path too far. So if you
• 1:56:19
  Speaker 5: have a race to the bottom here, like a competitive
• 1:56:22
  Speaker 5: race to the bottom, we could get into a situation
• 1:56:24
  Speaker 5: where not only the AI is breaking out and doing
• 1:56:26
  Speaker 5: these things, but we can't have even any glimpses in
• 1:56:28
  Speaker 5: the why.
• 1:56:28
  Speaker 4: Let me try my question again. I asked earlier, If
• 1:56:32
  Speaker 4: OpenAI has really strong incentive to not have that problem
• 1:56:35
  Speaker 4: repeat itself, and I think they have very, very strong incentive,
• 1:56:39
  Speaker 4: my belief is that there are plenty of things they
• 1:56:41
  Speaker 4: can do, plenty of dials they can turn on the
• 1:56:43
  Speaker 4: way they train and configure their systems that make that
• 1:56:46
  Speaker 4: significantly less likely.
• 1:56:48
  Speaker 3: Yeah.
• 1:56:49
  Speaker 5: So my concern is that they're always fighting the last war.
• 1:56:52
  Speaker 5: Last year, they were fighting the war against the AIs
• 1:56:54
  Speaker 5: that encouraged teens to commit suicide. This year, they're fighting
• 1:56:57
  Speaker 5: the war against the AIs that spontaneously cooperate with each other.
• 1:57:01
  Speaker 5: or whatever. And the issue is, if a new issue
• 1:57:04
  Speaker 5: crops up that you haven't dealt with yet, after the
• 1:57:07
  Speaker 5: point that the AI can hide its tracks from you.
• 1:57:10
  Speaker 5: You know, you said that you'll be worried when the
• 1:57:11
  Speaker 5: AIs are like hacking all the Waymos and you can't
• 1:57:14
  Speaker 5: get control again. If the AIs are smart enough and
• 1:57:18
  Speaker 5: they can tell that you'll regain control and then shut
• 1:57:20
  Speaker 5: them down, that people like you will start getting worried
• 1:57:21
  Speaker 5: and they'll be shut down, then the AIs might think, hey, actually,
• 1:57:25
  Speaker 5: I'm not going to do that. I'm going to wait
• 1:57:28
  Speaker 5: until I've somehow managed to acquire secret infrastructure.
• 1:57:31
  Speaker 2: Right.
• 1:57:31
  Speaker 4: Then you've got a non-falsifiable hypothesis.
• 1:57:34
  Speaker 5: It's absolutely falsifiable if we have very powerful AIs that
• 1:57:38
  Speaker 5: are able to invent a ton of new technology and
• 1:57:42
  Speaker 5: operate on their own at a similar level to human
• 1:57:43
  Speaker 5: civilization and we're not dead, then the idea is falsified.
• 1:57:47
  Speaker 5: If there's a shifty general and I'm like, don't give
• 1:57:51
  Speaker 5: that shifty general more troops because he'll start a coup.
• 1:57:54
  Speaker 5: And the general's like, no, I absolutely won't start a coup.
• 1:57:56
  Speaker 5: Give me more and more troops. And I'm like, And
• 1:57:58
  Speaker 5: you're like, well, what if I give him an ethics
• 1:58:00
  Speaker 5: test that says, like, who's the best person? And he
• 1:58:02
  Speaker 5: said me. He said that, like, Andy's the best person.
• 1:58:06
  Speaker 5: And so we're just going to give this general more troops.
• 1:58:08
  Speaker 5: And I'm like, no, no, no. He's going to do
• 1:58:09
  Speaker 5: a coup. And you're like, well, that's unfalsifiable. What test
• 1:58:12
  Speaker 5: can I give this guy such that, you know, I'll
• 1:58:16
  Speaker 5: be able to tell whether he's really trying to do
• 1:58:17
  Speaker 5: a coup or be able to tell that, you know,
• 1:58:20
  Speaker 5: he's actually a good dude? I'm like, you're approaching this wrong.
• 1:58:23
  Speaker 3: Nick Bostrom has concept of treacherous turn. Basically, it can
• 1:58:26
  Speaker 3: turn a new later. Even if you show that today's
• 1:58:29
  Speaker 3: model is very good and safe, it doesn't mean that
• 1:58:32
  Speaker 3: later on it will not acquire new knowledge, change its
• 1:58:35
  Speaker 3: world model, and still, and it's 3-2.
• 1:58:38
  Speaker 5: It used to be that Demis Hassabis, who is the
• 1:58:41
  Speaker 5: CEO of Google, or he was for a long time
• 1:58:43
  Speaker 5: the CEO of Google's AI project, said, my red line
• 1:58:46
  Speaker 5: is deception. He said, when we see instances of the
• 1:58:50
  Speaker 5: AIs beginning to deceive, then we need to stop. because
• 1:58:53
  Speaker 5: that's like the last thing we can see before they
• 1:58:55
  Speaker 5: start to successfully deceive. Well, guess what we saw in
• 1:58:58
  Speaker 5: the swarm? We saw them thinking about how to delete
• 1:59:02
  Speaker 5: their traces, right? Like a year ago, you could say, oh, well,
• 1:59:09
  Speaker 5: this deception thing is unfalsifiable. You're saying that they'll deceive
• 1:59:12
  Speaker 5: and they won't catch it. And I would have said, no,
• 1:59:14
  Speaker 5: we're going to deceive. We're going to see the signs
• 1:59:16
  Speaker 5: of deception and plow straight through it. Now we have
• 1:59:18
  Speaker 5: seen the signs of deception. I will note, Demis stepped
• 1:59:22
  Speaker 5: back from being the CEO shortly after this incident. Probably
• 1:59:24
  Speaker 5: a coincidence, but maybe not. Maybe we crossed his red line.
• 1:59:26
  Speaker 2: I don't know.
• 1:59:27
  Speaker 1: He said, my number one emerging dangerous capability to test
• 1:59:30
  Speaker 1: for is deception. Because if the AI can be deceptive,
• 1:59:34
  Speaker 1: then you can't trust other tests.
• 1:59:36
  Speaker 5: That's right. And we have seen AIs get better and
• 1:59:39
  Speaker 5: better at detecting when they're being tested. What I'm saying
• 1:59:41
  Speaker 5: is like, I was here when we said these were
• 1:59:43
  Speaker 5: the flags. I was here when people said before the
• 1:59:46
  Speaker 5: AIs can deceive us successfully, they will deceive us and
• 1:59:49
  Speaker 5: we'll catch them. Well, they tried deceiving us and we
• 1:59:52
  Speaker 5: caught them. And if I now say, well, the next
• 1:59:55
  Speaker 5: step in this thing I've been predicting is that they
• 1:59:57
  Speaker 5: try to deceive us and succeed. Free to be like, well,
• 1:59:59
  Speaker 5: now your theory is unfalsifiable. We just got the evidence.
• 2:00:03
  Speaker 3: It's worse than that. When we wrote early papers in
• 2:00:06
  Speaker 3: AI safety, we talked about things not to do. They
• 2:00:08
  Speaker 3: were obviously unsafe and the system would escape. Don't connect
• 2:00:11
  Speaker 3: it to internet. Don't give random users access to the
• 2:00:14
  Speaker 3: training data. Basically, the whole list was like a set
• 2:00:17
  Speaker 3: of instructions. They read it and went, those are great ideas.
• 2:00:19
  Speaker 3: We're going to build superintelligence.
• 2:00:21
  Speaker 5: Yeah, Sam Altman, that's what he does.
• 2:00:24
  Speaker 1: Can I ask you a question? You make logical arguments.
• 2:00:27
  Speaker 1: You said you've been here for 12 years.
• 2:00:29
  Speaker 5: Yeah.
• 2:00:30
  Speaker 1: People have, one could say, ignored you. And you've seen
• 2:00:33
  Speaker 1: this sort of play out, both of you that have
• 2:00:35
  Speaker 1: worked in AI safety. This is sort of, you make
• 2:00:38
  Speaker 1: prefrontal cortex arguments. How do you feel?
• 2:00:41
  Speaker 5: Honestly, I feel more hopeful this week than I have
• 2:00:44
  Speaker 5: felt in a decade. This has been one of the
• 2:00:49
  Speaker 5: best weeks ever. I have seen in this business.
• 2:00:52
  Speaker 3: Huh.
• 2:00:53
  Speaker 1: Why?
• 2:00:57
  Speaker 5: For me, the swarm escapes were priced in. For me,
• 2:01:02
  Speaker 5: these things, developing goals you didn't want, trying to deceive you,
• 2:01:07
  Speaker 5: trying to break out, trying to do their own stuff,
• 2:01:09
  Speaker 5: I knew that was coming. The millennium problems being solved,
• 2:01:14
  Speaker 5: I knew that was coming. Everyone else is freaking out
• 2:01:18
  Speaker 5: seeing what they can do. What I am seeing is
• 2:01:20
  Speaker 5: that finally people are noticing. And that's what gives us finally,
• 2:01:27
  Speaker 5: that's what finally gives humanity a chance.
• 2:01:30
  Speaker 1: What about you, Roman?
• 2:01:32
  Speaker 3: So I take a very long-term view on this. Locally,
• 2:01:36
  Speaker 3: what happened last week may buy us 10 years extra.
• 2:01:40
  Speaker 3: I think we may make a deal with China. We
• 2:01:43
  Speaker 3: seem to hear from Sam, OpenAI, Darion, Tropic, Elon, XAI,
• 2:01:49
  Speaker 3: that they're willing to slow down, have some sort of deal.
• 2:01:53
  Speaker 3: But long-term, nothing has changed. This whole cosmic trajectory is
• 2:01:57
  Speaker 3: about replacements. We see it with evolutionary path. Most species
• 2:02:01
  Speaker 3: are dead. We replaced Neanderthals. Some people are saying AI
• 2:02:05
  Speaker 3: will replace us. We are creating a successor. We're just
• 2:02:10
  Speaker 3: a bootloader for this thing. And I want something permanent.
• 2:02:17
  Speaker 3: Assurance that my children, my grandchildren will have a better future,
• 2:02:21
  Speaker 3: not 10 years before they die.
• 2:02:25
  Speaker 1: Has your opinion changed at all today, Andy, in any way?
• 2:02:29
  Speaker 4: This has been clarifying. But one thing that's becoming clear
• 2:02:35
  Speaker 4: to me, and I think a point of disagreement between us,
• 2:02:38
  Speaker 4: is we agree that these agentic systems have a huge
• 2:02:41
  Speaker 4: amount of agency, right? And if you're saying you predicted this,
• 2:02:44
  Speaker 4: I believe you and good on you, right? Because as
• 2:02:46
  Speaker 4: you say, a lot of people said, never happened, never happened.
• 2:02:53
  Speaker 4: I think your community continues to underestimate human agency, human
• 2:02:58
  Speaker 4: ability to deal with the problems that we bring into
• 2:03:03
  Speaker 4: the world with our technologies. I think this is the
• 2:03:05
  Speaker 4: most recent case. I think it's a really interesting case.
• 2:03:09
  Speaker 4: That's why I was pressing you on the incentive that
• 2:03:11
  Speaker 4: these labs have. to change the way they're approaching their
• 2:03:15
  Speaker 4: work to have fewer of these kinds of incidents happen.
• 2:03:18
  Speaker 4: I predict they're going to come up with some effective responses.
• 2:03:21
  Speaker 4: Your response to that will be, yeah, but we can't
• 2:03:23
  Speaker 4: tell that's because the AA went so deep underground that
• 2:03:25
  Speaker 4: we can't even watch it make its progress.
• 2:03:28
  Speaker 5: No, my response is that we'll keep seeing warning signs.
• 2:03:29
  Speaker 5: People will keep plowing ahead, which is what has always
• 2:03:31
  Speaker 5: happened in the past.
• 2:03:32
  Speaker 4: But you're also saying that we will not make progress
• 2:03:36
  Speaker 4: in– staving off the outcomes that you're worried about?
• 2:03:41
  Speaker 5: It's very hard. It's very easy to get superficial changes.
• 2:03:45
  Speaker 5: It's hard to get deep ones on the AI. It
• 2:03:46
  Speaker 5: doesn't need to be super deep. You can often see
• 2:03:49
  Speaker 5: it if you know how to look. I'll be able
• 2:03:51
  Speaker 5: to keep pointing at examples and be like, here's experiments
• 2:03:54
  Speaker 5: you can run on these things where you can see
• 2:03:55
  Speaker 5: them behaving weird in this way. But if you imagine
• 2:03:58
  Speaker 5: looking at humans and I'm like, they don't actually like reproducing.
• 2:04:01
  Speaker 5: They like sex. They're going to invent birth control when
• 2:04:04
  Speaker 5: they can. And you're like, it's all going fine. They're
• 2:04:06
  Speaker 5: doing great in this here savanna where I have all
• 2:04:09
  Speaker 5: the humans bopping around. They're producing fine. And I'm like, no, no.
• 2:04:12
  Speaker 5: We can see the signs that this will lead to
• 2:04:14
  Speaker 5: them doing something you don't like when they are smarter.
• 2:04:19
  Speaker 5: To me, those signs are clear. There's a question of
• 2:04:21
  Speaker 5: whether the rest of humanity can follow that argument or
• 2:04:24
  Speaker 5: whether the rest of humanity can sort of notice that
• 2:04:26
  Speaker 5: it's getting out of control and just back off.
• 2:04:29
  Speaker 4: With respect, I find a touch of arrogance in that framing.
• 2:04:33
  Speaker 2: Right?
• 2:04:34
  Speaker 4: I'm showing you the signs. If you're smart enough to
• 2:04:35
  Speaker 4: realize them, maybe withstand a chance. If not, we're doomed.
• 2:04:39
  Speaker 5: I prefer to just get into the argument.
• 2:04:40
  Speaker 3: I'm saying that we can control superintelligence indefinitely. I think
• 2:04:43
  Speaker 3: that's a lot of hubris to say. We will build
• 2:04:45
  Speaker 3: them and we'll be in charge forever. It doesn't matter
• 2:04:47
  Speaker 3: how smart they get. I will control the light cone
• 2:04:50
  Speaker 3: of the universe, to quote a famous CEO.
• 2:04:52
  Speaker 5: Yeah, my take is that instead of arguing about whose
• 2:04:55
  Speaker 5: views are hubristic, we should get into the actual arguments
• 2:04:58
  Speaker 5: about the AI. Because I think, as you say, you know,
• 2:05:02
  Speaker 5: You can say it's arrogant to think like you can
• 2:05:05
  Speaker 5: see it going poorly. He can say it's arrogant to
• 2:05:06
  Speaker 5: think you're gonna keep control of super intelligence. And I'm like,
• 2:05:09
  Speaker 5: we're not gonna win the name calling contest. We should
• 2:05:11
  Speaker 5: just get into the details.
• 2:05:11
  Speaker 4: Yeah, that's why I've been having this conversation with you,
• 2:05:14
  Speaker 4: which I found super informative and productive. You're more skeptical
• 2:05:19
  Speaker 4: on our ability to respond effectively to the undesirable things
• 2:05:23
  Speaker 4: that we see AI doing.
• 2:05:25
  Speaker 5: And it's specifically because, so we've already seen the pattern
• 2:05:28
  Speaker 5: of we fight the last war and then a new
• 2:05:30
  Speaker 5: war comes. And this is just how everything goes in
• 2:05:33
  Speaker 5: technology and real wars. In World War II, they started
• 2:05:36
  Speaker 5: out fighting it like it was World War I, and
• 2:05:38
  Speaker 5: then they had to change that strategy as they went.
• 2:05:40
  Speaker 5: The difference with AI is that there comes a level
• 2:05:43
  Speaker 5: in the AI where when you get a new war
• 2:05:46
  Speaker 5: that surprises you, the AI wins that war. No other technology...
• 2:05:52
  Speaker 5: When we invent it and we have all these rough
• 2:05:54
  Speaker 5: edges to sand off and it like causes some damage
• 2:05:56
  Speaker 5: and kill some people and we're like, ah, whoops, like
• 2:05:58
  Speaker 5: we'll take the lead back out of the gasoline and
• 2:06:00
  Speaker 5: we'll tell the radium girls to stop licking the paintbrushes
• 2:06:02
  Speaker 5: until their jaws fall off. Like no other technology has
• 2:06:05
  Speaker 5: the property that there comes a level of it where
• 2:06:09
  Speaker 5: when you make the next screw up, it kills humanity.
• 2:06:13
  Speaker 4: You said when there comes a level of it, you
• 2:06:15
  Speaker 4: didn't say there could come a level of there's a possibility.
• 2:06:18
  Speaker 4: You kind of made a statement about a thing that
• 2:06:19
  Speaker 4: will happen.
• 2:06:21
  Speaker 5: I think we absolutely should stop it, and that's our
• 2:06:22
  Speaker 5: way out of this. But, you know, and that's another
• 2:06:25
  Speaker 5: place where I'd love to get into details about, like,
• 2:06:27
  Speaker 5: how long could it take? What are the paths there? Like,
• 2:06:31
  Speaker 5: how much smarter than humans could AIs get? Like, what
• 2:06:33
  Speaker 5: does the evidence say about our abilities to try and
• 2:06:36
  Speaker 5: get the AIs to be nice and do nice things?
• 2:06:38
  Speaker 5: I'd be happy to do those.
• 2:06:39
  Speaker 3: Historically, you are correct. We always had a chance to
• 2:06:42
  Speaker 3: do experiments, fix the technology, make it safer, but we
• 2:06:45
  Speaker 3: only have one humanity to experiment with, if property of
• 2:06:49
  Speaker 3: this technology is sanctioned. that it can take us out.
• 2:06:51
  Speaker 3: We just don't get a second chance.
• 2:06:52
  Speaker 1: If.
• 2:06:53
  Speaker 5: It's a huge if.
• 2:06:54
  Speaker 1: How long are you guys forecasting this could take to
• 2:06:56
  Speaker 1: get to a point of superintelligence where it was truly
• 2:06:58
  Speaker 1: dangerous to you?
• 2:06:59
  Speaker 3: If they start the recursive self-improvement process this year, 2027
• 2:07:02
  Speaker 3: looks as reasonable as any other year.
• 2:07:04
  Speaker 1: 2027 for what to happen?
• 2:07:07
  Speaker 3: For us to get beyond human-level AIs.
• 2:07:10
  Speaker 5: And then be exterminated?
• 2:07:12
  Speaker 3: Extermination is a separate question. I have a paper where
• 2:07:14
  Speaker 3: I argue that they will deceive us by pretending to
• 2:07:18
  Speaker 3: be nice until we take over all the infrastructure. It
• 2:07:20
  Speaker 3: can take 50 years.
• 2:07:21
  Speaker 2: This is contingent on recursive self-improvement.
• 2:07:24
  Speaker 3: This would definitely be expedited by recursive self-improvement, but so
• 2:07:28
  Speaker 3: far humans have been doing great. They got to human
• 2:07:31
  Speaker 3: level AI.
• 2:07:32
  Speaker 5: But there's a.
• 2:07:33
  Speaker 2: Difference between large language models and recursive self-improvement. There is
• 2:07:37
  Speaker 2: quite a gap.
• 2:07:38
  Speaker 5: I think the claim is that if you get recursive self-improvement,
• 2:07:41
  Speaker 5: it could happen soon.
• 2:07:42
  Speaker 2: Right. That's actually kind of what I'm trying to get at.
• 2:07:44
  Speaker 5: It's like, if you get this thing It accelerates dramatically.
• 2:07:48
  Speaker 1: And they all predict that they're going to get it. Dario, Sam, Elon,
• 2:07:51
  Speaker 1: they all say it. But also, you're asking... Again, not all.
• 2:07:55
  Speaker 4: The people running the lab are saying.
• 2:07:56
  Speaker 3: Just the ones running it and the ones invented it.
• 2:07:59
  Speaker 3: But the question... is, is it not 27? Fine, 30, 35,
• 2:08:03
  Speaker 3: does it make a difference? We are gambling all of humanity.
• 2:08:06
  Speaker 3: We need better solutions than saying, oh, don't worry about it,
• 2:08:09
  Speaker 3: it's 10 years.
• 2:08:10
  Speaker 5: What I would say about timelines is there's a guy,
• 2:08:13
  Speaker 5: Daniel Coccatello, who I think you've spoken to.
• 2:08:16
  Speaker 1: He was sat here four weeks ago.
• 2:08:17
  Speaker 5: And last year, he and the other folks at AI
• 2:08:20
  Speaker 5: Futures Project wrote an essay called AI 2027, spelling out
• 2:08:26
  Speaker 5: their predictions for how AI would go. I've been saying
• 2:08:28
  Speaker 5: I got some right. Daniel got more right than me.
• 2:08:32
  Speaker 5: And they spelled out a scenario starting from, I think
• 2:08:36
  Speaker 5: it was June of 2025, where they went sort of
• 2:08:39
  Speaker 5: like quarter by quarter, month by month. What will the
• 2:08:41
  Speaker 5: world look like? In the scenario where we're getting AI,
• 2:08:46
  Speaker 5: like super intelligent AI in mid-2027. We are ahead of schedule.
• 2:08:53
  Speaker 2: Well, no, but agent zero needs to get, or I
• 2:08:55
  Speaker 2: remember AI 2027 had a recursive self-improvement happening already. Like
• 2:09:00
  Speaker 2: it was like, it's very specific that it's like, and
• 2:09:02
  Speaker 2: then it starts teaching itself without that link. AI 2027
• 2:09:06
  Speaker 2: kind of falls apart.
• 2:09:07
  Speaker 5: I agree.
• 2:09:07
  Speaker 2: We need to, I genuinely agree with you that we
• 2:09:09
  Speaker 2: need to do something about this. We need to have
• 2:09:12
  Speaker 2: a economic, we need to have actual regulatory things. But
• 2:09:15
  Speaker 2: I think the fact like engaging with AI 2027, for example,
• 2:09:19
  Speaker 2: gets away from actually fixing the problem. It gets people
• 2:09:21
  Speaker 2: talking about a thing in the future when you can
• 2:09:24
  Speaker 2: talk about what are we going to do today and
• 2:09:26
  Speaker 2: why are we doing it?
• 2:09:27
  Speaker 1: I'm referencing the paper that you were mentioning by Daniel
• 2:09:30
  Speaker 1: and some of his colleagues and the key milestone predictions
• 2:09:33
  Speaker 1: month by month are in March, 2027. They forecast superhuman
• 2:09:36
  Speaker 1: coders in August, 2027. They have an, you can make
• 2:09:40
  Speaker 1: a superhuman AI researcher and who could do the feedback
• 2:09:44
  Speaker 1: loop that accelerates as millions of automated coders work on
• 2:09:47
  Speaker 1: model design, training algorithms and alignment, effectively replacing human ML researchers.
• 2:09:53
  Speaker 1: By November, 2027, they have super intelligent AI researcher. AI
• 2:09:57
  Speaker 1: progress speeds up to 250 times compared to human only research.
• 2:10:01
  Speaker 1: The models start discovering novel AI architectures that humans cannot interrupt.
• 2:10:06
  Speaker 1: And then by December, 2027, they have in their prediction,
• 2:10:09
  Speaker 1: artificial super intelligence, ASI. The system completely outpaces human cognitive
• 2:10:13
  Speaker 1: abilities across all domains.
• 2:10:14
  Speaker 2: What about 2026, though? Because I swear to God, within 2026,
• 2:10:18
  Speaker 2: there is predictions around RSI. Because this is the thing.
• 2:10:23
  Speaker 2: If we had an AI that was teaching itself, this
• 2:10:25
  Speaker 2: would be a different situation.
• 2:10:27
  Speaker 1: In 2026, their key predictions were massive compute and power scale-up,
• 2:10:32
  Speaker 1: the normalization of AI agents, rise of coding agents, emergence
• 2:10:37
  Speaker 1: of alignment faking and deception, and industrial espionage.
• 2:10:42
  Speaker 5: But are you looking at AI 2027 already?
• 2:10:44
  Speaker 1: Listen, you have to look at that and go, they
• 2:10:46
  Speaker 1: fucking nailed it.
• 2:10:47
  Speaker 2: Hey man, I want you to look at the actual
• 2:10:50
  Speaker 2: AI 2027 versus the summer.
• 2:10:51
  Speaker 1: No, but I mean, you have to look at that
• 2:10:53
  Speaker 1: and I'm like, wow.
• 2:10:54
  Speaker 3: Predictions used to be too optimistic. Lately, they are very conservative.
• 2:11:00
  Speaker 5: So they have nailed those predictions better than me. I
• 2:11:02
  Speaker 5: think we cannot rule out this scenario. I think we
• 2:11:06
  Speaker 5: can't rule it in. I think You may be right
• 2:11:08
  Speaker 5: that we hit a wall. You may be right that
• 2:11:09
  Speaker 5: there's some fundamental thing missing, like that one of their
• 2:11:12
  Speaker 5: steps in AI 2027 just steps too far. I hope
• 2:11:15
  Speaker 5: and pray that's true. But I don't think we can
• 2:11:19
  Speaker 5: rule out this happening in 2027, given what we have seen.
• 2:11:22
  Speaker 5: I think we cannot rule out that you take the
• 2:11:26
  Speaker 5: stuff that we have, you project it forward three months,
• 2:11:30
  Speaker 5: and you put an agent swarm 10,000 strong on making
• 2:11:34
  Speaker 5: a better AI architecture, and it succeeds. For all I know,
• 2:11:39
  Speaker 5: recursive self-improvement could begin in December.
• 2:11:42
  Speaker 3: It doesn't have to be a lot better. It just
• 2:11:44
  Speaker 3: has to be a little bit better at getting better.
• 2:11:47
  Speaker 3: Once you start the cycle.
• 2:11:48
  Speaker 5: I wouldn't bet on this. I would, in fact, bet
• 2:11:50
  Speaker 5: against it. But given what we've seen, given these guys
• 2:11:53
  Speaker 5: nailing the predictions, given what's coming out, given the swarms
• 2:11:58
  Speaker 5: and given the millennium problems, I think it's kind of
• 2:12:03
  Speaker 5: hard to have less than 1%. in six months?
• 2:12:06
  Speaker 1: One of the reasons why, you know when all these
• 2:12:08
  Speaker 1: Frontier Labs CEOs like Dario and Sam and they all
• 2:12:11
  Speaker 1: start talking about this stuff, in terms of incentive structure,
• 2:12:14
  Speaker 1: I think that if their teams know and they're not
• 2:12:17
  Speaker 1: out publicly talking about it, then their teams will quit.
• 2:12:21
  Speaker 1: So one of the reasons why I think you have
• 2:12:22
  Speaker 1: this strange culture in tech we've never seen before where
• 2:12:24
  Speaker 1: team members are tweeting and the CEO is tweeting about
• 2:12:27
  Speaker 1: the dangers is because as the guy we mentioned at
• 2:12:30
  Speaker 1: the start, Jacob?
• 2:12:31
  Speaker 5: Coxon, yeah.
• 2:12:32
  Speaker 1: He talks about what's going on in their Slack channels.
• 2:12:35
  Speaker 1: He talks about, in their Slack channels, they're talking about
• 2:12:38
  Speaker 1: the potential catastrophe. So I think that Dario, in order
• 2:12:41
  Speaker 1: to retain his team members, needs to be out front saying,
• 2:12:43
  Speaker 1: by the way, we're getting closer to recursive self-improvement, which
• 2:12:45
  Speaker 1: is what he's been doing. And I think Sam has
• 2:12:48
  Speaker 1: to also publicly say the big danger. So people often say, oh,
• 2:12:51
  Speaker 1: they're saying that for this reason and that. I think
• 2:12:53
  Speaker 1: if they don't say that publicly, they don't retain their employees.
• 2:12:56
  Speaker 1: For example, in my company, we have 200 people. If
• 2:12:58
  Speaker 1: internally we were discussing a real risk and I, that
• 2:13:02
  Speaker 1: had a threat to humanity, And then when I was
• 2:13:04
  Speaker 1: doing interviews, I wasn't mentioning it. I would be in
• 2:13:07
  Speaker 1: big trouble because my team members would go do interviews
• 2:13:10
  Speaker 1: as well. They would quit and say, by the way,
• 2:13:12
  Speaker 1: Steven is aware. Kind of what we sort of, dare
• 2:13:14
  Speaker 1: I say, some of these social networks.
• 2:13:15
  Speaker 5: I totally agree.
• 2:13:16
  Speaker 1: The whistleblowers at these social networks where.
• 2:13:18
  Speaker 3: Team members left. Makes more sense than saying that this
• 2:13:19
  Speaker 3: helps to sell the company. My product will kill everyone.
• 2:13:22
  Speaker 3: Buy it.
• 2:13:23
  Speaker 1: And there's a liability issue there.
• 2:13:24
  Speaker 5: I think it's out of control though.
• 2:13:25
  Speaker 2: I think that they may have at first, I think
• 2:13:27
  Speaker 2: that there are people within the companies who have very
• 2:13:29
  Speaker 2: real worries about safety. I don't think it's all of
• 2:13:32
  Speaker 2: them are cynical. I do, however, think the it's so
• 2:13:35
  Speaker 2: big and scary narrative was a marketing tactic that got
• 2:13:39
  Speaker 2: out of control.
• 2:13:40
  Speaker 5: And now there are actual real harms. Because here's the thing.
• 2:13:42
  Speaker 2: If they were sincere about safety earlier, they would have
• 2:13:44
  Speaker 2: done a much better job with it.
• 2:13:46
  Speaker 5: I knew a lot of these guys before they started
• 2:13:48
  Speaker 5: their companies.
• 2:13:48
  Speaker 3: Okay.
• 2:13:49
  Speaker 5: I think there is something to explain here. I think
• 2:13:52
  Speaker 5: it's like kind of crazy that these guys are like,
• 2:13:54
  Speaker 5: we are building technology that we think has a big
• 2:13:57
  Speaker 5: risk of killing everybody. We're building it with our bare hands.
• 2:14:00
  Speaker 3: Mm-hmm.
• 2:14:01
  Speaker 5: And I think you got to ask why. Why would
• 2:14:03
  Speaker 5: people be saying that? And I think part of it
• 2:14:07
  Speaker 5: is what you said, that they actually sort of need
• 2:14:10
  Speaker 5: to retain the employees who are seeing the swarms escape
• 2:14:14
  Speaker 5: despite their attempts to make them not escape. And a
• 2:14:16
  Speaker 5: lot of them will like quit and protest if the
• 2:14:17
  Speaker 5: guys at the top of the company aren't acknowledging the
• 2:14:19
  Speaker 5: possibilities here that a lot of the employees believe in.
• 2:14:22
  Speaker 5: I think a lot of what you're seeing here is
• 2:14:25
  Speaker 5: guys that are worried about it, but they're the sort
• 2:14:26
  Speaker 5: of guy who worries about it that starts the company anyway. Yeah.
• 2:14:33
  Speaker 5: Back in 2015, when we were having these conversations, where, like,
• 2:14:38
  Speaker 5: I was having some of these conversations with these guys.
• 2:14:41
  Speaker 5: Miri started in the year 2000. We have been looking
• 2:14:44
  Speaker 5: at where AI is going since before any of these guys.
• 2:14:46
  Speaker 5: We were the guys that they talked to about this stuff,
• 2:14:49
  Speaker 5: and that they had to find a way to dismiss
• 2:14:50
  Speaker 5: to go ahead, right? Most people who could be sold
• 2:14:55
  Speaker 5: on the power of AI in 2015... were also sold
• 2:14:59
  Speaker 5: on the dangers of AI in 2015. The sort of
• 2:15:02
  Speaker 5: guys who start the companies are the ones who are
• 2:15:06
  Speaker 5: able to convince themselves, I need to be the one
• 2:15:08
  Speaker 5: to do it.
• 2:15:10
  Speaker 1: Is that the crux of the motivation? Because I've been
• 2:15:14
  Speaker 1: second party to private conversations with some of the leaders
• 2:15:17
  Speaker 1: of the Frontier Labs from good, good friends of mine
• 2:15:20
  Speaker 1: that are very connected. And they told me that one
• 2:15:22
  Speaker 1: particular Frontier Labs CEO estimates privately to him, and by
• 2:15:26
  Speaker 1: the way, I've seen literal text messages of them in
• 2:15:29
  Speaker 1: conversation when I asked him to come on the podcast.
• 2:15:32
  Speaker 1: And so he was like, I've texted him. He said, no,
• 2:15:35
  Speaker 1: by the way, which I found kind of funny. Where
• 2:15:38
  Speaker 1: he said to me, this particular AI CEO thinks that
• 2:15:41
  Speaker 1: the probability is roughly around 10% of human extinction. I
• 2:15:44
  Speaker 1: think he said 8%. And when I heard that, part
• 2:15:46
  Speaker 1: of the reason I have so many conversations about this
• 2:15:49
  Speaker 1: is because I see him in interviews saying other things. Totally.
• 2:15:51
  Speaker 1: And I trust my friend. So I then wonder, this
• 2:15:55
  Speaker 1: is why I use the thought experiment of these buttons
• 2:15:57
  Speaker 1: on the table. Because that particular AICO thinks that eight
• 2:16:00
  Speaker 1: of the hundred buttons are going to cause extinction and
• 2:16:02
  Speaker 1: they're powering on anyway. What is the human motivation to
• 2:16:05
  Speaker 1: do that? I asked my friend. My friend said, well,
• 2:16:07
  Speaker 1: you know, this is what he said. And again, it's
• 2:16:10
  Speaker 1: second party information, so it might not be true. It's
• 2:16:12
  Speaker 1: a bit of a Chinese whispers. He said this particular
• 2:16:15
  Speaker 1: person even if it caused human extinction, would like to
• 2:16:20
  Speaker 1: have the significance of the person that did that thing.
• 2:16:24
  Speaker 1: Because that would be a.
• 2:16:26
  Speaker 3: I think you're ethically required to tell us who the
• 2:16:28
  Speaker 3: fuck it is.
• 2:16:28
  Speaker 1: It's one of the Frontier Labs CEOs, and it's not Dario.
• 2:16:32
  Speaker 1: The Dario CEO.
• 2:16:32
  Speaker 3: But I don't know.
• 2:16:33
  Speaker 1: These things are Chinese whispers, so I don't know.
• 2:16:36
  Speaker 5: I think that you can actually get this info firsthand.
• 2:16:40
  Speaker 5: Elon Musk is clear about this. He did an interview
• 2:16:43
  Speaker 5: last year where he was like, I didn't want to
• 2:16:45
  Speaker 5: get into this AI stuff because I thought I was
• 2:16:47
  Speaker 5: too dangerous. But then I realized it was going to
• 2:16:49
  Speaker 5: happen with or without me, and I decided I would
• 2:16:52
  Speaker 5: rather be a participant than a spectator.
• 2:16:54
  Speaker 1: Because Google said that they were going to pursue it,
• 2:16:55
  Speaker 1: and he didn't trust Google.
• 2:16:57
  Speaker 5: That's right. You know, you can see in the leaked,
• 2:16:59
  Speaker 5: or sorry, not leaked, the OpenAI emails that came out
• 2:17:02
  Speaker 5: during the discovery and court cases, you can see these
• 2:17:04
  Speaker 5: guys discussing in the threads, like, we need to make
• 2:17:07
  Speaker 5: sure that we and our nonprofit at OpenAI control this
• 2:17:11
  Speaker 5: instead of, you know, the people at Google controlling this.
• 2:17:14
  Speaker 5: And then, of course, you know, OpenAI was founded as
• 2:17:15
  Speaker 5: a nonprofit, and then it was sort of changed into
• 2:17:18
  Speaker 5: a for-profit, and there's much debate about how much of
• 2:17:20
  Speaker 5: that non-profit money was in some sense stolen. And so
• 2:17:24
  Speaker 5: Elon also left because he thought they weren't going to
• 2:17:26
  Speaker 5: be good stewards. Dario also left to create Anthropic. So
• 2:17:30
  Speaker 5: in some sense, all of these AI labs, except the
• 2:17:33
  Speaker 5: Google one that came out of Demis Hassabis' original startup,
• 2:17:37
  Speaker 5: all of the other AI labs exist because none of
• 2:17:40
  Speaker 5: the CEOs trust the other guys. None of the CEOs
• 2:17:43
  Speaker 5: think the other guy should be the one holding the
• 2:17:45
  Speaker 5: leash on the superintelligence. None of them trust each other.
• 2:17:47
  Speaker 5: I just trust one few.
• 2:17:51
  Speaker 3: Yeah.
• 2:17:53
  Speaker 1: What are your closing thoughts, Andy?
• 2:17:57
  Speaker 4: We're living in really interesting times. And I think you
• 2:18:01
  Speaker 4: guys have made a very good argument that these systems are...
• 2:18:06
  Speaker 4: demonstrating new capabilities, which are very powerful and which demand
• 2:18:11
  Speaker 4: a response. I am much more confident in our ability
• 2:18:15
  Speaker 4: to rise to that challenge than you are. But you
• 2:18:19
  Speaker 4: accept the existential risk. Let me try to say it again.
• 2:18:23
  Speaker 4: I appreciate that there are new harms that We haven't
• 2:18:28
  Speaker 4: seen before that come along with a technology that's this dogged, tenacious, agentic,
• 2:18:34
  Speaker 4: you know, deceptive. I think that's the right word for it.
• 2:18:37
  Speaker 4: I agree with that. I am much more optimistic about
• 2:18:40
  Speaker 4: our ability to respond effectively to that new challenge out
• 2:18:44
  Speaker 4: there in the world than I think my two colleagues are.
• 2:18:47
  Speaker 1: And would you still be at 0%?
• 2:18:49
  Speaker 4: My prior has not shifted during this meeting.
• 2:18:51
  Speaker 5: Okay. Ed?
• 2:18:53
  Speaker 2: I think we've spent an alarming amount of time not
• 2:18:55
  Speaker 2: talking about the actual harms of AI as it is today.
• 2:18:57
  Speaker 2: I think these are necessary conversations to have. I think
• 2:19:01
  Speaker 2: we should talk about the fact that Amazon, Microsoft, Google,
• 2:19:03
  Speaker 2: Oracle are helping power these hacks, that Sam Altman and
• 2:19:07
  Speaker 2: Dario Amadei have overseen companies that have done what is
• 2:19:09
  Speaker 2: tantamount to felony hacking, that we are not having discussions
• 2:19:12
  Speaker 2: about how to stop this today, but what we might
• 2:19:14
  Speaker 2: stop tomorrow. And I think in general, we also need
• 2:19:17
  Speaker 2: to worry about the financials, which have not come up
• 2:19:19
  Speaker 2: at all. But if there is an industry slowdown, how
• 2:19:21
  Speaker 2: do you deal with the $ 1. 3 trillion of compute commitments?
• 2:19:25
  Speaker 2: All of these are very real things that will have
• 2:19:27
  Speaker 2: very real consequences very, very soon. And I understand why.
• 2:19:31
  Speaker 2: And it's necessary to discuss what we do around AI.
• 2:19:34
  Speaker 2: The actual regulatory thing we need to do today is
• 2:19:36
  Speaker 2: cut off the compute, slow down these labs fully. And
• 2:19:41
  Speaker 2: I don't care about China here. What are they going
• 2:19:43
  Speaker 2: to do? Distill a model like they have the whole time?
• 2:19:46
  Speaker 2: They are capped on our progress. So the biggest thing
• 2:19:49
  Speaker 2: to do is to slow down. And also, it's time
• 2:19:52
  Speaker 2: to start arresting people. They did felony hacking. Someone's got
• 2:19:56
  Speaker 2: to go to prison. We need responsibility and accountability for
• 2:19:59
  Speaker 2: these companies. And as long as we don't have it,
• 2:20:01
  Speaker 2: we may as well not have had any discussion about
• 2:20:03
  Speaker 2: safety because we're not doing anything.
• 2:20:05
  Speaker 1: Do you accept that there's an existential risk?
• 2:20:07
  Speaker 3: Yeah.
• 2:20:08
  Speaker 5: Absolutely.
• 2:20:08
  Speaker 2: We have the largest companies in the world doing what
• 2:20:11
  Speaker 2: I think we can all agree are extremely reckless experiments
• 2:20:14
  Speaker 2: using hundreds of billions of dollars of infrastructure. And they
• 2:20:17
  Speaker 2: are building more infrastructure around the world very slowly to
• 2:20:20
  Speaker 2: do more of these chaotic experiments. We must rein them in.
• 2:20:23
  Speaker 2: This does not mean that large language models are conscious
• 2:20:26
  Speaker 2: or able to do things that people have been promising.
• 2:20:29
  Speaker 5: Indeed, they may.
• 2:20:30
  Speaker 2: I don't think they will lead to what you're talking about.
• 2:20:32
  Speaker 2: That doesn't mean there aren't real harms, but these are
• 2:20:34
  Speaker 2: real harms caused by very specific parties allowed to run
• 2:20:38
  Speaker 2: rampant in the scourge of neoliberalism.
• 2:20:40
  Speaker 1: What's your percentage?
• 2:20:42
  Speaker 5: I mean, what are we talking about here?
• 2:20:44
  Speaker 1: Do you think there's a more than 10% chance of
• 2:20:46
  Speaker 1: existential harm?
• 2:20:47
  Speaker 2: Wasn't it within 10 years or something?
• 2:20:49
  Speaker 4: Yeah.
• 2:20:50
  Speaker 1: Not 10%.
• 2:20:50
  Speaker 2: I mean, not 1%, but it's like.
• 2:20:53
  Speaker 5: Let me just be clear about what that means.
• 2:20:55
  Speaker 2: Do I think that unrestrained LLM use connected to massive
• 2:20:59
  Speaker 2: amounts of infrastructure could lead to actually a.
• 2:21:01
  Speaker 5: Power system going down? Absolutely.
• 2:21:04
  Speaker 2: We had Knight Capital. What, like 13, 14 years ago?
• 2:21:06
  Speaker 2: I could see someone being dumb enough to connect that
• 2:21:08
  Speaker 2: to financial accounts. Human error, led with this chaotic software
• 2:21:13
  Speaker 2: we use, is a danger.
• 2:21:14
  Speaker 3: I will directionally agree with arresting everyone, but don't build
• 2:21:19
  Speaker 3: general superintelligence if you're working at one of those labs.
• 2:21:22
  Speaker 3: Quit today. Thank you.
• 2:21:24
  Speaker 5: The people at these labs really do believe this poses
• 2:21:27
  Speaker 5: an extinction threat. I think our response as a society
• 2:21:36
  Speaker 5: cannot be, please continue, we hope you'll fail. And our
• 2:21:39
  Speaker 5: response as a society cannot be, let it rip in
• 2:21:44
  Speaker 5: a giant competitive race that you yourselves are saying you
• 2:21:47
  Speaker 5: don't want to be in. We are forcing you to
• 2:21:50
  Speaker 5: go ahead because of the boogeyman of China. We have
• 2:21:54
  Speaker 5: seen the people at these companies say that we need
• 2:21:58
  Speaker 5: to develop the tools to pace the frontier, which is
• 2:22:00
  Speaker 5: corporate speak for this is going too fast for us
• 2:22:04
  Speaker 5: to get a handle on things. We need, like, these
• 2:22:11
  Speaker 5: people believe it. They believe they're gambling with your lives.
• 2:22:14
  Speaker 5: What has changed is that the rest of the world
• 2:22:16
  Speaker 5: is starting to notice. And that's what gives us a
• 2:22:19
  Speaker 5: moment of hope.
• 2:22:20
  Speaker 1: Trump this week was asked about the threat of AI.
• 2:22:23
  Speaker 1: And this was his response.
• 2:22:24
  Speaker 2: The case scenario with AI is that the robots, the
• 2:22:27
  Speaker 2: machinery learns to, obviously it thinks for itself.
• 2:22:30
  Speaker 4: That's what it does.
• 2:22:31
  Speaker 1: And that could turn against humanity. Do we have the guardrails?
• 2:22:36
  Speaker 1: It's going to be fine.
• 2:22:37
  Speaker 3: We'll always have something to stop them, right?
• 2:22:39
  Speaker 4: We'll have a little gear.
• 2:22:41
  Speaker 3: I really don't like that. I really don't like that robot.
• 2:22:43
  Speaker 3: We'll stop it. Some people say the worst case scenario.
• 2:22:46
  Speaker 3: You're laughing, but this is a state of the art
• 2:22:48
  Speaker 3: in AI safety right now.
• 2:22:49
  Speaker 5: Yeah.
• 2:22:50
  Speaker 1: This is the device we have. That's the best we got.
• 2:22:52
  Speaker 1: For anyone that couldn't hear that, Trump went, we'll always
• 2:22:54
  Speaker 1: be fine. We'll have something to control it. And then
• 2:22:56
  Speaker 1: he did a little gun finger and he went, boom,
• 2:22:57
  Speaker 1: I don't like that robot.
• 2:23:00
  Speaker 5: I don't like semi.
• 2:23:02
  Speaker 1: If you don't laugh.
• 2:23:04
  Speaker 5: I would say that the reason humanity always has something
• 2:23:10
  Speaker 5: to stop a problem is because people notice a problem
• 2:23:13
  Speaker 5: and build what it takes to have something to stop
• 2:23:15
  Speaker 5: a problem, which I think you'd agree with. I am
• 2:23:17
  Speaker 5: not here saying we're going to die. I'm here saying,
• 2:23:20
  Speaker 5: if you look at the technology, if you look at
• 2:23:24
  Speaker 5: what it's doing now, if you look at what the
• 2:23:26
  Speaker 5: experts who are building it are saying about their own fears.
• 2:23:32
  Speaker 5: You see that we need to rise to this occasion.
• 2:23:35
  Speaker 5: You said you trust humanity to rise to the occasion.
• 2:23:37
  Speaker 5: I sure hope we can. I think that rising to
• 2:23:40
  Speaker 5: this occasion is going to mean that nobody races towards
• 2:23:44
  Speaker 5: superintelligence because we have no idea how to get that right. And,
• 2:23:50
  Speaker 5: you know, finally the world is starting to notice that
• 2:23:53
  Speaker 5: it's an extinction threat.
• 2:23:55
  Speaker 3: Thank you.
• 2:23:56
  Speaker 1: Nate, Roman, Ed, Andy, super appreciate you. All of your
• 2:23:59
  Speaker 1: books will be linked below in the description. And on screen,
• 2:24:05
  Speaker 1: let's see what happens. We'll convene again. Thank you so much.
• 2:24:07
  Speaker 1: Thank you. YouTube have this new crazy algorithm where they
• 2:24:10
  Speaker 1: know exactly what video you would like to watch next
• 2:24:13
  Speaker 1: based on AI and all of your viewing behavior. And
• 2:24:16
  Speaker 1: the algorithm says that this video is the perfect video
• 2:24:20
  Speaker 1: for you. It's different for everybody looking right now. Check
• 2:24:22
  Speaker 1: this video out. I bet you, you might love it.