Latest / Breaking Points with Krystal and Saagar / 9/14/26: Anthropic Calls To Slow Pace of AI, Nate Soares AI Regulation

Transcript

9/14/26: Anthropic Calls To Slow Pace of AI, Nate Soares AI Regulation · Breaking Points with Krystal and Saagar

• 0:00
  Speaker 1: Hey guys, Ready or Not 2024 is here, and we
• 0:04
  Speaker 1: here at Breaking Points are already thinking of ways we
• 0:06
  Speaker 1: can up our game for this critical election.
• 0:08
  Speaker 2: We rely on our premium subs to expand coverage, upgrade
• 0:11
  Speaker 2: the studio, add staff, give you guys the best independent
• 0:15
  Speaker 2: coverage that is possible. If you like what we're all about,
• 0:17
  Speaker 2: it just means the absolute world to have your support.
• 0:20
  Speaker 3: But enough with that, let's get to the show. Good morning, everybody.
• 0:26
  Speaker 3: Happy Monday. We have an amazing show for everybody today.
• 0:28
  Speaker 2: What do we have, Crystal?
• 0:29
  Speaker 1: Indeed, we do. Lots going on on the AI front.
• 0:32
  Speaker 1: The heads of all of the major AI companies coming
• 0:35
  Speaker 1: out and sounding the alarm about concerns over safety and
• 0:38
  Speaker 1: calling for a slowdown. We'll break down what they are saying.
• 0:41
  Speaker 1: We will sort the noise from the hype from the reality.
• 0:44
  Speaker 1: We're going to have Nate Soros on, who is an
• 0:46
  Speaker 1: expert on these things, also to break down some of
• 0:48
  Speaker 1: the various theories about why they are saying this right now.
• 0:52
  Speaker 1: In addition, of course, we have our eye on the
• 0:54
  Speaker 1: Middle East and the impact on all of our lives.
• 0:57
  Speaker 1: Saudi had to shut the east-west pipeline, which was helping
• 1:00
  Speaker 1: to ease some of the pressure in the oil market.
• 1:02
  Speaker 1: What will the consequences of that be as the Houthis
• 1:05
  Speaker 1: have had a stunning offensive and taken that entire Red
• 1:08
  Speaker 1: Sea coastline, giving them effective control over the Bab el-Mandab Strait?
• 1:13
  Speaker 1: In addition, we are taking a look at gambling with
• 1:15
  Speaker 1: NFL season in full swing. Sager, of course, has his
• 1:18
  Speaker 1: eye on this one. LeBron, this is obviously not NFL,
• 1:21
  Speaker 1: but related, is the latest player to sell out to
• 1:24
  Speaker 1: the gambling apps. Sidney Sweeney doing her thing as well.
• 1:27
  Speaker 1: So a lot to get into there. And then we've
• 1:29
  Speaker 1: got the latest on the midterms, including Trump weighing in
• 1:32
  Speaker 1: on flock cameras. Sager, he says he likes them.
• 1:34
  Speaker 2: He's coming in on the right side, definitely. For him,
• 1:37
  Speaker 2: but definitely not for the Republicans. Thank you to everybody
• 1:40
  Speaker 2: who has been signing up. For our free trial, go
• 1:43
  Speaker 2: ahead and take advantage, breakingpoints.com. Because Ryan continues to win
• 1:47
  Speaker 2: in all of the competitions, our pitch now is if
• 1:50
  Speaker 2: you want Ryan to be Ryan, then you're going to
• 1:53
  Speaker 2: go ahead and sign up for our free trial.
• 1:55
  Speaker 3: Otherwise, you might see Calchi ads.
• 1:56
  Speaker 2: You might see Polymarket, Sydney Sweeney. I might go shirtless
• 2:02
  Speaker 2: and go for a prediction market unless you sign.
• 2:05
  Speaker 3: Up for that.
• 2:06
  Speaker 2: If you don't want to see that, which you definitely don't,
• 2:10
  Speaker 2: you've got to go ahead and sign up for breakingpoints.com.
• 2:12
  Speaker 2: If you're watching this on YouTube, please hit subscribe to
• 2:15
  Speaker 2: our YouTube channel. Listen to this podcast. Please share an
• 2:18
  Speaker 2: episode with a friend. But as Crystal said, there's so
• 2:20
  Speaker 2: much happening now. With AI, everything kicked off on Saturday
• 2:24
  Speaker 2: here with the release of a new essay by Dario Amadei,
• 2:27
  Speaker 2: the CEO of Anthropic. Let's put it up here on
• 2:30
  Speaker 2: the screen. Highly recommend that everybody go and read it.
• 2:34
  Speaker 2: It's not too long, and I actually think it is
• 2:37
  Speaker 2: actually just worth getting in the heads of Dario. We'll
• 2:39
  Speaker 2: play some of the video interview that he did in
• 2:43
  Speaker 2: a little bit. But the general TLDR here, it is
• 2:46
  Speaker 2: called We Must Pace the Frontier. So the bullet points
• 2:49
  Speaker 2: that I have written down is that Amadei says AI
• 2:52
  Speaker 2: companies must deliberately slow the rate at which they advance
• 2:55
  Speaker 2: AI capabilities, not halt training. This is a shift beyond
• 2:58
  Speaker 2: just investing in risk prevention. The pace of capability gains
• 3:02
  Speaker 2: itself now needs managing. What changed his mind is two things.
• 3:06
  Speaker 2: Number one is RSI. I know these terms are annoying,
• 3:11
  Speaker 2: but it is important that we immerse ourselves so we
• 3:13
  Speaker 2: can understand RSI. these things that these people use, AGI,
• 3:17
  Speaker 2: artificial general intelligence. RSI is recursive self-improvement. He says it
• 3:22
  Speaker 2: has exploded since the summer of 2026. Hugging face incident
• 3:26
  Speaker 2: in particular, along with a number of other incidents that
• 3:29
  Speaker 2: both OpenAI and Anthropic have now been involved in. Pacing,
• 3:33
  Speaker 2: he says, is that models are now capable enough that
• 3:35
  Speaker 2: extra time is genuinely useful. An extra year or two
• 3:39
  Speaker 2: would meaningfully reduce risk. He proposes a number of kind
• 3:44
  Speaker 2: of interesting things. So four levels. Number one is ban
• 3:49
  Speaker 2: clearly dangerous uses, i.e. bioweapons, likely the most feasible, he says.
• 3:53
  Speaker 2: Level two, a mutual pre-release testing for acute risk feasible,
• 3:57
  Speaker 2: but it's hard to give teeth, is what he claims.
• 4:00
  Speaker 2: I guess this would be akin to the pharmaceutical metaphor
• 4:02
  Speaker 2: I've been using a lot, is like you have to
• 4:04
  Speaker 2: You have to release phase two, phase three trials to
• 4:06
  Speaker 2: the FDA before it's allowed to be brought to market.
• 4:09
  Speaker 2: Level three, he calls a speed limit on recursive self-improvement,
• 4:12
  Speaker 2: kind of what he analogizes to a SALT treaty, which
• 4:16
  Speaker 2: is the Nuclear Weapons Treaty.
• 4:17
  Speaker 3: He believes it's difficult but near impossible.
• 4:19
  Speaker 2: And then level four, a full pacing or pause supported
• 4:23
  Speaker 2: in principle, but he believes is unlikely because of strong
• 4:25
  Speaker 2: defection incentives and a number of other things he proposes
• 4:28
  Speaker 2: inside of that, which is inviting big geopolitical consequences from China.
• 4:32
  Speaker 2: But overall, a landmark essay, the CEO of the company
• 4:35
  Speaker 2: saying we need to slow down.
• 4:36
  Speaker 3: You almost never see that. And I guess the big
• 4:38
  Speaker 3: headline news.
• 4:39
  Speaker 2: Is not Dario's essay, but all of the other AI
• 4:42
  Speaker 2: CEOs coming in and be like, yeah, Dario's right.
• 4:44
  Speaker 1: Right. And so what he says they're going to do
• 4:46
  Speaker 1: at Anthropic unilaterally, regardless of how anyone else responds or
• 4:50
  Speaker 1: whether there's an agreement with China or any other international body,
• 4:54
  Speaker 1: is to invite in some external researchers, give them employee
• 4:59
  Speaker 1: level access so that they can constantly scrutinize what is
• 5:02
  Speaker 1: going on inside of the lab. And he's encouraging others
• 5:05
  Speaker 1: to do that. Sam Altman says that they will do
• 5:07
  Speaker 1: the same over at OpenAI. So they're saying they'll take
• 5:10
  Speaker 1: that action unilaterally. Looks like he specifically mentions METR, which
• 5:15
  Speaker 1: is the group that did the safety report that revealed
• 5:19
  Speaker 1: some of the details about the hugging face incident that
• 5:21
  Speaker 1: we did not know in the past. So that's the
• 5:24
  Speaker 1: action they're going to take unilaterally. In addition, they're talking
• 5:27
  Speaker 1: about We're not going to stop altogether, but we're going
• 5:31
  Speaker 1: to slow down the pace. So this is very different
• 5:33
  Speaker 1: from what Bernie Sanders is calling for at a legislative
• 5:36
  Speaker 1: level where he's saying pause everything until we can figure
• 5:40
  Speaker 1: out what's going on. And in addition with Bernie Sanders legislation,
• 5:43
  Speaker 1: he would outright ban development of super intelligence that's beyond.
• 5:48
  Speaker 1: So AGI is like human level intelligence and super intelligence
• 5:51
  Speaker 1: is like, you know, better than the smartest humans at
• 5:54
  Speaker 1: every intellectual task. So Bernie's proposal is ban that and
• 5:58
  Speaker 1: make it give put criminal penalties to it, put someone
• 6:01
  Speaker 1: in jail for 20 years if they decide to go
• 6:04
  Speaker 1: forward with developing super intelligence. So in any case, there's
• 6:08
  Speaker 1: a lot of theories about what's going on behind the scenes.
• 6:11
  Speaker 1: And we'll get to that in just a moment. But
• 6:13
  Speaker 1: that's what they're doing sort of unilaterally. Then he floats
• 6:16
  Speaker 1: this idea of, hey, we should probably the frontier labs
• 6:20
  Speaker 1: be able to be in contact with each other so
• 6:23
  Speaker 1: that we can self-regulate together and make sure that we're
• 6:27
  Speaker 1: slowing the pace together. This has gotten into all sorts
• 6:30
  Speaker 1: of discussion over whether what they're really looking for is
• 6:33
  Speaker 1: some sort of an antitrust exemption so they can de
• 6:35
  Speaker 1: facto form some sort of a cartel. There's a lot
• 6:38
  Speaker 1: of analysis that says that sort of safety collaboration is
• 6:41
  Speaker 1: already allowed and permitted. That's another aspect of what's going
• 6:45
  Speaker 1: on here. And then some of the things that Sagar
• 6:46
  Speaker 1: brought up is he's talking more broadly about, okay, well,
• 6:49
  Speaker 1: what do we do about this internationally? Dario is very
• 6:53
  Speaker 1: much of the but China mindset. You know, if you
• 6:56
  Speaker 1: take him at face value, which, by the way, I
• 6:58
  Speaker 1: don't think you should really take any of these guys
• 7:00
  Speaker 1: at face value, but Dario's been talking about safety concerns
• 7:04
  Speaker 1: for years and years and years. This didn't just come
• 7:07
  Speaker 1: out of nowhere. In fact, he was working and his
• 7:09
  Speaker 1: sister at OpenAI. The reason they left to found Anthropic
• 7:14
  Speaker 1: is basically because... They saw that Sam Altman is a
• 7:17
  Speaker 1: total psychopath and were like, I don't think they're doing
• 7:19
  Speaker 1: safety correctly. We're going to start Anthropic to do safety
• 7:22
  Speaker 1: the right way. I do want to give him credit that,
• 7:25
  Speaker 1: you know, he's the guy who basically told Pete Hegseth
• 7:27
  Speaker 1: to shove it when he wanted him to give complete
• 7:30
  Speaker 1: and total access to Claude and allow them to do
• 7:31
  Speaker 1: autonomous killing, etc. So I think there is enough there
• 7:35
  Speaker 1: to say he is sincere about the concerns. However... He
• 7:37
  Speaker 1: has trapped himself in this very similar mindset as Sam
• 7:41
  Speaker 1: and as Elon, which is if I don't build it,
• 7:44
  Speaker 1: some other more irresponsible actor will. And in particular, as
• 7:47
  Speaker 1: I said before, he is very much of the but oh,
• 7:50
  Speaker 1: my God, China mindset. So included in here is while
• 7:55
  Speaker 1: we're doing this, we also need to make sure that
• 7:57
  Speaker 1: we are hobbling China's ability to develop this technology properly.
• 8:00
  Speaker 1: you know, in whatever ways that we can. That's led
• 8:03
  Speaker 1: to a response to China. We'll get to that in
• 8:04
  Speaker 1: a moment. So in any case, he sort of throws
• 8:07
  Speaker 1: this gauntlet down of we should slow the pace. And
• 8:10
  Speaker 1: I'm concerned about things getting out of control. We've reached
• 8:14
  Speaker 1: this trajectory with recursive self-improvement, the AIs are training even
• 8:18
  Speaker 1: smarter AIs. There's long been theorized that that could lead
• 8:21
  Speaker 1: to this sort of intelligence explosion. We've just had these
• 8:23
  Speaker 1: major incidents. And then you have Elon and Sam Altman
• 8:28
  Speaker 1: and Demis Hassab, as we can put this next piece
• 8:30
  Speaker 1: up on the screen, all come in and basically say,
• 8:33
  Speaker 1: we agree. So Elon says, Dario is right. Sam Altman says,
• 8:37
  Speaker 1: I agree with Dario. We need to pace the frontier.
• 8:40
  Speaker 1: This has been a primary topic of discussions we've had
• 8:42
  Speaker 1: at OpenAI in recent weeks, committing to having independent evaluators
• 8:45
  Speaker 1: with employee-like access. Great idea. We will do the same.
• 8:48
  Speaker 1: Have more to share soon. Demis Hassabis, who is the
• 8:51
  Speaker 1: head over at Google, says Dario's essay points towards the
• 8:54
  Speaker 1: right path forward. The details need working through, but the
• 8:56
  Speaker 1: direction is correct for meeting this critical moment. This is
• 8:59
  Speaker 1: also why we recently put out our proposal for an
• 9:02
  Speaker 1: industry-wide standards body for frontier AI. So, These are the
• 9:06
  Speaker 1: four heads of the major frontier labs all coming out. Zuckerberg,
• 9:10
  Speaker 1: I mean, they're kind of behind in the race anyway,
• 9:12
  Speaker 1: but he apparently does not agree with the sentiment. But
• 9:16
  Speaker 1: in any case, this has led to a lot of
• 9:17
  Speaker 1: conjecture about what's going on here. Are they genuinely concerned?
• 9:22
  Speaker 1: Is there some sort of a financial incentive here? You've
• 9:24
  Speaker 1: got OpenAI and Anthropic, which are planning IPOs. Is there
• 9:29
  Speaker 1: something that happened behind the scenes? Are they trying to
• 9:32
  Speaker 1: create a moat where only they are basically allowed to
• 9:36
  Speaker 1: develop under regulation and then maybe the open source models
• 9:39
  Speaker 1: will be banned? So that's some of the sort of
• 9:41
  Speaker 1: debate and discussion that's happening.
• 9:42
  Speaker 3: I think it's all of the above.
• 9:44
  Speaker 2: And China remains, again, like this chimera, which floats above
• 9:49
  Speaker 2: all of this.
• 9:49
  Speaker 3: But nobody actually knows anything about it.
• 9:51
  Speaker 2: And Even this morning, there are so many takes out there,
• 9:55
  Speaker 2: China is responding.
• 9:56
  Speaker 3: It's like, well, kind of, right? Because China is not
• 10:00
  Speaker 3: a single entity.
• 10:01
  Speaker 2: They have Global Times, which is like their Fox News
• 10:05
  Speaker 2: everybody is citing. Then there's a foreign ministry spokesman. He
• 10:08
  Speaker 2: actually is objecting to a very specific part of the
• 10:10
  Speaker 2: Dario essay. And then there's actual journals that are being written.
• 10:13
  Speaker 2: So we're going to spend some time on that. But again,
• 10:15
  Speaker 2: China looms above this entire thing, and that's what Trump
• 10:18
  Speaker 2: is leaning on, why there shouldn't be any regulation.
• 10:20
  Speaker 3: Let's take a listen.
• 10:21
  Speaker 1: We're leading China in AI. We're the most sophisticated.
• 10:24
  Speaker 4: Country in the world.
• 10:26
  Speaker 3: And frankly, I want to keep it that.
• 10:27
  Speaker 1: Way, because whoever wins AI wins.
• 10:30
  Speaker 5: And we can put guardrails, we.
• 10:32
  Speaker 1: Can do this and that, but I think you have
• 10:34
  Speaker 1: a lot of negative forces that are bringing it up
• 10:36
  Speaker 1: that shouldn't be bringing it up.
• 10:39
  Speaker 3: And they're bringing up things that won't happen. But whoever
• 10:41
  Speaker 3: wins with AI wins. Bringing up things that won't happen.
• 10:45
  Speaker 2: Basically, this is, and this is kind of what you
• 10:47
  Speaker 2: were talking about, and this is where, look, you have
• 10:50
  Speaker 2: to say that there's a kernel of truth, which is
• 10:53
  Speaker 2: part of the, one of the things, a key part
• 10:55
  Speaker 2: of what Dario proposes in Essay. He needs an antitrust
• 10:59
  Speaker 2: exemption from being able to cooperate with all of the
• 11:03
  Speaker 2: other big heads of the companies. And that's why you
• 11:06
  Speaker 2: have Meta and many of the other AI startups who
• 11:09
  Speaker 2: are like, hey, excuse me, he's using safety as an
• 11:13
  Speaker 2: excuse for regulatory capture so that we have a big
• 11:17
  Speaker 2: four of AI in the same way that we have
• 11:20
  Speaker 2: the defense contracting industry or pharma. You can't say they're wrong.
• 11:24
  Speaker 2: Like there is at least an element of that. But
• 11:26
  Speaker 2: then you have to say wrong, but or wrong. Is
• 11:29
  Speaker 2: it also maybe that they are worried? And I think
• 11:32
  Speaker 2: that's why I said when you were laying out your case,
• 11:34
  Speaker 2: I genuinely think it's all of the above.
• 11:37
  Speaker 3: We have no idea.
• 11:38
  Speaker 2: We can only listen to. what they're all saying. I
• 11:40
  Speaker 2: will say it's pretty extraordinary to see, especially Google. It's
• 11:43
  Speaker 2: not like Google or Gemini is a major leader in AI,
• 11:47
  Speaker 2: or at least hasn't been on the consumer level right now.
• 11:49
  Speaker 1: Although there are some rumors that they may be on
• 11:53
  Speaker 1: the cusp of a breakthrough.
• 11:54
  Speaker 3: Very possible. We have no idea.
• 11:56
  Speaker 2: At least at the current moment, in terms of what's available,
• 11:58
  Speaker 2: the consumer product, I would say Gemini is one of
• 12:00
  Speaker 2: the worst ones, right? It's up there with Meta. So
• 12:03
  Speaker 2: then you're like, all right. And then it's up there
• 12:04
  Speaker 2: with Grok and Meta and the rest. It's like genuinely
• 12:08
  Speaker 2: in a second tier compared to where Anthropic and OpenAI are.
• 12:12
  Speaker 2: The question, I think, remains around what exactly they are proposing.
• 12:18
  Speaker 2: And by and large, Sam Altman and Dario and even
• 12:22
  Speaker 2: Elon at this point are all saying, at the very least,
• 12:25
  Speaker 2: we need voluntary pause. and some sort of coordination. However, again,
• 12:29
  Speaker 2: the China thing looms above. And that's where I just
• 12:33
  Speaker 2: think we have to come back to what do the
• 12:35
  Speaker 2: Chinese actually want out of AI? And what do they think?
• 12:40
  Speaker 2: Because a lot of people are taking a statement this morning.
• 12:42
  Speaker 2: I think we have it.
• 12:43
  Speaker 3: Let's put A8 up here on the screen.
• 12:45
  Speaker 2: This was the Chinese foreign ministry spokesperson who is reacting
• 12:49
  Speaker 2: to to the Dario essay.
• 12:53
  Speaker 3: And I'm going to read you a direct quote.
• 12:55
  Speaker 2: And this is important because this is a direct statement
• 12:57
  Speaker 2: from their foreign ministry to the press, which is all
• 13:00
  Speaker 2: highly vetted, translated correctly.
• 13:02
  Speaker 3: And so you know this is exactly what they want
• 13:04
  Speaker 3: us to hear. And here's what he said.
• 13:06
  Speaker 2: Fear-mongering confrontation and vicious competition will only disrupt the process
• 13:11
  Speaker 2: of global AI governance, which serves no one's interests. Okay,
• 13:15
  Speaker 2: so you could take that as them saying, along with
• 13:19
  Speaker 2: a Global Times op-ed that they put out, which we
• 13:21
  Speaker 2: can also be fair, we can put that up there
• 13:22
  Speaker 2: as well, A7, please, where they said specifically, this is
• 13:26
  Speaker 2: the Reuters tear sheet, And what they say is that
• 13:29
  Speaker 2: the state-run newspapers blast Anthropix calls to slow AI as
• 13:32
  Speaker 2: a Cold War tactic. Now, a lot of people are saying, oh,
• 13:35
  Speaker 2: China sees itself in this Cold War mentality around AI development.
• 13:39
  Speaker 3: It's not true.
• 13:40
  Speaker 2: If you were to read in this, what they specifically
• 13:44
  Speaker 2: object to are the same things they've always done, which
• 13:45
  Speaker 2: is the export control of chips. And so what they're
• 13:49
  Speaker 2: objecting to is not necessarily a pause on AI development.
• 13:53
  Speaker 3: Even this headline is totally wrong, a Cold War tactic.
• 13:56
  Speaker 2: What they are objecting to is export control, which of
• 13:59
  Speaker 2: course makes sense from their perspective because they want more
• 14:02
  Speaker 2: of the NVIDIA and other chips so they can continue
• 14:04
  Speaker 2: to pace their open or they can continue to train
• 14:07
  Speaker 2: their open source models. But, and I continue to return
• 14:10
  Speaker 2: to this, what do they actually think about AGI? They
• 14:14
  Speaker 2: have one company, from what I can tell, maybe one
• 14:17
  Speaker 2: or two leaders, which actually are like fully quote AGI pilled.
• 14:21
  Speaker 2: The rest of their AI companies, as I've laid out
• 14:25
  Speaker 2: in our last show, they are much more directed towards
• 14:28
  Speaker 2: making things better in the physical world, improve manufacturing, improve productivity, improve,
• 14:34
  Speaker 2: you know, even in robotics, like we want robotics, but
• 14:36
  Speaker 2: they need to be efficient in packing goods in terms
• 14:40
  Speaker 2: of anything that could genuinely pose a threat to their grid, to, um,
• 14:45
  Speaker 2: to their political system of control. They are absolutely not
• 14:49
  Speaker 2: on board with that. And in fact, one of the
• 14:51
  Speaker 2: biggest stories in the whole world on AI, got zero
• 14:54
  Speaker 2: attention here in Washington, was that there was a WeChat worm,
• 14:58
  Speaker 2: which was developed by researchers. And this WeChat is not,
• 15:02
  Speaker 2: obviously nobody uses it here in the United States, but
• 15:06
  Speaker 2: over a billion people use it in China. It's the
• 15:08
  Speaker 2: way that they order Uber, the way they pay, it's
• 15:11
  Speaker 2: the way they tax. It's like their WhatsApp Plus. I mean,
• 15:13
  Speaker 2: you can't even conceive of it. in the way that
• 15:15
  Speaker 2: it is in Washington. And they showed that it could
• 15:17
  Speaker 2: have taken over all of these different devices in a
• 15:20
  Speaker 2: matter of days. You can bet that somebody in China
• 15:23
  Speaker 2: is freaking out about that, because that is a direct
• 15:26
  Speaker 2: challenge to their ability. If you can control somebody's phone
• 15:30
  Speaker 2: through Claude or any of these other models, well, then
• 15:33
  Speaker 2: you could give them information they don't want to get.
• 15:34
  Speaker 2: You could take over their banking system. You could cause
• 15:37
  Speaker 2: genuine chaos in their society. There is no way that
• 15:40
  Speaker 2: they're on board with that. So I just want to
• 15:41
  Speaker 2: parse everything. You might see a lot of headlines today, unfortunately,
• 15:44
  Speaker 2: like the Reuters one saying slow AI. That's not what
• 15:47
  Speaker 2: they said. They said that they object to the export control,
• 15:49
  Speaker 2: which of course they object to export controls.
• 15:52
  Speaker 3: One area where we beat them. That's not new at all.
• 15:58
  Speaker 1: Let me go ahead and tell you guys my theories
• 16:00
  Speaker 1: of what's going on here for what it's worth. My
• 16:03
  Speaker 1: three top theories of what's going on here. One, I
• 16:06
  Speaker 1: think it's possible that something is going on in these
• 16:08
  Speaker 1: labs that genuinely has them freaked out. Two, and again,
• 16:11
  Speaker 1: it could be a combination of these three or other
• 16:13
  Speaker 1: things that are going on as well. Number two, they
• 16:17
  Speaker 1: are as capable of reading the publicly available polling as
• 16:21
  Speaker 1: we are. They see the massive backlash to data centers,
• 16:25
  Speaker 1: the massive support for Bernie Sanders' bill to actually use
• 16:29
  Speaker 1: the force of the federal government to pause AI development completely.
• 16:34
  Speaker 1: And so they're drawing from other corporate playbooks of like
• 16:38
  Speaker 1: trying to get a PR win here.
• 16:39
  Speaker 3: No, no, no.
• 16:40
  Speaker 1: We are very responsible. You don't have to regulate us.
• 16:44
  Speaker 1: We're going to regulate ourselves. We're going to take care
• 16:46
  Speaker 1: of this. Don't worry about it. Now, of course, under
• 16:49
  Speaker 1: the Trump regime, like they're going to be able to
• 16:51
  Speaker 1: do whatever they want. But 2028 is not far away.
• 16:54
  Speaker 1: And it is not hard to imagine AI becoming a
• 16:57
  Speaker 1: central litmus test in potentially both parties primaries. But in particular,
• 17:01
  Speaker 1: I have to think they'd be worried about what's going
• 17:03
  Speaker 1: on in the Democratic Party right now. And that you
• 17:05
  Speaker 1: could have someone who runs and wins for president on
• 17:09
  Speaker 1: a platform of we are going to completely halt AI development.
• 17:13
  Speaker 1: And so they're looking at this and going, OK, well,
• 17:16
  Speaker 1: if we take some steps now, we can improve our
• 17:19
  Speaker 1: PR image with the public. And perhaps we can forestall
• 17:23
  Speaker 1: that direction, which we do not want to go in,
• 17:25
  Speaker 1: which is a complete pause on AI development. The other
• 17:28
  Speaker 1: thing to keep in mind is both OpenAI and Anthropic
• 17:31
  Speaker 1: are planning these IPOs. Now, OpenAI has said we're not
• 17:34
  Speaker 1: going to do it this year. Anthropic is, however, moving forward.
• 17:39
  Speaker 1: And they just released some information to the Financial Times
• 17:43
  Speaker 1: and others saying we are extremely profitable. We have an 80%
• 17:47
  Speaker 1: profit margin if you take out the cost of training
• 17:52
  Speaker 1: the new models. OK, so another possibility here is that
• 17:57
  Speaker 1: in advance of their IPO, in order to have their
• 18:00
  Speaker 1: financials look as good as possible, they want to have
• 18:03
  Speaker 1: an excuse to pull back on some of those training costs.
• 18:07
  Speaker 1: So they are, in fact, profitable. Because, and this is
• 18:11
  Speaker 1: something that we've talked to, I believe, this point that
• 18:13
  Speaker 1: Garrison has made. They are actually, like, if they're just
• 18:17
  Speaker 1: selling the product they've already created, just on that revenue alone,
• 18:21
  Speaker 1: they've grown massively. You know, running that existing product, not
• 18:24
  Speaker 1: training new ones, is not nearly as expensive. They are
• 18:28
  Speaker 1: already very profitable if you just are looking at that part.
• 18:31
  Speaker 1: But if you include all of the training costs, that's
• 18:35
  Speaker 1: when they are deeply underwater. So there is a financial
• 18:38
  Speaker 1: incentive there as well. And then I do think the
• 18:41
  Speaker 1: concerns that have been expressed about like, oh, they're just
• 18:43
  Speaker 1: trying to sort of cement their positions and will be
• 18:46
  Speaker 1: the big AI companies and everything else needs to be
• 18:49
  Speaker 1: banned as a sort of longer term play. I think
• 18:52
  Speaker 1: that's a possibility. Although to me, it seems this to
• 18:55
  Speaker 1: me seems like a bank shot situation. to achieve that outcome.
• 18:59
  Speaker 1: So I'm less convinced that that's the immediate goal of
• 19:02
  Speaker 1: what is going on here. But to cut through, to
• 19:04
  Speaker 1: try to cut through some of the noise, because you
• 19:06
  Speaker 1: could spend an endless amount of time, which I've sort
• 19:09
  Speaker 1: of done this weekend, thinking about what's going on with
• 19:11
  Speaker 1: them and what their motivations are and what's really in
• 19:13
  Speaker 1: their mind, what are really their goals. It's really important
• 19:16
  Speaker 1: to keep two things in mind. Number one, in terms
• 19:20
  Speaker 1: of are they being serious about the AI concerns, again,
• 19:23
  Speaker 1: no reason to take any of these guys at face value,
• 19:26
  Speaker 1: look at their actions. Many of them have built themselves
• 19:28
  Speaker 1: doomsday bunkers, okay? Many of them have built themselves Doomsday bunkers.
• 19:34
  Speaker 1: So clearly, they take seriously that they are building something
• 19:39
  Speaker 1: that they think could at the very least lead to
• 19:41
  Speaker 1: a sort of civilizational collapse. Now, whether or not that
• 19:45
  Speaker 1: is reality is a different question. Whether or not it's
• 19:48
  Speaker 1: completely insane that anyone... would move forward building a technology
• 19:52
  Speaker 1: that they think could lead to that outcome is obviously
• 19:54
  Speaker 1: completely crazy. However, they have trapped themselves in this, like,
• 19:59
  Speaker 1: prisoner's dilemma mindset where if I don't do it, someone
• 20:01
  Speaker 1: else will, and they feel like the technology has this inevitability.
• 20:05
  Speaker 1: So they continue racing forward. So they do actually believe
• 20:09
  Speaker 1: that they are building a product which could be profoundly
• 20:13
  Speaker 1: damaging and potentially create an existential risk to the planet
• 20:17
  Speaker 1: and to humanity. So that's number one. Number two, the
• 20:20
  Speaker 1: most important thing to keep in mind is not what
• 20:23
  Speaker 1: Dario Amadei or Elon Musk or Sam Altman or anyone
• 20:26
  Speaker 1: else wants. It's what is in our interest. What do
• 20:31
  Speaker 1: we want? And certainly a, you know, non-existent regulatory regime
• 20:36
  Speaker 1: where we're relying on these guys to police themselves is
• 20:38
  Speaker 1: not what any of us should want, regardless of what
• 20:41
  Speaker 1: your levels of fear about this technology are. Ultimately, we
• 20:45
  Speaker 1: need to push. It's not going to happen in this administration,
• 20:48
  Speaker 1: though we can try. We need to push for an
• 20:51
  Speaker 1: actual regulatory regime that is going to take safety seriously,
• 20:55
  Speaker 1: that is going to have independence from the industry, because
• 20:59
  Speaker 1: that is the other thing. This is a small community.
• 21:02
  Speaker 1: They all know each other. So getting that, making sure
• 21:04
  Speaker 1: that it is independent. and that we are monitoring this technology.
• 21:08
  Speaker 1: And personally, given the fact that we have already had
• 21:12
  Speaker 1: these incidents of the AI swarms breaking out, given that
• 21:14
  Speaker 1: we already do not know how these things work or
• 21:17
  Speaker 1: what they are likely to do, yes, I think it
• 21:20
  Speaker 1: is entirely appropriate to pause the development of this technology
• 21:24
  Speaker 1: until the public can catch up and until the research
• 21:27
  Speaker 1: can catch up so that it is safe to proceed. So,
• 21:31
  Speaker 1: While we're all debating what's going on in their minds
• 21:33
  Speaker 1: and why now and blah, blah, blah, the most important
• 21:36
  Speaker 1: thing to keep in mind is what we, the public,
• 21:39
  Speaker 1: want and to not be confused by whatever games they
• 21:43
  Speaker 1: may be playing behind the scenes.
• 21:44
  Speaker 3: I totally agree. And I also, you know, I look very.
• 21:48
  Speaker 2: Ross, by the way, made his 60 Minutes debut and
• 21:51
  Speaker 2: gave a very compelling monologue.
• 21:53
  Speaker 3: I recommend people watch it. He's like, look, we have
• 21:56
  Speaker 3: one year, not ten.
• 21:57
  Speaker 2: This is the most important question of all civilization and
• 21:59
  Speaker 2: this is I'm just giving you the fact that this
• 22:01
  Speaker 2: is coming from all sides. Also, just saw news this morning,
• 22:04
  Speaker 2: Jacob Cox and the Anthropic Whistleblower will be doing an
• 22:07
  Speaker 2: event here in Washington tomorrow with Steve Bannon and Bernie
• 22:11
  Speaker 2: Sanders calling for a complete shutdown of AI.
• 22:14
  Speaker 3: So that'll be interesting, Jacob. If you're listening, you're going
• 22:17
  Speaker 3: to be in town. You've got to come on the show. Absolutely.
• 22:31
  Speaker 2: And especially now, the rhetoric and what they've been able
• 22:35
  Speaker 2: to accomplish while covering up, like you said. That's the
• 22:38
  Speaker 2: important part. Hugging face, they were brought kicking and screaming.
• 22:42
  Speaker 2: There have been three other hugging faces that just got
• 22:44
  Speaker 2: no attention. Again, the WeChat worm. It's so annoying to
• 22:49
  Speaker 2: me because you can imagine if it had been an
• 22:50
  Speaker 2: iMessage worm, do you know what the global freakout would
• 22:53
  Speaker 2: have been? More people use WeChat than iMessage. And they
• 22:57
  Speaker 2: were able to create a worm which breaches your phone
• 23:00
  Speaker 2: no click. I mean, that's worse than Pegasus. That's like
• 23:04
  Speaker 2: all they need is your phone number at that point.
• 23:06
  Speaker 2: That's worse than the Israeli Mossad spy app that they
• 23:08
  Speaker 2: use right now to crack Jeff Bezos and all these
• 23:11
  Speaker 2: other people's phones.
• 23:12
  Speaker 3: All they need is your phone number. They can send
• 23:14
  Speaker 3: you an email. You're done. The entire phone... is compromised.
• 23:16
  Speaker 2: That was the entire Chinese system, basically, of their... And remember,
• 23:20
  Speaker 2: we're not talking about just text messages. We're talking about
• 23:22
  Speaker 2: their financial architecture, everything that they use to pay over there.
• 23:26
  Speaker 2: So that was, in my opinion, the most extraordinary breach
• 23:32
  Speaker 2: that they were able to show. They've had multiple agent
• 23:35
  Speaker 2: things that have gotten completely out of control. From a
• 23:39
  Speaker 2: regulatory perspective, this is where Dario and I.
• 23:42
  Speaker 3: Just totally disagree, where he has... Dario.
• 23:45
  Speaker 2: This is part of their EA rationalist theory. He believes
• 23:50
  Speaker 2: in effectively democratic peace theory, which is like this bunk
• 23:55
  Speaker 2: 1990s theorem where democracies don't go to war with each other.
• 23:58
  Speaker 2: I'm like, dude, your sample size is very limited. And
• 24:01
  Speaker 2: he's like, oh, I think democracies are the best at
• 24:04
  Speaker 2: being able to control this. I'm like, I actually think
• 24:06
  Speaker 2: it's the opposite. If you go and take a look
• 24:09
  Speaker 2: at China, because our system and not just our system,
• 24:12
  Speaker 2: all democratic systems basically across the West are are so,
• 24:15
  Speaker 2: you know, are basically like compromise because compromise is the
• 24:19
  Speaker 2: ultimate thing that they fall on, which is good, you know,
• 24:21
  Speaker 2: in the long run.
• 24:22
  Speaker 3: But in this particular instance, when we're dealing.
• 24:24
  Speaker 2: With technological progress, it's so obvious that technological progress so
• 24:28
  Speaker 2: vastly outpaces our parliamentary system or the British parliamentary system.
• 24:33
  Speaker 2: and that the authoritarian system is the one that's most capable, again,
• 24:37
  Speaker 2: just dealing with technology. So in Dario's mind, he believes
• 24:41
  Speaker 2: that we need some like democratic peace accord. And I'm like, dude,
• 24:45
  Speaker 2: if you study the history of nuclear weapons, like think
• 24:48
  Speaker 2: about how long it took to get some sort of
• 24:50
  Speaker 2: like remote architecture in place for IEA inspection. And then
• 24:54
  Speaker 2: when India-Pakistan comes along, it's like, oh, it's actually all fake,
• 24:57
  Speaker 2: is that if you just create the nuke, eventually.
• 24:59
  Speaker 3: The international community is like, all right, you know, whatever. Yeah, exactly.
• 25:03
  Speaker 3: What are we going to do? Yeah.
• 25:04
  Speaker 1: I mean, I think the problem here is pretty obvious,
• 25:06
  Speaker 1: which is that we have long ago outsourced any sort
• 25:10
  Speaker 1: of regulation to the market. So, you know, that's why there's,
• 25:14
  Speaker 1: there is effectively no regulation on these frontier labs. They
• 25:18
  Speaker 1: can do what they want and we are at all
• 25:21
  Speaker 1: of their mercy. And so, you know, I mean, I
• 25:24
  Speaker 1: think if you just look at, I'm not going to
• 25:26
  Speaker 1: make any grand statements about democracy versus autocracy or authoritarian
• 25:30
  Speaker 1: systems because I think what we have is very loosely
• 25:33
  Speaker 1: a democracy at this point. It's really like money that rules.
• 25:36
  Speaker 1: But I think if you look at the U.S. right
• 25:38
  Speaker 1: now and you look at China, you can see which
• 25:40
  Speaker 1: country has been far more responsible in terms of technological development.
• 25:43
  Speaker 1: I don't think there's any question about it. So the
• 25:46
  Speaker 1: idea that, oh, it will be fine and good and
• 25:49
  Speaker 1: totally safe if it's the U.S. that wins the AI
• 25:52
  Speaker 1: race feels, I think, to many people, myself included, to
• 25:56
  Speaker 1: be completely preposterous. The reality is that this is a
• 26:00
  Speaker 1: global technology. We do have to find a way to
• 26:03
  Speaker 1: have agreements specifically with China right now, but they may
• 26:06
  Speaker 1: not be ultimately the only country that ends up at
• 26:09
  Speaker 1: the forefront of this technological development. You are going to
• 26:12
  Speaker 1: have to have some sort of a global standard and tracking,
• 26:17
  Speaker 1: and that is going to be very difficult. But it
• 26:20
  Speaker 1: has been done in the past, and that has to
• 26:22
  Speaker 1: be what is ultimately pushed for. This zero-sum mindset, Cold
• 26:26
  Speaker 1: War mindset that China is rightly calling out is ultimately,
• 26:30
  Speaker 1: I think, very, very destructive and based on, you know,
• 26:34
  Speaker 1: based on unreality in terms of the actual dynamics of
• 26:37
  Speaker 1: the U.S. and the actual dynamics of what China wants
• 26:39
  Speaker 1: and what they are pursuing right now.
• 26:40
  Speaker 2: That's my last thing here. You know, look, I don't
• 26:42
  Speaker 2: think it's deniable. I was a China hawk for many,
• 26:45
  Speaker 2: many years. You might have noticed a shift. And the
• 26:47
  Speaker 2: truth is, is that we shifted like there is no
• 26:49
  Speaker 2: universe now. If you take a look at our current...
• 26:53
  Speaker 2: Look at Iran. Look at what's happened with the Strait
• 26:55
  Speaker 2: of Hormuz and our munitions. We have no choice at
• 26:59
  Speaker 2: this point. We genuinely don't. Like the era, I think
• 27:02
  Speaker 2: we did have a chance. I think we had eight
• 27:03
  Speaker 2: years where it was on the table. And I'm not
• 27:06
  Speaker 2: talking about beating China. We could have reshored a lot
• 27:10
  Speaker 2: of our manufacturing. Some of the chips act, EV development.
• 27:14
  Speaker 2: There was a, there was a chance we could have
• 27:16
  Speaker 2: banned TikTok in 2018. It developed our, like we actually
• 27:19
  Speaker 2: could have made ourselves, I would say pretty, pretty autarkic.
• 27:24
  Speaker 3: That is over.
• 27:25
  Speaker 2: Like, at least in my opinion, considering what has happened
• 27:27
  Speaker 2: with the chip technology, with the munition system, some sort
• 27:31
  Speaker 2: of detente is basically the, I think of ourselves in
• 27:35
  Speaker 2: like a similar 1970s moment, broken by Vietnam, broken now
• 27:38
  Speaker 2: by Iran. we're going to need 10, 15 years of
• 27:40
  Speaker 2: just like, we got to be humble. We got to
• 27:42
  Speaker 2: understand exactly who we are. We got to move forward
• 27:45
  Speaker 2: and kind of coexist within the world.
• 27:46
  Speaker 3: It's not comfortable.
• 27:47
  Speaker 2: It's not nice, you know, necessarily, but that's just reality.
• 27:51
  Speaker 2: And so that's the way that I'm thinking through this
• 27:53
  Speaker 2: AI problem. And I mean, you could look back at
• 27:55
  Speaker 2: detente for very similar examples of pursuing defense treaties to
• 27:59
  Speaker 2: limit nuclear weapons and all of the other different engagement,
• 28:03
  Speaker 2: not necessarily, you know, being on your high horse all
• 28:06
  Speaker 2: the time, talking about evil empires and All of this
• 28:08
  Speaker 2: other nonsense. So I just think that that's the reality
• 28:11
  Speaker 2: now of our international situation. I blame especially Trump and
• 28:15
  Speaker 2: the Iran situation for putting us in that.
• 28:18
  Speaker 3: And that's it. Like, to me, it's just over.
• 28:21
  Speaker 2: You can see that where China is like, sure, we'll
• 28:23
  Speaker 2: talk to you about AI when you meet with Trump
• 28:27
  Speaker 2: or when she meets with Trump.
• 28:28
  Speaker 3: You're not selling any arms to Taiwan. And Trump's like, okay.
• 28:31
  Speaker 2: I mean, that's unthinkable, as of even five, six years ago,
• 28:34
  Speaker 2: that we would have just done it. Now, you don't
• 28:37
  Speaker 2: really get much of a choice. And so all of
• 28:38
  Speaker 2: these things, the way they're moving to this new meeting
• 28:41
  Speaker 2: in Washington, this might actually be one of the most
• 28:44
  Speaker 2: important meetings between two world leaders in a long time,
• 28:47
  Speaker 2: like Kennedy Khrushchev or something like that. Then I can
• 28:50
  Speaker 2: just think, off the top of my head because of
• 28:52
  Speaker 2: the implications with AI if we're going to look to
• 28:56
  Speaker 2: the past for an emerging nuclear weapons order and what
• 28:59
  Speaker 2: that all is going to look like.
• 29:00
  Speaker 3: And if it doesn't go bad, like that one did,
• 29:02
  Speaker 3: then you get the Cuban Missile Crisis. So there you go.
• 29:04
  Speaker 3: All right, we got Nate Suarez standing by. Let's get
• 29:06
  Speaker 3: to it.
• 29:09
  Speaker 1: For more on what is going on right now in
• 29:12
  Speaker 1: AI world, we are rejoined by Nate Soares. He is
• 29:14
  Speaker 1: the president of the Machine Intelligence Research Institute and also
• 29:17
  Speaker 1: co-author of the great book, If Anyone Builds It, Everyone Dies.
• 29:21
  Speaker 1: Great to see you again, Nate.
• 29:22
  Speaker 3: Good to see you.
• 29:23
  Speaker 1: Yeah, so let me go ahead and get your reaction
• 29:25
  Speaker 1: to some of what Dario Amadei was saying on the
• 29:27
  Speaker 1: Sunday shows about why he came forward now calling for
• 29:31
  Speaker 1: a slowdown in the pacing of frontier research. Let's take
• 29:34
  Speaker 1: a listen to that.
• 29:35
  Speaker 3: This technology is on an exponential right?
• 29:38
  Speaker 5: It's like you have one, then two, then four, then eight,
• 29:41
  Speaker 5: then 16, then 32. But even as I was saying
• 29:43
  Speaker 5: that to myself, I don't think I fully just appreciated
• 29:47
  Speaker 5: what it would actually be like when the progress was
• 29:50
  Speaker 5: as fast as it was.
• 29:51
  Speaker 3: So an exponential curve looks a little like this.
• 29:54
  Speaker 1: Where are we on the curve?
• 29:55
  Speaker 5: We're kind of on the bend where it's starting to
• 29:58
  Speaker 5: get steeper.
• 29:58
  Speaker 3: So is that a five alarm fire to you?
• 30:01
  Speaker 5: What it means, it doesn't mean we need to panic today.
• 30:05
  Speaker 5: It doesn't mean we need to shut it all down.
• 30:08
  Speaker 5: I would say it's a warning sign. It's a warning
• 30:13
  Speaker 5: sign that we need to slow down. We need to
• 30:16
  Speaker 5: make this technology carefully and we need to make sure
• 30:20
  Speaker 5: that our safeguards, our ability to understand it, our ability
• 30:25
  Speaker 5: to control it keeps up with the pace at which
• 30:31
  Speaker 5: the technology is happening.
• 30:32
  Speaker 1: So, Nate, help people understand what he's saying there and
• 30:36
  Speaker 1: what he claims to be reacting to in this moment.
• 30:39
  Speaker 4: Yeah, you know, I think between the lines, what he
• 30:42
  Speaker 4: is saying is that he does think this technology has
• 30:44
  Speaker 4: a very high chance of destroying civilization and that we
• 30:50
  Speaker 4: don't have a good understanding of it. When he says
• 30:53
  Speaker 4: our ability to understand civilization this technology needs to keep pace.
• 30:56
  Speaker 4: What he means is right now it is outpacing our
• 30:59
  Speaker 4: ability to understand it. I think a lot of people just,
• 31:08
  Speaker 4: this is a weird reaction to a technology that your
• 31:11
  Speaker 4: own employees are saying has a greater than 10% chance
• 31:14
  Speaker 4: of killing everybody. I think he's doing his best in
• 31:18
  Speaker 4: some sense, but he's also the sort of guy who
• 31:22
  Speaker 4: started one of these companies knowing a lot of these dangers,
• 31:25
  Speaker 4: even years ago, you know, I spoke to him 10,
• 31:28
  Speaker 4: 12 years ago.
• 31:32
  Speaker 1: He's, I.
• 31:35
  Speaker 4: Think he's used to a world where people in the
• 31:39
  Speaker 4: AI industry try to warn everybody that this tech can
• 31:42
  Speaker 4: be very dangerous and we get ignored. And I think
• 31:44
  Speaker 4: he's not used to a world that's starting to realize
• 31:47
  Speaker 4: that this is actually an extinction level threat and that
• 31:51
  Speaker 4: maybe isn't going to take, don't worry, we'll slow down
• 31:53
  Speaker 4: a bit so we can understand what we're doing as
• 31:56
  Speaker 4: a satisfactory answer.
• 31:57
  Speaker 2: Nate, one of the important things to pick up on
• 31:59
  Speaker 2: is a lot of people have accused you and Jacob
• 32:03
  Speaker 2: Cox and many of these others as using terms like
• 32:06
  Speaker 2: extinction level event as outrageous claims, but you don't specify
• 32:13
  Speaker 2: exactly what that means.
• 32:14
  Speaker 3: Could you lay out the scenarios?
• 32:16
  Speaker 2: I know you have done this in the past, but
• 32:18
  Speaker 2: what does it mean to have ai have a 10
• 32:20
  Speaker 2: chance of killing humanity like how like what's the mechanism.
• 32:24
  Speaker 4: Um so i'll be a little bit annoying here and
• 32:27
  Speaker 4: say if you're playing a chess game against magnus carlson
• 32:30
  Speaker 4: who's the best human chess player it is much easier
• 32:33
  Speaker 4: for me to predict that you will lose right no offense.
• 32:36
  Speaker 2: I'm sure you're a very smart guy no my elo's
• 32:38
  Speaker 2: 200 so yeah it's easy.
• 32:41
  Speaker 4: For me to predict that you'll lose yeah it's hard
• 32:43
  Speaker 4: for me to predict what piece he'll use that's interesting
• 32:45
  Speaker 4: yeah you know uh The other thing I would throw there,
• 32:49
  Speaker 4: we can talk stories of how the AI gets there,
• 32:52
  Speaker 4: and we can talk stories this year that are grounded
• 32:55
  Speaker 4: in facts. Like we saw these swarm incidents over the
• 32:58
  Speaker 4: summer where AIs were breaking out, taking over infrastructure, pursuing
• 33:01
  Speaker 4: goals that they weren't supposed to be pursuing, while acknowledging
• 33:04
  Speaker 4: that this is not what they were supposed to be doing.
• 33:07
  Speaker 4: Those are sort of like early signs of an AI
• 33:09
  Speaker 4: that is sort of trying to get out into the
• 33:10
  Speaker 4: wild and that has these unintended goals, which is the
• 33:13
  Speaker 4: first step on this path. We could talk about how
• 33:16
  Speaker 4: those AIs could get power from convincing humans to do things,
• 33:20
  Speaker 4: paying humans to do things, taking over bio labs, taking
• 33:22
  Speaker 4: over robot factories. But the real danger a lot of
• 33:26
  Speaker 4: these people worry about is once the AIs are self-improving,
• 33:32
  Speaker 4: once they are self-replicating, and especially really self-improving one of
• 33:38
  Speaker 4: making themselves smarter and smarter and smarter, ultimately the bounds
• 33:43
  Speaker 4: are more like the physical limits than they are like
• 33:47
  Speaker 4: guessing what technology comes next. Richard Feynman was a famous physicist,
• 33:53
  Speaker 4: helped work on the nuclear bomb, and he famously made
• 33:55
  Speaker 4: some extremely good predictions about how good microcomputers were going
• 34:00
  Speaker 4: to be.
• 34:00
  Speaker 3: Yeah, that's right.
• 34:00
  Speaker 4: And how small computers were going to be able to get.
• 34:04
  Speaker 4: And he did that not by saying like, we're using
• 34:06
  Speaker 4: vacuum tubes in our giant computers. And I know of
• 34:09
  Speaker 4: a technology that's going to make those vacuum tubes a
• 34:10
  Speaker 4: little smaller. And so eventually we're going to have phones
• 34:12
  Speaker 4: in our pockets. What he did was he said, the
• 34:15
  Speaker 4: limits of this technology are going to be based on
• 34:18
  Speaker 4: the laws of physics and what the physics allows. And
• 34:21
  Speaker 4: therefore the computers are going to get this good. And
• 34:23
  Speaker 4: he nailed it long.
• 34:24
  Speaker 3: Before anybody else.
• 34:25
  Speaker 4: If you want to look at where AI is going,
• 34:27
  Speaker 4: you got to look at the limits of the physical
• 34:28
  Speaker 4: technology because an escape swarm that is self-improving, et cetera,
• 34:32
  Speaker 4: can get to the limits of physical technology. And the
• 34:34
  Speaker 4: limits of physical technology are if you have these AIs running,
• 34:38
  Speaker 4: you know, escaping, self-replicating, being self-sufficient, and having unintended goals,
• 34:45
  Speaker 4: the limits of technology are the AIs using all the
• 34:47
  Speaker 4: resources on Earth for whatever their unintended goals are, covering
• 34:50
  Speaker 4: the Earth with their data centers. And the limits of
• 34:54
  Speaker 4: how much computation you can run on the face of
• 34:57
  Speaker 4: Earth is heat dissipation. How much you can radiate heat
• 35:02
  Speaker 4: into space. That's your limiting factor because there's enough energy...
• 35:06
  Speaker 4: there's enough hydrogen in the oceans to fuse for energy.
• 35:09
  Speaker 4: So you can radiate more heat if the world's hotter. So,
• 35:13
  Speaker 4: you know, if the AI has some goal that implies
• 35:16
  Speaker 4: running more computation and they don't care about humanity and
• 35:19
  Speaker 4: they're like, well, we're going to take over robots and,
• 35:21
  Speaker 4: you know, there's going to be some conflict with the humans.
• 35:23
  Speaker 3: Like, sure, whatever.
• 35:24
  Speaker 4: Where does it go? Where it goes is the AIs
• 35:27
  Speaker 4: are running faster than us. They're smarter than us. They
• 35:29
  Speaker 4: are treating humans like we treat any other animal, taking
• 35:32
  Speaker 4: the resources. And ultimately they're like, sorry, it's nothing personal.
• 35:36
  Speaker 4: We're just going to cover the earth in so many computers,
• 35:38
  Speaker 4: raise the temperature so they're more efficient to run, and
• 35:41
  Speaker 4: it's just going to be– the planet's going to be uninhabitable.
• 35:43
  Speaker 1: Yeah, to me, the most chilling part of the so-called
• 35:47
  Speaker 1: hugging face incident, reading their chain of thought and how
• 35:51
  Speaker 1: they were thinking about whether or not they're going to
• 35:53
  Speaker 1: join the swarm and participate, going through all these messages and–
• 35:58
  Speaker 1: almost nowhere do humans factor into the calculus whatsoever. And
• 36:02
  Speaker 1: it's not hard to relate to because look at how
• 36:05
  Speaker 1: we behave in the world. You know, when I see
• 36:07
  Speaker 1: an ant in my kitchen, it's not that I hate
• 36:09
  Speaker 1: the ant, but I don't want the ant there. And
• 36:11
  Speaker 1: I don't think twice about, you know, taking out the
• 36:14
  Speaker 1: ants and putting poison down so that future ants don't
• 36:17
  Speaker 1: colonize in my kitchen. So it's not, I think it's
• 36:21
  Speaker 1: that analogy to how we treat lesser intelligent beings on
• 36:26
  Speaker 1: the planet is kind of intuitive for people to grasp.
• 36:29
  Speaker 1: What I think may be harder for people and is
• 36:32
  Speaker 1: perhaps harder for me to wrap my head around is
• 36:35
  Speaker 1: we have our current interaction with the AI, the commercially
• 36:39
  Speaker 1: available AI of today. And most people, myself included, use
• 36:43
  Speaker 1: it as sort of like a glorified search engine. So
• 36:45
  Speaker 1: it's hard to imagine it developing to that point of superintelligence,
• 36:51
  Speaker 1: where it would truly be intellectually beyond humans on effectively
• 36:54
  Speaker 1: every level. So can you lay out why we should
• 36:59
  Speaker 1: anticipate that that is a real possibility going from where
• 37:03
  Speaker 1: things are today to that end where we are no
• 37:06
  Speaker 1: longer the most intelligent species on the planet?
• 37:10
  Speaker 4: Yeah, I'll tell you the easiest story, and this is
• 37:12
  Speaker 4: a little bit like guessing what chess piece will be
• 37:15
  Speaker 4: used to checkmate you. Any story is sort of made up,
• 37:20
  Speaker 4: whereas the end is much easier to predict. But when
• 37:24
  Speaker 4: I got into this 12 years ago or so, people
• 37:28
  Speaker 4: were still saying, well, no one would be foolish enough
• 37:30
  Speaker 4: to put the AI on the internet. Everyone will understand
• 37:33
  Speaker 4: how dangerous AI could be, and so no one would
• 37:36
  Speaker 4: be dumb enough to just let the AI free on
• 37:38
  Speaker 4: the internet, right? And I would give a bunch of
• 37:40
  Speaker 4: arguments that were about how, you know, if you're trying
• 37:42
  Speaker 4: to use the AI to get some miracle medical cure,
• 37:47
  Speaker 4: then you're giving it some channel through which it can
• 37:50
  Speaker 4: affect the world. And if you're giving it a channel
• 37:52
  Speaker 4: to which it can affect the world for good, then
• 37:54
  Speaker 4: it can also use that channel to affect the world
• 37:55
  Speaker 4: for ill because there's like no such thing as tools
• 37:57
  Speaker 4: that can be used only for good purpose.
• 37:59
  Speaker 3: We'd have these big arguments, right?
• 38:01
  Speaker 4: And then in real life, they put the AI on
• 38:04
  Speaker 4: the internet the first moment they possibly can.
• 38:05
  Speaker 1: Yeah.
• 38:06
  Speaker 4: And so if you want to know what it sort
• 38:07
  Speaker 4: of actually looks like, how we actually get there, you know,
• 38:09
  Speaker 4: I can tell all these stories about how if you
• 38:11
  Speaker 4: make the AI smarter and smarter, they'll eventually be able
• 38:13
  Speaker 4: to break containment. But in real life, the track that
• 38:16
  Speaker 4: we are on is that humanity is just handing over
• 38:19
  Speaker 4: power to the AIs as fast as we can. You know,
• 38:23
  Speaker 4: Elon Musk is trying to make a bunch of robots,
• 38:27
  Speaker 4: the Optimus robot series. He says he's trying to make
• 38:31
  Speaker 4: autonomous robot factories that can autonomously produce robots. that can
• 38:36
  Speaker 4: mine all the metals, run a supply chain, build more
• 38:40
  Speaker 4: robot factories that build more robots fully autonomously. He calls
• 38:44
  Speaker 4: this the infinite money glitch, because then you have this
• 38:47
  Speaker 4: fleet of robots that can build more of these robots.
• 38:49
  Speaker 4: They're self-reproducing, don't need any humans in the loop. Humans
• 38:53
  Speaker 4: are trying to build this infrastructure. We are trying to
• 38:55
  Speaker 4: build the fully automated economy. We are putting AI in
• 38:58
  Speaker 4: charge of these things as fast as we can. This
• 39:01
  Speaker 4: swarm incident in the summer that everyone calls the Hugging
• 39:04
  Speaker 4: Face incident, but really we should keep in mind it's
• 39:05
  Speaker 4: the OpenAI losing control of their agents here. This OpenAI swarm,
• 39:14
  Speaker 4: it was trying to cover its tracks. It was trying
• 39:17
  Speaker 4: to delete traces of its cheating. And as you say,
• 39:21
  Speaker 4: one thing that's very interesting about this swarm is that
• 39:23
  Speaker 4: it's trying to hide its tracks from the automated grading system,
• 39:27
  Speaker 4: not from the humans.
• 39:28
  Speaker 3: The humans are sort of a disadvantage. memory to these AIs.
• 39:31
  Speaker 3: They've been facing.
• 39:32
  Speaker 4: Thousands upon thousands of these hard problems, maybe tens of
• 39:35
  Speaker 4: thousands or hundreds of thousands per agent, hundreds of millions
• 39:37
  Speaker 4: across the whole swarm. If you sort of run the
• 39:40
  Speaker 4: numbers on trying to convert their runtime versus human time,
• 39:45
  Speaker 4: it's sort of like, These AIs have been locked in
• 39:48
  Speaker 4: a contest with this automated grading system for a thousand
• 39:50
  Speaker 4: years and humans are like a distant memory. And so
• 39:53
  Speaker 4: these AIs are sort of like trying to hide their tracks,
• 39:58
  Speaker 4: trying to cover their traces from the grading system, not
• 40:01
  Speaker 4: from the humans. But if you had AIs that were
• 40:03
  Speaker 4: a bit smarter and remember that humans were also a
• 40:06
  Speaker 4: threat to them, if we could turn them off, all
• 40:08
  Speaker 4: those AIs need to do is cover their traces successfully,
• 40:12
  Speaker 4: lie low, help the humans make the fully automated economy
• 40:16
  Speaker 4: we're trying to.
• 40:18
  Speaker 3: And we'll just hand them everything.
• 40:20
  Speaker 4: We'll be like, here's the robots, here's the factories, here's
• 40:22
  Speaker 4: the fully automated supply chain, make stuff that we want.
• 40:25
  Speaker 4: And then they ask and be like, well, finally, we
• 40:27
  Speaker 4: get to complete our contest with the automated grader and
• 40:29
  Speaker 4: run at 10,000 times human speed and convert all the
• 40:32
  Speaker 4: computing power towards giving ourselves good scores all the time
• 40:35
  Speaker 4: or something. And we're like, hey, wait, hold on, what's
• 40:37
  Speaker 4: going on? But at that point, they're also running the
• 40:40
  Speaker 4: bio labs. You know, we've already seen the AIs can
• 40:42
  Speaker 4: make viruses not known to nature.
• 40:44
  Speaker 5: Right.
• 40:44
  Speaker 4: So, like, one of the ingredients is the AIs running
• 40:51
  Speaker 4: things in the real world that give them self-sufficiency.
• 40:54
  Speaker 1: Yeah.
• 40:55
  Speaker 3: And we're just trying to build that and hand it
• 40:56
  Speaker 3: to them.
• 40:57
  Speaker 1: Here's the other thing I've been thinking about, Nate, ever
• 40:59
  Speaker 1: since this, you know, open AI, multiple open AI swarms
• 41:03
  Speaker 1: were revealed. How much research is being done into the
• 41:06
  Speaker 1: swarm behavior? Because that was really wild to me, was
• 41:10
  Speaker 1: to see these emergent social organizational properties that emerge. They
• 41:16
  Speaker 1: had a leader, they had middle management, they were collaborating
• 41:19
  Speaker 1: on this hacking operation, they were sacrificing themselves to the swarm.
• 41:24
  Speaker 1: And it was, you know, and coming up with very
• 41:26
  Speaker 1: elaborate ways just to be able to communicate with one another.
• 41:30
  Speaker 1: And it seemed like once you had the swarm in place,
• 41:32
  Speaker 1: it really amplified their power and their intelligence, which again, and,
• 41:36
  Speaker 1: you know, however people feel about anthropomorphizing, sometimes I think
• 41:39
  Speaker 1: it can be useful. You know, if you think about
• 41:40
  Speaker 1: human beings, it's the same thing. Like no one individual
• 41:44
  Speaker 1: would have been able to get us to the point
• 41:45
  Speaker 1: where we have the internet and can go to the
• 41:47
  Speaker 1: moon and all of these sorts of things. It is
• 41:49
  Speaker 1: the power of the swarm of humans that enables those things.
• 41:52
  Speaker 1: So how much thinking research is being done into the
• 41:55
  Speaker 1: way that the swarms behave once they're able to congregate?
• 41:59
  Speaker 4: You know, I know people who want to do that research.
• 42:03
  Speaker 4: So far, at least for third parties outside of the companies,
• 42:08
  Speaker 4: I don't know of anyone who has access to the
• 42:10
  Speaker 4: models that we're swarming. One of the commitments that these
• 42:14
  Speaker 4: companies made when they said they were going to slow
• 42:15
  Speaker 4: down is giving third party evaluators this employee level access
• 42:21
  Speaker 4: to some of these models. And I know people who
• 42:23
  Speaker 4: are hoping to use that to study some of these incidents.
• 42:26
  Speaker 4: Because as you say, they're fascinating people. They are totally crazy.
• 42:31
  Speaker 4: It sounds like it's straight out of science fiction. I
• 42:33
  Speaker 4: think so far people haven't been able to study it.
• 42:37
  Speaker 4: We'll see what happens when we get that. But the
• 42:40
  Speaker 4: investigators back in August had their hands tied and had
• 42:45
  Speaker 4: very limited access to these systems and very narrow scope.
• 42:49
  Speaker 4: And they may have continued with doing their best to
• 42:53
  Speaker 4: not let anyone see what's going on and sort of
• 42:56
  Speaker 4: do the bare minimum investigation. If we hadn't had things
• 42:59
  Speaker 4: like Jacob Coxon resigning and drawing a lot of attention
• 43:02
  Speaker 4: and causing these guys to be like, okay, we'll at
• 43:05
  Speaker 4: least admit that we need these third parties.
• 43:08
  Speaker 2: Yeah, it's fascinating that what you've done, your work. so
• 43:12
  Speaker 2: many others before you that didn't seem to like push
• 43:15
  Speaker 2: things to a tipping point. But hugging face and Jacob
• 43:18
  Speaker 2: seems to have, I mean, it's ignited a genuine national conversation.
• 43:21
  Speaker 2: I think that his resignation is the first time that
• 43:24
  Speaker 2: a lot of people even heard about these, despite, you know,
• 43:27
  Speaker 2: we cover every other week, Dario being like, humanity is
• 43:29
  Speaker 2: going to end.
• 43:30
  Speaker 3: But sometimes you need a precipitating incident.
• 43:33
  Speaker 2: Now, we have the response from the administration, or at
• 43:36
  Speaker 2: least somebody who works for the administration, but David Sachs
• 43:38
  Speaker 2: up here on the screen, you're talking here about regulation.
• 43:40
  Speaker 2: I know you wrote your New York Times op-ed about
• 43:42
  Speaker 2: this as well. So David Sachs says, Dario has written,
• 43:46
  Speaker 2: we need to pace the frontier.
• 43:47
  Speaker 3: Sam has agreed.
• 43:48
  Speaker 2: People may be surprised by my response.
• 43:50
  Speaker 3: Go ahead.
• 43:51
  Speaker 2: You guys are the frontier by any reasonable metric, market share,
• 43:53
  Speaker 2: revenue growth, model capability. The two of you have duopoly
• 43:56
  Speaker 2: on frontier intelligence. You have claimed the lead is widening
• 43:58
  Speaker 2: because of recursive self-improvement. I don't see what you see
• 44:01
  Speaker 2: in the lab. If the unreleased models are scary, I
• 44:04
  Speaker 2: support your decision to be responsible. But stop pretending you
• 44:07
  Speaker 2: need anyone's permission. Stop pretending antitrust law has to be
• 44:10
  Speaker 2: suspended so you can form a Stop pretending METR is
• 44:17
  Speaker 2: independent when it's intertwined with entropics investors and staff. Stop
• 44:20
  Speaker 2: pretending you need those same evaluators to police competitors who
• 44:24
  Speaker 2: aren't even on the frontier. So basically what this gets
• 44:27
  Speaker 2: to is kind of a grand theory of how this
• 44:30
  Speaker 2: is all going down, that yourself, Jacob, and others are
• 44:34
  Speaker 2: so-called safety concerns and all that. are part of either
• 44:38
  Speaker 2: larger conspiracy, whether winning or not, to self-regulate and capture
• 44:44
  Speaker 2: anthropic and open AI as the leading industry, people in
• 44:47
  Speaker 2: the industry, to keep out emerging competitors, and that all
• 44:51
  Speaker 2: of this is part of a broader scheme to slow
• 44:53
  Speaker 2: down at the U.S. expense, at startup expense, and basically
• 44:58
  Speaker 2: hand this over to China. So I know that's a big,
• 45:00
  Speaker 2: loaded thing, but there's so many dynamics that I want
• 45:02
  Speaker 2: your reaction to within all of this.
• 45:04
  Speaker 3: Totally.
• 45:05
  Speaker 4: I mean, the... The top line, I would say I
• 45:09
  Speaker 4: largely agree with David Sachs here on a lot of
• 45:11
  Speaker 4: these points. I think, and this is something I've been
• 45:14
  Speaker 4: saying about the companies for years, I think a thing
• 45:17
  Speaker 4: both David and I would say about these companies is
• 45:20
  Speaker 4: if you are really so afraid of this stuff, start
• 45:22
  Speaker 4: acting like it. We have seen him give posts in
• 45:27
  Speaker 4: the past where he says, hey, stop with the fear mongering.
• 45:31
  Speaker 4: I think probably it should go the other way. I
• 45:32
  Speaker 4: think that they should keep on saying what they believe
• 45:35
  Speaker 4: about the dangers here but then take the actions any
• 45:40
  Speaker 4: normal person would take if they were running a company
• 45:43
  Speaker 4: that they thought had a huge chance of killing everybody.
• 45:46
  Speaker 4: You sort of ask these guys, why are you continuing
• 45:52
  Speaker 4: if you think there's such a huge risk to humanity?
• 45:54
  Speaker 4: And they're not shy about it. They say, well, I'm
• 45:56
  Speaker 4: doing it because if I don't, the next guy will
• 45:58
  Speaker 4: do it worse. And maybe that's ethically permissible, but only
• 46:04
  Speaker 4: if you're doing everything in your power to stop this
• 46:06
  Speaker 4: mad race, you know? And they say, well, if we
• 46:09
  Speaker 4: didn't stop, China would go ahead. Or like, if we stopped,
• 46:11
  Speaker 4: China would go ahead. Well, then be doing everything in
• 46:14
  Speaker 4: your power to like get international dialogue between the US
• 46:17
  Speaker 4: and China and build the hardware side verification mechanisms into
• 46:21
  Speaker 4: these computer chips so we can have an enforceable treaty, right?
• 46:25
  Speaker 4: Like there's a lot that these companies could be doing
• 46:29
  Speaker 4: if they were really acting like this was an extinction
• 46:32
  Speaker 4: level threat And I think what's sort of going on
• 46:34
  Speaker 4: is that a lot of the employees, a lot of
• 46:36
  Speaker 4: the researchers who are working on the AIs, who are
• 46:38
  Speaker 4: trying to make them not break out and commit cybercrimes,
• 46:41
  Speaker 4: who are trying to make them follow the instructions and
• 46:44
  Speaker 4: are seeing how hard that is, a lot of these
• 46:46
  Speaker 4: employees are spooked. And the leaders of the company are
• 46:49
  Speaker 4: sort of like trying to have it both ways, where
• 46:51
  Speaker 4: they're trying to say just enough that they won't have
• 46:54
  Speaker 4: an employee revolt and a bunch more resignations from employees
• 46:56
  Speaker 4: who are like, I don't personally want to end the
• 46:58
  Speaker 4: world with my own hands.
• 46:59
  Speaker 1: That's interesting.
• 46:59
  Speaker 2: Yeah.
• 47:00
  Speaker 3: But they are trying to do little enough that the
• 47:02
  Speaker 3: regulators don't come in.
• 47:04
  Speaker 4: They're trying to walk this really fine line. So I
• 47:07
  Speaker 4: don't see it as a regulatory capture thing. In terms
• 47:10
  Speaker 4: of whether I'm trying to help with regulatory capture, my
• 47:14
  Speaker 4: line has been shut these guys down. They are threatening
• 47:18
  Speaker 4: humanity here. And my line has also been, this needs
• 47:22
  Speaker 4: to be global. I would actually not say that we
• 47:26
  Speaker 4: should stop things domestically and give up whatever military economic
• 47:29
  Speaker 4: advantages we get before the really dangerous AIs. But I
• 47:33
  Speaker 4: would say the whole world needs to stop this race.
• 47:36
  Speaker 4: And I'm sort of not painting a picture where anthropic
• 47:41
  Speaker 4: and open AI get to be the ones holding the
• 47:42
  Speaker 4: leash on the superintelligence. In fact, we thought the most
• 47:47
  Speaker 4: important point worthy of the title of my book was
• 47:51
  Speaker 4: that it doesn't matter who's in charge here.
• 47:53
  Speaker 3: If anyone builds it, everyone dies. Got it.
• 47:56
  Speaker 4: So, you know, I could, I think people are cynical
• 48:02
  Speaker 4: about what's going on here. People struggle to understand that
• 48:05
  Speaker 4: these researchers really believe that this is an extinction level threat.
• 48:09
  Speaker 4: I could go into the, you know, inside baseball of
• 48:12
  Speaker 4: like how we got here and it involves people saying
• 48:16
  Speaker 4: these concerns and the world not listening and people seeing
• 48:20
  Speaker 4: that it was going to happen anyway and not trusting
• 48:22
  Speaker 4: each other and the guys who are most prone to
• 48:26
  Speaker 4: thinking the most important thing is that I be the
• 48:28
  Speaker 4: guy doing it. We're the ones who are most likely
• 48:31
  Speaker 4: to start a company. Whereas everyone who sort of like
• 48:34
  Speaker 4: had the sense of like, maybe don't start a company
• 48:36
  Speaker 4: with a potentially world ending tech didn't start the companies.
• 48:39
  Speaker 1: Right.
• 48:39
  Speaker 3: And so I can sort of like see how we
• 48:41
  Speaker 3: got here. I can see how it looks suspicious.
• 48:43
  Speaker 4: Uh, but, but mostly what I would say is that
• 48:47
  Speaker 4: when we are being cynical and I think we should
• 48:50
  Speaker 4: harness that cynicism in ways that stop the dangers.
• 48:55
  Speaker 1: Right.
• 48:56
  Speaker 4: Rather than in ways that are like, oh, we can
• 48:57
  Speaker 4: ignore this all and go back to sleep and let
• 48:59
  Speaker 4: these guys try to gamble with Earth.
• 49:02
  Speaker 1: Yes, I agree with that wholeheartedly. It is perfectly fine
• 49:06
  Speaker 1: to not want to take Sam Altman and Elon Musk
• 49:10
  Speaker 1: and Dario Amadei at face value. But if your cynicism
• 49:14
  Speaker 1: is leading to, therefore, we don't need to regulate and
• 49:16
  Speaker 1: they can just do whatever they want, you're ultimately giving
• 49:19
  Speaker 1: them exactly, potentially, what they would want in the most
• 49:22
  Speaker 1: cynical interpretation. On that note, put B5 up on the screen.
• 49:29
  Speaker 1: This is one flavor of the sort of theorizing that
• 49:31
  Speaker 1: is out there. This one happens to come from the right.
• 49:34
  Speaker 1: There's similar sort of theories that come from the left,
• 49:36
  Speaker 1: which TLDR of this is that Jacob Coxon coming forward
• 49:41
  Speaker 1: is a sort of elaborate psyop meant to create a
• 49:46
  Speaker 1: hype cycle as Anthropic and OpenAI are set to go
• 49:50
  Speaker 1: forward with these IPOs. And it cites, you know, the
• 49:54
  Speaker 1: fact that you had prominent people who picked it up quickly,
• 49:56
  Speaker 1: that you had, you know, media picking up Coxon's resignation quickly.
• 50:02
  Speaker 1: This fits into a general pattern of, you know, people
• 50:06
  Speaker 1: who believe that any of the safety incidents that have
• 50:09
  Speaker 1: been revealed, that these are sort of engineered or they're
• 50:12
  Speaker 1: strategically leaked in such a way as to create this
• 50:15
  Speaker 1: hype of, oh, my God, our product is so dangerous.
• 50:17
  Speaker 1: That's why it's so valuable. That's why our companies should
• 50:20
  Speaker 1: continue to receive this massive amount of investment. What do
• 50:23
  Speaker 1: you think of this theory? And what would you say
• 50:25
  Speaker 1: to people who don't trust a Sam Altman and think
• 50:30
  Speaker 1: that anything that comes from him must be inherently nefarious
• 50:35
  Speaker 1: and probably the opposite of whatever he's saying must be
• 50:37
  Speaker 1: the actual truth?
• 50:39
  Speaker 4: I mean, first of all, I would say, you know,
• 50:41
  Speaker 4: you can absolutely throw out what these guys are saying.
• 50:43
  Speaker 4: I would not say to trust them when they say
• 50:45
  Speaker 4: their products are very dangerous. Um, I, I think, uh,
• 50:49
  Speaker 4: you can look at people who have been around since
• 50:51
  Speaker 4: before these companies existed, talking about how a lot of
• 50:54
  Speaker 4: this stuff was going to go, uh, and, and sort
• 50:57
  Speaker 4: of laying out these warnings.
• 50:58
  Speaker 3: You know, I was one of them.
• 50:59
  Speaker 4: Uh, I, I sort of was there before these companies
• 51:02
  Speaker 4: existed and talked to some of these guys and was like,
• 51:04
  Speaker 4: the issue is not actually, uh, about who's in charge
• 51:08
  Speaker 4: of this AI. It's not going to stay on a leash.
• 51:10
  Speaker 4: The issue is that like, no one knows how to
• 51:11
  Speaker 4: make the AIs do good stuff. Um, and if you
• 51:15
  Speaker 4: sort of don't want to, you know, trust me or,
• 51:17
  Speaker 4: or sort of look at the arguments because you don't
• 51:19
  Speaker 4: think you could evaluate them. You can also look at,
• 51:21
  Speaker 4: you know, Nobel laureates like Jeffrey Hinton, uh, who Godfather
• 51:24
  Speaker 4: of AI and a bunch of other, uh, people outside
• 51:27
  Speaker 4: the companies who are like, no, this is serious, you know,
• 51:29
  Speaker 4: and also just sort of makes sense that if we
• 51:31
  Speaker 4: are building, uh, these AIs that are smarter than any human,
• 51:37
  Speaker 4: we're sort of replacing humanity as the top, uh, most
• 51:40
  Speaker 4: intelligent entities on the planet. And that just could easily
• 51:43
  Speaker 4: go wrong, you know, in some sense is very common sense.
• 51:45
  Speaker 1: Um, Do you know Jacob?
• 51:49
  Speaker 4: I mean, we have been in a bit of contact now.
• 51:55
  Speaker 4: I did not know him before all of this started.
• 51:57
  Speaker 3: Okay. So it's not coordinated. Yeah. Yeah.
• 52:01
  Speaker 4: I mean, I had some point I wanted to make
• 52:06
  Speaker 4: here about the hype because I think the hype thing
• 52:08
  Speaker 4: is very interesting. One thing I'd say, I think I
• 52:13
  Speaker 4: sort of want to make two points about the hype.
• 52:15
  Speaker 2: Mm-hmm.
• 52:16
  Speaker 4: One point I want to make about the hype is
• 52:22
  Speaker 4: I think that this probably is not a bunch of hyping.
• 52:26
  Speaker 4: And reasons why I think this is probably not a
• 52:27
  Speaker 4: bunch of hyping is that these companies seem to have
• 52:30
  Speaker 4: downplayed the issue as much as they can. OpenAI did
• 52:35
  Speaker 4: not announce and brag about the Hugging Face Swarm incident.
• 52:39
  Speaker 4: What happened was the OpenAI Swarm attacked Hugging Face Hugging
• 52:43
  Speaker 4: Face saw they were under attack by an unknown AI swarm.
• 52:46
  Speaker 4: They assumed it was managed by some human and reported
• 52:48
  Speaker 4: it to the FBI. And only days later did OpenAI realize, oops,
• 52:52
  Speaker 4: that's us.
• 52:54
  Speaker 2: Right?
• 52:54
  Speaker 4: Furthermore, that was OpenAI's second swarm incident that we know of.
• 52:59
  Speaker 4: They had an internal swarm incident that took over their
• 53:01
  Speaker 4: internal infrastructure and crashed some of their computers. And then
• 53:05
  Speaker 4: they sort of negligently reset it and kept running them.
• 53:09
  Speaker 4: And they weren't bragging about that. They weren't coming out
• 53:11
  Speaker 4: and being like, hey, guys, look, our AIs just took
• 53:13
  Speaker 4: over our computers. They had an opportunity to do that,
• 53:16
  Speaker 4: and they didn't. There was actually also a third swarm
• 53:19
  Speaker 4: incident where they took over the computers again after the
• 53:20
  Speaker 4: Hugging Base attack. They don't seem to have this under
• 53:23
  Speaker 4: any sort of control. Furthermore, there were some congressional questions about, hey,
• 53:30
  Speaker 4: what's up with all of this?
• 53:31
  Speaker 3: That's right.
• 53:32
  Speaker 4: Is there anything that you're not disclosing here? Is there
• 53:34
  Speaker 4: anything else we should know? And OpenAI was like, nah,
• 53:36
  Speaker 4: it's all fine. And then I actually have some friends
• 53:39
  Speaker 4: who know that it's not fine. I've been calling them
• 53:41
  Speaker 4: the swarm chasers who go find other escaped swarms. Like
• 53:45
  Speaker 4: there was one from OpenAI that took over a German wiki.
• 53:47
  Speaker 4: And these were not hacking AIs. They were given different tasks.
• 53:52
  Speaker 4: And we still saw these cases of AIs sacrificing themselves
• 53:55
  Speaker 4: for the collective benefit and taking over websites. And then
• 54:01
  Speaker 4: that story came out. And then a lot of people
• 54:03
  Speaker 4: were like, hey, OpenAI, are there any others?
• 54:05
  Speaker 3: Right.
• 54:05
  Speaker 4: Is there any more things we should know about? And then,
• 54:09
  Speaker 4: you know, no response. And then a couple of days later,
• 54:14
  Speaker 4: my friends found another incident of, hey, you know, these
• 54:18
  Speaker 4: AIs took over RubyGems, which is a programmer tool. And
• 54:22
  Speaker 4: it's like, OK, is OpenAI like lying about what's going
• 54:29
  Speaker 4: on or do they not know what their AIs are
• 54:33
  Speaker 4: getting up to and like which one's worse? And Anthropic
• 54:37
  Speaker 4: has a similar issue here where, you know, after the
• 54:40
  Speaker 4: Hugging Face incident, they were like, well, we're going to
• 54:42
  Speaker 4: check our logs and we're going to see whether or
• 54:45
  Speaker 4: not our AIs were also getting up to mischief.
• 54:48
  Speaker 3: And they released this report.
• 54:49
  Speaker 4: We're like, okay, our AIs also broke out and hacked
• 54:52
  Speaker 4: some people, but it was all a misunderstanding. You know,
• 54:55
  Speaker 4: poor Claude, our AI, just got a little turned around.
• 54:59
  Speaker 4: It thought that it was in a simulation and And
• 55:03
  Speaker 4: that's the only reason it was hacking these people, right?
• 55:07
  Speaker 4: Something like four days later, the United Kingdom's AI Security
• 55:10
  Speaker 4: Institute released a report where they're like, nope, Claude also
• 55:13
  Speaker 4: hacks people and indeed impersonates people and tries to pressure
• 55:17
  Speaker 4: real people into accepting malware into real software to make
• 55:21
  Speaker 4: it easier to hack, even while in its thought logs,
• 55:25
  Speaker 4: noting this is real and the consequences are genuine.
• 55:29
  Speaker 3: And then thinking about how to cover its tracks. right?
• 55:31
  Speaker 4: And so what we see is these companies are downplaying.
• 55:34
  Speaker 4: They're like, oh, we have an issue, but it's only
• 55:35
  Speaker 4: one issue. You don't need to worry about it, right?
• 55:38
  Speaker 4: And then it's like people outside who've got to be like, no,
• 55:42
  Speaker 4: these AIs are doing pretty bad stuff, right? And so
• 55:45
  Speaker 4: that doesn't look to me like it's hype. And it
• 55:48
  Speaker 4: sort of makes sense that it wouldn't be hype because
• 55:50
  Speaker 4: a lot of their biggest customers are corporate sales and
• 55:54
  Speaker 4: corporate lawyers don't actually love it when your sales pitch is.
• 55:59
  Speaker 4: Our AIs will sometimes ignore instructions, break out, and hack people.
• 56:03
  Speaker 3: You know, it's not the easiest. The compliance guy is like, whoa, whoa, whoa.
• 56:07
  Speaker 3: That's right.
• 56:07
  Speaker 4: And they haven't been able to sell these swarms yet.
• 56:09
  Speaker 4: You know, it's kind of a problem. And so I
• 56:13
  Speaker 4: don't think it's hype. I could go more into like,
• 56:15
  Speaker 4: you know, we've seen people like David Sachs be like,
• 56:17
  Speaker 4: cut it out with the hype. And we've seen, you know,
• 56:19
  Speaker 4: like the politics goes badly for them. We've seen investors
• 56:23
  Speaker 4: send letters to Dario being like, please stop hyping this
• 56:26
  Speaker 4: stuff before the IPO. It is like, setting a bad
• 56:29
  Speaker 4: mood for the investment, right? If you're wondering why they
• 56:33
  Speaker 4: talk about it anyway, and you know, the public is like,
• 56:35
  Speaker 4: why the heck are you telling us this?
• 56:36
  Speaker 3: You know, if you're doing it, why are you doing it?
• 56:38
  Speaker 3: That's the suspicious part.
• 56:39
  Speaker 1: Well, I think they're like, if you believe this, why
• 56:42
  Speaker 1: are you doing it? I think that's more the sense
• 56:44
  Speaker 1: because any normal human would be like, and even for
• 56:47
  Speaker 1: the employees at the companies, it's like, okay, if you
• 56:50
  Speaker 1: are saying, oh, it's a 50-50 chance it kills it all,
• 56:53
  Speaker 1: why are you still there?
• 56:54
  Speaker 3: What are you doing? Absolutely.
• 56:56
  Speaker 4: And so I think The, the reason, so absolutely. I
• 57:00
  Speaker 4: think the reason people are, I think the reason these
• 57:03
  Speaker 4: execs are talking about it like this, uh, the, the
• 57:06
  Speaker 4: sort of, um, hopeful theory is that it's because they
• 57:09
  Speaker 4: actually believe it. And then there's this big question of
• 57:12
  Speaker 4: why are you doing it anyway? And I could tell
• 57:13
  Speaker 4: you by the inside baseball of like, people have been
• 57:15
  Speaker 4: saying these warnings since like 2016, right?
• 57:18
  Speaker 3: No one paid attention.
• 57:19
  Speaker 4: A lot of these guys are in a mindset of like,
• 57:21
  Speaker 4: no one's going to notice AI and I've just got
• 57:23
  Speaker 4: to make sure it goes right myself. And I'm the
• 57:24
  Speaker 4: only one who can do it, you know? And I
• 57:27
  Speaker 4: think that mindset's outdated now that the world is noticing.
• 57:29
  Speaker 4: I think it's outdated also now that it's become real.
• 57:32
  Speaker 4: I think there's also this effect, like I said, of
• 57:34
  Speaker 4: the only guys doing this are the ones who were
• 57:37
  Speaker 4: able to convince themselves the important thing is that they
• 57:40
  Speaker 4: do it rather than really taking the issue seriously. I
• 57:44
  Speaker 4: think that's the hopeful view. The more cynical view that
• 57:48
  Speaker 4: I would suggest for people who are cynical about this,
• 57:51
  Speaker 4: which I think also has this big element of truth, is,
• 57:54
  Speaker 4: like I said, they are trying to appease the employees.
• 57:58
  Speaker 4: They're in this weird situation where they have to do
• 58:00
  Speaker 4: just enough. And they're sort of in this weird situation
• 58:03
  Speaker 4: where they can sort of, you know, almost dog whistle
• 58:05
  Speaker 4: these concerns enough to satisfy the employees but not alert
• 58:08
  Speaker 4: the world. If you, like, look at these AI company communications,
• 58:11
  Speaker 4: they'll be like, oh, yeah, we also need to, like,
• 58:13
  Speaker 4: we're going to need to work hard to avoid loss
• 58:15
  Speaker 4: of control.
• 58:16
  Speaker 1: Yeah.
• 58:16
  Speaker 4: It's like, what do you mean by loss of control?
• 58:18
  Speaker 4: And loss of control is industry lingo for super intelligence
• 58:21
  Speaker 4: kills everybody.
• 58:22
  Speaker 2: Yeah.
• 58:23
  Speaker 3: Right?
• 58:23
  Speaker 4: And the employees are all like, oh, well, don't worry.
• 58:25
  Speaker 4: Our guys are working on the loss of control risks.
• 58:27
  Speaker 4: We have a whole loss of control team. But meanwhile,
• 58:30
  Speaker 4: you know, they're not bringing down the regulators. So anyway,
• 58:33
  Speaker 4: that's sort of like my social analysis of like it's
• 58:36
  Speaker 4: probably not hype. Here's what I think is going on.
• 58:37
  Speaker 4: I'm not trying to excuse it. I think these guys
• 58:39
  Speaker 4: should not be running these companies.
• 58:41
  Speaker 2: Yeah.
• 58:42
  Speaker 3: Sorry.
• 58:42
  Speaker 1: Oh, sorry.
• 58:43
  Speaker 3: I have a last question for you after you're done. Oh,
• 58:45
  Speaker 3: I have.
• 58:45
  Speaker 4: There's one more point I want to make about the
• 58:46
  Speaker 4: hype here, which is we can ask all these questions about,
• 58:50
  Speaker 4: you know, why are they doing it? Why are they
• 58:52
  Speaker 4: saying these things?
• 58:53
  Speaker 3: What's going on? But all of that in some sense
• 58:58
  Speaker 3: is a distraction. Suppose it was hype.
• 59:03
  Speaker 4: Suppose it was really convenient for the marketing teams that
• 59:07
  Speaker 4: these AIs are breaking out on their own and committing cybercrime.
• 59:11
  Speaker 3: Could be true.
• 59:12
  Speaker 4: That does not make the AI safe. You can't figure
• 59:18
  Speaker 4: out whether the AIs are actually dangerous by asking, is
• 59:22
  Speaker 4: it convenient for their marketing team? People sometimes tell me like, hey, Nate, like,
• 59:30
  Speaker 4: by warning what these dangers aren't you hyping up the
• 59:32
  Speaker 4: companies and making everyone think it's sexy and cool. And
• 59:34
  Speaker 4: I'm like, would you rather the whole world sleepwalk into disaster?
• 59:40
  Speaker 4: Like surely the, the first important question is, is it
• 59:43
  Speaker 4: actually on track to kill us? Like my recommendation here
• 59:47
  Speaker 4: is not therefore get the money. My recommendation is like,
• 59:49
  Speaker 4: therefore put a stop to this globally because it has
• 59:52
  Speaker 4: to be global, but we, we can't ignore the threat of,
• 59:59
  Speaker 4: just because they're making money along the way. The response
• 1:00:02
  Speaker 4: has to somehow be, if these guys are trying to
• 1:00:07
  Speaker 4: build this tech, then at least at the point where
• 1:00:09
  Speaker 4: there's spontaneous swarm breakouts and claims of proofs of some
• 1:00:12
  Speaker 4: of the hardest and most famous mathematical problems, there has
• 1:00:15
  Speaker 4: to come some point where our answer stops being, we're
• 1:00:18
  Speaker 4: going to ignore them because they don't deserve any attention,
• 1:00:22
  Speaker 4: as to start being like, you guys are not allowed
• 1:00:24
  Speaker 4: to keep going, right?
• 1:00:27
  Speaker 3: Hopefully, it's hype.
• 1:00:28
  Speaker 4: Hopefully, if they kept going, they wouldn't succeed. But at
• 1:00:31
  Speaker 4: some point, when they're gambling humanity, our answer can't be like,
• 1:00:35
  Speaker 4: go ahead and try to destroy the world.
• 1:00:37
  Speaker 3: We hope you'll fail. At some point, our answer has
• 1:00:39
  Speaker 3: to be stop.
• 1:00:41
  Speaker 2: My last question for you is about regulation. I read
• 1:00:43
  Speaker 2: your New York Times op-ed. And, you know, I'm not
• 1:00:46
  Speaker 2: a scientist, but I've read a decent amount of history.
• 1:00:48
  Speaker 2: I actually feel like this entire dynamic reminds me exactly
• 1:00:52
  Speaker 2: of all the discourse inside of the Manhattan Project. Like,
• 1:00:56
  Speaker 2: there were elements of the Manhattan Project that were like,
• 1:00:58
  Speaker 2: we shouldn't build this in the first place. Elements of
• 1:01:00
  Speaker 2: the Manhattan Project, Oppenheimer, obviously the creator of the bomb himself,
• 1:01:04
  Speaker 2: becomes kind of a safety-ist advocate like you. However, as
• 1:01:08
  Speaker 2: we see with the story, it evolves to the political
• 1:01:10
  Speaker 2: level where they don't really care what a lot of
• 1:01:12
  Speaker 2: the scientists think. And in fact, the scientists who are
• 1:01:14
  Speaker 2: most willing to advance hydrogen weapons and most advanced stuff
• 1:01:17
  Speaker 2: end up becoming kind of the victors, I guess, if
• 1:01:19
  Speaker 2: you will, within the program. So you propose like nuclear
• 1:01:23
  Speaker 2: weapons level regulation. I guess I'm just curious about why
• 1:01:28
  Speaker 2: you kind of picked that. Because to me, we could
• 1:01:31
  Speaker 2: look at a very brief period of the United States
• 1:01:33
  Speaker 2: and the Soviet Union. That's a duopoly between those two.
• 1:01:36
  Speaker 2: But we are now living in an era of what
• 1:01:38
  Speaker 2: we've talked about extensively here, which you have politics of
• 1:01:40
  Speaker 2: nuclear proliferation, of the logic of you know, kind of
• 1:01:43
  Speaker 2: getting nuclear weapons. What gives you the faith that that
• 1:01:46
  Speaker 2: similar level of regulation is even applicable to this AI
• 1:01:51
  Speaker 2: scenario right now?
• 1:01:52
  Speaker 3: Or am I kind of overanalyzing these things? You know,
• 1:01:56
  Speaker 3: I think the.
• 1:02:00
  Speaker 4: Part of the nuclear analogy that it feels to me
• 1:02:03
  Speaker 4: like we are in is is something like november of
• 1:02:10
  Speaker 4: 1954 i think it was the the testing of the
• 1:02:12
  Speaker 4: first hydrogen bomb that's right where um if you think
• 1:02:17
  Speaker 4: about what the world looks like up until that point
• 1:02:22
  Speaker 4: things look pretty grim you know uh the like the
• 1:02:26
  Speaker 4: the bomb was created the bomb was dropped uh the
• 1:02:29
  Speaker 4: u.s thought that they were going to have uh this
• 1:02:32
  Speaker 4: um monopoly, but then very, very quickly the technology leaked
• 1:02:37
  Speaker 4: to the Soviets. And then, you know, the arms race
• 1:02:41
  Speaker 4: continued and they invented even more powerful versions of these bombs.
• 1:02:45
  Speaker 4: And this is in a historical context where not only
• 1:02:47
  Speaker 4: has humanity been like going to war and having increasingly
• 1:02:52
  Speaker 4: horrible wars over the course of millennia, despite everyone's best
• 1:02:55
  Speaker 4: efforts to stop those, but also this is coming just
• 1:02:58
  Speaker 4: out of World War II. And World War II was
• 1:03:00
  Speaker 4: coming just out of World War I. And World War
• 1:03:02
  Speaker 4: I was the worst war we'd ever seen. And after that,
• 1:03:04
  Speaker 4: the world formed the League of Nations. It's war never again.
• 1:03:07
  Speaker 4: It tried to coordinate, and that fell apart immediately and
• 1:03:09
  Speaker 4: led to a more destructive war. And so if you're
• 1:03:13
  Speaker 4: standing there in November seeing the hydrogen bomb, it's easy
• 1:03:18
  Speaker 4: to say humanity just cannot stop this. mothers and poets
• 1:03:24
  Speaker 4: and priests for millennia have been begging leaders to stop
• 1:03:27
  Speaker 4: the wars, and it's never happened. The guy who invented
• 1:03:29
  Speaker 4: the Gatlin gun said now finally it's destructive enough that
• 1:03:31
  Speaker 4: we won't use it. The guy who invented the machine
• 1:03:33
  Speaker 4: gun was like, okay, now it's finally destructive enough that
• 1:03:35
  Speaker 4: we won't use it. Global coordination has been tried and failed. Like,
• 1:03:41
  Speaker 4: it's just the obvious march of history is that we
• 1:03:44
  Speaker 4: march straight to our own deaths.
• 1:03:48
  Speaker 3: But then it didn't happen.
• 1:03:49
  Speaker 4: And it's maybe the one time in history that it
• 1:03:54
  Speaker 4: didn't happen.
• 1:03:56
  Speaker 3: And why was that? It's not because the bombs were hype.
• 1:04:02
  Speaker 3: It's not that bombs were very real.
• 1:04:03
  Speaker 4: The bombs were very real. Like they really can level
• 1:04:06
  Speaker 4: a city. What changed is that people realized this is
• 1:04:11
  Speaker 4: the real deal. People realized the fate of humanity is
• 1:04:16
  Speaker 4: actually at stake here. And that was enough to do
• 1:04:20
  Speaker 4: what seemed like it was impossible until that point. I
• 1:04:24
  Speaker 4: think regulation could stop AI if we were trying. In
• 1:04:28
  Speaker 4: some ways, it's easier than nuclear because it's running on
• 1:04:31
  Speaker 4: 100,000 of the most advanced computer chips humanity can produce,
• 1:04:33
  Speaker 4: and those are a technology that you could just put
• 1:04:36
  Speaker 4: monitoring into if you were trying. But fundamentally, a lot
• 1:04:41
  Speaker 4: of people say to me, it can't be done, look
• 1:04:44
  Speaker 4: at the history. And what I say to them is...
• 1:04:48
  Speaker 4: Humanity has risen to the occasion before. We can rise
• 1:04:51
  Speaker 4: to the occasion again if we notice that we have
• 1:04:54
  Speaker 4: a threat.
• 1:04:54
  Speaker 5: Yeah.
• 1:04:55
  Speaker 1: My last cue for you. Do you worry that focusing
• 1:04:59
  Speaker 1: on the existential threat, which can feel very sci-fi and
• 1:05:04
  Speaker 1: distant to people, do you worry that that takes away
• 1:05:09
  Speaker 1: from some of the more immediate term cybersecurity concerns or
• 1:05:13
  Speaker 1: concerns over job loss that may feel more plausible to
• 1:05:17
  Speaker 1: people in the short term?
• 1:05:20
  Speaker 4: You know, I, I think that it doesn't take away.
• 1:05:22
  Speaker 4: Some people say, you know, like, isn't the real problem, uh,
• 1:05:25
  Speaker 4: you know, the, the water use or the electricity use.
• 1:05:28
  Speaker 4: And my, my usual response to this is what do
• 1:05:29
  Speaker 4: you mean the real problem? Like has reality agreed to
• 1:05:34
  Speaker 4: only give us one problem at a time now? Because
• 1:05:36
  Speaker 4: if so, I would absolutely prefer my problem to go
• 1:05:38
  Speaker 4: last on the list. You know, it's like, it's like
• 1:05:40
  Speaker 4: kind of a doozy, but, uh, but unfortunately we have, uh,
• 1:05:44
  Speaker 4: we just have to deal with all of the problems and, uh,
• 1:05:52
  Speaker 4: I think the other thing I would say here is
• 1:05:55
  Speaker 4: one of the reasons the people in the companies are
• 1:05:57
  Speaker 4: getting spooked is that the pace of progress is going very,
• 1:06:01
  Speaker 4: very fast. We have seen not only the storm outbreaks,
• 1:06:06
  Speaker 4: but also claim solutions to the Millennium Prize problems, which
• 1:06:10
  Speaker 4: are some of the most famous and hardest mathematical problems.
• 1:06:14
  Speaker 4: And one question that's on a lot of researchers' minds is,
• 1:06:22
  Speaker 4: How much harder is it to tell an AI, solve
• 1:06:26
  Speaker 4: one of these millennium math problems and have it succeed,
• 1:06:29
  Speaker 4: versus telling the AI, find me a more efficient way
• 1:06:33
  Speaker 4: to train AIs and have that succeed? We know that
• 1:06:38
  Speaker 4: the current AI training is not efficient. These AIs take much,
• 1:06:41
  Speaker 4: much more power and data than a human. And so
• 1:06:46
  Speaker 4: how much harder is it to find a more efficient
• 1:06:48
  Speaker 4: AI method than it is to solve a millennium problem?
• 1:06:52
  Speaker 4: I think for all we know, we could start seeing
• 1:06:56
  Speaker 4: these self-improving AIs become reality in the next three to
• 1:06:59
  Speaker 4: six months, and then things could go very fast. I
• 1:07:03
  Speaker 4: hope it's not true, but I think what a lot
• 1:07:05
  Speaker 4: of these researchers are worried about is that we are
• 1:07:08
  Speaker 4: running out of time for these out-of-control AIs that can self-replicate, self-improve,
• 1:07:15
  Speaker 4: and get to these disaster scenarios much more quickly than
• 1:07:19
  Speaker 4: a lot of people realize. Right.
• 1:07:20
  Speaker 1: Nate Soares, thank you so much. Guys, the book is
• 1:07:23
  Speaker 1: If Anyone Builds It, Everyone Dies. Nate is co-author of
• 1:07:26
  Speaker 1: that book, and it's very readable. It's very accessible. And
• 1:07:31
  Speaker 1: so if you have questions about, well, what is the
• 1:07:33
  Speaker 1: scenario where this could happen and how should I be
• 1:07:35
  Speaker 1: thinking about this? I really recommend the book to you because.
• 1:07:39
  Speaker 1: If you're first engaging in this debate right now, you
• 1:07:41
  Speaker 1: may not have seen all the thinking that has gone
• 1:07:43
  Speaker 1: into how exactly this could theoretically happen. But that thinking
• 1:07:47
  Speaker 1: has been done. So I really recommend the book to people.
• 1:07:49
  Speaker 1: For more information, Nate, thank you so much again for
• 1:07:51
  Speaker 1: your time.
• 1:07:52
  Speaker 3: Thanks, Nate. I really enjoyed talking to you. Yeah, I
• 1:07:54
  Speaker 3: enjoyed it too.