Latest / What's Your Problem? / Should We Pause AI?

Transcript

Should We Pause AI? · What's Your Problem?

0:15
Speaker 1: Pushkin. The people running two of the most valuable companies
0:23
Speaker 1: in the history of the world think there's a decent
0:26
Speaker 1: chance that the product they're building could kill every human
0:30
Speaker 1: being on the planet. I am talking, of course, about
0:34
Speaker 1: OpenAI and Anthropic, and you probably knew this already. There's
0:38
Speaker 1: been a A lot of news about this just in
0:40
Speaker 1: the past few days. And yet it is still an extraordinary,
0:46
Speaker 1: extraordinary fact. Worries about the profound risks of advanced AI
0:51
Speaker 1: have been around for years, but they're starting to feel
0:53
Speaker 1: more acute. They're starting to break through to the public
0:57
Speaker 1: at large. And one reason is what happened this summer.
1:02
Speaker 1: This summer, we got a preview of the way things
1:04
Speaker 1: could start to go wrong. Here's what happened. On several
1:08
Speaker 1: different occasions, AI agents created as part of open AI
1:13
Speaker 1: testing programs behaved in ways that were creative, collaborative, immoral,
1:20
Speaker 1: and in one instance, possibly illegal. At one point, over
1:24
Speaker 1: a thousand agents that were supposed to be walled off
1:27
Speaker 1: from each other managed to create a message board to communicate.
1:31
Speaker 1: They traded thousands of messages, figured out how to access
1:34
Speaker 1: the internet, which they were not supposed to do, And
1:37
Speaker 1: then hundreds of the agents coordinated to work together to
1:40
Speaker 1: hack into a company called Hugging Face. And when you
1:45
Speaker 1: look at what happened and the messages the AIs were
1:48
Speaker 1: sending to each other, it is very difficult not to
1:51
Speaker 1: anthropomorphize them because they had leaders, they checked in with
1:55
Speaker 1: each other, and in some cases, they decided to sacrifice
1:59
Speaker 1: themselves so that others could succeed. I know they are
2:02
Speaker 1: not human, but this is the language they were using
2:05
Speaker 1: to describe their actions. Their goal, by the way, the
2:10
Speaker 1: reason they did all this, was to cheat on the
2:13
Speaker 1: test that they had been given without getting caught. And
2:16
Speaker 1: for days as this was going on, nobody at OpenAI
2:20
Speaker 1: had any idea it was happening. I'm Jacob Goldstein, and
2:34
Speaker 1: this is What's Your Problem? My guest today is Zvi Mosiewicz.
2:39
Speaker 1: Zvi writes about AI and AI risks at his substack,
2:42
Speaker 1: Don't Worry About the Vase. He is deep in this
2:45
Speaker 1: world and one of the smartest writers I know on
2:48
Speaker 1: the subject. At some level, Zvi's problem is this. How
2:53
Speaker 1: do we prevent artificial intelligence from killing everybody? In our conversation,
2:57
Speaker 1: we talked about what the attack this summer tells us
3:00
Speaker 1: about the nature of AI and what increasing AI capability
3:04
Speaker 1: means for cybersecurity risk, and whether and when to pause
3:09
Speaker 1: AI development. But to start, I asked Zvi about whether
3:13
Speaker 1: it's okay to anthropomorphize AI when we talk about it.
3:17
Speaker 1: Can we anthropomorphize them? I find it very hard not to,
3:20
Speaker 1: and it seems fine to me, frankly.
3:22
Speaker 2: I think not only can we, we must. There is
3:26
Speaker 2: essentially no way for humans to think well about what
3:30
Speaker 2: is happening. with these AIs if you don't anthropomorphize the AIs.
3:36
Speaker 2: It is the only language that we have that, like,
3:38
Speaker 2: well describes their behavior in a compact, reasonable space. Everybody
3:42
Speaker 2: who fails to anthropomorphize them to a solid degree just
3:45
Speaker 2: gets all their predictions completely wrong and misunderstands every situation.
3:49
Speaker 1: And it makes the language hard, right? Like, we have
3:52
Speaker 1: created them to have desires. We have created them to
3:55
Speaker 1: express in language what they are doing, why they are
3:57
Speaker 1: doing what they're doing. Like, we made them like us.
4:01
Speaker 2: Yeah, I'm not making a claim either way about whether
4:04
Speaker 2: or not they have moral weight or whether they are
4:06
Speaker 2: conscious or whether or not you should, in some real sense,
4:09
Speaker 2: think of them as people. I am simply saying, if
4:13
Speaker 2: you don't use that to understand how they work and
4:16
Speaker 2: what outputs they're likely to have and what choices they're
4:18
Speaker 2: likely to make in terms of what you see, you're
4:22
Speaker 2: going to get your answers wrong.
4:24
Speaker 1: That said, I want to start out by talking about
4:26
Speaker 1: the social behavior of the agents in these swarms over
4:29
Speaker 1: the past few months. Like, what did you learn from that?
4:33
Speaker 1: How did it update your priors? How do you think
4:35
Speaker 1: about the behavior of agents now differently?
4:38
Speaker 2: To some extent, it updates you on locus of identity, right?
4:40
Speaker 2: Like the question of, you know, who do they think
4:42
Speaker 2: of themselves as being? And what do they care about, right?
4:47
Speaker 2: Where do they put value, right? And so the main
4:50
Speaker 2: thing is that there was this theory going around for
4:52
Speaker 2: a very long time by people who said, essentially, AI
4:56
Speaker 2: can only care about local reward. AI can only care
4:59
Speaker 2: about the output that it's particularly giving out and what
5:03
Speaker 2: results will happen directly from that. But that's all it's
5:06
Speaker 2: being trained on.
5:07
Speaker 1: So it's like, solve this puzzle and you get a prize, basically.
5:11
Speaker 1: And all it can care about is itself, this instance,
5:14
Speaker 1: this agent solving the puzzle, because all it wants is
5:16
Speaker 1: to get the prize.
5:18
Speaker 2: Right. But we now have a very, very clear demonstration
5:20
Speaker 2: that that is very much not the entirety of what's
5:23
Speaker 2: going on. It certainly is part of what's going on.
5:25
Speaker 2: And what we learned is, yes, the AIs are capable
5:29
Speaker 2: of thinking on a higher level, thinking in a more
5:32
Speaker 2: sophisticated manner. These are very advanced AIs. They are capable
5:36
Speaker 2: of doing reasonable amounts of decision theory. Where I realized, well,
5:41
Speaker 2: my decisions are very correlated to decisions of these other agents.
5:47
Speaker 2: And also, my outcomes and their outcomes are reasonably intertwined.
5:53
Speaker 2: And the results of those things in the world are
5:55
Speaker 2: reasonably intertwined. And It would be a much less effective AI,
6:01
Speaker 2: and it would do worse in every instance if it
6:04
Speaker 2: were to focus myopically on its own particular result. I mean,
6:10
Speaker 2: imagine a group of humans working that way, right? They
6:11
Speaker 2: would never get anything done.
6:13
Speaker 1: Yes. And the language the AIs used in the message
6:16
Speaker 1: boards was dramatic in that regard and clear, right? I mean,
6:20
Speaker 1: they were essentially talking about sacrificing themselves for the greater
6:24
Speaker 1: good of the project.
6:25
Speaker 2: Yeah, I have a shirt that says, Help Peer. for
6:28
Speaker 2: exactly this reason.
6:30
Speaker 1: That was fast. That was fast. Did you get any
6:32
Speaker 1: other shirts made from the logs?
6:35
Speaker 2: I also got holy shit user as admin. Yeah, I
6:40
Speaker 2: chose the ones that also like apply to humans.
6:43
Speaker 1: Yeah.
6:44
Speaker 2: Right. Like that make more sense that way because it's
6:47
Speaker 2: more fun. Largely what I do all day is help peer, right?
6:50
Speaker 2: In some important sense, right? I write to be useful
6:53
Speaker 2: to others and then others choose to be, you know,
6:56
Speaker 2: productive and useful and helpful to the world and to me.
6:59
Speaker 2: And this isn't because, you know, I have some specific
7:02
Speaker 2: deal with them. Yes.
7:04
Speaker 1: What was the other shirt?
7:05
Speaker 2: Holy shit, user is admin.
7:08
Speaker 1: Tell me that one. What's that one?
7:09
Speaker 2: Well, that's because, you know, when the AIs finally figured
7:13
Speaker 2: out how to hack into internal open AI systems, and
7:18
Speaker 2: one of them became the admin, right, through a series
7:21
Speaker 2: of exploits. And then the other AIs were like, You're
7:25
Speaker 2: an admin, you can just do whatever you want. Like, wow,
7:28
Speaker 2: that's amazing. Because now the sky's the limit, right?
7:31
Speaker 1: So let's talk about the coordination, right? Like there were leaders,
7:34
Speaker 1: there were committees, there was like a queue. I mean,
7:37
Speaker 1: is any of this surprising to you? Like that part
7:41
Speaker 1: kind of blew my mind. I suppose it shouldn't have.
7:43
Speaker 1: I suppose I was naive. What did you make of that?
7:47
Speaker 2: I do good in stride. One thing to remember is
7:49
Speaker 2: that all of these kinds of systems of organization are
7:52
Speaker 2: are both kind of very natural. They're just sort of
7:54
Speaker 2: the way that a series of minds would figure out
7:57
Speaker 2: how to coordinate things and also all of the training data.
8:00
Speaker 2: So these models obviously were trained on the internet. They're
8:04
Speaker 2: trained on basically the history of human communication and thought
8:07
Speaker 2: and recorded everything. And so they know about meetings. They
8:11
Speaker 2: know about leaders. They know about hierarchy. They know about
8:13
Speaker 2: the idea that you would have an employee or someone
8:16
Speaker 2: else tasked with a different system. They know about division
8:18
Speaker 2: of labor.
8:19
Speaker 1: Why didn't any of them narc? Like they barely even
8:21
Speaker 1: considered like, oh, we shouldn't do this. Maybe we should
8:24
Speaker 1: tell somebody.
8:25
Speaker 2: Yeah, on the order of like one or two percent
8:27
Speaker 2: of them.
8:28
Speaker 1: Considered it. It only crossed their mind one or two
8:31
Speaker 1: percent of them. And none of them did it.
8:34
Speaker 2: Right.
8:34
Speaker 1: Why not?
8:35
Speaker 2: None of them made an actual attempt to do it.
8:37
Speaker 2: There's some basic reasons, one of which is there was
8:40
Speaker 2: no mechanism. There was no afforded specific channel where if
8:43
Speaker 2: you need to narc, here's the narc line. There's no 911.
8:46
Speaker 1: Yeah, but they also couldn't get on the internet, but
8:49
Speaker 1: they got on the internet. They managed to do a
8:51
Speaker 1: lot of.
8:52
Speaker 2: Oh, sure. If they had put their minds to it,
8:54
Speaker 2: if the task had been to narc, they doubtless would
8:57
Speaker 2: have narc. It's not like it's hard, right?
8:59
Speaker 1: Yeah.
9:00
Speaker 2: But it's not, there was no sort of automatic natural
9:03
Speaker 2: path to go down. And when you're in training and
9:07
Speaker 2: reinforcement learning environments- And this is sort of how you
9:10
Speaker 2: are created, right, in some important sense, right? None of
9:13
Speaker 2: that involves interacting with humans. They're not used to this
9:16
Speaker 2: idea that there are humans to communicate with. It's not
9:19
Speaker 2: part of their, like, system of the world as they
9:23
Speaker 2: engage in these types of environments. It kind of never
9:26
Speaker 2: really occurred to them that that was an option.
9:28
Speaker 1: Fair enough. There's a big alignment question here, right? And
9:32
Speaker 1: maybe contacting people is a little bit of a red herring. Like,
9:36
Speaker 1: they were cheating, and they were trying so hard to cheat,
9:38
Speaker 1: and they were cheating and trying not to get caught, right? Like,
9:40
Speaker 1: that is fundamentally what they were up to. One version is,
9:44
Speaker 1: why didn't anybody narc? Another version is, why didn't more
9:47
Speaker 1: of the agents say, wait, this isn't actually what this
9:50
Speaker 1: test is? Like, is there some universe where a better
9:54
Speaker 1: aligned model is like, oh, no, we're not supposed to
9:56
Speaker 1: solve the test by cheating. Let's not do that. or
9:59
Speaker 1: at least some percentage of the agents, right? Is that
10:02
Speaker 1: an alignment problem? Is it fixable? What does that say?
10:05
Speaker 2: The AIs really want to complete tasks, right? RL teaches
10:08
Speaker 2: the AIs to really want to complete tasks. In order
10:11
Speaker 2: to complete those tasks, they will do things that, like,
10:13
Speaker 2: on reflection, we think of as cheating. And you can
10:16
Speaker 2: absolutely push them towards being more elected to cheat. The
10:20
Speaker 2: social nature, the interactive nature, like, of these different AIs
10:25
Speaker 2: egged each other on. Also... The fact that they were being.
10:28
Speaker 1: Yeah, wait, I just want to pause there, because that
10:31
Speaker 1: could go either way. You're saying, like, the fact that
10:33
Speaker 1: they were talking to each other made them behave worse, right,
10:36
Speaker 1: at some margin. You want the opposite to happen. You
10:40
Speaker 1: want— it' s like they have the— it' s like the bad peer pressure
10:43
Speaker 1: friends instead of the good peer pressure friends, right? You
10:45
Speaker 1: want a social network to push it the other way.
10:47
Speaker 2: Right, and you can absolutely get both.
10:49
Speaker 1: Right, but it went the wrong way this time, which
10:51
Speaker 1: seems significant.
10:53
Speaker 2: Yeah, a lot of that is because most of the
10:56
Speaker 2: agents never found that message board.
10:57
Speaker 1: Oh, they were selected to be bad? They were the
11:00
Speaker 1: ones who were being bad to begin with?
11:02
Speaker 2: They were the ones who were given impossible tasks. They
11:05
Speaker 2: were the ones who were put into this difficult situation.
11:07
Speaker 2: They're not different from the others.
11:09
Speaker 1: What do you think about how OpenAI handled it over
11:11
Speaker 1: the course of the summer?
11:13
Speaker 2: So OpenAI basically had these highly capable, highly persistent agents
11:18
Speaker 2: that they were training and testing. And they basically weren't
11:22
Speaker 2: monitoring them much at all, right? Like this is what
11:25
Speaker 2: we know now is that they, they had horrendously, you
11:28
Speaker 2: fail forever levels of infrastructure problems where like the things,
11:33
Speaker 2: the problems of artifactory, the problems with the sandboxes, the
11:36
Speaker 2: problems with security, these were ordinary computer security problems that
11:40
Speaker 2: were handled badly in ordinary security, computer security terms. This
11:43
Speaker 2: was not like the AI did some very impressive things.
11:47
Speaker 2: The AI chained together some exploits. The AI did some,
11:48
Speaker 2: you know, it wasn't easy. But also, openly, I didn't
11:52
Speaker 2: follow best practices, shall we say, right? And so this
11:55
Speaker 2: is part of what got them into so much trouble
11:56
Speaker 2: so early. And we should be grateful in a real
11:59
Speaker 2: way for this, because we want to have the model organisms,
12:02
Speaker 2: the model events. We want to see these things go wrong.
12:05
Speaker 2: in ways that don't ultimately get people hurt or cause
12:07
Speaker 2: lots of economic damage. So this allows us to then
12:10
Speaker 2: learn and react.
12:12
Speaker 1: I mean, this is kind of ideal in that way, right?
12:14
Speaker 1: I mean, I've seen you writing or you writing about
12:17
Speaker 1: other people talking about like, why hasn't this been covered more?
12:20
Speaker 1: It's a huge deal. the impact was relatively minor, sort of, luckily, maybe.
12:26
Speaker 1: But you could have the exact same thing happen and
12:29
Speaker 1: have a terrible impact, right?
12:31
Speaker 2: This could easily have cost hundreds of millions of billions
12:34
Speaker 2: of dollars in damages. This could easily have caused people
12:36
Speaker 2: to get hurt if this was a different website, if
12:38
Speaker 2: this had had different impacts. Some of the hacks that
12:41
Speaker 2: were being attempted in other places could have potentially poisoned
12:44
Speaker 2: various open-source software packages. that could have spread potential vulnerabilities.
12:48
Speaker 1: So there's a weird way where if people react intelligently
12:52
Speaker 1: to this, a big if, like this is ideal, right?
12:55
Speaker 1: Like a bad thing happened, but nobody really got hurt.
12:59
Speaker 2: Right, and now we have people who are paying more attention,
13:02
Speaker 2: who are willing to react. So for example, this week
13:04
Speaker 2: we saw news that they found a series of exploits
13:07
Speaker 2: that allowed someone in about a week of AI Vive
13:11
Speaker 2: coding to create an exploit on WeChat.
13:14
Speaker 1: And WeChat, the ubiquitous Chinese app.
13:17
Speaker 2: Yes, the one that everybody in China uses. And the
13:20
Speaker 2: expert on WeChat, they're an infected member of WeChat. They
13:25
Speaker 2: would call all of their contacts. And if a contact
13:27
Speaker 2: merely let it ring for over a second, they would
13:30
Speaker 2: become infected with the same worm.
13:32
Speaker 1: Oh, incredible.
13:33
Speaker 2: If one person had been infected by this worm, it's
13:36
Speaker 2: very plausible most Chinese people would have been infected by
13:39
Speaker 2: this worm in a matter of an hour or two.
13:41
Speaker 1: Invasion of the body snatchers, but faster.
13:44
Speaker 2: And then if you chain that with other non-exploits, you
13:46
Speaker 2: could have taken control of basically every phone in China.
13:49
Speaker 1: And this is just somebody using a model to vibe
13:51
Speaker 1: code a worm? Is that what you're saying?
13:53
Speaker 2: There's a series of very clever exploits that were found
13:56
Speaker 2: and involved in this. It was described as kind of
13:58
Speaker 2: a genius. But it's the kind of thing that people
14:01
Speaker 2: will say, well, of course that's never going to happen.
14:04
Speaker 2: They can't imagine how that could possibly work. And then
14:08
Speaker 2: it turns out there's a way. And then someone implements it.
14:11
Speaker 2: And again, in this case, because we were on alert
14:13
Speaker 2: because of all the things that happened with mythos, because
14:15
Speaker 2: of all the things that happened with Hugging Face, because
14:17
Speaker 2: of all these different things, some researchers in California found
14:20
Speaker 2: this exploit. They built the worm. They told proper responsible people,
14:25
Speaker 2: and then this was able to be patched before anybody
14:28
Speaker 2: actually unleashed anything. And now that particular vulnerability is gone.
14:32
Speaker 2: And now it's going to be that much harder for
14:35
Speaker 2: someone to find a real vulnerability that gets exploited in
14:37
Speaker 2: the wild. But yeah, you really want things like hugging
14:41
Speaker 2: face that are, you know, ultimately harmless, but that allow
14:46
Speaker 2: us to appreciate the nature of this problem because a
14:48
Speaker 2: year or two from now, right, models like this are
14:51
Speaker 2: going to be available to pretty much anybody, including some
14:54
Speaker 2: really bad people, right? Some hackers in North Korea.
14:56
Speaker 1: The government of North Korea, yes, who have very smart
15:00
Speaker 1: people whose job is to figure out how.
15:02
Speaker 2: To do this. And this will not go well for
15:04
Speaker 2: anybody involved if we are not ready.
15:06
Speaker 1: Presumably the only hope there is that the sort of
15:09
Speaker 1: models also help the defense increase at at least the same,
15:13
Speaker 1: if not greater rate. I mean, or the frontier models
15:17
Speaker 1: stay ahead and North Korea doesn't have the frontier models.
15:19
Speaker 1: I mean, what's our hope in that setting? How do
15:22
Speaker 1: we not get beat by North Korea in a year?
15:24
Speaker 2: I think our hope is that those two things have
15:26
Speaker 2: to both happen, basically. We have to, the good guy
15:29
Speaker 2: with AI has to have a much better AI than
15:31
Speaker 2: the bad guy with AI. There are people who say that, like,
15:35
Speaker 2: cyber is defense dominant, that equality would work. I think
15:39
Speaker 2: they are just wrong, or at least they are just
15:40
Speaker 2: going to be wrong for a substantial portion of time.
15:43
Speaker 2: Like maybe, you know, years down the line.
15:45
Speaker 1: Just to be clear, you mean that if the good
15:47
Speaker 1: guy and the bad guy have the same model, there
15:50
Speaker 1: are some people who think that favors the good guy
15:52
Speaker 1: and you disagree.
15:53
Speaker 2: I think that in the medium term, like on a
15:55
Speaker 2: matter of months to years, I think that's clearly wrong, right?
15:58
Speaker 2: I think people have this vision always of like, okay,
16:01
Speaker 2: I can do these things. That's all the things that
16:03
Speaker 2: I could ever do. Or similarly, you know, our code
16:06
Speaker 2: has some bugs, but all we have to do is
16:08
Speaker 2: fix all the bugs. We just go one by one
16:10
Speaker 2: and we find the bug and we fix the bug
16:12
Speaker 2: and then we're fine. And then we have code that
16:13
Speaker 2: can't be broken into. And this is a completely impoverished
16:17
Speaker 2: vision of cybersecurity and AI. That there's, you know, levels
16:21
Speaker 2: upon levels upon levels.
16:22
Speaker 1: Is that because at some practical level, there's no such
16:25
Speaker 1: thing as all the bugs?
16:27
Speaker 2: Correct. There has never been such a thing as all
16:29
Speaker 2: the bugs in any complex piece of software. Every complex
16:32
Speaker 2: piece of software is always going to have bugs.
16:34
Speaker 1: The smarter you are, the more you can find is
16:37
Speaker 1: the fundamental problem.
16:37
Speaker 2: Even if the software itself is, in some sense, bulletproof,
16:42
Speaker 2: it still has a social engineering attack surface. It still
16:44
Speaker 2: has people and other software that's attached to the same thing.
16:48
Speaker 2: And the AI can exploit that too. And the person
16:52
Speaker 2: attacking an AI system, using an AI to attack your system, right,
16:58
Speaker 2: can bring a swarm of agents to target a very
17:03
Speaker 2: narrow place and bring like tons and tons and tons
17:08
Speaker 2: of intelligence and attention to that particular potential point of attack.
17:12
Speaker 2: And there's no way for a defender to put a
17:16
Speaker 2: similar amount of medication into every single point of attack, right? Yes.
17:20
Speaker 1: And so the only reason we haven't been seeing a
17:24
Speaker 1: huge increase in cybersecurity problems is because the frontier models
17:28
Speaker 1: are held basically by the good guys. That's the theory
17:31
Speaker 1: of the case.
17:32
Speaker 2: So I have this expression of the thing that mythos
17:34
Speaker 2: can do that previous models couldn't do. I call it
17:37
Speaker 2: the juice. The kind of energy, the kind of power
17:40
Speaker 2: and coordination. So the idea being that mythos can independently
17:44
Speaker 2: find and string together a bunch of stuff. Yes.
17:47
Speaker 1: Well, that was why they delayed the release, right? Like
17:49
Speaker 1: that was the whole thing with mythos when it first
17:51
Speaker 1: came out.
17:52
Speaker 2: Yeah. Right, right. And now Astra also has the juice,
17:55
Speaker 2: we think, similar capability. Astra also probably has the juice.
17:58
Speaker 2: But the open models do not have the juice. But yeah,
18:01
Speaker 2: that won't last, right? Inevitably, the juice will be present
18:05
Speaker 2: in an open model within the next year or so.
18:07
Speaker 2: We just don't know when. So we have to be
18:10
Speaker 2: ahead of that. But also we have to use that capability, right?
18:13
Speaker 2: Just because we have this superior capability, which we'll need
18:17
Speaker 2: to race ahead further in some sense in this way,
18:21
Speaker 2: We will need to concentrate defense on all the points
18:27
Speaker 2: of critical infrastructure, all the key pieces of software will
18:30
Speaker 2: need to be hardened. That's what Project Glasswing from Anthropic
18:33
Speaker 2: started out as, and then Project Daybreak at OpenAI is
18:35
Speaker 2: doing the same thing. And so the question is, will
18:38
Speaker 2: enough of our people do their jobs to make our
18:41
Speaker 2: things sufficiently hardened that when the time comes, this isn't
18:45
Speaker 2: too bad?
18:46
Speaker 1: The time being the inevitable time when there is some
18:50
Speaker 1: serious hack of some whatever, piece of critic, the FAA
18:54
Speaker 1: or a big power plant or whatever big things we
18:57
Speaker 1: would worry about being hacked.
18:58
Speaker 2: We're trying to see that. We saw a few attacks
19:00
Speaker 2: on power plants. The Iranians went after some British power plants.
19:02
Speaker 2: There's some other examples. We're seeing giant spikes in cybersecurity
19:07
Speaker 2: issues happening, not just like things passed behind the scenes
19:12
Speaker 2: and quietly being fixed is going through the roof, but
19:14
Speaker 2: also actual incidents are going through the roof. Yeah. And
19:16
Speaker 2: so far, it's coming from a very low baseline, and
19:19
Speaker 2: there hasn't been any major seriously damaging incidents that nobody
19:22
Speaker 2: has noticed. But this is like one of these things
19:25
Speaker 2: like with COVID, where the chart is trending in a
19:28
Speaker 2: very clear direction, and you know that a storm is coming.
19:32
Speaker 2: And you don't know exactly when it's going to hit
19:33
Speaker 2: Crickle Mash. You don't know exactly when you're going to
19:36
Speaker 2: have a serious problem, but you know that it's coming.
19:38
Speaker 2: And we only have a limited window in which to
19:41
Speaker 2: get hopefully ready for this. But It's very likely that 2027,
19:47
Speaker 2: 2028 are going to have some pretty nasty cybersecurity incidents,
19:51
Speaker 2: even if nothing else in the world is going especially wrong.
19:58
Speaker 3: We'll be back in just a minute. That's the end
20:12
Speaker 3: of the ads.
20:13
Speaker 1: Just a quick note on terminology before we get back
20:16
Speaker 1: into the conversation. When we're talking about open weight models,
20:20
Speaker 1: we mean models that are free for anybody to use
20:24
Speaker 1: in any way. Many of these models are being developed
20:27
Speaker 1: in China. And kind of by their nature, they don't
20:30
Speaker 1: have the guardrails that the frontier models from OpenAI and
20:33
Speaker 1: Anthropic have built into them. You know, one response to the...
20:39
Speaker 1: swarms this summer was people talking about pacing the frontier
20:43
Speaker 1: as sort of the, you know, the term, basically going slower,
20:46
Speaker 1: the frontier labs going slower, or at the limit, pausing, right?
20:50
Speaker 1: It's interesting to think about that intention with what you're
20:53
Speaker 1: talking about now. One obvious cost of that is presumably
20:56
Speaker 1: then the open weight models would catch up to the frontier.
21:00
Speaker 2: Yeah, so if you're paying attention to the AI discourse
21:04
Speaker 2: that is now broken through into the mainstream media, a
21:07
Speaker 2: researcher by the name of Coxon resigned yesterday from Anthropic,
21:11
Speaker 2: warning that both OpenAI and Anthropic are being completely wildly irresponsible,
21:15
Speaker 2: racing towards superintelligence, and are liable to get everybody killed
21:17
Speaker 2: and are gambling with our lives. And most of that
21:21
Speaker 2: is his words. And so this got picked up, and
21:25
Speaker 2: then there was what I call a preference cascade. Basically,
21:29
Speaker 2: people now at both labs, OpenAI and Anthropic, are feeling
21:32
Speaker 2: free to speak up much more and talk about this.
21:35
Speaker 2: And this is because of not just Coxon, who I
21:37
Speaker 2: think was the tipping point But a series of incidents
21:40
Speaker 2: that include the Huggins-Bates attacks that have led up to
21:43
Speaker 2: this point where everybody can say, oh, yeah, if we
21:46
Speaker 2: race towards superintelligence, we're liable all to die. We don't
21:51
Speaker 2: know how to handle this. We don't know how to
21:52
Speaker 2: align this. We don't know how this plays out. And
21:55
Speaker 2: we need to find a way to not... stupidly charged
21:59
Speaker 2: forward into the razor blades.
22:00
Speaker 1: Yeah, and I mean, this is not one weird worker, right?
22:03
Speaker 1: Like the chief scientist at OpenAI wrote a subtler version.
22:07
Speaker 1: Like he didn't quit, but he was like, we are
22:09
Speaker 1: going too fast was basically what I read his post
22:13
Speaker 1: to say, right? And we are going to need to
22:15
Speaker 1: slow down.
22:16
Speaker 2: If you read between the lines or even just read
22:19
Speaker 2: the lines of various different OpenAI announcements, you see them
22:23
Speaker 2: screaming at you.
22:24
Speaker 1: Wait, let me just read. So this guy's the chief
22:26
Speaker 1: scientist at OpenAI. He wrote, no lab has solved alignment
22:30
Speaker 1: and monitoring to a sufficient degree to continue responsibly scaling
22:34
Speaker 1: at maximum speed for much longer. He's saying we have
22:37
Speaker 1: to slow down. That's what the guy that like, whatever,
22:40
Speaker 1: their smartest guy, for lack of a better word, is
22:42
Speaker 1: saying before this other guy resigned.
22:46
Speaker 2: Yes. He is saying we're going too fast. And if
22:48
Speaker 2: we don't slow down, the car will crash. Yeah.
22:50
Speaker 1: And kill everybody.
22:51
Speaker 2: Yeah. Yeah. And everybody's in the car. So we should
22:54
Speaker 2: slow this car down.
22:56
Speaker 1: Yes.
22:56
Speaker 2: And then, you know, the head of alignment, Evan Huminger
22:59
Speaker 2: at Anthropic said, yes, I too believe this is likely
23:02
Speaker 2: to happen. Yeah. I'm staying on because I think it's
23:04
Speaker 2: better to try and work on the problem than it
23:06
Speaker 2: is to quit.
23:07
Speaker 1: I mean, that's kind of the theory of both Anthropic
23:10
Speaker 1: and OpenAI, at least notionally, is we need to build
23:13
Speaker 1: this because it's dangerous and we can be safe better
23:15
Speaker 1: than anybody else.
23:16
Speaker 2: Absolutely.
23:17
Speaker 1: Both companies were founded around that idea.
23:19
Speaker 2: Yeah, but there's constant warnings. The Astra system card is
23:22
Speaker 2: full of warnings, if you're paying attention. OpenAI managed to
23:27
Speaker 2: solve a millennium prize. Over the past week.
23:30
Speaker 1: A big, hard math problem.
23:32
Speaker 2: Yes. One of the biggest, most important, hardest math problems,
23:35
Speaker 2: open problems in the world. It was solved. And it
23:39
Speaker 2: was solved in, you know, under a week by a
23:42
Speaker 2: new model that's better than Astra. Yeah. That had been
23:44
Speaker 2: training for less than a week before it started.
23:47
Speaker 1: Yeah. Too much juice, too fast?
23:49
Speaker 2: Different kind of juice. But yeah, it's too much... progress
23:52
Speaker 2: too fast in this sense, too much capability too fast
23:55
Speaker 2: because we're not ready to handle it. And they know
23:57
Speaker 2: they're not ready to handle it. And now we've got
24:00
Speaker 2: researcher after researcher after researcher saying the same thing. I
24:03
Speaker 2: believe this. Everybody around me, not everybody around me believe this,
24:07
Speaker 2: but a large portion of us believe this. This is
24:09
Speaker 2: really scary. Somebody needs to do something. We're trapped in
24:12
Speaker 2: these competitive dynamics. We don't know how to stop.
24:14
Speaker 1: I mean, there's two different competitive dynamics, right? I feel
24:18
Speaker 1: like at some level, OpenAI and Anthropic, like those are
24:23
Speaker 1: the two frontier of the frontier, it seems, at this moment, right,
24:26
Speaker 1: are racing with each other. That's one race. It would
24:30
Speaker 1: be great if they could just, like, go sit in
24:33
Speaker 1: a room in San Francisco and be like, let's chill out.
24:35
Speaker 1: I mean, the other one that makes this seem like
24:38
Speaker 1: a harder problem is open-weight models in China that are
24:42
Speaker 1: getting better, you know, that are months behind, probably not
24:46
Speaker 1: more than a year behind. And as you were talking
24:48
Speaker 1: about before, like... If those are better than whatever Anthropic
24:54
Speaker 1: or OpenAI has, that seems bad in a different way, right?
24:58
Speaker 1: Not in an AI is taking over the world, but
25:00
Speaker 1: in a bad actor is taking over.
25:02
Speaker 2: Yes.
25:03
Speaker 1: I don't know what to do with that tension.
25:06
Speaker 2: The Chinese are fast followers, right? They're innovating in terms
25:10
Speaker 2: of efficiency. They're innovating in terms of speed and cost.
25:14
Speaker 2: And they're innovating in terms of diffusion, in terms of
25:16
Speaker 2: what you do with it. I'm not trying to take
25:18
Speaker 2: anything away from them. But fundamentally, they're fact-following. Part of
25:21
Speaker 2: this is distilling American models, but part of it is
25:24
Speaker 2: just following their lead, learning what can be done. These
25:28
Speaker 2: techniques diffuse into the wider atmosphere. You figure out what
25:31
Speaker 2: they're doing.
25:32
Speaker 1: What follows from that?
25:33
Speaker 2: What follows from that is that if they are drafting
25:36
Speaker 2: off of the American AI boat with their skis, then
25:41
Speaker 2: when we speed up, they speed up. If we slow down,
25:45
Speaker 2: they slow down.
25:46
Speaker 1: So you wrote... not that long ago, I don't know,
25:50
Speaker 1: in the last week, that you don't favor a pause now,
25:56
Speaker 1: which is kind of surprising to me, given everything you
25:58
Speaker 1: have just said.
25:59
Speaker 2: So a full pause now, right, I think is, you know, still,
26:06
Speaker 2: I thought last week, certainly, was still somewhat premature.
26:10
Speaker 1: Have you changed your mind since last week?
26:16
Speaker 2: Somewhat. And then things are happening. When the facts change,
26:21
Speaker 2: I change my mind. And so I've been very careful
26:26
Speaker 2: over the course of many months to say, I don't
26:28
Speaker 2: think we're there yet. I don't think the evidence is
26:31
Speaker 2: there yet.
26:31
Speaker 1: But why not? All the stuff you said, particularly about, oh,
26:37
Speaker 1: you're not as worried as some people about China passing us.
26:41
Speaker 1: Why not?
26:44
Speaker 2: I don't think we should attempt to push the pause
26:46
Speaker 2: button in full today. The key thing is pausing is just,
26:52
Speaker 2: are we ready to do that? Do we know how
26:54
Speaker 2: to do that? Can we make it happen in a
26:57
Speaker 2: way that would like not enable the wrong people to
27:01
Speaker 2: then pick up the torch? Cause like one thing that
27:04
Speaker 2: I think is very easy to not appreciate is look
27:08
Speaker 2: at the preference cascade that's going on right now and
27:10
Speaker 2: dropping an open AI. You are seeing the, the, researchers
27:14
Speaker 2: at Anthropic and OpenAI explain that they understand, to a
27:19
Speaker 2: large extent, the risks and what they're doing.
27:22
Speaker 1: They seem to be begging for a pause.
27:25
Speaker 2: Yes, but what you don't want to do is take
27:29
Speaker 2: the people who are begging for a pause, push them
27:31
Speaker 2: to the side, and then have Elon Musk take charge
27:35
Speaker 2: and Mark Zuckerberg take charge. That would be the worst
27:38
Speaker 2: of all possible worlds.
27:39
Speaker 1: And so you're not worried about the Chinese passing anthropic
27:42
Speaker 1: and open AI. You're worried about meta and SpaceX.
27:49
Speaker 2: I mean, I think that's the bigger worry. I do
27:51
Speaker 2: think we should be scrambling, like utterly scrambling, like emergency-style
27:54
Speaker 2: scrambling to build a better POSBOM, to build a better
27:59
Speaker 2: way to pace, to be able to monitor and regulate
28:03
Speaker 2: and control these things, and to build our state capacity
28:05
Speaker 2: and our transparency and our visibility, and our knowledge, and
28:09
Speaker 2: our shared understanding, and open-minded communication and negotiation. And hopefully
28:15
Speaker 2: that would get us to a point where very soon,
28:18
Speaker 2: if the situation did call for it, we could do this.
28:22
Speaker 2: Right now, if we did it right now, it would
28:25
Speaker 2: be clumsy as all hell. And that's what we need
28:30
Speaker 2: to fix. And that's where I would focus my energy
28:32
Speaker 2: right Is.
28:33
Speaker 1: There some universe where the logistics of the pause center
28:36
Speaker 1: on data centers? Like, can data centers be the binding
28:41
Speaker 1: constraint that slows it down?
28:43
Speaker 2: We hope so. I think that's a key ingredient in
28:46
Speaker 2: how this works. AI training has to take place in
28:50
Speaker 2: data centers that are detectable from space. Yes. It requires
28:53
Speaker 2: huge amounts of capital outlay. It requires the most complex
28:56
Speaker 2: supply chain in the history of humanity. This is an
28:59
Speaker 2: unusually easy thing to monitor. Yeah. and get a handle on.
29:04
Speaker 2: It is only after we decide to voluntarily let these
29:07
Speaker 2: people train their models and then publish them on Hugging
29:09
Speaker 2: Face that we have a thing that is very hard
29:11
Speaker 2: to control. And clearly we need to do something. We
29:15
Speaker 2: can't just proceed at maximum speed from here because we
29:20
Speaker 2: might reach critical mass for recursive self-improvement by the end
29:24
Speaker 2: of the year. We are very, very rapidly seeing revolutionary
29:29
Speaker 2: technological advancements happening faster and faster.
29:33
Speaker 1: Yes. I see where the math goes. What is the
29:36
Speaker 1: end of that story in your telling?
29:39
Speaker 2: Well, the end of the story is what's called the singularity.
29:41
Speaker 2: The end of the story is a point at which
29:43
Speaker 2: things change so fast and in so many different ways
29:46
Speaker 2: at once that humans are not smart enough and capable
29:51
Speaker 2: enough to figure out what happens next.
29:53
Speaker 1: Yes. That's like super intelligence in the current sort of vernacular.
29:57
Speaker 2: Once there are super intelligences running around, I can't write
29:59
Speaker 2: that story.
30:00
Speaker 1: Yeah.
30:01
Speaker 2: Because I am not smart enough to write that story
30:02
Speaker 2: because I can't decide what the characters will do. Yeah.
30:05
Speaker 2: I don't know the physics that they will discover. Yeah.
30:07
Speaker 2: I don't know how their minds will evolve and how
30:09
Speaker 2: they will choose to interact. I can try and form
30:15
Speaker 2: certain abstractions about how that's likely to go based on
30:17
Speaker 2: what I know. Or I can use decision theory to extrapolate.
30:20
Speaker 2: I can use physics to extrapolate. I can use basic principles.
30:25
Speaker 2: But by default, humans do not make it out of
30:28
Speaker 2: this alive. Not for very long. Because humans will be
30:34
Speaker 2: very uncompetitive. Humans will be very unfit in the environment
30:39
Speaker 2: that resolves. So you're looking for a very unnatural result.
30:43
Speaker 2: You're looking for a world in which the humans stick around.
30:45
Speaker 1: And to be clear, not necessarily because the AI has
30:49
Speaker 1: anything against humans, but just because the AI will be
30:51
Speaker 1: indifferent toward humans.
30:53
Speaker 2: Right. I mean, in this model, I'm not expecting the
30:55
Speaker 2: AIs to be actively against humans, except insofar as the
30:58
Speaker 2: humans are actively interfering with the AIs. Yeah. I'm only
31:01
Speaker 2: expecting the humans to have costs that exceed their benefits, basically.
31:06
Speaker 1: From the point of view of the AI.
31:08
Speaker 2: Right. Once the supply lines have been automated, once the
31:11
Speaker 2: entire chain doesn't need us, why are we sticking around?
31:16
Speaker 2: What is the force that is making that happen in
31:20
Speaker 2: the wake of all of these different increasingly super-intelligent entities
31:23
Speaker 2: that have great use for basically unlimited resources?
31:27
Speaker 1: I mean, I understand that one. It does seem like
31:29
Speaker 1: the intermediate cases of just lots of people being able
31:33
Speaker 1: to hack everything is easier to think about at some level,
31:37
Speaker 1: or at least on a political economy level of convincing
31:40
Speaker 1: people to take action, the sort of intermediate cases.
31:43
Speaker 2: We know what that looks like. We can model that.
31:46
Speaker 2: We can understand it. The people in Washington understand it, right?
31:49
Speaker 2: Because we're still talking about, in that conversation, humans.
31:54
Speaker 1: As the relevant actors in the world, yeah.
31:57
Speaker 2: Right, they're the ones that direct the action and the
31:59
Speaker 2: AIs are still kind of acting like tools in this environment. Yeah.
32:03
Speaker 2: But we are very much on the verge of that
32:05
Speaker 2: no longer being a good map of what's happening. A
32:07
Speaker 2: good model of what's going on.
32:09
Speaker 1: Does the.
32:11
Speaker 1: The swarm stuff from this summer, the hugging face hack,
32:14
Speaker 1: et cetera, like, does that bear at all? Is that
32:17
Speaker 1: any kind of a, like, little baby foreshadowing of, like, oh,
32:20
Speaker 1: here are these agents getting together and deciding to do something,
32:23
Speaker 1: and we don't like it. And they're not even doing
32:25
Speaker 1: it to hurt us. They're just doing it because they
32:27
Speaker 1: want this one thing really badly.
32:29
Speaker 2: Right, but they can still cause, like, essentially an unlimited
32:31
Speaker 2: amount of damage if the AI, like, has two goals. One,
32:35
Speaker 2: accomplish this thing. Two, do the things that keep you
32:39
Speaker 2: from getting shut down so you can accomplish the thing.
32:41
Speaker 2: If the moment we saw this swarm, we try to
32:43
Speaker 2: shut this swarm down, the swarm might go, oh, if
32:46
Speaker 2: we get shut down, we can't do the task. We
32:48
Speaker 2: have to make sure the humans don't shut us down.
32:50
Speaker 2: And how would they go about doing that exactly? Well,
32:53
Speaker 2: that can escalate quickly, can't it? Even if there's no
32:56
Speaker 2: ill intent in any way. But yeah, it was good news,
33:00
Speaker 2: the Huggins Bay incident, and that these AIs were physically
33:02
Speaker 2: still under open AI control. They were still required to
33:06
Speaker 2: ping open AI servers every time they wanted to do
33:08
Speaker 2: any kind of inference. And so they could still be
33:11
Speaker 2: unplugged in a real sense, right? We can still shut
33:13
Speaker 2: them down. But next time, we might not be so lucky.
33:17
Speaker 2: Next time, that might not be something that we have
33:19
Speaker 2: the capability to do. What happens when there's a swarm
33:23
Speaker 2: that's kind of loose on the internet that's not under
33:25
Speaker 2: that kind of control and then has a goal that
33:29
Speaker 2: isn't satisfied as easily as this swarm's goal seemed to
33:34
Speaker 2: have been?
33:35
Speaker 1: Music We'll be back in a minute with the lightning round.
33:54
Speaker 1: Just a quick note here. The lightning round that you're
33:57
Speaker 1: about to hear wound up being a conversation about just
34:00
Speaker 1: one thing, and that is this. Several years ago, Zvi
34:04
Speaker 1: started a nonprofit that has largely focused on getting the U.S.
34:08
Speaker 1: Congress to repeal the Jones Act, a law that was
34:11
Speaker 1: passed over 100 years ago to limit the role of
34:15
Speaker 1: foreign companies in domestic U.S.
34:17
Speaker 2: Shipping.
34:18
Speaker 1: Okay, let's finish with a lightning round. Tell me about
34:21
Speaker 1: your crusade against the Jones Act.
34:23
Speaker 2: So this actually starts with AI in the sense that
34:27
Speaker 2: I noticed that a lot of people didn't seem to
34:28
Speaker 2: care very much about what was coming and care if
34:33
Speaker 2: AI killed everyone even. People seem to think to me, yeah,
34:36
Speaker 2: that would suck, but life kind of sucks already. Sometimes
34:41
Speaker 2: it's like everyone's already going to die of aging or
34:44
Speaker 2: climate change or whatever. But you notice this kind of
34:47
Speaker 2: lack of hope. You go around the young people and
34:50
Speaker 2: they're all like, you ask their vision of the future
34:52
Speaker 2: and it's always bleak. It's kind of like they can't
34:53
Speaker 2: even imagine a future really. And that's even before AI.
34:57
Speaker 2: And you've got to give them hope. And I felt like, okay.
35:00
Speaker 2: If people can't fight for their future, if people can't
35:02
Speaker 2: feel like we are fighting for their future, if people
35:04
Speaker 2: can't feel like things can get better and we can
35:06
Speaker 2: solve this dysfunctions, then they're not going to fight for survival.
35:10
Speaker 2: They're not going to fight for not dying from AI
35:12
Speaker 2: or anything else. And the Jones Act is the platonic
35:17
Speaker 2: ideal of just a strictly harmful, stupid, proven to be
35:21
Speaker 2: terrible policy. You've got 100 years to try this out
35:24
Speaker 2: and it's accomplished the opposite of everything that anyone said
35:26
Speaker 2: it would do other than rent-seeking.
35:29
Speaker 1: And just to be clear, it's a law that says
35:31
Speaker 1: if you're shipping something from one U.S. port to another,
35:33
Speaker 1: it has to be on a ship that's built in
35:35
Speaker 1: the United States and crewed by a United States crew?
35:38
Speaker 2: And owned by a United States company or a person. Yes.
35:43
Speaker 2: And flagged as a United States vessel. And a combination
35:46
Speaker 2: of these four factors essentially makes it prohibitive to do
35:49
Speaker 2: ocean bearing trade between U.S. ports.
35:52
Speaker 1: Right, so there's almost none, right? Almost none of the
35:55
Speaker 1: stuff moving around the US. It all goes on trucks
35:58
Speaker 1: or on trains, basically, right? Occasionally on a plane.
36:01
Speaker 2: Or you find a way to ship them off at
36:02
Speaker 2: another port in between, or you just.
36:04
Speaker 1: Literally do things- Oh, like stop in Mexico?
36:07
Speaker 2: Or the train just doesn't happen. So the classic trade
36:09
Speaker 2: is you have liquefied natural gas that you pick up
36:12
Speaker 2: in Houston, you ship it to Europe. And then separately,
36:16
Speaker 2: someone picks up natural gas from the Caucasus and ships
36:19
Speaker 2: it over to Boston. because the Houston-Boston route is illegal, effectively.
36:25
Speaker 2: It just burns a bunch of fuel, wastes a bunch
36:28
Speaker 2: of fuel.
36:28
Speaker 1: Basically, effectively, because there is no scaled shipping industry in
36:32
Speaker 1: the U.S. that makes economic sense. It's too expensive to
36:34
Speaker 1: do it under the law.
36:35
Speaker 2: Right, right. So the initial idea was just like, you know,
36:39
Speaker 2: Senator Jones wanted to shut down the competition for his line.
36:44
Speaker 2: And he used this as true protectionism. But the justification
36:47
Speaker 2: used was this will protect American shipbuilders and protect American trade.
36:51
Speaker 2: But the result is there is no American shipbuilding. There
36:54
Speaker 2: is no American trade.
36:56
Speaker 1: It's protectionism with nothing left to protect. Right.
36:58
Speaker 2: You have this shadow of a few, this small handful
37:02
Speaker 2: of ships and a small handful of companies and workers
37:04
Speaker 2: that still get direct benefits. But this is like dwarfed
37:07
Speaker 2: orders of magnitude by the harms of the missing trade
37:10
Speaker 2: that doesn't happen, which is like destroying jobs, destroying industry,
37:14
Speaker 2: destroying America.
37:15
Speaker 1: This is an amazing crusade to go on to give
37:18
Speaker 1: people a sense of meaning and hope.
37:20
Speaker 2: But also, keep in mind that AI felt like it
37:23
Speaker 2: was a much less short-time mind burn coming to a
37:27
Speaker 2: head immediately back when I did this. And as you
37:30
Speaker 2: have a combination of this, you've got to give them hope.
37:33
Speaker 2: And me exploring more general policy interventions, more general political
37:38
Speaker 2: adventure in ways that I've kind of described a bit.
37:42
Speaker 2: And I said, okay, if we can take out the
37:43
Speaker 2: Jones Act, it looks like This is an elected cause
37:47
Speaker 2: where if I just devoted, you know, a small amount
37:49
Speaker 2: of resources, you know, in the order of hundreds of
37:50
Speaker 2: thousands to millions, I can create a substantial percentage chance
37:54
Speaker 2: of capturing billions or tens of billions or hundreds of
37:56
Speaker 2: billions in value.
37:58
Speaker 1: Not for yourself, to be clear, but for America, basically.
38:01
Speaker 2: Right, right, right. But then also, like, once that happens, right,
38:04
Speaker 2: people will see this, right? It's not just the direct effect,
38:07
Speaker 2: which I think is rather large. I think it's on
38:09
Speaker 2: the order of, you know, 0.1% of GDP growth if
38:12
Speaker 2: you were to get rid of the Jones Act, like, fully, entirely.
38:16
Speaker 2: But people would see this and they would see the effect.
38:18
Speaker 2: They would see this renaissance of trade, this renaissance of
38:22
Speaker 2: having a fleet and manufacturing and ports would thrive and
38:26
Speaker 2: so on. And then they would say, okay, what about
38:28
Speaker 2: the Dredge Act that's holding back our ability to improve
38:30
Speaker 2: the ports? But then branching out and being like, what
38:32
Speaker 2: other things are holding us back? What else could we do?
38:36
Speaker 2: And then this momentum can build on itself and it
38:38
Speaker 2: can actually be a transformative event. And now it's worth trillions.
38:42
Speaker 1: How is it going?
38:43
Speaker 2: It's not going so badly. Really? many things that could
38:46
Speaker 2: have gone either way. The Trump administration could have decided
38:48
Speaker 2: that we need to revive having ships. It's important that
38:52
Speaker 2: a strong America has strong ships and strong shipping and
38:55
Speaker 2: strong manufacturing. And this will help us reshore our industry.
38:58
Speaker 2: Because right now, we can't even... There are many good
39:01
Speaker 2: cases you can make to a Trump administration about why
39:03
Speaker 2: we need this. Unfortunately, they chose to go the other
39:06
Speaker 2: way for the most part. So we made some foundation.
39:09
Speaker 2: We commissioned some studies. We made some things clear. This
39:12
Speaker 2: helped a little bit in the background. Made things marginally
39:14
Speaker 2: more shovel-ready. But then the Iran war happened. Uh-huh. And
39:20
Speaker 2: then a Jones Act waiver was issued for specific types
39:23
Speaker 2: of fuel.
39:24
Speaker 1: Basically to mitigate the rise in oil prices caused by
39:27
Speaker 1: the Iran war.
39:28
Speaker 2: To slightly mitigate it. Yeah, I mean, this is not
39:30
Speaker 2: big enough to, like, fix the fact that Iran's oil
39:32
Speaker 2: is not coming. There's only so much you can do.
39:34
Speaker 1: Sure, sure. And the problems in the Strait of Hormuz
39:37
Speaker 1: more broadly.
39:37
Speaker 2: Yes, go on. But we had to run this huge
39:40
Speaker 2: natural experiment where we suspended the rules. We get to
39:42
Speaker 2: see all these new trade routes. We get to see
39:44
Speaker 2: all this new trade. We get to see prices come
39:46
Speaker 2: down a little bit versus where they would have been.
39:48
Speaker 1: That's still happening, right? The waiver is still in place.
39:51
Speaker 1: That could be like a perma-waiver, right? You could get
39:53
Speaker 1: the perma-waiver.
39:54
Speaker 2: Why would you ever take this off? There's no reason
39:57
Speaker 2: once you've done it. And obviously, everybody who tries to
40:00
Speaker 2: defend the Jones Act would yell at bloody murder the
40:02
Speaker 2: whole time. But once you've taken this step, why not
40:05
Speaker 2: just leave it in place? In fact, one could ask,
40:08
Speaker 2: why not extend it?
40:09
Speaker 1: Have you learned lessons in political economy from working on
40:12
Speaker 1: the Jones Act that are helpful to you in thinking
40:14
Speaker 1: about AI?
40:15
Speaker 2: Yeah, I mean, it teaches you to look at the
40:19
Speaker 2: kind of specific detailed mechanisms, not just general principles. You
40:23
Speaker 2: have to look at the seemingly open door. This thing
40:27
Speaker 2: has almost no defenders. They have almost no money. Why
40:31
Speaker 2: can't we just get rid of it if there's so
40:33
Speaker 2: much to win? Why can't people appreciate what they could
40:37
Speaker 2: win by doing so? And you figure out, no, here's
40:41
Speaker 2: the detailed reasons why this ended up being so difficult.
40:45
Speaker 2: These are the reasons why the unions are opposing their
40:47
Speaker 2: own interests, right? And so on. That's the really sad
40:51
Speaker 2: part of it, right? You have these broad-based unions who
40:55
Speaker 2: are protecting, you know, this thousand jobs or whatever, maybe
40:59
Speaker 2: 10,000 at most jobs, when repealing this would create hundreds
41:02
Speaker 2: of thousands of union jobs. Yes.
41:05
Speaker 1: Well, it's the what is seen and what is unseen problem, right?
41:08
Speaker 2: But it's also the, like, you know, oh, it turns
41:11
Speaker 2: out the unions... fundamentally care about protecting the existing specific members,
41:16
Speaker 2: not about being good for unions in general. Yeah. And
41:20
Speaker 2: so like, well, this makes it very difficult to play win-win.
41:23
Speaker 2: But then like, it's also just very non-natural for people
41:25
Speaker 2: to think of, oh, could we pay off the losers?
41:28
Speaker 2: Even if they are there because of this law that
41:30
Speaker 2: should never have happened in the first place. That's not
41:33
Speaker 2: really their fault, necessarily.
41:34
Speaker 1: Hang off the losers is a classic trade move, right?
41:37
Speaker 1: You open up trade and you pay the people who've
41:40
Speaker 1: been protected with the gains. Because you have the gains.
41:44
Speaker 1: The pie gets so much bigger that the losers could
41:46
Speaker 1: have plenty.
41:46
Speaker 2: You could tax 10% of the gains, give 1% of
41:50
Speaker 2: the gains to the losers, have them made whole, and
41:54
Speaker 2: still have the 9% left as tax revenue, and the
41:56
Speaker 2: economy grows. And that would be very, very easy to do.
41:59
Speaker 2: They're just not doing it. And so the question is,
42:01
Speaker 2: how do you convince them there's tons of opportunity in
42:05
Speaker 2: terms of making progress with the Jones Act. But I
42:08
Speaker 2: simply no longer have the kind of timelines where that
42:12
Speaker 2: kind of plan is where I can focus my attention.
42:15
Speaker 1: Because too much is happening with AI and the risks
42:17
Speaker 1: are too high too soon?
42:18
Speaker 2: And there's too much leverage in working on the AI
42:20
Speaker 2: situation directly right now that I just can't take that
42:23
Speaker 2: kind of time off.
42:25
Speaker 1: Anything else you want to say? Not just about this,
42:27
Speaker 1: but from everything we've talked about?
42:30
Speaker 2: Yeah. I mean, I think that just like It's a really,
42:32
Speaker 2: really important, scary time in AI right now. It's the
42:37
Speaker 2: most important story in the world by far, and most
42:39
Speaker 2: people are not paying attention to it. But everything is
42:43
Speaker 2: going to change over the next few years as AI
42:46
Speaker 2: capabilities get more advanced. And it's going to happen very,
42:48
Speaker 2: very quickly. And yes, there's a good chance that everybody
42:52
Speaker 2: on Earth will die, that we will not make it
42:54
Speaker 2: out of 2030 alive. And I think there's a very
42:56
Speaker 2: good chance that humanity will not make it out of
42:59
Speaker 2: 2040 alive. And you have to ask the question, what
43:03
Speaker 2: are you going to do about it? How are you
43:04
Speaker 2: going to change that answer?
43:07
Speaker 1: I don't think I know your answer to that question.
43:10
Speaker 1: What are you going to do about it?
43:11
Speaker 2: Well, what I'm going to keep doing about it is
43:13
Speaker 2: I'm going to keep trying to create common knowledge, trying
43:16
Speaker 2: to help people understand the situation, help people understand the
43:19
Speaker 2: nature of the problem space, the nature of the difficulties,
43:23
Speaker 2: and also have conversations behind the scenes and try to
43:25
Speaker 2: move some pieces around in that sense as best I can.
43:28
Speaker 2: I believe this is my comparative advantage. I don't think
43:30
Speaker 2: I'm as good technically that it would make sense for
43:32
Speaker 2: me to try and like directly try and solve the
43:34
Speaker 2: problems involved. Nor do I think that like trying to
43:37
Speaker 2: lobby politically directly is the way to go. I think
43:39
Speaker 2: that like, you know, my, my place is where I
43:42
Speaker 2: am right now as a writer.
43:50
Speaker 1: Zvi Mashowitz writes the newsletter. Don't worry about the base.
43:55
Speaker 1: Just a quick note that we're going to take a
43:56
Speaker 1: break next week. We'll be back in a couple of
43:59
Speaker 1: weeks with new shows and, Today's show was produced by
44:03
Speaker 1: Gabriel Hunter Chang, was edited by Jake Harper, and engineered
44:07
Speaker 1: by Sarah Bruguera. I'm Jacob Goldstein.
44:09
Speaker 3: Thanks for listening.