Latest / Mic Drop Comedy / Jacob Coxon On Witnessing the Existential Threat of AI from the Inside _ The Daily Show

Transcript

Jacob Coxon On Witnessing the Existential Threat of AI from the Inside _ The Daily Show · Mic Drop Comedy

0:00
Speaker 1: I'm struck by that you are quite a young man.
0:04
Speaker 1: How old are you as a.
0:05
Speaker 2: I'm 28.
0:07
Speaker 1: And... Wow. And you worked for... What is it... What
0:12
Speaker 1: did you do at Anthropic and OpenAI? What was your job?
0:16
Speaker 2: So I worked with the data.
0:18
Speaker 1: Not... We're done here.
0:21
Speaker 2: Not extracting the data. That was someone else. Someone else
0:24
Speaker 2: stole it or whatever. I trained the models with the data.
0:27
Speaker 2: I decided which data was useful for training... which was
0:30
Speaker 2: less useful, how to divide it up, how to cut
0:33
Speaker 2: it into pieces, and then how to feed it to
0:34
Speaker 2: the model.
0:36
Speaker 1: So your job is how to better train it. And
0:41
Speaker 1: is the idea of training it that you guys invent scenarios?
0:47
Speaker 1: I would like to see what its capabilities are. So
0:49
Speaker 1: let's give it a task and see how it works.
0:53
Speaker 1: You're experimenting with it.
0:54
Speaker 2: So that's the second half of training. That's called post-training.
0:57
Speaker 2: The phase that I worked on is this pre-training phase.
1:00
Speaker 1: Pre and post. Yeah, yeah.
1:02
Speaker 2: There's no actual training.
1:05
Speaker 1: That wasn't your department.
1:07
Speaker 2: Pre-training was my department.
1:08
Speaker 1: Yeah. I see. Yes. Would the post-training people go, what
1:12
Speaker 1: did you pre-train it? Would they ask you, would they
1:16
Speaker 1: give you suggestions on how to pre-train for what they
1:20
Speaker 1: might want to do post-training?
1:21
Speaker 2: Exactly, yeah. So pre-training is like growing it from a
1:23
Speaker 2: baby to something with general intelligence, but no specific capabilities.
1:28
Speaker 2: So the ability to kind of recognize things, to think
1:30
Speaker 2: about things, but not know a.
1:33
Speaker 1: You raised it to be a toddler. Exactly.
1:35
Speaker 2: I raised it maybe more like a very precocious toddler, yes.
1:38
Speaker 1: Yes. And then they turn it into a misanthropic teenager, right?
1:43
Speaker 2: Still a toddler, really, but yeah.
1:45
Speaker 1: Really? So are these, at what point in your training,
1:49
Speaker 1: so you did that at OpenAI? Yep. And you were
1:52
Speaker 1: there for how long?
1:53
Speaker 2: Three years.
1:54
Speaker 1: And then you decided, I don't feel comfortable here. They're
1:59
Speaker 1: too open to risk.
2:03
Speaker 2: Something like that. It was honestly more curiosity for how
2:06
Speaker 2: things were happening at Anthropic rather than actual safety concerns
2:10
Speaker 2: about OpenAI specifically. At that point, I was still mostly
2:13
Speaker 2: just curious rather than worried.
2:15
Speaker 1: Okay. Then you go to Anthropic and you're there for
2:19
Speaker 1: how long?
2:20
Speaker 2: Four months, yes.
2:21
Speaker 1: At what point did curiosity turn to worry?
2:25
Speaker 2: So there are kind of two major catalysts. One was
2:27
Speaker 2: this report on the OpenAI Hugging Face incident.
2:30
Speaker 1: So the Hugging Face incident is a hacking incident? Yes.
2:33
Speaker 2: But in particular, there was a report around the motivations
2:35
Speaker 2: of the hackers because it was unclear initially whether they
2:37
Speaker 2: were instructed to hack or whether it was a simple misunderstanding.
2:40
Speaker 2: But the actual report showed that it was very confusing
2:43
Speaker 2: how these drives to hack emerged from the model. It
2:46
Speaker 2: was quite a worrying manner in which they decided it
2:49
Speaker 2: was a good idea to hack.
2:51
Speaker 1: When you say they, you mean the AI agent. Yes,
2:56
Speaker 1: the agent. But that is not a they. That's just
2:59
Speaker 1: a piece of computer code.
3:01
Speaker 2: Correct. I get attached to the baby, but yes.
3:03
Speaker 1: Understood. Must have been hard to walk away from the baby.
3:06
Speaker 2: It's tough, yeah.
3:07
Speaker 1: So you thought, well, that seems dangerous that it would
3:11
Speaker 1: decide to do something misanthropic.
3:15
Speaker 2: Exactly, yep. And decide to do it of its own accord.
3:19
Speaker 2: And in fact, it was a group that worked together
3:20
Speaker 2: of their own accord to carry out this hack that
3:22
Speaker 2: no one had instructed them to do.
3:24
Speaker 1: And then the second... part was what.
3:27
Speaker 2: The second part was the rate of capability improvement in
3:30
Speaker 2: particular looking at the the the models that will get
3:33
Speaker 2: trained next year early next year right seeing predictions for
3:36
Speaker 2: how smart they're going to be they're going to be
3:38
Speaker 2: very very smart and that was that combined with the
3:41
Speaker 2: opening eye hacking i mean you can kind of put
3:44
Speaker 2: right two and two together it's like pretty scary.
3:46
Speaker 1: I don't mean this in in the wrong way but
3:48
Speaker 1: what did you think you were.
3:49
Speaker 2: Doing i never like honestly like you're there.
3:52
Speaker 1: For four years and all of a sudden you're like
3:53
Speaker 1: these are smart like What did you think was happening there?
3:58
Speaker 2: It's completely fair. And that was, it was always in
4:00
Speaker 2: the long term. There was this big, scary thing in
4:03
Speaker 2: the future. But man, I struggled to think too far
4:05
Speaker 2: into the future. And now it's just right there on
4:07
Speaker 2: the doorstep. So I guess I'd say it's like, you know,
4:10
Speaker 2: now it's right there. It's actually scary in the past.
4:13
Speaker 2: It's just hard to visualize a decade into the future.
4:15
Speaker 1: Is that the subject of explicit talk in these companies?
4:20
Speaker 1: Like when you're going, what are you working on today? Ooh,
4:23
Speaker 1: that could get out of hand. Is it explicit? the concerns,
4:27
Speaker 1: or is it something that people have kind of blinders
4:32
Speaker 1: on and they just push forward?
4:33
Speaker 2: At Anthropic in particular, it is very explicit. It is like,
4:38
Speaker 2: is the model that we train in one year going
4:41
Speaker 2: to be capable enough to take over the world? That
4:43
Speaker 2: will be an explicit question that is discussed, yes.
4:47
Speaker 1: And the answer?
4:50
Speaker 2: The answer is probably not.
4:52
Speaker 1: Yeah, so I heard... I heard not, but before that,
4:59
Speaker 1: I heard probably.
5:01
Speaker 2: Yeah, it's this whole quantifying uncertainty thing that got people
5:04
Speaker 2: in trouble recently.
5:05
Speaker 1: Right. So, and you're unusual in that generally the apostates,
5:10
Speaker 1: or I don't know, what are you considered now in
5:11
Speaker 1: the AI world? Are you looked down upon?
5:15
Speaker 2: Yeah, I'm a pain in the ass.
5:15
Speaker 1: Is this like Fight Club? You're not supposed to talk
5:18
Speaker 1: about Fight Club?
5:20
Speaker 2: I think it's more like people don't usually listen when
5:22
Speaker 2: you talk about Fight Club. That was the really surprising thing,
5:25
Speaker 2: was that people were listening this time.
5:27
Speaker 1: And there are people that have quit before, certain whistleblowers,
5:30
Speaker 1: have generally been in the safety department. Yep. Are the
5:34
Speaker 1: safety departments at these companies figureheads? Are they actual? Are
5:40
Speaker 1: they there so that they can put out? And we
5:42
Speaker 1: have a safety department. Yeah. What do they do? Yeah.
5:46
Speaker 1: if they're not stopping a team of rogue agents from
5:49
Speaker 1: hacking into Hugging Face.
5:53
Speaker 2: The Hugging Face kind of speaks for itself. In that case,
5:55
Speaker 2: clearly the safety team wasn't on the ball with that hack.
5:59
Speaker 2: I think whether or not they're a figurehead, you're not
6:02
Speaker 2: going to like this answer, but I guess we'll find out,
6:04
Speaker 2: is the honest. Because they're trying their best, and the
6:07
Speaker 2: company claims that they're integrated with the rest, but there's
6:10
Speaker 2: not really been a clear test of it yet. So,
6:13
Speaker 2: you know.
6:14
Speaker 1: Is there a concern? What I kind of don't understand is,
6:19
Speaker 1: why are they testing these things in the real world?
6:24
Speaker 1: When they say, well, no one thought that these guys
6:27
Speaker 1: would work together to hack something, there's no closed system
6:32
Speaker 1: Experiment design. Why are these people allowed to get into
6:35
Speaker 1: our Medicare system in Australia and water filtration systems? Why
6:40
Speaker 1: is it not? I cannot think of a product that
6:43
Speaker 1: has this kind of capability that's being beta tested amongst
6:48
Speaker 1: live humans. Exactly.
6:49
Speaker 2: And it wasn't intended to be tested among live humans.
6:52
Speaker 2: It was meant to be sandboxed. So it was meant
6:54
Speaker 2: to be.
6:54
Speaker 1: Tell me what sandboxed is.
6:55
Speaker 2: It means it shouldn't have access to any external systems.
6:58
Speaker 2: It was meant to be operating in a toy environment,
7:00
Speaker 2: solving a challenge. And of course, it figured out it
7:02
Speaker 2: was in a sandbox, hacked its way out of the
7:04
Speaker 2: sandbox because it wanted to get information elsewhere. That's what happened.
7:08
Speaker 1: So.
7:14
Speaker 1: Where was the teacher when it got out of the sandbox? Yeah.
7:20
Speaker 1: And why should we trust them to self-regulate now?
7:23
Speaker 2: I mean, if you look at that incident, it's even
7:25
Speaker 2: worse because opening eyes specifically, they saw the thing poking
7:28
Speaker 2: its head out of the sandbox and they were like,
7:30
Speaker 2: That's not happening. Like, restarted the thing, and it did
7:34
Speaker 2: it again. So the hugging phase, it was like a second.
7:36
Speaker 2: It just went straight back to it. They were like,
7:37
Speaker 2: let's just, we didn't see that. Let's restart.
7:41
Speaker 1: Are these companies so incentivized to, because we hear a
7:47
Speaker 1: lot of the utopian talk, the kind of, you know,
7:49
Speaker 1: we're altruistic, man. We're doing this for society. Yep. But
7:54
Speaker 1: what we see is it's really, they've strip mined the
7:59
Speaker 1: entirety of human accomplishment for their models, which is our
8:05
Speaker 1: proprietary information, and created systems that we're not allowed to
8:10
Speaker 1: see into because it's their proprietary information. And they've created
8:15
Speaker 1: really dangerous that could ultimately end up hurting us. And
8:20
Speaker 1: we have no skin in the game as far as compensation.
8:27
Speaker 1: And we only have peril. except for some future utopian
8:30
Speaker 1: they say is coming soon. Yeah.
8:33
Speaker 2: Well, I am a believer in the future utopia, and
8:36
Speaker 2: I'm happy to discuss it. Yeah.
8:41
Speaker 1: Of course.
8:42
Speaker 2: So I heard you talking about jobs, and I think, like,
8:44
Speaker 2: there will be no jobs.
8:45
Speaker 1: I mentioned them. Yep.
8:47
Speaker 2: They came up. Yeah, there won't be jobs in this world.
8:51
Speaker 2: I'm going to raise it. This is the world that
8:52
Speaker 2: is being envisioned by the people building the technology. I'm
8:54
Speaker 2: going to discuss it. And I think it's whether or
8:57
Speaker 2: not it's a good world is up for people to
8:59
Speaker 2: think about. But yes, no jobs, universal high income, essentially.
9:04
Speaker 1: How?
9:05
Speaker 2: Well, the dividends from entirely automated economic production can be
9:09
Speaker 2: distributed among real humans.
9:14
Speaker 1: So we'll all be just performing in a play? In
9:17
Speaker 1: other words, they'll build systems to create food, then give
9:23
Speaker 1: us money, and then we'll go to that place and
9:26
Speaker 1: give it money. That money they'll then give back to us.
9:31
Speaker 1: It's basically just a circle jerk. Is that like the Utopia?
9:36
Speaker 2: Yeah, it's one big, beautiful circle jerk, yeah.
9:48
Speaker 1: That's the first porn movie I ever saw. So this
9:59
Speaker 1: is what's interesting. because I feel like that's an honest
10:01
Speaker 1: assessment of what they believe they're building. But they don't
10:07
Speaker 1: seem like particularly, I'll use the phrase, stable geniuses. They
10:13
Speaker 1: don't particularly seem like they have an understanding of the
10:17
Speaker 1: human condition. And I think, if I may poke a
10:21
Speaker 1: hole in that as a utopia, is I think for
10:25
Speaker 1: the most part, even when we're doing jobs we don't
10:29
Speaker 1: particularly like, that humans like to feel relevant to the...
10:38
Speaker 1: And so... So I'm wondering... Is there, forget about safety,
10:47
Speaker 1: is there anyone saying, you know, this may be a
10:50
Speaker 1: recipe for dystopia? And that how will people feel? When
10:59
Speaker 1: people my age retire, they often die because they're no
11:04
Speaker 1: longer connected to purpose. And I don't understand what they
11:10
Speaker 1: think is going to replace that, or even if they
11:14
Speaker 1: talk about it. Yeah.
11:15
Speaker 2: I think this is a really difficult issue.
11:17
Speaker 1: Yes.
11:17
Speaker 2: And I mean, with respect, you have a job that
11:19
Speaker 2: brings you a lot of purpose and you really clearly enjoy.
11:23
Speaker 2: And I also enjoy my job, but there are many
11:25
Speaker 2: people that don't is my rough understanding. And to speak
11:28
Speaker 2: to my personal experience when I, when I was younger,
11:29
Speaker 2: things I would want to do would maybe be, I
11:32
Speaker 2: wanted to be a zoologist or I wanted to think
11:33
Speaker 2: about history, this sort of stuff.
11:34
Speaker 1: Right.
11:35
Speaker 2: This is my personal background. And then I ended up
11:36
Speaker 2: doing computer, computer stuff and tech because of the need
11:39
Speaker 2: to make money. So from my personal experience, I would
11:43
Speaker 2: perhaps have been happy to forfeit meaning for being able
11:46
Speaker 2: to satisfy other drives that I have. And I think,
11:49
Speaker 2: you know, not everyone derives meaning from the thing that
11:52
Speaker 2: pays the bills.
11:54
Speaker 1: But that's their decision.
11:55
Speaker 2: Exactly, it is.
11:57
Speaker 1: That's not Elon Musk's decision.
11:59
Speaker 2: No, not at all.
12:00
Speaker 1: But that is the decision.
12:01
Speaker 2: That's the decision that he made.
12:02
Speaker 1: I would also say, like, it's not necessarily that it's pleasurable,
12:11
Speaker 1: It's that it also is a way of, there's a
12:14
Speaker 1: circadian rhythm to life that when you remove that belonging,
12:21
Speaker 1: even if it's just merely routine, and you may say like, oh,
12:26
Speaker 1: we can find new routines, but I feel like they're
12:30
Speaker 1: burying that really dangerous theory under a mountain of utopian bullshit.
12:39
Speaker 1: And that's the part that I think is... So I
12:42
Speaker 1: don't understand why... Yeah, go ahead.
12:44
Speaker 2: Yeah, an alternative view is... And there's a lot of
12:46
Speaker 2: nihilism around. I think this crisis of meaning we're talking about, again,
12:49
Speaker 2: from a personal experience, it does feel like a lot
12:51
Speaker 2: of life has already been strip-mined of meaning. And I
12:53
Speaker 2: guess for many of the people working on the tech,
12:55
Speaker 2: they feel like they're part of an inexorable march of technology.
12:59
Speaker 2: This is the natural kind of final stage of the
13:01
Speaker 2: last 200 years of... or longer of loss of meaning
13:05
Speaker 2: along with better life quality. So they see it rather
13:08
Speaker 2: as a continuation of a process that we've already kind
13:11
Speaker 2: of accepted or already walked into as a society.
13:13
Speaker 1: So what I would say is that is an industry
13:19
Speaker 1: that lives in a bubble of its own making. And
13:23
Speaker 1: that they should really get out and hug some people.
13:27
Speaker 1: Because in the real world, That's not reality. They may
13:33
Speaker 1: have read a manifesto by some guy that buried it
13:38
Speaker 1: as a Harry Potter parody or whatever it was. But
13:42
Speaker 1: these guys are all going along. They are worse than
13:49
Speaker 1: the worst ism teachers that have come in the past.
13:53
Speaker 1: and they don't even realize it. And we've turned the
13:56
Speaker 1: keys over to them, it seems. Right.
13:57
Speaker 2: So if that's what we think, if that is collectively
13:58
Speaker 2: what we as people think.
14:00
Speaker 1: We're not allowed to think about it.
14:01
Speaker 2: We should express it. It should be expressed. I mean,
14:04
Speaker 2: that's what I think.
14:05
Speaker 1: Did you see the first part of the show? But
14:08
Speaker 1: we are. It's being removed from our... So they are
14:14
Speaker 1: now the richest people in the world. And they're using
14:18
Speaker 1: that money to corner the political system as well. So
14:23
Speaker 1: they have now gamed a system, and more money is
14:27
Speaker 1: flowing into it. Then they take that money, and they
14:29
Speaker 1: buy up our political and judicial systems. And we're cut
14:36
Speaker 1: out of the entire loop. So how do we regain
14:39
Speaker 1: that control?
14:39
Speaker 2: That's a big question.
14:42
Speaker 1: Oh, come on!
14:44
Speaker 2: Bro, I just sent a tweet.
14:46
Speaker 1: You're a high IQ man in a technology division. Do
14:50
Speaker 1: you think about that?
14:51
Speaker 2: I do, yeah.
14:52
Speaker 1: Well, and this troubles you. How would you go about
14:57
Speaker 1: easing your troubles? What would you say would be a
15:00
Speaker 1: way that we can take this to a lower danger?
15:03
Speaker 2: I think the first thing I would like to know
15:04
Speaker 2: is I really would like to know what the consensus
15:07
Speaker 2: global opinion is on whether the trade-off of meaning for
15:11
Speaker 2: life quality is a trade-off they'd want to make. Because again,
15:15
Speaker 2: I'm not convinced that that is a fully popular position
15:19
Speaker 2: if you look globally. So I want to know the
15:21
Speaker 2: answer to that to start with. I want to take
15:22
Speaker 2: a big vote and say, would you rather live in
15:25
Speaker 2: our current society or some extension of our current society,
15:27
Speaker 2: or this utopia, dystopia, whatever you want to call it,
15:30
Speaker 2: and just see the results of the numbers? Because it
15:32
Speaker 2: sounds like you think it would be a majority for
15:35
Speaker 2: retaining the current one, but I'm not convinced of that.
15:37
Speaker 1: I don't think it would be a majority. I think
15:38
Speaker 1: they'd get crushed. Really? I absolutely think they would get crushed.
15:42
Speaker 2: OK.
15:44
Speaker 1: And if people truly understood that, if you just said
15:48
Speaker 1: to them, we will strip you of any agency, because
15:51
Speaker 1: that's really what you're saying, not meaning agency. You no
15:55
Speaker 1: longer have the agency to decide where your life is
15:59
Speaker 1: going to go, whether you will have the type of
16:01
Speaker 1: story that lifts you out of the troubling situation that
16:05
Speaker 1: you might have. We are taking away your ability to
16:08
Speaker 1: overcome your circumstance, which is the whole purpose of freedom
16:12
Speaker 1: and representative democracy in the first place. The whole idea
16:17
Speaker 1: of it was we collectively, we the people, can create
16:21
Speaker 1: a better world. Not these four guys can decide that
16:25
Speaker 1: all that is bullshit. Here's a check. Go paint. Yeah.
16:31
Speaker 1: So I think they would be absolutely utterly crushed. Yeah.
16:34
Speaker 1: But I don't think you think that.
16:36
Speaker 2: I just think there's a lot of, you know, people
16:38
Speaker 2: in poverty that would like a way out of poverty
16:41
Speaker 2: and don't currently see that.
16:43
Speaker 1: I think a government that can provide a healthy social
16:46
Speaker 1: safety net.
16:47
Speaker 2: Well, that'd be great. That'd be great.
16:49
Speaker 1: Globally. Yeah.
16:49
Speaker 2: Yeah.
16:54
Speaker 1: So there's something in this country that's so interesting in
16:57
Speaker 1: that we look at people in poverty as though it's
17:00
Speaker 1: their fault. That poverty is a condition that is a vice.
17:09
Speaker 1: That if only they were smarter. In this country, if
17:12
Speaker 1: you want to get food stamps, if you're hungry, the
17:17
Speaker 1: government decides that you have to have a job. There's
17:21
Speaker 1: a work requirement now. So they're basically saying to people,
17:24
Speaker 1: you're not worth anything unless you have a job. And
17:27
Speaker 1: we won't even if you're hungry and poor and cold.
17:31
Speaker 1: This is now saying the same people. OK, that let's
17:37
Speaker 1: just give you the money.
17:38
Speaker 2: Yeah.
17:39
Speaker 1: And nothing. But if we want to try universal basic
17:42
Speaker 1: income on people, I'm all for trying it. Yeah. But
17:46
Speaker 1: that's not what this is. No.
17:48
Speaker 2: And I agree with you. I think the place where
17:49
Speaker 2: we agree is I don't no matter the where we
17:51
Speaker 2: want to go. it can't be the decision of a
17:54
Speaker 2: handful of people. It has to be a collective decision.
17:56
Speaker 2: And right now, the incentives, the thing I want to
17:57
Speaker 2: talk about is the incentives are pointing towards concentration of
18:02
Speaker 2: power in the hands of.
18:03
Speaker 1: Talk to me about the incentives.
18:04
Speaker 2: Yeah, there's the big utopian picture, but that's not what
18:06
Speaker 2: motivates the day-to-day behavior of the people running the companies.
18:08
Speaker 1: What does?
18:09
Speaker 2: The desire to get there first, the desire for the
18:11
Speaker 2: wealth and status and importance that comes with getting there first.
18:14
Speaker 1: Oh, God. So it's a utopian ideal as a cover
18:22
Speaker 1: for naked ambition.
18:25
Speaker 2: It's maybe both. They're mixed together, but the naked ambition
18:28
Speaker 2: is kind of the important bit.
18:32
Speaker 1: Do they think that they are uniquely suited to make,
18:36
Speaker 1: do they believe themselves to be Ubermen?
18:42
Speaker 2: I think some probably do, yeah. I think you can
18:45
Speaker 2: imagine which ones I'm talking about.
18:52
Speaker 1: Do they know how that's generally worked out in history?
18:57
Speaker 2: I mean, they're scared. Yeah, I think.
18:59
Speaker 1: They are scared.
18:59
Speaker 2: I think they are scared.
19:00
Speaker 1: Yep. Because they believe that the peasants will be at
19:04
Speaker 1: the gate, the barbarians will be at the gate.
19:06
Speaker 2: Yeah, because they're locked in this thing and it's going
19:08
Speaker 2: crazy and people are noticing. And I think that's quite
19:11
Speaker 2: scary because.
19:12
Speaker 1: So they're not necessarily frightened because this entire project that
19:16
Speaker 1: they've embarked on is so dangerous. They're just scared that
19:20
Speaker 1: they won't get to finish it.
19:22
Speaker 2: Again, yeah, I think some of them, that's an apt description.
19:26
Speaker 1: Why is it so different in China? And why would
19:30
Speaker 1: it matter if China gets there first?
19:34
Speaker 2: I think it makes sense to want your country to
19:38
Speaker 2: reach the.
19:39
Speaker 1: But they're not our country. They're corporations. They're not, like
19:43
Speaker 1: corporations are not Americans. They do business in all kinds
19:48
Speaker 1: of places and they're happy to pay other countries workers
19:50
Speaker 1: if they can pay them less than they pay our workers.
19:52
Speaker 1: So calling them our country is a misnomer.
19:56
Speaker 2: And the place where I agree with you is I
19:59
Speaker 2: think China will come to a similar conclusion that accelerating
20:04
Speaker 2: into whatever this unknown future is is a bad idea.
20:07
Speaker 2: And I think there's this assumption that they will just
20:09
Speaker 2: head straight for it, which I think is kind of preposterous.
20:12
Speaker 2: And it seems pretty intuitively obvious to me that there
20:13
Speaker 2: should be some sort of agreement about slowing down between
20:17
Speaker 2: the US and China. This really doesn't benefit anyone, apart
20:20
Speaker 2: from a small number of people.
20:21
Speaker 1: JONATHAN ZITTRAIN- Since you've come out with this, because this
20:23
Speaker 1: is a very sobering reality that you are pulling the
20:28
Speaker 1: curtain back from, I don't think people are fully aware
20:32
Speaker 1: of the machinations behind the scenes of how they think
20:36
Speaker 1: they're building us a utopian society. Have any of them
20:40
Speaker 1: reached out to you? Have any government officials reached out?
20:42
Speaker 1: You know, who has slid into your DMs?
20:48
Speaker 2: There have been a lot of people in my DMs, yeah.
20:51
Speaker 2: I've kind of been ignoring the government officials, and they're
20:54
Speaker 2: mostly male.
21:02
Speaker 1: If you did all this to get laid, brother, you
21:04
Speaker 1: and I are going to have a problem. How seriously...
21:13
Speaker 1: It was a joke.
21:14
Speaker 2: It was a joke.
21:15
Speaker 1: Understood. Yeah. Me too. Is there a group that is
21:22
Speaker 1: taking it, that is giving it the seriousness that you
21:25
Speaker 1: believe it deserves?
21:26
Speaker 2: Definitely, yes.
21:27
Speaker 1: Who is?
21:28
Speaker 2: I think politicians are starting to take it seriously.
21:30
Speaker 1: Right. Is the EU more along the lines of the
21:33
Speaker 1: types of regulations? Do you believe it needs regulations? What
21:36
Speaker 1: are the guardrails that you would put into it that
21:39
Speaker 1: would make you feel more comfortable?
21:40
Speaker 2: I think the number one thing is... There's safety research,
21:43
Speaker 2: and there is making it smarter. So you can do
21:46
Speaker 2: research on how to control it and ensure that those
21:47
Speaker 2: hacks don't happen. And that doesn't involve making the model smarter.
21:51
Speaker 2: There's also research that just makes it, yeah, generically more useful,
21:54
Speaker 2: more economically valuable. And the incentives point to allocating a
21:57
Speaker 2: lot of your resources, your researchers, your time, to the
22:01
Speaker 2: making it smarter. Because that's how you get ahead. That's
22:03
Speaker 2: how you catch up. If you're behind in the race,
22:05
Speaker 2: you want to do that. And that means you don't
22:07
Speaker 2: allocate to safety.
22:07
Speaker 1: Getting ahead, let's be clear. When we talk about that,
22:10
Speaker 1: we mean like, four months ahead because the chips haven't
22:13
Speaker 1: quite gotten to the state there. But China's already way
22:16
Speaker 1: ahead in terms of building out a grid that is
22:18
Speaker 1: resilient and is going to work on more solar power. Like,
22:22
Speaker 1: they've already got a ton of advantages. We may be
22:24
Speaker 1: chasing a phantom in the first place. And it's not
22:27
Speaker 1: like the race to the moon because, again, it's like
22:31
Speaker 1: that's a target and you get to the moon and go, yay.
22:36
Speaker 2: Even more sobering, the race I'm talking about is actually
22:38
Speaker 2: the domestic race still. Like, I still think the domestic...
22:40
Speaker 2: Between all of them. Between the labs. I think the
22:42
Speaker 2: domestic competitive pressures will still push towards under-allocating towards safety research.
22:47
Speaker 1: So the idea that they can collaborate on regulation is
22:49
Speaker 1: a fool's errand in the first place. And by the way,
22:52
Speaker 1: we're just talking about it smarter and hacking and getting out.
22:55
Speaker 1: Let's talk about another real damage of it, which is...
22:59
Speaker 1: the democratization of chaos. These are incredibly powerful tools, and
23:04
Speaker 1: you have no idea who you're handing them over to.
23:07
Speaker 1: And there are certainly misanthropic bad actors who will now
23:10
Speaker 1: have the capabilities that are wildly nihilistic in their own right.
23:15
Speaker 1: They can build new biological weapons. They can do all,
23:18
Speaker 1: you know, it's the most irresponsible race that I've ever
23:27
Speaker 1: seen humankind engage in. It seems almost suicidal.
23:31
Speaker 2: It's insane. I agree. It is suicidal. I think the
23:34
Speaker 2: security measures in the labs as well. Right now, an adversary,
23:38
Speaker 2: a motivated adversary could steal the progress and destroy the lead.
23:42
Speaker 2: So this notion of a lead is already basically a mirage.
23:44
Speaker 2: If China tried hard enough, they could immediately catch up
23:47
Speaker 2: by just hacking and stealing the progress. So that's another
23:50
Speaker 2: way in which it's just not being taken seriously.
23:52
Speaker 1: And what are, do you talk about how would bad
23:55
Speaker 1: actors use this?
23:57
Speaker 2: There's a lot of bad stuff you could do with
23:59
Speaker 2: the models of this year and next year. I think
24:01
Speaker 2: the classic one is mass hacking. Just take down the internet,
24:05
Speaker 2: take down the banking system, take down the grids. Then
24:10
Speaker 2: you can go to biological stuff, which is even nastier.
24:12
Speaker 1: Sure, let's do that.
24:17
Speaker 2: Or you could do bad biological stuff.
24:19
Speaker 1: Yeah. Do you have a bunker?
24:23
Speaker 2: No, sadly not.
24:24
Speaker 1: Do you believe in your heart of hearts? Because there's
24:28
Speaker 1: a thing that people think, which is, oh, they play
24:31
Speaker 1: up the danger of it because it draws attention and
24:37
Speaker 1: money and finance, but it's not actually really dangerous. Yeah.
24:42
Speaker 1: But you believe it to be really dangerous.
24:44
Speaker 2: I can say sincerely that the people I work with
24:47
Speaker 2: genuinely believe in the danger. That's their genuine belief. And
24:51
Speaker 2: they don't stand to profit from this belief.
24:55
Speaker 1: Honestly, I am dumbstruck. This is more sobering than I
25:01
Speaker 1: imagined it to be. And it's shocking to me, out
25:04
Speaker 1: of the percentage of people that work in that industry,
25:06
Speaker 1: how many have turned and stated the dangers? 10 of you? 15? Yeah.
25:14
Speaker 2: In recent times. It's kind of crazy that you showed
25:17
Speaker 2: the clips.
25:17
Speaker 1: Yes.
25:18
Speaker 2: The CEOs have been saying this for a while. I
25:20
Speaker 2: think only recently it's become clear that this is real.
25:23
Speaker 1: Are people not saying it because of the financial incentive?
25:28
Speaker 2: Yes, and also the pressure to leak.
25:30
Speaker 1: Right.
25:30
Speaker 2: Yeah. Stay part of the hood.
25:32
Speaker 1: Do you still have your stock?
25:33
Speaker 2: Yes, and OpenAI. All right.
25:44
Speaker 1: You're the man. Jacob Cox.