Latest / Faster, Please! β The Podcast / π€ Superintelligence and national security: My chat (+transcript) with AI expert Dan Hendrycks
Transcript
- 0:06SPEAKER_00: Welcome to Faster Please, the podcast.
- 0:09SPEAKER_00: I'm your host, Jim Pethikoukis.
- 0:12SPEAKER_00: Several times a month,
- 0:13I'll feature a lively conversation with a fascinating and
- 0:16provocative guest about
- 0:18SPEAKER_00: how to make the world a better place through scientific discovery,
- 0:21SPEAKER_00: technological innovation,
- 0:23SPEAKER_00: and economic growth.
- 0:24SPEAKER_00: You're also going to want to check out my Faster Please newsletter.
- 0:27SPEAKER_00: You're on Substack throughout the week for fresh essays,
- 0:30SPEAKER_00: Q&As,
- 0:31SPEAKER_00: and stories from around the internet and around the world.
- 0:36SPEAKER_00: As we seemingly grow closer to achieving artificial general intelligence,
- 0:41SPEAKER_00: machines that are smarter than humans at basically everything,
- 0:44SPEAKER_00: we might be incurring some serious geopolitical risks.
- 0:48SPEAKER_00: In the paper, Superintelligence Strategy,
- 0:51SPEAKER_00: his joint project with former Google CEO Eric Schmidt and Alexander Wang,
- 0:56SPEAKER_00: Dan Hendricks introduces the idea of mutual assured AI malfunction,
- 1:01a system of deterrence where any state's attempt at total AI
- 1:05dominance is sabotaged
- 1:07SPEAKER_00: by its peers.
- 1:09SPEAKER_00: Today on Faster Please,
- 1:10SPEAKER_00: the podcast,
- 1:11I talk with Hendricks about the potential threats posed by
- 1:15super intelligent AI in
- 1:16the hands of state and rogue adversaries and what a strong
- 1:20deterrent strategy might
- 1:21SPEAKER_00: look like.
- 1:23SPEAKER_00: Hendricks is the executive director of the Center for AI Safety.
- 1:26He's an advisor to Elon Musk's XAI and Scale AI and is a
- 1:32prolific researcher and writer.
- 1:34How would you compare your view of AI today as a powerful
- 1:42technology with economic,
- 1:44SPEAKER_00: national security,
- 1:45SPEAKER_00: and broader societal implications?
- 1:48How would you compare your view of AI today versus November of
- 1:542022 when OpenAI
- 1:55SPEAKER_00: rolled out ChatGPT?
- 2:00I think that the main difference now is that we have the
- 2:06reasoning paradigm.
- 2:08SPEAKER_01: So back in 2022,
- 2:11GPT couldn't think for an extended period of time before
- 2:15answering and try out
- 2:16SPEAKER_01: multiple different ways of dissolving a problem.
- 2:19SPEAKER_01: So the main new capability is its ability to handle more complicated,
- 2:26reasoning and science,
- 2:28technology,
- 2:28engineering,
- 2:29mathematics sorts of tasks.
- 2:31SPEAKER_01: So it's a lot better at coding.
- 2:33It's a lot better at graduate school mathematics and physics
- 2:36and virology.
- 2:38And so an implication of that for national security is that AIs
- 2:45have some virology
- 2:46SPEAKER_01: capabilities that they didn't before,
- 2:48which has β and virology is dual use that can be used for
- 2:51civilian applications and
- 2:52SPEAKER_01: weaponization applications.
- 2:54SPEAKER_01: So that's a new concerning capability that they have.
- 2:58SPEAKER_01: But I think overall, the agents β or excuse me, the AI systems β
- 3:03SPEAKER_01: are still fairly similar in their capabilities profile.
- 3:06SPEAKER_01: They're better in lots of different ways, though, but not substantially.
- 3:08SPEAKER_01: I think the next large shift is when they can be agents,
- 3:14SPEAKER_01: when they can operate more autonomously,
- 3:16SPEAKER_01: when they can book you flights reliably,
- 3:19SPEAKER_01: make PowerPoints,
- 3:20SPEAKER_01: play through long form games for extended periods of time.
- 3:24SPEAKER_01: And that seems like it's potentially on the horizon this year,
- 3:28SPEAKER_01: that it didn't seem like that two years ago.
- 3:32So that's something that a lot of people are keeping an eye on
- 3:35and think could be
- 3:36SPEAKER_01: arriving fairly soon.
- 3:37SPEAKER_01: But overall,
- 3:39SPEAKER_01: I think the capabilities profile is mostly the same,
- 3:41except now it has some dual-use capabilities that they didn't
- 3:47have earlier,
- 3:48SPEAKER_01: in particular,
- 3:48SPEAKER_01: virology capabilities.
- 3:49SPEAKER_00: Yeah.
- 3:52SPEAKER_00: What extent,
- 3:54SPEAKER_00: and we'll get deeper into this,
- 3:55are your national security concerns based on the capabilities
- 4:00of the technology as
- 4:02SPEAKER_00: it is today versus where you think it will be in five years?
- 4:09And I guess this is also a way of be asking about to the extent
- 4:13that you view AGI
- 4:16SPEAKER_00: as a useful framing device.
- 4:18SPEAKER_00: So I guess this is also a question about your timeline.
- 4:22SPEAKER_00: Yeah.
- 4:23SPEAKER_01: So I think that mostly the AI systems aren't that impressive.
- 4:27SPEAKER_01: Currently,
- 4:30SPEAKER_01: people use them to some extent,
- 4:32but I'd more emphasize the trajectory that we're on rather than
- 4:36the current capabilities.
- 4:38SPEAKER_01: They still can't do very interesting cyber offense, for instance.
- 4:44SPEAKER_01: The virology capabilities is very recent.
- 4:46SPEAKER_01: We just, I think maybe a week ago, put out a study with
- 4:50SPEAKER_01: a secure bio from MIT,
- 4:52SPEAKER_01: where we had Harvard,
- 4:53SPEAKER_01: MIT,
- 4:53SPEAKER_01: virology,
- 4:54SPEAKER_01: postdocs doing wet lab skills,
- 4:57SPEAKER_01: trying to work on viruses.
- 4:58SPEAKER_01: So like, here's a picture of my Petri dish.
- 5:01SPEAKER_01: I heated it to 37 degrees.
- 5:03SPEAKER_01: What went wrong?
- 5:04SPEAKER_01: Helped me troubleshoot.
- 5:05SPEAKER_01: Helped me guide me through this step by step.
- 5:08And we were seeing that it was getting around 95th percentile
- 5:13compared to those
- 5:15SPEAKER_01: Harvard-MIT virology postdocs in their area of expertise.
- 5:18SPEAKER_01: This is not a capability that the models had two years ago.
- 5:22SPEAKER_01: So that is a national security concern.
- 5:24But I think most of the national security concerns of where
- 5:27it's strategically relevant,
- 5:30SPEAKER_01: where it can be used for more targeted weapons,
- 5:34SPEAKER_01: where it affects the basis of a nation's power,
- 5:39I think that's something that happens in the next,
- 5:42say,
- 5:43two to five years.
- 5:45SPEAKER_01: And I think that's what we mostly need to be thinking about,
- 5:49not particularly trying to raise the alarm saying that the AI
- 5:52systems right now are
- 5:53extremely scary in all these different ways because they're not
- 5:56even agential.
- 5:57SPEAKER_01: They can't book flights yet.
- 5:59all right so that that two to five year timeline this and again
- 6:02you can uh i guess
- 6:03you can debate with again this is a good way of thinking about
- 6:06it is is that is
- 6:07that is that a trajectory or timeline to something that could
- 6:11be called a human
- 6:12SPEAKER_00: level ai and you can define that any way you want and what are then the
- 6:17capabilities that make ai potentially dangerous and a in a in a
- 6:22strategic player
- 6:24SPEAKER_00: when thinking about national security
- 6:26SPEAKER_01: Yeah,
- 6:27so I think having a monolithic term for AGI or for advanced AI
- 6:34systems is a little difficult,
- 6:37SPEAKER_01: largely because there's been a consistently moving goalpost.
- 6:41SPEAKER_01: So
- 6:42SPEAKER_01: Right now,
- 6:43SPEAKER_01: people say AIs are dumb because they can't do,
- 6:45SPEAKER_01: they can't play video games at the level of,
- 6:49SPEAKER_01: you know,
- 6:50SPEAKER_01: a teenager.
- 6:51SPEAKER_01: They can't code for a day-long project and things like that.
- 6:57SPEAKER_01: So neither can, like...
- 6:59SPEAKER_01: my grandmother.
- 7:02SPEAKER_01: That doesn't mean that she's not human level intelligence.
- 7:05SPEAKER_01: It's just like a lot of people don't have some of these capabilities.
- 7:11SPEAKER_01: So I think when thinking about the future of AI,
- 7:14SPEAKER_01: it's useful to think,
- 7:16SPEAKER_01: especially when thinking about national security,
- 7:17it's useful to think in terms of specific capabilities as
- 7:20opposed to strategically
- 7:22SPEAKER_01: relevant capabilities,
- 7:24SPEAKER_01: as opposed to
- 7:27SPEAKER_01: When is it truly intelligent or something like that?
- 7:31SPEAKER_01: So this is because the capabilities of AI systems are very jagged.
- 7:37SPEAKER_01: So they're good at some things and terrible at others.
- 7:39SPEAKER_01: They can't fold clothes that reliably.
- 7:42SPEAKER_01: Most of the AIs can't.
- 7:44SPEAKER_01: And they're okay at driving in some cities but not others.
- 7:48SPEAKER_01: But they can solve really difficult mathematics problems.
- 7:54They can write really long essays and provide pretty good legal
- 7:59analysis very rapidly.
- 8:01And they can also forecast geopolitical events I think better
- 8:04than most forecasters.
- 8:06SPEAKER_01: So it's a really weird capabilities profile.
- 8:13And so when I'm thinking about national security from a
- 8:15malicious use standpoint,
- 8:17SPEAKER_01: I'm thinking about bio weapon capabilities.
- 8:20SPEAKER_01: I'm thinking about cyber offensive capabilities,
- 8:22SPEAKER_01: which they don't yet have,
- 8:23SPEAKER_01: but that's an important one to track.
- 8:25SPEAKER_01: And I'm thinking about the...
- 8:28SPEAKER_01: Outside of malicious use,
- 8:30I'm thinking about what's their ability to do AI research and
- 8:33how much of that can
- 8:33SPEAKER_01: they automate?
- 8:34SPEAKER_01: Because if they can automate AI research,
- 8:37then you could just run 100,000 of these artificial AGI
- 8:41researchers to build the
- 8:42SPEAKER_01: next generations of AGI.
- 8:44SPEAKER_01: And that could get very explosive extremely quickly.
- 8:47You're moving from human speed research to machine speed
- 8:49research to typing 100
- 8:50SPEAKER_01: times faster than people.
- 8:52SPEAKER_01: They're running tons of experiments simultaneously.
- 8:54SPEAKER_01: That could be quite explosive.
- 8:56And that's something that the founders of AI pointed at as a
- 9:00really relevant capability,
- 9:01SPEAKER_01: like Alan Turing and others,
- 9:03where that's where you could have a potential loss of control
- 9:07type of event is with
- 9:09this sort of runaway process of AIs building future generations
- 9:12of AIs quite rapidly.
- 9:14SPEAKER_01: So that's another capability.
- 9:15SPEAKER_01: What fraction of AI research can they automate?
- 9:19SPEAKER_01: And for weaponization,
- 9:22SPEAKER_01: I think if it gets extremely smart,
- 9:25SPEAKER_01: able to do research in lots of other sorts of fields,
- 9:28then that would raise concerns of its ability to be used to
- 9:33disrupt the balance of power.
- 9:35SPEAKER_01: For instance, if it can do research well,
- 9:38Perhaps it could come up with a breakthrough that makes oceans
- 9:42more transparent so
- 9:42that we can find where nuclear submarines are or find the
- 9:45mobile launchers
- 9:46extremely reliably or a breakthrough in driving down the cost
- 9:52by some orders of
- 9:52SPEAKER_01: magnitude of anti-ballistic missile systems,
- 9:54SPEAKER_01: which would disrupt having a secure second strike.
- 9:57SPEAKER_01: And these would be very geopolitically salient.
- 9:59SPEAKER_01: To do those things,
- 10:00SPEAKER_01: though,
- 10:00that seems like a bundle of capabilities as opposed to a
- 10:03specific thing like cyber
- 10:04SPEAKER_01: offensive capabilities.
- 10:06But those are the things that I'm thinking about that can
- 10:09really disrupt the
- 10:11SPEAKER_01: geopolitical landscape.
- 10:12SPEAKER_00: If we put them in a bucket called,
- 10:14SPEAKER_00: as you use your phrase,
- 10:16SPEAKER_00: strategically relevant capabilities,
- 10:18SPEAKER_00: are we on a trajectory of just sort of...
- 10:23SPEAKER_00: a data and computing power driven trajectory to those capabilities?
- 10:29SPEAKER_00: Or do there need to be one or two key innovations before those relevant
- 10:36SPEAKER_00: capabilities are possible?
- 10:39SPEAKER_01: It doesn't seem like it currently that we need some new big insights,
- 10:47SPEAKER_01: in large part because the rate of improvement is pretty good.
- 10:53SPEAKER_01: So if we look at their coding capabilities,
- 10:56SPEAKER_01: if we just extract,
- 10:57SPEAKER_01: there's a benchmark called SWE Bench Verified,
- 10:59SPEAKER_01: which
- 11:01SPEAKER_01: So we use like software engineering.
- 11:03SPEAKER_01: So given a set of coding tasks,
- 11:08SPEAKER_01: and this benchmark was made some years ago,
- 11:10SPEAKER_01: the models are poised to get something like 90% on this this summer.
- 11:16SPEAKER_01: Right now they're in the 60% range.
- 11:19SPEAKER_01: So if we just extrapolate the trend line out some more months,
- 11:23Then they'll be doing 9 out of 10 of those software engineering
- 11:29tasks that were set
- 11:30SPEAKER_01: some years ago.
- 11:31SPEAKER_01: That doesn't mean that that's the entirety of software engineering.
- 11:34SPEAKER_01: Still need coders.
- 11:35SPEAKER_01: It's not 100%, obviously.
- 11:37But that suggests that the Cape Village is still improving
- 11:45fairly rapidly in some
- 11:47SPEAKER_01: of these domains.
- 11:50Likewise with like their ability to play things that take games
- 11:54that take like 20 plus hours.
- 11:57SPEAKER_01: Um,
- 11:58SPEAKER_01: A few months ago, they couldn't.
- 11:59SPEAKER_01: Like Pokemon,
- 12:01SPEAKER_01: for instance,
- 12:01SPEAKER_01: is something that kids play,
- 12:03SPEAKER_01: and that takes like 20 hours or so to beat.
- 12:06SPEAKER_01: Right now, the models from a few months ago couldn't beat the game.
- 12:11Now the current models can beat the game,
- 12:13but it takes them a few hundred hours.
- 12:14It would not surprise me if in a few months they'll get it down
- 12:17to around human
- 12:20SPEAKER_01: level on the order of tens of hours.
- 12:23SPEAKER_01: And then from there,
- 12:26they'll be able to play harder and harder sorts of games that
- 12:28take longer periods
- 12:29SPEAKER_01: of time.
- 12:30And I think that this would be indicative of higher agential capabilities.
- 12:37I think that there's a lot of steam in the current way that
- 12:42things are being done.
- 12:43And I think that they've been trapped at the floor in their
- 12:48agent capabilities for a while.
- 12:50SPEAKER_01: But I think we're starting to see the shift.
- 12:54And I think that most people at the major AI companies would
- 12:58also think that agents
- 13:00SPEAKER_01: are on the horizon.
- 13:01SPEAKER_01: And I don't think they were thinking that.
- 13:03SPEAKER_01: myself included, a year ago.
- 13:05SPEAKER_01: We were not seeing the signs that we're seeing now.
- 13:07SPEAKER_00: What we're talking about is AIs having,
- 13:10SPEAKER_00: use your phrase,
- 13:11SPEAKER_00: which I like,
- 13:11strategically relevant capabilities on a timeline that is soon
- 13:16enough that we
- 13:17should be having the kinds of conversations and the kind of
- 13:22thinking that you put
- 13:23SPEAKER_00: forward in superintelligence.
- 13:26SPEAKER_00: We should be thinking about that right now very seriously.
- 13:31SPEAKER_00: Yeah,
- 13:32SPEAKER_01: I mean,
- 13:33it's very difficult to wrap one's head around because unlike
- 13:37other domains,
- 13:39SPEAKER_01: AI,
- 13:40SPEAKER_01: I think,
- 13:40SPEAKER_01: is much more general and broad in its impacts.
- 13:45SPEAKER_01: So if one's thinking about nuclear strategy,
- 13:48you obviously need to think about bombs going off and
- 13:50survivability and second strike.
- 13:55SPEAKER_01: The failure modes are...
- 13:58One state strikes the other and then there's also in the
- 14:02civilian applications,
- 14:03SPEAKER_01: you know,
- 14:04SPEAKER_01: fissile material leaking or there being a nuclear power plant meltdown.
- 14:08SPEAKER_01: Like that's kind of the scenario space.
- 14:11What states can do and then there's also some of these civilian
- 14:14application issues.
- 14:15SPEAKER_01: Meanwhile, with AI.
- 14:17SPEAKER_01: We've got much more than power plants melting down or bombs going off.
- 14:20SPEAKER_01: We've got to think about how it transforms the economy,
- 14:24SPEAKER_01: how it transforms people's private life,
- 14:26SPEAKER_01: the sort of issues with them being sentient.
- 14:30We've got to think about it potentially disrupting mutual
- 14:33assured destruction.
- 14:35SPEAKER_01: We've got to think about the AIs themselves being threats.
- 14:38We've got to think about regulations for autonomous AI agents
- 14:44and who's accountable.
- 14:46SPEAKER_01: We've got to think about this open weight, closed weight issue.
- 14:49We've got,
- 14:50I think,
- 14:52a larger host of issues that touch in all the important spheres
- 14:56of society.
- 14:58SPEAKER_01: So it's not a very delimited problem.
- 15:01SPEAKER_01: And I think
- 15:02SPEAKER_01: it's a very large pill to swallow,
- 15:04SPEAKER_01: this possibility that it will be not just strategically relevant,
- 15:08SPEAKER_01: but strategically decisive this decade.
- 15:11SPEAKER_01: So I think, consequently,
- 15:17SPEAKER_01: thinking a little bit beforehand about it is useful.
- 15:20SPEAKER_01: Otherwise,
- 15:21SPEAKER_01: if we just ignore it,
- 15:21I think reality will slap us across the face and AI will hit us
- 15:25like a truck.
- 15:26SPEAKER_01: And then we're going,
- 15:26SPEAKER_01: wow,
- 15:26SPEAKER_01: I wish we did something,
- 15:28SPEAKER_01: had some more break glass measures at a time right now.
- 15:32But the cupboard is bare in terms of strategic options because
- 15:37we didn't do some
- 15:38prudent things a while ago or we didn't even bother thinking
- 15:40about what those are.
- 15:41SPEAKER_01: I keep thinking of the situation room in two years.
- 15:45And they get news that China's doing some new big AI project,
- 15:48and it's fairly secretive.
- 15:50SPEAKER_01: And then in the situation, they're thinking like, okay, what do we know?
- 15:55SPEAKER_01: And the answer is nothing.
- 15:57SPEAKER_01: We don't relate anybody on this.
- 15:59SPEAKER_01: We're not collecting any information about this.
- 16:02SPEAKER_01: We didn't have many concerted programs in the ICE really tracking this.
- 16:06SPEAKER_01: So we're flying blind.
- 16:09SPEAKER_01: I really don't want to be in that situation.
- 16:11SPEAKER_00: Now,
- 16:12SPEAKER_00: as I'm sure you know,
- 16:13SPEAKER_00: throughout the course of the Cold War,
- 16:15there was considerable amount of time and money spent on
- 16:21thinking about these kinds
- 16:24SPEAKER_00: of problems.
- 16:25SPEAKER_00: I went to college just before the end of the Cold War,
- 16:28SPEAKER_00: and I took an undergraduate class on sort of nuclear war theory.
- 16:35SPEAKER_00: So there was a lot of thinking.
- 16:38SPEAKER_00: To what extent
- 16:40Does that sort of volume of sort of research and analysis over
- 16:46the course of a half century,
- 16:48SPEAKER_00: to what extent is that helpful for what you're trying to accomplish here?
- 16:53SPEAKER_01: I think it's very fortunate that because of the Cold War,
- 16:56a lot of people started getting more of a sense of game theory
- 17:00and when it's
- 17:01SPEAKER_01: rational to conflict versus...
- 17:04negotiate and the just you know offensive offense can provide a
- 17:09good defense some
- 17:10of these counterintuitive things I mean I think mutual sure
- 17:12destruction is an ugly
- 17:13reality that took decision makers a long time to internalize
- 17:19but that's just what
- 17:19the game theory sort of showed would make the most sense so I
- 17:26think hopefully we'll
- 17:28SPEAKER_01: do a lot better with AI because
- 17:31SPEAKER_01: strategic thinking can be a lot more precise.
- 17:35SPEAKER_01: And some of these things that are initially counterintuitive,
- 17:39SPEAKER_01: if you reason through them,
- 17:40SPEAKER_01: you go,
- 17:40SPEAKER_01: actually,
- 17:40SPEAKER_01: no,
- 17:41SPEAKER_01: this makes a lot of sense.
- 17:42We're trying to shape each other's intentions in this kind of
- 17:44complicated way.
- 17:45So I think that makes us much better poised to address these
- 17:52geopolitical issues than last time.
- 17:57I think of the Soviets,
- 17:58for instance,
- 17:58when talking about anti-ballistic missile systems.
- 18:01At one point,
- 18:03I forget who said that offense is immoral,
- 18:06defense is moral.
- 18:08So pointing these nuclear weapons at each other,
- 18:10this is the immoral thing.
- 18:11SPEAKER_01: We need missile defense systems.
- 18:14SPEAKER_01: That's the moral option.
- 18:15SPEAKER_01: It's just like, no, this is going to...
- 18:17SPEAKER_01: create some,
- 18:17this is just going to eat up all of our budget because we're
- 18:21going to keep building
- 18:22SPEAKER_01: these defense systems.
- 18:23SPEAKER_01: It's not going to make us safer.
- 18:24SPEAKER_01: We're just going to be spending more and more.
- 18:27SPEAKER_01: That was not intuitive.
- 18:30Offense does feel viscerally more mean,
- 18:34hostile,
- 18:35but that's what you want.
- 18:37SPEAKER_01: So that's what you want to preserve for strategic stability.
- 18:40SPEAKER_01: So
- 18:41SPEAKER_01: I think that a lot of the thinking is helpful with that.
- 18:46And I think the education for appreciating the strategic dynamics,
- 18:50it's more in the water.
- 18:51SPEAKER_01: It's more diffused across the decision makers now.
- 18:54SPEAKER_01: I think that that's great.
- 18:56SPEAKER_00: I was recently reviewing a scenario slash,
- 19:01SPEAKER_00: I guess,
- 19:01SPEAKER_00: world building kind of exercise among technologists and
- 19:07SPEAKER_00: economists, forecasting people.
- 19:10SPEAKER_00: And they were looking at various scenarios,
- 19:12SPEAKER_00: assuming that we're able to,
- 19:15SPEAKER_00: on a rather short timeline,
- 19:18SPEAKER_00: develop what they would,
- 19:19SPEAKER_00: I think they termed AGI.
- 19:21SPEAKER_00: And one of the scenarios was that the U.S.
- 19:25SPEAKER_00: gets there first.
- 19:27And it proves to be such a powerful I mean,
- 19:30it's probably not by very long,
- 19:31but U.S.
- 19:32SPEAKER_00: got there first.
- 19:33SPEAKER_00: I don't know how far China was behind,
- 19:35but that gave us the capability to sort of dictate terms to
- 19:40China about what their
- 19:41SPEAKER_00: foreign policy would be.
- 19:43SPEAKER_00: You're not you know, you're going to leave Taiwan alone.
- 19:45SPEAKER_00: You're going to.
- 19:46SPEAKER_00: So it gave us an amazing strategic advantage.
- 19:49SPEAKER_00: And I'm sure there are a lot of.
- 19:52SPEAKER_00: American policymakers who would read that scenario and say,
- 19:55SPEAKER_00: that's the dream,
- 19:57SPEAKER_00: that we are able to accelerate progress,
- 20:00SPEAKER_00: that we are able to get there first,
- 20:02SPEAKER_00: we can dictate foreign policy terms to China,
- 20:05SPEAKER_00: game over,
- 20:06SPEAKER_00: we win.
- 20:08SPEAKER_00: If I've read superintelligence correctly,
- 20:12That scenario is far more complicated than it would play out in
- 20:17a far more
- 20:17SPEAKER_00: complicated way than what I've just described.
- 20:20SPEAKER_01: Yeah, I think so.
- 20:21I think any bid for being a not just unipolar force,
- 20:27but having a sort of.
- 20:30near strategic monopoly on power and able to cause all other
- 20:34superpowers to
- 20:34SPEAKER_01: capitulate in arbitrary ways is concerns the other superpowers.
- 20:40SPEAKER_01: So I think and there is also a risk that China builds it first.
- 20:46So I think what we want to do in the US is build up the
- 20:50capabilities to surgically
- 20:53SPEAKER_01: prevent them if they are near
- 20:58or eminently going to gain a decisive advantage that would
- 21:03become durable and
- 21:05SPEAKER_01: sustained over us,
- 21:08SPEAKER_01: we want the ability to prevent that.
- 21:10SPEAKER_01: I mean, there's a variety of ways one can do things.
- 21:13There's the classic grayer ways like arson and cutting wires
- 21:17and data centers and
- 21:18SPEAKER_01: things like that.
- 21:19SPEAKER_01: Or for power plants nearby.
- 21:21SPEAKER_01: There's cyber offense.
- 21:22SPEAKER_01: There's other sorts of kinetic sabotage.
- 21:25SPEAKER_01: But we want it nice and surgical and having a good,
- 21:30credible threat so that we can deter that from happening and
- 21:34shaping their intentions.
- 21:36SPEAKER_01: I think it will be difficult to limit their impact.
- 21:38SPEAKER_01: capabilities, their ability to build these powerful systems.
- 21:43But I think being able to shape their intentions is something
- 21:46that is more tractable.
- 21:48SPEAKER_01: They will be building powerful AI systems.
- 21:50But if they are making an attempt at leapfrogging us and in a
- 21:58way that we never
- 21:59SPEAKER_01: catch up and lose our standing,
- 22:03SPEAKER_01: And they get AIs that could also potentially disrupt MAD, for instance.
- 22:09SPEAKER_01: We want to be able to prevent that.
- 22:12SPEAKER_01: So that is,
- 22:13SPEAKER_01: I think,
- 22:13an important strategic priority is developing a credible
- 22:16deterrent and saying
- 22:17SPEAKER_01: there's some AI scenarios that are totally unacceptable to us.
- 22:21SPEAKER_01: And we want to block them off through credible threats.
- 22:24SPEAKER_01: They will do the same to us as well.
- 22:26SPEAKER_01: And they can do it more easily to us.
- 22:28SPEAKER_01: They know what's going on at all of our AI companies.
- 22:31And this will not change because we have a double-digit
- 22:34percentage of the employees
- 22:35SPEAKER_01: are Chinese nationals,
- 22:36SPEAKER_01: easily extortable.
- 22:37SPEAKER_01: They have family back home.
- 22:38SPEAKER_01: And the companies do not have good information security.
- 22:43That will probably not change because that will slow them down
- 22:45if they really try
- 22:46and lock them up and move everybody to North Dakota or wherever
- 22:50to work in the
- 22:50SPEAKER_01: middle of nowhere and have everything air-gapped.
- 22:53SPEAKER_01: So,
- 22:55SPEAKER_01: We are an open book to them,
- 22:57and I think they can make very credible threats for sabotage
- 23:02and preventing that
- 23:03SPEAKER_01: type of outcome.
- 23:04If we are making a bid for having them dictating their foreign
- 23:11policy and all of this,
- 23:13SPEAKER_01: if we're making a bid for a sort of strategic monopoly on power,
- 23:17SPEAKER_01: they will not sit idly by,
- 23:19SPEAKER_01: they will not take kindly to that when they recognize the stakes.
- 23:22SPEAKER_01: If the U.S.
- 23:23SPEAKER_01: were to do a
- 23:24SPEAKER_01: a $500 billion program to achieve this faster than them.
- 23:29SPEAKER_01: That would not go unnoticed.
- 23:31SPEAKER_01: There's not a way of hiding that.
- 23:32SPEAKER_01: But we are trying to achieve it faster than them.
- 23:34SPEAKER_01: Well, we are trying to.
- 23:36SPEAKER_01: So I would distinguish between...
- 23:38trying to develop just generally more capable AI technologies
- 23:43than some of these
- 23:43strategically relevant capabilities or some of these
- 23:45strategically relevant programs,
- 23:48like if we get AI systems that are generally useful for health
- 23:55care and for
- 23:58SPEAKER_01: whatever your pet cause area,
- 24:00SPEAKER_01: we can have that.
- 24:02That is different from applying the AI systems to rapidly build
- 24:06the next generation
- 24:07SPEAKER_01: of AIs and the next generation of that.
- 24:09Like,
- 24:09just imagine if you have right now open-ends and we've got a
- 24:11few hundred AI researchers.
- 24:13Imagine if you've got ones that are at that level that are
- 24:17artificial AGI type of researcher,
- 24:20SPEAKER_01: are artificial researchers.
- 24:22SPEAKER_01: You run...
- 24:2410,000,
- 24:24100,000 of them,
- 24:25they're operating around the clock at 100x speed.
- 24:28SPEAKER_01: I think,
- 24:29SPEAKER_01: you know,
- 24:30expecting a decade's worth of development compressed or
- 24:32telescoped into a year,
- 24:34SPEAKER_01: that seems very plausible.
- 24:37SPEAKER_01: Not certain, but...
- 24:39SPEAKER_01: certainly double digit percent chance.
- 24:42And that is China or Russia,
- 24:46for instance,
- 24:47would perceive that as this is really risky.
- 24:49They could get a huge leap from this because the rate of
- 24:53development will be so
- 24:54SPEAKER_01: high that we could never catch up.
- 24:55SPEAKER_01: And they could use their new gains to collaborate.
- 24:59SPEAKER_01: Or if they don't control it, then we're also going
- 25:04SPEAKER_01: dead or lose our power.
- 25:07SPEAKER_01: So if the US controls it,
- 25:10China would reason that our survival is threatened and how we
- 25:15do things is threatened.
- 25:16SPEAKER_01: And if they lose control of it, our survival is also threatened.
- 25:19SPEAKER_01: So either way, provided that
- 25:23this sort of automated AI research and development loop
- 25:26produces some extremely
- 25:27SPEAKER_01: powerful AI systems,
- 25:29SPEAKER_01: China would be fearing for their survival.
- 25:31SPEAKER_01: And it's not just China,
- 25:32SPEAKER_01: India,
- 25:33SPEAKER_01: the Global South,
- 25:34SPEAKER_01: all the other countries,
- 25:35SPEAKER_01: if they're more attuned to this situation,
- 25:37SPEAKER_01: would be very concerned.
- 25:39SPEAKER_01: Russia as well.
- 25:40SPEAKER_01: Russia doesn't have the hope of outcompeting.
- 25:43SPEAKER_01: They don't have $100 billion data centers.
- 25:48SPEAKER_01: They are busy with Ukraine.
- 25:50SPEAKER_01: And,
- 25:50SPEAKER_01: you know,
- 25:51SPEAKER_01: when they're finished with that,
- 25:53SPEAKER_01: they may reassess it,
- 25:55SPEAKER_01: though,
- 25:55SPEAKER_01: but they're too many years behind.
- 25:56So I think the best they can do is actually try and shape other
- 26:02states' intents
- 26:03SPEAKER_01: rather than try to make a bid for outcompeting them.
- 26:08SPEAKER_00: Right.
- 26:08SPEAKER_00: So if we're thinking about,
- 26:10SPEAKER_00: you know,
- 26:10SPEAKER_00: deterrence and what you call mutually assured AI malfunction,
- 26:14SPEAKER_00: so that...
- 26:16SPEAKER_00: So there's a capability aspect that we want to make sure we have,
- 26:20SPEAKER_00: you know,
- 26:21that we would have the capability to check that kind of dash
- 26:27for dominance.
- 26:28But there's also a communication aspect where both sides sort
- 26:32of have to understand
- 26:34SPEAKER_00: and trust what the other side is trying to do,
- 26:37SPEAKER_00: which was a key part of sort of,
- 26:39SPEAKER_00: you know,
- 26:40SPEAKER_00: classic Cold War deterrence.
- 26:42SPEAKER_00: Is that happening?
- 26:44SPEAKER_01: Well, so I think so information problems.
- 26:47SPEAKER_01: Yeah.
- 26:47SPEAKER_01: If there's if there's worse information, then that can lead to conflict.
- 26:51I think China doesn't really need to worry about their access
- 26:54to information of what's going on.
- 26:57SPEAKER_01: I think the U.S.
- 26:58SPEAKER_01: will need to develop more of its capabilities to
- 27:01SPEAKER_01: have more reliable signals abroad.
- 27:06But I think there's different ways of getting information and
- 27:10producing misunderstandings,
- 27:12SPEAKER_01: like the confidence building measures,
- 27:13SPEAKER_01: all these sorts of things.
- 27:14SPEAKER_01: But I think that the unilateral one is just espionage.
- 27:16And then the multilateral one is verification mechanisms and
- 27:19building some of that institutional
- 27:22SPEAKER_01: or international infrastructure.
- 27:25And I think the first step in all of this is the states needed
- 27:31to at least take
- 27:32SPEAKER_01: matters into their own hands by building up these unilateral options,
- 27:36the unilateral option to prevent adversaries from doing a dash
- 27:40for domination and
- 27:45SPEAKER_01: also know what's going on with each other's projects.
- 27:49SPEAKER_01: So I think that's what the U.S.
- 27:51SPEAKER_01: should focus on right now.
- 27:53SPEAKER_01: Later on,
- 27:54SPEAKER_01: as the salience of AI increases,
- 27:55I think then international discussions to increase more
- 28:01strategic stability around
- 28:03SPEAKER_01: this would be more plausible to emerge.
- 28:05But if they're not trying to take basic things to defend
- 28:09themselves and protect
- 28:11SPEAKER_01: their own security,
- 28:11then I don't think this is international stuff that makes that
- 28:15much sense.
- 28:16SPEAKER_01: It's kind of out of order.
- 28:16SPEAKER_01: Yeah.
- 28:17SPEAKER_00: I mean,
- 28:18SPEAKER_00: I have,
- 28:18I have in my notes here that you think there's an 80% chance
- 28:21that a AI arms race
- 28:23SPEAKER_00: would result in a,
- 28:25SPEAKER_00: that a catastrophe that would,
- 28:26SPEAKER_00: you know,
- 28:27SPEAKER_00: kill most of humanity.
- 28:29SPEAKER_00: Do I have that right?
- 28:30SPEAKER_01: I think, I think it's not necessarily just the race.
- 28:34I think,
- 28:35or let's,
- 28:36let's sort of think of people's probabilities for this.
- 28:39SPEAKER_01: So there's a, there's a wide spectrum of probabilities.
- 28:40SPEAKER_01: So yeah,
- 28:42SPEAKER_01: Elon, who I work with at XAI, a company I advise, XAI is his company.
- 28:49SPEAKER_01: Elon thinks it's generally on the order of 20% to 30%.
- 28:54And Dario Amidai,
- 28:55the CEO of Anthropic,
- 28:57I think,
- 28:58thinks it's around 20% as well.
- 29:01SPEAKER_01: Sam Altman around 10%.
- 29:03I think it's more likely than not that this doesn't go that
- 29:09well for people.
- 29:11SPEAKER_01: But there's a lot of tractability and a lot of volatility here.
- 29:14SPEAKER_01: So...
- 29:16If our institutions wake up to this more and do some of the
- 29:21basic stuff of knowing
- 29:23SPEAKER_01: what's going on and sharpen your ability to have credible threats,
- 29:27SPEAKER_01: credible targeted threats to prevent another state dominating the other,
- 29:33SPEAKER_01: I think that will make this go quite a bit better.
- 29:37SPEAKER_01: So it's sort of like a nuclear β I think if we went back in time in the β
- 29:44SPEAKER_01: 1940s and were saying,
- 29:45do we think that this whole nuclear thing is going to turn out
- 29:48well in 50 years?
- 29:50I think it would have actually been,
- 29:51I think we actually got a little lucky.
- 29:53I mean,
- 29:53the Cuban Missile Crisis itself was...
- 29:58There were a lot of bad moments in the 60s.
- 29:59SPEAKER_00: There were quite a few...
- 30:01SPEAKER_01: Yeah.
- 30:01So I think it's more likely than not,
- 30:03but there's substantial attractability.
- 30:04And it's important not to be sort of fatalistic about it or
- 30:09just deny it's an issue itself.
- 30:11SPEAKER_01: I think it's sort of like, do we think AI will go well?
- 30:13SPEAKER_01: I don't know.
- 30:14SPEAKER_01: It depends on what our policy is.
- 30:16SPEAKER_01: Right now, we're in the very early days and I'm still not noticing...
- 30:22much of the many of our institutions that are rising to the
- 30:25occasion that I think is warranted.
- 30:27SPEAKER_01: But this could easily change in a few months with some larger event.
- 30:31You know,
- 30:32not to be science fictional or anything,
- 30:34but you talk about a catastrophe.
- 30:35SPEAKER_00: Are you talking about,
- 30:37SPEAKER_00: you know,
- 30:37SPEAKER_00: you know,
- 30:38SPEAKER_00: AI creates some sort of biological weapon,
- 30:41SPEAKER_00: you know,
- 30:42SPEAKER_00: back and forth cyber attacks,
- 30:45SPEAKER_00: you know,
- 30:46destroy all the electrical infrastructure for China and the
- 30:49United States.
- 30:51SPEAKER_00: So all of a sudden, you know, we're back into the 1800s.
- 30:53SPEAKER_00: Are you talking about,
- 30:55SPEAKER_00: you know,
- 30:55SPEAKER_00: some sort of,
- 30:56SPEAKER_00: you know,
- 30:57SPEAKER_00: more Terminator-like scenario,
- 30:58SPEAKER_00: rogue AI?
- 30:59SPEAKER_00: Like when you think about the kind of catastrophe that could be that,
- 31:03SPEAKER_00: you know,
- 31:04SPEAKER_00: that dangerous humanity,
- 31:06SPEAKER_00: what do you think about?
- 31:07SPEAKER_01: We have three risk sources.
- 31:10I think one are states,
- 31:12the other are rogue actors like terrorists and pariah states.
- 31:15SPEAKER_01: And then there's the AIs themselves.
- 31:17SPEAKER_01: The AIs themselves are not relevant right now,
- 31:20but I think could be quite capable of causing damage on their
- 31:25own in even a year or two.
- 31:28SPEAKER_01: So I think that's the space of threat actors.
- 31:31SPEAKER_01: And so, yes, AIs could...
- 31:39In the future,
- 31:40I don't see anything that makes them logically not controllable.
- 31:44SPEAKER_01: They're mostly controllable right now.
- 31:47SPEAKER_01: Maybe it's one out of a hundred,
- 31:48one out of a thousand of the times you run these AI systems and
- 31:51deploy them in some
- 31:52SPEAKER_01: sort of environments.
- 31:53SPEAKER_01: They do try like breaking free.
- 31:55That's a bit of a problem later on when they actually gain the
- 31:57capability to break
- 31:58SPEAKER_01: free and when they are able to operate autonomously.
- 32:02SPEAKER_01: So there's been lots of studies on this.
- 32:03And you can see this in OpenAI's reports whenever they release
- 32:06new models.
- 32:06SPEAKER_01: It's like, oh, there's only a 0.1% chance of it trying to break free.
- 32:10SPEAKER_01: But if you run a million of these AI agents,
- 32:12SPEAKER_01: that's a lot of them that are going to be trying to break free.
- 32:14SPEAKER_01: They're just not very capable currently.
- 32:16SPEAKER_01: So I think that they are actually, that the AIs themselves are risky.
- 32:20And if you're having humanity going up against AIs that aren't
- 32:22controlled by
- 32:23SPEAKER_01: anybody or AIs that broke free,
- 32:25SPEAKER_01: that could get quite dangerous if you also have,
- 32:29SPEAKER_01: as we're seeing now,
- 32:30China and others building more of these humanoid robots in the
- 32:33next few years.
- 32:34This could make them be concerning in that they could just by
- 32:38themselves create
- 32:39SPEAKER_01: some sort of bioweapon.
- 32:40SPEAKER_01: You don't need even human hands to do it.
- 32:42SPEAKER_01: You can just
- 32:43SPEAKER_01: Instruct a robot to do it and disperse it.
- 32:46SPEAKER_01: And I think that's a pretty easy way to take out biological opposition,
- 32:51SPEAKER_01: so to speak,
- 32:52SPEAKER_01: in kind of an eccentric way.
- 32:54SPEAKER_01: But so I think that that's a concern.
- 32:56SPEAKER_01: Rogue actors themselves doing this,
- 32:59SPEAKER_01: you know,
- 33:00SPEAKER_01: them reasoning that,
- 33:00SPEAKER_01: oh,
- 33:01SPEAKER_01: this,
- 33:01SPEAKER_01: you know,
- 33:01SPEAKER_01: bioweapon gives us a secure second strike.
- 33:03SPEAKER_01: Yeah.
- 33:04SPEAKER_01: Things like that would be a concern from from rogue actors.
- 33:09SPEAKER_01: Then,
- 33:09SPEAKER_01: of course,
- 33:09states using this to make make an attempt to crush the other
- 33:14state or develop a
- 33:15SPEAKER_01: technology that lets them disables and adversaries secure second strike.
- 33:23SPEAKER_01: I think these are these are real problems.
- 33:25SPEAKER_00: Let me finish with this.
- 33:26SPEAKER_00: I.
- 33:29SPEAKER_00: I want continuing AI progress such that we can cure,
- 33:36SPEAKER_00: you know,
- 33:37SPEAKER_00: all the major chronic diseases that we can make,
- 33:40SPEAKER_00: that we can get,
- 33:41SPEAKER_00: you know,
- 33:42SPEAKER_00: commercial nuclear fusion,
- 33:45SPEAKER_00: that we can get,
- 33:46SPEAKER_00: you know,
- 33:46SPEAKER_00: faster,
- 33:47SPEAKER_00: you know,
- 33:47SPEAKER_00: rocket,
- 33:48SPEAKER_00: all the kinds of like optimistic stuff,
- 33:51SPEAKER_00: you know,
- 33:53SPEAKER_00: accelerate economic growth to a pace that we've never seen.
- 33:56SPEAKER_00: But I want all of that.
- 33:59Can I get all of that and also avoid the kinds of snares you're
- 34:04worried about
- 34:05SPEAKER_00: without turning sort of the optimistic AI project into...
- 34:11something that arrives at the end of the century rather than
- 34:14arrives mid-century.
- 34:16SPEAKER_00: I'm just worried about slowing down all that progress.
- 34:18SPEAKER_01: No, I think we can.
- 34:21SPEAKER_01: So in the superintelligence strategy, we have three parts to that.
- 34:25SPEAKER_01: We have the deterrence part,
- 34:26SPEAKER_01: which in speaking about here,
- 34:27and we have the making sure that the capabilities aren't
- 34:30falling in the hands of
- 34:31SPEAKER_01: rogue actors.
- 34:32And I think this isn't that difficult,
- 34:34like good export controls and add some...
- 34:36SPEAKER_01: Just basic safeguards of like,
- 34:38we need to know who you are if we're going to be helping you
- 34:41manipulate viruses,
- 34:42SPEAKER_01: things like that.
- 34:44SPEAKER_01: That's easy to handle.
- 34:45SPEAKER_01: And then on the competition aspect,
- 34:47SPEAKER_01: there are many ways the US can make itself more competitive,
- 34:52SPEAKER_01: like having more guaranteed supply chains for AI chips.
- 34:56So more manufacturing here or in allied states instead of all
- 35:01of it being in Taiwan.
- 35:02SPEAKER_01: Currently, all the cutting-edge AI chips are made in Taiwan.
- 35:05SPEAKER_01: So if there's a Taiwan invasion, the US loses in this AI race.
- 35:11SPEAKER_01: They lose.
- 35:12SPEAKER_01: And this is double-digit probability.
- 35:13SPEAKER_01: This is very foreseeable.
- 35:17SPEAKER_01: trying to robustify our manufacturing capabilities, quite essential.
- 35:21SPEAKER_01: Likewise for making robotics and drones.
- 35:25SPEAKER_01: So I think there's still many axes to compete in.
- 35:29I don't think it makes sense to try and compete in building a
- 35:33sort of super
- 35:33SPEAKER_01: intelligence versus one of these sort of
- 35:35SPEAKER_01: potentially mutual assured destruction, disrupting AIs.
- 35:39SPEAKER_01: I don't think you want to be building those,
- 35:41SPEAKER_01: but I think you can get all,
- 35:42SPEAKER_01: you can have your AIs for healthcare,
- 35:44you can have your AIs doing all the complicated math you want
- 35:49and whatever,
- 35:51SPEAKER_01: and all this coding and driving your vehicles and folding your laundry.
- 35:56SPEAKER_01: You can have all of that.
- 35:59SPEAKER_01: So I think it's definitely feasible.
- 36:04SPEAKER_01: What we did in the Cold War with the prospect of nuclear weapons,
- 36:07we obviously got through it and we had deterrence through
- 36:10mutual issue destruction.
- 36:11We had nonproliferation of fissile materials to lesser states
- 36:15and rogue actors.
- 36:16SPEAKER_01: And we
- 36:17SPEAKER_01: had containment of the Soviet Union.
- 36:19SPEAKER_01: And I think the superintelligence strategy is somewhat similar.
- 36:22SPEAKER_01: You deter some of the most destabilizing AI projects.
- 36:25You make sure that some of these capabilities are not
- 36:29proliferating to random rogue actors.
- 36:31And you increase your competitiveness relative to China through
- 36:36things like improving,
- 36:39SPEAKER_01: incorporating AI in your military.
- 36:41SPEAKER_01: By like,
- 36:41SPEAKER_01: for instance,
- 36:42improving your ability to manufacture drones and improving your
- 36:44ability to reliably
- 36:46SPEAKER_01: get your hands on AI chips,
- 36:47SPEAKER_01: even if there's a Taiwan conflict.
- 36:49SPEAKER_01: So I think that's the strategy.
- 36:53SPEAKER_01: And this doesn't make us uncompetitive.
- 36:57SPEAKER_01: We're still focusing on competitiveness,
- 36:59but this does put barriers around some of the threats that
- 37:04different states could
- 37:05pose to us and that rogue actors using AIs could pose to us
- 37:10while still shoring up
- 37:12economic security and positioning ourselves if AI becomes
- 37:15really relevant.
- 37:15SPEAKER_00: I lied.
- 37:16SPEAKER_00: I had one more short question.
- 37:17SPEAKER_00: If we avoid the dire scenarios, what is...
- 37:22SPEAKER_00: What does the world look like in 2045?
- 37:24SPEAKER_01: I would guess that would be utterly transformed.
- 37:33I mean,
- 37:33I wouldn't expect people would be working then as much,
- 37:36hopefully.
- 37:37If you've controlled it well,
- 37:39there could be many ways of living as there is now.
- 37:42SPEAKER_01: And people would have resources to do so.
- 37:44SPEAKER_01: So it's not like there's one way of living there.
- 37:48SPEAKER_01: That seems bad because there's many different values to pursue.
- 37:54SPEAKER_01: So letting people pursue their own values,
- 37:57SPEAKER_01: so long as it doesn't destroy the system and things like that,
- 38:00SPEAKER_01: as we have today,
- 38:04SPEAKER_01: It seems like an abstract version of the picture.
- 38:07SPEAKER_01: People keep thinking like, are we in zoos?
- 38:10SPEAKER_01: Are AI's keeping us in zoos or something like that?
- 38:13SPEAKER_01: It's like, no.
- 38:13Or like,
- 38:14are we just all in the Zuckerberg sort of virtual reality AI
- 38:20friend thing,
- 38:21though?
- 38:21SPEAKER_01: It's like, no, you can choose to do otherwise as well.
- 38:24SPEAKER_01: So I think we want to preserve that ability.
- 38:27SPEAKER_01: Good news, we won't have to fold laundry.
- 38:28SPEAKER_00: Bad news in zoos.
- 38:30SPEAKER_00: There's many scenarios.
- 38:31SPEAKER_00: Yeah.
- 38:33SPEAKER_01: Yeah, yeah, yeah.
- 38:34So I think what we want to shoot for is one where people have
- 38:40enough resources and
- 38:42SPEAKER_01: the ability to just live their lives in ways as they self-determine.
- 38:50SPEAKER_01: subject to not harming others in severe ways.
- 38:54But people tend to think there's some sort of forced dichotomy
- 38:58of it's going to be
- 38:59a Wally world where everybody has to live the same way or
- 39:03everybody's in zoos or
- 39:04SPEAKER_01: everybody's just pleasured out and drugged up or something.
- 39:07SPEAKER_01: It's forced choices.
- 39:09SPEAKER_01: Some people do that.
- 39:10SPEAKER_01: Some people choose to have drugs and they don't really...
- 39:13SPEAKER_01: We don't hear much from them.
- 39:14And others choose to flourish and pursue projects and raise
- 39:19children and so on.