Latest / Faster, Please! โ The Podcast / ๐ My chat (+transcript) with researcher Toby Ord on existential risk
Transcript
- 0:06SPEAKER_01: Welcome to Faster Please, the podcast.
- 0:09SPEAKER_01: I'm your host, Jim Pethikoukis.
- 0:12SPEAKER_01: Several times a month,
- 0:13I'll feature a lively conversation with a fascinating and
- 0:16provocative guest about
- 0:18SPEAKER_01: how to make the world a better place through scientific discovery,
- 0:21SPEAKER_01: technological innovation,
- 0:23SPEAKER_01: and economic growth.
- 0:24SPEAKER_01: You're also going to want to check out my Faster Please newsletter.
- 0:27SPEAKER_01: You're on Substack throughout the week for fresh essays,
- 0:30SPEAKER_01: Q&As,
- 0:31SPEAKER_01: and stories from around the internet and around the world.
- 0:36The 2020s have so far been marked by pandemic,
- 0:39war,
- 0:40and startling technological breakthroughs.
- 0:43SPEAKER_01: Conversations around climate disaster,
- 0:46SPEAKER_01: great power conflict,
- 0:47SPEAKER_01: and malicious AI are seemingly everywhere.
- 0:51SPEAKER_01: It's enough to make anyone feel like the end might be near.
- 0:55Toby Ord has made it his mission to figure out just how close
- 0:58we are to catastrophe,
- 0:59SPEAKER_01: and maybe not close at all.
- 1:01Orr is the author of the 2020 book,
- 1:04The Precipice,
- 1:05Existential Risk and the Future of Humanity.
- 1:08SPEAKER_01: In 2024,
- 1:09SPEAKER_01: he delivered his talk,
- 1:10SPEAKER_01: The Precipice Revisited,
- 1:12in which he reassessed his outlook on the biggest threats
- 1:15facing humanity.
- 1:17SPEAKER_01: Today on Faster Please,
- 1:18SPEAKER_01: the podcast,
- 1:19SPEAKER_01: Orr and I address the lessons of COVID,
- 1:21SPEAKER_01: our risks of nuclear war,
- 1:23SPEAKER_01: potential pathways for AI,
- 1:25SPEAKER_01: and much more.
- 1:26SPEAKER_01: Orr is a senior researcher at Oxford University.
- 1:30Let's just start out by taking a brief tour through the
- 1:33existential landscape and
- 1:35SPEAKER_01: how you see it,
- 1:37SPEAKER_01: how you see it now versus first wrote the book,
- 1:39SPEAKER_01: The Precipice,
- 1:40SPEAKER_01: which I've mentioned frequently in my writings.
- 1:43SPEAKER_01: I love that book.
- 1:43SPEAKER_01: Love to see a sequel at some point, maybe once in the works.
- 1:46SPEAKER_01: Let's start with the existential risk,
- 1:49SPEAKER_01: which is sort of dominated,
- 1:51SPEAKER_01: you know,
- 1:53SPEAKER_01: many people's thinking for the past quarter century,
- 1:55SPEAKER_01: which is climate change.
- 1:57SPEAKER_01: My sense is,
- 1:59SPEAKER_01: not just you,
- 2:00SPEAKER_01: but many people are somewhat less worried than they were five years ago,
- 2:05SPEAKER_01: 10 years ago.
- 2:07SPEAKER_01: Perhaps they see at least the most extreme outcomes less likely.
- 2:11SPEAKER_01: How do you see it?
- 2:12SPEAKER_00: I would agree with that.
- 2:13SPEAKER_00: I'm not sure that everyone sees it that way,
- 2:17but there were two really big and good pieces of news on
- 2:21climate that were rarely
- 2:23SPEAKER_00: reported in the media.
- 2:25SPEAKER_00: One of them is that
- 2:27SPEAKER_00: there's the question about how many emissions there'll be, right?
- 2:30We don't know how much carbon humanity will emit into the
- 2:34atmosphere before we get
- 2:35SPEAKER_00: it under control.
- 2:36SPEAKER_00: And there are these different emissions pathways,
- 2:38SPEAKER_00: you know,
- 2:38SPEAKER_00: these RCP 4.5 and things like this you will have heard of.
- 2:42SPEAKER_00: And often when people would give a sketch of how bad things could be,
- 2:47SPEAKER_00: they would talk about RCP 8.5,
- 2:48SPEAKER_00: which is the worst of these pathways.
- 2:51SPEAKER_00: And we're very clearly not on that.
- 2:54SPEAKER_00: And we're also, I think, pretty clearly now not on RCP6 either.
- 2:59SPEAKER_00: So the two worst pathways, we're pretty clearly not on.
- 3:04And so that's pretty good news that we're kind of headed more
- 3:08towards one of the
- 3:09SPEAKER_00: better pathways in terms of the emissions that we'll put out there.
- 3:14SPEAKER_01: What are we doing right?
- 3:17SPEAKER_00: Ultimately,
- 3:18SPEAKER_00: some of those pathways were based on business-as-usual ideas,
- 3:21SPEAKER_00: that there wouldn't be climate change as one of the biggest issues in the
- 3:26SPEAKER_00: international political sphere over decades.
- 3:30SPEAKER_00: So ultimately,
- 3:31nations have been switching over to renewables and low-carbon
- 3:37forms of power,
- 3:38SPEAKER_00: which is good news.
- 3:43SPEAKER_00: back when we initially created these things,
- 3:45I think we would have been surprised and happy to find out that
- 3:47we were going to
- 3:47SPEAKER_00: end up among the better two pathways instead of the worst ones.
- 3:52SPEAKER_00: The other big one,
- 3:53SPEAKER_00: I think,
- 3:53SPEAKER_00: is that as well as how much we'll emit,
- 3:58there's the question of how bad is it to have a certain amount
- 4:02of carbon in the atmosphere?
- 4:03SPEAKER_00: So in particular, how much warming does it produce?
- 4:07SPEAKER_00: And this is something of which there's been massive uncertainty.
- 4:10SPEAKER_00: The general idea is that
- 4:12that we're trying to predict if we were to double the amount of
- 4:15carbon in the
- 4:15SPEAKER_00: atmosphere compared to pre-industrial times,
- 4:18SPEAKER_00: how many degrees of warming would there be?
- 4:20SPEAKER_00: And the best guess,
- 4:21SPEAKER_00: actually,
- 4:23SPEAKER_00: since I was born,
- 4:24SPEAKER_00: the year I was born,
- 4:25SPEAKER_00: 1979,
- 4:26SPEAKER_00: has been three degrees of warming.
- 4:28But the uncertainty has been somewhere between one and a half
- 4:30degrees and four and a half.
- 4:31SPEAKER_01: Is that Celsius or Fahrenheit, by the way?
- 4:33SPEAKER_00: Oh, sorry.
- 4:34SPEAKER_00: Yes, this is all Celsius.
- 4:36SPEAKER_00: So the climate community has kept the same uncertainty,
- 4:40SPEAKER_00: you know,
- 4:40SPEAKER_00: from 1979 all the way up to 2020.
- 4:44SPEAKER_00: And this wild level of uncertainty,
- 4:46four and a half degrees of warming is three times one and a
- 4:48half degrees of warming.
- 4:50SPEAKER_00: So the range is,
- 4:51SPEAKER_00: you know,
- 4:51up to triple these levels of degrees of warming based on this
- 4:55amount of carbon.
- 4:56SPEAKER_00: So massive uncertainty that hadn't changed over many decades.
- 4:59And now they've actually revised that and have actually brought
- 5:04in the range of uncertainty.
- 5:08Now they're pretty sure that it's somewhere between two and a
- 5:11half and four degrees.
- 5:13SPEAKER_00: And this is based on better understanding of climate feedbacks.
- 5:16And this is good news if you're concerned about worst case
- 5:20climate change.
- 5:20It's saying it's closer to the central estimate than we
- 5:23previously thought,
- 5:24whereas previously we thought that there was a pretty high
- 5:27chance that it could
- 5:28SPEAKER_00: even be higher than four and a half degrees of warming.
- 5:31SPEAKER_00: You know,
- 5:31when you hear these targets of one and a half degrees of
- 5:34warming or two degrees of warming,
- 5:38SPEAKER_00: they sound quite precise and
- 5:40SPEAKER_00: But in reality,
- 5:42we were just so uncertain of how much warming would follow from
- 5:45any particular
- 5:45SPEAKER_00: amount of emissions that it was very hard to know.
- 5:49SPEAKER_00: And that could mean that things are better than we'd thought,
- 5:51SPEAKER_00: but it could also mean things would be much worse.
- 5:53SPEAKER_00: And if you're concerned about existential risks from climate change,
- 5:57then those kind of tail events where it's much worse than we
- 6:01would have thought are
- 6:01SPEAKER_00: the things that would really get you.
- 6:03SPEAKER_00: And we're now pretty sure that we're not on one of those...
- 6:07SPEAKER_00: extreme emissions pathways,
- 6:09and also that we're not in a world where the temperature is
- 6:11extremely sensitive to
- 6:12SPEAKER_00: those emissions.
- 6:14What do you make of this sort of emerging nuclear power revival
- 6:18you're seeing
- 6:19SPEAKER_01: across Europe,
- 6:20SPEAKER_01: Asia,
- 6:20SPEAKER_01: and in the United States?
- 6:22These United States partially being driven by the need for more
- 6:25power for these AI data centers.
- 6:27SPEAKER_01: How does it sort of change your perception of risk in a world where
- 6:33many rich countries or maybe even not so rich countries start
- 6:36sort of re-embracing
- 6:37SPEAKER_01: nuclear energy?
- 6:39SPEAKER_00: Yeah.
- 6:39SPEAKER_00: So in terms of the local risks with the power plants,
- 6:43SPEAKER_00: so risks of meltdown or other types of harmful radiation leak,
- 6:50SPEAKER_00: I'm not too concerned about that.
- 6:52Ultimately,
- 6:53when it comes to the deaths caused by different power sources,
- 6:57I've
- 6:57SPEAKER_00: Coal,
- 6:58even setting aside global warming just through particulates
- 7:02being produced in the soot,
- 7:03SPEAKER_00: killed many more people than nuclear does.
- 7:07SPEAKER_00: Much, much more.
- 7:09SPEAKER_00: And so nuclear is a pretty safe form of energy production as it happens,
- 7:14SPEAKER_00: contrary to popular perception.
- 7:17SPEAKER_00: So I'm in favor of that.
- 7:19SPEAKER_00: But the proliferation concerns,
- 7:21SPEAKER_00: if it is countries that didn't already have nuclear power,
- 7:24then the possibility that they would be able to use that to
- 7:27start a weapons program
- 7:29SPEAKER_00: would be concerning.
- 7:30SPEAKER_01: And it's sort of a mechanism for more clean energy.
- 7:35SPEAKER_01: Do you view nuclear as clean energy?
- 7:37SPEAKER_00: Yes, I think so.
- 7:39SPEAKER_00: It's certainly not carbon producing energy.
- 7:42SPEAKER_00: I think that it has various downsides,
- 7:44SPEAKER_00: including the difficulty of knowing exactly what to do with the fuel,
- 7:47SPEAKER_00: that would be a very long lasting problem.
- 7:51But I think it's become clear that the problems caused by other
- 7:54forms of energy are
- 7:55SPEAKER_00: much larger,
- 7:56and we should switch to the thing that has fuel problems rather
- 7:59than more problems.
- 8:00SPEAKER_01: I recently read a book,
- 8:01SPEAKER_01: finished a book called nuclear war,
- 8:03SPEAKER_01: a scenario,
- 8:04SPEAKER_01: which is kind of a,
- 8:04SPEAKER_01: a minute by minute look at how a nuclear war could break out.
- 8:09SPEAKER_01: And if you've read the book,
- 8:11the book is you know the book is terrifying because it really
- 8:13goes into a lot of
- 8:14and i live near washington dc so when it gives its various
- 8:17scenarios uh certainly
- 8:19uh my house is included in the blast zone so a really uh a
- 8:23frightening book but
- 8:25when it tried to explain like how a war would start i didn't
- 8:27find it a particularly
- 8:28compelling book the scenarios were actually starting a conflict
- 8:32i didn't think
- 8:33SPEAKER_01: sounded particularly realistic i
- 8:35SPEAKER_01: Do you feel,
- 8:36SPEAKER_01: and obviously we have,
- 8:36SPEAKER_01: you know,
- 8:37Russia invade Ukraine and loose talk by Vladimir Putin about
- 8:41nuclear weapons.
- 8:42Do you feel more or less confident that we'll avoid a nuclear
- 8:46war than you did when
- 8:47SPEAKER_01: you wrote the book?
- 8:48SPEAKER_00: Much less confident, actually.
- 8:51I guess I should say when I wrote the book,
- 8:53it came out in 2020,
- 8:54finished the writing in 2019.
- 8:55SPEAKER_00: And, uh,
- 8:57SPEAKER_00: you know, ultimately we're in a time of relatively low nuclear risk.
- 9:01SPEAKER_00: And I feel that the risk has risen.
- 9:04That said,
- 9:05I was trying to provide estimates for the risk over the next
- 9:08hundred years.
- 9:08And so I wasn't assuming that the low risk period would
- 9:10continue indefinitely.
- 9:12But it was quite a shock to end up so quickly back in this
- 9:16period of heightened
- 9:17SPEAKER_00: tensions and threats of nuclear escalation.
- 9:21SPEAKER_00: The type of thing I thought was really from my parents' generation.
- 9:23SPEAKER_00: And
- 9:26SPEAKER_00: So,
- 9:26SPEAKER_00: yes,
- 9:27I do think that the Ukraine war in particular has created a lot
- 9:32of possible flashpoints.
- 9:34SPEAKER_00: That said,
- 9:34SPEAKER_00: there's been,
- 9:35SPEAKER_00: you know,
- 9:35SPEAKER_00: the temperature has come down on the conversation in the last year.
- 9:38SPEAKER_00: So that's something.
- 9:40Of course,
- 9:41the conversation might heat right back up if you see a Chinese
- 9:44invasion of Taiwan.
- 9:46SPEAKER_01: I've been very, very bullish about like the U.S.
- 9:49SPEAKER_01: economy and world economy over the rest of this decade.
- 9:52SPEAKER_01: But sort of the exception is as long as we don't have a war with China,
- 9:55SPEAKER_01: you know,
- 9:56SPEAKER_01: from an economic point of view.
- 9:58But I certainly also from a nuclear point of view,
- 10:01I mean,
- 10:02two nuclear armed powers in conflict.
- 10:04That would not be an insignificant event from from from from
- 10:07the risk existential
- 10:08SPEAKER_01: risk perspective.
- 10:10SPEAKER_00: I mean,
- 10:10it is good that China has a smaller nuclear arsenal than the US
- 10:14or Russia,
- 10:15SPEAKER_00: but it could easily be a great tragedy.
- 10:18SPEAKER_01: The book comes out during the pandemic.
- 10:20Did our response to the pandemic make you more or less
- 10:24confident in our ability and
- 10:28SPEAKER_01: willingness to confront that kind of outbreak,
- 10:31SPEAKER_01: the worst one that we saw in 100 years?
- 10:34SPEAKER_00: Yeah, overall, it made me much less confidence.
- 10:37There'd been general thought by those who look at these large
- 10:41catastrophic risks that,
- 10:43SPEAKER_00: you know,
- 10:44SPEAKER_00: when the chips are down and the threat is imminent,
- 10:47that people will see it and will band together and put a lot of
- 10:50effort into it.
- 10:52SPEAKER_00: You know,
- 10:52SPEAKER_00: that once,
- 10:52SPEAKER_00: you know,
- 10:53SPEAKER_00: you see the asteroid in your telescope and it's headed for you,
- 10:55SPEAKER_00: you know,
- 10:56SPEAKER_00: that then things will really get together,
- 10:57SPEAKER_00: you know,
- 10:58SPEAKER_00: a bit like in the action movies or what have you.
- 11:01SPEAKER_01: That's where I take my cue from.
- 11:03SPEAKER_01: Exactly.
- 11:04SPEAKER_00: And with COVID, it was kind of staring us in the face.
- 11:10Those of us who follow these things closely were quite alarmed
- 11:16a long time before
- 11:18SPEAKER_00: the national authorities were.
- 11:21SPEAKER_00: Overall, a lot of countries really just muddled through not very well.
- 11:25SPEAKER_00: And the
- 11:27SPEAKER_00: large institutions that were supposed to protect us from these things,
- 11:31SPEAKER_00: like the CDC and the WHO,
- 11:33SPEAKER_00: didn't do a great job either.
- 11:35That said,
- 11:36scientists,
- 11:37particularly developing RNA vaccines,
- 11:40did better than I expected.
- 11:43In the years leading up to the pandemic,
- 11:44certainly we'd seen other outbreaks.
- 11:46SPEAKER_01: You know, they had the avian flu outbreak.
- 11:48SPEAKER_01: And you know,
- 11:48SPEAKER_01: as well as I do,
- 11:49there were how many white papers or scenario planning exercises
- 11:54for just this sort
- 11:55SPEAKER_01: of event.
- 11:56SPEAKER_01: I think I recall a story where in 2018,
- 11:59Bill Gates had a conversation with President Trump during his
- 12:02first term about the
- 12:03SPEAKER_01: risk of just such an outbreak.
- 12:05SPEAKER_01: So it's not as if this thing came out of the out of the blue.
- 12:09SPEAKER_01: In many ways, we saw the asteroid.
- 12:10SPEAKER_01: It was just maybe pretty far away.
- 12:12SPEAKER_01: But to me,
- 12:12SPEAKER_01: that says something,
- 12:14SPEAKER_01: again,
- 12:14SPEAKER_01: about as humans,
- 12:15our ability to deal with severe but infrequent risks and
- 12:22obviously not having a
- 12:23SPEAKER_01: true global nasty outbreak in 100 years.
- 12:26SPEAKER_01: Where should we focus our effort?
- 12:27SPEAKER_01: And sort of preparation, making sure we have enough
- 12:31SPEAKER_01: you know, ventilators or our ability to respond?
- 12:35SPEAKER_01: Because it seems like the preparation route will only go so far.
- 12:38And the reason it wasn't a much worse outbreak is because we
- 12:41have a really strong
- 12:42SPEAKER_01: ability to respond.
- 12:45SPEAKER_00: Yeah, I'm not sure if it's the same across all risks as to how it
- 12:51SPEAKER_00: how preparation versus ability to respond, which one is better.
- 12:55SPEAKER_00: In some risks,
- 12:56SPEAKER_00: there's also other possibilities,
- 12:57SPEAKER_00: like avoiding an outbreak,
- 12:59SPEAKER_00: say an accidental outbreak happening at all,
- 13:01SPEAKER_00: or avoiding a war,
- 13:02SPEAKER_00: a nuclear war starting,
- 13:04SPEAKER_00: and not needing to actually respond at all.
- 13:07So yeah,
- 13:08I'm not sure if there's an overall rule as to which one is better.
- 13:11SPEAKER_01: Do you have an opinion on the outbreak of COVID?
- 13:15SPEAKER_00: I don't know whether it was a lab leak.
- 13:17I think it's a very plausible hypothesis,
- 13:18but plausible doesn't mean it's proven.
- 13:21SPEAKER_01: And does the sort of post-COVID reaction,
- 13:23SPEAKER_01: at least in the United States,
- 13:25SPEAKER_01: to vaccines,
- 13:26does that make you more or less confident in our ability to
- 13:30deal with the kind of
- 13:32SPEAKER_01: societal cohesion and confidence to tackle a big problem,
- 13:37SPEAKER_01: to have enough trust?
- 13:38SPEAKER_01: And maybe, again, maybe our leaders don't deserve that trust.
- 13:41But what do you make from sort of this kind of pushback against
- 13:44vaccines and our,
- 13:45SPEAKER_01: at least in the United States,
- 13:46SPEAKER_01: our medical authorities?
- 13:48SPEAKER_00: You know,
- 13:49SPEAKER_00: when COVID was first really striking Europe and America,
- 13:53it was generally thought that while China was locking down the
- 13:58Wuhan area,
- 13:59SPEAKER_00: that Western countries wouldn't be able to lock down,
- 14:02SPEAKER_00: that it wasn't something that we could really do.
- 14:05SPEAKER_00: But then various governments did order lockdowns.
- 14:08SPEAKER_00: That said,
- 14:09SPEAKER_00: if you look at the data on movement of citizens,
- 14:12SPEAKER_00: it turns out that citizens stopped moving around prior to the lockdowns.
- 14:15So the lockdown announcements were more kind of like the tail
- 14:19rather than,
- 14:20you know,
- 14:21the dog.
- 14:21SPEAKER_00: But over time,
- 14:23citizens wanted to kind of get back out and interact more and
- 14:26the rules were
- 14:27SPEAKER_00: preventing them.
- 14:29if a large fraction of the citizens were under something like
- 14:31house arrest for you
- 14:33know the better part of a year would that lead to some fairly
- 14:37extreme resentment uh
- 14:39and some you know backlash uh some of which was fairly
- 14:43irrational yeah that is
- 14:45actually exactly the kind of thing uh that you would expect you
- 14:49know it's very
- 14:49difficult to get a whole lot of people to row together and take
- 14:52the same kind of
- 14:52response you know that we needed to coordinate the response to
- 14:55prevent the spread
- 14:56SPEAKER_00: And pushing for that had some of these bad consequences,
- 15:01SPEAKER_00: which are also going to make it harder for next time.
- 15:04SPEAKER_00: We haven't exactly learned the right lessons.
- 15:07SPEAKER_01: We're more than halfway through our chat,
- 15:08and now we're going to get to the topic that probably most
- 15:10people would like to
- 15:12SPEAKER_01: hear about.
- 15:13SPEAKER_01: After the robots take our jobs, are they going to kill us?
- 15:16SPEAKER_01: What do you think?
- 15:17SPEAKER_01: What is your concern about AI risk?
- 15:20SPEAKER_00: I'm quite concerned about it.
- 15:21SPEAKER_00: Ultimately,
- 15:23SPEAKER_00: when I wrote my book,
- 15:24SPEAKER_00: I put AI risk as the biggest existential risk,
- 15:28SPEAKER_00: albeit the most uncertain as well.
- 15:31SPEAKER_00: And I would still say that.
- 15:33SPEAKER_00: That said, some things have gotten better since then.
- 15:37SPEAKER_01: I would assume what makes you less confident is one,
- 15:40SPEAKER_01: what seems to be the rapid advance,
- 15:44SPEAKER_01: not just the rapid advance of the technology,
- 15:46but you have the two leading countries in a geopolitical
- 15:50globalization also being
- 15:52SPEAKER_01: the leaders in the technology and not wanting to slow down.
- 15:55SPEAKER_01: I would imagine that would make you...
- 15:58SPEAKER_01: more worried that we'll move too quickly,
- 16:02what would make you more confident that we would avoid any
- 16:06serious existential downsides?
- 16:08SPEAKER_00: I mean, I agree with your supposition.
- 16:10Attempts by the US and China to turn this into some kind of
- 16:15arms race are quite concerning.
- 16:17SPEAKER_00: But here are a few things.
- 16:22were things like AlphaGo,
- 16:24if you remember that,
- 16:25or the Atari playing systems.
- 16:27SPEAKER_00: Quite, quite, quite.
- 16:29SPEAKER_00: Yeah,
- 16:30but it was very zero-sum reinforcement learning-based game
- 16:34playing where these
- 16:36SPEAKER_00: systems were learning directly to behave adversarially to other systems.
- 16:41And they could only understand a kind of limited aspect about
- 16:44the world and
- 16:45SPEAKER_00: struggle and overcoming your adversary.
- 16:47SPEAKER_00: That was really all they could do.
- 16:48And the idea of teaching them about ethics or how to treat
- 16:51people and the diversity
- 16:53SPEAKER_00: of human values seemed almost impossible.
- 16:56SPEAKER_00: How do you tell a chess program about it?
- 16:59SPEAKER_00: But,
- 17:05SPEAKER_00: trying to maximize something.
- 17:07They're rather,
- 17:09you know,
- 17:09you ask them questions and they blurt out some answers.
- 17:14And these systems have read,
- 17:16you know,
- 17:17more books on ethics and moral philosophy than I have.
- 17:21SPEAKER_00: And they've read all kinds of books about the human condition,
- 17:23SPEAKER_00: you know,
- 17:24SPEAKER_00: almost all novels that have ever been published.
- 17:26And pretty much every page of every novel involves people
- 17:29judging the actions of
- 17:30SPEAKER_00: other people and having some kind of opinions about them.
- 17:33And so there's a huge amount of data about human values and how
- 17:38we think about each
- 17:39SPEAKER_00: other and what's inappropriate behavior.
- 17:40SPEAKER_00: And if you ask the systems about these things,
- 17:43they're pretty good at judging whether something's
- 17:44inappropriate behavior if you
- 17:45SPEAKER_00: describe it.
- 17:47SPEAKER_00: The real challenge remaining is to get them to care about that.
- 17:51SPEAKER_00: But at least the knowledge is in the system.
- 17:53SPEAKER_00: And that's something that previously seemed extremely difficult to do.
- 17:57SPEAKER_00: And also these systems,
- 17:59there are versions that do reasoning and so that spend longer
- 18:05with a kind of
- 18:05SPEAKER_00: private text stream where they think,
- 18:09it's kind of like sub-vocalizing thoughts to themselves before
- 18:12they answer.
- 18:14And when we do that,
- 18:15these systems are thinking,
- 18:16you know,
- 18:17in English,
- 18:18in plain English.
- 18:19SPEAKER_00: And that's something that we really didn't expect.
- 18:21SPEAKER_00: If you look at all of the weights of a neural network,
- 18:23SPEAKER_00: it's quite inscrutable,
- 18:25SPEAKER_00: famously difficult to know what it's doing.
- 18:27But somehow we've ended up with systems that are actually
- 18:30thinking in English and
- 18:31SPEAKER_00: where that could be inspected by some oversight process.
- 18:34SPEAKER_00: So there are a number of ways in which things are better than I'd feared.
- 18:40SPEAKER_01: What is your actual existential risk scenario look like?
- 18:45SPEAKER_01: Like this is what you're most concerned about happening with AI.
- 18:49SPEAKER_00: Yeah,
- 18:49SPEAKER_00: so I think it's quite hard to be all that concrete on it at the moment,
- 18:54SPEAKER_00: partly because things change so quickly.
- 18:58I don't think that there's going to be some kind of existential
- 19:01catastrophe from AI
- 19:02SPEAKER_00: in the next couple of years,
- 19:04partly because the current systems require so much compute in
- 19:08order to run them.
- 19:09SPEAKER_00: that they can only be run at very specialized and large places,
- 19:12SPEAKER_00: of which there's only a few in the world.
- 19:13And so that means the possibility that they kind of break out
- 19:17and copy themselves
- 19:18SPEAKER_00: into other systems is not really there,
- 19:21in which case the possibility of turning them off is much more
- 19:25possible as well.
- 19:26Also,
- 19:27they're not yet intelligent enough to be able to execute a
- 19:31lengthy plan.
- 19:33SPEAKER_00: If you have some kind of complex task for them that requires,
- 19:36SPEAKER_00: say,
- 19:36SPEAKER_00: 10 steps,
- 19:37SPEAKER_00: for example,
- 19:38booking a flight on the internet by clicking through all the
- 19:41appropriate pages and
- 19:42finding out when the times are and managing to book your ticket
- 19:45and fill in the
- 19:45SPEAKER_00: special codes they send to your email and things like that.
- 19:48That's a somewhat laborious task and the systems can't do
- 19:51things like that yet.
- 19:52SPEAKER_00: They're still at the case that
- 19:54a 90% chance of completing any particular step,
- 19:58that the 10% chances of failure add up.
- 20:01And eventually,
- 20:02it's likely to fail somewhere along the line and not be able to recover.
- 20:06SPEAKER_00: They'll probably get better at that.
- 20:08SPEAKER_00: But at the moment,
- 20:09The inability to actually execute any complex plans does
- 20:12provide some safety.
- 20:14SPEAKER_00: Ultimately,
- 20:16SPEAKER_00: the concern is that,
- 20:17SPEAKER_00: you know,
- 20:17the more abstract level is that we're building systems which
- 20:22are smarter than us at
- 20:23SPEAKER_00: many things,
- 20:23and we're attempting to make them much more general and to be
- 20:26smarter than us
- 20:27SPEAKER_00: across the board.
- 20:28SPEAKER_00: If you know that one player is a better chess player than another,
- 20:30SPEAKER_00: I suppose Magnus Carlsen's playing me at chess,
- 20:34SPEAKER_00: I can't predict exactly how he's going to beat me,
- 20:36SPEAKER_00: but I can know with quite high likelihood that he will end up beating me.
- 20:40SPEAKER_00: I'll end up in checkmate,
- 20:42even though I don't know what moves will happen in between here
- 20:44and there.
- 20:45SPEAKER_00: And I think that it's similar with AI systems.
- 20:47If we make things that are smarter than us and are not
- 20:51inherently able to control their value
- 20:55rules to work within,
- 20:57then we should expect them to ultimately be calling the shots.
- 21:01SPEAKER_01: I frequently check out the Metaculous online prediction platform.
- 21:07SPEAKER_01: And I think currently that platform, 2027...
- 21:12SPEAKER_01: for what they would call weak AGI,
- 21:16SPEAKER_01: artificial general intelligence,
- 21:17a date which has moved up two months in the past week as we're
- 21:22recording this.
- 21:22SPEAKER_01: And then I think 2031 also has accelerated for strong AGI.
- 21:27SPEAKER_01: So this is pretty soon, 2027 or 2031, quite, quite soon.
- 21:33SPEAKER_01: Is that kind of what you're assuming is gonna happen,
- 21:37that we're gonna have to deal with very powerful technologies
- 21:41quite quickly?
- 21:42SPEAKER_00: Yeah,
- 21:43SPEAKER_00: I think that those are good numbers for the typical case,
- 21:46SPEAKER_00: like what you should be expecting.
- 21:48SPEAKER_00: I think that a lot of people wouldn't
- 21:53obstacle that slows down progress and takes longer before it
- 21:57gets overcome.
- 21:58But it's also wouldn't be surprising at this point if there are
- 22:00no more big
- 22:00obstacles and it's just a matter of scaling things up and doing
- 22:03fairly simple
- 22:05SPEAKER_00: processes to get it to work.
- 22:07SPEAKER_00: There's also, you know, it's now a multi-billion dollar industry.
- 22:11SPEAKER_00: And so there's a lot of money,
- 22:13SPEAKER_00: you know,
- 22:14SPEAKER_00: focused on ironing out any kinks or overcoming any obstacles on the way.
- 22:19SPEAKER_00: So I expect it to move pretty quickly.
- 22:22SPEAKER_00: And those timelines sound very realistic, maybe even sooner.
- 22:27SPEAKER_01: When you wrote the book,
- 22:28SPEAKER_01: what did you put as the risk to human existence over the next 100 years?
- 22:32SPEAKER_01: And what is it now?
- 22:33SPEAKER_00: Yeah.
- 22:34SPEAKER_00: So when I wrote the book, I thought it was about one in six.
- 22:37SPEAKER_01: So it's still one in six?
- 22:38SPEAKER_00: Yeah, I think that's still about right.
- 22:40SPEAKER_00: And I would say that most of that is coming from AI.
- 22:43SPEAKER_01: This isn't, I guess, a specific risk, but...
- 22:47To the extent that being positive about our future means also
- 22:54being positive on our
- 22:55SPEAKER_01: ability to work together,
- 22:57SPEAKER_01: countries working together.
- 22:59What do you make of sort of society going in the other
- 23:01direction where we see more
- 23:02SPEAKER_01: suspicious of other countries?
- 23:05SPEAKER_01: Even in the United States,
- 23:06SPEAKER_01: more suspicious of our allies,
- 23:10SPEAKER_01: more suspicious of international agreements,
- 23:11SPEAKER_01: whether they're trade or military alliances.
- 23:14SPEAKER_01: To me, I would think that the age of globalization
- 23:17SPEAKER_01: would have a net lowered that risk to one in six.
- 23:22And if we're going to have less globalization,
- 23:25to me,
- 23:25that would tend to increase that risk.
- 23:28SPEAKER_00: Yeah, they could be right.
- 23:30Certainly increased suspicion to the point of paranoia or
- 23:35cynicism about other
- 23:36SPEAKER_00: nations and their ability to say,
- 23:39SPEAKER_00: form deals on these things is not going to be helpful at all.
- 23:44SPEAKER_00: Ultimately, existential risks are, you know,
- 23:49problems uh uh this continued functioning of human civilization
- 23:53uh is this global
- 23:55public good uh and existential risk you know is the opposite uh
- 23:59and so these are
- 23:59things where uh like one way to look at it is that the us has
- 24:03about four percent of
- 24:04the world's people um so one in 25 people live in the us uh and
- 24:08so an existential
- 24:09SPEAKER_00: risk
- 24:12SPEAKER_00: So if every country is just interested in itself,
- 24:15SPEAKER_00: they will undervalue it by a factor of 25 or so.
- 24:19SPEAKER_00: And the countries need to work together in order to overcome that kind of
- 24:22SPEAKER_00: that kind of problem.
- 24:24SPEAKER_00: So ultimately, if one of us falls victim to these risks, then we all do.
- 24:29SPEAKER_00: And so it definitely does call out for international cooperation.
- 24:32SPEAKER_00: And I think that it has a strong basis for international cooperation.
- 24:35SPEAKER_00: It is in all of our interests.
- 24:38SPEAKER_00: And so I think there are also verification possibilities and so on.
- 24:43And I'm actually quite optimistic about treaties and other ways
- 24:48to move forward.