Latest / Trumponomics / Why the US Must Engage China on AI Safety Before It’s ‘Game Over’

Transcript

Why the US Must Engage China on AI Safety Before It’s ‘Game Over’ · Trumponomics

• 0:02
  Speaker 1: Bloomberg Audio Studios, podcasts, radio news. The fact is China
• 0:08
  Speaker 1: has cutting edge AI labs. There are only six months
• 0:11
  Speaker 1: behind the frontier in the US. And if they get
• 0:15
  Speaker 1: a really powerful model that can do cyber hacking, for example,
• 0:18
  Speaker 1: and they put it in their systems and their systems
• 0:20
  Speaker 1: are open source, it's game over. This is just going
• 0:23
  Speaker 1: to proliferate over all around the world. So that is
• 0:26
  Speaker 1: a very bad outcome. So let's try and be creative
• 0:29
  Speaker 1: about at least trying with China to talk to them.
• 0:32
  Speaker 1: You're going to need both sides to slow down. But
• 0:35
  Speaker 1: my point about the discussion in China, which I've came
• 0:37
  Speaker 1: into touch with when I was there, is that there
• 0:40
  Speaker 1: is at least a discussion and openness to think about safety.
• 0:44
  Speaker 1: The door is opened, just a crack, and so it's
• 0:46
  Speaker 1: up to American negotiators to go there and try and
• 0:49
  Speaker 1: give it a nudge and open it more.
• 0:59
  Speaker 2: I'm Stephanie Lander's head of Government and Economics at Bloomberg,
• 1:02
  Speaker 2: and this is Trumpnomics, the podcast that looks at the
• 1:05
  Speaker 2: economic world of Donald Trump, how he's already shaking up
• 1:08
  Speaker 2: the global economy. Modern Earth is going to happen next
• 1:15
  Speaker 2: last week we looked ahead to this week's summit between
• 1:18
  Speaker 2: the US and Chinese presidents and recording this on Tuesday,
• 1:22
  Speaker 2: May twelfth, and by the time you listen to this,
• 1:25
  Speaker 2: President Trump should already have touched down in Beijing ready
• 1:28
  Speaker 2: for what feels like a high stakes encounter on Thursday
• 1:33
  Speaker 2: with President Chi Jinping. Even though the expectations for concrete
• 1:37
  Speaker 2: policy outcomes, as we heard last week, are firmly under
• 1:40
  Speaker 2: control now. Along with trade, one of the big issues
• 1:43
  Speaker 2: hanging over that conversation will be US China competition in
• 1:48
  Speaker 2: advanced technologies and the race to dominate AI. Tesla's Elon
• 1:52
  Speaker 2: Musk and Apples. Tim Cook are in the group of
• 1:54
  Speaker 2: top executives Trump decided to invite with him to China,
• 1:58
  Speaker 2: and there is some mild speculation that President Trump might
• 2:01
  Speaker 2: further loosen the export controls on US cutting edge semiconductors
• 2:06
  Speaker 2: for Chinese companies. But the safety of this race, protecting
• 2:10
  Speaker 2: the world from the misuse of these increasingly powerful tools
• 2:15
  Speaker 2: both countries are now capable of producing, well, that doesn't
• 2:19
  Speaker 2: seem to be high on the agenda. This week's guest
• 2:22
  Speaker 2: on Trumpnomics thinks that is a big and potentially dangerous
• 2:26
  Speaker 2: missed opportunity. Sebastian Mallaby is a senior fellow at the
• 2:30
  Speaker 2: Council and Farm Relations, author of multiple award winning books,
• 2:34
  Speaker 2: and most recently of The Infinity Machine. Demisseivis Deep Mind
• 2:39
  Speaker 2: and the Quest for Superintelligence. I sat down with him
• 2:42
  Speaker 2: in our New York studio a few weeks ago now
• 2:45
  Speaker 2: to talk about his new book, The Future of AI
• 2:49
  Speaker 2: and the Role of China, which he has recently visited.
• 2:52
  Speaker 2: I think you'll find what he had to say fascinating,
• 2:56
  Speaker 2: whether hopeful or pretty alarming. Well, i'lli that to you. Sebastian. Welcome,
• 3:06
  Speaker 2: Thank you very much for being here in the flesh
• 3:08
  Speaker 2: in the New York studio, which is always fun. There's
• 3:12
  Speaker 2: so many themes coming out of your book, The Infinity Machine.
• 3:15
  Speaker 2: I wanted to start with something that we talk about
• 3:17
  Speaker 2: a fair bit on Trumpanomics, which is the impact of
• 3:21
  Speaker 2: AI on the economy, society, I guess especially jobs. But
• 3:25
  Speaker 2: I think given the themes of the book, and I
• 3:28
  Speaker 2: know things that you've been thinking about since it came out,
• 3:31
  Speaker 2: will move on fairly quickly to the question of safety.
• 3:34
  Speaker 2: How worried would we be about the pace of the
• 3:36
  Speaker 2: improvement in AI's capacity, but also how likely is it
• 3:40
  Speaker 2: that governments or anyone else will be able to bend
• 3:43
  Speaker 2: it to their will? Humanities will So first thing first,
• 3:48
  Speaker 2: we had a conversation with darren Os and Moglu about
• 3:51
  Speaker 2: it recently. I mean economists. I've noticed they tend to
• 3:54
  Speaker 2: divide into those who say, AI is going to have
• 3:58
  Speaker 2: a very radical impact on business models, on the economy,
• 4:02
  Speaker 2: on society, but that's going to take a long time
• 4:06
  Speaker 2: and we will have time to prepare. There's another group
• 4:09
  Speaker 2: who say, no, no, this is all going to happen
• 4:11
  Speaker 2: much sooner, but it will probably be less seismic. You
• 4:16
  Speaker 2: had all these years researching this book really up close,
• 4:20
  Speaker 2: looking at the evolution of this technology. What's your best
• 4:23
  Speaker 2: guess on that?
• 4:24
  Speaker 1: Yeah. I think the second view, which is that there'll
• 4:26
  Speaker 1: be an explosion in the power of the models and
• 4:29
  Speaker 1: it will immediately affect the economy, is put about by
• 4:32
  Speaker 1: the technologists who are just focused on the algorithms and
• 4:36
  Speaker 1: on the models and what the base model can do,
• 4:38
  Speaker 1: and it completely underestimates the rollout challenges. If you think
• 4:43
  Speaker 1: about this idea of recursive self improvement, which preoccupies a
• 4:47
  Speaker 1: lot of people in Silicon Valley, once you've got a
• 4:49
  Speaker 1: machine that is good enough to write the code for
• 4:52
  Speaker 1: the next iteration of the machine, and then it recursively
• 4:55
  Speaker 1: self improves. It'll just explode upwards. And that is true
• 4:59
  Speaker 1: so far as the computer science goes. It's not remotely
• 5:02
  Speaker 1: true as far as actually applying it to the real
• 5:04
  Speaker 1: economy goes, because you need to build the computing capacity
• 5:07
  Speaker 1: to serve those models. So you need all the chips,
• 5:11
  Speaker 1: and there's a shortage of chips. You then need all
• 5:12
  Speaker 1: the power to fuel all those chips. The people who
• 5:16
  Speaker 1: are in the center of this a talking about having
• 5:18
  Speaker 1: to launch data centers in space. That just gives you
• 5:21
  Speaker 1: a sense of how difficult this is going to be.
• 5:23
  Speaker 1: And that's not to say anything about the institutional frictions
• 5:26
  Speaker 1: of integrating AI into say a law firm or a
• 5:30
  Speaker 1: bank or whatever. I mean, that's a non trivial challenge.
• 5:33
  Speaker 1: You've got to get all your clients to sign off
• 5:35
  Speaker 1: on the confidentiality of their information and so forth. So
• 5:38
  Speaker 1: I do think that rolling it out such that it
• 5:40
  Speaker 1: makes a difference in terms of job displacement or productivity
• 5:43
  Speaker 1: is more kind of a decade plus than it is
• 5:45
  Speaker 1: a kind of three to five year thing.
• 5:47
  Speaker 2: And on your point about the exponential improvement, I guess
• 5:50
  Speaker 2: the other thing and the colleagues who are much more
• 5:52
  Speaker 2: focused on this than the economics team have also said
• 5:55
  Speaker 2: this to me that when you think about sort of
• 5:58
  Speaker 2: pie chart of all the kinds of time asks that
• 6:00
  Speaker 2: humans do that are part of the workplace, that exponential
• 6:04
  Speaker 2: improvement is really only in a certain subset of those tasks,
• 6:08
  Speaker 2: and there are other areas where they're really quite far
• 6:10
  Speaker 2: behind that are quite important for jobs. Yeah.
• 6:12
  Speaker 1: Another way of saying the same thing is that there
• 6:15
  Speaker 1: are sectors which are probably going to be quite difficult
• 6:18
  Speaker 1: to disrupt, and the public sector is pretty slow at
• 6:20
  Speaker 1: adopting tech, and that's a pretty big chunk of some economies,
• 6:24
  Speaker 1: right construction, healthcare. There's a whole slew of economic sectors
• 6:30
  Speaker 1: which don't seem to be very right for disruption at all.
• 6:33
  Speaker 1: And then the ones that are right for disruption the
• 6:35
  Speaker 1: archetypical people working on a screen doing knowledge work and analysis,
• 6:39
  Speaker 1: which is very similar to what AI does. As I say,
• 6:42
  Speaker 1: you've got to get the customers okay with it, and
• 6:46
  Speaker 1: that's quite a challenge.
• 6:48
  Speaker 2: Railways is the example that's often used, and it takes
• 6:51
  Speaker 2: a long time for business models to adapt to that.
• 6:54
  Speaker 2: If I think of something like the smartphone, which is
• 6:57
  Speaker 2: extraordinarily new by historical standards, less than a decade old,
• 7:02
  Speaker 2: day to day, all of us have been transformed by that.
• 7:05
  Speaker 2: So is it comparable to that that it's not necessarily
• 7:08
  Speaker 2: having a big change on the economy or on productivity,
• 7:10
  Speaker 2: but day to day we're all being affected really quite quickly.
• 7:14
  Speaker 1: Yeah. I think there's a lot of truth of that.
• 7:15
  Speaker 1: That people will be using it. They already are using it.
• 7:18
  Speaker 1: It'll occupy quite a lot of their headspace, but it's
• 7:22
  Speaker 1: not necessarily making them more productive. One of my favorite
• 7:25
  Speaker 1: stories about the anthropic agent agentic system, which was out
• 7:30
  Speaker 1: in January of this year. This is before Mythos, but
• 7:33
  Speaker 1: the earlier one. People got very excited about, Oh, this
• 7:36
  Speaker 1: agent can organize my email, it can create a bespoke
• 7:39
  Speaker 1: new summary for me in the morning. And so a
• 7:42
  Speaker 1: friend of mine was implementing all these productivity enhancing hacks,
• 7:46
  Speaker 1: and finally, after doing about four of these, she said
• 7:49
  Speaker 1: to the agent, would you please now sync up my
• 7:53
  Speaker 1: calendar such that you check that the train timetables on
• 7:58
  Speaker 1: average are not too late, or some sort of extra
• 8:00
  Speaker 1: productivity thing to which Anthropic answered or Claude answered, Are
• 8:05
  Speaker 1: you sure you're not just giving me this taskause you're
• 8:07
  Speaker 1: trying to avoid your own work, you know? So, I mean,
• 8:11
  Speaker 1: I do think there's a difference between the excitement around
• 8:15
  Speaker 1: it and the adoption of it to actually make you
• 8:18
  Speaker 1: more productive. And then even in science, Alpha fold, which
• 8:22
  Speaker 1: was the deep mind system published in twenty twenty end
• 8:25
  Speaker 1: of twenty twenty for which de mister Sabis got the
• 8:28
  Speaker 1: Nobel Prize, which unraveled protein folding. This was said at
• 8:32
  Speaker 1: the time to be the breakthrough for faster drug discovery,
• 8:36
  Speaker 1: and it probably will be over the longer term. But
• 8:39
  Speaker 1: now we're six years later, there isn't a single drug
• 8:41
  Speaker 1: that's been discovered thanks to alpha fold. Not to say
• 8:44
  Speaker 1: that there hasn't been progressed, because now there's a much
• 8:46
  Speaker 1: more sophisticated build out of alpha fold, with other aspects
• 8:50
  Speaker 1: of the drug discovery process being automated by AI. So
• 8:54
  Speaker 1: I think on a kind of ten to fifteen year horizon,
• 8:57
  Speaker 1: yes it will massively accelerate drug discovery, but on a
• 9:00
  Speaker 1: six year horizon hasn't.
• 9:02
  Speaker 2: That's a really interesting example. There is a wonderful one
• 9:04
  Speaker 2: of the good documentaries about that whole period. It's almost
• 9:07
  Speaker 2: a sort of companion to your book or a part
• 9:09
  Speaker 2: of your book. But there's a wonderful bit of footage
• 9:11
  Speaker 2: where they have decided to just identify all of the
• 9:15
  Speaker 2: proteins and then and just put them onto the Internet
• 9:17
  Speaker 2: and have them allow researchers anywhere to access them. And
• 9:20
  Speaker 2: they're watching in real time the uploads from all over
• 9:24
  Speaker 2: the world on a map of the world, and it's
• 9:26
  Speaker 2: very moving because you're sort of seeing in India and
• 9:28
  Speaker 2: Africa or all these researchers a lot more than they
• 9:30
  Speaker 2: thought in the space of a few hours uploading all
• 9:32
  Speaker 2: this information. But it's fascinating to hear that they have
• 9:35
  Speaker 2: not actually produced any new drugs on the back of it.
• 9:38
  Speaker 2: I mean that goes to something and we move into
• 9:40
  Speaker 2: the broader sort of policy and safety territory a little bit.
• 9:43
  Speaker 2: But Darrenus and Moglu was on the show recently and
• 9:48
  Speaker 2: I was checking in with him because we talked to
• 9:51
  Speaker 2: him over time on this very issue, and his focus
• 9:55
  Speaker 2: and his advice to governments had been to think about
• 10:00
  Speaker 2: out you can't alter the nature of the technology, but
• 10:02
  Speaker 2: you can change the way it's adopted, and that it
• 10:05
  Speaker 2: was adoption was going to be the main thing that
• 10:08
  Speaker 2: determined how jobs were affected, whether you're enhancing productivity of
• 10:15
  Speaker 2: humans or merely just allowing businesses to replace humans. I
• 10:19
  Speaker 2: later have a conversation with Jason Furman about that, the
• 10:21
  Speaker 2: former head of the Council of Economic Advisors, and he
• 10:23
  Speaker 2: thought that was just completely wishful thinking that you can't
• 10:25
  Speaker 2: tilt the technology even to that extent that you can
• 10:28
  Speaker 2: encourage it to be more complementary to labor as opposed
• 10:31
  Speaker 2: to just substituting for labor. So what do you think
• 10:33
  Speaker 2: about that?
• 10:35
  Speaker 1: You know, I think in a race dynamic where you
• 10:36
  Speaker 1: have multiple labs in multiple countries producing this stuff, the
• 10:40
  Speaker 1: idea that you're going to control all of them in
• 10:43
  Speaker 1: some direction that makes the technology complementary to labor, not replacing,
• 10:49
  Speaker 1: does strike me. I'm kind of on Jason's side on that.
• 10:51
  Speaker 1: The other thing to say is that the labs are
• 10:53
  Speaker 1: going to create what the consumers want, and if the
• 10:56
  Speaker 1: consumer's meaning in some cases the business consumers, the enterprises
• 11:00
  Speaker 1: have an opportunity to replace labor, they want to take that,
• 11:03
  Speaker 1: and if they don't, their competitors and some other country will,
• 11:06
  Speaker 1: And so that the laws of competition and capitalism, I
• 11:10
  Speaker 1: think are very much on Jason's side.
• 11:12
  Speaker 2: I think part of that is this dynamic which is
• 11:14
  Speaker 2: also very relevant to the book of the race with
• 11:18
  Speaker 2: other countries, particularly the race with China. But if you're
• 11:22
  Speaker 2: a country, if you're the UK for example, or could
• 11:26
  Speaker 2: be any European country that's not necessarily going to be
• 11:28
  Speaker 2: at the forefront of the technology, but clearly does want
• 11:31
  Speaker 2: to be at the forefront or close to the front
• 11:35
  Speaker 2: in adoption in a way that also helps the economy,
• 11:38
  Speaker 2: helps their workers. Is there any advice from your book
• 11:42
  Speaker 2: or from the research that you've done that you would
• 11:45
  Speaker 2: have for government who are trying to do that?
• 11:47
  Speaker 1: Yeah, very much. I mean, I think there is a
• 11:48
  Speaker 1: lot of commentators, especially in Britain, draw exactly the wrong
• 11:52
  Speaker 1: lesson about the story of deep Mind. So the classic
• 11:56
  Speaker 1: complaint is deep Mind is a British company founded in London,
• 12:00
  Speaker 1: and main founder is British. She's very patriotic about it.
• 12:03
  Speaker 1: He refuses to move to Silicon Valley, and it turns
• 12:06
  Speaker 1: out to be the most consequential sort of pioneer AI
• 12:09
  Speaker 1: lab in the world. That's amazing, But these idiots sell
• 12:12
  Speaker 1: it to Google in twenty fourteen and it becomes a
• 12:15
  Speaker 1: subsidiary of an American company, and that's terrible, and I
• 12:19
  Speaker 1: think that's wrongheaded in the sense that really the sale
• 12:21
  Speaker 1: to Google by deep Mind in twenty fourteen was a trick,
• 12:25
  Speaker 1: one could say, to get the American parent to pour
• 12:28
  Speaker 1: nearly a billion dollars a year of research and development
• 12:31
  Speaker 1: money into London, with enormous spill of effects for the
• 12:34
  Speaker 1: tech ecosystem in London. So I think that's a win
• 12:37
  Speaker 1: for Britain. The company stayed in London, the founder stayed
• 12:40
  Speaker 1: in London. That's great for Britain. What was terrible for Britain,
• 12:44
  Speaker 1: and about which Britain should beat itself up mightily, is
• 12:48
  Speaker 1: that My book also tells the story of deep Mind
• 12:50
  Speaker 1: trying to help the National Health Service, and both Demis's
• 12:54
  Speaker 1: mother trained as as a nurse, his co founder, me
• 12:56
  Speaker 1: Stuffer Sulliman, his mother trained as a nurse. For this reason,
• 13:00
  Speaker 1: they really wanted to help the NHS. They were willing
• 13:03
  Speaker 1: to do it pro bono for the first five years,
• 13:06
  Speaker 1: and they started helping and rolling out AI algorithms that
• 13:09
  Speaker 1: recognized early signs of blindness, early signs of cancer. This
• 13:13
  Speaker 1: was just super positive, helpful stuff that was going to
• 13:16
  Speaker 1: save the NHS time and help h based patients, and
• 13:19
  Speaker 1: it got blown up by a stupid backlash over privacy
• 13:23
  Speaker 1: or privacy depending on which side of the Atlantic youuron,
• 13:26
  Speaker 1: which I mean, I really kicked the tires on this.
• 13:28
  Speaker 1: The complaint about leaks of patient data were fictitious and
• 13:33
  Speaker 1: so Britain because of this political climate of suspicion of tech,
• 13:37
  Speaker 1: particularly suspicion of the subsidiary of the American tech company,
• 13:41
  Speaker 1: created such a backlash against this collaboration between the NHS
• 13:45
  Speaker 1: and deep Mind that both the doctors and the computer
• 13:49
  Speaker 1: scientists at deep Mind got cold feet and the collaboration collapsed,
• 13:53
  Speaker 1: and Britain could have been really the cutting edge country
• 13:56
  Speaker 1: in the world on medical AI and they blew it.
• 14:00
  Speaker 2: Putting together the two pieces of what you said, they
• 14:03
  Speaker 2: are related because one of the reasons that there was
• 14:05
  Speaker 2: a lack of trust was the foreign ownership, the US ownership,
• 14:09
  Speaker 2: and particularly the association with major tech companies in Silicon
• 14:14
  Speaker 2: Valley who have not necessarily earned our trust. Many people
• 14:18
  Speaker 2: would say, over the last twenty years, they've made promises
• 14:22
  Speaker 2: about privacy, about sharing of information that have been broken.
• 14:26
  Speaker 2: Every single tech company has broken them, including Google. They
• 14:30
  Speaker 2: have changed their terms on which even they were developing AI.
• 14:34
  Speaker 2: In different cases there were promises even that had been
• 14:37
  Speaker 2: built originally into the deep Mind Charter around safety and
• 14:41
  Speaker 2: other things, which then got overridden as they needed just
• 14:45
  Speaker 2: more and more money from the parents. So it seems
• 14:47
  Speaker 2: to me, yes, in principle it would have been great
• 14:50
  Speaker 2: to have the UK on the cutting edge, but also
• 14:53
  Speaker 2: completely understandable that having given tech companies a benefit of
• 14:57
  Speaker 2: the doubt of various other times and rather regressed that
• 15:00
  Speaker 2: they wouldn't want to do it in the case of
• 15:01
  Speaker 2: this very sensitive subject.
• 15:03
  Speaker 1: Yeah, that's all true. At the same time, it would
• 15:05
  Speaker 1: have been nice if the critics looked at the specifics
• 15:08
  Speaker 1: or what was going on in this collaboration between deep
• 15:11
  Speaker 1: Mind and the National Health Service, and if they'd look
• 15:13
  Speaker 1: more carefully the specifics, they would have seen that this
• 15:16
  Speaker 1: was positive for Britain. And if you just have a
• 15:18
  Speaker 1: blanket view that anything to do with American tech is
• 15:21
  Speaker 1: ipso facto a matter of suspicion because there's been bad
• 15:25
  Speaker 1: behavior in other instances, I'm not in favor of that.
• 15:28
  Speaker 1: I think that Britain did miss an opportunity there, but
• 15:30
  Speaker 1: I think the issue that arises out of this is
• 15:32
  Speaker 1: that it's in the interest of the tech companies, especially
• 15:35
  Speaker 1: now with this super powerful technology AI, to get governance right,
• 15:40
  Speaker 1: to be proactive in inviting governments with democratic legitimacy to
• 15:46
  Speaker 1: hold sway over the technology. Because if you just take
• 15:49
  Speaker 1: it unto yourself as a youngish startup and you say
• 15:53
  Speaker 1: this is how healthcare is going to evolve, I'm doing it,
• 15:56
  Speaker 1: it's not legitimate and people won't accept it.
• 15:58
  Speaker 2: I mean, some of this gets to that sort of
• 16:01
  Speaker 2: broader question of our degree of control and also the
• 16:05
  Speaker 2: degree to which any of the people involved in this
• 16:08
  Speaker 2: can shape the outcome in line with what they themselves
• 16:11
  Speaker 2: want when they're in the competitive context. You have a
• 16:15
  Speaker 2: very striking phrase in your book towards the end, as
• 16:17
  Speaker 2: inventors dream of shaping the technology that they create, often
• 16:21
  Speaker 2: the technology shapes them. The technology plus the business, political,
• 16:24
  Speaker 2: and geopolitical currents that it unleashes. There's a bit in
• 16:28
  Speaker 2: your book where it starts to feel I felt like
• 16:30
  Speaker 2: it's some of the historical analysis that you see of
• 16:33
  Speaker 2: the start of World War One. You have everyone individually
• 16:37
  Speaker 2: in your story had potentially the right incentives. Was very
• 16:43
  Speaker 2: focused on being careful, identifying risks ahead of time, managing
• 16:48
  Speaker 2: them ahead of time, not inflicting things on society before
• 16:52
  Speaker 2: we were ready for them. The end result has not
• 16:54
  Speaker 2: been that at all.
• 16:56
  Speaker 1: That's right. The central character in my book, demister Sabez,
• 17:00
  Speaker 1: is a good person, has good values, but whether he
• 17:04
  Speaker 1: can actually do good even if he is good, is
• 17:07
  Speaker 1: the central conundrum. And he really wanted to make things safer.
• 17:11
  Speaker 1: That's why he fought a secret three year war with
• 17:13
  Speaker 1: Google trying to force them to put more safety guardrails
• 17:17
  Speaker 1: around AI. That's why he suggested to Rishie Sunak the
• 17:21
  Speaker 1: idea of a Bletchley Park AI safety conference, which happened
• 17:25
  Speaker 1: in late twenty twenty three. So he does care about safety,
• 17:29
  Speaker 1: but when other labs are willing to go fast, there
• 17:33
  Speaker 1: isn't much a single AI leader can do. And what,
• 17:36
  Speaker 1: of course that points you to is that the collective
• 17:38
  Speaker 1: action problem needs to be solved by government.
• 17:41
  Speaker 2: But what we've seen in a lot of these cases
• 17:44
  Speaker 2: is the researchers involve the companies involved say, yes, we
• 17:48
  Speaker 2: think it's very important to have a collective agreement. But
• 17:51
  Speaker 2: if a government comes along with concrete rules, whether it's
• 17:54
  Speaker 2: the I'm sure you probably feel strong about but the
• 17:57
  Speaker 2: EU Internet Safety Rules or AI Safety rules. President Biden
• 18:02
  Speaker 2: had a rather different set of rules. There tends to
• 18:05
  Speaker 2: be the response, how could governments possibly understand this technology?
• 18:09
  Speaker 2: We want some agreement, but not this one. So how
• 18:11
  Speaker 2: do you solve that?
• 18:12
  Speaker 1: Yeah, look, you're right that you know at the moment
• 18:15
  Speaker 1: there is lots of state level initiatives to regulate AI
• 18:19
  Speaker 1: in the US, and for example, open AI is actually
• 18:22
  Speaker 1: funding the opposition to those regulatory efforts. So there is
• 18:25
  Speaker 1: hypocrisy sometimes.
• 18:26
  Speaker 2: And after the fact they often say yes, actually it's
• 18:29
  Speaker 2: quite good and we've adopted it, but they never want
• 18:30
  Speaker 2: it in advance.
• 18:32
  Speaker 1: Yeah. On the other hand, there is a counterexample, which
• 18:34
  Speaker 1: is that in the Biden administration when they set up
• 18:38
  Speaker 1: the AI Safety Institute, and it included a mandatory requirement
• 18:43
  Speaker 1: for all the frontier labs to submit their models to
• 18:46
  Speaker 1: this institute for review. The institute didn't have a veto,
• 18:49
  Speaker 1: which it should have done in my view, but still
• 18:52
  Speaker 1: it was mandatory to submit. And I've spoken to the
• 18:56
  Speaker 1: Biden people who created that executive order and they say
• 19:00
  Speaker 1: that there was zero resistance from the labs. All the
• 19:04
  Speaker 1: labs work more in the mode of saying, look, you
• 19:07
  Speaker 1: need to be aware something's coming down the pike. It's
• 19:10
  Speaker 1: very powerful. You need to get ahead of it. And
• 19:12
  Speaker 1: when they did get ahead of it, or they started
• 19:14
  Speaker 1: to at least move on it, there wasn't resistance. That's
• 19:17
  Speaker 1: what the Biden people say, So I think it depends.
• 19:20
  Speaker 1: It's a bit like with Google and deep Mind and
• 19:23
  Speaker 1: the National Health Service in Britain. There can be good
• 19:25
  Speaker 1: instances of technology company behavior. And again, if we paint
• 19:29
  Speaker 1: this is true of anything, like if you paint the
• 19:31
  Speaker 1: Chinese as uniformly bad, you can never collaborate with them
• 19:35
  Speaker 1: on anything, and that's actually a bad outcome in itself.
• 19:38
  Speaker 1: Same with the tech companies.
• 19:40
  Speaker 2: Mentioned China in a US context and the group of
• 19:43
  Speaker 2: companies that you're writing about in your book. I could
• 19:45
  Speaker 2: totally see how you have a kind of circle of
• 19:47
  Speaker 2: trust among those companies and shared incentives. But the thing
• 19:51
  Speaker 2: that I think God in the way then and gets
• 19:53
  Speaker 2: in the way often in practice, is this fear of
• 19:56
  Speaker 2: China the outsider, which is not necessarily part of this system.
• 20:00
  Speaker 2: And if you want to not be applying the breaks,
• 20:03
  Speaker 2: if you want to not have these kind of control,
• 20:04
  Speaker 2: all you need to say is what China is going
• 20:06
  Speaker 2: to get ahead of us.
• 20:07
  Speaker 1: Yeah. So what I think about that is that the
• 20:10
  Speaker 1: West really needs to remember what happened in the Cold War.
• 20:14
  Speaker 1: What happened in the Cold War in the nineteen fifties
• 20:16
  Speaker 1: was you had simultaneously very moments of extreme tension, the
• 20:20
  Speaker 1: Hungarian uprising, Soviet tanks going to Hungary. You have the
• 20:24
  Speaker 1: Sewis Canal crisis at the same time, which prompts the
• 20:27
  Speaker 1: Soviets to actually threaten the West with nuclear attack because
• 20:30
  Speaker 1: of the Sewis intervention was viewed as imperiodists by the Russians.
• 20:34
  Speaker 1: And simultaneously with those events you had the creation of
• 20:38
  Speaker 1: the IAEA, the Institute for Atomic Energy and forgetting them
• 20:44
  Speaker 1: but basically counting all the nuclear material and trying to
• 20:47
  Speaker 1: prevent the nuclear material from going into loose nukes. Then
• 20:50
  Speaker 1: in the sixties you get the Cuban missile crisis, enormously
• 20:54
  Speaker 1: tense moment, and six years later you get the Nuclear
• 20:56
  Speaker 1: Non Proliferation Treaty. So you can do what Reagan said,
• 21:00
  Speaker 1: which is sort of trustpor verified, do arms control, even
• 21:03
  Speaker 1: if there is competition with China. And when I went
• 21:06
  Speaker 1: to China in March, because my book came out there
• 21:09
  Speaker 1: first and spent eight days talking to AI leaders, both
• 21:12
  Speaker 1: in academia and in the private sector. It was stunning
• 21:15
  Speaker 1: to me how often they raised the subject of safety spontaneously.
• 21:19
  Speaker 1: They would say, these agents are becoming powerful. We have
• 21:22
  Speaker 1: a window of opportunity before they are too powerful to
• 21:25
  Speaker 1: make them aligned with human incentives. The government should be
• 21:28
  Speaker 1: warning people about dangers with these models. And in fact,
• 21:32
  Speaker 1: while I was there, open Claw, this open source agent,
• 21:36
  Speaker 1: which is not a good thing to put in your
• 21:38
  Speaker 1: computer because it can do whatever it wants with all
• 21:40
  Speaker 1: of the data that was being downloaded by Chinese mums
• 21:44
  Speaker 1: and pups onto their laptops, and the government warned them
• 21:47
  Speaker 1: not to. So the idea that China doesn't care about
• 21:50
  Speaker 1: safety is wrong. There is a debate there, and they
• 21:53
  Speaker 1: love regulating the Internet. That's what they're very good at
• 21:56
  Speaker 1: in China. So why wouldn't we talk to them and say,
• 21:59
  Speaker 1: look to probably compete on a sort of grand geostrategic
• 22:03
  Speaker 1: military level, but we don't want this technology leaking out
• 22:08
  Speaker 1: all over the world and being used by terrorists, criminals,
• 22:12
  Speaker 1: nonstate actors to crash the web and do cyber hacking
• 22:16
  Speaker 1: and build a bio weapon. Nobody wants that.
• 22:31
  Speaker 2: You talked about the importance of trust earlier. I guess
• 22:33
  Speaker 2: the question is have we got to a point with
• 22:35
  Speaker 2: China where there is enough trust? I mean, the comparison
• 22:38
  Speaker 2: with Russia is an interesting one, and that you could
• 22:40
  Speaker 2: say in a sense that was a semi stable equilibrium.
• 22:44
  Speaker 2: The trust to verify was one of the things that
• 22:45
  Speaker 2: made it stable, but it was an established competition of superpowers.
• 22:50
  Speaker 2: China is a much more in flux. We've had administrations
• 22:54
  Speaker 2: have quite different approaches to controlling or competing or working
• 22:59
  Speaker 2: with China. A lot of people will even listen to
• 23:02
  Speaker 2: what you've said about your trip and say yes, they
• 23:04
  Speaker 2: discourage their own people from using these things, just as
• 23:07
  Speaker 2: they have much tougher controls on social media with their
• 23:09
  Speaker 2: own population, and they don't tend to be so worried
• 23:13
  Speaker 2: when they export all these products to the rest of
• 23:15
  Speaker 2: the world. So do you think there's trust.
• 23:16
  Speaker 1: That I think at a high level, what we should
• 23:19
  Speaker 1: first get on the table is that the alternative to
• 23:22
  Speaker 1: not even trying with China is that we just resign
• 23:25
  Speaker 1: ourselves to having AI go viral all over the world
• 23:30
  Speaker 1: with zero controls. The fact is China has cutting edge
• 23:33
  Speaker 1: AI labs. There are only six months behind the frontier
• 23:36
  Speaker 1: in the US. And if they get a really powerful
• 23:39
  Speaker 1: model that can do cyber hacking, for example, and they
• 23:42
  Speaker 1: put it in their systems, and their systems are open source,
• 23:45
  Speaker 1: it's game over. This is just going to proliferate over
• 23:48
  Speaker 1: all around the world. So that is a very bad outcome.
• 23:50
  Speaker 1: So let's try and be creative about at least trying
• 23:53
  Speaker 1: with China to talk to them. And I think the
• 23:56
  Speaker 1: notion that it was easier in the Cold War not really.
• 23:59
  Speaker 1: I mean, look, Krischev was this sort of mercurial, unpredictable,
• 24:03
  Speaker 1: semi stable l leader, and the Russians would go to
• 24:05
  Speaker 1: the UN and take their shoes off and bang them
• 24:07
  Speaker 1: on the table. It was not an easy relationship in
• 24:10
  Speaker 1: the sixties, right, and yet there was the nuclear non
• 24:13
  Speaker 1: Proliferation Treatise. I really don't think we should give up.
• 24:16
  Speaker 2: Okay. So the latest example we've had, and we've discussed
• 24:19
  Speaker 2: it while back on Trumponomics, is mythos And there's obviously
• 24:24
  Speaker 2: been a care in the way that it's been introduced,
• 24:29
  Speaker 2: controls on who can have it. I know you feel
• 24:31
  Speaker 2: strongly about the difference between the sort of safety differences
• 24:35
  Speaker 2: between open source and closed source technologies, talk us through
• 24:39
  Speaker 2: how some of the stuff we've just been talking about,
• 24:41
  Speaker 2: how that's been illuminated or not by the example of Mythos. Yeah.
• 24:45
  Speaker 1: So I think what's happened here is that whilst there
• 24:48
  Speaker 1: was progress towards some AI safety regulation by governments in
• 24:53
  Speaker 1: twenty twenty three and twenty twenty four, with national AI
• 24:56
  Speaker 1: safety institutes being set up in multiple countries, a meeting
• 24:59
  Speaker 1: of all those institutes in San Francisco in the fall
• 25:03
  Speaker 1: of twenty twenty four, Once President Trump came into office,
• 25:07
  Speaker 1: the whole system shifted towards acceleration, competition, and the momentum
• 25:14
  Speaker 1: for regulation diet even though the technology itself was speeding ahead.
• 25:20
  Speaker 1: So now we find ourselves in twenty twenty six and
• 25:22
  Speaker 1: what we all knew was going to happen has happened.
• 25:25
  Speaker 1: There is a model that, if in the wrong hands,
• 25:28
  Speaker 1: could be used to basically steal everybody's money from their
• 25:31
  Speaker 1: online bank accounts, crash internet systems, control dams, and so forth.
• 25:36
  Speaker 1: Is a pretty scary prospect, and there is no government
• 25:40
  Speaker 1: mechanism to stop that from leaking out, and there's no
• 25:44
  Speaker 1: international mechanism because you need to do it internationally because
• 25:47
  Speaker 1: this will be copied by the Chinese at some point,
• 25:50
  Speaker 1: so it's a big problem. So Anthropics response to this
• 25:53
  Speaker 1: is to say, there's no governmental governance, so we're going
• 25:57
  Speaker 1: to do private governance. We Anthropic, a five year old
• 26:00
  Speaker 1: startup in San Francisco, are going to arrogate unto ourselves,
• 26:04
  Speaker 1: the power to say these are the forty entities that
• 26:06
  Speaker 1: will get access to this technology, and the rest of you,
• 26:10
  Speaker 1: cyber peons, you're outside the castle, outside our protection, and
• 26:14
  Speaker 1: you can just stuff it. I don't think that's a
• 26:17
  Speaker 1: sustainable position for a private company to be in, to
• 26:20
  Speaker 1: be the arbiter of who is safe on the Internet
• 26:22
  Speaker 1: and who isn't. But it's fort demurur. It's like there
• 26:25
  Speaker 1: is no governance from the government, We're left with this
• 26:28
  Speaker 1: private solution, which is not sustainable. So I think what
• 26:31
  Speaker 1: this is a wake up call in terms of our
• 26:34
  Speaker 1: earlier conversation is we don't have an alternative but to
• 26:36
  Speaker 1: try and do something with China because they have these
• 26:40
  Speaker 1: open source labs which are putting this AI into everybody's
• 26:45
  Speaker 1: hands in a way that is totally not controlled. So
• 26:49
  Speaker 1: it's not theoretical now that we might have an AI
• 26:52
  Speaker 1: that's dangerous. We have one that's dangerous. So I think
• 26:55
  Speaker 1: mythos is a wake up call, and looking back at
• 26:58
  Speaker 1: the history of AI in my book, there are other
• 27:00
  Speaker 1: moments when private labs thought that they could do governance themselves.
• 27:05
  Speaker 1: Open AI thought that it had a kind of for profit,
• 27:09
  Speaker 1: nonprofit capitalist hybrid that would make it all safe. It
• 27:13
  Speaker 1: didn't work. Deep Mind tried the same thing with Google.
• 27:16
  Speaker 1: It didn't work. Basically, individual labs just can't deliver governance.
• 27:20
  Speaker 1: It needs to be government.
• 27:22
  Speaker 2: If you think that the open source putting and putting
• 27:27
  Speaker 2: open source technology into people's hands is probably the single
• 27:30
  Speaker 2: most dangerous thing around at this level of technology, and
• 27:33
  Speaker 2: China is still doing that despite, as you say, being
• 27:36
  Speaker 2: so concerned about safety. Doesn't that suggest that there's the
• 27:40
  Speaker 2: same dynamic operating in China that despite much talk and
• 27:43
  Speaker 2: concern about safety, they are still hardwired to be moving ahead. Yeah.
• 27:49
  Speaker 1: I think there's a couple of things there. I mean,
• 27:50
  Speaker 1: one is that any lab that is not right at
• 27:54
  Speaker 1: the frontier can't compete by telling customers we've got the
• 27:58
  Speaker 1: best technology. So they compete by saying my technology is
• 28:02
  Speaker 1: more available to you. You can download it into your
• 28:04
  Speaker 1: server and do whatever you want with it. And that's
• 28:07
  Speaker 1: what Meta does in the US their open source, that's
• 28:10
  Speaker 1: what Mistra does in France they are open source. And
• 28:13
  Speaker 1: that's what all the Chinese labs do because all of
• 28:16
  Speaker 1: these players have in common the fact that they are
• 28:18
  Speaker 1: not one of the big three anthropic open AI Google
• 28:22
  Speaker 1: Deep Mind, and so this is just what followers do,
• 28:26
  Speaker 1: and to stop them from doing that again you need
• 28:28
  Speaker 1: some government action. And yes, the Chinese do have a
• 28:33
  Speaker 1: sort of race mentality. They do want to build AI
• 28:36
  Speaker 1: as fast as possible because they're frightened the falling behind
• 28:39
  Speaker 1: the West, and that's just the normal, natural, obvious thing
• 28:42
  Speaker 1: for them to do. In the face of an American
• 28:45
  Speaker 1: leadership which is also racing, you're going to need both
• 28:48
  Speaker 1: sides to slow down. But my point about the discussion
• 28:51
  Speaker 1: in China, which I've i came in to touch with
• 28:55
  Speaker 1: when I was there, is that there is at least
• 28:57
  Speaker 1: a discussion and openness to think about safe. The door
• 29:00
  Speaker 1: is opened, just a crack, and so it's up to
• 29:02
  Speaker 1: American negotiators to go there and try and give it
• 29:05
  Speaker 1: a nurge and open it more.
• 29:06
  Speaker 2: It's very striking to me the conversations about AI safety,
• 29:10
  Speaker 2: the Bletchley summit that Chally Soon I had, there was
• 29:13
  Speaker 2: a lot of focus on killer robots, existential threats, the
• 29:17
  Speaker 2: sort of much darker potential scenarios, the elimination of humanity,
• 29:22
  Speaker 2: all of those things, And there was a general feeling
• 29:25
  Speaker 2: in the industry and even among some policymakers that the
• 29:28
  Speaker 2: focus on that was to some extent counterproductive because it
• 29:33
  Speaker 2: was distracting from the sort of short term things that
• 29:35
  Speaker 2: actually might cause us more harm and the impact of
• 29:40
  Speaker 2: chatbots on children, all of those kind of things. It
• 29:43
  Speaker 2: feels like we've now swung completely in favor of thinking
• 29:47
  Speaker 2: about those things, those kind of social harms. Even today
• 29:52
  Speaker 2: we had a devastating story about AI generated child porn
• 29:56
  Speaker 2: and how that's making extremely difficult for investigators to to
• 30:00
  Speaker 2: identify the true abuse of children and the footage and
• 30:05
  Speaker 2: just well, you we get very depressed reading it. But
• 30:08
  Speaker 2: sort of ironic to me that just as we've been
• 30:10
  Speaker 2: focused on those social harms, actually those slightly more existential
• 30:15
  Speaker 2: harms have become all the risks have become much greater.
• 30:19
  Speaker 1: Yeah, I mean, I think that the existential risks are
• 30:23
  Speaker 1: still there. And when I talked to Demissabis, as I
• 30:27
  Speaker 1: did again this week, he still worries about that stuff
• 30:30
  Speaker 1: both and by existential, what I really mean is two things.
• 30:33
  Speaker 1: First is that bad guys get the technology and do
• 30:36
  Speaker 1: something really catastrophic with it, build a bioweapon, do an
• 30:39
  Speaker 1: enormous cyber attack which bloods a city by hacking a
• 30:42
  Speaker 1: dam or something. Second, the machine itself goes rogue and
• 30:46
  Speaker 1: starts attacking humans. And I always thought while I was
• 30:49
  Speaker 1: writing this book that that second thing was science fiction nonsense.
• 30:53
  Speaker 1: The machines will be more intelligent than humans, but they
• 30:57
  Speaker 1: don't have a motive to attack humans because not evolved
• 31:01
  Speaker 1: to survive. Unlike humans, we're always watching out for the
• 31:04
  Speaker 1: lion on the savannah because we want to pass on
• 31:06
  Speaker 1: our DNA. Machines don't have DNA. But then I had
• 31:09
  Speaker 1: this rather disturbing conversation with the computer scientist Jeffrey Hinton.
• 31:13
  Speaker 1: I went up to Toronto and sat in his kitchen
• 31:15
  Speaker 1: for two hours, and he said to me, look, supposing
• 31:17
  Speaker 1: you have an AI which is very powerful, but you're
• 31:20
  Speaker 1: worried that the enemy AI will attack it. What are
• 31:22
  Speaker 1: you going to do? You're going to tell your AI
• 31:24
  Speaker 1: to defend itself if you see the attack coming, counterattack.
• 31:27
  Speaker 1: Make sure you don't get killed. You've just given it
• 31:30
  Speaker 1: a survival instinct. So now are you so comforted, Sebastian,
• 31:35
  Speaker 1: do you think that AI won't have a survival instinct.
• 31:38
  Speaker 1: I think this existential stuff is real. It can be
• 31:42
  Speaker 1: addressed by engineering what they call alignment into the models,
• 31:46
  Speaker 1: where you're basically giving instructions to the system not to
• 31:52
  Speaker 1: turn against humans. And that's an engineering challenge which I
• 31:55
  Speaker 1: think properly is solvable, but you need time to solve it.
• 31:59
  Speaker 1: And the thing about alignment is that what we really
• 32:02
  Speaker 1: care about is aligning not necessarily today's model, but the
• 32:05
  Speaker 1: the tougher, more dangerous, more powerful one which we might
• 32:08
  Speaker 1: have in six months or a year. And you can't
• 32:10
  Speaker 1: invent the alignment of the future model until you've got
• 32:14
  Speaker 1: it in the lab and now you're trying to align it.
• 32:17
  Speaker 1: So the faster that progress goes, the less likely we
• 32:20
  Speaker 1: are to do alignment. And that's another reason why we
• 32:23
  Speaker 1: need government controls.
• 32:24
  Speaker 2: The government we have currently, as you pointed out, the
• 32:28
  Speaker 2: US administration, and obviously it's the US government that's going
• 32:30
  Speaker 2: to have the biggest influence by definition on this at
• 32:33
  Speaker 2: the moment, is one that is not minded towards safety,
• 32:38
  Speaker 2: is very focused on the arms race aspect with China.
• 32:41
  Speaker 2: Given what you've been saying about the pace of change,
• 32:43
  Speaker 2: the fact that we already have potentially enormously damaging instruments
• 32:48
  Speaker 2: that could be tomorrow in the wrong people's hands, is
• 32:53
  Speaker 2: it going to really matter a lot to all of
• 32:55
  Speaker 2: us that this administration, with those instincts was in power
• 32:58
  Speaker 2: for at least the next two Yeah.
• 33:01
  Speaker 1: I think there's a real danger that will turn out
• 33:03
  Speaker 1: to be decisive on the upside. It's interesting that with Mythos,
• 33:07
  Speaker 1: the Treasury Secretory, Scott Vesant, immediately worried that this was
• 33:12
  Speaker 1: going to be catastrophic for the stability of the banking
• 33:14
  Speaker 1: system because bank accounts would be hacked, and he called
• 33:17
  Speaker 1: in the heads of the big banks and told them
• 33:19
  Speaker 1: to take it seriously. And he apparently inside the administration
• 33:24
  Speaker 1: favored a nationalization of Mythos because he thought it was
• 33:29
  Speaker 1: so scary. So I think it's possible that as these
• 33:32
  Speaker 1: models become more threatening, the Trump administration, which is flexible,
• 33:37
  Speaker 1: one could say, might do the kind of u turn.
• 33:41
  Speaker 1: It's done on the notion that you shouldn't intervene in
• 33:43
  Speaker 1: other countries and start wars. It clearly forgot that principle,
• 33:47
  Speaker 1: and so perhaps it will change on AI as well.
• 33:49
  Speaker 2: Even the instincts that we're seeing on that seem to
• 33:52
  Speaker 2: be much more about protecting American banks and maybe UK banks.
• 33:55
  Speaker 2: But to your point about Anthropic really controlling who has this,
• 33:59
  Speaker 2: is it really realistic that you can be just picking
• 34:02
  Speaker 2: and choosing countries that get this defense.
• 34:04
  Speaker 1: I didn't think it's realistic for Anthropic to continue to
• 34:06
  Speaker 1: do this. Therefore, I think we have to hope that
• 34:11
  Speaker 1: the sheer evidence of the disruptive power of the models
• 34:14
  Speaker 1: eventually gets the US government to change his position, and
• 34:18
  Speaker 1: then we could see a sort of norm cascade where
• 34:22
  Speaker 1: other countries also change.
• 34:24
  Speaker 2: Well, you say in your book that you start off
• 34:25
  Speaker 2: as an optimist on these things, and I hope that
• 34:27
  Speaker 2: you've still managed to sustain a little bit of optimism.
• 34:29
  Speaker 2: But Sebastian, thank you so much.
• 34:31
  Speaker 1: So nice to be with you.
• 34:31
  Speaker 2: Stephanie, thanks for listening to Trumponomics from Bloomberg. It was
• 34:42
  Speaker 2: hosted by me Stephanie Flanders. I was joined by Sebastian Mallaby,
• 34:46
  Speaker 2: Senior Fellow at the Council and Foreign Relations and author
• 34:49
  Speaker 2: of The Infinity Machine. DeNisi Sibi's Deep Mind and the
• 34:53
  Speaker 2: Quest for super Intelligence. Trumponomics was produced by Samasai and
• 34:57
  Speaker 2: Moses and with help from Amy Keane, and sound design
• 35:01
  Speaker 2: was by Blake Maples and Kelly Garrett. To help others
• 35:05
  Speaker 2: find this show and enjoy it, please rate and review
• 35:08
  Speaker 2: it highly wherever you listen.