Latest / Uncomfortable Conversations with Josh Szeps / A.I., Iran, & Automated War with Prof. Toby Walsh

Transcript

A.I., Iran, & Automated War with Prof. Toby Walsh · Uncomfortable Conversations with Josh Szeps

• 0:00
  Get a humans, welcome to the safe space for dangerous ideas and a lot of dangerous ideas
• 0:09
  floating around at the moment, Iran, being the biggest of them, a lot of people asking
• 0:15
  me what I think about the American and Israeli strike on Iran.
• 0:20
  Most of the coverage and analysis seems to be, look, he was a bad guy, Khamenei and the
• 0:26
  regime was terrible, but this is illegal and so and we don't know what is going to come
• 0:31
  with it.
• 0:33
  And I think I just broadly agree with that rather boring and banal interpretation, unfortunately.
• 0:38
  If the Iranian people are a fabulous culture and an amazing people and incredibly industrious
• 0:46
  and innovative and culturally self aware and sophisticated and so I've long felt that
• 0:51
  they've been imprisoned for the past nearly four decades under crazy, you know, dictatorial
• 0:58
  theocratic idiots, jihadist death cultists who just want the worst for Western civilization
• 1:05
  and for Jews and women and the list goes on.
• 1:09
  So on a narrow level was a little part of me happy when Khamenei was reported dead
• 1:15
  absolutely.
• 1:17
  And I sort of admire the gumption of finally trying to do something about the situation,
• 1:22
  but it's clear that there is no rationale either under international law or under American
• 1:26
  law for doing this around pose no threat.
• 1:30
  In fact, the very fact that it was so unthreatening at the moment because it had been so successfully
• 1:34
  degraded in the strikes in the middle of last year against its nuclear facilities and
• 1:39
  so thoroughly degraded by a succession of incredible Israeli security operations like
• 1:43
  the exploding pages on Hezbollah in Lebanon and targeted assassinations in Iran.
• 1:50
  You know, that's why it made this such a low cost and achievable outcome for the Americans
• 1:55
  and the Israelis.
• 1:57
  So the very grounds on which one might want to base a rationale for a preemptive war,
• 2:02
  namely that your adversary is formidable under an imminent threat are the very reason why
• 2:08
  this undertaking was taken now, I mean, the fact that those conditions were not met.
• 2:15
  It's not just legalistic personality nonsense to care about international law and American
• 2:19
  law.
• 2:20
  These are the international law.
• 2:24
  Perhaps one can make an argument is easier to ignore since even the Iraq war was in
• 2:29
  contravention of international law.
• 2:31
  It didn't have security council approval.
• 2:35
  Perhaps it's unreasonable to expect Russia and China to sign off on every international
• 2:40
  adventure.
• 2:42
  But Americans should insist on abiding by American law, you know, the Constitution was very
• 2:48
  good at breaking up the rights, the abilities, the capabilities of a monarch into a bunch
• 2:55
  of different buckets and giving some of those capabilities to one of three branches of
• 3:01
  government, the judiciary and the legislative branch and the executive.
• 3:05
  And when it came to war to war, they thought, well, why don't we break it up into the
• 3:08
  declaring of war and then the running of a war?
• 3:11
  So we'll split it apart.
• 3:13
  We'll make sure that no individual branch has all of the powers of any particularly important
• 3:17
  thing and we'll give the legislative branch the ability to declare a war and then once
• 3:21
  a war is underway, it's the executive branch that figures out how to run the war, which
• 3:24
  is the basic way that the American system is supposed to work anyway.
• 3:27
  It's supposed to be Congress, the House of Representatives and the Senate together that
• 3:32
  make the decisions about, you know, they pass the laws about what's going to happen and
• 3:36
  then the executive branch, the president and whoever he picks are the people who then
• 3:39
  do whatever Congress has instructed them to do and they do the running of it.
• 3:43
  They do the operational side of things, whereas the big picture stuff is all taken care
• 3:46
  of by the legislative branch.
• 3:47
  I mean, that just doesn't exist anymore and it should worry Americans that it doesn't
• 3:52
  exist anymore because you are supposed to live in a republic.
• 3:55
  You're not supposed to live in an elected monarchy where for four years, one person gets
• 4:00
  to do whatever they want, including things that are explicitly allocated to the other
• 4:05
  branches and the mere fact that one of the branches is so dickless that they're unwilling
• 4:10
  to stand up for themselves and insist on that it's their right to declare war and nobody
• 4:15
  else's does not mean does not make it okay.
• 4:21
  So Trump should have gone to Congress and he should have gotten approval, congressional
• 4:25
  approval, at least George W. Bush did that before the Iraq war, even though he snubbed
• 4:31
  his nose at the fact that the UN Security Council voted against it.
• 4:36
  And so I worry about the legal precedence in the United States of essentially one person
• 4:43
  being able to take out any other country just because that person doesn't like it without
• 4:48
  having to provide a cogent or coherent rationale, not just to the international community, but
• 4:52
  even to the elected representatives of the American people.
• 4:56
  And the other thing that worries me a bit about it is just the uncertainty and the
• 5:00
  riskiness of it.
• 5:01
  I admire the Hutzpah, I admire the sense that nobody has been able to solve this situation
• 5:06
  so we're just going to roll the dice and take out chances and see what happens.
• 5:10
  And if it becomes possible for the Iranian people to rise up and if American and Israeli
• 5:15
  firepower is able to neuter the Iranian Revolutionary Guard to such an extent that those
• 5:20
  uprisings are successful and there is a genuine people's revolution, then it will all
• 5:24
  have been worth it.
• 5:25
  And that is fabulous.
• 5:28
  But that is not currently, I would say, the most likely outcome and there are a bunch
• 5:34
  of other possibilities on the table that gave pause to Trump's predecessors.
• 5:40
  So I don't think it's as simple as saying, look, Trump has managed to do what none of his
• 5:45
  predecessors managed to pull off, which is, you know, destabilize the Iranian regime,
• 5:50
  this terrible bloodthirsty, pariah state of religious lunatics.
• 5:56
  You know, the reason why it's not like his predecessors didn't know how, they knew
• 6:00
  that they could authorize this kind of military action and it's not that they didn't have
• 6:04
  the balls.
• 6:05
  It's that they thought that the possible ramifications were bad enough that the status
• 6:11
  quo was preferable to rolling those dice.
• 6:14
  The calculus is a bit different for Israel because it sees Iran's nuclear program and in
• 6:19
  fact, the the theocracy in Iran itself as an existential threat to Israel.
• 6:23
  And they think that certain factions in Iran, the ruling faction up until this week would
• 6:29
  essentially be suicidal in wanting to take out Israel.
• 6:33
  We don't know whether that's Israeli paranoia or not, but it certainly puts a different
• 6:38
  slant on things than they have in the United States.
• 6:41
  So I just feel watch this space.
• 6:43
  I'm both impressed and also worried about the whole thing.
• 6:48
  The only thing I can say about the media coverage is it is so tiresome to me to see reports
• 6:55
  about like people's flights being disrupted because they were flying Emirates Airlines
• 6:59
  and then they you know, they couldn't they weren't going to be able to their flight was canceled
• 7:03
  through Dubai and there was another piece in the in the the paper.
• 7:08
  And these are like this like the second most important piece in the online news and then
• 7:14
  like the fourth most important the fourth biggest piece about Iran was there's a Sydney
• 7:18
  woman whose mother is an Iranian in an Iranian prison and she's worried about her mother's
• 7:23
  welfare in this Iranian prison.
• 7:26
  One of the reasons why I got out of the legacy media was this endless relentless attempt
• 7:30
  to sort of find like these banal parochial human interest stories and everything.
• 7:36
  I remember in 2017 when I was sort of having a job interview with some of the producers
• 7:40
  at the public broadcaster and they said, well, so what would you want to?
• 7:44
  What would be a give us an example of a story that you would want to pursue now?
• 7:48
  This was 2017 and my first word that I said was Russia, which was not a fun thing to talk
• 7:54
  about in 2017 or not even on people's radar very much in 2017 because the Crimean war had
• 7:59
  happened and the attack on Georgia had happened, but I just knew that Russia was going to be
• 8:04
  a big problem.
• 8:05
  I didn't know it would unfold exactly as Ukraine has unfolded, but I knew that it was going
• 8:10
  to be a problem.
• 8:11
  I think we need to be paying more attention to Russia.
• 8:13
  I don't think I think things are worse than we think they are.
• 8:18
  The lead producer, there was an awkward silence as everyone thought this guy doesn't have
• 8:22
  a clue what he's talking about.
• 8:23
  He doesn't understand the way that you deliver news to the public, to the little man, to
• 8:28
  the little man who doesn't care about Russia.
• 8:30
  She said, so, imagine I'm a housewife and I'm driving along in Sydney and I turn on
• 8:38
  the radio, how does what I'm hearing impact my life as, say, a mother driving along in
• 8:48
  Russia hour in Sydney?
• 8:52
  And I thought, well, if that's the prism that you're going to put all information through,
• 8:57
  how does it affect the life of a woman driving along in Sydney?
• 9:04
  And most of the most important things that we need to be talking about would be off the
• 9:07
  table because they might not have an immediate impact on her afternoon.
• 9:13
  But I think there's something condescending about thinking that that woman wouldn't be
• 9:16
  interested in knowing about a country that sure enough five years later launched the
• 9:25
  first war in Europe since World War II.
• 9:29
  I think she might have liked to have, for that to have been on her radar and I think it
• 9:34
  might have been the responsibility of the public broadcaster to put it on her radar in
• 9:37
  2017.
• 9:38
  Needless to say, I didn't do stories about Russia because I was constantly encouraged
• 9:42
  to do stories that would have a real human bread and butter impact and be relatable to
• 9:47
  people.
• 9:48
  Well, the success of this show and shows like it show that people like you who consume
• 9:53
  this show are more interested in things bigger than just what is going to affect your afternoon.
• 9:58
  Otherwise, we would just do the weather all day.
• 10:01
  So my sense now is it's time for thinking about the sort of big picture stuff, the rules
• 10:07
  of war, the international rules, legal rules, whether or not we're likely to get anywhere
• 10:16
  with an administration whose idea of regime change in Venezuela was just to replace one
• 10:20
  person with that person's deputy and then claim success does not give me huge hope that
• 10:27
  what will eventually aid in Iran is going to be much more than a cosmetic change.
• 10:31
  But I could be wrong.
• 10:32
  So who knows, I'd like to feel this is a moment like the fall of the Berlin Wall.
• 10:36
  But as I read one commentator say, the successor to Hamanay is unlikely to be a Nelson Mandela.
• 10:44
  It's more likely to be a Gorbachev.
• 10:46
  But even if it was a Gorbachev, I would take it.
• 10:49
  Today we're talking about something related to Iran, but not exactly following, oh, I should
• 10:56
  mention, if you want the sort of moment-by-moment strategic breakdown of where we are in Iran,
• 11:03
  then on Wednesday evening, if you're listening to this in the few days after it posts Wednesday
• 11:07
  evening at 8 p.m., East and time in the United States, that's Thursday morning in Australia.
• 11:14
  We'll be speaking with David Frum, who is currently at the Atlantic.
• 11:17
  He was George W. Bush's speech writer, one of the architects of the Iraq War back in
• 11:21
  the day, and as an anti-Trump Republican, he knows a lot about the potential ramifications
• 11:28
  of Middle Eastern conflicts and the way that such engagements can go sideways.
• 11:33
  So you'll be able to tune in live for that.
• 11:35
  If you're a primo subscriber, you'll be able to watch it after the fact or you'll be
• 11:37
  able to listen to it on the podcast feed on Friday, Australian time, Thursday in the US.
• 11:44
  Today's conversation is about the way that artificial intelligence is evolving in warfare
• 11:48
  and the sort of dystopian and asymmetric way.
• 11:52
  It could impact all of our lives in terms of the conflicts that we find ourselves
• 11:58
  embroiled in and exactly what they look like and the civilian casualties therein as well.
• 12:03
  Toby Walsh is probably Australia's leading artificial intelligence expert.
• 12:08
  He actually gave a National Press Club speech last week, which is sort of our annual
• 12:12
  speech-speechifying honor, saying that regulation of artificial intelligence is, you know,
• 12:18
  risks leading to supercharged AI, which will inflict supercharged harms on all of us.
• 12:26
  He's the chief scientist at the AI Institute at the University of New South Wales.
• 12:29
  He's a professor of AI at UNSW. He has a degree in theoretical physics and mathematics
• 12:35
  from Cambridge University and a PhD in artificial intelligence.
• 12:39
  He was the editor-in-chief of the Journal of Artificial Intelligence Research
• 12:44
  and of AI Communications. So this is his bag and he was kind enough to stop by
• 12:50
  the uncomfortable conversation studios literally on his way to the airport to fly to Geneva
• 12:56
  to take part in a United Nations conversation about artificial intelligence in war.
• 13:03
  Now, the reason why I wanted to talk about Toby this week was because the biggest story of
• 13:08
  the week had it not been for the invasion of Iran would be, it's funny to say, invasions,
• 13:13
  since there are no troops on the ground, but you can't really call it a war because the war
• 13:16
  wasn't declared either. It's so funny how we live in this hybrid shredding a state of war and
• 13:22
  not war at the same time. But the reason why we wanted to talk to him this week was because this
• 13:26
  huge story happened a few days ago where Anthropic, which is probably the producer of the best AI
• 13:33
  right now, Claude is their consumer facing product, but they have had a long standing contract
• 13:39
  with the Pentagon. They refused to renew their contract with the Pentagon unless the Pentagon
• 13:44
  was willing to accept a couple of limits on what it was allowed to use artificial intelligence
• 13:49
  for. It wasn't allowed to create autonomous killing machines and it wasn't allowed to spy on
• 13:53
  Americans. The Pentagon refused. Anthropic called their bluff and said, well, you can't have it
• 13:58
  then and Trump counter called the bluff and said, all right, everything's off. Anthropic has to
• 14:02
  be removed from the no collaboration with the Pentagon, no collaboration with any American
• 14:06
  agencies at all. And then most recently Anthropics competitor, open AI, jumped in and replaced,
• 14:12
  is going to supposedly replace them and signed a contract with the Pentagon instead.
• 14:16
  Massive massive business, massive reverberations around Silicon Valley, massive reverberations
• 14:21
  around the way that military strategists think about insinuating and embedding AI into their
• 14:27
  systems. Toby is a delightful person to talk to. I hope that you'll also enjoy the conversation
• 14:32
  with David from about Iran on Wednesday or Thursday. Please enjoy. As much as I did, the one the only
• 14:37
  Toby Walsh. It's weird that this is not the biggest story of the week. But in a weird way,
• 14:52
  tangentially, it's sort of almost is. Let's start with Iran, because why not? To what extent
• 14:59
  is technology responsible for what we're seeing in Iran? I don't think you would see what we're
• 15:05
  seeing today without the advances that technology thinks like intelligence. The precision that you
• 15:11
  can attack now and the scale and speed, transformed by the technology. So the fact that you can actually
• 15:18
  go in and take out the leader. Do we know how that was done? I assumed it was human. I assumed it
• 15:24
  was old fashioned. Massade spy was in the Iranian Revolutionary Guard Corps and new where he was.
• 15:30
  But that might just be because I watched too many Cold War era. I think you're right. It was
• 15:34
  probably embedded spies, Israeli embedded spies in the regime who fed back information very quickly
• 15:43
  to where he was. And they would have probably had to have confirmation from independent sources
• 15:50
  that that was where he was. I also had some rumor that the precision of the strike. Right, I see.
• 15:55
  You mean just the actual technical precision of being able to aim the bomb right there?
• 16:00
  Although I suspect the US doesn't have many tomahawks left now.
• 16:04
  You think they've been using the same way with gay abandonment a little too much?
• 16:07
  Like half the reserve. Right. This is going to go on for any length of time.
• 16:14
  I mean it helps when your adversary doesn't have anything either. They don't have a credible
• 16:18
  defense. It's funny isn't that they've spent so much. I mean I heard an analyst talking about how
• 16:22
  Iran has really invested all of its defense budget in proxies who can create mischief throughout
• 16:28
  the region in the nuclear program and sort of these are the sort of big ticket things that don't
• 16:34
  actually help you protect the homeland but they don't have like missile launches. They might
• 16:41
  have 400 missile launches or something like that which every time you fire at the Americans
• 16:45
  or the Israelis take it out. So you better be better you better pick your 400 wisely and make sure
• 16:50
  they're not shot out of the sky. What do you make of it just politically and globally the moment
• 16:56
  that we're in? As usual Trump you wonder what is his endgame? Does he actually thought through how
• 17:02
  is this going to end? I don't see staring this horn it's messed up. I mean they're saying it's
• 17:09
  regime change and the regime has changed because you've taken out the leader, the supreme leader but
• 17:15
  supposedly there's just hard liners are going to come behind. So I don't see the Iranian people
• 17:20
  rising up and taking back control. Yes I mean civilians are unarmed and the Revolutionary
• 17:27
  Guard Corps is very very very arm brutal and totally brutal. Yeah I know. I mean
• 17:34
  it's obviously one is pleased that someone is evil as that has gone but I didn't see that that being
• 17:40
  necessary changing the situation on the ground in a positive set so long term. I mean it's fantastic
• 17:46
  distraction for the president. Well it's funny isn't it when you think about what he didn't
• 17:51
  Venezuela which is just take out the bad guy but then let his deputy let the bad guys deputy just
• 17:56
  continue ruling in the same party and the same thing getting the same here again. Yeah so it's hard
• 18:00
  to know. The president does have a lot of follow through. That's not even one of his strengths. I
• 18:06
  wouldn't say that that's his top five. No because he hasn't got a big game plan. Attention to detail.
• 18:11
  All right it hasn't got a big game plan. No but I guess it's like whatever someone whispered in his
• 18:16
  ear last. I mean to be generous to him I do think that he just senses that if things haven't worked
• 18:22
  for almost 40 years then why don't you try the risky option that nobody else wanted to try and see
• 18:27
  what happens because you might end up with someone you know the cleric who replaces Hamanay. If you
• 18:35
  shake the dice enough times eventually you do it exactly. That's right roll the dice it's a
• 18:40
  craption is a craps table. You may end up with someone who is still a dictator but is more
• 18:45
  amenable to the West all you may end up with someone the clerics may be answered by the IRGC
• 18:53
  who may and you may end up with a more fanatical more fundamentalist terrorist sympathizing
• 19:00
  character who just wants to invest all of their money in funding proxies to carry out spectacular
• 19:07
  mega-terrorism operations in western cities and you just don't know what you which one you're
• 19:11
  going to get or a decade of civil war. Yeah I think it's interesting there to see that the allies
• 19:18
  western allies have not come behind this except for one exception the only the only country
• 19:25
  that the country that could. The country that could. Exactly any country that has really
• 19:32
  backed it. The others have been conspicuous by their silence and by saying yes he was an evil
• 19:39
  guy but you know we want to we have they haven't seen it hasn't been a full-throwed
• 19:44
  system. No and did you see that? They haven't used UK bases apparently for example. Right
• 19:49
  right and they a journalist asked I think with the foreign minister or the defense minister
• 19:55
  here in Australia after the government said oh this is great is a great idea we love this attack
• 20:00
  on Iran they said did you do you do you do you do to provide any tactical support in any way or was
• 20:04
  there any intelligence or anything that was gathered on Australian soil and they said no no
• 20:09
  we weren't told or asked but we want to get the credit from the American administration anyway
• 20:15
  for at least having supported it. We have turned up to every conflict since the second world war
• 20:20
  to support the Americans this is true. So artificial intelligence in warfare the thing that made me
• 20:25
  reach out to you over the weekend was this I was fascinated by these internet and battle between
• 20:31
  anthropic and the Pentagon and Trump and Palantir and I didn't quite know how they all linked together
• 20:39
  and watch it is that war makers want to be able to use AI for that even AI bosses themselves who
• 20:48
  are not known for their heedfulness when it comes to experimentation in new technologies.
• 20:54
  I'm really about. Thropic their credit they do try and advertise themselves as the most careful
• 21:00
  the most conscientious safe they do AI company around indeed you know it was some refugees from
• 21:06
  open AI who left because they didn't feel that open AI was being safe enough who set up anthropic.
• 21:12
  Yes I mean but they did also have to didn't they originally say that if they felt like AI was
• 21:18
  getting out of control that they would shut it down and then last year they said actually it
• 21:24
  is getting out of control but we're not going to shut it down because if we shut it down then
• 21:27
  everybody else will go full steam ahead anyway so we're better to do it than everybody else.
• 21:31
  Even more current than that indeed you know one cynical interpretation of the fact that the
• 21:36
  anthropic have stood up against the Pentagon so vigorously it lost a 200 million dollar contract
• 21:43
  with the Pentagon is as a distraction from what they announced just a week or two back.
• 21:51
  Which is that they're going to backtrack on their firm safety promise which is they would not
• 21:56
  release any leading AI models if they couldn't guarantee that they were going to be safe and
• 22:02
  they've watered that down say well we'll only guarantee they're safe if we're at the frontier with
• 22:09
  the leading maker of models if we're behind if we're playing catch up then we don't care.
• 22:14
  Right because there are worse people out there anyway because they're already worse things out there.
• 22:19
  Yeah oh god okay so what were the dynamics of the actual disagreement between anthropic and
• 22:26
  the Pentagon what did the Pentagon want to be able to do.
• 22:28
  Was the anthropic had actually got in the bed with the DOD very quickly they were the first
• 22:35
  language model to be proved to be run on secure hardware and there I think many people think
• 22:45
  they've probably got slightly better model than most people and suddenly by all accounts within
• 22:51
  the Pentagon they prefer the preferred model they like using.
• 22:55
  At this point I know a lot of creatives who think that Claude is superior to anything which is
• 22:59
  the anthropic user consumer facing LLM.
• 23:04
  And as far as we know the Venezuela military action was planned with the help of anthropic.
• 23:10
  Cool.
• 23:10
  Really?
• 23:10
  Yes.
• 23:11
  Right I guess it must have been.
• 23:13
  Do we know how?
• 23:16
  Respect to some of the high level planning thinking you more gaming,
• 23:19
  strategizing planning how to coordinate the activities.
• 23:24
  It's quite a good tool for doing those sorts of things.
• 23:26
  Yes.
• 23:27
  It also seemed to plan out some of Trump's tariffs if you looked at the countries that were targeted
• 23:32
  for the tariffs you remember that last year when it was like there were these bizarre people
• 23:36
  were like why would you get those tariffs on those countries and then someone plugged into
• 23:40
  I think it was ChatGPT that the principle formula for tariffs.
• 23:45
  Yeah it was like the difference it was the trade imbalance or something divided by something
• 23:49
  or other and then you got these numbers though.
• 23:51
  Yes and indeed if you've applied that formula to Australia we should have had a negative tariff
• 23:56
  Yeah exactly.
• 23:57
  We got a trade surplus.
• 23:59
  We didn't get a trade negative tariff.
• 24:02
  So anyway the Pentagon's using AI to okay so for war games planning like what's the best way to
• 24:10
  get this guy and whatever else but at what point does anthropic stop saying this far but no further?
• 24:17
  They have a contract with the Department of Defense.
• 24:21
  It's not clear why they were renegotiating the terms of the contract you know.
• 24:26
  Cinnick would say Pete Hesketh was looking for some good publicity
• 24:31
  but anyway anthropic said there are two things that we will not permit our AI bottles to be used for.
• 24:38
  One was for mass domestic surveillance which of course is against the fourth amendment
• 24:44
  any rain and the second was for lethal autonomous weapons.
• 24:48
  As far as we know they're not actually using the language models for lethal autonomous weapons
• 24:54
  other types of AI would be used so not clear why that was such a contentious point in the sand
• 25:01
  but certainly you can imagine that I would be unsurprised if the security services aren't
• 25:06
  some of the three-letter agencies aren't using language models to help them shift through
• 25:11
  huge amounts of domestic surveillance and so on.
• 25:16
  I suppose from the Pentagon well both of those technically already governed by existing law
• 25:24
  so the fourth amendment prohibits domestic surveillance.
• 25:27
  That's the unreasonable search and seizure and the US military have a
• 25:37
  directive three thousand and nine which requires appropriate human oversight of AI
• 25:43
  in using autonomous weapons systems.
• 25:45
  Oh right.
• 25:46
  When's that from?
• 25:47
  So that's the US military who themselves decided the first
• 25:53
  first bit of major AI weapon regulation themselves and it's not required
• 25:59
  but that was set up in 2012.
• 26:06
  It's been re-approved a various times most recently in 2023 and they revised it up again.
• 26:12
  Oh I mean you wouldn't have trusted a computer in 2012.
• 26:14
  You wouldn't have trusted like MS DOS in 2012 to figure out the difference between us.
• 26:23
  A terrorist and a shopkeeper but in 2026 you might.
• 26:32
  We've all seen AI make silly mistakes and that's one of the challenges here which is that
• 26:38
  it is not 100% reliable and if it makes mistakes who's going to be held accountable?
• 26:43
  Why wouldn't the same chain of command be held accountable that is supposedly held accountable
• 26:46
  when humans make mistakes?
• 26:47
  Well it is in the US military.
• 26:48
  It's according to this directive requires the humans to be held accountable and saying
• 26:53
  chain of command but the problem is if you minimize the role of humans in the decision-making loop
• 27:01
  and the problem is that decision-making is getting faster and faster
• 27:07
  then can they really be held accountable?
• 27:11
  Well they're not held accountable anyway.
• 27:14
  I mean like Abu Ghraib happens and some low level functionaries have the book thrown at them
• 27:21
  but the culture that permitted the abuse of detainees,
• 27:27
  chaining in Rumsfeld and Bush are fine and so are the generals.
• 27:31
  Oh that's interesting so ad for the defense training college here for military officials,
• 27:41
  military officers. They did a survey of young military officers and they were most concerned
• 27:47
  about the uptake of AI because they realized they were the ones that were going to be held accountable
• 27:52
  for the mistakes. Not the, as you say, not the generals.
• 27:55
  Right, they're going to be held accountable and so they were the most concerned.
• 28:00
  But suppose you had a system where there was accountability and that if there is systemic issue
• 28:06
  like waterboarding of prisoners or something like that one of the obvious examples
• 28:11
  that we can think of from recent history in which militaries have done things that were
• 28:14
  in contravention of human rights and probably international law.
• 28:20
  If you assume that you had a system where there was accountability all the way up to the authorizing
• 28:25
  figure whether that's a general or the Minister of Defense or the Secretary of Defense,
• 28:30
  why would it make a difference whether the person on the ground who was in,
• 28:35
  let's pivot to say a drone who was pushing a button for a drone to blow up a
• 28:40
  an Iranian military facility whether that was a person making the decision or whether that was an AI
• 28:48
  making the decision as long as accountability for the person giving the command that authorised
• 28:53
  that underling to make the call is ultimately accountable.
• 28:58
  Because I fear you'll be throwing the person under the bus because you know two of the fundamental
• 29:03
  princes principles of international humanitarian law that we under which we fight war under which
• 29:09
  you can prosecute people at the you know the Hague and so on are the principle of distinction
• 29:15
  that you can't target civilians you have to target valid military targets and if we're handing
• 29:22
  over those decisions can we be sure that the machine is making the right decision and then the
• 29:27
  second principle one of proportionality which is that you're only allowed to to use as much force
• 29:33
  as proportional to the threat you face and that's a that's a very subtle thing and it's not clear
• 29:39
  that machines have the capability to make those sort of subtle cause it's only when you look at
• 29:46
  you know what was happening in Gaza you look at you know AI was being used to make the targeting
• 29:50
  decisions systems like Lavender and West Adi that were that were processing vast amounts of military
• 29:57
  intelligence, social media and all the information they were getting back into just to where
• 30:02
  where the hammer targets were going to be you saw the collateral damage you know it was 10 50 to
• 30:08
  one that you're thinking well that doesn't seem to me like that's representing proportional to
• 30:14
  the threat that you are allowed to you know kill 50 civilians for the one high target.
• 30:20
  Yeah but I sometimes feel like I mean speaking of people being scapegoats I sometimes feel like
• 30:23
  that the the systems are being used as scapegoats as well because that disproportionality in Gaza
• 30:30
  was taking place in a context of a government in Israel that had absolutely no care apparently for
• 30:38
  proportionality of their conflicts right I mean so that when we were on instruction just as a
• 30:44
  human would have been maybe not on overt instruction but was operating in a context in which it was
• 30:51
  clear to everybody who was looking at it that the position of the Israeli government was
• 30:56
  we're sick of this shit we don't get a fuck about any of them we're going to do whatever we have to
• 31:00
  do because they've got out people and with no one's going to treat us like this like there was that
• 31:04
  obvious revenge kind of you know knee jerk hostile that they weren't they weren't taking time to think about
• 31:12
  okay well maybe we could get the women and children out and we could put them in the Israeli sign
• 31:16
  eye and we could you know let's love the tunnels and just target that it was like we're going to do
• 31:21
  what has to be done and we're not going to be good guys about this so then of course the AI
• 31:26
  that you deploy in that context behaves that way but I would put it to you that a human a human
• 31:32
  Israeli that Netanyahu would have done that as well he would the difference is that it then changes
• 31:42
  the cadence of battle so that they were spitting out targets far quicker than the human
• 31:47
  allies could ever spit out the targets right so it becomes a rattatat machine gun of sort of AI
• 31:54
  targets yeah I mean I remember when the numbers start when you started see the the data coming out
• 31:59
  I was from saying as far we know we don't know that much about these systems but either they're
• 32:04
  inaccurate or the idea of do not care I think the reality is it was both right maybe yeah I suppose
• 32:14
  we don't know if that would let's path the inaccuracy then let's assume that we know that it's
• 32:19
  more accurate right but the other thing that you know behind that is it's changing the character of
• 32:23
  war right in a way that that is you know as we've seen technologies in the past change the character
• 32:29
  war in a way that makes war possibly a more a more deadly a more difficult proposition
• 32:40
  right in one way more deadly and difficult well I mean you see it in Ukraine you see that you know
• 32:49
  it's some way some strange mixture between the first world war and some twenty second century war
• 32:56
  right it's some you know they're bogged down I mean in that sense the technology has been a good
• 33:01
  thing the technology has allowed the Ukraine's to to resist and oppress you know and attack those
• 33:08
  ten times their size has got ten times the economic might they've been able to hold them back
• 33:15
  by embracing the technologies but it's turned it into a terrible war of attrition where any
• 33:21
  humans in daylight are immediately attacked by the drones I remember seeing do you remember that
• 33:28
  moment a few years ago in the Ukraine war when Ukraine managed to put a bunch of killer drones
• 33:35
  into a a lorry a semi-trailer truck that was in Russia you know a thousand kilometers from
• 33:43
  Ukraine it was miles away but it was near it was going to delivering some kind of payload to a
• 33:49
  military base if I recall correctly and there were fighter jets and migs and things on the base and
• 33:56
  when and unbeknownst to the truck driver when the when the the lorry opened up the back of the semi-trailer
• 34:05
  erupted with a whole bunch of drones and the drones went and blew up the the Russian jets
• 34:12
  yeah on the on the the airfield and everyone's tired which is why you need AI and so this is
• 34:18
  incredible no this Ukrainian drones yes it was a targeted thing you know we thought well this is
• 34:23
  great this is like the little guy suddenly being able to it's the democratization of war in
• 34:28
  effect because now the little guy can use drones to take out these hundreds of millions of dollars
• 34:33
  worth of military heart military hardware and I read a piece by an American military
• 34:38
  analyst saying don't celebrate too quickly because we are the country that has the best
• 34:43
  most expensive military hardware and someone else is going to use exactly the same tactic to use
• 34:48
  drones against us you know in you we don't actually want a democratization of warfare if we're
• 34:53
  the United States we want old-fashioned brute force to be the dominant thing because that's what we
• 35:00
  have that's what America's good at but America doesn't need to be nimble America just needs to be
• 35:05
  you know America wants a world in which being having a massive big fist to punch with works I agree
• 35:13
  I mean there's two things to learn from that that incident which one of which is that we were
• 35:18
  thousands of kilometers deep into Russian territory which is why increasingly you want the drones
• 35:23
  to be have autonomous capabilities because you you can't control them from a thousand kilometers
• 35:28
  a way they want to be able to make their own decisions and target themselves so that's first
• 35:33
  what's why there is an incentive military incentive to make this system of increasing autonomous
• 35:38
  and second is the realization that one you just mentioned which is we are going to be on the
• 35:42
  receiving end of these technologies they're not going to be it's not like buying F-35s or nuclear
• 35:47
  submarines these are cheap bits of kit the intelligence comes from the software which is going to be
• 35:54
  is not going to be expensive to obtain or steal from someone else and so we will have to defend
• 35:59
  ourselves against these capabilities so it is going to be terribly destabilizing to the current
• 36:07
  balance of power it won't be just you know the US that can send in the aircraft carriers and
• 36:15
  the Sprite drones yeah Iran was becoming a major drone power Turkey has become a major drone
• 36:23
  power the latest ship that the Turkish Navy launched was a was a drone carrier really yes
• 36:29
  who needs an aircraft carrier exactly when you're going to have a drone carrier wow
• 36:34
  you can project your force with a drone carrier amazing in fact you mentioned it it's interesting
• 36:39
  how how did Turkey become a major drone superpower it's they they provide you know some of many of
• 36:46
  the drones in Ukraine are Turkish drones now possibly to both sides even yeah so weapons to both sides
• 36:55
  but they so they tried to buy drones from the US they tried to buy you know who the Turks did
• 37:02
  the Turks did right and they were turned out on on human rights and various other you know we're
• 37:07
  don't trust you too well yeah so they said okay we'll build them ourselves and the
• 37:12
  Sun in law of Erdogan the Turkish President had just done his em it does his PhD at MIT on
• 37:19
  aeronautics wow and he's now the technical lead on the on developing drones for Turkey when I
• 37:28
  imagine if your Sun in law of Erdogan you know resources not a problem no exactly it was a
• 37:33
  open for you foreign university students and you I didn't trust them now we've got that's exactly
• 37:38
  why we go back to the home countries and create drone armies you mentioned Iranian drones I was
• 37:43
  reading that the a lot of the strikes that are taking place right now around the throughout the golf
• 37:47
  states are from these Iranian drones actually that these are yeah like Bahrain was here to buy
• 37:57
  Airport and a fancy hotel in Abu Dhabi or some of the famos that's right yeah the famos in Abu
• 38:04
  Dhabi was struck by these Iranian drones which you may know more than I do about this but the
• 38:11
  report I was saying that these Iranian drones were good enough in their day that they that the
• 38:17
  Americans reverse engineered them actually and America part of America's drone program was a
• 38:24
  reverse engineering of these Iranian drones and re-batching them with a different name and the
• 38:28
  Iranians were also selling them to Russia too so yes I mean this is not just about ballistic missiles
• 38:35
  being fired is it is it is ready that drones yeah as with those drone attacks deep into Russian
• 38:42
  territory increasingly those drones aren't radio controlled because if they're radio controlled
• 38:48
  they're easy to stop you can jam the radio right and if you're in the Ukraine if their radio control
• 38:54
  do you find where the signals of the radio signals are coming from and you send artillery down
• 38:58
  on the drone pilots they're the most hated people on the battlefield so by making them autonomous
• 39:03
  you make them a much more capable much more difficult weapon to defend against and by autonomous
• 39:08
  presumably they still need GPS no no so that I need satellite either well so they they do use GPS
• 39:15
  that's something a fantastic way of navigating but GPS is easy to deny yes that's where I was going
• 39:23
  so and they know that that's going to happen yeah so how do they know where to go just like your
• 39:27
  Tesla car does it uses cameras visual recognition of the landmarks underneath them oh so that
• 39:33
  really gives AI it's not just it's not just a navigation routine yeah I mean I mean
• 39:39
  because there's also some inertial navigation right so you know where you start you know where you
• 39:42
  have you know what right how far you're going but that's that's imprecise but they said they'd
• 39:50
  be using a lot of increasingly using visual computer vision to identify landmarks on the ground
• 39:56
  and when you said that you suspected that something as precise as the taking out of the supreme leader
• 40:01
  of Iran you know is only possible because of technological advancements yeah and I said I thought
• 40:06
  it was sort of human intelligence what's the intersection there what is your hunch about how well
• 40:10
  it would it would have been human human intelligence that gave away the coordinates of
• 40:15
  commune and they would have I'm sure they would have any done it when they've got it confirmed by
• 40:20
  two different sources but then to be able to offer the precision and speed of attack that particular
• 40:27
  coordinate would have used some you know autonomous guided systems of some form yeah but I thought
• 40:34
  we had those pre-AI like when we saw the skies igniting over Baghdad in 2003 yeah I mean that was
• 40:43
  the beginning of yeah those were precision yeah those were self-guided missiles as well weren't they
• 40:48
  they weren't just you have to you have to aim it in the correct direction and then hope that it
• 40:51
  gets there they would find their way through the yeah through the window and things like this yes
• 40:56
  right so what are the big differences that people are working on now I mean when we're talking about
• 41:00
  anthropic not wanting but I think the big difference is it's being democratized so as you say
• 41:05
  Iran has got these these weapons it I mean it used to be you had to spend a couple of million and
• 41:09
  buy yourself a Tomahawk missile yeah only only the USA Tomahawk missiles now everyone can you know
• 41:17
  buy a cheap drone and and get that sort of accuracy technologically speaking yes what are the big
• 41:26
  advances because when I think about you know the when when I first heard the term chat GPT you know
• 41:32
  which was probably 2022 maybe something like that and it was this novelty where you could say you
• 41:39
  know tell me you know say give me tell say Mary had a little lamb but in the style of an American
• 41:46
  revolutionary war soldier or something and it would you know it was a party trick and then it got
• 41:51
  better and better and better and only you know within the past six months has it become something
• 41:58
  that I interact with as if it was a sentient thing capable of yeah well I'm not saying it is but
• 42:04
  but the problem is we're easy forward yeah yeah and we increasingly we will think it sent it yes
• 42:12
  and people are you know this is what troubles me that people are starting to form what they
• 42:18
  feel is meaningful relationships with these sentient beings yeah I don't have a meaningful relationship
• 42:24
  to it but I do have a relationship that's akin to a relationship I might have to research
• 42:32
  assistant who I don't really like very much but you know I don't socialize with but you know
• 42:37
  they're just there it's not like one of my beloved in a course not like my producer or my you know
• 42:43
  someone who I actually have a so human relationship with it's you don't say please and thank you
• 42:48
  and worry about not me I mean I think it's nice I used to and then I just sort of fell off the
• 42:53
  bandwagon and I think I feel like if I'm being demanding I say please but if it's just a normal
• 42:57
  request then I do well I mean one of the problems there is if that if that flows over into our
• 43:06
  real lives yes so we become more demanding yes we do worry about you become more demanding of
• 43:10
  your real human existence because it's a thing it's a thing yeah no but do I mean do you not engage
• 43:19
  as if the AI was I only mean engages if it was sentient in the sense that is able to iterate
• 43:26
  and understand my commands as well as a human is when we're bouncing ideas back and forth
• 43:35
  usually and so I guess I just mean to say that that is so far from Mary had a little lamb in the
• 43:43
  style of the American Revolutionary War soldier that I try to transpose that onto the battlefield
• 43:48
  and think about well what is it that that armies and the Pentagon and you know the brightest people
• 43:54
  in the Pentagon are trying to mold these systems into doing and possibly succeeding in doing
• 44:04
  that is as different from the you know the night sky flashing up over Baghdad in 2003
• 44:10
  well there's my relationship to church they're starting to use them to help them war game and
• 44:15
  strategize and that that itself is troubling I mean there's been troubling research that's come
• 44:22
  out of King's College London recently where they they got all of the leading models to play war
• 44:28
  games and said we know what happens if you put them in some you know well developed military
• 44:33
  situations and 95% of the time they resorted to tactical nuclear weapons
• 44:42
  did you have to add the caveat that we don't want everyone to die in the world that wouldn't
• 44:46
  nuclear holocaust is a bad thing maybe they forgot that it's only tactical nuclear weapons
• 44:51
  sorry just half the time what could possibly go on half the time was all out nuclear war I think
• 44:56
  yeah 95% well when you start with a tactical nuclear weapon it is not not to for that to
• 45:00
  evolve but not to evolve in some sense I you know it's like I'm unsurprised right these are trained
• 45:06
  on the internet in the internet they were mirror of what they're trained on and they're also
• 45:12
  trained on every military strategy book that's ever been read yes but they're trained on a half
• 45:16
  the internet and people are pretty aggressive and confrontational on the internet so unsurprising
• 45:23
  they're not very you know they're not very diplomatic they don't back down and as any of you
• 45:30
  we've seen for ourselves when you you have a disagreement on the internet it quickly becomes you know
• 45:35
  yeah yeah but I don't think AI is Twitter I don't think AI has to
• 45:41
  be straight on Twitter in part yes in part yes but if I if you go to Twitter and you say I'm having
• 45:50
  a problem with my uncle beer cause my uncle is a big fan of the far right and I don't know how
• 45:58
  to communicate with someone who is that racist then Twitter will come back and say you're the
• 46:04
  fucking problem your uncle is you know is great you just lib tards need to die but if you go
• 46:10
  away I and say that it will give you useful information about human communication and about
• 46:16
  again they're trained to be rather sycophantic to be rather confirming and so they don't
• 46:23
  necessarily push back as humans do against you which is why you know I think there's troubling that
• 46:28
  we are trusting upon them is troubling that we are having relationships with them because relationships
• 46:33
  you know sometimes you're the person who's making your people have to push back against the
• 46:38
  M.I.D. asshole thread on the internet yeah it's like yeah you off and on yeah that's interesting
• 46:43
  I mean it does when you think about war games it feels that to me like a lot of the cases that
• 46:49
  you've made thus far in this conversation have to do with the claim that AI just isn't as good
• 46:54
  as we think it is that we shouldn't trust AI on warfare because it's not as sophisticated or
• 47:01
  nuanced as people are the wall gaming for example we shouldn't trust it on pushing the button
• 47:06
  to kill someone because it's going to make mistakes more than people would but I'm my resolution
• 47:12
  should go beyond that though yes because I'm changing the interest in the case even if it
• 47:16
  were better like even if it were more well I think at some point we should entertain the idea
• 47:22
  we better yes humans are terrible yes right we're we make terrible at making irrational decisions yeah
• 47:27
  let's suppose we can and we make we make vile decisions as well if you look at the history of
• 47:32
  warfare people have done obscene things we are apes we are primates so we know and so if so you can
• 47:38
  get let's suppose you have an AI that controls drones yes and fire that can do the what's going
• 47:43
  on in Iran right now or you know what was happening in Gaza and you can guarantee because there's
• 47:49
  been such a run so many virtual tests of this that it's 10x more accurate in minimizing human
• 47:57
  casualties than human beings are it's not perfect does still happen but you're going to get 10 times
• 48:02
  fewer less collateral damage if you let the drones do the killing yes then human operators do
• 48:08
  the killing humans just can't see what's in that building the way that drones can they can't make
• 48:12
  the inferences that drones can they don't have the scope the scale of information and the speed of
• 48:17
  judgment that the sorry the AI not the drone does then what's the objection well I think there are
• 48:23
  two two objections that the first one is these weapons will fall into the hands of our opponents
• 48:30
  and they'll maybe have no qualms about removing those safeguards so now we've got this incredibly
• 48:35
  deadly weapon without those moral safeguards that's the first one and the second is the way
• 48:41
  that it changes the character of war that it will make warfare a much quicker much deadlier
• 48:48
  activity and yeah we've seen that in the past you know we see that there's various things that we
• 48:53
  can invent we've invented that would have made warfare more deadly activity and we've decided to
• 49:00
  ban because of what chemical weapons biological weapons yeah even the very first one of the very
• 49:06
  first things that was banned dumb dumb bullets they were a more effective way of called a dumb
• 49:10
  dumb bullets they bullets that when they go into your body they they explode and cause huge
• 49:16
  trapping damage to your internal organs so they cause they're much much more destructive land of
• 49:23
  mines yeah we're trusty munitions I mean there's a there's a host of things where we decide
• 49:29
  actually know that's changed the character of war in a way that we find distasteful
• 49:33
  and we've decided therefore we should try and limit to some extent and I think the same will be
• 49:38
  because otherwise I could I could tell you what warfare will will be which is as soon as war starts
• 49:43
  all the humans are dead because we don't have the the precision the speed the accuracy that they
• 49:48
  will have and so humans will be any humans that happen to be around the battlefield will be dead
• 49:59
  right that's not a good outcome I know we fight war according to some rules which is strange
• 50:08
  which is what we do and you know to try and you know constrain you know the horrors of warfare
• 50:17
  but it would be incredibly horrible at that point as far as I can see hmm but
• 50:23
  then the it strikes me that the real problem is the asymmetry and the fact that it would fall
• 50:28
  into the hands of unscrupulous probably non-state actors because the reality is that and it would be
• 50:36
  if Putin wanted all the Ukrainians dead then he could have just gone in and he could have made it
• 50:43
  more like Grozny I mean he could have just gone around assassinating everybody from day one
• 50:48
  and he could have dropped a tactical nuke on apparently they've been trying to recess
• 50:53
  I mean I'm not saying since day what well Zolansky sure he's the leader but I just mean the civilian
• 50:59
  they could have been were appalled by the lack of concern for civilians in Ukraine and in Gaza
• 51:05
  we should also be appalled by the lack of concern for the Russian soldier certainly absolutely
• 51:10
  grinding congratulations soldiers someplace yes because they don't care yeah they really just
• 51:16
  don't care anymore than they care about Ukrainian lives but but this is still happening in a
• 51:21
  context in which there is at least some level of lip service to the idea that you don't intentionally
• 51:25
  target civilians the way the terrorist does exactly and so if he had if Putin or Netanyahu had
• 51:34
  AI that was capable of immediately killing everybody on the battlefield presumably they wouldn't
• 51:39
  use it because they haven't used the tools that they do have that would come close to that
• 51:46
  so isn't that considered actually just that people we were a terrorist would get it I mean isn't that
• 51:51
  but it's not just terrorists Josh we began the podcast by by talking about taking out of the
• 51:56
  supreme leader we're all they're doing exactly that they are taking out people individually
• 52:04
  I don't have a huge moral problem we're taking out the supreme leader I have a problem with
• 52:08
  international law I have a problem with you know of all many problems with the way that Trump
• 52:13
  conducts foreign policy but but we were talking about how the character of war is changing which
• 52:17
  is now we have technologies and precision to be able to do things like that isn't that good in a
• 52:22
  second world we didn't take a second world war you would have had to blow up all of Tehran to get
• 52:27
  the exactly get Hitler and going you would have to we weren't able we didn't have the technology
• 52:34
  to be able to say let's take out Hitler I mean there wasn't a assassination attempt from within
• 52:38
  Germany and Hitler but we didn't have the precision and also the intelligence as well to be
• 52:43
  able to say okay Hitler's here we can we can take him out tomorrow no you're winning me over to
• 52:48
  the opposite side of your position Toby right now I'm thinking it would have been great if we could
• 52:52
  just take it out what I'm persuading you is war is changing right now we have because of
• 52:58
  technology like I we have the precision to do these things now whether that's the right thing from
• 53:03
  a stability of the world perspective from a military perspective from a moral perspective it
• 53:11
  is another thing right but we now have technologies that change how we can fight war where we can
• 53:15
  say we can take out the leader and it won't be long yeah it won't be long before the US
• 53:20
  president is constantly under threat of being taken out themselves because there's no secret service
• 53:24
  that's going to be able to make them independent well fun enough when I started to get involved in
• 53:29
  the debate around it on the sweaters I thought this was going to be the the easy argument I thought
• 53:34
  politicians were quickly realized they are going to be the targets yeah therefore it's in their
• 53:38
  interest right to ensure these technologies are socially regulated because otherwise it's them yeah
• 53:43
  why have they why have they not got that memo well I think I think they might be started to get
• 53:48
  that memo now took them longer to realize I you know 10 years ago I realized that that was right
• 53:54
  to end up yeah I mean you know people are already afraid in the United States in the Republican
• 54:00
  party of standing up to Donald Trump because he'll shout at them on truth social and all of a
• 54:04
  sudden some you know maybe one of his crazy followers will drive halfway across the country to
• 54:10
  the lawmakers house and you know try to inflict violence on them well if you lived in a world in
• 54:16
  which everybody can get online a killer drone that can go and assassinate that lawmaker without
• 54:22
  them having drive halfway across the country and as was the case for Nancy Pelosi's unfortunate
• 54:26
  husband get smashed in the head with a hammer by a deranged you know lunatic if they could just press
• 54:31
  a button and a drone could do it for them then it would probably have the effect on political
• 54:37
  assassinations that guns have had on everyday killings in the United States or is it just makes it
• 54:42
  much much much more common because it's so much easier oh it probably start with someone's AI agent
• 54:47
  which misguidedly decides that what we wanted it was was to go onto the dark web and order a drone
• 54:53
  attack on right and they have to call bonus yes that's that's a more strange love and outcome perhaps
• 55:00
  I mean I suppose it comes to the point which is we're already seeing the breakdown of the rules
• 55:05
  based order but we're seeing technology which is going to come along and change what we can do
• 55:10
  and I think you know make the world again a slightly less safe place and more yes difficult
• 55:17
  place so I guess the argument that you probably hear a lot from democratic leaders is not dissimilar
• 55:24
  in its shape to the argument that anthropic makes about its other adversaries it's other competitors
• 55:31
  in the AI race which is if we don't keep developing the technology other people are going to and
• 55:36
  will be better than this is priceless exactly it's sort of you will end up in a terrible place
• 55:42
  that no one wants to be yeah no no one can trust the other person well exactly if we don't make
• 55:47
  these drones the Chinese and the Russians and the Iranians aren't going to stop so we'd better be
• 55:52
  as well equipped as they are yes some arms race it is but you know the analogy I like to use when
• 56:00
  people suggest if utility of avoiding this terrible outcome this terrible equilibrium outcome
• 56:08
  that we get into you say well what about chemical weapons because chemical weapons it's not a
• 56:12
  technology that's that's difficult to tell you can go to your hardware storm get the rudimentary
• 56:20
  basis of a very simple chemical weapon it's not knowledge that's you know it's high school chemistry
• 56:26
  graduate level chemistry you don't need a lot of technical sophistication you don't need a lot of
• 56:31
  resources and yet chemical weapons are not widely used despite the fact that it's you know it's a
• 56:37
  technology which could easily be misused by lots of bad actors largely because I think you know
• 56:44
  because we haven't actually regulated this out it's use and if people do and the occasionally they
• 56:48
  do in Syria or elsewhere use chemical weapons there's universal condemnation there's resolutions
• 56:56
  on the floor of the UN there's front-page articles in the New York Times it's a threat of a
• 57:00
  distant court in the Hague and most importantly arms companies aren't openly selling them because
• 57:05
  they don't want to end up on a UN blacklist because there's plenty of ways they can make money selling
• 57:10
  legitimate arms which they do why would they want to risk being sanctioned by the United Nations
• 57:16
  or someone for selling chemical weapons so chemical weapons are not openly sold and terrorist
• 57:22
  organizations are not known for having large research organizations to develop chemical weapons
• 57:27
  and means to deliver them so they buy what's readily available on the black markets of the world
• 57:33
  and so I think we know we could hope for the same with autonomous weapons we could hope at the same
• 57:39
  which is it's not going to be a technology that we can constrain but if we decide collectively
• 57:45
  as a planet that that's morally reprugnant and put in some safeguards then maybe that will
• 57:52
  limit the worst successes what's the morally repugnant thing though oh it's easy it's easy
• 57:59
  I say to people it's easy to imagine what it's going to be morally repugnant each going to look
• 58:04
  like a bad Hollywood movie you're going to turn on your TV and they're going to go to the news
• 58:08
  reports in Gaza or Ukraine or Dohar where it is and you see a swarm of drones taking out a
• 58:17
  school yard of children and you say oh my god that's being done in my name
• 58:21
  well you know we're not doing anything about preventing that happening
• 58:25
  because it will be a you know a bad Hollywood movie yep I'm still stuck on the point that we
• 58:29
  can already do that and we don't we can do what we can already take out a school yard of children
• 58:35
  yeah and we don't so whatever the rules are that we're obeying because then why wouldn't we
• 58:42
  just impose the same rules on the AI I hope we do but then isn't the AI not the problem but our
• 58:52
  military and political decision-making like I don't understand why autonomous weapons equals
• 58:58
  mowing down children because they changed the speed if we already have the power to mow down children
• 59:04
  in a school but they don't choose to do that during war they're making it ever quicker ever more
• 59:08
  accurate to do that so previously if you wanted to fly you know several drones and you needed
• 59:15
  several pilots now you don't you can just fire them autonomous yes I mean again this is coming back
• 59:20
  to where I think the real right the US Air Force implies more drone pilots now the pilots of any
• 59:27
  other type of aircraft and soon it will be more drone pilots than all the other pilots added together
• 59:33
  so there'll be the US drone force not the US I understand and none of the meds or ever has
• 59:39
  mown down children in a school yard you know just as a as a strategy of targeting civilians during war
• 59:46
  well there are some leaked videos to suggest that all the targets are legitimate but but anyway
• 59:53
  that's a that's a challenge for the the US military which is you've got to train those people you
• 59:58
  have to employ those people and those people are harmed I mean interestingly enough
• 1:00:03
  PTSD post-traumatic stress disorder amongst drone pilots is greater it occurs at a greater rate
• 1:00:10
  than amongst fighter pilots it's more damaging to fight those and kill those people on a TV screen
• 1:00:16
  that it is actually to be there in the cockpit doing it yourself that's interesting but I mean
• 1:00:21
  there's you know there's various reasons you know why why you could expect that to happen
• 1:00:25
  why the disconnect right so you're you're in a container in Nevada flying these drone
• 1:00:30
  killing people on the ground you know terror hopefully terrorists but you know whoever they are
• 1:00:35
  then you go back to your to your family home and have dinner in the evening
• 1:00:38
  and the strange yeah yeah strange world that you're in yeah you're having to switch between
• 1:00:45
  and there's no danger and then the distance between yeah maybe if you're in a fighter jet at least
• 1:00:51
  you're feeling like you're part of a battle yeah and you could get shot at it's gone right you
• 1:00:56
  you justified yourself the fact that you're killing people because you your own life is a risk
• 1:01:00
  yeah exactly yeah it's me all that yeah yeah yeah it's obviously got to be them but you know
• 1:01:06
  sitting in a container pressing a button on joystick I've got my skin in the game right you
• 1:01:10
  just have to urinate into a bag that's the worst that has to happen in your container in Nevada yeah
• 1:01:16
  yeah I mean so it strikes me that we're still confusing the democratization of warfare and the
• 1:01:23
  risk that it will that these technologies will carry if they fall into the hands which they
• 1:01:29
  invariably will have terrible either terrible states or non-state actors like terrorists
• 1:01:37
  with a light the likely outcome of western democratic militaries using AI to improve their
• 1:01:47
  objectives whatever those objectives might be right right that's why I'm sort of moving
• 1:01:52
  stuck on the point about like you know shooting like children may get attacked by swarms of drones
• 1:01:57
  but it's unlikely that that's going to happen because of western military decided that
• 1:02:02
  it wanted to inflict suffering on a civilian population to achieve a political end right
• 1:02:06
  because we already don't do that but let's let's agree before we move on that they already
• 1:02:11
  falling into the hands of the wrong people right so the fact that you're on can retaliate yes and take
• 1:02:16
  out you know airport hotel in Dubai or wherever it was is because they're using drugs but previously
• 1:02:28
  they would not have any capability to respond at all but now they have got a capability to respond
• 1:02:34
  and cause harm yeah yeah and that's only going to increase over time yeah I'm still stuck on the
• 1:02:42
  prisoner's dilemma of feeling like if they're going to be doing it then I want us to have the best
• 1:02:49
  equipment with which to respond and maybe autonomous drones are included in that toolkit why is
• 1:02:57
  this any different to chemical weapons we should be using chemical weapons well I think chemical
• 1:03:02
  weapons is tricky isn't it because the hardware is different like the hardware with chemical weapons
• 1:03:07
  is chemicals yes but the hardware with a lot of these systems is identical to the stupid version of
• 1:03:12
  the thing isn't it so you know drone is a drone whether or not it has a clever brain in it
• 1:03:20
  you know clever AI brain in or not so it's not such a potent weapon right so regular drones today
• 1:03:26
  in Ukraine are useless right because this electronic jamming is so sophisticated now you
• 1:03:31
  you cannot fly them immediately get stopped so they're either you know flying off fiber optics
• 1:03:38
  which can't be jam so easily or they're increasing the autonomous so what kind of a model of
• 1:03:44
  restriction is best from your point of view because I because of that problem that the hardware is
• 1:03:51
  often the the same stuff essentially how do you police I can imagine policing chemical
• 1:03:59
  weapons and certainly policing you know rare types of weapons like uranium and stuff like that
• 1:04:05
  plutonium what exactly does one police in the enforcement suppose we had an international
• 1:04:11
  set of rules around AI I think there are a few easy places to begin so come on to control
• 1:04:18
  nuclear weapons I think that's like that's a no-brainer even if the AI is better than people
• 1:04:25
  yes let's keep nuclear weapons humans in the loop in deciding to start a nuclear war right I think
• 1:04:31
  you know we could all agree to that one but just to point out we haven't all agreed to that so
• 1:04:37
  the Russia is developing got prototypes of maybe is even in in in service now at the
• 1:04:45
  precise and autonomous underwater submarine things the size of size of a bus it's nuclear powered
• 1:04:52
  so it can travel unlimited distances very high speed it's much smaller than conventional submarines
• 1:05:00
  because you know it's it's uncrewed so you don't need all of the of that so it's very stealthy
• 1:05:06
  and it's believed that it will contain a a cobalt tipped nuclear device so you could drive
• 1:05:13
  this thing into city harbour or up the Hudson and take out half of New York City wow and not
• 1:05:22
  unused nuclear missile launcher but a nuclear device on it so that the submarine itself explodes
• 1:05:27
  but itself up take out half of Manhattan because it's a submarine you can't communicate
• 1:05:32
  with it radio waves don't work underwater if it comes to the surface it reveals itself easy
• 1:05:38
  gets destroyed so it's decision-making will be using algorithms and it will have set off one two
• 1:05:46
  three a week ago before from the Baltic Sea to travel stealthily across the Atlantic to go up the
• 1:05:53
  Hudson and then it doesn't realize that in the meantime the premier of the Soviet Union it
• 1:06:00
  gives us reminds me of strange love I mean it's literally though it's literally the plot of
• 1:06:05
  strange life the Soviet Union devises a doomsday machine that cannot be turned off that we will
• 1:06:10
  just detonate itself automatically that humans are kept out of the loop on you do ask yourself
• 1:06:16
  what is going through the minds of the Russian military or Putin to think that this is a useful
• 1:06:22
  weapon I mean what circumstances what sort of what sort of capability is this well why would
• 1:06:30
  you what why would you ever conceive of using this or how does this give you military
• 1:06:36
  advantage just the existence of this thing it's hard to understand I guess it's probably a defensive
• 1:06:41
  strategic tool much like all nuclear weapons are which is that they're not actually very useful
• 1:06:45
  to use but having them is really useful to stop actors from doing other conventional things
• 1:06:51
  that you don't want them to do yeah exactly I mean yeah so the reason I mean partly we know about
• 1:06:56
  this is because Russia hadn't internet naming competition for it here's a tip lad if you
• 1:07:03
  want to keep something secret don't don't invite don't invite public solutions on the name
• 1:07:08
  so that's why it's called besides in the god of the god of the sea should have been called
• 1:07:12
  sub-emix subface yes you know new key mcnew you can make new face wow that's amazing but
• 1:07:19
  is that yep so yes so you've got systems like that so I think that's an easy place to start yeah
• 1:07:23
  and another place that you might want to start is autonomous weapons that target humans as
• 1:07:31
  opposed to autonomous weapons that target military hardware so when we when we subset we do already
• 1:07:36
  have autonomous weapons that target you know the radio radar costs cross section of an incoming
• 1:07:41
  fighter missile or a tank you know on the ground but things that are designed to target humans in
• 1:07:48
  particular things that are designed to target specific humans based upon specific characteristics
• 1:07:54
  maybe their face right so I think in this way I thought politicians would be swayed
• 1:07:58
  right when the when it's targeted with you know it's using facial recognition of you know
• 1:08:04
  particular people's faces to do the targeting yeah that is frightening isn't it I mean I
• 1:08:09
  came into the country yesterday and you know you've been many years since you had to talk to a
• 1:08:13
  human being when you come into Australia and many other countries you just stand in front of a
• 1:08:17
  facial recognition thing and the gates open it let's you in so if it's that easy and that
• 1:08:22
  cheap to scan your face and identify you with the classic problem we have with our intelligence
• 1:08:30
  which is the same algorithm is on your smartphone opens the face with your face yeah that's a great
• 1:08:36
  convenience and it's secure because you know it's your face so means you don't have to remember
• 1:08:41
  past words and do it when you put that on a drone right so just put that on a drone just to say
• 1:08:46
  that's a good thing right it's made a lot of life more convenient it's great people with
• 1:08:57
  disability you can use phones now without having two hands to do things great but equally we'll
• 1:09:04
  take the same algorithm we put it on a drone we start killing particular people and not just
• 1:09:10
  particular people but I think we've spoken in the past about you know the possibility of if you
• 1:09:14
  were up to ethnic cleansing for example yes you could just get drones to target all Asian people
• 1:09:19
  and you know pinky people like us or white people or whatever you can have them or Chinese people
• 1:09:25
  whatever it is right any whatever characteristic or all women or all children you know all people
• 1:09:30
  with an interpreter and all purple yeah all people you know with a bit whatever it is yeah this is
• 1:09:35
  like pole pot who rounded up everybody who wore glasses and you know put them in concentration
• 1:09:41
  candles or killed them because wearing glasses meant that you were an enemy of the people
• 1:09:44
  because you're a snobby elitist intellectual which is one reason why I say in some sense this is
• 1:09:49
  more troubling than previous technology I mean we've had other weapons of mass destruction we've
• 1:09:54
  talked about chemical weapons nuclear weapons those are very they're not very discriminatory
• 1:10:01
  chemical weapon will kill everyone in that they do it right but suddenly we've got a weapon
• 1:10:05
  of mass destruction because it will be able to you know great thing about computers if you could do
• 1:10:09
  something once you could do it a hundred times a thousand times you just put it a loop around it
• 1:10:12
  you know if you've got a drug that killed one person a hundred drones will kill a hundred people
• 1:10:16
  a thousand drones will kill a thousand people so suddenly we can scale warfare um so you know
• 1:10:22
  in some sense that is a weapon of mass destruction but now we've got an incredibly high precision
• 1:10:27
  discriminatory weapon of mass destruction where we can see exactly who we kill and who we don't
• 1:10:32
  right and so the advocates of that would say that's great because we can kill Hitler
• 1:10:37
  or Khamenei and we can target them but I suppose the opponents would say
• 1:10:45
  but then it lowers the bar to doing that because you don't you might be reticent to wipe out an
• 1:10:51
  entire city with a nuclear weapon yes but you might be less reticent to pull something more targeted
• 1:10:57
  so yes I mean that's we haven't talked about that but potentially low is the barriers to war
• 1:11:02
  right so if you feel that you can now we can do things that you previously could do and we
• 1:11:06
  we see that in Iran it's going on right now yeah we previously we thought oh well it's
• 1:11:11
  possible we can't can't guarantee we're going to take out the leadership well now we can
• 1:11:16
  pretty much guarantee we are going to take it in this ship that's right and if the collateral
• 1:11:19
  damage was a hundred X what it is today 10 years ago then you would also worry about alienating
• 1:11:25
  the or any in people more than one and the more questions of killing civilians and how many civilians
• 1:11:31
  are we allowed to kill yeah that lead it well if it's very few then it justifies doing that
• 1:11:37
  yes yes I heard I heard something on the radio about some civilian apartment complex being hit in
• 1:11:42
  Iran and you know some dozens of civilians dying in these Israeli and American strikes
• 1:11:49
  and whilst I was sympathetic I did somewhat sort of think it's not very many in the context
• 1:11:56
  of historical wars is it like I mean that you know it's it's terrible for the people but as far
• 1:12:00
  as war goes I would be I mean the real news would be if not a single civilian lost their life
• 1:12:07
  when the world's greatest superpower you know launched a major combat operation against another
• 1:12:12
  country it reinforces the point that technology is changing the character of war what we thought
• 1:12:18
  of as war right so war had war used to be about you know how much pain you were willing to put
• 1:12:24
  up with and it was it was against your civilian population as much now we have you know
• 1:12:30
  the surgery in place where we can precision strike things yeah so I mean there's
• 1:12:35
  lying lowering the barrier to going to war there's also lowering the cost of war both financially
• 1:12:42
  and in terms of your civilian lives yeah compared to the other civilian lives and this comes back
• 1:12:48
  to the question of asymmetry is I mean as long as we live in a world in which one side has vastly
• 1:12:56
  more technologically advanced military equipment than another they're going to be able to go to
• 1:13:02
  war with much greater impunity because they're obviously going to care less about their adversary
• 1:13:05
  civilians than they are about their their own so if you can guarantee that almost none of your
• 1:13:12
  guys lose their lives then it becomes somewhat beside the point how many of your enemies
• 1:13:19
  do it doesn't it depending on where the moment though there's a relay symmetry and this is part
• 1:13:27
  of the problem which is there's an asymmetry for the aggressor against the defender it's actually
• 1:13:32
  quite hard to defend yourself yes that's the we don't have many good defenses that we had against
• 1:13:37
  a swarm of drones now you can overwhelm most conventional means of defense if you come out someone
• 1:13:46
  with 50 drones right you can take out a few of the drones but some of them will get through so
• 1:13:51
  it's it's not so easy to defend yourself and it then incites you to well I should be the aggressor
• 1:13:57
  though right turns everyone into a paranoid yes yeah that's interesting isn't it because
• 1:14:05
  yeah it's one thing for Iran to retaliate by lobbing a missile into the suburbs of Jerusalem as they
• 1:14:12
  did and killing you know a few dozen Israelis but if there was a swarm of a thousand killer drones
• 1:14:19
  flying along a Tel Aviv beach blowing up civilians that would be a very different story and
• 1:14:29
  we may not be that far away from a world in which that's not more extensive than the
• 1:14:33
  missile that they yeah dropped yeah all right so no AI on nukes no no AI that is that can target
• 1:14:46
  humans human beings only infrastructure and assets yeah is there any other rule of thumb that
• 1:14:51
  you would want to impose and that's a good place to try and start the world is to set the
• 1:14:56
  something we could do in this space as opposed to you know often people as an issue reaction which
• 1:15:02
  is like oh well yeah we can't possibly do this we the no one else can agree with us and we
• 1:15:08
  didn't just put ourselves as a technical disadvantage and are there forums in which these things are
• 1:15:13
  properly being discussed but funny enough I'm getting up I didn't actually mean to tee you up to
• 1:15:18
  your impending journey this afternoon to the United Nations Geneva who for the last 10 years
• 1:15:24
  have been talking about this slowly slowly trying to build some consensus in the international
• 1:15:29
  community around yeah we should do something about this this as we've done things about other
• 1:15:35
  technologies that it would be worthwhile trying to get some consensus together yeah the problem
• 1:15:40
  there of course is that United Nations works by looking for you know complete consensus and the US
• 1:15:49
  China to a certain extent Russia certainly and a few other nations have probably felt that they've
• 1:15:56
  got a technical lead that they should try and hold on to it I think that's foolish because if you
• 1:16:01
  look at the history of military technologies no one has held on to a technical leave for any
• 1:16:05
  like the time you know the US thought they were going to have a technical lead with the hydrogen
• 1:16:09
  the Russians had stolen within a couple of years no technology has ever been kept to one side of
• 1:16:20
  that's interesting yeah I mean and there were voices weren't there after the Manhattan
• 1:16:23
  projects saying that instead of racing for a hydrogen bomb they should immediately you know
• 1:16:30
  hold to the American nuclear program and talk to the Russians about not getting into an arms race
• 1:16:34
  the world probably would have been a lot safer if there weren't thousands of hydrogen bombs
• 1:16:38
  sitting around yeah um is there any receptiveness from any countries that matter
• 1:16:48
  well over a hundred nations have called for as it wants that matter
• 1:16:52
  the European Union the African Union okay that matter Austria Belgium to a certain extent
• 1:17:00
  Germany the German Foreign Minister so the problem is you know we don't exactly live in a
• 1:17:08
  raw space order anymore no does it if you know the US is positioned which it seems to be at the moment
• 1:17:15
  which is we are going to be forceful and force wins over everything else and you know it's a
• 1:17:26
  reflection of the world that we're in do you think that the Pentagon's um desire not to or
• 1:17:32
  refusal to accept anthropics limitation on the use of autonomous killing machines was indicates
• 1:17:41
  that they are doing research on it or do you think it just indicates a natural sort of military
• 1:17:47
  desire not to be pushed around and not have to follow rules because we're going to do what we want
• 1:17:51
  to do and you can't tell us what's it I'm pretty convinced that the US military didn't want this to
• 1:17:58
  happen this is the Secretary of War they didn't want the break to happen because Claude is their
• 1:18:06
  favorite tool yeah and the two limitations that the anthropic we're insisting upon were probably
• 1:18:16
  not constraining them in any meaningful way so what and indeed you know they've
• 1:18:23
  they've designed a $200 million contract now with open AI yeah to do exactly the same
• 1:18:29
  so it's just inconvenient but they haven't accepted the limitations right because I mean the open
• 1:18:35
  AI is willing to allow the Pentagon to do well I will claim that they've got those they've got
• 1:18:41
  uh those two clauses in the contract but it's a weakened water down version of those two clauses
• 1:18:48
  yeah so I'm just a little bit baffled about what happened it was was it just a matter of
• 1:18:51
  a few some some of them and as stabbed Dario Modi in the back right you he was negotiating
• 1:18:57
  no I mean I understand what happened with the open I mean of course open AI is going to come in
• 1:19:01
  and pick up the $200 million yes of course I mean why would why would they not but I'm what I mean
• 1:19:06
  is I am still unclear about what the actual contours of the dispute between the Pentagon and the
• 1:19:11
  Trump administration on one side and anthropic on the other side were if the two conditions that
• 1:19:15
  anthropic was insisting on were sort of moot either way why did the Pentagon take such a harsh
• 1:19:21
  stance on not agreeing to them why didn't they just agree always this just Trump trying to
• 1:19:28
  be TASKET for first he'd hexeth trying to boot anthropic yes in the tick kick him in the teeth yeah
• 1:19:34
  god you know I mean this usual Trump term term two presidency which is you you try and bully people
• 1:19:44
  uh how do you get but it's I mean it's unprecedented they've never used the supply chain act to
• 1:19:52
  go against a US company before um you know supposedly they were in bed with big tech they all
• 1:19:59
  lined up in the inauguration behind the president suddenly they're discovering is you know it's not
• 1:20:04
  so easy to be a friend with the Trump these days I'm a reliable friend I mean I would have thought
• 1:20:08
  that once the once a company once an AI company is working with the Pentagon and they's
• 1:20:13
  simulated all of their systems already well they are and you know Trump's now saying
• 1:20:18
  every part of government has to withdraw from using anthropic not just the military
• 1:20:22
  it's not going to happen is that I mean why would you harm you know one of the most successful AI
• 1:20:30
  companies that's helping drive the economy in the US by but also how much money you're going to
• 1:20:36
  spend on uninstalling it from every system that you've got the DOD contract was about 10% of their
• 1:20:41
  revenue wow non-trivial amount of money then if it goes to all forms of governments it's
• 1:20:48
  focused on enterprise that's going to be a significant chunk of their revenue
• 1:20:53
  but equally you know what I'm gonna see how it's going to unfold but it's
• 1:21:00
  I feel like I need to go out and get a paid subscription to Claude just as a principled attempt to
• 1:21:07
  bolster anthropic for having stood up to them and there's a reciprocal campaign now to
• 1:21:15
  do subscribe from chat you yeah yeah good and supposedly I look this morning hundreds of
• 1:21:20
  thousands of people had do subscribe to you as a as a response to them you know signing the contract
• 1:21:27
  with the DOD what's the worst case in sort of best case scenario that we end up with on AI and
• 1:21:36
  warfare like you've sort of painted the worst case you painted the worst case you painted the worst
• 1:21:42
  case example is a bad Hollywood movie where we see swarms of drones being used to attack
• 1:21:48
  civilians being used by terrorist organizations being used by rogue states to cause you know
• 1:21:56
  significant harms and a world that's really destabilized by this technology because it's not
• 1:22:01
  as we keep on saying it's not going to be just the military powers that have the capabilities it
• 1:22:06
  would be everyone and their dog who can call upon these technologies and if you're a terrorist
• 1:22:11
  interesting thing is the US military can say oh we're going to use this technology responsibly
• 1:22:16
  and I'm sure they will they test there's you know there's significant testing and they go through
• 1:22:22
  before they actually use any artificial intelligence in the US military to make sure that it's being
• 1:22:26
  used in lawful ways but the problem is that if you're a rogue state or a terrorist organization
• 1:22:31
  and you've got a weapons system that's only 50% accurate what do you care it's some more
• 1:22:38
  terrible weapon if it kills half the people it kills and all the intended targets well that's
• 1:22:42
  what that's what terror is about yeah you don't care if you're the US military you're only going to
• 1:22:46
  deploy it when it's 99.99 you know five nines or whatever it is accurate right so the terrorists
• 1:22:52
  will do it first yes yeah and there's no way that you can defend it's not like you can create an
• 1:22:57
  iron dome no it's missiles out of the sky if what you're dealing with it we're back to say symmetry
• 1:23:03
  it's much easier to attack to launch a wave of you know hundreds of cheap drones and one of them
• 1:23:09
  will get through and it's actually much easier to defend yourself against a nuclear ICBM that's
• 1:23:14
  fought flying towards you then against you know a bunch of a thousand drones the drones that are
• 1:23:19
  swarming around at least there's a chance of shooting the ICBM out of the sky there's nothing you
• 1:23:23
  can do about yeah thousand drones yeah all right the positive picture is you know it's used for
• 1:23:30
  mind clearing it's used to get supplies into contested territories it's used to you know help
• 1:23:36
  you more efficiently more effectively defend yourself yeah so I mean we think AI is recent but AI
• 1:23:43
  has been around for 56 years I remember when desert storm happened I was at one of the big AI
• 1:23:48
  conference that year and the US military have always been the biggest funder of AI research
• 1:23:53
  historically I remember various generals turning up to that conference saying you paid the desert
• 1:23:59
  storm paid for all of AI research ever as they say what a way to do pay for US research so
• 1:24:04
  what we used one scheduled AI scheduling algorithm to get the stuff out to to the golf get the soldiers
• 1:24:11
  the bullets the the ready pack meals everything out in the right order the right amount at the right
• 1:24:18
  time and we did it so much more efficiently than the last conflict that the billions of dollars
• 1:24:24
  that we've ever spent on AI research has been paid for wow wow and Jeff Bezos woke up and thought
• 1:24:31
  I could do that to sell books and I used the logistics to you know on time delivery I mean it is
• 1:24:37
  incredible because militaries are essentially you know logistics with fighting capability yeah right it's
• 1:24:42
  all about you know armies fight on their stomachs is about making sure you've got enough bullets
• 1:24:47
  meal it's FedEx it's FedEx so it's a major process operation yeah exactly and it's a very
• 1:24:53
  demanding cloud because if you turn up without the bullets you're in trouble yeah it's not good yeah
• 1:24:58
  no it isn't I mean it isn't so you can see there's yeah there's going to be some fantastic yeah
• 1:25:03
  I've always said we're not going to not have AI in the military I accept fully the AI is going to
• 1:25:09
  do some really useful stuff in terms of defending ourselves in terms of we're effectively delivering
• 1:25:14
  you know helping to disaster zones and also saving lives no human should ever go and clear a
• 1:25:21
  minefield put a robot on the job absolutely indeed we're developing data 61 Sarah developing a
• 1:25:28
  mind clearing robot so yeah if it goes wrong robot gets blown up gonna get another robot yeah no
• 1:25:34
  one should risk your life for them clearing a minefield fabulous um lastly Toby I just want
• 1:25:40
  your thoughts about sort of how we govern ourselves and um badly is it apocryphal the old line about
• 1:25:53
  was it Gandhi who has asked what he thought of Western civilization oh be good if when it comes
• 1:25:56
  yeah yeah I think it'll be a great idea but no in the future like so as AI gets better at
• 1:26:04
  um noticing patterns and helping us to strategize and helping security agencies identify threats
• 1:26:12
  before they've become threats maybe even going into demographic and sociological things where
• 1:26:19
  we know this person who has these genes that are normally associated with violent behavior who
• 1:26:24
  was a troubled child see we're going down to minority reports yeah all the sort of minority reports
• 1:26:29
  shit right you know let's keep an eye on I mean look the Bondi terrorist attack right this is still
• 1:26:33
  an open wound here in Sydney and there's a lot of recrimination about the intelligence failures
• 1:26:37
  that led up to that why was a you know why was a person who went to a very extreme mosque and was
• 1:26:43
  obviously very very devout he went to the southern Philippines which is a hotbed of terrorism
• 1:26:48
  and then came back and was allowed to own six guns like why why were then why was there not an AI
• 1:26:54
  that was pinging or a human but an AI is you know cheaper and can ingest more data that flagged
• 1:27:02
  you know you might want to take a look at these at these guys earlier um this is a great debate we
• 1:27:07
  need to have yeah so suppose because yeah you're right with some joined up intelligence we could have
• 1:27:13
  easily stopped that you know the the 9-11 terrorists were no one yes and again joined up
• 1:27:18
  intelligence would have would have identified them the attack happened right so yes but the problem
• 1:27:24
  is the same joined up intelligence will it can be misused by authoritarian governments for stopping
• 1:27:30
  any dissent yeah right so how do we walk that line that's that's that's not a technical question
• 1:27:35
  that's a question about governance and democracy yes that's why I said I want to ask you about
• 1:27:41
  politics because if you if you get good enough data yeah about all this stuff you can imagine a
• 1:27:47
  world where there's a significant push towards a more authoritarian form of governance to keep
• 1:27:55
  us all safe especially if individuals are super empowered to pull off mega terrorism quite
• 1:28:01
  cheaply thanks to drones like if it if it does become possible for an individual to launch
• 1:28:08
  drone warfare biological warfare chemical warfare using nothing but the dark web and things
• 1:28:14
  that they can buy at a hardware store and the AI that helps them plan it then even if only
• 1:28:24
  1000th of 1% of the population wants to do that the need to track those people and catch them
• 1:28:31
  beforehand given the scale of what they can pull off will be so pressing that you can imagine
• 1:28:38
  people tolerating a large infraction of their civil liberties in terms of what governments do
• 1:28:42
  you know you said that the Pentagon might already be surveilling Americans it's entirely possible
• 1:28:46
  that will submit voluntarily to AI surveying us all in order to know and show that they were
• 1:28:52
  yes exactly that's right yes the note that's the Snowden stuff yes they're already doing it on
• 1:28:57
  during the Bush administration right and now they've got art of intelligence on top of that
• 1:29:02
  which can do it in a much more specific way that was a scandal so it's not a foregone conclusion
• 1:29:06
  that they would do it unsolved consciously but they're certainly willing to do it so they were
• 1:29:11
  tapping into you know telephone calls and various other communications now you can put on top
• 1:29:15
  of that art of intelligence which allows you in a much more semantics sophisticated way to listen
• 1:29:21
  to those conversations well they were they were they were just to be you know perfectly
• 1:29:26
  pedantic about it they were looking at the methods art of conversations they weren't listening in
• 1:29:30
  on Americans conversations without a warrant but they were nonetheless you could interpret it
• 1:29:34
  as an unlawful seizure to look at all of the telephone records of all Americans to tell us
• 1:29:39
  nothing who's trying to do exactly quite telling yes that's right but how do you so how do you
• 1:29:45
  think about that in your role as sort of thinking about the future of AI about how do you how do you
• 1:29:51
  reduce the risk of rogue individuals from becoming super empowered to do evil at the same time as
• 1:29:59
  you maintain civil liberties and don't allow governments to become big AI institutions that are
• 1:30:07
  using mega data to spy on us all and catch the bad guys and in the process in slavus all accidentally
• 1:30:16
  I mean these these are tough questions but that because technology is helping
• 1:30:22
  give us privacy as well let's not forget it's not one way straight right so cryptography is
• 1:30:28
  you know we're getting better and better tools to actually guard our privacy to stop people
• 1:30:35
  listening to us and to you know to think dangerous thoughts because democracy dies if we
• 1:30:40
  can no longer think dangerous thoughts well uncomfortable conversations is the name of the show
• 1:30:46
  Toby it's always lovely to talk to you thanks for being with us.