Latest / Ethical Machines / In Defense of Killer Robots

Transcript

In Defense of Killer Robots · Ethical Machines

0:01
[SPEAKER_00]: Usually when you hear the phrase killer robots, you think that's a really bad thing.
0:04
[SPEAKER_00]: We don't want killer robots.
0:06
[SPEAKER_00]: We don't want AI targeting human beings.
0:09
[SPEAKER_00]: We don't want AI choosing which targets are the appropriate ones to kill or blow up.
0:15
[SPEAKER_00]: And last season, that's exactly what we talked about.
0:18
[SPEAKER_00]: I spoke with Rosario today, oh, from Oxford, who argued exactly that.
0:22
[SPEAKER_00]: This conversation is the exact opposite.
0:24
[SPEAKER_00]: Today, I'm speaking with Michael Horowitz.
0:26
[SPEAKER_00]: Michael, and this is a long bio, but it's relevant.
0:29
[SPEAKER_00]: He's the director of the Perry World House and Richard Perry professor at the University of Pennsylvania and a senior failure for a technology and innovation at the council on foreign relations.
0:38
[SPEAKER_00]: And before he was a professor, he was a deputy assistant secretary of defense for a forced development and emerging capabilities.
0:44
[SPEAKER_00]: and director of the emerging capabilities policy office.
0:47
[SPEAKER_00]: So he's been with the DOD.
0:50
[SPEAKER_00]: He's also a professor and he roughly are used in defense of killer robots.
0:54
[SPEAKER_00]: He thinks that's a bit exaggeration.
0:55
[SPEAKER_00]: You'll see but he thinks that it is good to have AI integrated into systems that manifest some degree of autonomy with regards to deciding whether someone should live or die in a military context.
1:09
[SPEAKER_00]: the conversation is obviously fascinating and I think you'll find that it's a compelling case.
1:15
[SPEAKER_00]: It's a very compelling case and so while maybe our instincts are to say, no, AI with a capacity to the autonomy, to kill people in warfare, maybe we have the incident that's terrible,
1:27
[SPEAKER_00]: There's good reason to think actually maybe those instincts are all wrong.
1:30
[SPEAKER_00]: So that's the topic of this conversation.
1:32
[SPEAKER_00]: Obviously it's extremely fascinating and extremely important since we really do need to decide what we should allow in a military context and what the real danger is or since obviously they could be existential.
1:44
[SPEAKER_00]: All right, that's what this episode's about.
1:45
[SPEAKER_00]: Don't forget to subscribe on Apple or Spotify by my book, Ethical Machines.
1:49
[SPEAKER_00]: Go to my sub-stack, read BlackBand out, sub-stack.com, email me at EM at readblackman.com.
1:54
[SPEAKER_00]: If you have questions, try and share the criticism of the suggestions for guests or issues you want to hear discussed.
1:59
[SPEAKER_00]: And that's it.
2:00
[SPEAKER_00]: Let's go talk about Killer Robots with Michael.
2:19
[SPEAKER_00]: So Michael, if you want to scare someone and talking about AI and military context, you might talk about killer robots.
2:25
[SPEAKER_00]: But you want to, it seems, you want to defend killer robots, whatever that means.
2:29
[SPEAKER_00]: And so I guess my first question is, why do you hate humanity so much?
2:33
[SPEAKER_00]: It's a great question.
2:34
[SPEAKER_02]: My wife and family asked me that all the time.
2:36
[SPEAKER_02]: I don't know if anybody's really pro killer robots.
2:39
[SPEAKER_02]: It's like a central tenant of pop culture that like, if you give a robot a brain and a gun that it decides to destroy all of humanity, yeah, which I think says more about how we think about ourselves, frankly, then about the like robots like per se, although like the murder bot show, currently on television, actually.
2:58
[SPEAKER_02]: I don't know if I can just say is an exact it's like spoiler alert for the first two minutes of murder
3:03
[SPEAKER_02]: the robot hacks its own system and is free and does not immediately decide to destroy all of humanity.
3:09
[SPEAKER_02]: Does not, but does not.
3:11
[SPEAKER_02]: I'd be like the first time besides Wally in the history of film and television.
3:16
[SPEAKER_02]: You've seen the sequel though.
3:18
[SPEAKER_02]: sequel of Wally.
3:18
[SPEAKER_00]: Yeah, he goes
3:29
[SPEAKER_02]: And by that, what I mean is we used to think about victory and defeat on the battlefield is about having more, like give a bigger army, more tanks, more planes, etc.
3:37
[SPEAKER_02]: And then for the last 40 years, we've been really focused on precision.
3:41
[SPEAKER_02]: Radar guided missiles, heat-seeking missiles, etc.
3:44
[SPEAKER_02]: etc.
3:44
[SPEAKER_02]: All about instead of mass, let's fire one missile at a specific building in a village after a drone hovers above it for a week when we know there's only one person
3:58
[SPEAKER_02]: in there and it's the target.
4:00
[SPEAKER_02]: The challenge is that precision technology is now four to five decades old.
4:05
[SPEAKER_02]: Those precision weapons can cost millions of dollars.
4:09
[SPEAKER_02]: And what we see on the battlefield every day with Ukraine in the context of the war between Ukraine and Russia suggests that there is a new angle here
4:24
[SPEAKER_02]: commercial manufacturing and basic precision guidance.
4:28
[SPEAKER_02]: Me, that almost anybody around the world now can do precision strike in one way or another.
4:35
[SPEAKER_02]: And this is, I think, the most important feature of it.
4:38
[SPEAKER_02]: They increasingly need the ability to survive for their weapons to survive in dangerous environments where they can be jammed or hacked or otherwise disrupted.
4:50
[SPEAKER_02]: In these kinds of new conventional wars, it isn't the war in terrorism or the forever war.
4:55
[SPEAKER_02]: This is a world where you might lose access to satellites and you need the ability of weapon systems to operate even when you don't have access.
5:04
[SPEAKER_02]: And in that way, having more autonomy built into the system gives you the ability to successfully operate even in that communications, degraded, or denied environment, as I think what an expert might say about that.
5:18
[SPEAKER_02]: But more important, you can do this in ways that are 100% consistent with international humanitarian law.
5:25
[SPEAKER_02]: Because what we really always want is a human responsible or accountable for the use of force.
5:34
[SPEAKER_02]: the machine doesn't make a decision, the human makes a decision.
5:37
[SPEAKER_02]: And you can do that even with an autonomous weapon system.
5:41
[SPEAKER_02]: And it can have advantages too.
5:44
[SPEAKER_02]: Imagine a world where you have a radar guided missile, and that radar guided missile's been launched to try to destroy an enemy radar, and it turns out that radar is on top of a school or on top of a hospital.
5:53
[SPEAKER_02]: That radar guided missile is hitting the school or the hospital.
5:56
[SPEAKER_02]: That's super dangerous, and a lot of civilians are going to die.
5:59
[SPEAKER_02]: Now imagine instead, there's an algorithm on board, it gets close to the radar and it's a, hey, that radar is on top of a school.
6:05
[SPEAKER_02]: Maybe it shouldn't hit the, not to like, anthropomorphize the best way, I shouldn't hit the radar.
6:09
[SPEAKER_02]: and then diverts, we should want our systems to be smart enough to be able to determine what the legitimate targets are after a human has designated that it's legal for them to be able to operate in a given situation.
6:25
[SPEAKER_00]: Okay, this is what this sounds like to me.
6:26
[SPEAKER_00]: It's something like this.
6:28
[SPEAKER_00]: Don't worry about giving AI military weapon really increased autonomy.
6:34
[SPEAKER_00]: I can see the functional argument is really useful because it loses contact with satellites, blah, blah, blah, blah, get that.
6:41
[SPEAKER_00]: Someone might think, oh, you're just going to let it be autonomous.
6:44
[SPEAKER_00]: So you're not going to have control over it.
6:46
[SPEAKER_00]: Isn't that insanely dangerous and reckless?
6:49
[SPEAKER_00]: And you're sort of reply that you built in as you are particularly that objection was, don't worry.
6:54
[SPEAKER_00]: There's still going to be plenty of control because we'll be sending in out with such and such algorithms.
6:59
[SPEAKER_00]: As you put it, parameters, guidance, instructions, et cetera, such that it will know how to make decisions
7:07
[SPEAKER_00]: the right way because we've told it how to make decisions the right way.
7:09
[SPEAKER_00]: So we still have sufficient control such that we are accountable for the goans on of the AI.
7:17
[SPEAKER_02]: I agree with most of that except I would push back against the use of the word decision in that I don't think that the machine in this context is ever making a decision and in some ways I think it gives us a lot more control over whether a weapon actually hits its target then before.
7:33
[SPEAKER_02]: Here's what I mean.
7:35
[SPEAKER_02]: If a human fires a gun, nobody asks, can we control the bullet all the way to impact?
7:41
[SPEAKER_02]: If a human fires a radar-guided missile,
7:44
[SPEAKER_02]: Yeah, there's the school analogy that I mentioned before where you can have a lot of collateral damage.
7:48
[SPEAKER_02]: It can be super dangerous.
7:50
[SPEAKER_02]: What I'm talking about is a world where you can fire an autonomous weapon systems or set of systems.
7:58
[SPEAKER_02]: And those systems have been say programs so that the valid targets are say the tanks of your adversary.
8:07
[SPEAKER_02]: And they get close to where the adversaries are, like the Seeker pops open, like, oh, that's a tech hit the tech and they can't be jammed and they can't be hacked.
8:17
[SPEAKER_02]: And so they're really, and they're really resilient and in some ways, they're more likely to be accurate in some ways.
8:22
[SPEAKER_02]: I think the conversation about autonomous weapon systems or killer robots or route, whatever you want to call them is really a battle of analogies.
8:30
[SPEAKER_02]: If what you're talking about is, all right, here's an individual person on the battlefield on the ground.
8:37
[SPEAKER_02]: Is that person a, like, okay, person to hit?
8:41
[SPEAKER_02]: That would be an incredibly difficult thing for any algorithm to be able to do, which is frankly why it's like the last place, I think you would imagine seeing autonomous weapons systems on the battlefield.
8:52
[SPEAKER_02]: But if the question is in a less cluttered, say naval war or air war,
8:58
[SPEAKER_02]: Or a conflict, a Russia Ukraine conflict where you have a stable front and Russian tanks say are approaching the front line and that kind of situation.
9:06
[SPEAKER_02]: It's not like there are lots of civilians like running about and you're trying to figure out like is this human and a valid target.
9:12
[SPEAKER_02]: It's oh, there's a fighter jet.
9:15
[SPEAKER_02]: Like, it's not our fighter jet.
9:17
[SPEAKER_02]: That's a veteran.
9:18
[SPEAKER_02]: Right.
9:18
[SPEAKER_02]: And like in that kind of situation, in some ways, it's about the alignment between what the technology can do and what you're asking it to do.
9:25
[SPEAKER_02]: And the important thing is to keep that alignment in check.
9:27
[SPEAKER_02]: And I think the technology has reached the level with there are some scenarios where autonomous weapon systems can be incredibly useful.
9:35
[SPEAKER_02]: And we see evidence of that in Ukraine every day.
9:39
[SPEAKER_00]: Well, let me offer two lines of thought.
9:41
[SPEAKER_00]: One line of thought is, I don't know that the comparison with the bullet is the right comparison.
9:48
[SPEAKER_00]: Is do we ask how much control it was under over the bullet after you've hit?
9:51
[SPEAKER_00]: And I think that's because we're pretty decent at predicting where it's going to go.
9:55
[SPEAKER_00]: It's got to go straight, you know, boring some God-like wind.
10:00
[SPEAKER_00]: It's a little fairly straight.
10:01
[SPEAKER_00]: And we might miss, we are A might not be completely accurate,
10:06
[SPEAKER_00]: We exhibit, I would say, a high degree of control over the trajectory of that bullet or where that bullet ends up, not complete.
10:13
[SPEAKER_00]: So it's not as though we're talking about control, yes or no, it's degrees of control.
10:18
[SPEAKER_00]: And I think what most people are going to say with the bullet is we have a sufficient control over where that bullet ends up, such that we've got no problem assigning responsibility to the person to shot it.
10:29
[SPEAKER_00]: We feel like, don't our due diligence on gun safety, such that we know how to use it safely,
10:35
[SPEAKER_00]: This is a bit bizarre.
10:36
[SPEAKER_00]: If you had a gun, and then the bullet quasi-magically, no, maybe just magically, zigzagged a lot.
10:42
[SPEAKER_00]: And sometimes it's zig, sometimes it's zag and eventually tries to get to where you're aiming it.
10:46
[SPEAKER_00]: You might be a little bit more hesitant to say you've got sufficient control over that bullet such that blah, blah, blah, blah.
10:53
[SPEAKER_00]: And I wonder if people think I'm not sure this is justifiable that the degree of control you have over an autonomous system is more like a zigzaggy bullet where, yeah, it's got some and destination and you're aiming for it, but because it can zig and zag and so many unpredictable ways, that makes it the case that it's not as, you do the logistics not as done as it is with say sugar gun.
11:16
[SPEAKER_00]: You do have a higher standard in order to deploy the autonomous act.
11:21
[SPEAKER_00]: Is that
11:22
[SPEAKER_02]: that totally makes sense.
11:23
[SPEAKER_02]: I think that in some ways this is all a play on the way that military's do testing and evaluation.
11:29
[SPEAKER_02]: Like frankly, the weakest point in the claim I just made is that if you think that militaries are going to cut corners in the way that they test their systems.
11:38
[SPEAKER_02]: Yeah, then yeah, you could have your like the zigzaggy bullet equivalent and then it won't work and like that would be like a bad idea.
11:43
[SPEAKER_02]: Yeah, but I would posit that's not a bad idea because it's autonomous or AI enabled or like whatever term we want to use it's bad because it's a bad weapon.
11:51
[SPEAKER_02]: When I say this, I've served in the U.S.
11:53
[SPEAKER_02]: Department of Defense, I've not served in any other militaries and so I'm only familiar with the American process in great detail.
11:59
[SPEAKER_02]: But the defense department is the largest bureaucracy in the world, arguably, and the testing and evaluation process for weapon systems is extremely detailed in that any weapon system needs to meet specific standards.
12:14
[SPEAKER_02]: And so suppose you were going to deploy an autonomous weapon system that was something like a missile, and like a really fancy, really smart missile essentially, that missile would
12:27
[SPEAKER_02]: as any other kind of this, or to use your example, if you were gonna fire a zigzaggy bullet, like an AI enabled zigzaggy bullet, that gun in that process would have had to go through the same testing as firing an M16 or something like that.
12:40
[SPEAKER_02]: So by the time anybody actually uses it on the battlefield, yeah, it's reliable, because frankly, weapon systems that aren't safe,
12:49
[SPEAKER_02]: don't work and weapons systems that don't work aren't useful for military commanders like military operators like none of them wanted to play systems that yeah don't work on the battlefield.
12:59
[SPEAKER_00]: Yeah this reminds me of the future life Institute.
13:03
[SPEAKER_00]: It doesn't exist anymore.
13:05
[SPEAKER_00]: They do.
13:06
[SPEAKER_00]: One of them doesn't exist anymore.
13:06
[SPEAKER_00]: Anyway, they put out a video.
13:08
[SPEAKER_00]: Maybe it was them.
13:09
[SPEAKER_00]: I think it was them.
13:09
[SPEAKER_00]: They put out a video that was meant to show how we get to catastrophe.
13:15
[SPEAKER_00]: Some AI system launches something without sufficient oversight and the causes.
13:19
[SPEAKER_00]: The enemy, to also launch nuclear weapon and reply and ignore all dead.
13:23
[SPEAKER_00]: But what was so funny, I thought it was comical in a way because
13:28
[SPEAKER_00]: when they were telling the story in the video about how the military came to acquire the weapon, it was like a plucky startup pitching at at a conference table and the military, you brass it and they go, yeah, that's great thumbs up and that's the procurement process.
13:42
[SPEAKER_00]: That's another procurement process.
13:43
[SPEAKER_00]: That is.
13:44
[SPEAKER_00]: is represented in the video and of course if it were like a plucky start up talking to a couple people in the military and they liked their picture now the onboard AI nuclear weaponry.
13:54
[SPEAKER_00]: Yeah that would be really worrying but presumably to your point the procurement process is slightly more rigorous than that.
14:02
[SPEAKER_02]: And just be clear, if you don't think that the procurement process is rigorous, and you don't think it delivers systems that, you know, it can do the things that they say that they're going to do, then they can absolutely, that there would be great risk and I think one could a counter argument to the point that I just made or be like, oh, you know, what if you're fighting a war and then you like cut corners to speed something onto the battlefield, my answer to that would be like that totally happens.
14:26
[SPEAKER_02]: But again, that's not about the AI.
14:28
[SPEAKER_02]: Yeah, because that happens with lots of systems, look at the way that, say, like, Ukraine is updating the software in their weapon systems on a weekly basis or sometimes even faster than that, because they're trying to evade Russian jamming and building more AI into those systems to try to make them more effective.
14:45
[SPEAKER_02]: They're facing an existential threat, reacting and reacting accordingly, and
14:51
[SPEAKER_02]: We'd probably do the same thing.
14:52
[SPEAKER_00]: Oh, they ask you this about the due diligence testing or the safety testing.
14:56
[SPEAKER_00]: How feasible is it that it gets done sufficiently comprehensively?
15:02
[SPEAKER_00]: What I'm thinking of is autonomous systems we use painted it as a lot of these things are coarse grain scenarios where it's a fighter jet or it's not a sort of gray area case.
15:12
[SPEAKER_00]: It's not a form of situation.
15:14
[SPEAKER_00]: Yes, you had easy for yourselves.
15:15
[SPEAKER_00]: But the fog of ore is shit's crazy and sometimes yes, it's clear.
15:19
[SPEAKER_00]: That's a fighter jet.
15:20
[SPEAKER_00]: It's not ours fire.
15:22
[SPEAKER_00]: Fine.
15:22
[SPEAKER_00]: But I take it that one of the issues is
15:26
[SPEAKER_00]: things get extremely complicated on the ground, things get very fuzzy, and they get fuzzy in, I would think fairly unpredictable ways.
15:36
[SPEAKER_00]: And so in order to do safety testing of the autonomous system, you're going to have to what simulate all the ways in which the fuzzy things might occur, the set of all fuzzy things, and then blah, blah, blah.
15:47
[SPEAKER_00]: There's just so many variables.
15:49
[SPEAKER_00]: I guess I
15:56
[SPEAKER_00]: comprehensive safety testing, given all the novel situations in which an autonomous system might find itself.
16:01
[SPEAKER_02]: Absolutely.
16:02
[SPEAKER_02]: Now imagine how hard it is to train a person to think through those situations.
16:05
[SPEAKER_02]: And all the mistakes that people make in war when people get angry and people get tired or there's a situation on the battlefield that somebody isn't prepared for.
16:14
[SPEAKER_02]: Yeah.
16:14
[SPEAKER_02]: And those definitively lead to challenges in war.
16:18
[SPEAKER_02]: One of the reasons, one of the main reasons the US military is the best in the world is because our people are the best in the world.
16:23
[SPEAKER_02]: What you're worried about is what if your data is poisoned, what if there's a scenario the algorithm hasn't encountered, what if there's pixelated camouflage in a way that distorts how accurate your algorithm is?
16:36
[SPEAKER_02]: Those are all things that you could know about and test for, but this in some ways is a question of how you build the failsafe on those kinds of systems.
16:44
[SPEAKER_02]: Like in theory, a smart way to build those kinds of systems would be, all right, what do you do if you're confused?
16:53
[SPEAKER_02]: in that kind of case, unless you are in like a, like the enemy is coming over the hills, like in Vade, it's like a different kind of situation.
17:02
[SPEAKER_02]: But I also think that in some ways, like this is a case where it's so hard for us to grasp some of these kinds of situations because we're so use used to the use of force scenarios that we encountered in the forever war.
17:16
[SPEAKER_02]: Yeah.
17:17
[SPEAKER_02]: Oh, like we have a week while an mq9 Reaper hangs around this like village waiting for like the one minute this militant like goes into a house and their family has gone and there's nobody else in the house and whatever and that's not what.
17:32
[SPEAKER_02]: that's not what the war that say like Russian Ukraine or fighting like looks like.
17:35
[SPEAKER_02]: Yeah.
17:36
[SPEAKER_02]: And we need to be ready for those kinds of conflicts.
17:40
[SPEAKER_00]: Yeah.
17:40
[SPEAKER_00]: Okay, so
17:43
[SPEAKER_00]: You sort of answered the question in two ways.
17:44
[SPEAKER_00]: I actually think I like the first one more than second one.
17:47
[SPEAKER_00]: The first thing they're both legit actually.
17:49
[SPEAKER_00]: The first one is humans are also confused in situations as well.
17:53
[SPEAKER_00]: And so then it's not a sort of monitoring issue or a fail safe issue.
17:59
[SPEAKER_00]: Then it's what's the benchmark here?
18:00
[SPEAKER_00]: It's the issue is not can we get perfection out of the autonomous system.
18:04
[SPEAKER_00]: The issue is can we get
18:06
[SPEAKER_00]: something like on average better results with the autonomous system than we can with the people.
18:10
[SPEAKER_00]: And since people like all sorts of mistakes, what we need to do is first measure how shitty people are in confusing situations.
18:17
[SPEAKER_00]: And then see how they reply, figure out how to measure such a thing, and then put autonomous systems in an at least simulate a situation that are like those and see if they perform better or worse than the humans.
18:27
[SPEAKER_00]: And then yeah, you might it might be the case that
18:30
[SPEAKER_00]: There are situations which are human would perform better than the autonomous system, but if on average, or something along those lines, the autonomous system outperforms the human in complex foggy confusing situations, then that looks like a pretty compelling reason to perform the autonomous system over the people.
18:47
[SPEAKER_02]: I would say that like for folks that are really worried about autonomous weapon systems, their concern is that a military's will accept performance that is not as good as humans.
18:57
[SPEAKER_02]: Yes.
18:58
[SPEAKER_02]: In some ways, like I think I have more faith in how at least a military like the United States would make those decisions.
19:03
[SPEAKER_02]: Yeah.
19:03
[SPEAKER_02]: And so the at least the center that like I would use would be a system that is as good or better than
19:10
[SPEAKER_02]: what a human could do in a given situation, which frankly is what military should be incentivized to use because again, if it's not that good, it's going to be less effective and then why would you use it?
19:20
[SPEAKER_00]: So here's, okay, so here's a reason.
19:22
[SPEAKER_00]: I'm not sure how compelling it is, but suppose I could see someone giving the following kind of argument.
19:27
[SPEAKER_00]: Yeah, look, it's not quite as good as people.
19:29
[SPEAKER_00]: We don't want to lose lives.
19:30
[SPEAKER_00]: Better to lose metal than it is to lose lives.
19:33
[SPEAKER_00]: Yeah, it's more inefficient.
19:35
[SPEAKER_00]: Yeah, it's not as good.
19:37
[SPEAKER_00]: That is impactful as humans, but I'll take the lower efficiency for greater lives saved if we could throw more money at it more AI at it to make up for its relative inferiority to people.
19:50
[SPEAKER_02]: That's totally reasonable, except I think this in some ways goes back to my belief that some of these like really complicated ground warfare scenarios where maybe you're like literally trading is a person knocking on the door.
20:04
[SPEAKER_02]: versus like some kind of machine or something like that, like where you have that kind of trade if that's the last kind of place you would use in its anonymous web and system on the battlefield.
20:12
[SPEAKER_02]: Simply because it would be such a confusing environment that it'd be difficult for those kind of systems to be as good or better than a human.
20:18
[SPEAKER_02]: Yeah.
20:19
[SPEAKER_02]: It's why we're more likely, I think, like when I say that I think that we've entered the era of precise mass and this is a world in which lots of people can now do this.
20:27
[SPEAKER_02]: Yeah.
20:28
[SPEAKER_02]: And a country like the United States has no choice somewhat, but to respond.
20:31
[SPEAKER_02]: It's because of the way that I think this plays out in more sort of conventional war environments when where everybody's in agreement that like, hey, the bad guys are over there.
20:41
[SPEAKER_02]: Like, we have to go fight.
20:42
[SPEAKER_02]: Yeah.
20:43
[SPEAKER_02]: I wish I realized it's like a little bit of a cop out to like the point that you're making, but I don't disagree with you.
20:48
[SPEAKER_02]: I just think those are the kind of scenarios in which we're at least likely to use those kinds of AI enabled capabilities.
20:55
[SPEAKER_00]: Let me ask you a different kind of question now, so we're focused on the case in which we're
21:03
[SPEAKER_00]: Ask, should we use the autonomous system or should we use a bunch of humans or something along those lines?
21:07
[SPEAKER_00]: Or just non-autonomous systems, those matter.
21:10
[SPEAKER_00]: And what have I asked a different kind of question that's about something like the set of autonomous systems?
21:16
[SPEAKER_00]: Maybe I'm worried about a much shorter word, or I think I might be worried about the scale of potential destruction or something along those lines or errors being amplified when a human screws up, the scale of the screw up is
21:29
[SPEAKER_00]: unless they have their figure on the button, relatively minor, but since AI acts at scale, it's going to do what it does at scale.
21:38
[SPEAKER_00]: I would think you might think that having a bunch of AI's is way more dangerous than a bunch of humans.
21:48
[SPEAKER_00]: And so the sort of it's not the kind of
21:58
[SPEAKER_02]: totally reasonable point.
21:59
[SPEAKER_02]: Okay.
22:00
[SPEAKER_02]: I think in some ways then that's not a question of an autonomous weapon.
22:03
[SPEAKER_02]: I would pivot a little bit to say that and I think your concern is not unreasonable that you could imagine a world in which if you have a bunch of different AI systems stacked on each other and you're not sure exactly how they interact with each other is that then essentially that would increase your error rate.
22:18
[SPEAKER_02]: Now, in theory, the testing of the systems would help you address that, what we're talking about then in some ways are there are a couple angles here like one is it sounds to me like what you were talking about essentially or AI decision support systems combined with what would be AI enabled weapons on the battlefield and the question then is like where is the human in the process and that the sure like I'm
22:45
[SPEAKER_02]: is one where there is a responsible and accountable human, which means you have trusted confidence that the systems are going to operate the way that they're supposed to.
22:55
[SPEAKER_02]: Yeah.
22:56
[SPEAKER_02]: I actually worry a bit about countries that are not the United States and that don't trust their people as much as we do, how the logic of control say in an authoritarian military,
23:12
[SPEAKER_02]: might lead to in some ways the kind of situation that you're worried about or you cut people out because like you didn't trust on the first place.
23:19
[SPEAKER_02]: So if you trusted your people in the first place, maybe you have a different kind of government, frankly.
23:23
[SPEAKER_02]: Yeah.
23:24
[SPEAKER_02]: And so there's like an authoritarian logic to that kind of layering that I'm like a little less worried about, and at least a military like the United States.
23:31
[SPEAKER_02]: Like that's like part one.
23:33
[SPEAKER_02]: I have to say part two is, if on the other hand, I totally misinterpreted your question and what you're really worried about is, but couldn't that swarm go out there and be a ton of mess and do really bad stuff?
23:43
[SPEAKER_02]: then the answer is in theory, I guess, but that thing just to me comes back to, well, how effectively is a thing been tested?
23:51
[SPEAKER_02]: Yeah.
23:51
[SPEAKER_02]: And do you actually have trusted confidence that it works the way that it's supposed to?
23:55
[SPEAKER_02]: And if you don't, you shouldn't be deploying it.
23:57
[SPEAKER_00]: Yeah.
23:57
[SPEAKER_00]: I guess my question in my head is really a structure of a question as opposed to a question in a way.
24:02
[SPEAKER_00]: What I have in mind are things like, so suppose we say, let's say, it's ethically permissible for single individual to, I don't know, toss a single piece of plastic into the ocean.
24:13
[SPEAKER_00]: or something along those lines, or to drive a car, and create the amount of pollution that a car creates.
24:18
[SPEAKER_00]: So that's the thing, if I'm going to say, is that ethical permissible for Mike or Reed to drive Michael or Reed to drive their car, but of course it is.
24:25
[SPEAKER_00]: But then you take this set of permissible actions and you wind up with an ethical outcome that is not good for anyone, let's say.
24:34
[SPEAKER_00]: So that's the case with pollution.
24:36
[SPEAKER_00]: There's lots of other kinds of cases where there's individual actions that are ethically permissible,
24:42
[SPEAKER_00]: add up to a sum of actions that has about impact.
24:45
[SPEAKER_00]: So I guess I'm just wondering if there's a kind of analogous case here where we say, yeah, look, it's ethically permissible in this case to use an autonomous system to shoot down those planes or to identify that building instead of a person.
24:58
[SPEAKER_00]: But I wonder if the set of ethically permissible actions of performing the autonomous system over the human leads to some situation that is analogous to climate change or something like that.
25:10
[SPEAKER_02]: I am less worried about that.
25:12
[SPEAKER_02]: I think in some ways, because I am less worried about the, in some ways, because I'm confident in the way in the ability of military bureaucracies to stymie like that kind of situation and put another way.
25:25
[SPEAKER_02]: From a US perspective, I'm much worried about the US going too slowly and adopting these technologies, then too quickly.
25:30
[SPEAKER_02]: Yeah.
25:30
[SPEAKER_02]: So I realized takes an abstract point that you made it makes it like a concrete about one military, which is actually so.
25:35
[SPEAKER_00]: I was like, what did you do?
25:36
[SPEAKER_02]: Which is like convenient for the point that I'm making, but by that what I mean is the, if you're talking about a bureaucracy where everybody's been trained their entire career to operate a certain way and to have humans involved in lots and lots of different parts of the process, I think that the incentives are more toward those individual use cases happening.
25:57
[SPEAKER_02]: Oh, there's I think the, in some ways the burden of proof is on the, and whether it should be this way is a different question.
26:05
[SPEAKER_02]: The burden of proof is on, I think, something like an autonomous weapon system in a military bureaucracy like the United States, because the bias in some ways is against it.
26:16
[SPEAKER_02]: And so I'm less worried about that kind of aggregation over time.
26:20
[SPEAKER_02]: Look, on the other hand, if what you're telling me is that the technologies and proved to the level where these systems are just fundamentally better,
26:30
[SPEAKER_02]: then humans in all of those individual cases, then the question becomes, then what kind of human oversight do you have over that system?
26:38
[SPEAKER_02]: Yeah.
26:38
[SPEAKER_02]: And if the human oversight of that system still means that if something goes awry, you can point to the, like, that commander said the mission was okay and that commander was supposed to, like, well, the U.S.
26:49
[SPEAKER_02]: Department of Defense calls like appropriate human judgment over the use of force.
26:54
[SPEAKER_02]: That as long as you have that, it's not clear that there's necessarily a problem there,
26:59
[SPEAKER_02]: The problem would be if you have unanticipated, lots and lots of unanticipated behaviors that happen in a way that nobody had planned for, yeah, where you went from, think about this is going from like zero to 60 as I've most of the crawl walk run.
27:13
[SPEAKER_00]: Yeah, okay, let me, I have two kinds of questions in mind.
27:17
[SPEAKER_00]: One is, do you think that, given that different,
27:24
[SPEAKER_00]: units of the military would be using different autonomous systems is there a kind of an orchestration problem or an orchestration issue that doesn't exist when we're not talking about autonomous systems because each different units are sending out different autonomous systems that are doing different things.
27:39
[SPEAKER_00]: And we may have tested each individual autonomous system and it's sufficiently safe, but we haven't tested it when it interacts with another autonomous system either of the same kind or a different kind.
27:51
[SPEAKER_00]: So one is there an orchestration problem which makes it, and if so, or just doesn't that just generally make controlling the AI's more complicated, then controlling non-autonomous systems.
28:03
[SPEAKER_00]: That's one question.
28:04
[SPEAKER_00]: The other question is
28:07
[SPEAKER_00]: these AI, these autonomous systems go through this really complex internal supply chain where
28:16
[SPEAKER_00]: One company or one unit or one team made this part of the AI system another part made this part of the AI system another part made this part of the system so it's just this complicated it's a very complicated whole.
28:28
[SPEAKER_00]: We're different people are responsible for different parts and this speaks to the accountability issue of okay so things go sideways how do we assign responsibility given that the thing is composed of all these parts.
28:41
[SPEAKER_00]: And for each part was made as the result of hundreds, thousands of decisions across dozens, hundreds of teams.
28:50
[SPEAKER_00]: So I guess those are two, two, two very different questions.
28:52
[SPEAKER_02]: Yeah, no, those are both really good questions.
28:54
[SPEAKER_02]: I think the first one, the, to me, you wouldn't want to use these systems in combination on the battlefield unless you had tested, unless you had tested beforehand.
29:04
[SPEAKER_02]: Yep.
29:05
[SPEAKER_02]: To the extent that, like, this is not a case of practice makes, like, perfect more like practice makes permanent, or like the way you practice is the way that you play.
29:12
[SPEAKER_02]: If you're going to use,
29:14
[SPEAKER_02]: systems that would interact with each other on the battlefield.
29:16
[SPEAKER_02]: You would certainly want to test that beforehand to understand how they interact.
29:20
[SPEAKER_02]: There might be some level of unpredictability there that's intrinsic, but that's also true in a non-autonomous world.
29:27
[SPEAKER_02]: Where you have, say, a marine unit, an air force unit, an army unit, all like converging on a spot on the battlefield.
29:34
[SPEAKER_02]: They all have maybe like slightly different orders.
29:37
[SPEAKER_02]: And you're counting on their understanding of the situation to de-conflect it and
29:42
[SPEAKER_02]: in many cases like what we're like macro often talking about here those human machine teaming.
29:47
[SPEAKER_02]: Not just like you're sending the robots.
29:50
[SPEAKER_02]: Yeah, the robots are like going to take the hill.
29:53
[SPEAKER_00]: Yeah.
29:53
[SPEAKER_02]: In that human machine teaming situation, you still have Marines out there.
29:58
[SPEAKER_02]: You still have.
29:59
[SPEAKER_00]: Sure.
30:00
[SPEAKER_02]: We still have folks out there.
30:01
[SPEAKER_02]: And so I think that help that the at least in theory helps address that along with the testing, along with in theory, like again, like testing how you would play.
30:10
[SPEAKER_02]: Yeah.
30:11
[SPEAKER_02]: Sorry, remind me the second question again.
30:13
[SPEAKER_00]: I will
30:15
[SPEAKER_00]: But first I want to say, yeah, I think your answer is a fair one because earlier we were citing that look, you want to compare the autonomous system against a human, what how poorly do humans act when they're confused because the AI or autonomous system act anymore poorly when it's called unquote confused.
30:30
[SPEAKER_00]: And so the relevant benchmark is not perfection, but human error rates, something along those lines.
30:36
[SPEAKER_00]: And what I was trying to draw attention to with the orchestration issue is,
30:41
[SPEAKER_00]: A kind of skepticism about our ability to comprehensively engage in safety assessments, or to rather to engage in comprehensive safety assessments.
30:51
[SPEAKER_00]: And your point was, yeah, what's the benchmark?
30:54
[SPEAKER_00]: The Royal Adventure Park is not.
30:56
[SPEAKER_00]: 100% comprehensive safety assessment because that's not even possible, right, cause any other area.
31:02
[SPEAKER_00]: So the question really, is it sufficiently comprehensive such that we have sufficient confidence to give a green light decision or something along those lines?
31:10
[SPEAKER_00]: And then what counts as sufficient is going to depend upon at least in part what counts as sufficient in a non-autonomous scenario as well.
31:17
[SPEAKER_00]: So the thresholds can say the same.
31:18
[SPEAKER_02]: And this is actually, I think an important distinction that matters
31:24
[SPEAKER_02]: the U.S.
31:25
[SPEAKER_02]: Department of Defense is sort of definition of an autonomous weapon system and policy on autonomous weapons systems.
31:31
[SPEAKER_02]: Yeah, part of a key part of that is something just about the technology.
31:34
[SPEAKER_02]: It's also at how you use it.
31:36
[SPEAKER_02]: You could lawble weapon out there, sure that, essentially, you're not in that, you could lawble weapon out there that ends up being an autonomous weapon system, even though it wasn't designed for that.
31:47
[SPEAKER_02]: There's a, this is essentially about where the technology meets your concept of operation.
31:52
[SPEAKER_02]: like how you plan to use the technology, and you need to plan both needs to be capable of being autonomous, and you need to actually plan to use it autonomously.
32:02
[SPEAKER_02]: So your second question, which I remembered, now, it was about essentially all the different things that go into a system and how you would then determine accountability or responsibility.
32:12
[SPEAKER_02]: To me, I'm gonna go back on to the question of what accountability looks like in that non-autonomous scenario.
32:17
[SPEAKER_02]: Yeah, so imagine say there's a helicopter trash.
32:19
[SPEAKER_02]: There's a tragic helicopter crash.
32:21
[SPEAKER_02]: couple of military personnel, a couple of military personnel die.
32:24
[SPEAKER_02]: There'll be an inquest.
32:25
[SPEAKER_02]: So there'll be a process try to figure out what happened.
32:27
[SPEAKER_02]: What are they trying to look for then?
32:29
[SPEAKER_02]: They're trying to look for.
32:30
[SPEAKER_02]: All right, did the pilots make a mistake?
32:33
[SPEAKER_02]: Did they get bad information about what was happening?
32:36
[SPEAKER_01]: Yeah.
32:36
[SPEAKER_02]: Was there a mechanical error?
32:38
[SPEAKER_02]: And then if there was a mechanical error, was it a mechanical error that was
32:44
[SPEAKER_02]: In some ways, acceptable, because there is some sort of error rate for whatever the system is, or maybe a mechanic should have caught it before the helicopter actually got in the air, or was this an error that no one had heard before, no one had prepared for, but the manufacturer should have known about and told people about.
33:03
[SPEAKER_02]: And I think you would end up going through the same process.
33:07
[SPEAKER_02]: for an AI enabled system on or close to the battlefield, which is if there was a tragic accident, you would then try to figure out, like, all right, did the system do what it was supposed to?
33:17
[SPEAKER_02]: And what it was supposed to do, led to say, like civilian harm, or did the system not do what it was supposed to?
33:24
[SPEAKER_02]: And if it didn't do what it was supposed to?
33:25
[SPEAKER_02]: Is that because it got bad data?
33:28
[SPEAKER_02]: in which case is that the fault of the system or is that the fault of whoever whatever got it the data or was it like an actual information processing sort of issues like on board like with a chip and like the the process of figuring out accountability which is already well established in cases of accidents and things like crashes or if a ship runs a ground or something like that.
33:49
[SPEAKER_02]: I think a military would go through the same process.
33:51
[SPEAKER_01]: Yeah.
33:52
[SPEAKER_02]: I think one of the things that ends up confusing about this or like difficult about this and why it's so important to have these kinds of conversations is that the folks that normally do that stuff, those are not AI experts.
34:03
[SPEAKER_02]: In the AI community are not like super well-trained in how militaries do accident investigations.
34:08
[SPEAKER_02]: Of course, yep.
34:09
[SPEAKER_02]: But when really like you need both of those bodies of knowledge then to be able to do that effectively.
34:13
[SPEAKER_02]: Yeah.
34:15
[SPEAKER_00]: I take it that this whole conversation started with the implicit idea
34:22
[SPEAKER_00]: Precision weapons are good, and they're good because they tend to harm the people that we think ought to be harmed and don't harm the people that we think ought not to be harmed.
34:31
[SPEAKER_00]: So there's something in favor, something strong in favor of precision weapons, and this is a kind of further manifestation of precision weaponry, something along those lines.
34:41
[SPEAKER_02]: Yeah, more or less we think about precision, the like when we tend to talk about precision from like an ethical or philosophical perspective, we talk about it more in terms of reducing collateral damage reducing civilian harm, but yeah, why did say the what why did the US military invent precision weapons in the first place, so they could hit targets more accurately.
34:59
[SPEAKER_02]: Sure, like the reason for precision was not like the OG for precision is not civilian harm the OG is if you can drop get the job one wrong to destroy a target as opposed to 10 or 100 like all the better right issue now is that the one bomb you drop is super expensive and if you could fire maybe 10 things.
35:19
[SPEAKER_02]: each of which costs like 5% of the cost of that one bomb using cheaper manufacturing and some like AI capabilities on board.
35:28
[SPEAKER_02]: Then now you're in a it requires thinking about the use of force differently, but if you can keep that accuracy, but now generate.
35:37
[SPEAKER_02]: the same effect at lower cost by firing a few more things, then there's potentially opportunity there and the rest of the world is doing that.
35:45
[SPEAKER_02]: It's not like we have this and nobody else does, which was like more like the case we thought about like in the 90s.
35:52
[SPEAKER_00]: So I take it that
35:54
[SPEAKER_00]: Most of the objections are most of the considerations that we would talk about so far have to do with something like are you sure you can have sufficient control over the thing are you sure that This isn't going to go totally sideways in an unexpected way or it's like we're not sure enough and your point is constantly been to say
36:12
[SPEAKER_00]: Yeah, look, there's plenty of empirical reason to think that it'll be tested in all the ways that all the other things have been tested and it'll meet the appropriate risk thresholds and blah, blah, blah, blah, blah.
36:21
[SPEAKER_00]: That's been the point so far.
36:22
[SPEAKER_00]: There's another line of argument, which I know you're familiar with and I'm pretty sure I know what you're doing to say to this, but I'll ask you anyway.
36:29
[SPEAKER_00]: There's some people who are
36:32
[SPEAKER_00]: I think there's just a pose to an autonomous system killing people, something on those lines, that it's somehow contrary, you can see that I don't really get the thrust of this argument.
36:44
[SPEAKER_02]: So I might ask you to, oh, there's something like inherent human dignity that is violated by an autonomous weapon system.
36:50
[SPEAKER_00]: I don't, I'll just, I should say, there's a yuck factor that I understand, but,
36:58
[SPEAKER_00]: The reason that I don't really feel the pull is precisely because of something that you said earlier, which is, look, we're still programming the AI to act in this way in such and such contexts.
37:09
[SPEAKER_00]: And it's still, if you like, our decision.
37:12
[SPEAKER_00]: for it to kill person X and situation Y. So when the autonomous system does it, it's not, as you said earlier, making a decision to kill someone.
37:22
[SPEAKER_00]: I don't, so is that your view something like, yeah, I think that's basically right.
37:26
[SPEAKER_02]: I also think that the sort of, what I would call maybe the human dignity objection to somebody like an autonomous weapon system.
37:33
[SPEAKER_02]: I think there's two issues I would call out, maybe in that context beyond what you said.
37:38
[SPEAKER_02]: Yeah, one is that the systems, if they work effectively and this is to be clear is like a very like utilitarian like argument that it's not clear to me if what we are talking about is fighting a war where like things are destroyed and people die.
37:53
[SPEAKER_02]: It's not clear to me that there's something like inherently wrong there, especially if there's still a human that is responsible for the use of force.
38:02
[SPEAKER_02]: If you pull the human out completely from responsibility, then that's a different kind of situation.
38:08
[SPEAKER_00]: But then the objection would be the person who decided to pull the human out altogether.
38:12
[SPEAKER_02]: And then we could always respect accountability in some way if like I really wanted to.
38:16
[SPEAKER_02]: But I also think that there's sometimes a failure to understand how autonomous modern weapons are already.
38:23
[SPEAKER_02]: So there's an air missile called the A120 or the M-RAM.
38:28
[SPEAKER_02]: A very stylized description of how an M-RAM works essentially is, you know, imagine like you're a pilot, you're flying a plane, and over the horizon, so you can't see it.
38:39
[SPEAKER_02]: You get a ping that you believe is an adversary,
38:46
[SPEAKER_02]: You then press a button.
38:49
[SPEAKER_02]: You fire the missile.
38:50
[SPEAKER_02]: When the missile gets close enough, it opens up a seeker.
38:55
[SPEAKER_02]: That seeker is not like an algorithm program with a target library or something.
38:59
[SPEAKER_02]: It's got the maybe it's like radar seeking in one, but it's got some, imagine like dumb smart technology.
39:06
[SPEAKER_00]: Essentially.
39:06
[SPEAKER_00]: It's like photo recognition software.
39:09
[SPEAKER_02]: So it recognizes that not, a photo recognition would be much more than the way.
39:15
[SPEAKER_02]: your average like air to air weapon would have.
39:17
[SPEAKER_02]: Okay.
39:17
[SPEAKER_02]: And then so imagine like your sensor open.
39:19
[SPEAKER_02]: So let's just for simplicity.
39:20
[SPEAKER_02]: So let's call it like say it's like radar guided.
39:23
[SPEAKER_02]: The adversary plane has its radar on.
39:25
[SPEAKER_02]: You turn on your seeker like the missile turns on its seeker radars like on in the pilot who fired the missile like watches as their missile then like watches like the blip of their missile hit the blip of the plane.
39:37
[SPEAKER_02]: Yeah, sure.
39:38
[SPEAKER_02]: And then there are no blips.
39:43
[SPEAKER_02]: the adversary.
39:44
[SPEAKER_02]: Yeah, like, didn't know for sure, except that like they know that they're like fighting a war and that there's it's like legal for them to fire.
39:49
[SPEAKER_02]: Yeah, and for them to fire a weapon.
39:52
[SPEAKER_02]: Yeah, the maybe if they had figured depending on how exactly the system works and what the like rules of engagement are.
39:58
[SPEAKER_02]: Yeah, maybe if they at the last minute, if they discovered that it wasn't actually an adversary, they could call off the missile.
40:04
[SPEAKER_02]: Yeah, maybe not.
40:06
[SPEAKER_02]: Frankly, and so there's a lot of autonomy built into that system and the judgment role of the human in some ways is one of trust in the system and they're training to know when you're supposed to fire or not.
40:20
[SPEAKER_02]: And like that in some ways is the special sauce in a way that I don't think goes away.
40:26
[SPEAKER_02]: even with most most of what the like realistic kinds of autonomous weapons, yeah, that you would use on the battlefield.
40:34
[SPEAKER_02]: But when you put it in that context, I think this seems a lot, it could be scarier, not scary, but it's a lot less different.
40:42
[SPEAKER_02]: Yeah.
40:42
[SPEAKER_02]: Then the sort of semi-autonomous systems that are deployed with regularity today.
40:48
[SPEAKER_00]: It feels like
40:50
[SPEAKER_00]: your response to the objection is that the person who lobbies the objection has a mental model of what warfare looks like that's unrealistic.
40:57
[SPEAKER_00]: Fundamental.
40:58
[SPEAKER_00]: Yeah.
41:00
[SPEAKER_00]: Okay.
41:02
[SPEAKER_02]: Yeah.
41:02
[SPEAKER_02]: I mean, look, I'm just being like, like, blunt and honest, but he has a short think about it though.
41:06
[SPEAKER_02]: If you look at the, and I'm sure somebody will write in to complain about this, the point that I'm about to make.
41:12
[SPEAKER_02]: If you think about where there's an NGO group called the campaign to stop killer robots,
41:19
[SPEAKER_01]: Yeah.
41:19
[SPEAKER_02]: If you look at where the campaign to stop killer robots came from and where some of the countries that were first describing concerns about some of these capabilities were coming from, they were really concerned with drone strikes, which were remotely piloted.
41:34
[SPEAKER_02]: And in some ways, the they review that some of those strikes were legal, they're like, they'll be even worse.
41:40
[SPEAKER_02]: If that MQ9 Reaper,
41:44
[SPEAKER_02]: was just had the algorithm.
41:45
[SPEAKER_02]: It was like, oh, like, that person's a bad guy.
41:47
[SPEAKER_02]: Now we can kill them.
41:48
[SPEAKER_02]: What started this in some ways was a or started like some of this conversation was a very specific.
41:56
[SPEAKER_02]: use case in some ways that folks objected to, but the cat was out of the bag already on drone strikes.
42:03
[SPEAKER_02]: And they were looking at what was next.
42:06
[SPEAKER_02]: That would be of greatest concern.
42:08
[SPEAKER_02]: Yeah.
42:09
[SPEAKER_02]: I think one of the issues is that it turns out that the kinds of use cases that are most likely for autonomous weapon systems are actually not the one.
42:18
[SPEAKER_02]: that initial NGO folks had in mind, the actual use cases are much more like conventional like something in the Indo-Pacific between the US and China or something between Russia and Ukraine rather than like a tax against militant groups.
42:35
[SPEAKER_00]: Yeah.
42:36
[SPEAKER_00]: The next one last question I think, which is something like, where does all this end or maybe end is too ominous?
42:43
[SPEAKER_00]: Where does all this go?
42:47
[SPEAKER_00]: Let's say the people of the campaign to stop killer robots.
42:49
[SPEAKER_00]: I imagine that something that they would say is something like, if all the country start using all these autonomous systems, things are going to go just super sideways.
43:01
[SPEAKER_00]: I imagine they're thinking existential risk kind of stuff.
43:04
[SPEAKER_00]: So it's just,
43:06
[SPEAKER_00]: One milled hair will throw their robots at the other military will throw their robots and it's going to be a war of the robots against robots and this is just going to lead to an insane level of destruction.
43:17
[SPEAKER_00]: But you add to that things like
43:20
[SPEAKER_00]: The speed at which AI makes decision quote-unquote makes decisions is insanely fast.
43:26
[SPEAKER_00]: We can't possibly monitor it, not the decision making in real time, given the speed.
43:32
[SPEAKER_00]: If especially if it's interacting with other AI's, including enemy AI's, the speed of warfare, has to increase at something like an exponential rate, or at least a rate that's too fast for us to really
43:43
[SPEAKER_00]: audit in real time, or monitor in real time, and what about that?
43:48
[SPEAKER_00]: And this actually speaks back to my pollution kind of case.
43:51
[SPEAKER_02]: If that's the best version of your pollution argument.
43:56
[SPEAKER_02]: So what's the response to the best version the I would say two things one I actually wrote a whole article about how if one was going to be worried like that would be the thing to be like a little more about which essentially the speed trap of too much autonomy on the battlefield, but I think in some ways that is less an autonomous weapon system issue it's more if AI decision support becomes AI decision making.
44:19
[SPEAKER_02]: I think the issue is, or not the issue, the reason why I am personally less worried about that, like at this point, I don't think that's like coming to a movie theater near you, or maybe it's only coming to a movie theater near you.
44:31
[SPEAKER_02]: Yeah, in the short term, or even potentially in the medium term.
44:35
[SPEAKER_02]: There's like a lot to work out with how these systems actually operate.
44:38
[SPEAKER_02]: Yeah, I think we're a lot more likely to see, at the separate, think about a weapon system where the tactical level of war,
44:46
[SPEAKER_02]: And the kind of risks that you're talking about involve more if you were overseeing lots of different forces on the battlefield, more the operational level of war, the to totally turn that over to an AI would imply a degree of confidence in the performance of that system that so far exceeds.
45:05
[SPEAKER_02]: what existing sort of technology, what existing technology like looks like if that technology is so good that it's way better than humans from all the testing and evaluation than hey, maybe we should just praise our robot overlords.
45:18
[SPEAKER_02]: I think we are less likely functionally to see that on the battlefield one caveat.
45:24
[SPEAKER_02]: I do worry about those kinds of risks more as I and I preview this before when it comes to more autocratic countries because I think that due to the their like lack of what is
45:40
[SPEAKER_02]: often, not always, you know, what in some ways isn't intrinsically lower level of trust in their people.
45:46
[SPEAKER_02]: Sure.
45:47
[SPEAKER_02]: And in their personnel, the like logic in a country like the United States and most democratic countries is human machine teaming.
45:54
[SPEAKER_02]: Like our people are great.
45:55
[SPEAKER_02]: We use technology to make our people better.
45:57
[SPEAKER_02]: Maybe even there are fewer people.
46:00
[SPEAKER_02]: but the role of technology is to enhance especially when it comes to making choices or decisions.
46:05
[SPEAKER_02]: Yeah, the role of technology to enhance the ability of the people to make a decisions.
46:10
[SPEAKER_02]: If you don't trust your people, you might be more likely to cut more and more of them out of the decision-making loop.
46:14
[SPEAKER_01]: Yeah.
46:15
[SPEAKER_02]: Which is actually why like the thing I worry about the most in this kind of scenario is actually not,
46:20
[SPEAKER_02]: like killer robots run a rye on the battlefield in a way that leads to the flash crash war sort of scenario.
46:28
[SPEAKER_02]: It's more in autocratic country using autonomy to cut people out of the loop for purposes of like mass domestic repression and destruction.
46:38
[SPEAKER_02]: Like the and that one of the key checks against the ability of autocrats to essentially like murder lots of their citizens historically is the army eventually says no, but if you can cut more and more people out of that,
46:49
[SPEAKER_02]: Like that to me is a scenario I'm much more worried about than, like conventional war, like autonomous weapons systems interacting and then like disaster strikes.
47:01
[SPEAKER_00]: Yeah.
47:02
[SPEAKER_00]: I hear you on the totalitarian state clamping down on citizens, I guess when it comes to that other scenario that you're less worried about.
47:13
[SPEAKER_00]: I could see someone giving a kind of slippery slope argument, but you know, this isn't that that's not this and that, you know, and then we're in a hellscape, so then my question to you with something like, what would be a plausible milestone for us to look out for a restaurant like, okay, we've hit this point in the development, maybe it's going to be in five years, maybe it's going to be in 10 years, maybe it's going to be in 50 years.
47:32
[SPEAKER_00]: I don't know, but here's a milestone where we would want to say hold on a second, have we slipped too far down this low, what would that milestone look like?
47:41
[SPEAKER_02]: I will say just like philosophical, yeah, I don't love slippery slippery events, but the like if the but to play along because why not the I think you could reject a whole thing.
47:49
[SPEAKER_00]: It's not a slippery slope.
47:50
[SPEAKER_00]: It's already a hard and fast line.
47:52
[SPEAKER_00]: I don't think it's a really relationship here.
47:54
[SPEAKER_02]: I think I get the point that you're making in the US military.
47:58
[SPEAKER_02]: There are combatant commands, like Indo-Pacific commands, like northern command, like European command, like, like, actually, like, like, African command, etc.
48:06
[SPEAKER_02]: Sure, central command, like covering Middle East.
48:08
[SPEAKER_02]: If you had a situation in which the judgment of the commander of, say, Indo-Paycom,
48:15
[SPEAKER_02]: was no longer the primary factor, making decisions surrounding military operations.
48:21
[SPEAKER_02]: And instead, it was essentially like an AI, the commander's just like, let her just pressing a button agree or not.
48:27
[SPEAKER_02]: Yeah, that's a situation in which, like, I've been bound to run a class.
48:31
[SPEAKER_01]: You need to do that.
48:31
[SPEAKER_02]: Unbelievable.
48:32
[SPEAKER_02]: Where I said, if you were going to be worried about that, that's like the point in which I would have a little more, might have a little more concern.
48:38
[SPEAKER_02]: I just don't think that's the vision for how at least a country like the United States will use these kinds of systems.
48:44
[SPEAKER_00]: Yeah, okay.
48:44
[SPEAKER_00]: So let me just slide me free.
48:45
[SPEAKER_00]: There are people who say, Oh, there's going to be an AI CEO of a company.
48:50
[SPEAKER_00]: And so your point is something like in the military context, that would be the milestone of the red flag or the holy shit moment.
48:57
[SPEAKER_00]: But secondly, while the odds of that happening in a business context are not that slim in a way.
49:03
[SPEAKER_00]: Some people are claiming to have it in the military context.
49:06
[SPEAKER_00]: It's extraordinarily unlikely.
49:08
[SPEAKER_00]: So there's a milestone and it's extraordinarily unlikely that will ever hit it.
49:12
[SPEAKER_00]: Yeah, I think that's probably for cool.
49:15
[SPEAKER_00]: Michael, this has been super fascinating.
49:16
[SPEAKER_00]: Thanks so much for coming on and thanks for defending the killer robots because if you don't want to know for the killer robots, who will somebody has to somebody has to exactly this is what I'm just me there.
49:26
[SPEAKER_02]: I don't it's not staying up for the killer robots.
49:28
[SPEAKER_02]: I got arguments against them.
49:31
[SPEAKER_02]: Yeah, yeah, just.
49:32
[SPEAKER_02]: not based in the reality of how military technology and military process works.
49:38
[SPEAKER_00]: Yes, no.
49:38
[SPEAKER_00]: I hear that point loud and clear and I'm sympathetic.
49:40
[SPEAKER_00]: I'm launching objections, but I'm broadly sympathetic to the view.
49:44
[SPEAKER_00]: And so thanks for joining me in the pro killer robots camp.
49:47
[SPEAKER_00]: Yeah, we'll do the other non-profit the campaign for killer robots camp and for killer robots.
49:52
[SPEAKER_02]: I don't know that doesn't really roll off the tongue in this.
49:54
[SPEAKER_00]: I think it's got a nice ring to it, ominous, but it's a ring.
49:56
[SPEAKER_00]: Yeah.
49:57
[SPEAKER_00]: Thanks again.
49:58
[SPEAKER_00]: Absolutely.
49:59
[SPEAKER_00]: Thanks a lot.
50:01
[SPEAKER_00]: Thanks everyone for listening.
50:03
[SPEAKER_00]: I hope you enjoyed the episode, and I want to thank a few people who make this all possible.
50:07
[SPEAKER_00]: Delano Alexander, he's my producer, he does such an amazing job on everything that he touches.
50:11
[SPEAKER_00]: Garrett Lang is the sound engineer.
50:13
[SPEAKER_00]: That's honestly, he's just, he's too good for me.
50:15
[SPEAKER_00]: Off me a gyro on his joined us regionally, he's helping us find the best guests to bring on to this podcast.
50:21
[SPEAKER_00]: And to actually watch the Lanko help us organize all this, get people on the schedule, get people's bios, headshots, and really keeps it straight on the track.
50:27
[SPEAKER_00]: So thanks guys, really appreciate all the hard work you do.
50:41
[UNKNOWN]: Thank you.