Latest / The Road to Accountable AI / Michael Horowitz, UPenn: Governing AI That's Designed to Kill

Transcript

Michael Horowitz, UPenn: Governing AI That's Designed to Kill · The Road to Accountable AI

0:09
Werbach: Hi, I'm Kevin Warbeck, professor of Legal Studies and
0:11
Business Ethics at the Wharton School of the University of Pennsylvania.
0:15
For decades, I've studied emerging technologies from broadband to blockchain.
0:19
Today, AI is promising to transform our world, but AI needs accountability
0:26
mechanisms to ensure it's developed and deployed in responsible,
0:29
safe, and trustworthy ways.
0:32
On this podcast, I speak with the experts leading the charge for accountable ai.
0:41
Philanthropics dispute with the Pentagon and recent military conflicts around
0:45
the world have brought attention to autonomous weapon systems.
0:49
What does accountable AI mean when AI is part of systems whose
0:52
primary function is to kill?
0:55
Fortunately, one of my colleagues at the University of Pennsylvania is among
0:58
the world's top experts on this topic.
1:01
Michael Horowitz is a political science professor at Penn, director of Perry
1:04
World House and a former senior official at the US Department of Defense, where
1:08
he was the principal author of the US Political Declaration on Responsible
1:12
Military Use of AI and Autonomy.
1:16
Horowitz: Mike, pleasure to have you on the podcast.
1:18
Thanks so much for having me.
1:21
Werbach: First, uh, can you talk about just how aggressively the
1:25
US military and other militaries are leaning into AI today?
1:30
Horowitz: It's a, it, it, it's a good question and I think the, the way I
1:34
would think about artificial intelligence from a military perspective is to
1:38
put uses of AI into three bucket.
1:42
The first are uses of AI that essentially are the same as the way a private sector
1:48
company or a university or something like might use artificial intelligence.
1:52
So think about this in terms of hr, payroll processing, basic logistics,
1:58
like business functions essentially that like even for the Pentagon, don't differ
2:02
like that much from what you know and what, what some other business might do.
2:06
So that's bucket one.
2:07
Bucket two is what I would call intelligence surveillance
2:11
and reconnaissance.
2:12
And that is all of the vast information that the US military gets from all the
2:16
satellites that are up in space, all the airborne sensors, maybe information
2:21
from, from human intelligence sources, information from battlefields,
2:25
all sort, all of that data and the volume of that data is overwhelming.
2:31
And so for, for many years, I mean, this isn't a like.
2:34
Today thing, it's more like a decade ago story.
2:37
The militaries have been interested in using older AI tools like computer
2:41
Vision to try to process all of that data, extract the signal from the
2:46
noise, get accurate information faster, and so that's just like the dial's
2:50
just been turned way up on that.
2:51
Now in the current period, that's bucket two.
2:54
Bucket three is closer to the pointy end of the spear.
2:59
And that in their two like little subcategories sort of there.
3:03
The, the first is what are called AI decision support, uh, systems, sort of
3:08
like a Maven Smart system, which is a tool that, a dashboard essentially that was
3:14
developed by a company called Palantir.
3:17
And that's been in the news a bit, a bit in the news a bit, uh, lately.
3:20
Uh, 'cause Palantir is essentially, the Palantir dashboard is available
3:24
for every senior US military leader, uh, around the world.
3:28
That's helping them essentially process information and do their
3:31
operational workflow faster.
3:33
We, we can get into that if you want.
3:35
And then there's the far like pointy end of the spear in a category of what, what
3:39
are called autonomous weapon systems.
3:41
And those are the use of artificial intelligence or frankly, simpler
3:46
autonomy to select and engage targets without human oversight, you know, after
3:52
being activated by a person, but where essentially you would have an algorithm.
3:57
Saying, say you have an algorithm trained on a data set of Russian tanks
4:02
or something, and, uh, in targeting them, so we have those three buckets.
4:06
And so military see lots of advantages to this.
4:08
They're all the same kind of efficiency process, advantages that you might see
4:12
in the corporate world, but then in a, in a world where making better decisions
4:17
and making faster decisions, better can, can be the difference between
4:20
winning and losing on the battlefield.
4:22
There are lots of advantages to militaries in effectively incorporating ai.
4:28
Werbach: So let's put the first bucket to the side.
4:30
It's, it's important to mention that obviously the, you know, the
4:32
Pentagon is a huge, maybe the biggest enterprise and, and all of those
4:36
use cases are, are significant for
4:37
ai.
4:37
Horowitz: So big.
4:37
It's one of the largest office, but the Pentagon itself is one of
4:40
America's largest office buildings.
4:41
Essentially.
4:42
Like people, you know, people like, people know that the military is big, but it's
4:45
a huge bureaucracy, a lot of efficiency.
4:47
You can.
4:48
Sorry.
4:49
Werbach: So all that's critically important, um, and all the
4:53
issues that we talk about with AI governance and the private sector
4:56
are going to be relevant there too.
4:57
The, the second two, though, seem to be potentially distinctive.
5:01
And what I wanna ask about is, um, you know, with this shift from, uh, predictive
5:07
machine learning to generative ai, we're now dealing with systems which first of
5:11
all are, are really at the cutting edge.
5:13
We don't really understand how they work.
5:15
You know, they're, they're non-deterministic.
5:17
They, you know, they have this inherent error, uh, capability or, or possibility.
5:21
Um, how much concern is there in the military about deploying those kinds
5:26
of technologies in these use cases?
5:29
Horowitz: I think, I mean, and, and I'm, I'm most familiar with the American
5:32
military since I spent a couple of years working in the, I've done a couple
5:35
of different since in the Pentagon, but most recently from 2022 to 2024.
5:39
And I think there is a both simultaneous recognition that it
5:43
kind of has to happen like this.
5:45
Like mm-hmm.
5:45
This is inevitable.
5:47
Like either you're for it or you're gonna get run over, but
5:50
also some concern about it.
5:52
And, and by that what I mean is militaries are really little sea conservative
5:57
institutions, and they're really, and they're little sea conservative
6:00
institutions because the decisions that they make lives are on the line.
6:04
Like literally.
6:05
I mean, it's the, it's the job of the military to, you know,
6:07
essentially to kill people.
6:08
Like the business of the Pentagon is the business of war.
6:11
And because of that, there's no institution in some ways more
6:16
incentivized to ensure their technology works appropriately than the military.
6:21
Because when they get it wrong, people die and they can die.
6:26
That leads to some natural conservatism when it comes to
6:29
integration of emerging technologies.
6:31
Now it's like, that's like a little bit different if you're
6:33
like being invaded the way that Ukraine is or something like that.
6:35
But in, but in, in general.
6:38
And so there is, uh, that sort of natural conservatism like, and especially with
6:42
in the context of these large language model tools, if there they're black boxes,
6:46
you know, if they're not simply like predictive either for vision or text.
6:50
How do we know if they're working?
6:52
How do we know if we can trust them?
6:54
How do we know if they're reliable?
6:55
That leads to then some questions about, alright, like what are the kinds of
6:58
situations where you can trust them and you can, what you try to do then is
7:03
prove that they're reliable at a certain level so that even if you're not a
7:08
hundred percent sure how it's operating you, you understand how reliable it is.
7:13
But, and so the, and there's then an interest in, in seeking to take these
7:18
tools because they're so powerful and throw them at especially large, uh,
7:22
data sets and complicated problems, and especially operational workflows
7:27
that are still almost entirely analog to try to speed those things along.
7:31
But it's balanced by that kind of conservatism.
7:35
Werbach: Hmm.
7:36
Alright, so let's, let's talk about the, the pointy end of the spear, as you
7:39
say, the autonomous weapons where you, you know, as much about this as anyone?
7:43
Uh, I think
7:45
Horowitz: for
7:45
Werbach: better or worse, and there's been, there's been a lot of discussion
7:47
with Philanthropics dispute with the Pentagon, where one of their red lines was
7:52
use of Claude for, uh, autonomous weapons.
7:54
Um, you know, how, how, first of all, how, how worried should we be that the
7:59
killer robots are, are here or they're coming, uh, you know, how, how widespread.
8:03
Is this practice, um, how much is it a hypothetical today
8:07
versus a reality say for the us?
8:11
Horowitz: Don't worry about the killer robots, I promise.
8:13
The, I mean, or I mean whatever.
8:15
You can worry.
8:15
Here, here, here's, here's the thing.
8:17
So there's, uh, there're two different kinds of weapon systems I wanna
8:21
differentiate between semi-autonomous weapon systems are are weapon systems
8:26
where a human aims them at a target.
8:29
Then the weapon system can essentially go hit the target basically on its own.
8:33
These are all the precision weapons that the US military militaries around the
8:36
world and sort of used for decades, and that we think are both more effective
8:39
and reduce collateral damage in civilian harm because they're just way more
8:43
accurate than say like World War II area bombing or, or something like that.
8:47
So you have semi-autonomous weapons, and within that category you have
8:50
some of them, like say the Tomahawk Missile that's been in the news because
8:54
of the US using it against Iran.
8:56
There's essentially a human, almost like riding shotgun on
8:59
it up to the point of impact.
9:00
'cause it's GPS guided, you know, maybe you have real-time satellite,
9:03
you've got like some access to information about how it's going.
9:06
There's also this whole category though of, of semi autonomous
9:10
weapons with no human supervision, like a radar guided missile.
9:13
For example, a pilot sees a ping and they're flying an aircraft.
9:17
They see a ping that says there's an enemy radar nearby.
9:20
They launch a a radar guided missile and that.
9:22
It goes for a certain distance, it turns on a seeker, it detects
9:26
the radar and it hits the radar.
9:27
That radar might be on top of a school.
9:29
That radar might be on top of a hospital like the missile doesn't know, and
9:33
like that's, that kind of weapon has essentially been around since the
9:36
seventies or early eighties, and dozens of countries like have that kind of weapon.
9:41
And that's only a semi-autonomous weapon system.
9:43
And then we have autonomous weapon systems.
9:45
And again, they could be human supervised or non-human supervised.
9:48
And here's the rub.
9:51
For more than 40 years, the United States military has deployed a system
9:55
called the Close in Weapon System.
9:57
It's an enormous Gatling gun that basically sits on US Navy ships to
10:01
protect it from a large, you know, large numbers of attacks coming in.
10:05
It has an automatic mode.
10:08
Where if an, if there are too many threats coming in for an operator, even to point
10:12
and click on, the operator can turn on the automatic mode, and by algorithm it
10:17
can select and engage targets with the hu.
10:19
The Huey could turn it off if they want, but even has nothing to do and that and
10:22
that, and it's been publicly acknowledged that system has been used in the red seat.
10:27
In the last five years to protect, you know, US ships
10:29
from, uh, from Houthis attacks.
10:31
And, and again, this is an algorithm selecting and engaging targets on its own.
10:36
And that's tech, the US military.
10:38
And by the way, 40 militaries around the world have been using since the eighties.
10:43
And so the, what's new in the context of the philanthropic conversation
10:47
in some ways is this potential category of autonomous weapon
10:50
systems with no human supervision.
10:53
There's some antecedent stuff that we can talk about as well.
10:56
Then especially this question of what kinds of algorithms are most
10:59
appropriate and are, um, large language models ready for the task of
11:05
guiding an autonomous weapon system?
11:07
I'll be honest, I don't really think they're there yet.
11:09
Like if, if you were gonna design an autonomous weapon system, the last
11:13
thing that you would use is like Claude trained on the slop of the internet.
11:17
Like before you did that, like you, what you really want
11:20
is say a bespoke algorithm.
11:22
Trained on a very particular data set.
11:24
Like these are Chinese fighters.
11:26
These are Russian tanks using like computer vision essentially to detect,
11:30
you know, something gets in an area like turns on, essentially the turns
11:35
on its sensor and it's like, oh, that is a thing I've been trained to hit.
11:38
I can go hit that and, and that is not a thing I've been trained to hit.
11:41
I won't go hit that.
11:42
Mm-hmm.
11:43
That's pretty determined, you know, that that's a far way different than like rock
11:47
or quad or like chat GPT or something.
11:50
But those, those things can exist and like frankly, have
11:52
existed for like quite a while.
11:54
We just haven't really talked about them.
11:57
Werbach: Um, well, ironically, that's one of the big arguments that philanthropic
12:01
is making, why they don't want Claude to be used in these use cases that they
12:05
don't believe it's accurate enough.
12:07
Horowitz: I mean, I, I don't, I don't think they're wrong about that now.
12:10
I mean, I think that the, it, I, I, I do think it's interesting and, and
12:13
happy to talk about the, the dispute a little bit in that, you know, and
12:18
Andro on Anthropic, CEO on the, the Thursday evening before they got labeled
12:23
a supply chain risk, you know, released what's probably the longest statement
12:26
they have ever, uh, had on, uh, on autonomous weapon systems in particular.
12:32
And by the way, public service OU announcement.
12:34
The correct term of art from a US government perspective
12:37
is autonomous weapon system.
12:39
When philanthropic talks about fully autonomous weapons, I think it generates
12:43
confusion sometimes because it's not exactly clear when they say that,
12:47
like what kinds of systems they're talking about and, and whatever.
12:51
There's like endemic terminological issues in this area, just like, like
12:54
in any, just like in any area, but like this is one where it actually
12:57
caused some misunderstanding between philanthropic and the Pentagon.
13:00
But what philanthropic actually said is they don't have a philosophical
13:03
objection to autonomous weapons systems.
13:05
They don't even have a philosophical objection to large language models being
13:08
used for autonomous weapons systems.
13:09
And they're willing to work with the Pentagon on making them work.
13:13
They just don't think they're ready now.
13:15
And like that is the tragedy of this dispute in some ways.
13:19
To me, like there's no use case that Anthropic is currently
13:23
doing for the Pentagon where the either side is upset about.
13:27
There is no ask the Pentagon is made of anthropic that Anthropic won't
13:31
do yet they've now been labeled like a, like a Chinese company
13:36
as a threat to national security.
13:37
Like, how did we get here?
13:39
Werbach: Yeah.
13:40
Uh, no, no.
13:41
There's a lot, a lot to be said about that and I don't think most people
13:44
realize that, that anthropic is the only, or, or the first of the large
13:48
language battle companies that's even available on these classified systems.
13:51
It wasn't that they were resisting these military uses.
13:54
Horowitz: Yeah, I mean the, you know, the, the AI companies had been
13:57
doing some unclassified work, like poking around the edges for a while.
13:59
But this is an, you know, you know, I think lots of people know the story.
14:02
You know, a story about Silicon Valley was funded by defense dollars, like way
14:06
back in the day, like during the Cold War.
14:08
But the AI companies that are now the, you know, like titans of industry in
14:12
some ways, and this, this like generation of American, of American business, you
14:16
know, they didn't start as companies that thought they were going to be
14:19
doing business with the military.
14:21
But as advances in artificial intelligence have increasingly made clear that it's
14:26
a general purpose technology that's gonna be everywhere, like everywhere
14:29
in the economy, everywhere in society, that includes militaries then.
14:32
Then you have all of these companies that never thought of themselves
14:36
as likely to do business with the American military or in China, the
14:39
Chinese military, like whatever, who.
14:41
All of a sudden you're building tech that the government's like,
14:43
this is really important, like we need this for national security.
14:47
And philanthropic was the first one in the door willing to not just do
14:51
unclassified work but to do classified work, which is why it's so like tragic
14:55
in some ways from a US national security perspective that this feud is happening.
15:01
Werbach: So let's get back to autonomous weapon systems specifically.
15:04
Um, what kinds of, uh, frameworks or agreements are there today for
15:12
militaries to, to use these technologies?
15:15
Horowitz: It's a good question.
15:15
So there's no, the, the place that I would start is, and, and that I
15:20
think it's left out of the public conversation a lot, is that due
15:23
to international humanitarian law.
15:26
Requirements to to have the use of force have distinction where you're
15:30
distinguishing between say, a military or civilian and proportionality or using
15:34
force and proportion to like how much you need to, like, et cetera, et cetera.
15:38
There is an implied requirement there that the US military firmly believes it
15:43
exists for human responsibility over the use of force and the requirement for human
15:48
responsibility over the use of force.
15:50
Exists whether you are F, whether it's a bow and arrow, whether it's a machine
15:54
gun, whether it's a radar guided missile, whether it's an autonomous weapon system.
15:58
If you start from there, I think the pieces actually start to fall into place
16:01
a little bit because the thing that people sometimes worry about in the context
16:05
of autonomous weapon systems is that there's going to, that there's no good,
16:09
there's gonna be an accountability gap.
16:10
There's no person responsible or accountable.
16:13
In some ways, if you ended up in that situation, you would
16:16
already be talking about a use of force that would be illegal.
16:18
Mm-hmm.
16:19
From the perspective of the American military, the, and so the Pentagon
16:24
then has a policy that, that my old office in the Pentagon wrote,
16:27
uh, while I was there, called, uh, department of Defense Directive,
16:30
3000.09 autonomy and weapon systems.
16:34
And that governs essentially the development.
16:39
So like doing the research and development and moving toward
16:41
procuring the system and then fielding like, all right, we bought it.
16:45
Like how do we get it to the army unit, the Navy unit, et cetera.
16:48
The development and fielding of semi-autonomous systems, weapon
16:53
systems, the all the precision weapons, and then autonomous weapons systems.
16:57
And what it does is lay out both the requirements for
17:01
those systems, like all the.
17:03
All of the testing and evaluation that you have to do to prove they work.
17:06
'cause you remember what I said before.
17:08
So since that don't work, aren't useful to the military and commanders
17:12
and operators won't trust them.
17:14
And the, and it it, it also sets up that for some types of autonomous
17:19
weapons systems, those that don't have human supervision essentially.
17:24
There's an extra layer of review beyond the really stringent reviews that every
17:29
weapon system goes under, where it's not just that the system has to go through
17:32
all the tests to prove that it works, it then also has to be essentially like run
17:37
by uh, senior leaders like the number two in the entire American military and
17:43
Werbach: a couple of very senior civilian know Senate
17:45
Horowitz: confirmed civilian officials to just do that double check to make sure
17:49
that, hey, this is actually a good idea.
17:51
Like, and it works.
17:52
So, I mean like the Pentagon, you know, I was joking before about
17:55
the Pentagon is a bureaucracy.
17:56
There's always process.
17:57
Werbach: There's so much process
17:59
Horowitz: and so the, and that process in this case in some ways should be
18:04
reassuring because it makes it actually a lot more unlikely than people would
18:09
think that the Pentagon would field some kind of like wacky killer robot that
18:15
like didn't work or would put people in harm's way that shouldn't be, et cetera.
18:21
Werbach: What exactly does that phrase human responsibility
18:24
for use of force mean?
18:25
Is that analogous to when people talk about human in the
18:28
loop, uh, in AI governance?
18:30
Uh, how, how should we understand it?
18:32
Horowitz: That is a good question.
18:33
The, I would say human responsibility, I would think about more in terms
18:37
of the law of armed conflict.
18:38
Like if you had a, if you have a, like, like for example in the, in the outset
18:44
of the Iran uh operation, it appears that based on publicly available sources, now
18:49
it appears as though a US Tomahawk missile may have tragically hit a school in Iran
18:54
leading to a lot of civilian casualties.
18:57
The American military right now is doing a really detailed after
19:00
action investigation to try to figure out exactly what happened.
19:04
And it is, I mean, in putting aside the politics of the current moment, the it
19:09
is, it is what normal practice would be for the military is to figure out who the
19:14
military actually will have a rubric to figure out who are the people that should
19:18
have known, how did this mistake happen?
19:22
Who are the people that should have known, or who are the people
19:24
that should have caught the error?
19:25
However the error happen.
19:27
Then like, what is the, and then depending on how, what the error was and what
19:31
part of the process, et cetera, those people will probably be reprimanded
19:34
or punished in one way or another.
19:35
Look like sometimes accidents genuinely do happen, but the, the military has
19:41
really detailed systems for that.
19:42
So by responsibility and accountability, what we mean is there is a person that
19:46
said, go use this piece of technology.
19:49
So they're taking res, they're taking responsibility for it,
19:52
and there's accountability in that if there's an accident.
19:55
That like, like they're the ones on the line.
19:58
Like they're the ones that authorize the use of force.
20:00
And so that's why they need to be, according to Pentagon policy, they need
20:04
to have, um, you know, what's called appropriate human judgment over the use
20:07
of force, meaning they're trained and understand both the upsides and limits of
20:12
any autonomous weapon system so they can make an informed decision about using it.
20:17
Werbach: But does that only happen at the front end when a system's
20:20
being developed or fielded?
20:22
Um, what happens when it's out there, you know, in a battlefield situation.
20:27
Horowitz: So that responsibility exists anytime there's the,
20:30
the actual use of force.
20:31
Mm-hmm.
20:31
And so that, that has to exist regardless.
20:34
The, the decision making surrounding the use of force though is, is
20:37
in some ways technology agnostic.
20:40
The same responsibility and accountability provisions that mean.
20:45
If you were, if you were firing, you know, if you had like mounted knight that were
20:49
gonna later like, like throw spears at somebody or something, the, those same
20:54
kind of provisions would apply in the context of an autonomous weapon system
20:57
being sort of used in the battlefield.
20:59
Used in the battlefield as well.
21:02
The, it essentially requires militaries come up with like new
21:04
standard operating procedures to like figure out how this works in a
21:08
world of autonomous weapon systems.
21:10
But again, there have been really simple versions of these that have
21:12
been around for decades and so.
21:14
Militaries actually have a lot more experience using simple versions
21:18
of artificial intelligence than people I think tend to realize.
21:23
Werbach: Hmm.
21:24
And so we have these, uh, these doctrines and these policies,
21:28
uh, in the US military.
21:29
What, what happens if, you know, say China, Russia, North Korea,
21:32
take your pick the, the adversary.
21:34
They don't wanna play by those rules.
21:36
Horowitz: Good question.
21:38
Um, the, I think one of the really interesting this context in the u you
21:41
know, we, we talk about the US and China a lot in the AI context, obviously, because
21:44
the US and China are competing for global leadership in AI across lots of things,
21:48
like not just in the military arena.
21:50
And one of the really interesting things is if you get.
21:52
US and Chinese, uh, either like mil, like folks that do a military
21:59
AI sort of together, they actually don't sound all that different.
22:03
Like, since I've left the government, I've been in some track two discussions
22:06
with Chinese interlocutors and the kinds of things they say about responsible
22:10
AI and about ensuring that systems are trustworthy are not frankly, that
22:14
like they, they use like the words in like slightly different orders or
22:16
whatever, but they're not expressing views that are fundamentally different.
22:20
The thing that a, say the United States worries about when it comes to a country
22:25
like China's use of AI from a military perspective though is, is different
22:30
and relates to, uh, China's political regime in that, because China is an
22:36
autocracy, obviously, like led by Led, led by Xi, the, the Chinese government
22:40
in some ways, like the, doesn't trust their people in the first place.
22:44
If they trusted their people, they might have a different kind of political regime.
22:47
And so the incentive structure, and you see this in China
22:49
actually all the time now.
22:50
The incentive structure is like G tries to consolidate power is to cut people
22:54
out of the decision making process.
22:57
And so the thing you worry about in an autocratic context is that
23:01
you end up with essentially unsafe centralization of decision making.
23:07
The where like too many people are cut out.
23:09
Essentially to be able to make, uh, to be able to make good decisions, or at
23:12
least that that's the concern sometimes.
23:14
But on paper, like the way the US talks about using AI from a
23:18
military perspective, the way China talks about using AI from
23:20
a military perspective, think not as different as you might imagine.
23:23
Hmm.
23:25
Werbach: So there are two, two recent conflicts where there's been a, a lot
23:28
of, um, you know, apparent use of, uh, AI and various autonomous technologies
23:34
for, uh, targeting being, you know, Israel's conflict in Gaza and Russia,
23:39
Ukraine, um, where, you know, lots of use of, of, of drones and these, you know,
23:43
semi-autonomous systems and so forth.
23:45
Uh, what's the sense of, of, you know, what we're learning from what's
23:49
happened in in those two conflicts?
23:51
Horowitz: I mean, I think we are seeing the.
23:55
We are seeing, obviously, intense public interest in these systems and, and
23:59
also how, how nascent some of them are.
24:02
The, like, for example, like there's been a lot of controversy about
24:05
Israel's lavender system in the context of the Gaza War, some of these
24:09
accusations that the lavender system is.
24:12
Essentially it's like queuing up targets for people that have been
24:15
like a couple of seconds to either like, approve it or not approve it.
24:18
And like then maybe there's a strike that is against a civilian.
24:21
Uh, that's not really how lavender works, I think.
24:24
And there's like good public reporting that suggests that.
24:26
I think it's much further back in the decision making process.
24:29
It's like one input that goes into a, uh, a, a decision that's going through
24:33
several human checks before somebody's using lethal force it to be clear.
24:39
That doesn't mean that it's risk free.
24:42
There's certainly the risk of automation bias in that context where say if there's
24:46
a recommendation from lavender and that recommendation is like, I dunno, suppose
24:50
it's 75% accurate, but people treat it as if it's 99% accurate, you're then gonna
24:55
introduce the more errors into the system.
24:57
Um, but like, I think part of this, I think part of the Israel
25:01
conversation is actually a like rules of engagement conversation.
25:05
And that if what Israel is doing and like, I don't actually
25:08
know the answer to this, like.
25:09
Per se.
25:10
But it seems sometimes like what Israel's doing is essentially taking, or what
25:14
Israel was doing in the Gaza context was essentially taking output maybe from
25:18
algorithms that wasn't, you actually didn't have super high confidence in.
25:23
And then like going ahead with operations on that basis,
25:27
that's not the algorithms fault.
25:28
Like if the algorithm's like, say the algorithm's like 55% accurate and you
25:31
know it's 50% accurate, and you're like, well that's good enough, let's do it.
25:34
Like, that's not, uh, that's not an algorithm issue.
25:36
That's a, that's a rules of engagement issue in some ways.
25:40
And although like whatever, but that a lot of that's uncertain in the US context.
25:44
Now we are seeing the, uh, real like debut of a platform called a Maven Smart
25:50
System from, uh, a company Palantir.
25:53
And one of the plugins, uh, one of the many plugins to Maven
25:56
Smart System is Claude Thero, uh, the Anthropic technology.
26:01
What's that?
26:02
The way that it essentially works is imagine it as a
26:06
paralegal helping a lawyer.
26:08
So all sorts of like background research and assessment and uh, and
26:13
and analysis that like simplifies a ton of the workflow to queue things up
26:19
for senior leaders to make decisions.
26:21
But like no matter how good your paralegal is, it's still the lawyer going to court.
26:25
It's still the lawyer making the decision.
26:28
I think that's what the way that this kind of system works for the United States,
26:32
um, right now is it's essentially doing dramatic work to speed up and to make
26:40
more efficient the workflow involved in, uh, in macro target selection.
26:46
But that output then goes through a bunch of layers of scrutiny, including
26:50
a final decision by a, by a commander.
26:53
Prior to, prior to force being, prior to force being used.
26:58
So it's simultaneously actually like a really big deal from a
27:01
military planning process and there are still heavy human involvement.
27:06
Werbach: Mm-hmm.
27:08
And, uh, what, what about Ukraine with the, the drone warfare
27:11
that's, that's happening there?
27:12
Horowitz: I think in the Ukraine context, we're seeing something a little bit
27:15
different in the Ukraine context, we.
27:18
There's been increasing use of AI on the battlefield and you know, like necessity
27:23
has been the mother of invention for the Ukrainian since the drop, like
27:26
since Russia's invasion in, in 2022.
27:29
And they have, uh, I think pioneered a lot of uses of, of essentially AI
27:35
decision support tools, kind of like Maven ish to try to help them queue up.
27:41
Um, respo, like more automated responses to attacks from the attacks
27:46
from the Russians and for, and for target prioritization and, and also
27:50
in u between Ukraine and Russia.
27:52
Now, you know, 60 to 70% of the casualties between Ukraine and
27:56
Russia on the front lines are driven by, uh, one-way attack drones.
28:01
Now, those are generally remotely piloted.
28:04
They're not, they're not autonomous.
28:05
They're not necessarily AI driven.
28:07
One of the things that Ukraine has started to do is build in essentially
28:10
last mile autonomy, because if somebody is, if somebody fires a weapon at
28:15
you and they need a, there's a data link between, say, an operator with
28:20
a, you know, if we're talking about one of these, like first person view
28:22
drones, there's a data link between the operator and then the drone that's
28:24
gonna say, gonna hit a Russian tank.
28:26
What Washington tries to do then is jam the data link, disrupt the data
28:29
link, so then the, you know, the human operator loses control of that drone
28:33
and like it falls out of the sky or whatever, whatever it's gonna do.
28:37
And one of the things that, that Ukraine's been doing is programming these drones.
28:42
They've been doing lots, they've been using fiber optic cables and like all
28:44
sorts of things like make this stuff work.
28:47
But if you lose connectivity generating LA last mile autonomy so that if it's
28:51
close enough, it can then turn on a sensor and be like, oh, that's a Russian tank.
28:55
And it can hit it.
28:56
And so we're really seeing some of this stuff start to play out in the
28:59
Russia, Ukraine context in a way that, you know, I think a lot of people
29:02
have written about, but we see it.
29:05
And just the drone war in general at an industrial scale.
29:08
Mm-hmm.
29:09
That is, that is unprecedented.
29:10
Mm-hmm.
29:12
Werbach: Alright, so one last thing just to get back to the,
29:14
you know, the LLMs and, and the philanthropic situation and so forth.
29:17
As, as you mentioned, a lot of Silicon Valley came out of defense spending
29:21
and things like, like DARPA being what, what basically, you know, brought us
29:25
the original technology of the internet.
29:27
Um, and now we have these private sector companies.
29:32
Like, you know, the big AI companies who are absolutely at the frontier
29:36
in terms of technology, including for, for military purposes.
29:40
Um, so, you know, how does that change things?
29:43
You know, what, what do you see, you know, beyond this, this particular controversy
29:46
with, with philanthropic and the Pentagon?
29:48
How is that relationship likely to develop and what does it mean for,
29:53
uh, our military going forward?
29:55
Horowitz: I think there are some real challenges that have been introduced
29:57
by this controversy that A, and it's not clear exactly how they'll pan
30:01
out since you know, it, it, if the government didn't like philanthropics
30:06
the conditions that philanthropic wanted to put in its contract, they
30:08
could certainly decide not to renew.
30:09
They could decide to cancel the contract.
30:11
Like that's a thing that happens in the business world all the time in
30:14
between like governments and businesses.
30:15
But the, but the Pentagon has essentially decided to salt the earth through
30:19
labeling anthropic as a supply chain risk.
30:23
The challenge then is it could signal to other companies that, and, and you know,
30:28
in other, there's some companies that are already working with the Pentagon or
30:31
thinking about working with the Pentagon.
30:32
But for someone that hasn't made that choice, if there's now a non-zero risk
30:36
that if you have a contract dispute with the Pentagon over terms of service
30:40
essentially, that that might lead to them not just canceling the contract
30:45
but attempting to destroy your business.
30:47
Outside of that, you, you could certainly imagine a world in which
30:50
that has a bit of a chilling effect.
30:52
Because there's been a lot of improvement in the relationship between Silicon Valley
30:56
and the Pentagon in over, over many, uh, over many years as these companies
31:01
have realized, and I think especially realized when they saw Ukraine using
31:07
essentially some of their tools to protect itself against Russia's invasion.
31:10
They saw the use case for their technology in a fight for good.
31:15
And, and I think that helped shape sort of greater receptivity.
31:19
It's not that companies aren't gonna step in the breach now, you know,
31:22
uh, uh, XAI has now said it'll do classified work at the Pentagon.
31:25
OpenAI is now working on a deal, uh, with the Pentagon as well.
31:29
And OpenAI actually, I think, genuinely wishes for whatever deal
31:33
they sign to also apply to Anthropic.
31:35
Like they, uh, you know, obviously those companies compete, but it, it's not, and.
31:41
The way that this has happened with Anthropic is not good for open AI for all
31:45
the reasons that I just said essentially.
31:46
And so I think they would actually like a deal that could also
31:49
work for Anthropic potentially.
31:51
Um, and, but so, so companies will get in the door in the short term, but this
31:55
is not gonna be quite as simple as just like, this is not just like, oh, you're
31:59
using this chatbot one day and now I'm gonna use this other chat bot another day.
32:03
Cloud is now embedded in classified workflows that the
32:06
military uses around the world.
32:09
Disentangling that, and then plugging in and testing another
32:12
system and validating it.
32:13
Like, that's gonna take time.
32:15
And I mean, I, I hope at least that that philanthropic in the government can come
32:19
to some kind of agreement since this, this controversy seems like totally
32:23
needless from, from my perspective.
32:26
But no, I mean, this does, this does really shake up relations between Silicon
32:30
Valley and the Pentagon once again.
32:33
Werbach: We'll see.
32:33
No, I, I agree with you and I think it's, it's a really significant turning
32:37
point in a lot of ways about how we go forward with AI in this country.
32:41
Mike, thank you so much for speaking with me.
32:43
Horowitz: Thanks so much for having me.
32:44
Happy to come back anytime.
32:48
This has been The Road to Accountable ai.
32:50
If you like what you're hearing, please give us a good review and
32:52
check out my substack for more insights on AI accountability.
32:56
Thank you for listening.
33:01
Werbach: If you want to go deeper on AI governance, trust
33:03
and responsibility with me.
33:05
Another distinguished faculty of the world's top business school.
33:09
Sign up for the next cohort of Wharton's Strategies for Accountable
33:12
AI online executive education program, featuring live interaction
33:16
with faculty expert interviews and custom designed asynchronous content.
33:21
Join fellow business leaders to learn valuable skills you can
33:24
put to work in your organization.
33:26
Visit exec ed.wharton.upen.edu/acai for full details.
33:33
I hope to see you there.