Latest / The Overlap / Killer Robots: The UN Autonomous Weapons Non-Treaty

Transcript

Killer Robots: The UN Autonomous Weapons Non-Treaty · The Overlap

0:00
In twenty thirteen, a South African human rights lawyer named Christoph Haines stood
0:05
in front of the UN Human Rights Council in Geneva and warned a room full
0:10
of diplomats that governments were getting ready to give the decision
0:13
to kill a person entirely over to software.
0:16
He was the special reporter on extrajudicial executions,
0:21
which made him the person the UN sends to count the bodies after
0:25
a government decides the law doesn't apply to certain people.
0:29
He wrote the first UN report on what he called lethal autonomous robotics.
0:33
And what he asked for was pretty modest a moratorium.
0:38
Stop building these until someone works out whether a machine
0:42
can legally decide that you should die.
0:45
On September 5th, 2026,
0:48
just 10 days ago, the UN group that grew out of that warning finished
0:54
its three year mandate in Geneva.
0:55
It's called the Group.
0:57
Of governmental experts on lethal autonomous weapons systems.
1:02
And states have been meeting on this subject for some form for about twelve years.
1:08
The document they produced at the end is nonbinding.
1:12
In the final hours, the United States and Russia,
1:14
two countries that agree on almost nothing right now,
1:17
except on who's the President of the United States,
1:21
both push to delete language from the text.
1:25
References to ethics came out,
1:27
and so did references to predictability.
1:29
So did references to the design and development stage,
1:33
which is the only point where anyone could stop these weapons before they exist.
1:37
As of this month, seventy-six states say they want to open negotiations
1:43
on a binding treaty. That's a a large block and still can't force anything because
1:48
the room runs on consensus. One country can block the result and two showed
1:52
up ready to. Meanwhile,
1:55
In the first five months of 2026,
1:56
Venture Capital put more than 14.6 billion into autonomous weapons companies.
2:03
AndyRail Industries closed a five billion dollar round in May at
2:08
60 at a sixty-one billion dollar valuation.
2:12
It booked two point two billion in revenue last year and is building
2:16
a one billion dollar factory in Ohio to mass produce autonomous systems.
2:22
While diplomats argued over whether the word ethics could stay in a paragraph,
2:26
investors had already decided what they expected Geneva to produce.
2:30
Haynes died in twenty twenty one before the Years Review Conference collapsed
2:36
and before Gaza even started. He never got his moratorium.
2:39
This isn't a story about a treaty that failed,
2:41
because there is no treaty. There never has been a treaty.
2:44
There is not right now in September of two thousand twenty six as we
2:48
sit here a single binding international rule anywhere on earth that says
2:53
a human being has to be the one to pull the trigger.
2:56
Which leaves questions about who decided and when that the rest of
3:00
us had already accepted this.
3:19
Hi Will.
3:20
Hi Joshua.
3:21
This is the overlap. We cover systems of power,
3:23
labor, and American injustice,
3:26
and today's system is the UN disarmament process,
3:29
which I realize sounds a little bit like homework.
3:33
It does, but some of us like homework.
3:35
And then you'll enjoy a room in Geneva where for a decade a handful
3:39
of powerful countries have used a procedural rule to make sure nothing legally
3:44
binding ever gets written about machines that kill people on their own.
3:48
The timing is why we're covering it right now.
3:49
The mandate that ended September 5th was a three-year cycle,
3:53
and there's a review conference in November of 2026.
3:56
That's when the states pushing for negotiations either force the issue
3:59
or lose another five years. In 2023,
4:02
Secretary General Antonio Guterres and the President of
4:06
the International Committee of the Red Cross jointly called for
4:09
a legally binding instrument on autonomous weapons by 2026.
4:13
We're about three and a half months out from the end of that deadline,
4:16
and what exists is a non-binding text with the word,
4:19
among others, with the word ethics cut out.
4:22
So in the US, this debate usually gets framed one very particular way.
4:26
The rest of the world wants to tie our hands,
4:29
and we can't accept that because China won't.
4:34
You hear it on think tank panels and in our Pentagon statements.
4:38
I think that framing is a little bit backwards.
4:41
Of all of the countries involved,
4:44
the United States should be the most worried about a world with no rules here.
4:50
Why is that, Joshua?
4:52
Well a a technological lead lasts a few years,
4:55
right? And and a precedent lasts decades.
4:58
Every rule the US deletes from a UN document in twenty twenty six is
5:03
a rule that likely won't exist or have an opportunity to until twenty thirty four,
5:07
when somebody else has the better system.
5:10
We already ran this experiment with drones actually.
5:13
~ the US built a legal rationale for targeted killing by aircraft,
5:17
used it for twenty years and called it.
5:20
acceptable because we were the only ones flying,
5:23
right? Like we only we were the only ones who had the technology to do it.
5:26
And now Turkish, Iranian and Russian drones are pretty much everywhere,
5:31
~ not to mention Ukrainian drones.
5:33
The the legal objection we would need today is one we spent basically
5:38
two decades tearing apart.
5:40
And that's the problem that we see with the you know the nuclear option
5:44
in the Senate and things like that.
5:45
It's like when you're in charge and you make the rules to benefit your side,
5:48
you have to be prepared for them to serve the other side when they're in charge.
5:51
but autonomy here goes further than drones did,
5:54
right? Drones are bad enough, but a drone strike still had a person in
5:57
a trailer in Nevada watching a screen.
5:59
And don't get me wrong, the outcomes were often terrible,
6:02
but there was a person whose name could end up on a document.
6:05
Somebody had to to authorize the attack.
6:08
An autonomous system removes that person.
6:10
When the machine picks a target and gets it wrong,
6:12
who gets charged? The operator who wasn't operating?
6:15
The commander who approved the setting?
6:17
The engineer in Ohio, or some model that nobody can fully explain or understand?
6:21
And predictability, meaning like the ability to know in advance what
6:26
the system is actually going to do,
6:28
we call that eval in the AI world,
6:30
was one of the words was that was actually cut from the text just this month.
6:36
Which is strange because the companies building AI have said for a couple
6:39
of years now that they don't fully understand the behavior of what they're shipping.
6:42
They report capability jumps they didn't predict and behaviors they didn't train
6:46
for, and they fund research teams whose whole job is to figure
6:49
out what their products are doing.
6:51
So the builders describe this this risk in public.
6:56
I mean, we're hearing it everywhere,
6:57
and the government of the country where most of them are headquartered went
7:01
to Geneva and just like struck sh struck the word entirely from the document.
7:06
So basically Geneva keeps failing on this in a very particular way,
7:12
and and the reason goes back to a treaty that took effect in nineteen eighty three.
7:15
The history. gosh, wouldn't you believe it?
7:19
We're gonna go back and cover the history of it all.
7:21
1983, it was a different world.
7:23
But the convention on ~ I didn't.
7:24
Let's not call that history. Some of us live through that.
7:26
~
7:28
I
7:28
was that's history for me. That's ancient runic history,
7:31
sir. But the convention on certain conventional weapons,
7:36
CCCW, usually called just the C C W
7:41
It came out of the nineteen seventies,
7:43
right? You had Vietnam, you had the napalm and mustard gas and bomb fragments that
7:48
that didn't show up on an X-ray.
7:49
The Geneva Conventions mostly covered how you treat people after capture,
7:54
right? And had a lot less to say about what happened and when and whether
8:00
or not you're allowed to just shoot them with a thing.
8:03
The Red Cross and a group of
8:04
As long as they die first, the Geneva
8:05
Convention doesn't care.
8:06
Well, I mean, to be fair, we do fund most of the UN.
8:10
~ the Red Cross and a group of states pushed for something to actually fill that
8:16
gap. The CCW is a a framework convention,
8:20
basically a frame that bans nothing on its own,
8:23
with specific rules attached to it as protocols.
8:26
Think of it like guidelines. It's like a a ~ framework for how we're supposed
8:29
to handle warfare with one another,
8:32
especially UN countries. Protocol one
8:36
Covers weapons that leave fragments undetectable in the body.
8:40
All of these are incredibly scary to talk about,
8:44
like it feels gross even saying them out loud,
8:46
like Protocol One covers just random shit that might be left over in your body after
8:52
war. but that's exactly what it covers.
8:55
Terrifying is that that's the first one that came up.
8:58
Of course. I mean that's that's but it came out of Vietnam.
9:00
I mean it makes sense, right? Like lots of strapnel.
9:02
Right.
9:03
So protocol two covers mines and booby traps.
9:06
And then protocol three covers incendiary weapons like flamethrowers
9:11
and things like that.
9:12
Sure, your regular call of duty trifecta.
9:14
Alright, so the idea was that it could grow to encompass new weapons when they
9:18
appeared and states write a new protocol attach and attach
9:21
And it grew once, right, in a in a way that almost nobody remembers,
9:24
including myself. In in nineteen ninety five,
9:27
Protocol four banned ~ blinding laser weapons before any military
9:33
had actually used them in the field.
9:35
So prototypes existed that would basically permanently blind soldiers
9:40
and and states banned them in advance.
9:41
For good reason, right? Like I mean that's that's a pretty terrible thing to
9:45
do to a human being, but
9:47
The the major militaries actually signed on for it.
9:49
It was a a clear case of the world seeing a weapon coming and agreeing like we've
9:54
got to stop this. Like before it even actually does it,
9:56
no before anybody puts it on a you know,
9:59
on a on a resolution anywhere or or a truck or a Hum V,
10:03
we need to ban this because this is objectively,
10:06
ethically wrong.
10:08
Course there was little pushback on the ~ blind blinding laser weapons because
10:12
nobody was really getting rich off of those.
10:14
However, I'll say they did pivot to selling to high school kids so that they
10:17
go around and try and blind their friends.
10:19
Apparently.
10:19
And pilots in the sky.
10:21
And pilots, yeah.
10:23
So I mean with no money behind it,
10:24
nobody was there to argue about the ~ there's a laser lobby basically.
10:28
No venture capital pouring into permanent blindness technology.
10:30
No no laser lobby, I like that one.
10:31
Yeah.
10:32
That's what I'm gonna
10:33
call my my next band, the laser lobby.
10:35
The laser
10:35
lobby. Nice. the military benefit was marginal and the horror was easy to picture.
10:40
I mean you blind a nineteen year old conscripted to fight for his country
10:43
for the next sixty years of his life and you don't even win the battle.
10:46
That ban cost nobody anything.
10:48
The contrast sits in the same treaty in the same decade though.
10:52
The C C W's mine protocol, mines protocol,
10:55
the protocol two that we talked about before,
10:56
was full of exceptions and where feasible language and exempted nearly everything
11:01
armies actually used. So it you know it it
11:05
Said it on paper, but it had no teeth,
11:06
right? And it was amended in nineteen ninety six and stayed weak.
11:09
So the campaigners, the survivors,
11:11
and the doctors doing amputations on children's in children in Cambodia,
11:14
Angola, and Mozambique made a tactical decision that changed arms control forever.
11:19
They left the C C W.
11:21
And then went to of all places Ottawa,
11:24
in Canada.
11:25
Is that still named Ottawa? I'm I'm losing track these days.
11:28
Okay.
11:28
It it is. He he i they're
11:30
not another state yet.
11:32
Got it. ~ in December nineteen ninety seven,
11:34
a hundred and twenty two states signed the Mine Band Treaty in Ottawa.
11:38
It was negotiated outside the UN consensus process,
11:41
with civil civil society and mid sized countries driving it.
11:44
Casu stayed out of it. The US,
11:46
Hmm.
11:46
Russia, and China.
11:48
Giant surprise. I think I think that's probably why people call it a failure now.
11:54
Right, but on the other hand, look at the production and trade numbers though.
11:57
So when people stopped when the rest of the world stopped wanting to use mines,
12:01
~ the global trade and anti personnel mines mostly collapsed.
12:05
Manufacturers quit the business,
12:07
banks divested, countries that never signed stopped exporting
12:10
and started clearing minefields.
12:12
The stigma changed behavior in places the signatures never reached.
12:16
I I don't know, I would push back a little bit on how well that actually transfers
12:19
to reality. So a norm or a a a precedent without like
12:26
the the biggest powers in the world,
12:28
right, that actually covers them except the countries that fire the most.
12:32
Ottawa succeeded partly because mines are pretty cheap,
12:37
right? They're crude, they're low tech.
12:38
I was I was reading the other day ~ about a a
12:43
Something they did Vietnam where they would take a hand grenade,
12:45
remove the pen with the keep the clip still on,
12:48
cover it in mud, and let the mud dry.
12:51
So basically you could just step on the and break the mud,
12:54
and essentially you have yourself a an explosive or an IED of of sorts that that
13:00
worked in the same way that a mine did.
13:02
But no major powers security strategy actually depended upon them,
13:08
right? I mean, even the Viet Cong,
13:10
especially coming out of this era.
13:12
~ they used they used ambush techniques and they used really low-tech weapons that
13:18
are pretty awful, like objectively,
13:20
if you read about them. ~ and and some of those I did read about in
13:24
the research for this episode.
13:26
But ~ basically autonomous targeting,
13:31
right? This basically AI targeting is is incredibly different.
13:35
So since 2014, the Pentagon
13:38
Has pursued what it's called the third offset strategy.
13:41
And it's basically a formal bet that AI and autonomous weapons systems would define
13:48
the next era of the American military advantage.
13:51
You'd be asking the US to basically give up its entire plan for the next several,
13:57
you know, decades, well, at least one generation of kids,
14:01
warfare, right? And that's pretty weird.
14:06
True, and it does explain why the consensus rule is the problem.
14:09
The bargain was that the big militaries get a veto and in exchange they stay
14:14
in the room and the rules apply to them.
14:15
The problem is the rules never get written under the veto or the consensus rule.
14:20
So what that bargain bought was their attendance and nothing else.
14:23
So they show up at the meetings but they they don't they prevent anything from
14:27
happening.
14:28
I think that's kind of the the the hitch in my crawl,
14:31
you know, the ~ the burr in my sock.
14:34
That rule itself, right,
14:37
the the CCW consensus means nobody in the room objects.
14:42
And that's that's how consensus model is supposed to work.
14:45
~ I'm I'm a board member on a local choir here and that's how we we operate.
14:50
It's a consensus. If if anybody has any objections,
14:53
we keep talking it out or we keep renegotiating until everybody agrees yes.
14:58
puts their hand up. ~ a majority doesn't actually count.
15:02
So it's not like a two-thirds or a supermajority.
15:05
There's no there's no rule like that.
15:06
Everybody needs to agree before we move forward.
15:09
So when a headline basically says the UN failed to agree,
15:14
it usually means more than a hundred countries agreed and one said no,
15:20
right? Or had no vote available to actually overrule it.
15:25
And that rule has been enforced
15:28
Since nineteen eighty three, all the way until about ten days ago.
15:32
Let's lay out the dates on our current situation because in sequence it looks like
15:36
a tactical move.
15:37
In sequence, sure. I in ~ that's I think one of the the benefits of of hindsight,
15:42
right? Like being able to look back into history and see it as opposed
15:45
to looking for it. But in in twenty thirteen,
15:47
Haynes delivered his report, and in twenty fourteen the C
15:51
CW starts informal meetings on autonomous weapons.
15:55
In twenty sixteen, states formally created the the group of governmental experts,
16:00
which is a body with a mandate to basically study the issue and it
16:05
has studied it for years.
16:07
During which in march twenty twenty,
16:09
a Turkish made loitering munition may have killed someone in Libya on its own.
16:13
We'll get to that in a minute though.
16:15
Yeah, so w then comes the sixth review conference in December of twenty twenty one,
16:21
when years of meetings were supposed to become a negotiating mandate.
16:26
So dozens of states basically were like,
16:29
Hey, like we want this to stick.
16:31
We we need this to have some teeth.
16:33
It needs to be a binding resolution and and the UN panel of experts report already
16:39
documented what happened in Libya.
16:41
On that final day, a small group of of countries sort of stripped
16:45
the substantive content, all of the teeth,
16:48
~ from the the conference report and basically ended with
16:52
no mandate for negotiations. ~ I think ~ in a lot of ways this is
16:57
how the GOP tends to operate, right?
17:00
It it seems like an innocuous overriding of a of a procedure or a norm.
17:06
But it has long lasting impacts and consequences.
17:09
Maybe they know, maybe they don't know,
17:11
and and it and it sort of bites in the butt later.
17:13
And and I think we're kind of F A F Oing right now on that.
17:17
Well, one thing we know is that waiting until the last day is deliberate.
17:20
If you object in the second month of this long drawn out process
17:24
and and everybody else has a chance to organize around you and respond,
17:27
if you object in the final hours when delegates have flights to catch home
17:31
and and alternative to your weak text is no text,
17:33
you get what you want.
17:35
Yeah, I I mean I I definitely draw the parallels there,
17:38
especially with the Senate and Congress,
17:40
right? Like they they want a filibuster at the last minute when they could have been
17:43
having the negotiations all along.
17:45
~ and so I I I agree that it can definitely be used as its own negotiating tactic.
17:52
But the nineteen eighty three rules actually allow that.
17:57
Like as of in nineteen eighty three,
17:59
the rules allowed for the ability to to use that as a tact.
18:02
Right. So fast forward to ~ fast forward forty years to nine sorry,
18:07
to twenty twenty three, at the annual meeting of the treaties member states,
18:10
Russia blocks adoptions of the rules of procedure.
18:13
So now we're talking about without adopted rules,
18:16
the meeting was it goes into informal mode.
18:18
So they're not even blocking the substance,
18:19
they're blocking the procedural rules to have the meeting,
18:22
~ to to you know, have it in front of the public.
18:25
And so they shut out basically the campaigners,
18:28
researchers, and survivor organizations that track where all these weapons turn up.
18:32
Yeah, a a lot like you're seeing ~ when these data centers go up and
18:36
and things that the city council is just keeping people from talking,
18:41
blocking public comment for whatever reason.
18:44
~ that was actually the same year that Gutierrez and the ICRC called
18:48
for a something with actual teeth by twenty twenty six.
18:52
So we have the UN Secretary General asking for a law by twenty twenty six,
18:55
and a permanent member of the Security Council responds by clearing
18:58
the observers out of the room.
19:00
Yeah. That's a forty three year system that proved that once in nineteen ninety five
19:07
that it could ban a weapon before it existed,
19:09
right? The the lasers we were talking about earlier,
19:11
and then spent the next thirty years showing that it could only manage that when
19:16
the weapon wasn't being made by anybody for money.
19:21
Right, when there couldn't be a profit derived from it.
19:24
But in May of this year, autonomous weapons had drawn at least fourteen point
19:29
six billion.
19:29
Ba billion in venture capital in five months.
19:34
And one company explains a lot about where that money went.
19:36
Andoril Industries was founded by Palmer Lucky,
19:40
who you may know from the Oculus headset,
19:43
VR headset, which he sold to Facebook,
19:45
then left Facebook and started a defense company.
19:48
Brian Schemph is now the CEO of Andoril and their core product is
19:52
a software called Lattice. That's not not Lattice like you eat,
19:56
but Lattice, like, you know, ~ the side of your house,
19:59
the lattice work. ~ it takes the sensor data from towers,
20:02
drones, and radar.
20:04
fuses it, creates a lattice, if you will,
20:07
identifies objects and tells the operator or the machine what's out there
20:11
and what to do about it. Now Androil sells directly to the Pentagon and
20:15
to our NATO allies. In May 2026,
20:18
Android closed a $5 billion Series H round.
20:23
Series H, I didn't even know they went that far.
20:26
At a $61 billion,
20:27
They go all the way to Z, apparently.
20:29
apparently, ~ $61 billion valuation on that round.
20:33
Roughly double its previous valuation.
20:35
It had two point two billion in revenue in twenty twenty five and it's spending
20:39
a billion dollars on the Ohio factory it's now creating.
20:42
Yeah, that Round H closed about four months before the Geneva mandate expired,
20:48
without e without a a ~ rule or a binding resolution of any kind.
20:54
Investors, however, in these organizations read the same news that we do.
21:00
After a decade of Geneva with the US and Russia both able to block anything,
21:06
and the twenty twenty one collapse on the record.
21:10
It was a pretty rational bet that no binding rule is coming.
21:13
Somewhere there's probably a PowerPoint or a pitch deck that lists
21:18
the regulatory risk as low. And so far that deck has proven to be correct.
21:24
And as you touched on earlier,
21:25
the usual explanation here is the arms race,
21:27
right? The US can't stop because China won't stop,
21:30
and China can't stop because the US won't stop.
21:32
That dynamic does exist. I'm not saying it's it's completely manufactured,
21:36
but I think the stronger driver in that autonomy is the fastest growing part
21:40
of the defense business. Ships take a decade to build.
21:43
Fighter program runs twenty years and costs over a trillion dollars across its life.
21:47
Autonomy is software, so you can ship a version this quarter and sell
21:51
it to eleven countries at software margins.
21:54
Venture Capital spent five years looking for the next category after consumer apps
21:57
and crypto, and it found a customer with the largest military budget
22:00
on earth that rarely cancels anything.
22:02
As unfortunate as that is, the fiscal twenty twenty six US defense budget,
22:08
the one we talked about a couple episodes ago,
22:11
was over nine hundred billion dollars with a growing share specifically directed
22:17
toward autonomy and AI.
22:21
That's almost a t trillion dollars.
22:23
Sorry, this doesn't sound as good on when I do it there,
22:27
but I thought I could claim that one.
22:27
It doesn't. It it's that T
22:28
R that T R vowel is hard to repeat.
22:30
Yeah. ~ so when someone says we can't regulate because of the arms race,
22:35
what I hear is there's a customer with enormous money and no rules,
22:38
and Silicon Valley noticed about that same same time.
22:41
Yeah, and the diplomacy in this scenario runs in in sort of parallel.
22:46
On September twenty fourth, ~ just about a year ago,
22:50
the UN Security Council held a high level debate
22:54
on AI and international peace and security,
22:58
with heads of state and foreign ministers attending and and Slovenia's president
23:02
at the time, Natasha Pirce-Musar presiding.
23:06
Gutierrez basically said AI could do enormous good.
23:12
Which I think all of us would likely agree with that.
23:14
And if you've been following us,
23:15
there are good things that it can be used for.
23:18
But just like anything else, it can be weaponized without guardrails.
23:23
And he called for human control over every use of force.
23:28
He's not saying like we shouldn't use AI.
23:30
You can use AI to target, but it needs to be backed up by a human being
23:35
so that we have a way to blame.
23:36
It so that we have ~ a method enforced
23:41
with teeth to to blame this on a person rather than I don't know what happened
23:47
inside of this AI system. Right?
23:50
That request for for that use of force,
23:54
that's just a minimum standard.
23:56
And nothing, nothing we've seen coming doing the research for this,
24:01
nothing in any of the the UN reports,
24:03
nothing in even in the United States,
24:05
actually contains any information about
24:08
The idea that the use of force should ~ eventually come back to a human being.
24:13
So on december first, twenty twenty five,
24:16
the General Assembly adopted a resolution on lethal autonomous weapons systems with
24:22
a hundred and sixty-four in favor,
24:24
six against, and seven completely abstained.
24:29
Not sure I need to ask, but who voted no?
24:31
Well, they're all the people you'd think.
24:34
So Belarus, Burundi, North Korea,
24:38
Israel, Russia, and the United States of America.
24:44
There we are. So one hundred and sixty four countries,
24:46
including nearly every US ally.
24:48
Every other NATO member, the EU,
24:52
Japan, and most of Latin America and Africa voted for rules.
24:55
Only six voted against and two of them have permanent security council seats
24:59
and can block consensus in Geneva.
25:01
Right. So in that system, those two,
25:05
specifically China, Russia, or three,
25:08
I guess, China, Russia, the United States,
25:10
they don't need to win the argument.
25:12
Basically nobody can outvote them at all.
25:16
Right? So on september fifth of this year,
25:20
the three year mandate ended with non binding consensus text that people
25:26
in the room describe as much weaker.
25:29
than the draft from just forty-eight hours before that.
25:31
So the US and Russia forced late eradications,
25:37
deletions, w w erasures of ethics,
25:41
predictability, and design and development.
25:45
And each one of those deletions individually tells you something.
25:48
Now you can defend cutting ethics if you argue that
25:51
the international humanitarian law is the legal standard and ethics is a philosophy.
25:55
I disagree, but it's an argument that can be made.
25:58
But design and development has real consequences.
26:01
If obligations only start at the moment you use the weapon,
26:04
then building it, training it,
26:06
selling and exporting it are all untouched.
26:08
The arms control regimes that succeeded regulated production,
26:12
transfer and stockpiles.
26:14
Deleting design and development deletes the only part that would ever reach
26:16
a factory in Ohio.
26:18
And predictability.
26:19
As you said, predictability is a technical term which asks whether a system will
26:23
do will do the same thing twice given similar inputs,
26:25
right? And whether a commander can know in advance what it's going to do.
26:29
For three years the AI industry has described unpredictability as
26:33
a property of these systems, an inherent,
26:35
~ ineradicable. Like it's it's just in there.
26:38
We can't predict what they're doing.
26:39
There's there's such a big black box.
26:41
And something they
26:41
Right.
26:42
don't expect to patch out, right?
26:44
So labs publish papers on capabilities they didn't anticipate
26:47
and run interval evaluations.
26:49
Because training doesn't tell them how a model's gonna behave.
26:51
But some companies have said development should slow down and they want over outside
26:55
oversight. In fact, I think this last weekend all three of the
26:58
big frontier model companies, or three of the big frontier model companies,
27:02
I shouldn't say all three. Because I think Meta was not on board.
27:05
But the other
27:06
Of course not.
27:07
the the others said that you know they won outside oversight.
27:11
Now I I don't know if all that was sincere,
27:12
but it is on the record.
27:14
I mean, I I do think that that they let the publicity do a little bit
27:17
of work on that, right? Like the idea that that hey look,
27:22
we stood against this. We said we need to slow down despite the fact that we're not,
27:26
you know, I mean whatever. ~ but the US delegation actually took
27:31
a position directly on predictability that its own industry
27:36
is contradicting in public. Like the president specifically said this week
27:41
on a plane that
27:43
It doesn't need guardrails, we don't need and w when they asked what he used it for,
27:47
he just said so many things, but I'm not gonna tell you what any of those are.
27:52
Right, and but from memory serves I believe he called in on a podcast
27:55
to say all this. ~ but
27:57
No, this was
27:58
on this was on Air Force One. ~ he was asked specifically outside of
28:02
Air Force One by a reporter, I think it was it was ABC News.
28:05
That's why it was, 'cause he asked he asked like,
28:08
Who are you with? And she and she said,
28:11
we're with ABC News and he was like,
28:13
~ ABC fake news, like it it's but couldn't literally answer what he even uses
28:18
AI for, but said how amazing it was.
28:20
Right.
28:21
Right. So I mean we have we have here the the administration,
28:26
~ which you know, I know Donald Trump is extremely smart and high,
28:30
very high intelligence, but he hasn't
28:32
Highest intelligence actually.
28:33
the highest, right? I mean, but he hasn't studied AI at a at a doctoral level
28:37
as far as I know. but maybe he just asked OpenAI or asked meta's ~ model,
28:43
but anyway or Grok,
28:43
Or Grok, most likely Grok.
28:45
probably Grok, yeah, if we're for being honest there.
28:47
But ~
28:48
But basically we have an administration disputing the facts with
28:52
the companies that are the ones driving this whole thing.
28:54
And they're they're th th they're now of a disagreement not over the values
28:57
of what should be done, but actually what the facts are.
29:00
Are these things predictable or not?
29:02
Yeah, and there are there are people on the ground who are are reporting on this.
29:06
I mean there are there are specific like attaches that are are their only
29:10
job is to work directly inside the UN to report on these things,
29:14
to to discuss what they're discussing and talking about.
29:17
And ~ a a lot of the alarm for this podcast and some of the driver
29:23
for it was specifically one of those reporters.
29:25
but people in the room allegedly tried to stop this.
29:31
Right. They and the coverage tends to describe these outcomes as as if like,
29:36
well, nobody was really responsible for it.
29:38
But Verity Coyle of Amnesty International was actually there in the room,
29:43
and so was ~ Nicole Van Ruihen,
29:46
who leads the Stop Killer Robots Coalition,
29:50
which that's great. Stop killer robots.
29:53
~ and
29:54
Cool business card.
29:55
Robert in Den Bosch, the Dutch diplomat who chaired the group,
30:01
Humans human rights watch documented the session ~ on December seventh.
30:05
by the close of it, seventy six states said that they support moving
30:09
to negotiations before the the November review conference.
30:14
And of course in most bodies that block would be enough to move forward.
30:17
In the UN it counts for very little because the threshold is every state
30:20
in the room.
30:21
Right. And the thing that's probably the most confusing about this
30:25
is that Russia and the United States positions actually look identical on paper.
30:31
Both say that national guidelines are are fine.
30:35
Like we just need national guidelines and and you know,
30:38
restricted at a national level,
30:39
it doesn't need to happen at the UN level,
30:41
basically saying that existing humanitarian laws already apply here and that
30:47
no new binding instruments are actually needed.
30:50
because those, you know, existing human rights laws already apply and they have they
30:56
have their own teeth.
30:57
So they they may get to pretty much the same result,
30:59
but they got there from different directions,
31:01
right? Russia is openly obstructive.
31:03
It blocked the rules of procedure,
31:04
it kept civil society out, and it's fighting a war right now where drone autonomy
31:09
is developing on both sides faster than anyone planned.
31:12
They want no observers and no rules,
31:14
and they've made that painfully clear.
31:17
The American position is a little bit stranger,
31:20
and I think actually more dangerous because officials mean it.
31:22
The US believes it can regulate itself better than a treaty would.
31:26
There are defense departments directives,
31:28
~ review or sorry, Department of War directives,
31:31
~ the review processes and lawyers at every level.
31:34
And many of the people who wrote those policies have thought about this harder than
31:37
their critics.
31:38
But tell why.
31:40
A national guideline is a policy,
31:41
and a policy can be revised, waived,
31:43
or classified by whoever's in office.
31:45
It binds no other country, has no enforcement mechanism,
31:49
and can vanish after a single election.
31:50
To keep it, the US gave up a binding rule that would have constrained
31:54
its adversaries to, probably for a generation,
31:57
and it called that sovereignty.
31:58
So that's I think that's the the biggest hidden point in all of this right there
32:04
is ultimately that the US keeps saying we have to keep going,
32:09
because if we don't, China's gonna overtake us or Russia might overtake us.
32:13
But they're they actually had the opportunity to constrain them as well
32:19
and still fought against that.
32:21
And the the fourteen point six billion dollars in
32:25
Venture capital is not waiting on any of these policies.
32:29
Let's let's take a turn back to Libya,
32:31
which we discussed briefly earlier.
32:33
That's the incident everyone in this debate cites,
32:35
and almost no one describes it carefully,
32:37
so we're gonna go slowly through it.
32:38
In March 2020, so this is six years ago,
32:41
~ outside Tripoli, a civil war was underway between
32:45
the internationally recognized government and forces loyal to
32:48
the commander Khalifa Haf Haftar.
32:50
And Haftar's units broke and retreated,
32:53
according to a UN panel of experts report published in March twenty twenty one.
32:58
five hundred and forty eight page report,
32:59
the retreating forces were hunted down and remotely engaged
33:02
by unmanned combat aerial vehicles and lethal autonomous weapon systems.
33:07
So literally we're talking about ~ retreating soldiers basically hunted down
33:12
and gunned down. The system
33:13
Exactly like
33:15
boats, you know, in the in the Gulf of Mexico and or sorry,
33:18
Gulf of America and the Pacific and Atlantic Oceans.
33:22
I mean, i this is really this is really dark stuff here,
33:27
but like hunted down and remotely engaged.
33:30
Now I will say we did not re read this five hundred and forty eight page report.
33:33
It it we AI summarized it, ~ to be fair.
33:38
But th these are real the this is something that actually happened.
33:42
Sorry to interrupt.
33:42
Right.
33:43
No worries. And the system named the system in this incident was named
33:49
the STM Cargo two, a Turkish loitering munition,
33:54
which is a small drone that carries an explosive charge,
33:56
and it just circles until it finds a target and then flies to that target,
34:00
and one soldier can carry one of these.
34:02
So the report says it was programmed to attack targets without needing
34:05
a data connection between operator and munition,
34:08
a capability the report calls fire,
34:10
forget, and find.
34:12
Ugh. This is so gross. I
34:14
All right.
34:14
mean, these are literally like kamikaze drones,
34:17
but at least in the kamikaze world,
34:21
like it was it was one for one,
34:23
right? Like when during during World War Two,
34:24
Right. A person who did.
34:25
the
34:26
Japan was one for one, right? Like it was a real person choosing to
34:30
to carry out that action. Now,
34:32
depending on who's citing it, right,
34:34
that incident gets presented as either like
34:38
Proof that this is this is a real situation that we really are faced with,
34:42
or they just wave it off like it's a rumor.
34:45
Both of those readings are somewhat wrong.
34:47
The report does not confirm that anyone died.
34:50
So it's not proof, but it it is,
34:52
you know, it's more than just rumor.
34:54
Zachary Callenborne, the analyst who brought it into public debate,
34:58
wrote it up in the Bulletin of the Atomic Scientist in May 2021 under
35:02
a headline that ended with a question mark asking whether a flying killer robot
35:06
was used in Libya. His answer was quite possibly.
35:09
But the uncertainty is the finding.
35:12
We don't know whether a machine killed a person in Libya.
35:15
But there was a UN panel, a five hundred and forty eight page report,
35:17
and a team of investigators, and the answer is still unknown.
35:20
And those were combatants like on an active like battlefield where
35:25
the law provides not very many protections,
35:28
right, than it does for like civilians.
35:30
So if investigators can't establish what actually happened
35:36
to those fighters, a strike on a residential building will be even harder
35:42
to reconstruct or try to figure out after the
35:45
And that's the accountability gap.
35:46
The men in that report also have no names.
35:49
They appear as retreating half tar affiliated forces.
35:52
Most are probably young as soldiers usually are,
35:55
and they were running, which the law of war treats as a situation with specific
36:00
limits on what you may do to a person.
36:02
The judgment about them ended up as a question mark in a UN document,
36:05
and we'll likely never know who they were.
36:08
Timeline is what makes this systemic,
36:11
right? So a report came out in March 2021,
36:13
and the sixth review conference ended in December of 2021.
36:18
That means diplomats in Geneva had the phrase literally,
36:23
and we quote this hunted down and remotely engaged in a UN report right
36:29
in front of them. And they still stripped the substance from their own text
36:34
on the last day.
36:35
And deployment of these weapons has outrun negotiations ever at every stage since
36:40
then. The second case we have is Gaza,
36:43
where AI generated the targets and a person approved them in the end.
36:46
In April 2024, the Israeli journalist Yuval Abraham published an investigation
36:52
in ~ Pl plus nine seven two magazine ~ in local call based
36:58
on interviews with six Israeli intelligence officers who served during the war.
37:03
he had already reported on a system called the Gospel,
37:05
ironically, which
37:06
~ huh.
37:07
generated building targets. this investigation was about a system called Lavender.
37:11
Love these names.
37:13
According to those officers, Lavender was a machine learning opera system that
37:17
scored people in Gaza on how likely they were to be Hamas
37:20
or Islamic jihad operatives. And it produced a list of about thirty seven thousand
37:25
people.
37:26
And how much d like human oversight and human review do you think that last list
37:31
actually got?
37:33
Well the officers involved said reviews sometimes took seconds per target.
37:36
One described his role as a rubber stamp,
37:39
checking mainly that the target was a male.
37:40
There was also a companion system the officers called Where's Daddy,
37:44
which tracked flagged individuals
37:45
~ man.
37:47
and signaled when they entered their family homes.
37:49
That's awful.
37:50
That
37:50
is that is terrible. Meaning strikes were timed for when those
37:55
men were literally walking into their houses to greet their family.
38:01
That's what the reporting describes.
38:02
One documented case was the attempt to kill a Hamas commander named Ibrahim Biari.
38:06
~ it's logged as incident six seven two in the AI incident database.
38:11
The public actor archive researchers ~ used to record harms by ~ caused
38:15
by AI systems.
38:16
I I wanna take a second and just sort of mention that that does exist.
38:20
the AI incident database, but it is a public archive that that researchers use,
38:25
so it's not something that's maintained by any organization or government.
38:28
But Biari's case shows how the parts actually fit together,
38:33
right? According to the r this report,
38:35
the location intelligence was wrong and heavy weapons were used.
38:40
Not we're not talking bullets,
38:41
we're talking bombs, right? Like explosives or ~
38:45
You know, RPGs, those sorts of things,
38:47
and the strike caused destruction and civilian deaths across
38:51
an incredibly dense urban landscape.
38:54
A machine-generated target, meaning AI created target.
39:00
A person looked at it for a few seconds,
39:03
a gigantic heavy weapon system was selected,
39:07
and the information at the start of the chain was entirely incorrect.
39:13
Each step had its own independent justification as far as the reporting shows.
39:17
Nobody, not one person in the chain broke their own organizational procedure,
39:23
but the procedure produced that outcome.
39:27
AI also increases the number of decisions in addition to their speed.
39:30
So most of the alignment focuses on accuracy,
39:33
whether the model is right 90% of the time,
39:35
whether a human would do better.
39:36
The bigger change is volume. A human targeting cell can develop a limited number
39:41
of targets a week because people take time to be sure.
39:44
That limit worked as a moral constraint,
39:46
even though it looked logistical,
39:48
and nobody wrote it into policy because nobody had to.
39:50
You don't have to slow down the human team.
39:53
But install a system that outputs thirty seven thousand names
39:56
and that limit disappears. A decision to strike which used to be rare because
40:00
it took so much human effort becomes cheap.
40:03
So this report is also part of the the Geneva record.
40:07
Human Rights Watch has actually cited in submissions to
40:11
the General Assembly consultations and and Abraham's investigations
40:16
are actually referenced in UN level debate.
40:19
So the connection between Gaza and the words deleted on september fifth,
40:25
twenty twenty six is directly documented in a file.
40:28
And ethics, predictability, and design and development are the three provisions most
40:32
likely to have applied to a system like Lavender.
40:35
So what that means is a few powerful countries keep using the consensus rule
40:41
to strip the teeth out of every document with that the UN is producing,
40:47
every every resolution that the UN is creating at the very last minute.
40:53
And that happened on the final day in the twenty twenty one review conference
40:57
and again in the twenty twenty three ~ conference when Russia blocked
41:01
the rules of procedure and then shut out
41:04
All of civil society. And then again,
41:06
when the US and Russia forced erasures hours before the mandate expired
41:13
on September 5th, 2026. That's the same exact MO,
41:17
same pattern, same tactic, three times,
41:21
and that's only five years. And we're talking about AI.
41:24
So this is going to start happening more and more faster and faster.
41:28
Deployment.
41:30
Also keeps running ahead of the diplomacy,
41:32
right? A UN panel documented the Cargo 2 incident in March of 2021,
41:37
before the expert group had produced any regulatory text,
41:42
whatever whatever, you know, whatsoever.
41:43
Lavender and the gospel were actually in use during a war in 2024.
41:50
That Ander Rails lattice software is deployed within NATO right now.
41:57
And that mandate closed in September with a non binding document.
42:00
And the money keeps showing up wherever the law has a little loophole,
42:06
a little gray area. The Pentagon announced the third offset in twenty fourteen.
42:10
That's the same year the CCW started talking about autonomous weapons systems.
42:14
In twenty twenty six, venture funding passed fourteen point six billion dollars
42:21
in five months, and Andy Rill's valuation doubled to sixty one billion dollars.
42:27
And the treaty process ended with nothing binding anyone to anything.
42:33
And those three trends you're talking about actually reinforce each other.
42:36
The consensus rule produces predictable failure.
42:40
Investors read that failure as a signal,
42:42
and capital builds the technology faster.
42:44
The faster it spreads, the more governments decide they can't accept constraints,
42:48
which strengthens the states using the consensus rule.
42:51
Each cycle makes the next one easier.
42:53
And people in active conflicts in Libya,
42:55
Sudan, Ukraine, and Gaza, they're exposed to AI assisted targeting with
43:01
no oversight, no rule requiring a person to make the decision
43:07
to kill multiple people.
43:09
And the US actually believes it to head here,
43:11
and I think that's a mistake.
43:13
W w let's hear what what do you why do you think it's a mistake?
43:17
Well, it's kinda like what you said earlier.
43:18
Any lead is temporary and shorter than most people assume.
43:21
This is software, and software spreads faster than almost any other technology.
43:25
The cargo too is Turkish. Loitering munitions are being improvised
43:29
and updated weekly in Ukraine workshops,
43:31
Ukrainian workshops by people with no national laboratory.
43:34
A machine that can identify a human figure and fly into it.
43:37
Now it takes a hobby drone, an off the shelf model,
43:40
and someone who can code. The advantage the US protects by refusing rules
43:44
is it disappearing on its own.
43:46
The US is also unusually exposed.
43:48
It's an open society with an enormous area to defend.
43:51
Ports, the power grid, rail lines,
43:54
airports with fences you could climb.
43:56
Autonomous weapons favor attackers since they're cheap,
43:59
expendable, scalable, and indifferent to their own losses.
44:01
A country whose defense strategy depends on projecting force abroad while staying
44:05
safe at home has more to lose from cheap autonomous weapons than just about anybody.
44:10
I I think there's definitely some irony there when it comes to the United States.
44:13
What
44:14
Mm-hmm.
44:14
what do you say of the sort of the AI industry's own concerns and warnings?
44:19
As we talked about, they're they're saying in public that these systems
44:22
are unpredictable. They can't fully explain model behavior,
44:25
and they discover capabilities after release.
44:27
Some have said that development is outpacing understanding,
44:30
that outside oversight is needed and that slowing down is reasonable.
44:33
I don't think they say that purely out of virtue.
44:35
Some of it is positioning, and some of it looks like regulatory capture.
44:39
The technical claim underneath is still true though,
44:41
and serious researchers don't dispute it.
44:43
Now, give that system a weapon and put it on a battlefield full of inputs
44:47
and nobody trained on it trained it on?
44:49
And facing another country's unpredictable system at machine speed,
44:52
with humans just too slow to step in.
44:54
Yeah, think of it like two black boxes,
44:57
right? Like escalating against each other before anyone can wake up the president.
45:02
And I mean the black box that like,
45:04
you know, the nuclear codes and all that stuff.
45:08
Right. And the killer robot imagery has damaged this debate because nobody serious
45:12
is worried about a metal skeleton with a rifle yet.
45:15
the worry is a flash
45:16
I mean
45:16
crash with munitions. two automated systems interacting in a
45:20
way that neither operator predicted for eleven seconds,
45:22
and at the end of those eleven seconds a war has started.
45:25
unless it's never been declared.
45:27
What the US
45:28
Right.
45:28
defends in Geneva is kind of the the option,
45:32
right? It's the it's the decision,
45:34
the right to figure it out later.
45:36
But for eighty years the American approach to international law has been just avoid
45:41
signing anything, right? Keep your options open and assume that this ambiguity
45:46
is gonna eventually favor the strongest country,
45:49
in which case it's mostly been us.
45:52
That approach has a a
45:54
A record a mile long. Drone strikes became normal,
45:57
and this indefinite detention became something that governments just do now.
46:02
Each time the US keeps its options open,
46:05
someone eventually used that open option against
46:11
us or against people we told that we would protect.
46:14
And the objection we needed was one we'd already given up.
46:18
And it it bears saying again, in case you missed it before,
46:22
after three years of this mandate,
46:24
the result is a document nobody has to follow,
46:27
missing three important words ethics,
46:30
predictability, and design and development.
46:32
Three most important words. I think over the next few months,
46:35
if you watch the November Review Conference and the count of the countries that
46:42
are asking for negotiations, which as of September is 76 countries,
46:46
see whether it grows before November and whether those states actually stay inside
46:53
the consensus process or start discussing a separate track in the corridors.
46:59
A separate track is basically how
47:01
The whole Canadian Ottawa thing happened.
47:03
122 different countries in 1997 who decided to go out of the process
47:09
to build a thing that could actually stop horrible things from happening.
47:14
Right? If diplomats start saying alternative forum or general assembly process,
47:20
they've basically decided that they're not going to ask permission anymore.
47:23
And I think that's a good thing for all of them.
47:25
And the same goes for deletions in any negotiation.
47:28
A story that says ~ talks failed to reach agreement usually leaves
47:32
out that most participants did agree.
47:34
In this case, 164 countries agreed.
47:36
~ asks which words were removed and by whom,
47:39
since the words removed words show the actual policy.
47:42
The United States and Russia are are further apart than at any point since
47:47
the Cold War. Except their leaders.
47:50
They are handling things very much the same.
47:53
They disagree about borders and energy,
47:56
who shot down what, the basic facts of the recent history that we're actively living
48:01
through. In a room in Geneva, in the last hours,
48:05
they both agreed that the same word,
48:09
ethics.
48:10
has to come out of this UN document about AI that kills people.
48:16
So under a rule where everyone has to agree,
48:18
the agreement produced that deletion.
48:20
Now you can act on this without calling anyone.
48:22
You know, I don't know how much we can really do right now
48:28
in our current environment because we don't have a whole lot of control over the UN.
48:33
And we certainly don't have control over the people who are supposed
48:36
to be representing our viewpoints at the UN.
48:38
But the UN Office for Disarmament Affairs actually runs a free public document
48:45
library with working papers, national statements,
48:47
and draft texts.
48:48
You can actually pull the draft from before the final session and the text that
48:52
was adopted after it and kind of compare them,
48:54
do a diff like left to right, side by side.
48:56
It takes about 40 minutes, and I'm not gonna lie,
48:59
it's not fun. But afterwards you will know then more than most of
49:04
the attaches and the reporters that are actually covering this in the trenches,
49:08
right? The post is the comparison that any of us can make because those deletions.
49:14
Stay invisible until somebody actually reads them and publishes them
49:18
and says something to somebody else about it.
49:20
If you're one of our listeners who works in the software industry,
49:23
get the end use terms in writing before you write the code.
49:26
Email your employer and ask what the export classification is
49:29
on what you're building, whether it's dual use,
49:31
and what the contract says about integration into targeting systems.
49:35
Most engineers never ask, and often nobody inside the company knows the answer,
49:39
which tells you something too.
49:41
An email creates a paper trail that can come out in discovery,
49:44
which an open letter never will.
49:45
Also go local, ~ because the same companies sell in both directions,
49:50
right? So firms that build autonomous military systems also build flock cameras
49:56
and surveillance towers and sensor fusion and border and police technology
50:01
and ice vehicles and ice systems and city council procurement meetings
50:07
are among the least attended and most winnable rooms in the country.
50:12
You can actually read every single contract.
50:14
You can ask about data retention.
50:15
You can ask about the vendor software,
50:17
whether they make automated decisions about people and get answers on the record.
50:21
A county commission can be made to read a contract out loud,
50:24
and that changes what gets put bought sometimes.
50:27
The mayor of Oklahoma City doesn't apparently give a shit,
50:29
but it can make a difference.
50:33
So ending sort of where we began,
50:35
Christoph Haynes asked for a moratorium in twenty thirteen.
50:38
In the thirteen years since, there's been a probable killing machine killing
50:42
in Libya, a list of thirty seven thirty seven thousand names in Gaza,
50:46
and fourteen point six billion dollars in venture funding in five months,
50:51
and nobody has granted it.
50:53
And as of this month, seventy six countries are asking to negotiate that.
50:57
And the review conference is still coming up in November.
51:00
that's the overlap for this week.
51:02
Joshua, where can they find us if they wanted to continue this discussion?
51:06
FOF dot foundation, that is FOF Friends of the Future.
51:10
Foundation, no dot com, no dot org,
51:14
no dot AI, nothing like that. These are made by real people,
51:18
you and me. ~ you can also check us out on every podcast app of your choosing.
51:24
That can be ~ Apple Podcasts or Spotify or Podbean or wherever you can find
51:29
~ the overlap. If you have trouble finding us,
51:31
Or your friend had trouble finding us,
51:33
tell us tell them to type in F O F F Friends of the Future Foundation
51:37
~ after the overlap and they will be able to find us quickly that way.
51:44
All right. Thanks for being here with me,
51:46
Will.
51:47
Thank you, Joshua. Talk to you next time.
51:49
Talk to you guys next time.