Latest / Kvetch & Release / One Minute to Midnight: Al Hallucination Almost Triggered a US-China War

Transcript

One Minute to Midnight: Al Hallucination Almost Triggered a US-China War · Kvetch & Release

• 0:00
  Speaker 1: It's Monday, and time for Litke's Got a Take. John
• 0:03
  Speaker 1: Litke is here. Good morning, John.
• 0:05
  Speaker 2: Good morning, Jim.
• 0:06
  Speaker 1: Well, we talk a lot about artificial intelligence here on
• 0:08
  Speaker 1: this show, but there's a story coming out, I don't
• 0:11
  Speaker 1: think everybody's heard about it, that could have, and I'm
• 0:14
  Speaker 1: not trying to be alarmist, sparked something like the beginning
• 0:17
  Speaker 1: of World War III. Now, we hear a lot about
• 0:20
  Speaker 1: artificial intelligence messing up high school essays and legal submissions,
• 0:25
  Speaker 1: but an issue came out of the U.S. military. where
• 0:28
  Speaker 1: an AI hallucination almost triggered an armed incident with China.
• 0:34
  Speaker 1: I know the story, but for the uninitiated, walk us
• 0:36
  Speaker 1: through what almost happened.
• 0:39
  Speaker 2: Thanks for having me, Gene. Yeah, this is chilling. So
• 0:42
  Speaker 2: an intelligence analyst in the U.S. military fed raw data
• 0:47
  Speaker 2: about a Chinese cargo ship into an AI chatbot for
• 0:51
  Speaker 2: analysis and to pull it together. And the chatbot completely hallucinated,
• 0:56
  Speaker 2: telling the outputting that the ship was actually carrying nuclear
• 1:01
  Speaker 2: weapon components to the Middle East. Now, that was then
• 1:06
  Speaker 2: compiled again with AI into a report, and the next
• 1:09
  Speaker 2: thing you know, it's passed up the chain, and service
• 1:11
  Speaker 2: members are gearing up to board the vessel, fighter jets
• 1:14
  Speaker 2: are scrambled into the air, and we're on the brink
• 1:17
  Speaker 2: of an international crisis, all because a computer made up
• 1:20
  Speaker 2: a story. They only stopped it because someone finally double-checked
• 1:24
  Speaker 2: the source data at the last second, and a source
• 1:27
  Speaker 2: inside the report actually completely nailed this, saying, AI allows
• 1:31
  Speaker 2: you to get to a bad idea faster.
• 1:34
  Speaker 1: Wow. Military spokespeople say the system worked because an actual
• 1:39
  Speaker 1: human being double-checked the data before any shots were fired
• 1:43
  Speaker 1: or the Chinese boat was boarded. Is that a good assessment,
• 1:46
  Speaker 1: and is that good enough, though?
• 1:48
  Speaker 2: I think it's pure bureaucratic spin. Sure, on paper, the
• 1:51
  Speaker 2: official line is that it worked because a human was
• 1:54
  Speaker 2: in the loop. But let's be real. We crossed the
• 1:56
  Speaker 2: Rubicon here in a massive way, scrambling fighter jets and
• 2:00
  Speaker 2: being ready to board a nuclear adversary's vessel. because a
• 2:06
  Speaker 2: machine's fever dream is way past acceptable margins of safety.
• 2:10
  Speaker 2: In the fog of war, you're operating on a razor's edge, and,
• 2:13
  Speaker 2: you know, one nervous pilot or a standard radio delay problem,
• 2:18
  Speaker 2: and that fake report could become a real war. Philip K.
• 2:21
  Speaker 2: Dick wrote a book, Do Androids Dream of Electric Sheep?
• 2:24
  Speaker 2: It turns out in 2026, AI dreams of Chinese cargo
• 2:28
  Speaker 2: ships hauling nukes.
• 2:30
  Speaker 1: I'm wondering, is this just a failure of technology, or...
• 2:35
  Speaker 1: Is it a classic case of human laziness where an
• 2:38
  Speaker 1: analyst just trusted a chatbot instead of doing their job properly?
• 2:43
  Speaker 2: You know, they're integrating AI throughout all components of the military,
• 2:47
  Speaker 2: but I would say it's both. At its core, it's
• 2:49
  Speaker 2: a catastrophic failure of basic intelligence tradecraft. But what the
• 2:53
  Speaker 2: AI did was launder garbage data into authoritative intelligence. The
• 2:58
  Speaker 2: analysts didn't just ask the chatbot. They used it, as
• 3:01
  Speaker 2: I said, a second time to reformat it into a
• 3:03
  Speaker 2: polished executive draft. So it gave flat-out fiction the institutional
• 3:08
  Speaker 2: shine of a high-level military document, stripping away the time
• 3:12
  Speaker 2: and instincts human beings usually rely on to say, wait
• 3:15
  Speaker 2: a minute, does this actually make any sense?
• 3:17
  Speaker 1: Wow. You know, when I think about global conflicts, I
• 3:20
  Speaker 1: usually worry about rogue actors and miscalculations. So now how
• 3:25
  Speaker 1: dangerous is adding AI into that mix? And I think
• 3:28
  Speaker 1: I already know the answer.
• 3:30
  Speaker 2: I think we're pouring diesel onto the fire here. In
• 3:34
  Speaker 2: geopolitical crises, miscalculations are already the biggest threat that we face. Historically,
• 3:40
  Speaker 2: we worry about a young officer panicking under pressure. Now
• 3:44
  Speaker 2: we're feeding automated hallucinations directly into that pressure cooker. In
• 3:49
  Speaker 2: this instance, Gene, we were literally at, it seems, one
• 3:52
  Speaker 2: minute to midnight with another nuclear superpower, not because of
• 3:56
  Speaker 2: real-world troop movements or actual threats, but because a software
• 4:00
  Speaker 2: program misread it. Oh, yikes.
• 4:04
  Speaker 1: Now, this happened back in the spring. It has been confirmed.
• 4:07
  Speaker 1: It's only coming to light right now. Started out with
• 4:10
  Speaker 1: a few leaks here and there. I'm wondering why we're
• 4:13
  Speaker 1: only hearing about this now, or were they trying to
• 4:15
  Speaker 1: keep on under wraps?
• 4:17
  Speaker 2: Yeah, it's very interesting. This wasn't a DOD, Department of Defense,
• 4:21
  Speaker 2: press release issued out of transparency. This is a leak
• 4:25
  Speaker 2: that's made up of multiple sources. So what it says
• 4:27
  Speaker 2: is that there's massive silent panic inside defense and intelligence
• 4:32
  Speaker 2: circles about how fast the tools are being deployed without
• 4:35
  Speaker 2: proper guardrails. And I don't think there's any surprise that
• 4:38
  Speaker 2: we're hearing this story now, you know, just week after
• 4:41
  Speaker 2: week after week of other really scary stories about AI.
• 4:44
  Speaker 2: It seems like we're building towards something. So someone on
• 4:47
  Speaker 2: the inside saw how close we came to disaster, realized
• 4:51
  Speaker 2: the leadership was pushing speed over accuracy, and decided to
• 4:54
  Speaker 2: blow the whistle before a hallucination gets people killed.
• 4:59
  Speaker 1: Now, there's a bit of a double-edged sword here, I believe.
• 5:02
  Speaker 1: Does a high-stakes near-miss like this force the U.S. military
• 5:06
  Speaker 1: to hit the brakes, or do they double down to
• 5:10
  Speaker 1: stay ahead of China in the AI race?
• 5:12
  Speaker 2: Well, they're doubling down with double-doubles on this one, Gene.
• 5:15
  Speaker 2: I mean, they'll call this a near-miss learning opportunity and
• 5:18
  Speaker 2: patch the software, but then they're going to press the accelerator.
• 5:21
  Speaker 2: Washington sees AI as a zero-sum, must-win Cold War against China.
• 5:28
  Speaker 2: Under the Pentagon's current posture, the mandate is to flood
• 5:31
  Speaker 2: these tools down to millions of service members throughout the
• 5:34
  Speaker 2: entire apparatus as fast as humanly possible.
• 5:38
  Speaker 1: The U.S.
• 5:39
  Speaker 2: Is playing to win the AI game, and their fear
• 5:42
  Speaker 2: of falling behind China completely eclipses their fear of software
• 5:46
  Speaker 2: glitches causing real-world problems.
• 5:48
  Speaker 1: Yeah, we really need some boundaries and some guardrails. Final
• 5:52
  Speaker 1: question here. I always love asking you things like this.
• 5:55
  Speaker 1: Where does this leave... Canadian observers watching this tech continue
• 5:59
  Speaker 1: to roll out in every corner of society. How does
• 6:02
  Speaker 1: it affect us right here in Canada?
• 6:04
  Speaker 2: Thinking about how it impacts us as Canadians is something
• 6:07
  Speaker 2: that I really put some thought into. I mean, we're
• 6:09
  Speaker 2: sitting directly in the wake of this arms race right now,
• 6:12
  Speaker 2: and we're taking part not in the same way, but
• 6:16
  Speaker 2: we're also, you know, dialed in. We're structurally, though, tied
• 6:19
  Speaker 2: into U.S. intelligence networks and NORAD, so if Washington starts
• 6:23
  Speaker 2: making operational decisions based on automated hallucinations, our national security
• 6:29
  Speaker 2: gets instantly dragged into that fallout. But beyond defense, I mean,
• 6:32
  Speaker 2: this is a stark warning. We are outsourcing high stakes
• 6:36
  Speaker 2: human judgment to algorithms built for speed over accuracy. These
• 6:40
  Speaker 2: tools meant to help us make better decisions faster are
• 6:44
  Speaker 2: actually robbing decision makers of the one thing that they
• 6:47
  Speaker 2: need most. Time to pause, verify, and then prevent a catastrophe. So,
• 6:52
  Speaker 2: you know, let's pump the brakes a little bit here
• 6:54
  Speaker 2: because we don't want to rush into World War III,
• 6:57
  Speaker 2: or at least I don't want to.
• 6:58
  Speaker 1: No, I don't want to either. You know, I'm going
• 7:01
  Speaker 1: to be talking tomorrow about another AI story. Just don't
• 7:04
  Speaker 1: have time for it today. But the former CEO of
• 7:09
  Speaker 1: Google is out with his warning about artificial intelligence. And
• 7:14
  Speaker 1: I'll just give you a little tease about what I'm
• 7:16
  Speaker 1: talking about tomorrow. He says, pull the plug right now. Wow.
• 7:23
  Speaker 2: Well, we're hearing it from the highest levels of the industry. So,
• 7:27
  Speaker 2: you know, it says something.
• 7:29
  Speaker 1: Yep. Great take this morning, John. We'll be talking to
• 7:32
  Speaker 1: you tomorrow.
• 7:33
  Speaker 2: Thank you, Gene.
• 7:34
  Speaker 1: John Letke with Letke's Got a Take.