Latest / MBA Training AI / AI hallucinations: the near-strike on fabricated intel

Transcript

AI hallucinations: the near-strike on fabricated intel · MBA Training AI

• 0:00
  Host: Leaders' insights. Today, one hallucinated component list almost started a U.S. military
• 0:10
  Host: strike. The mistake almost everyone makes is treating a hallucination like a bug you
• 0:14
  Host: can patch out. It isn't. It's the machine doing exactly what it was built to do.
• 0:20
  Expert: Right. A large language model, software that predicts the next plausible word, not the
• 0:25
  Expert: next true one, doesn't know the difference between a real Chinese nuclear component and
• 0:30
  Expert: a convincing sounding fake. It never did. Fluency and accuracy are separate axes and
• 0:36
  Host: we keep confusing them. So walk me through what actually happened. A military unit nearly
• 0:41
  Expert: ordered a strike off a fabricated component list? The intelligence packet included specifications
• 0:47
  Expert: for hardware that, as far as anyone can verify, does not exist. Part numbers, tolerances, the
• 0:54
  Expert: works, all generated, all wrong, all sitting inside a document that looked identical to
• 0:59
  Expert: human sourced analysis. Nobody flagged it because the output carried no marker saying
• 1:04
  Host: I made this up. That's the part that chills me. It doesn't
• 1:08
  Expert: hedge. It hedges when it's least dangerous and
• 1:11
  Expert: asserts when it's most dangerous. Ask it your dinner plans. You get maybe consider.
• 1:17
  Expert: Ask it for a weapons spec. You get seven confident decimal places.
• 1:21
  Host: Let me bring you three things people believe about this. First one, hallucinations are
• 1:26
  Host: getting rarer as models get bigger. So this problem solves itself.
• 1:30
  Expert: Half true and the half that's false is the one that'll hurt you. Google's own numbers
• 1:36
  Expert: suggest their newer models hallucinate less on benchmark questions, though flag that Google
• 1:41
  Expert: is an AI lab selling these systems. And we have no independent audit to cross check
• 1:46
  Expert: them today. Even taking the figure at face value, rarer
• 1:51
  Expert: means the errors are harder to catch, not fewer at scale. A mistake that shows up one
• 1:56
  Expert: time in 500, spread across a million intelligence documents, is 2000 fabrications you now trust
• 2:03
  Expert: more because they're surrounded by correct ones.
• 2:06
  Host: So bigger models make the survivors more camouflaged.
• 2:09
  Expert: Precisely. You've bred a better liar, not an honest one.
• 2:13
  Host: Some believe this was a human failure, not an AI one. Someone should have checked.
• 2:18
  Expert: Wrong, and I'll die on this hill. The system was designed so that checking was optional.
• 2:24
  Expert: When you drop machine output into the same format, same font, same folder, as verified
• 2:29
  Expert: human work, you've engineered the shortcut. Blaming the analyst is like installing a
• 2:34
  Expert: door with no lock and blaming the guy who walked through it.
• 2:38
  Expert: The failure is upstream. In whoever decided AI-generated text needed no provenance tag.
• 2:44
  Host: No, this came from a model-y stamp. Provenance meaning where the information
• 2:48
  Expert: came from. The paper trail. Who or what produced each
• 2:52
  Expert: claim? In finance, a trade you can't source gets flagged automatically.
• 2:58
  Expert: In this pipeline, a claim you couldn't source sailed straight toward a launch authorization.
• 3:03
  Host: Uncomfortable question, then. If provenance is that simple, why doesn't every serious
• 3:09
  Expert: operation already do it? Because it's boring and it slows things down.
• 3:13
  Expert: And slow is the one thing high-stakes decision chains are punished for.
• 3:18
  Expert: Everybody wants the speed of the model and the accountability of the old process.
• 3:22
  Expert: You can't have both for free. You buy accountability with friction, and
• 3:26
  Expert: most organizations quietly decided friction was the enemy.
• 3:30
  Host: Third belief, put a human in the loop and you're safe.
• 3:33
  Expert: Half true. A human in the loop who's reviewing 40 documents
• 3:37
  Expert: an hour is not a safeguard. He's a rubber stamp with a pulse.
• 3:41
  Expert: The research on automation bias, the human tendency to defer to whatever the machine
• 3:46
  Expert: says, is brutal. People override a correct instinct because the
• 3:51
  Expert: confident output made them doubt themselves. So the loop only works if the human has
• 3:56
  Expert: the time, the authority, and the provenance data to actually push back.
• 4:01
  Host: Strip any one of those, and the loop is theater. So what does someone running one of these
• 4:05
  Expert: pipelines do Monday morning? Tag every machine-generated claim so it's
• 4:10
  Expert: visually distinct from verified human sourcing. Different color, a banner, anything.
• 4:16
  Expert: If a reader can't instantly tell what came from a model, your review process is fiction.
• 4:21
  Expert: That one change would have stopped this incident cold, and it costs you nothing but
• 4:25
  Host: a little friction. Friction being the price of not launching
• 4:29
  Expert: a missile at a rumor. Cheapest insurance in the building.
• 4:32
  Host: Sources for today's episode, The Decoder TechCrunch AI, ours Technica AI, Google AI,
• 4:38
  Host: vendor AI Lab. That's your five. The full AI track, structured
• 4:42
  Host: and free of fluff, is at MBA-training.com.