Latest / MBA Training AI / AI automation bias: when human oversight stops working

Transcript

AI automation bias: when human oversight stops working · MBA Training AI

• 0:00
  Host: Welcome to the Leaders Insights podcast.
• 0:07
  Host: Today's episode, AI systems causing real harm before oversight can catch up.
• 0:12
  Host: Someone almost started a nuclear exchange because a piece of software made something
• 0:15
  Host: up.
• 0:16
  Host: Tell me that's an exaggeration.
• 0:18
  Expert: I wish it were.
• 0:19
  Expert: This week, a military AI system, a decision support tool feeding target recommendations
• 0:24
  Expert: to commanders, hallucinated, meaning it produced a confident output with no basis in reality
• 0:30
  Expert: and flagged Chinese nuclear infrastructure as an active threat.
• 0:35
  Expert: It nearly triggered a US response before a human caught it.
• 0:38
  Host: Nearly.
• 0:39
  Host: So the human oversight worked.
• 0:41
  Host: Isn't this a success story?
• 0:43
  Expert: That's the seductive reading, and it's wrong.
• 0:45
  Expert: It worked once, under scrutiny, with time on the clock.
• 0:49
  Expert: The durable lesson here isn't few.
• 0:52
  Expert: The human caught it.
• 0:54
  Expert: It's that we've built systems whose speed and confidence outrun the humans supposedly
• 0:57
  Expert: watching them.
• 0:59
  Expert: Oversight only counts if it can keep pace, and increasingly, it can't.
• 1:03
  Host: Break that down.
• 1:05
  Host: Why can't a trained operator keep pace?
• 1:07
  Expert: Two reasons.
• 1:08
  Expert: First, volume.
• 1:10
  Expert: When a model spits out a thousand recommendations a day, all fluent, all authoritative sounding,
• 1:16
  Expert: your reviewer stops reviewing and starts rubber stamping.
• 1:19
  Expert: That's automation bias.
• 1:21
  Expert: The human tendency to trust the machine precisely because it's a machine.
• 1:26
  Expert: Second, the good models are right often enough that you relax.
• 1:30
  Expert: A system that's wrong 10% of the time is more dangerous than one that's wrong half
• 1:35
  Expert: the time because you learn to trust it.
• 1:38
  Host: That's counterintuitive.
• 1:39
  Host: Explain.
• 1:40
  Expert: If a colleague was wrong every other sentence, you'd check everything they said.
• 1:44
  Expert: If they're wrong once a month, you stop checking.
• 1:47
  Expert: And the one time they're wrong is the one that hurts.
• 1:51
  Expert: Google's own AI safety team published figures this year suggesting frontier models, the
• 1:56
  Expert: biggest, most capable ones, hallucinate on complex factual queries somewhere between 3
• 2:02
  Expert: and 8% of the time.
• 2:05
  Expert: Worth noting, Google sells these models, so they've every reason to make that number
• 2:09
  Expert: look small.
• 2:10
  Expert: Independent researchers, I trust, think the real figure under adversarial conditions is
• 2:15
  Expert: meaningfully higher.
• 2:17
  Expert: But even 3% at military scale is terrifying.
• 2:20
  Host: So what do the best operators do differently?
• 2:23
  Host: Because everyone claims they have a human in the loop.
• 2:26
  Expert: Human in the loop is the most abused phrase in this industry.
• 2:30
  Expert: Putting a person at the end of a pipeline they can't meaningfully interrogate isn't
• 2:34
  Expert: oversight.
• 2:35
  Expert: It's theater.
• 2:37
  Expert: The operators who get this right do three specific things.
• 2:41
  Expert: They cap throughput deliberately, so a reviewer sees few enough cases to actually think.
• 2:46
  Expert: They force the system to show it's working.
• 2:49
  Expert: The sources, the confidence, the counter-argument, not just the conclusion.
• 2:54
  Expert: And they build in friction, a second independent check for anything above a certain stake's
• 2:59
  Expert: threshold.
• 3:00
  Host: Friction sounds like the opposite of what these systems are sold to deliver.
• 3:04
  Host: Speed.
• 3:05
  Expert: Exactly.
• 3:06
  Expert: And that's the tension nobody wants to say out loud.
• 3:08
  Expert: You buy the AI for speed, then the responsible thing is to slow it back down at the point
• 3:13
  Expert: of consequence.
• 3:14
  Expert: The near miss this week happened because the loop was optimized for fast recommendations,
• 3:19
  Expert: not for catchable errors.
• 3:22
  Expert: There was no requirement that the system expose why it flagged that target.
• 3:26
  Expert: The operator got a conclusion, not a chain of reasoning.
• 3:30
  Host: Play devil's advocate against yourself.
• 3:32
  Host: Isn't the real fix just better models that hallucinate less?
• 3:35
  Expert: Tempting and partly true.
• 3:38
  Expert: Models are improving.
• 3:39
  Expert: But it's a trap.
• 3:41
  Expert: Every efficiency gain from a more reliable model gets spent on giving it more autonomy
• 3:46
  Expert: and less supervision.
• 3:47
  Expert: You never bank the safety, you spend it.
• 3:50
  Expert: That's why this isn't a technology problem you wait out.
• 3:53
  Expert: It's a design discipline.
• 3:55
  Expert: The gap between capability and oversight widens on its own unless you actively hold it open.
• 4:01
  Host: So for someone running AI in a far less dramatic setting, a hospital, a bank, a hiring
• 4:06
  Host: tool, what's the one thing they do Monday morning?
• 4:09
  Expert: Make your highest stakes AI decision and ask, can the person signing off actually see
• 4:14
  Expert: why the system concluded what it did?
• 4:17
  Expert: And do they have time to disagree?
• 4:19
  Expert: If the answer to either is no, you don't have oversight.
• 4:23
  Expert: You have a liability with a login.
• 4:25
  Expert: Fix the reasoning visibility first before you touch the model.
• 4:28
  Host: A liability with a login, that'll do it.
• 4:31
  Host: Sources for today's episode.
• 4:33
  Host: Mars Technica AI, TechCrunch AI, The Decoder, Google AI, Vendor, AI Lab, KD Nuggets.
• 4:41
  Host: Done.
• 4:42
  Host: Want to know where you actually stand?
• 4:44
  Host: Take the AI self-assessment at MBA-training.com.