Latest / MBA Training AI / Clinical AI explainability: why clinicians ignore alerts

Transcript

Clinical AI explainability: why clinicians ignore alerts · MBA Training AI

• 0:00
  Welcome to the Leaders Insights podcast.
• 0:08
  Today's episode, why frontline clinicians don't trust clinical AI and what actually
• 0:13
  changes that.
• 0:14
  Here's something that should embarrass the entire clinical AI industry.
• 0:18
  Most of these tools don't fail because the model is wrong.
• 0:21
  They fail because nobody in the room knows when to believe it.
• 0:24
  Right.
• 0:25
  And that's the part everyone gets backwards.
• 0:28
  Nurses obsess over accuracy on a validation set, a curated batch of test cases where the
• 0:33
  model already knows the answer.
• 0:36
  Then it hits a real ward and the nurse just… ignores it.
• 0:40
  Why ignore it though?
• 0:42
  If the thing is right 94% of the time, isn't skepticism the irrational move?
• 0:47
  Because right 94% of the time is useless at the bedside.
• 0:51
  The clinician doesn't get a lifetime average.
• 0:54
  They get one patient, one alert, right now.
• 0:57
  And no way to tell if this is one of the 94 or one of the 6.
• 1:01
  So it's a trust problem, not a math problem.
• 1:04
  It's a calibration problem.
• 1:06
  Calibration meaning.
• 1:07
  Does the model know when it doesn't know?
• 1:09
  A good system says, I'm 60% confident, and here's why I'm shaky.
• 1:14
  A bad one barks the same confident tone, whether it's certain or guessing.
• 1:19
  Clinicians spot that fast, and once they catch it lying with a straight face once,
• 1:24
  Give me the concrete example.
• 1:26
  Where did this actually blow up?
• 1:28
  The classic is Epic's sepsis prediction model.
• 1:31
  Sepsis being that runaway infection response that kills fast.
• 1:35
  It got deployed across hundreds of US hospitals.
• 1:38
  An external study out of Michigan found it missed most sepsis cases and flooded staff
• 1:43
  with false alarms.
• 1:44
  And here's the durable lesson.
• 1:46
  The failure wasn't just accuracy.
• 1:48
  It was that clinicians had no window into why it fired.
• 1:52
  So they turned it into wallpaper.
• 1:54
  Alert fatigue.
• 1:55
  Alert fatigue, which is a polite way of saying you trained your best people to ignore the software.
• 2:01
  That's worse than no software, because now you've spent the budget and eroded trust for the next tool too.
• 2:07
  Okay, so explainability is the fix.
• 2:09
  But explainability is one of those words that means everything and nothing.
• 2:13
  What does it actually look like on a screen?
• 2:16
  Two things and they're humble.
• 2:18
  One, show the evidence not the reasoning.
• 2:21
  Don't tell me the neural network's internal weights.
• 2:24
  Tell me flagging this because lactate is climbing and heart rate's been up for six hours.
• 2:30
  That's a sentence a doctor can argue with.
• 2:33
  Two, show uncertainty honestly.
• 2:36
  Confidence bans, not a single number pretending to be gospel.
• 2:39
  You said argue with.
• 2:41
  That's interesting.
• 2:42
  You want the doctor to be able to overrule it.
• 2:44
  Every time.
• 2:46
  The tools that get adopted treat the clinician as the decision maker
• 2:49
  and the AI as a loud, well-read intern.
• 2:53
  The ones that get abandoned try to be the attending physician.
• 2:56
  Nobody trusts an intern who won't show their work.
• 2:59
  Now, the numbers.
• 3:00
  I know OpenAI has been pushing figures on clinical accuracy this year.
• 3:04
  They have and their medical benchmark results are genuinely strong.
• 3:08
  But flag it.
• 3:09
  OpenAI is an AI lab selling the models.
• 3:12
  So treat their internal scores as a sales brochure until someone independent checks them.
• 3:18
  On the adoption side, I lean on O'Reilly's radar surveys, which track what practitioners
• 3:24
  actually deploy, not what vendors wish they'd deploy.
• 3:28
  And the gap between model performs and model gets used is enormous.
• 3:33
  How enormous.
• 3:34
  Ballpark.
• 3:35
  O'Reilly's work has consistently shown that the blocker isn't model quality.
• 3:40
  It's trust, integration, and workflow.
• 3:43
  A brilliant model bolted onto a broken workflow dies.
• 3:47
  A decent model that explains itself and fits how people already work gets used daily.
• 3:52
  So the operators who win aren't the ones with the best model.
• 3:55
  They're the ones who understood before GoLeave that trust is a feature you design, not
• 4:00
  a thing that shows up after launch because your accuracy is high.
• 4:04
  The best health system AI leaders run a trust review alongside the clinical validation.
• 4:11
  Separate sign off?
• 4:12
  Give me the one thing a listener does Monday morning.
• 4:15
  Before you approve any clinical AI, sit a front line clinician in front of it and ask,
• 4:20
  can you tell me when this is wrong?
• 4:22
  If they can't answer, you don't have a tool.
• 4:25
  You have expensive wallpaper.
• 4:27
  Sources for today's episode.
• 4:28
  TechCrunch AI, Ars Technica AI, The Decoder, O'Reilly Radar, OpenAI, Vendor, AI Lab,
• 4:36
  KD Nuggets.
• 4:37
  Done.
• 4:38
  Want to know where you actually stand?
• 4:40
  Take the AI self-assessment at MBA-training.com.