Latest / The Clinical Realist / The Wrong Diagnosis: Why Health Systems Keep Failing at AI

Transcript

The Wrong Diagnosis: Why Health Systems Keep Failing at AI · The Clinical Realist

• 0:00
  SPEAKER_00: Welcome to the Clinical Realist.
• 0:02
  SPEAKER_00: I'm Dr.
• 0:02
  SPEAKER_00: Sarah Matt, physician and digital health strategist.
• 0:05
  SPEAKER_00: This week I want to talk about something I see repeatedly when health systems hit a wall with clinical AI.
• 0:09
  SPEAKER_00: Not a tech wall, a thinking wall.
• 0:12
  SPEAKER_00: I was at SUNY Upstate speaking to some ophthalmology residents last week about AI evaluation, smart physicians, smart physicians, generally engaged, asking good questions about specific tools.
• 0:22
  SPEAKER_00: And I stopped the conversation and asked them, well, before we talk about a tool, can you tell me what framework your department uses to decide whether a tool is safe for your workflow?
• 0:30
  SPEAKER_00: They didn't know.
• 0:31
  SPEAKER_00: Not because they weren't intelligent, but because no one ever asked them that question.
• 0:35
  SPEAKER_00: And that's exactly the problem I want to talk about today.
• 0:38
  SPEAKER_00: When a clinical AI initiative fails, the postmortem always, always sounds like this.
• 0:43
  SPEAKER_00: The tool is not ready, or the vendor overpromised, or adoption was lower than expected.
• 0:48
  SPEAKER_00: These are findings, not diagnoses.
• 0:51
  SPEAKER_00: And they describe what happened.
• 0:52
  SPEAKER_00: They don't tell you why.
• 0:54
  SPEAKER_00: So here's what usually actually happened.
• 0:56
  SPEAKER_00: The health system started probably with the wrong question.
• 0:59
  SPEAKER_00: They probably asked, is this tool good?
• 1:01
  SPEAKER_00: And then they spent six to 18 months trying to answer that question through a pilot.
• 1:05
  SPEAKER_00: The tool was good.
• 1:07
  SPEAKER_00: The pilot worked.
• 1:08
  SPEAKER_00: And then the implementation fell apart.
• 1:11
  SPEAKER_00: Because the tool being good is not the relevant question at the point of adoption.
• 1:15
  SPEAKER_00: The relevant question is, does this organization have the capacity to integrate this tool into the way we actually make decisions?
• 1:22
  SPEAKER_00: Those are the same question.
• 1:24
  SPEAKER_00: And conflaying them is the misdiagnosis.
• 1:26
  SPEAKER_00: So let me be concrete.
• 1:28
  SPEAKER_00: A diagnostic AI tool has 91% sensitivity for a specific condition.
• 1:33
  SPEAKER_00: That's generally impressive and it outperforms the average doc on this task.
• 1:37
  SPEAKER_00: The pilot data confirms it.
• 1:39
  SPEAKER_00: You run it in your, say, cardiology department, and it works.
• 1:42
  SPEAKER_00: But now you try to scale it.
• 1:44
  SPEAKER_00: And you discover that the physicians in your system who are not pilot enthusiasts don't know how to weigh the AI recommendation against their own clinical judgment.
• 1:52
  SPEAKER_00: They either ignore it completely or defer to it without appropriate skepticism.
• 1:56
  SPEAKER_00: And neither behavior is what you wanted.
• 1:59
  SPEAKER_00: The tool's not the problem.
• 2:01
  SPEAKER_00: The tool's still 91% sensitive.
• 2:03
  SPEAKER_00: The problem is that you did not build a framework for how your physician should integrate an AI recommendation into a clinical decision before you deploy the tool.
• 2:11
  SPEAKER_00: You assume the training was explicit.
• 2:15
  SPEAKER_00: It wasn't.
• 2:17
  SPEAKER_00: That assumption is the misdiagnosis.
• 2:20
  SPEAKER_00: So what's the right question and when should you be asking it?
• 2:23
  SPEAKER_00: The right question comes before you even talk to a vendor, before the demo, before the pilot proposal, before the RFP.
• 2:30
  SPEAKER_00: The right question is: does this organization have the governance infrastructure to make a sound decision about clinical AI tools?
• 2:37
  SPEAKER_00: And that question has two to four components.
• 2:40
  SPEAKER_00: First, do you have a defined evaluation framework?
• 2:44
  SPEAKER_00: Not a pilot protocol, a framework.
• 2:46
  SPEAKER_00: A set of criteria you apply to any AI tool before you actually agree to run a pilot.
• 2:51
  SPEAKER_00: What evidence standards do you require?
• 2:53
  SPEAKER_00: What validation data set do you need?
• 2:56
  SPEAKER_00: And what failure modes are you disqualified?
• 2:59
  SPEAKER_00: If you don't have this framework, you're not really ready to evaluate any tool.
• 3:03
  SPEAKER_00: You're just ready to be impressed by a vendor pitch.
• 3:06
  SPEAKER_00: Second, do you have a clinical decision maker with explicit accountability?
• 3:11
  SPEAKER_00: Not a physician who's kind of involved, a clinician with a title, a scope of authority, and explicit accountability for the decision and its downstream clinical consequences.
• 3:20
  SPEAKER_00: This person's career path is connected to this outcome.
• 3:23
  SPEAKER_00: That connection creates that kind of rigor that enthusiasm doesn't.
• 3:28
  SPEAKER_00: Third, do you have a workflow integration plan that actually predates the pilot?
• 3:32
  SPEAKER_00: Not a go-live plan, a plan for how clinicians are going to incorporate the IA recommendation into an existing decision-making workflow.
• 3:39
  SPEAKER_00: This is the gap between the tool makes a recommendation and the clinician uses the recommendation appropriately.
• 3:45
  SPEAKER_00: This is not filled by training slides.
• 3:47
  SPEAKER_00: It's filled by deliberate workflow design.
• 3:51
  SPEAKER_00: Fourth, do you have a governance structure that can say no?
• 3:54
  SPEAKER_00: And this is one most health systems completely skip.
• 3:57
  SPEAKER_00: They build an evaluation process that's designed to approve tools, not reject them.
• 4:02
  SPEAKER_00: The innovation or digital office, they want to win.
• 4:05
  SPEAKER_00: The vendor wants deployment.
• 4:06
  SPEAKER_00: So everyone in the room is incentivized to say yes.
• 4:10
  SPEAKER_00: The governance structure needs a physician, a nurse, a pharmacist, a clinician with a standing authority to say, this tool is not ready for our workflow and have that decision hold.
• 4:21
  SPEAKER_00: If your governance structure can't say no, it's not from the governance structure, is it?
• 4:26
  SPEAKER_00: It's theater.
• 4:28
  SPEAKER_00: So I want to spend a few minutes on the clinician's role in this because I think clinicians consistently underestimate their authority in this conversation.
• 4:36
  SPEAKER_00: When a health system forms a committee to evaluate a clinical AI tool, the typical composition is usually the CIO or a tech leader, the innovation or digital officer, a data scientist or IT director, and one or two doctors, nurses, providers.
• 4:51
  SPEAKER_00: In most of those rooms, the technology leaders dominate the conversation.
• 4:55
  SPEAKER_00: They have the vocabulary, they have the slide deck, and they have an evaluation framework that they use for technology procurement.
• 5:03
  SPEAKER_00: The clinician sits in those meetings and often feels like an advisor, a clinical consultant who gets asked, does this make sense from a clinical perspective?
• 5:11
  SPEAKER_00: Rather than a decision maker who is standing to actually shape the entire process.
• 5:15
  SPEAKER_00: That's wrong.
• 5:16
  SPEAKER_00: It's wrong in a specific, demonstrable, and documentable way.
• 5:21
  SPEAKER_00: A clinician is the only person in that room who's professionally and legally accountable for a clinical AI failure.
• 5:26
  SPEAKER_00: The CIO is not going to be named a malpractice suit if the AI gives a wrong recommendation.
• 5:31
  SPEAKER_00: The innovation officer doesn't have a license on the line.
• 5:34
  SPEAKER_00: The data scientist is not standing in front of the medical board.
• 5:38
  SPEAKER_00: The clinician is.
• 5:39
  SPEAKER_00: That accountability creates authority.
• 5:42
  SPEAKER_00: The physician or nurse or pharmacist, they don't need permission to ask hard questions at the evaluation table.
• 5:48
  SPEAKER_00: They don't need the CIO's buy-in to say, we need a better validation standard before we run this pilot.
• 5:54
  SPEAKER_00: Their professional accountability gives them the authority to shape those decisions.
• 5:59
  SPEAKER_00: Most clinicians are not claiming that authority at all.
• 6:02
  SPEAKER_00: They're showing up to the meeting, listening to the vendor pitch, then asking questions about clinical relevance.
• 6:08
  SPEAKER_00: They're not saying, before we look at any vendor data, let me tell you what evidence standard I require to recommend this tool to my colleagues.
• 6:16
  SPEAKER_00: That sentence said at the first meeting changes everything.
• 6:20
  SPEAKER_00: It establishes that doc, that nurse, that pharmacist, as a gatekeeper, not just an advisor.
• 6:28
  SPEAKER_00: That the evaluation standard is clinical, not just technical.
• 6:31
  SPEAKER_00: And that the approval authority rests with the person who carries the accountability.
• 6:36
  SPEAKER_00: You have that authority.
• 6:38
  SPEAKER_00: Use it early.
• 6:40
  SPEAKER_00: So let me close with what it looks like when an organization actually gets it right.
• 6:44
  SPEAKER_00: The health systems that are scaling clinical AI well are not the ones with the biggest innovation budgets or the newest tech infrastructure.
• 6:50
  SPEAKER_00: They're the ones that have built their governance structure before they even started talking to vendors.
• 6:54
  SPEAKER_00: They have a right evaluation framework.
• 6:56
  SPEAKER_00: It was built by a clinician-led team before any vendor comes in.
• 7:00
  SPEAKER_00: And every vendor that enters the evaluation process is evaluated against that framework, not just against its own pitch.
• 7:06
  SPEAKER_00: They have a clinical executive who owns the AI governance function, and that person has a title, a seat at the senior leadership table, and explicit authority to delay or halt implementation if the evidence standard isn't met.
• 7:18
  SPEAKER_00: And they're not just a committee chair, they're a decision maker.
• 7:21
  SPEAKER_00: They have a workflow integration process that happens in parallel with the pilot, not after.
• 7:26
  SPEAKER_00: And while the clinical team is evaluating the tool's accuracy, the ops team is designing the workflow changes.
• 7:31
  SPEAKER_00: The two tracks converge before go live.
• 7:35
  SPEAKER_00: And they have a governance body that's structurally capable of actually saying no.
• 7:39
  SPEAKER_00: And they've said no before.
• 7:41
  SPEAKER_00: The organization knows that saying no is a legitimate outcome of an evaluation process.
• 7:46
  SPEAKER_00: So these organizations don't have more tech talent than appears.
• 7:49
  SPEAKER_00: They have better diagnostic instincts.
• 7:51
  SPEAKER_00: They ask the right questions first.
• 7:53
  SPEAKER_00: That's the diagnosis.
• 7:56
  SPEAKER_00: So the right question is not, is this tool good?
• 7:59
  SPEAKER_00: The right question is, do we have the framework to evaluate whether this tool is right for our workflow?
• 8:05
  SPEAKER_00: Get the diagnosis right before you touch a vendor.
• 8:09
  SPEAKER_00: Build the governance structure before the pilot.
• 8:12
  SPEAKER_00: Put a clinician in the room with authority, not just presence.
• 8:16
  SPEAKER_00: That's what I help health systems do.
• 8:18
  SPEAKER_00: The framework, the governance design, the evaluation criteria, the clinical leadership coaching.
• 8:23
  SPEAKER_00: If your organization is in the early stages of a clinical AI eval, that's the moment to build it correctly.
• 8:28
  SPEAKER_00: So I'm Dr.
• 8:29
  SPEAKER_00: Saramatt, and this is the clinical realist.
• 8:31
  SPEAKER_00: I'll see you next week.