Latest / The Clinical Realist / The 90% Problem: Why Clinical AI Pilots Don't Scale

Transcript

The 90% Problem: Why Clinical AI Pilots Don't Scale · The Clinical Realist

• 0:00
  SPEAKER_00: Every AI pilot succeeds.
• 0:03
  SPEAKER_00: And that's not actually a compliment.
• 0:04
  SPEAKER_00: It's a design flaw.
• 0:06
  SPEAKER_00: So when a health system announces that a pilot demonstrated positive results and they're expanding deployment system-wide, they've often proven very little about whether that tool is actually going to work in the real clinical environment they're about to send it into.
• 0:18
  SPEAKER_00: The gap between what pilots prove and what health systems need to actually know before expansion is where most AI deployments go wrong.
• 0:25
  SPEAKER_00: So I'm Dr.
• 0:26
  SPEAKER_00: Sarah Matt, and this is the Clinical Realist, where we work through what AI and healthcare actually requires.
• 0:31
  SPEAKER_00: Not what vendors promise and not what conference panels describe, but what the operational and clinical reality look like when you're the one accountable for making those tools work.
• 0:39
  SPEAKER_00: So today we're talking about clinical validation in healthcare AI, specifically the structural reasons why most AI pilots and health systems are designed to succeed, and why that success actually tells you almost nothing about whether you should expand deployment.
• 0:52
  SPEAKER_00: This episode is for CMIOs, CMOs, quality officers, physician executives, CNOs, executives that are clinical in any way, anyone who is the clinical validation process owner or who's being handed pilot results and asked to sign off on them system-wide for a rollout.
• 1:10
  SPEAKER_00: So if you've ever looked at a pilot summary deck and felt like something was off but couldn't name exactly what it was, this episode's for you.
• 1:18
  SPEAKER_00: So let me start with the structure of a standard healthcare AI pilot, because the structure is actually the problem.
• 1:23
  SPEAKER_00: A typical pilot runs on a motivated clinical team, usually the team that requested the tool or champion the vendor relationship.
• 1:30
  SPEAKER_00: And it runs on a controlled patient population over a very defined time frame, with the vendor providing implementation support that will not be present at scale.
• 1:37
  SPEAKER_00: These conditions are not representative of the real-world deployment at all.
• 1:42
  SPEAKER_00: They're actually optimal conditions.
• 1:44
  SPEAKER_00: So the pilot is designed to demonstrate capability, not to predict performance under realistic operational pressure.
• 1:50
  SPEAKER_00: Three forces combine to guarantee that most pilots produce positive results, regardless of how effective the underlying tool actually is.
• 1:57
  SPEAKER_00: The first is what we call the Hawthorne effect in clinical AI.
• 2:01
  SPEAKER_00: When clinical staff know they're being observed and evaluated, their behavior changes.
• 2:06
  SPEAKER_00: They document more carefully, they follow protocols more consistently, they engage with the tool more deliberately.
• 2:12
  SPEAKER_00: So the pilot results capture that behavioral change as much as they capture the tool's performance.
• 2:17
  SPEAKER_00: So when the pilot ends and the observation ends with it, performance often regresses toward the mean.
• 2:23
  SPEAKER_00: What the pilot measured as a tool performance improvement was partially a staff behavior improvement that was never sustainable at scale.
• 2:30
  SPEAKER_00: The second is volunteer bias.
• 2:33
  SPEAKER_00: The clinical teams who participate in AI pilots are almost always the teams most interested in the technology.
• 2:38
  SPEAKER_00: They're not representative of the clinical staff who will be required to use the tool system-wide.
• 2:43
  SPEAKER_00: Pilot results capture the performance ceiling for motivated early adopters.
• 2:47
  SPEAKER_00: Rollout performance is going to reflect the median clinical staff members' engagement with a tool they did not volunteer to use, were not involved in selecting, and were told is now part of their workflow.
• 2:57
  SPEAKER_00: Those are not the same people.
• 2:59
  SPEAKER_00: They do not produce the same numbers.
• 3:02
  SPEAKER_00: The third is vendor support as a confounder.
• 3:05
  SPEAKER_00: During pilots, vendors are typically providing implementation support, prompting really quick troubleshooting, and active optimization.
• 3:13
  SPEAKER_00: That level of support does not continue at scale.
• 3:16
  SPEAKER_00: What the pilot measures as tool performance is often partially vendor support performance.
• 3:20
  SPEAKER_00: So when the vendor steps back after Go Live, the tool performs differently.
• 3:24
  SPEAKER_00: And not because the tool changed, but because a significant component of its effective performance was the vendor's involvement in making it work.
• 3:32
  SPEAKER_00: And I want to give you a concrete example of what that looks like in practice.
• 3:35
  SPEAKER_00: So a large academic medical center piloted an AI-assisted radiology workflow tool on one of their highest performing radiology teams.
• 3:42
  SPEAKER_00: So the team had requested the tool, had a physician champion who had been following the vendor for two years, and received 12 weeks of embedded vendor support during the pilot.
• 3:51
  SPEAKER_00: The pilot showed results that showed 22% reduction in turnaround time and strong clinician satisfaction scores.
• 3:58
  SPEAKER_00: The system approved full radiology department rollout.
• 4:01
  SPEAKER_00: Now, rollout performance at six months showed an 8% reduction in turnaround time on the original team and essentially no measurable effect on the other seven radiology teams.
• 4:11
  SPEAKER_00: The tool had not changed.
• 4:13
  SPEAKER_00: What had changed was removal of the three conditions that made the pilot work.
• 4:18
  SPEAKER_00: The motivated team, the vendor support, and the observation effect.
• 4:22
  SPEAKER_00: The pilot had proved that the tool could work under optimal conditions.
• 4:26
  SPEAKER_00: It had proved nothing about whether it would work under normal conditions.
• 4:30
  SPEAKER_00: And that distinction cost the system approximately 18 months of remediation effort and a second vendor engagement.
• 4:38
  SPEAKER_00: Even when pilots are conducted in good faith, and most are, let's be honest, they're definitely there's no bad actors here.
• 4:46
  SPEAKER_00: But there's still four common design flaws that are going to make their results statistically and operationally uninterpretable as evidence for system-wide rollout.
• 4:54
  SPEAKER_00: Design flaw one is sample size and population match.
• 4:57
  SPEAKER_00: Most health system AI pilots run for 60 to 90 days on one or two clinical units.
• 5:02
  SPEAKER_00: And that sample size is rarely adequate to detect meaningful effect sizes for the outcomes that actually matter: adverse events, near misses, downstream clinical outcomes.
• 5:11
  SPEAKER_00: Pilots are sized to demonstrate workflow integration, not to produce statistically valid clinical outcome data.
• 5:18
  SPEAKER_00: So when a pilot summary reports improved outcomes, it's almost always reporting directional trends in small samples, not validated evidence of clinical effect.
• 5:27
  SPEAKER_00: There's a meaningful difference between we saw a positive trend and we have validated evidence.
• 5:32
  SPEAKER_00: Most pilot summaries describe the first and kind of present it as the second.
• 5:37
  SPEAKER_00: Design flaw two, the comparator problem.
• 5:41
  SPEAKER_00: So what is the AI tool being compared to?
• 5:44
  SPEAKER_00: In most pilots, it's compared to the baseline performance of the participating clinical team before the tool was introduced.
• 5:50
  SPEAKER_00: And that's not a rigorous comparator.
• 5:52
  SPEAKER_00: The baseline performance of a team that just completed training on a new tool that is being actively observed and that has extra vendor support available is not the same as the counterfactual performance of that team without the tool.
• 6:05
  SPEAKER_00: Without a true control group, a comparable clinical team in the same period operating without the tool, the pilot can't really establish that the tool caused the improvement.
• 6:14
  SPEAKER_00: It can only establish that the improvement occurred.
• 6:16
  SPEAKER_00: And those are not equivalent claims.
• 6:19
  SPEAKER_00: Design flaw three, outcome selection.
• 6:23
  SPEAKER_00: Pilots typically measure what is measurable in 60 to 90 days, throughput, documentation time, alert acknowledgement rates, clinician satisfaction scores.
• 6:32
  SPEAKER_00: And these are not the outcomes that matter most to your patients or your malpractice carrier.
• 6:36
  SPEAKER_00: They're the outcomes that are convenient to measure in a short time frame.
• 6:40
  SPEAKER_00: The clinical outcomes that determine whether a tool is actually safe and effective, downstream adverse event rates, diagnostic accuracy at scale, long-term workflow challenges, those require longer time frames than most pilots run.
• 6:53
  SPEAKER_00: A pilot that shows improved documentation time is not a pilot that shows improved patient outcomes.
• 6:58
  SPEAKER_00: That equivalence is often implied in pilot presentations and almost never justified.
• 7:04
  SPEAKER_00: Design flaw four, missing the workflow failure modes.
• 7:07
  SPEAKER_00: So pilots catch workflow success.
• 7:09
  SPEAKER_00: They rarely surface workflow failure modes because failure modes appear at low frequencies in edge cases and in the combinations of patient complexity and workflow pressure that are underrepresented in control pilots.
• 7:21
  SPEAKER_00: The failure mode that matters in a real clinical deployment is the one that appears in the 400th patient, not the first 50.
• 7:28
  SPEAKER_00: Pilots don't get to 400 patients under realistic conditions.
• 7:32
  SPEAKER_00: So the failure mode goes undetected until it appears in the rollout at a point where the organizational commitment to the deployment is already fully made and reversal is politically difficult.
• 7:42
  SPEAKER_00: So I've seen this pattern with clinical decision support tools specifically.
• 7:45
  SPEAKER_00: The failure mode that tends to matter is not the tool giving a clearly wrong answer in an obviously complex case.
• 7:52
  SPEAKER_00: Clinicians catch those.
• 7:54
  SPEAKER_00: The failure mode that matters is actually the tool giving a subtly insufficient answer on a case that looks routine but is not.
• 8:00
  SPEAKER_00: At the moment in a shift where workflow pressure is highest, at verification is likely the least.
• 8:06
  SPEAKER_00: That combination does not appear in a six-day pilot on a motivated team with a vendor representative available for questions.
• 8:12
  SPEAKER_00: It appears in month seven of a system-wide rollout on a night shift on a Sunday.
• 8:18
  SPEAKER_00: So you don't have to run a randomized control trial before you deploy AI.
• 8:22
  SPEAKER_00: I'm not saying that.
• 8:28
  SPEAKER_00: So here's what that framework could look like.
• 8:31
  SPEAKER_00: The most important single change is prospective criteria defined before the pilot starts.
• 8:38
  SPEAKER_00: Before the pilot begins, define in writing what results would lead to expansion and what results would lead to stopping.
• 8:45
  SPEAKER_00: Not, we'll review the data and decide.
• 8:48
  SPEAKER_00: Specific performance thresholds defined in advance, agreed to by the clinical and operational teams.
• 8:54
  SPEAKER_00: This forces the question of what you actually need to see before the vendor's pilot results are on the table and the organizational pressure to expand is already in motion.
• 9:02
  SPEAKER_00: So when the pilot summary arrives, you're not deciding whether the results are good enough.
• 9:07
  SPEAKER_00: You're checking whether results meet the criteria you already specified.
• 9:11
  SPEAKER_00: Those are very different conversations with very different political dynamics.
• 9:15
  SPEAKER_00: The second element is stage deployment, not pilot to full rollout.
• 9:20
  SPEAKER_00: The binary of pilot succeeded, now we expand, is a false choice.
• 9:25
  SPEAKER_00: A stage deployment, one unit, then two units, then one facility, then a second facility, it gives you multiple cycles of performance data under increasingly realistic conditions.
• 9:33
  SPEAKER_00: Each stage allows you to see whether the pilot results hold when the conditions are made, that the pilot succeeded are progressively removed.
• 9:41
  SPEAKER_00: So you're not betting the organization on a single 90-day data set.
• 9:45
  SPEAKER_00: You're building an evidence base for expansion decisions.
• 9:48
  SPEAKER_00: The third element is pre-specified failure mode identification.
• 9:52
  SPEAKER_00: So before deployment, ask the vendor for the documented failure modes their internal testing identified.
• 9:58
  SPEAKER_00: Ask what patient populations the model underperformed on in validation.
• 10:02
  SPEAKER_00: And ask what workflow conditions produce the worst outputs.
• 10:06
  SPEAKER_00: This information exists.
• 10:07
  SPEAKER_00: Vendors generate it in the course of building the tool.
• 10:10
  SPEAKER_00: Most health systems never actually ask for it.
• 10:13
  SPEAKER_00: So the ones that do can design monitoring that specifically watches for the failure modes the vendor already knows about rather than discovering them when they appear in the deployment.
• 10:22
  SPEAKER_00: So I've pushed a vendor on this in a pre-deployment conversation once.
• 10:26
  SPEAKER_00: And the initial response was that their tool had performed consistently across all patient populations in their validation study.
• 10:32
  SPEAKER_00: I asked for the validation documentation.
• 10:34
  SPEAKER_00: And the documentation showed that their validation study had a 43% exclusion rate for patients with more than three comorbidities, which was absolutely the patient population we were deploying into.
• 10:44
  SPEAKER_00: And so the tool had been validated on a population that was structurally less complex than the patient panel we wanted to use it for.
• 10:50
  SPEAKER_00: And that's not a vendor deception.
• 10:52
  SPEAKER_00: It was a validation scope limitation that would not have surfaced unless I had actually asked for the documentation.
• 10:57
  SPEAKER_00: So we negotiated a modified monitoring structure that specifically tracked performance on high comorbidity patients in the first 90 days of the deployment.
• 11:06
  SPEAKER_00: The failure mode appeared on schedule at roughly week 11, and we caught it before it became a patient safety event.
• 11:13
  SPEAKER_00: The fourth element is clinician-reported friction as a leading indicator.
• 11:17
  SPEAKER_00: The earliest signal that a tool will fail at scale is often the friction clinicians report during the pilot.
• 11:23
  SPEAKER_00: Not friction as complaint, but friction as workflow signal.
• 11:26
  SPEAKER_00: Where does a tool create more work than it saves?
• 11:29
  SPEAKER_00: Where does the output require the most verification?
• 11:32
  SPEAKER_00: And where do clinicians report overriding the tool most frequently?
• 11:36
  SPEAKER_00: Those friction points in the pilot are the preview of the failure modes in the rollout.
• 11:41
  SPEAKER_00: Most pilot evaluation frameworks treat clinical friction as a change management problem to address with training.
• 11:47
  SPEAKER_00: A more useful frame is that friction is diagnostic data about where the tool is not working as designed, or where it's working as designed and it was designed poorly.
• 11:59
  SPEAKER_00: Raising clinical validation concerns about an AI pilot, it's not anti-I and it's not anti-progress.
• 12:05
  SPEAKER_00: It's the clinical leader's job.
• 12:07
  SPEAKER_00: The challenge is framing it so it lands in that way.
• 12:11
  SPEAKER_00: The framing problem is real.
• 12:13
  SPEAKER_00: And when clinical executives, physician executives, raise clinical validation concerns about AI pilots, they're often read as skeptical about AI in general, or as protecting turf or slowing progress.
• 12:24
  SPEAKER_00: And that reading is wrong.
• 12:25
  SPEAKER_00: And it's also completely unnecessary.
• 12:27
  SPEAKER_00: The frame that works is this.
• 12:32
  SPEAKER_00: That requires us to know what the pilot actually proved.
• 12:35
  SPEAKER_00: Here's what I need to see in the data before I can sign off on expansion.
• 12:40
  SPEAKER_00: That's not a no.
• 12:41
  SPEAKER_00: That's a conditional yes with specified conditions.
• 12:44
  SPEAKER_00: The difference matters because it positions the clinical executive as a partner in making the deployment succeed rather than as an obstacle to get to the launch mode.
• 12:56
  SPEAKER_00: The distinction between technical validation and clinical validation is also worth naming explicitly in these conversations.
• 13:03
  SPEAKER_00: IT invers are equipped to validate technical performance, uptime, integration, output generation, they're not necessarily equipped to validate clinical performance.
• 13:12
  SPEAKER_00: Whether the outputs are clinically appropriate, whether the workflow integration is safe, whether the failure modes are acceptable in the patient population you're serving.
• 13:19
  SPEAKER_00: So clinical validation is not a check the box exercise.
• 13:23
  SPEAKER_00: It's a substantive review that requires real clinical judgment.
• 13:27
  SPEAKER_00: So the clinical executive who owns clinical validation, they're not doing duplicative work.
• 13:32
  SPEAKER_00: They're doing work that no one else in the organization can do.
• 13:36
  SPEAKER_00: What to ask for from the vendor before signing off on expansion?
• 13:40
  SPEAKER_00: Well, the full validation documentation, not the pilot summary deck, the performance data by patient subgroup, the failure mode documentation from their internal testing, the comparator methodology the pilot used.
• 13:52
  SPEAKER_00: If the vendor can't or will not prove these, that is information about the tool's readiness for expansion.
• 13:58
  SPEAKER_00: Specifically, that the vendor does not have the documentation that would justify the expansion you are being asked to approve.
• 14:04
  SPEAKER_00: And let me address the institutional standing question directly, because I know what's going to come up.
• 14:09
  SPEAKER_00: Physician executives, nursing executives, and other clinical executives who are accountable for clinical outcomes have both the standing and the obligation to ask for validation evidence that supports that accountability.
• 14:20
  SPEAKER_00: This is not optional due to diligence.
• 14:23
  SPEAKER_00: If a patient harm event occurs after expansion and it emerges that the validation data supporting expansion was adequate, that clinical executive who signed off, they're in the review meeting.
• 14:33
  SPEAKER_00: That context is the backing for every hard validation conversation.
• 14:38
  SPEAKER_00: So you're not being difficult when you ask for the evidence base.
• 14:41
  SPEAKER_00: You're doing the job that your accountability actually requires.
• 14:45
  SPEAKER_00: So the goal of this episode is not to make you more skeptical of AI.
• 14:49
  SPEAKER_00: It's to make you more precise about what validation evidence you actually need before your accountability is attached to a deployment.
• 14:55
  SPEAKER_00: Pilots are not evidence of readiness, they're evidence of possibility.
• 15:00
  SPEAKER_00: The gap between these two things is governance, specifically the governance decisions that determine what you need to see before expansion begins.
• 15:08
  SPEAKER_00: Who has authority to slow or stop a deployment if the post-launch data diverges from the pilot?
• 15:13
  SPEAKER_00: And how clinical validation concerns get to the decision table in time to matter.
• 15:18
  SPEAKER_00: So if your health system is currently in a gap between a successful pilot and a system-wide rollout decision, this is the moment to ask for the data.
• 15:26
  SPEAKER_00: Not after.
• 15:27
  SPEAKER_00: Not after the expansion is approved.
• 15:29
  SPEAKER_00: Now.
• 15:30
  SPEAKER_00: So I'm Dr.
• 15:30
  SPEAKER_00: Sarah Matt.
• 15:31
  SPEAKER_00: New episodes of the Clinical Realist come out every Wednesday.
• 15:34
  SPEAKER_00: And if this was useful for you, share it with a clinical leader who is about to be handed a pilot results deck and asked to sign off the expansion.
• 15:40
  SPEAKER_00: They need this conversation before that meeting, not after.
• 15:44
  SPEAKER_00: See you next time.