Latest / The Clinical Realist / What Physicians Actually Need to Know About AI Contracts Before They Sign Anything

Transcript

What Physicians Actually Need to Know About AI Contracts Before They Sign Anything · The Clinical Realist

• 0:00
  SPEAKER_00: A physician executive at a mid-sized health system called me after their AI assisted diagnostic tool produced a recommendation that contributed to a delayed diagnosis.
• 0:10
  SPEAKER_00: The vendor's response was very textbook.
• 0:12
  SPEAKER_00: The tool performed within its validated parameters.
• 0:16
  SPEAKER_00: The physician was the one accountable in the peer review.
• 0:19
  SPEAKER_00: She had signed the clinical validation form 11 months earlier.
• 0:23
  SPEAKER_00: She had not read the indemnification clause, because almost no physician actually does.
• 0:29
  SPEAKER_00: That conversation is why I'm recording this episode.
• 0:33
  SPEAKER_00: Welcome back.
• 0:34
  SPEAKER_00: I'm Dr.
• 0:34
  SPEAKER_00: Sarah Matt, and this is the show for physician executives and health system leaders navigating the real operational and strategic implications of AI and clinical settings.
• 0:43
  SPEAKER_00: Today's episode is about AI contracts.
• 0:46
  SPEAKER_00: Specifically, what's in them that physicians need to understand before their name goes on the clinical validation line.
• 0:53
  SPEAKER_00: This is not a legal lecture.
• 0:54
  SPEAKER_00: I'm not a lawyer, thank goodness, and nothing here is legal advice.
• 0:58
  SPEAKER_00: What I am is someone who sat in rooms where the aftermath of those contracts lighted squarely on physicians who had no idea what they had agreed to.
• 1:07
  SPEAKER_00: So if you're a CMO, a CMIO, a physician executive, a CNIO, a CHIO, whoever you are, if you're a clinician executive who has been asked to sign off on an AI deployment in the past 18 months, this episode is for you.
• 1:22
  SPEAKER_00: So let me describe a procurement workflow that will probably sound very familiar.
• 1:26
  SPEAKER_00: IT and legal negotiate the contract, finance signs off on the spend, and the clinical executive is brought in to validate clinical appropriateness and sign an internal attestation.
• 1:36
  SPEAKER_00: None of that internal process is the actual vendor contract, but the vendor contract almost always includes language that assigns post-deployment clinical oversight responsibility to a named physician, physician role, or other clinician.
• 1:50
  SPEAKER_00: And that person becomes the accountable party when performance questions arise.
• 1:55
  SPEAKER_00: So why don't physicians read the vendor contracts?
• 1:58
  SPEAKER_00: Well, there's three big reasons.
• 2:00
  SPEAKER_00: First of all, they're long.
• 2:01
  SPEAKER_00: They're written in technology and commercial law language that takes actual legal training to parse.
• 2:06
  SPEAKER_00: And the physician is typically given 72 hours to validate a tool that took the vendor six months to build a business case for in the first place.
• 2:14
  SPEAKER_00: There's there's no structural space in that process for the physician to actually engage with the contract appropriately.
• 2:20
  SPEAKER_00: So the assumption is that legal has handled the legal parts and legal has for the business terms, but what legal has not always handled is the clinical accountability structure, because legal is not in a position to assess clinical risk 100%.
• 2:35
  SPEAKER_00: So I want to give you a concrete example of how this plays out.
• 2:38
  SPEAKER_00: Now, I'm using the word physician executive, but again, we all know that other clinicians may be in this spot, whether you're nurses, physical therapists, pharmacists, you name it.
• 2:47
  SPEAKER_00: So remember that as I go through this talk.
• 2:50
  SPEAKER_00: So a physician executive was asked to sign off on an ambient documentation tool.
• 2:54
  SPEAKER_00: She had 48 hours notice, a demo that showed impressive workflow, impressive efficiency numbers, and an internalization form from her CMO's office.
• 3:03
  SPEAKER_00: She signed it.
• 3:04
  SPEAKER_00: 18 months later, when a documentation gap in a complex patient case was traced to the AI's transcription of a physician's verbal assessment, she was the name clinical oversight officer in the vendor contract.
• 3:17
  SPEAKER_00: The vendor's position was that the tool had transcribed accurately.
• 3:20
  SPEAKER_00: The clinical judgment about what to transcribe and verify remained the physician's actual responsibility.
• 3:27
  SPEAKER_00: So she hadn't known that she was named in the vendor contract at all, much less as the clinical oversight officer.
• 3:33
  SPEAKER_00: And that's not a vendor betrayal.
• 3:35
  SPEAKER_00: That's what the contract said.
• 3:36
  SPEAKER_00: She just had not read the contract.
• 3:39
  SPEAKER_00: So the accountability mismatch that this creates is not subtle.
• 3:42
  SPEAKER_00: You can be named in a contract you never saw for deployment you had limited authority over.
• 3:47
  SPEAKER_00: And that combination, accountability without visibility, responsibility without genuine authority, that's the structural problem this episode is designed to help you prevent.
• 3:58
  SPEAKER_00: So you don't need to read every page of an AI vendor contract.
• 4:02
  SPEAKER_00: You do need, you absolutely do need to know what to look for in four specific areas.
• 4:05
  SPEAKER_00: So the first is indemnification and liability allocation.
• 4:09
  SPEAKER_00: So who's responsible when the tool produces a bad outcome?
• 4:13
  SPEAKER_00: And that outcome, that output actually influences a clinical decision.
• 4:17
  SPEAKER_00: Almost every AI vendor contract contains language that says the vendor is absolutely not liable for clinical decisions made using the tool's output because clinical judgment remains the physician's responsibility.
• 4:28
  SPEAKER_00: And that is standard and in isolation somewhat reasonable.
• 4:31
  SPEAKER_00: The physician is the one with the license and the clinical training.
• 4:36
  SPEAKER_00: What's not standard and what most physicians never see coming is the contract language that names the physician validator personally as a clinical accountability layer rather than the organization.
• 4:46
  SPEAKER_00: So when a peer review or regulatory inquiry traces a patient outcome to an AI tool, there's a significant legal difference between the accountability being held by your institution and the accountability being held by the physician whose name appears on the validation form.
• 5:00
  SPEAKER_00: Know which one you're signing.
• 5:03
  SPEAKER_00: The second clause is validation scope and limitation.
• 5:06
  SPEAKER_00: So the vendor validated their tool in a specific population on a specific data set under very specific conditions.
• 5:13
  SPEAKER_00: And that validation scope is in the contract.
• 5:15
  SPEAKER_00: When your organization deploys a tool outside that scope, so a different patient population, a different EHR integration, a different workflow context, the vendor's validated performance data no longer applies.
• 5:27
  SPEAKER_00: And the physician who signed the clinical validation form did so against the vendor's stated validation scope.
• 5:32
  SPEAKER_00: So if deployment has drifted beyond that scope since the form was signed, and of course it's going to, the validation form is now essentially meaningless.
• 5:40
  SPEAKER_00: And the physician is still the one accountable.
• 5:44
  SPEAKER_00: The third clause covers post-deployment performance monitoring requirements.
• 5:48
  SPEAKER_00: So most AI vendor contracts include a requirement for ongoing clinical performance monitoring.
• 5:52
  SPEAKER_00: The question is, who owns that monitoring obligation?
• 5:56
  SPEAKER_00: If the contract names the CMO's office, that's an organizational commitment.
• 6:00
  SPEAKER_00: If it names a specific role and that role turns over, the monitoring obligation does not automatically transfer to whoever fills the role next.
• 6:08
  SPEAKER_00: In practice, many health systems sign contracts with explicit monitoring obligations and then stand up whatever monitoring structure they have bandwidth and budget for, which is often not the structure the contract specified.
• 6:19
  SPEAKER_00: And the absence of required monitoring can become a liability amplifier when you need the documentation trail to demonstrate due diligence.
• 6:26
  SPEAKER_00: The fourth is the black box access clause, or more accurately, the lack of one.
• 6:31
  SPEAKER_00: So when something goes wrong with an AI output, can your organization actually access the model's reasoning?
• 6:37
  SPEAKER_00: Some contracts explicitly limit explainability.
• 6:40
  SPEAKER_00: Others are just simply silent on it, which creates ambiguity at the worst possible moment.
• 6:45
  SPEAKER_00: So if your contract does not explicitly give you the right to request an explanation of how the model produced a given output, you may not have that right in a peer review or regulatory context.
• 6:55
  SPEAKER_00: And that's a negotiating point that almost no health system thinks to raise before signing, because it only becomes urgent after something has already gone terribly wrong.
• 7:03
  SPEAKER_00: The clause I see cause the most consistent confusion in health system conversations is the monitoring clause, specifically the gap between what the contract requires and what the organization has actually built.
• 7:14
  SPEAKER_00: And I've been in more than one conversation where the CMO was learning in real time that their vendor contract specified monthly clinical performance reviews routed to a named executive, and the organization had been submitting quarterly reports to a committee that was not the named executive.
• 7:29
  SPEAKER_00: So the contract had been out of compliance in letter for two years while being compliant in spirit.
• 7:35
  SPEAKER_00: That's the kind of gap that looks very different before an incident than it does after one.
• 7:42
  SPEAKER_00: So you don't need to become a technology lawyer.
• 7:45
  SPEAKER_00: Heaven forbid, we don't need more technology lawyers.
• 7:47
  SPEAKER_00: You need five specific questions answered in writing before your name is on any clinical validation document.
• 7:54
  SPEAKER_00: Question one What's the validated scope of this tool?
• 7:58
  SPEAKER_00: Specific means the specific population, the specific data set, and the specific workflow context the vendor used in their validation studies.
• 8:06
  SPEAKER_00: In writing, if the vendor can't answer this question with precision, that's not a detail gap.
• 8:11
  SPEAKER_00: That's a signal about the rigor of their validation process.
• 8:14
  SPEAKER_00: So you're being asked to validate clinical appropriateness.
• 8:17
  SPEAKER_00: You can't do that without knowing what the clinical appropriateness claim is actually based on.
• 8:22
  SPEAKER_00: Question two.
• 8:24
  SPEAKER_00: What happens to our organization's clinical accountability if the tool is deployed outside its validated scope?
• 8:30
  SPEAKER_00: Ask your legal team, not the vendor.
• 8:32
  SPEAKER_00: The vendor's answer will reflect optimism about their product's generalizability.
• 8:36
  SPEAKER_00: Your legal team's answer will reflect the contract terms, and those are often significantly different.
• 8:41
  SPEAKER_00: So you want the legal answer before the deployment is planned, not after the expansion is approved.
• 8:47
  SPEAKER_00: Question three.
• 8:48
  SPEAKER_00: Who in this organization is named in the vendor contract as the clinical accountability owner?
• 8:53
  SPEAKER_00: And do they know?
• 8:55
  SPEAKER_00: This is often a CMO role placeholder that the current CMO has never seen in writing.
• 8:59
  SPEAKER_00: It's not uncommon common for a physician executive to be contractually named as the clinical accountability owner for a deployment they joined 18 months after the contract was signed, because the role was named rather than the individual.
• 9:11
  SPEAKER_00: And that physician has no context for the original validation conditions, no institutional memory of the procurement, conversation, and full contractual accountability.
• 9:19
  SPEAKER_00: So the answer to this question should be a specific named person who has seen the contract language that names them.
• 9:26
  SPEAKER_00: Question four, what are the post-deployment monitoring requirements in the contract?
• 9:30
  SPEAKER_00: And has the organization already built the infrastructure to meet them?
• 9:34
  SPEAKER_00: The second half of that question is the one that almost never gets asked.
• 9:38
  SPEAKER_00: Organizations sign contracts with monitoring requirements and then build whatever monitoring structure they have bandwidth for.
• 9:44
  SPEAKER_00: And if the answer to the second half of that is that you're going to figure it out, well, that's a flag, especially because the monitoring requirement is usually what produces the documentation trail that either demonstrates due diligence or establishes the absence of it.
• 9:59
  SPEAKER_00: And question five: if this tool contributes to a patient harm event, what does our vendor contract say about our right to access the model's reasoning for peer review?
• 10:08
  SPEAKER_00: Ask your legal team to point you to the specific clause before you sign, not after.
• 10:13
  SPEAKER_00: This question is uncomfortable to ask because it sounds like you're anticipating failure.
• 10:18
  SPEAKER_00: You're not.
• 10:18
  SPEAKER_00: You're doing the same thing you would do when you review a consent form or verify a medication before administration.
• 10:23
  SPEAKER_00: You're doing the work that the situation actually requires.
• 10:27
  SPEAKER_00: So let me give you a specific example of what happens when these questions are not asked.
• 10:33
  SPEAKER_00: So a health system deployed a clinical decision support tool for sepsis screening in their ER.
• 10:37
  SPEAKER_00: The pilot results were strong.
• 10:40
  SPEAKER_00: Expansion was approved at the board level based on the pilot summary, and the vendor contract named the CMIO as a clinical accountability officer.
• 10:47
  SPEAKER_00: The CMIO in that role had been there for eight months when the contract was signed, two years earlier by the previous CMIO.
• 10:54
  SPEAKER_00: The post-deployment monitoring requirement in the contract specified weekly performance reviews routed to the CMIO directly.
• 11:01
  SPEAKER_00: Now, the actual monitoring structure that had been built was a monthly committee report that did not go to the CMIO at all.
• 11:07
  SPEAKER_00: So when a sepsis screening failure was traced to a model drift event, the review revealed that model drift had been visible in the performance data for 11 weeks.
• 11:15
  SPEAKER_00: The weekly monitoring, the contract required, would have caught it probably at week two.
• 11:20
  SPEAKER_00: The current CMIO was named in the contract.
• 11:23
  SPEAKER_00: So the monitoring gap was documented.
• 11:25
  SPEAKER_00: The combination of those two facts defined what the subsequent months look like for that physician and that institution.
• 11:31
  SPEAKER_00: None of the five questions above have been asked when the contract was signed.
• 11:36
  SPEAKER_00: So here's something that's true and rarely stated clearly.
• 11:40
  SPEAKER_00: Physician executives have significant leverage in AI contract negotiations.
• 11:45
  SPEAKER_00: Most don't use it.
• 11:46
  SPEAKER_00: Not because they lack the standing to do so, but because they have been brought into the process way too late to deploy it effectively.
• 11:54
  SPEAKER_00: So AI vendors need physician executive sign-off to close health system deals.
• 11:58
  SPEAKER_00: That's genuine leverage, not symbolic influence.
• 12:01
  SPEAKER_00: Health systems that can't demonstrate physician leadership in AI governance are increasingly unable to get board approval for AI investment.
• 12:08
  SPEAKER_00: And vendors know this.
• 12:09
  SPEAKER_00: So the physician executive is imprimature is part of the business case.
• 12:14
  SPEAKER_00: So the question is when you deploy that leverage and when you don't.
• 12:18
  SPEAKER_00: Post-term sheet, you have almost no link, no leverage whatsoever.
• 12:21
  SPEAKER_00: The business case has been built, the executive relationships have been established, the vendor has already invested significant time and resources in reaching the point of term sheet.
• 12:31
  SPEAKER_00: So asking to renegotiate identification language at that stage will essentially be treated as a procurement delay, not a legitimate clinical concern.
• 12:40
  SPEAKER_00: So you'll be managed rather than heard.
• 12:42
  SPEAKER_00: Now, pre-RFP, you have real leverage.
• 12:45
  SPEAKER_00: If the physician exec is involved in defining the vendor evaluation criteria before the RFP goes out, the clinical performance requirements, the accountability structure, the explainability standards, those requirements become part of the vendor's proposal and the eventual contract negotiation.
• 13:01
  SPEAKER_00: So you're not asking for changes to a contract that's already structured.
• 13:04
  SPEAKER_00: You're defying the terms the vendor is agreeing to when they choose to respond.
• 13:09
  SPEAKER_00: So the practical implication is that a specific organizational conversation needs to occur.
• 13:14
  SPEAKER_00: If you're not being brought into AI procurement before the RFP is issued, that's the conversation to have with your CEO and COO.
• 13:21
  SPEAKER_00: Not I want to be involved earlier.
• 13:23
  SPEAKER_00: That framing asks for a favor.
• 13:24
  SPEAKER_00: The framing that works is here is the clinical accountability structure that will govern this deployment.
• 13:30
  SPEAKER_00: Here is what happens to this organization's liability posture if that structure is not defined before the vendor relationship is established.
• 13:37
  SPEAKER_00: Here is what I need to be in a position to protect both the organization and our patients.
• 13:42
  SPEAKER_00: That's not asking to be included.
• 13:44
  SPEAKER_00: That's defined the conditions under which your accountability can be actually meaningful.
• 13:49
  SPEAKER_00: So I went into a procurement cycle early enough to shape the RFP once, and I have gone in late enough to only review a finished contract twice.
• 13:56
  SPEAKER_00: And I will tell you the difference.
• 13:58
  SPEAKER_00: Early involvement looks like this.
• 14:00
  SPEAKER_00: I was brought in before vendors were solicited.
• 14:02
  SPEAKER_00: We spent two hours defining what clinical accountability would mean for this type of deployment, what explainability requirements we would include, and what monitoring structure the contract would need to specify.
• 14:12
  SPEAKER_00: And those requirements, they went right into the RFP.
• 14:15
  SPEAKER_00: Three of the four vendors who responded had not faced those requirements before.
• 14:19
  SPEAKER_00: Two adjusted their contract terms to meet our specifications before the first demo.
• 14:23
  SPEAKER_00: And one declined to respond.
• 14:25
  SPEAKER_00: So the fourth had robust documentation for every requirement we listed.
• 14:29
  SPEAKER_00: And that fourth vendor got the deal.
• 14:31
  SPEAKER_00: Partly because their documentation made the legal and clinical review significantly faster.
• 14:35
  SPEAKER_00: And that's what early involvement looks like.
• 14:38
  SPEAKER_00: It's not slower, it's faster because the accountability structure is defined before the first demo, not negotiated after the contract is drafted.
• 14:47
  SPEAKER_00: So nothing I've said to you today requires you to become a technology procurement expert.
• 14:51
  SPEAKER_00: What it requires is a specific posture shift from I will validate what has been selected to I will define the clinical accountability requirements before selection begins.
• 15:02
  SPEAKER_00: So the contracts are written by people who understand tech and commercial law.
• 15:06
  SPEAKER_00: They're not written by people who understand the daily reality of clinical operations.
• 15:10
  SPEAKER_00: They don't understand workflow pressure or what happens to a physician's career when a peer review goes the wrong way.
• 15:16
  SPEAKER_00: That's your expertise.
• 15:17
  SPEAKER_00: The only way it gets into the contract is if you are in the room when the contract is being shaped.
• 15:23
  SPEAKER_00: That conversation about who gets to be in the room and when, that's what episode eight was about.
• 15:28
  SPEAKER_00: So if you haven't listened to that one, go back and start there.
• 15:31
  SPEAKER_00: Now I'm Dr.
• 15:32
  SPEAKER_00: Sarah Matt.
• 15:33
  SPEAKER_00: New episodes drop every Wednesday.
• 15:34
  SPEAKER_00: And if this was useful, share it with a physician exec who's heading into the AI procurement cycle.
• 15:39
  SPEAKER_00: Or forward it to the CMO who just got handed a vendor contract to validate.
• 15:43
  SPEAKER_00: They're gonna thank you.