Latest / The Clinical Realist / When the Algorithm Is a Device: AI, the FDA, and the Question Almost Nobody Is Asking

Transcript

When the Algorithm Is a Device: AI, the FDA, and the Question Almost Nobody Is Asking · The Clinical Realist

• 0:00
  SPEAKER_00: So I want to start with a question that almost no buyer of clinical AI in this country has been asked directly.
• 0:06
  SPEAKER_00: Here it is.
• 0:07
  SPEAKER_00: The last AI tool you started using, the one your group bought last quarter, or the one bundled into the EHR upgrade, the one your hospital pirated on your unit, the one your investor company is selling to the health system you advise.
• 0:22
  SPEAKER_00: Was it cleared by the FDA as a medical device?
• 0:25
  SPEAKER_00: See, most of you don't even know the answer now.
• 0:28
  SPEAKER_00: And that's not a physicism.
• 0:29
  SPEAKER_00: The product probably did not tell you.
• 0:32
  SPEAKER_00: A vendor's fish deck almost certainly did not lead with it.
• 0:35
  SPEAKER_00: It was in there somewhere, maybe.
• 0:36
  SPEAKER_00: And the trainer who walked your team through the workflow definitely didn't bring it up.
• 0:41
  SPEAKER_00: The procurement committee that approved the contract may not have known to ask.
• 0:46
  SPEAKER_00: But the answer matters because if a tool is a medical device under FDA regulations, the rules that govern it, what it can claim, how it has to perform, and what happens when it fails, who's liable when something goes wrong, and what evidence the manufacturer had to produce before it reached your practice are categorically very different from the rules that govern most software in your stack.
• 1:08
  SPEAKER_00: So this is a clinical realist of Dr.
• 1:10
  SPEAKER_00: Sarah Matt.
• 1:10
  SPEAKER_00: On location here at a Rostar.
• 1:13
  SPEAKER_00: Amazing.
• 1:14
  SPEAKER_00: So today we're going to walk through these questions.
• 1:16
  SPEAKER_00: Every clinician, every health system buyer, and every health tech founder in the audience needs to be able to answer clearly.
• 1:24
  SPEAKER_00: How do you tell when an algorithm is a micro device?
• 1:26
  SPEAKER_00: What do the three FDA approvals pathways actually mean about the evidence behind a product?
• 1:31
  SPEAKER_00: And what does professional liability look like when the device is the one making the thought?
• 1:37
  SPEAKER_00: So this episode pairs directly with this week's conversation on medical economics.
• 1:41
  SPEAKER_00: There, I argued that most healthcare AI fails on financial grounds because the user, the decider, and the payer are three different people with three different incentives.
• 1:51
  SPEAKER_00: Today, we're adding a fourth seat to that tape.
• 1:54
  SPEAKER_00: They're regulators.
• 1:55
  SPEAKER_00: And the regulator's answer to the question of whether your AI is a voice shapes everything the other three were operating saying.
• 2:02
  SPEAKER_00: So let's start.
• 2:03
  SPEAKER_00: The FDA defines a medical device very broadly, but statutory language has been in place since 1976 in the medical device amendments.
• 2:13
  SPEAKER_00: So a medical device is any instrument, apparatus, or other article intended for the use in the diagnosis, cure, mitigation, treatment, or prevention of disease, or intended to affect the structure or function of the body.
• 2:26
  SPEAKER_00: Wow, that's a big one.
• 2:27
  SPEAKER_00: The web definition is older than most of the technology use, and maybe older than us.
• 2:31
  SPEAKER_00: It's been doing double duty in the AI era.
• 2:34
  SPEAKER_00: So when the FDA looks at an AI tool, the question is not, is this software?
• 2:39
  SPEAKER_00: Software is everywhere.
• 2:41
  SPEAKER_00: The question is, what is this software claiming to do?
• 2:44
  SPEAKER_00: If the tool is nuclear diagnosis, subtoriate diagnosis, recommended treatment, predicting an outcome that drives a clinical decision or marketing on a clinical condition.
• 2:54
  SPEAKER_00: It's functioning as a medical device.
• 2:55
  SPEAKER_00: So the official regulatory category is software as a medical device, or to say MD.
• 3:01
  SPEAKER_00: Now, first of all, I'm gonna put out there, I am not a lawyer nor a regulator.
• 3:05
  SPEAKER_00: So no matter what, you always have to check.
• 3:08
  SPEAKER_00: In the meantime, though, let's see if we can make sure we all understand this a little bit better.
• 3:12
  SPEAKER_00: So three quick examples to make this concrete.
• 3:14
  SPEAKER_00: Example one.
• 3:16
  SPEAKER_00: So say you have an AI tool that reads a retinal image and tells you that the patient has signs of diabetic retinopathy.
• 3:23
  SPEAKER_00: That's medical device.
• 3:25
  SPEAKER_00: It's making a diagnosis.
• 3:27
  SPEAKER_00: The FDA has cleared at least one product in this category, IDX-DR, and the clearance process took years.
• 3:34
  SPEAKER_00: Now, example two, an AI tool that monitors your charting and suggests an alternative diagnosis you might consider.
• 3:41
  SPEAKER_00: That could be a medical device too.
• 3:43
  SPEAKER_00: It's supported a diagnosis.
• 3:45
  SPEAKER_00: The clearance pathway depends on what the tool claims to detect and how confident the manufacturer is willing to be in writing.
• 3:52
  SPEAKER_00: Example three, an AI scribe that listens to your visit and drafts your no.
• 3:56
  SPEAKER_00: Is that a medical device?
• 3:58
  SPEAKER_00: Probably not.
• 3:59
  SPEAKER_00: Again, and this is currently positioned by most vendors.
• 4:02
  SPEAKER_00: The scribe is drafting documentation, but it's not making it or supporting a clinical decision.
• 4:06
  SPEAKER_00: So the FDA has signaled that ambient documentation tools really sit outside an open advice pathway as long as the physician is the one signing them.
• 4:14
  SPEAKER_00: But and this is the part most procurement teams miss, that boundary is not stable.
• 4:19
  SPEAKER_00: The moment that ambient scribes start suggesting a bone code or surfacing a clinical alert or flagging a possible diagnosis you didn't consider, they may have crossed the line.
• 4:28
  SPEAKER_00: So the tool you bought last quarter as a documentation aid might actually be performing as a clinical decision support.
• 4:33
  SPEAKER_00: And clinical decision support, depending on how the manufacturer markets it, may or may not need VTA clearance.
• 4:38
  SPEAKER_00: So this is also where the 21, 21st century Cures Act starts to matter.
• 4:42
  SPEAKER_00: So Cures passed in 2016 and carved out certain clinical decision support tools from FDA oversight under very specific conditions.
• 4:49
  SPEAKER_00: One of the central positions, the physician has to be able to independently review the basis for recommendation.
• 4:54
  SPEAKER_00: If the AI is a black box and a dot can't see the reasoning, exemption, if the AI is starting to see recommendation and clear logic, the physician can independently evaluate, it might be.
• 5:05
  SPEAKER_00: So the line is drawn in transparency, not on the underlying technology.
• 5:09
  SPEAKER_00: So for the clinicians in the audience, the practical takeaway is sure.
• 5:12
  SPEAKER_00: Ask the question.
• 5:13
  SPEAKER_00: Ask that question of every AI tool in reverse flow.
• 5:16
  SPEAKER_00: At least the medical device.
• 5:17
  SPEAKER_00: If a vendor can't answer, or the answer is we are exempt, ask them to point you to the specific exemption.
• 5:23
  SPEAKER_00: They are a real same.
• 5:24
  SPEAKER_00: For the health tech founders in the audience, the same question applies in reverse.
• 5:27
  SPEAKER_00: So you should be able to answer it clearly and clean.
• 5:30
  SPEAKER_00: Your customers are going to start asking.
• 5:32
  SPEAKER_00: The buyers who get good at this are going to drive procurement standards across Clistry inside the next 18 months.
• 5:38
  SPEAKER_00: And for the health system procurement teams in the audience, the failure mode is uniform.
• 5:42
  SPEAKER_00: Most AI vendors' due diligence today still leads with security and data hit mode.
• 5:46
  SPEAKER_00: And those are necessary.
• 5:47
  SPEAKER_00: They're not necessarily sufficient, though.
• 5:50
  SPEAKER_00: The regulatory status of the product belongs in the first 10 questions of the procurement conversation.
• 5:54
  SPEAKER_00: And if it's not there in your process, that's a process gap to close.
• 5:58
  SPEAKER_00: So you don't need to become a regulatory expert to operate well in this environment.
• 6:02
  SPEAKER_00: What you need to know is that you need to know enough to push the question, and you need the discipline to push it on every patron.
• 6:07
  SPEAKER_00: And once a tool is classified as a medical device, the FDA evaluates it through one of three primary pathways.
• 6:13
  SPEAKER_00: And each pathway means something very different about how reduced the tool was tested before it reached your practice.
• 6:18
  SPEAKER_00: Now remember, just because it's cleared by the FDA doesn't mean it's going to improve critical outcomes.
• 6:23
  SPEAKER_00: It doesn't mean it's going to be ready for your workflow.
• 6:27
  SPEAKER_00: It just means it's cleared from a regular perspective.
• 6:30
  SPEAKER_00: So the audience for this episode, clinicians, buyers, founders should know how to read these.
• 6:35
  SPEAKER_00: First is pathway one, 510K clearance.
• 6:38
  SPEAKER_00: And this is the most common pathway.
• 6:39
  SPEAKER_00: It's also the one that people misunderstand.
• 6:42
  SPEAKER_00: It's not the same as approval.
• 6:44
  SPEAKER_00: A 510K means the manufacturer demonstrated the device is substantially equivalent to a predicate device, one that's already on the game.
• 6:51
  SPEAKER_00: Now bars comparison, not safety and efficacy in the trial sense that we usually associate with the FDA proof role.
• 6:57
  SPEAKER_00: So for AI tools, the 510K pathway is currently the most common route, and it's faster than the alternatives, often six to nine months if you're lucky.
• 7:05
  SPEAKER_00: The trade-off is that the evidence the FDA reviewed before clearance might be relatively limited.
• 7:10
  SPEAKER_00: So if you see a press release that says a clinical AI tool is FDA cleared, and you don't see any other qualifier, that almost means always that it was a 510K.
• 7:18
  SPEAKER_00: So translate that as the manufacturer demonstrated similarity to something else the FDA has previously seen.
• 7:24
  SPEAKER_00: Doesn't mean the tool was studied in a large multi-site clinical shot.
• 7:28
  SPEAKER_00: So this is also where the most common procurement misread happens.
• 7:31
  SPEAKER_00: The phrase FDA cleared gets used in DEX all the time as if it means the same thing as FDA approved.
• 7:36
  SPEAKER_00: And it really doesn't.
• 7:38
  SPEAKER_00: A health system that does not know the difference is going to make different decisions than one that does.
• 7:42
  SPEAKER_00: So the good news is the difference is easy to teach.
• 7:44
  SPEAKER_00: And bad news is that most procurement teams have not been taught.
• 7:48
  SPEAKER_00: Now, what is pathway two?
• 7:50
  SPEAKER_00: Well, this is de novo.
• 7:52
  SPEAKER_00: And the de novo pathway is for novel, lower to moderate risk devices that don't have a predicate to compare against.
• 7:58
  SPEAKER_00: And many of the early AI tools that the FDA cleared, including the ones for diabetic retinopathy detection that I mentioned earlier, went through de novo.
• 8:07
  SPEAKER_00: And de novo process requires a manufacturer to make a risk-based case from scratch and to commit to specific performance characteristics.
• 8:14
  SPEAKER_00: So it's a lot more rigorous than 510K and establishes a new device type that subsequent products can then use as a predicate.
• 8:21
  SPEAKER_00: Now, the de novo pathway is the one to look for if you're evaluating a tool in a clinical area that did not exist as a device category before AI.
• 8:29
  SPEAKER_00: The clearance letter usually contains performance specifications.
• 8:33
  SPEAKER_00: And this is information you can ask the vendor to share.
• 8:35
  SPEAKER_00: As for it, the letter is a public document.
• 8:38
  SPEAKER_00: If they aren't gonna provide it, well, that itself is a signal worthwhile.
• 8:42
  SPEAKER_00: And then there's pathway three, pre-market approval, PMA.
• 8:45
  SPEAKER_00: So this is the most stringent pathway.
• 8:47
  SPEAKER_00: PMA is required for class three devices, the highest risk category.
• 8:51
  SPEAKER_00: PMA requires clinical trial evidence.
• 8:54
  SPEAKER_00: So it's what most physicians have in mind when they think of FDA approval.
• 8:57
  SPEAKER_00: But very few AI tools are currently going through PMA because most are positioned as class two or lower risk class three.
• 9:04
  SPEAKER_00: So if a tool you're evaluating cleared PMA, you can ask for trial data, and it exists by regular requirement.
• 9:11
  SPEAKER_00: So the takeaway from this is very short.
• 9:14
  SPEAKER_00: When a vendor tells you their AI tool is FDA cleared, ask which pathways.
• 9:19
  SPEAKER_00: The answer will tell you what kind of evidence the FDA review.
• 9:24
  SPEAKER_00: And remember, F10K.
• 9:27
  SPEAKER_00: And remember, 510K is a floor, de novo is more substantial, and PMA is the gold standard.
• 9:33
  SPEAKER_00: But none of them is a guarantee that the tool is going to perform well on your specific patient population.
• 9:38
  SPEAKER_00: But all three give you a starting point to evaluate the depth of the regulatory review behind the product.
• 9:44
  SPEAKER_00: Now, for the founders in the audience, the pathway you choose at submission time is a signaling decision.
• 9:48
  SPEAKER_00: Substantially equivalent claims are fast and easy to underwrite, harder to defend in sophisticated procurement room.
• 9:55
  SPEAKER_00: A de novo or PMA track product carries a much higher cost at a longer timeline and substantially harder evidence on the back end.
• 10:03
  SPEAKER_00: So the right answer really depends on the buyer you're selling to and the bar the buyer is going to hold you to.
• 10:08
  SPEAKER_00: Now, if the buyer's sophisticated, maybe the most rigorous pathway you can credibly secure.
• 10:13
  SPEAKER_00: The shortcut is more expensive than it looks.
• 10:17
  SPEAKER_00: So now we get to the question that most nobody is talking about in the AI procurement meeting.
• 10:22
  SPEAKER_00: Liability.
• 10:23
  SPEAKER_00: The standard of care in medicine is what a reasonable physician will similar train and experience would do under similar circumstances.
• 10:29
  SPEAKER_00: And what's the framework that governs malpractice cases?
• 10:32
  SPEAKER_00: It's also what your insurer is underrated against.
• 10:34
  SPEAKER_00: And it was designed way before any clinical workflow involved that black box algorithm.
• 10:39
  SPEAKER_00: So there's three liability questions every provider using an AI tool should be able to answer.
• 10:44
  SPEAKER_00: And there are also three parallel questions every health system procurement team should be able to answer about every AI tool they're solving for.
• 10:50
  SPEAKER_00: I'm gonna walk through all three from both angles.
• 10:54
  SPEAKER_00: So question one if the AI gets it wrong, who's responsible?
• 10:58
  SPEAKER_00: You, the manufacturer, or both?
• 11:01
  SPEAKER_00: Now, the kernel medial answer in most jurisdictions is it really depends on whether you reasonably relied on the tool and whether the tool was being used as the manufacturer.
• 11:08
  SPEAKER_00: If you use the tool off-label, though, or if you ignored a clinical signal that contraindicated the AI output, the picture gets hard.
• 11:15
  SPEAKER_00: The standard of care does not vanish because an algo got involved.
• 11:18
  SPEAKER_00: It just gets harder to articulate.
• 11:20
  SPEAKER_00: Now, on the procurement side, does the vendor indemnify the health listed in the event of an AI-driven inherent event?
• 11:27
  SPEAKER_00: And under what conditions?
• 11:29
  SPEAKER_00: Now, almost every AI vendor contract I've read in the last 12 months tries to push that liability back onto the practice and institution.
• 11:35
  SPEAKER_00: And some of it's very reasonable, but some of it may not be.
• 11:37
  SPEAKER_00: So the clauses to look for are the indemnification scope, the cap on vendor liability, and the exceptions list.
• 11:43
  SPEAKER_00: Now the exceptions list is usually where the operative risk lies.
• 11:47
  SPEAKER_00: Now, question two When the LGO updates, are you using a different device?
• 11:52
  SPEAKER_00: And now this one's unique to AI because most medical devices are static.
• 11:55
  SPEAKER_00: And the fibrillator does not get smarter over time.
• 11:58
  SPEAKER_00: Many AI tools do.
• 12:00
  SPEAKER_00: So the FTA is a guidance side of what it calls predetermined change control, which allows manufacturers to update certain device brands without triggering a brand new clearance.
• 12:10
  SPEAKER_00: Updated software can gain meaningfully different performance.
• 12:14
  SPEAKER_00: And if the AI you trusted six months ago has been retrained, maybe on new data, the new reliance on its output is reliant on a different model.
• 12:22
  SPEAKER_00: So the vendor should be able to tell you when the model is off-updated.
• 12:25
  SPEAKER_00: And if they can't, that's a question with raising.
• 12:27
  SPEAKER_00: The procurement side version of this is: does the contract require the vendor to notify the health system in material model updates?
• 12:34
  SPEAKER_00: Does it give the health system any right to evaluate performance on the new model before it goes live?
• 12:39
  SPEAKER_00: Now, a lot of contracts don't carry that clause by default, and not a lot of procurement teams are asking for it.
• 12:45
  SPEAKER_00: And that's probably a gap you should definitely look into.
• 12:48
  SPEAKER_00: Then there's question three.
• 12:49
  SPEAKER_00: What did the AI actually see and can you reconstruct its reasoning?
• 12:52
  SPEAKER_00: So if you ever face a malpractice action involved in an AI assisted decision, you're gonna be asked what information AI had access to, what its output was, and why you act about it.
• 13:02
  SPEAKER_00: And if the tool does not produce an audit trail, you may be not able to answer those questions.
• 13:07
  SPEAKER_00: Some AI vendors maintain detailed logs of every input and output, others don't.
• 13:12
  SPEAKER_00: The data retention policy would be in the contract.
• 13:14
  SPEAKER_00: So read it before something goes wrong, not after.
• 13:17
  SPEAKER_00: And on the procurement side, does the contract require the vendor to retain inference logs at a level of detail the illegal team can actually use in the position?
• 13:26
  SPEAKER_00: The duration of your statute of limitations exposure.
• 13:29
  SPEAKER_00: Now, most contracts will probably retain logs for 90 or 180 days.
• 13:33
  SPEAKER_00: However, most state malpractice statutes of limitations are two to seven years.
• 13:37
  SPEAKER_00: So the mismatch could be very structural.
• 13:39
  SPEAKER_00: The cure is in the contract red log.
• 13:42
  SPEAKER_00: So I'm not telling you any of this to scare you off of using clinical AI.
• 13:46
  SPEAKER_00: Used well, these tools are saving logs.
• 13:48
  SPEAKER_00: And the diagnostic AI that identifies a melanoma you would have missed is an unambiguous patient benefit.
• 13:54
  SPEAKER_00: The triage tool that flags a sepsis case six hours earlier is a clinical win.
• 13:58
  SPEAKER_00: And AI in clinical practice is here.
• 14:00
  SPEAKER_00: It's going to expand, and the institutions learn how to use it well.
• 14:04
  SPEAKER_00: Will the regulatory and liability of scaffolding in place are going to deliver much better care to more patients while maintaining and managing the downside?
• 14:12
  SPEAKER_00: What I am telling you is that because the regulatory and liability environment is moving faster than the average procurement conversation, permissions, leaders, et cetera, are being handed AI tools and asked to use them.
• 14:24
  SPEAKER_00: And health systems are being asked to sign through-year contracts on products whose model versions are gonna change inside of that contract life, maybe a whole bunch of times.
• 14:33
  SPEAKER_00: Founders for building are being asked to define more evidence-based to procurement teams that are getting more and more sophisticated as reported.
• 14:39
  SPEAKER_00: And the conversations about FDA status, model versioning, audit trails, and standards of care are happening in legal offices, not necessarily in exam rooms or in board rooms.
• 14:48
  SPEAKER_00: And they should be happening all three.
• 14:50
  SPEAKER_00: So I'm gonna close with the operational layer.
• 14:53
  SPEAKER_00: Three S for each of the three seats at this table.
• 14:56
  SPEAKER_00: If you're a clinician using AI tools in your practice this week, ask your vendor whether the tool is regulated as an auto-device.
• 15:02
  SPEAKER_00: If they say yes, switch pathway.
• 15:05
  SPEAKER_00: Each tells you something different about the evidence behind the tool.
• 15:08
  SPEAKER_00: If they say no, we'll ask them to point you to a specific exemption they're relying on.
• 15:12
  SPEAKER_00: Write it down.
• 15:13
  SPEAKER_00: And if a question ever comes up about a clinical decision, if the tool informed, you will want that documentation.
• 15:19
  SPEAKER_00: Now, if you're a health system procurement leader, open your three largest AI contracts, the active ones, this week.
• 15:26
  SPEAKER_00: Read the identification, model versioning, and audit log retention clauses.
• 15:30
  SPEAKER_00: Compare the audit log retention period to your state statute of limitations.
• 15:34
  SPEAKER_00: And if there's a gap document, if there's a gap, document it and route it back to a vendor for renegotiation of the next renewal law.
• 15:44
  SPEAKER_00: If you don't have a renewal coming up soon, document the gap and route to your legal team and your insurance proper.
• 15:49
  SPEAKER_00: The risk doesn't vanish because the contract doesn't address it.
• 15:53
  SPEAKER_00: Now, if you're a health tech founder selling a clinical AI product, the procurement teams you are selling to are about to get really good at this.
• 16:00
  SPEAKER_00: Now, buyers who learn the questions in this episode are gonna drive this standard across the restraint inside the next 15 months.
• 16:06
  SPEAKER_00: So if your product can't answer cleanly, you're gonna lose deals you would have won six months ago.
• 16:10
  SPEAKER_00: The fix is really straightforward.
• 16:12
  SPEAKER_00: It's right a one-page regulatory posture.
• 16:14
  SPEAKER_00: Make sure you name your FDA pathway.
• 16:16
  SPEAKER_00: Make sure your model versioning policy is clear and your audit log retention is also documented.
• 16:22
  SPEAKER_00: Remember, you're gonna wanna negotiate a demnification posture.
• 16:27
  SPEAKER_00: When you have a really hot prospect, make sure they have it.
• 16:29
  SPEAKER_00: We have first procurement conversation.
• 16:31
  SPEAKER_00: The deals are gonna make tons faster and you're gonna sleep a lot better.
• 16:35
  SPEAKER_00: Now, if you're an investor sitting on the board of a health AI company, ask the file of these questions at your next board meeting.
• 16:40
  SPEAKER_00: And if the answers are not raised, it's your data point.
• 16:43
  SPEAKER_00: The valuation premium for clean regulatory posture is going to widen substantially this year.
• 16:48
  SPEAKER_00: So help your portfolio company close the gap before the next round.
• 16:53
  SPEAKER_00: So I want to bring this back to the question I started with.
• 16:57
  SPEAKER_00: The last AI tool you started using was accurate by the FDA as a medical device.
• 17:04
  SPEAKER_00: Most of you still don't know the answer, but after this episode, you should know what to find, how to find it, and what the answer tells you, and what the follow-up question should be.
• 17:12
  SPEAKER_00: That's the goal.
• 17:14
  SPEAKER_00: So clinical AI is not slowing down.
• 17:16
  SPEAKER_00: The regulatory environment is going to keep moving, the liability frameworks are going to keep evolving, and the institutions and the founders who treat this question as foundational, not optional, are gonna define how this technology gets deployed in the next decade.
• 17:30
  SPEAKER_00: You absolutely don't need to be a regulatory expert, but you need to be regulatorily literalish.
• 17:35
  SPEAKER_00: There's a big difference.
• 17:49
  SPEAKER_00: I'll see you next Wednesday.
• 17:50
  SPEAKER_00: Thanks for joining the Clinical Realist.