Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Doctor Has a Conflict of Interest
Transcript
- Lucas: So there's this study that came out in the Journal of Medical Ethics last month that I think is going to be one of those watershed papers people reference for years. Luna: I saw that one — it's about financial ties between AI developers and drug companies, right? Lucas: Exactly. Researchers looked at 47 clinical AI tools that had been cleared by the FDA over the past three years, and they found that roughly 30 percent of them were co-developed or sponsored by a pharmaceutical company whose products the algorithm was likely to recommend. Luna: So the algorithm is basically nudging doctors to prescribe the drug from the company that helped build it. That seems like a pretty clear conflict of interest. Lucas: It is, and the paper walks through some concrete examples. One that stood out to me was an AI system designed to detect sepsis early in hospital patients. Sepsis is time-critical — every hour of delayed treatment increases mortality by something like 7 percent. Luna: Right, so speed is everything. And the algorithm is supposed to help clinicians act faster. Lucas: It did flag patients faster, but the study found that the treatment recommendations it surfaced were disproportionately for a specific brand of broad-spectrum antibiotic — the same brand manufactured by the company that had a financial stake in the AI vendor. The generic alternative was equally effective and cheaper, but the system just didn't surface it as prominently. Luna: So the bias isn't that the algorithm gives wrong advice — it's that it gives tilted advice. And in a time-pressured environment like sepsis care, a clinician might just go with what the system suggests without second-guessing. Lucas: That's the part that worries the authors most. They call it 'algorithmic steering' — and they argue that because the FDA's current clearance pathway for ai based software as a medical device doesn't require developers to disclose sponsorship or ownership ties, hospitals often have no idea these conflicts exist. Luna: I want to dig into that more. But first, let's make sure listeners know what we mean by 'cleared by the FDA.' This isn't the same as a drug going through clinical trials, right? Lucas: No, it's a much lighter process. Most clinical AI tools go through what's called the 510 pathway, where the developer just has to show the new device is 'substantially equivalent' to something already on the market. They don't have to prove clinical effectiveness for the specific use case. And they definitely don't have to reveal who funded the development. Luna: So a sepsis AI that might influence treatment decisions for thousands of patients can get cleared without anyone checking whether the company behind it also sells the drug it's recommending. Lucas: That's exactly the loophole the study highlights. And it's not just sepsis. They found similar patterns in tools for radiology — one algorithm that helps radiologists prioritize which scans to read first was shown to flag scans that were more likely to require a specific contrast agent manufactured by the same parent company. Luna: Interesting. So the contrast agent is more expensive, but the algorithm makes it look like the preferred option. Lucas: Right. And here's the thing: a radiologist might never know that the algorithm is steering them. They just think the AI is helping them triage. The paper's lead author, Dr. Mariana Chen, told me in an interview that when she presented these findings to hospital administrators, most of them were shocked because their procurement processes never even asked about financial ties. Luna: And I imagine the AI vendors push back. Probably say the algorithm is objective, that the training data is clean, that there's no explicit instruction to favor any product. Lucas: They do. And technically, they might be right — the bias might not be coded into the rules. It could emerge from the training data. If the algorithm was trained on electronic health records from hospitals that predominantly used that one brand of antibiotic, then the model learns that pattern and amplifies it. The vendor can plausibly say 'we didn't design it to favor our sponsor's drug.' But the outcome is the same. Luna: So it's a structural bias, not a malicious one. But the financial relationship still created the conditions for it. That feels like exactly the kind of thing ethics regulations should catch. Lucas: And there is some movement. There's a bipartisan bill introduced in Congress last month called the AI Transparency in Healthcare Act. It would require any AI tool used in clinical decision-making to disclose financial relationships between the developer and any company whose products the algorithm might influence. It also calls for a public database of those disclosures. Luna: That's promising, though I imagine the AI industry will lobby hard against it. They'll say it creates unnecessary burden and slows innovation. Lucas: They already are. The Coalition for Health AI, which includes some of the biggest tech companies and hospital systems, put out a statement arguing that the bill's disclosure requirements would be 'overly broad' and could reveal proprietary information. But Dr. Chen's counterargument is that physicians already have to disclose conflicts of interest when they give talks or publish research — why should an algorithm be held to a lower standard than a human doctor? Luna: That's a really clean way to frame it. And I think it gets at a deeper question: we trust AI in healthcare because we assume it's objective, but if the system that builds it has financial incentives, that assumption falls apart. Lucas: If today's conversation gave you something useful — something you might think about the next time you hear about a hospital adopting a new AI tool — that's exactly what this show tries to do. And the way we keep it ad-free is through listener support. So if you found value in this, consider buying us a coffee at buy me a coffee dot com slash fexingo. It genuinely helps us keep digging into stories like this. Luna: Yeah, and it's a small way to say this kind of independent journalism matters. Every contribution adds up. Lucas: So back to the bill — let's talk about what it would actually change in practice. Luna: One of the things I'm curious about is whether it would cover algorithms that aren't technically 'medical devices' — like the kind that insurance companies use to approve or deny coverage. Lucas: That's a great point. The current bill focuses on clinical decision support software, which is used by providers. But there's a whole separate category of AI used by payers — prior authorization algorithms, claim denial systems — and those aren't covered. Some critics say the bill doesn't go far enough. Luna: So it's a first step, but not a comprehensive fix. Still, if it passes, it would at least force hospitals and doctors to know what they're working with. Lucas: Exactly. And I think that transparency alone could change behavior. If a hospital purchasing committee sees that an AI tool is sponsored by a drug company, they might ask harder questions. Or they might negotiate better pricing. Or they might choose a different tool. Luna: I also wonder about the role of professional societies. The American Medical Association has ethics guidelines for physicians on conflicts of interest, but do they have anything similar for AI? Lucas: They do, but they're relatively new. In 2024, the AMA published a set of principles for augmented intelligence, and one of them is about transparency — but it's voluntary. The bill would make it mandatory for certain tools. Luna: Voluntary disclosure in a for-profit system — we've seen how that works in other industries. Usually, not much happens until regulators step in. Lucas: Right. And that's the tension here. Healthcare AI is a rapidly growing market — projected to hit $67 billion by 2028, according to one estimate. There's a lot of money at stake. And the companies developing these tools are often startups backed by venture capital, so there's pressure to show returns. Luna: That pressure can lead to shortcuts. And when the shortcut involves a conflict of interest, it's patients who pay the price — either through less effective treatment or higher costs. Lucas: One more thing I want to mention: the study also looked at how often these AI tools were updated after deployment. They found that tools with pharmaceutical ties were significantly less likely to be updated when new evidence came out that challenged the effectiveness of the sponsor's drug. So the algorithm becomes a kind of anchor, locking in older practices. Luna: That's almost worse than the initial bias — it means the system resists correction. Even if a doctor starts to question the recommendations, the algorithm keeps reinforcing the same pattern. Lucas: Precisely. And that's why Dr. Chen argues that disclosure alone isn't enough — she wants to see independent auditing of clinical AI tools, similar to how financial auditors check public companies. But that's a much heavier lift politically and logistically. Luna: For now, the bill is a start. And I think for listeners, the takeaway is: next time you're in a doctor's office and they mention an AI tool that helped with your diagnosis, it's okay to ask 'Who built it? And who funded it?' Lucas: That's a great practical tip. And honestly, the more patients ask those questions, the more pressure there will be for transparency — even before the law catches up. Luna: Alright, I think we've covered a lot. Thanks, Lucas. Lucas: Thanks, Luna. And to our listeners — keep asking the hard questions. We'll be back next time with another angle on responsible AI.