Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Recommends a Doctor You Can't Afford
Transcript
- Lucas: So picture this — you go to your primary care doctor with a nagging symptom, they run some basic tests, and then an algorithm decides which specialist you get referred to. Luna: Which sounds efficient on the surface. But the question is — efficient for who? Lucas: Exactly. And that question is at the heart of a really troubling study that came out of UCSF last year. Researchers looked at referral patterns across three large hospital networks and found that the AI systems recommending specialists were significantly more likely to send privately insured patients to expensive, high-reputation specialists — while patients on Medicaid got routed to whoever was available, regardless of specialty fit. Luna: I remember that study. The gap was around 40 percent, right? Even when the clinical presentation was identical? Lucas: Yeah, 40 percent. That's massive. And the researchers controlled for every variable they could think of — age, severity, comorbidities — and the only thing that predicted the referral tier was the insurance type. Luna: So the algorithm learned that from historical data. If the training data reflected years of doctors referring their better-insured patients to fancier specialists, the model just amplified that pattern. Lucas: Right. And here's the thing — the hospitals didn't build these algorithms themselves. They bought them from third-party vendors. Epic, Cerner, a few others. So the hospitals themselves may not even know exactly what features the model is using. Luna: Which makes auditing nearly impossible. You can't just ask the vendor, 'Hey, did you include insurance type as a feature?' Because the model is a black box even to them sometimes. Lucas: There was one case that really stuck with me. A woman in Phoenix — her name was redacted in the study, but her story is in the appendix. She had persistent headaches, went to a clinic that used an AI triage tool. The algorithm recommended an MRI with contrast, costing about $2,000. She had a high-deductible plan, so she'd have to pay out of pocket. She couldn't afford it, so she waited. Luna: And what happened? Lucas: Turns out it was a vitamin B12 deficiency. Cost about twenty dollars in supplements to fix. The MRI was completely unnecessary for that diagnosis, but the algorithm — trained on data where patients with her demographic profile were often referred for imaging — pushed her down that path. Luna: So the algorithm wasn't just biased — it was also clinically wasteful. It generated unnecessary costs for the patient and the system. Lucas: Exactly. And that's the part that gets lost in a lot of these conversations. Algorithmic bias isn't just a fairness problem — it's also a quality problem. When the referral is wrong, the patient gets worse care. Luna: And it's not just about insurance. There are examples of racial bias in these systems too. A 2023 study in Health Affairs found that Black patients were less likely to be referred to cardiologists by an AI system, even when their risk factors were identical. Lucas: Yeah, that study was a wake-up call for a lot of people. And the thing is, these systems are being deployed at scale. The Centers for Medicare and Medicaid Services estimated that by the end of 2025, over 70 percent of large hospital systems had some form of ai driven clinical decision support. Luna: So what's being done about it? I know CMS proposed a rule last year requiring hospitals to audit their referral algorithms for bias. Lucas: They did. The proposed rule — it's still in comment period, so not final yet — would require hospitals to run what they're calling an 'equity impact assessment' before deploying any algorithm that influences clinical decisions. And they'd have to repeat that assessment annually. Luna: Which sounds good, but who does the auditing? The hospitals themselves? That feels like asking the fox to count the chickens. Lucas: Right. And there's a practical problem — most hospitals don't have the data science staff to do a rigorous audit. A 2024 survey by the American Hospital Association found that fewer than 15 percent of hospitals had a dedicated data ethics team. Luna: So we're in this awkward middle where the regulation exists on paper, but the infrastructure to implement it doesn't. Lucas: Exactly. And meanwhile, the vendors keep selling these systems. There's a startup called ReferWell that raised $80 million last year on the promise that its AI could 'optimize specialist matching.' The pitch deck they showed investors talked about reducing wait times and improving revenue capture — not once did it mention equity. Luna: So the financial incentive is to optimize for efficiency and profit, not for fairness. Lucas: And that's the structural problem. As long as hospitals are paid more for expensive procedures, the algorithm that maximizes those referrals will look like a good algorithm to the hospital's CFO. Luna: Which brings us back to the core ethical question: should an algorithm be making these decisions at all? Or should it just be a recommendation that a human doctor can override? Lucas: That's the thing — most of these systems are designed as decision support, not autonomous. But in practice, studies show that doctors override the algorithm's recommendation less than 10 percent of the time. Partly because they're busy, partly because they trust the system. Luna: So the human in the loop is a theoretical safeguard that doesn't really function in practice. Lucas: Exactly. And that's why the onus has to be on the algorithm itself. You can't just say 'well, the doctor can always say no' if the doctor never actually does. Luna: You know, this episode actually gave me something really concrete to think about. If today's conversation was useful to you too, one way to keep these ad-free episodes coming is listener support. It's buy me a coffee dot com slash fexingo. Lucas: Yeah, that's right. No ads, no sponsors — just a direct way for people who find value in this to help keep it going. And we really appreciate that. Luna: So back to the audit problem. One thing that gives me a little hope is that some of the bigger vendors are starting to build fairness toolkits. Epic released what they call the 'FairRefer' module last month — it's a set of dashboards that let hospitals see referral patterns by insurance and race. Lucas: I saw that. It's a step, but the transparency only helps if hospitals actually use it to change behavior. And right now, there's no penalty for ignoring the data. Luna: So we're back to needing regulation with teeth. Lucas: Yeah. And I think a really interesting test case is going to be how the FDA handles this. They've been talking about regulating clinical decision support software as a medical device, but they've been very slow to act. Luna: Because if they classify it as a device, then any update to the algorithm would require a new approval. That would slow down innovation significantly. Lucas: Right. But it might also force vendors to be more careful about what they put into the model in the first place. You can't just ship a biased algorithm and say 'we'll fix it in the next update.' Luna: So the question is — do we want faster innovation with more risk of harm, or slower innovation with more safety? Lucas: That's the classic trade-off. But in healthcare, the harm isn't just a bad user experience — it's people not getting the care they need. Luna: And that's why this conversation matters. It's easy to think of algorithmic bias as an abstract problem, but when it's your mom or your friend who gets the wrong referral, it's very concrete. Lucas: Absolutely. So I think the takeaway here is not that AI in healthcare is bad — it's that we need to build these systems differently. The data we feed them, the metrics we optimize for, the oversight we require — all of that has to be rethought. Luna: And maybe the first step is just admitting that efficiency is not the only goal. Lucas: Yeah. It's a good place to leave it. Next time, let's talk about AI in criminal sentencing — there's a new algorithm being tested in three states that claims to be race-neutral. Luna: Looking forward to it.