Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Diagnosis Misses Rare Disease Patients
Transcript
- Lucas: So there's this stat that keeps me up at night: rare diseases collectively affect about 300 million people worldwide. That's roughly the population of the United States. And yet, when we train AI diagnostic tools, we almost exclusively feed them data on common conditions. Luna: Right—because that's where the bulk of electronic health records and imaging data live. But that means the AI is practically blind to the people who most need pattern recognition. Lucas: Exactly. A 2025 study in the Journal of the American Medical Informatics Association—JAMIA—put a number on it. They evaluated five commercial diagnostic AI models across a range of conditions. For common diseases like diabetes or hypertension, accuracy hovered around 90 percent. But for rare diseases—defined as affecting fewer than 200,000 people in the U.S.—that accuracy dropped to about 60 percent. Luna: So a 30-point gap. And false negatives must be brutal in that group. Lucas: They are. The study found false-negative rates as high as 60 percent for conditions with fewer than a thousand documented cases. That means the AI is telling six out of ten patients that they don't have a disease they actually do have. And those patients are often told 'your tests are normal' or 'it's probably anxiety.' Luna: Which is devastating—because for many rare diseases, early intervention is critical. I'm thinking of conditions like pulmonary arterial hypertension or certain pediatric metabolic disorders. A six-month delay can change the prognosis entirely. Lucas: The structural problem is simple: rare diseases are, by definition, rare. So training datasets have vanishingly few examples. Most models are built on data from large academic medical centers, which have plenty of diabetes and heart failure cases but maybe a dozen cases of, say, Erdheim-Chester disease or stiff person syndrome. Luna: And that's assuming the rare disease is even captured in the structured data. Many rare conditions don't have a clean ICD code, so they get lumped into vague categories like 'other specified disorders.' The AI never learns the pattern. Lucas: It's a feedback loop. The AI is trained on common cases, so it's good at common cases. It's terrible at rare cases, so clinicians get false reassurance from a negative AI readout, and they don't dig further. The rare disease stays undiagnosed, no data point is generated, and the model never improves for that population. Luna: And there's a perverse incentive issue too. Companies developing diagnostic AI—like those behind the models in the JAMIA study—have little financial motivation to optimize for rare diseases. The addressable market is tiny per condition, and the data collection cost is high. Lucas: That's a really important point. If you're a startup trying to get FDA clearance, you're going to demonstrate efficacy on common conditions because that gets you through the process faster. Rare disease validation is expensive and doesn't move the needle on your valuation. Luna: But the FDA started paying attention. In March 2026, they released a draft guidance specifically on AI in medical devices, and it included a section on 'performance across subpopulations.' They're asking developers to stratify results by disease prevalence. It's not a requirement yet, but it signals the direction. Lucas: That's promising. And there are technical approaches emerging. One is federated learning—where multiple hospitals train a shared model without sharing raw patient data. That lets rare-disease registries pool their cases without privacy headaches. A pilot from the NIH's Rare Diseases Network showed a 15 percent accuracy improvement on five rare cancers using federated learning across 20 sites. Luna: Another approach is synthetic data—generating artificial but clinically plausible rare-disease cases to augment training sets. There's a team at Stanford that used generative adversarial networks to create synthetic retinal scans for rare eye conditions. They boosted detection rates from 55 percent to 78 percent. Lucas: So there are paths forward. But the core question is whether the ecosystem will prioritize this. The JAMIA study's lead author, Dr. Priya Menon, said something that stuck with me: 'If we only build AI for the common, we are building a system that serves the many and abandons the few.' And the 'few' in this case is three hundred million people. Luna: That's a powerful framing. And it ties into a bigger conversation about what we consider 'good enough' performance. If a model is 90 percent accurate overall but fails systematically on a subset, it's not a good model for everyone. Lucas: Yeah. And look—if today's conversation gave you something usable, a new angle on why AI bias isn't just about race or gender but also about disease prevalence, then honestly, if it was worth a coffee to you, that's the link—buy me a coffee dot com slash fexingo. Listener support is what keeps this show ad-free and focused on these deeper cuts. Luna: It's a small way to say this kind of conversation matters. And it really does help us keep digging into stories like this one. Lucas: So—back to the rare disease challenge. One thing that struck me in Dr. Menon's paper was that even when a rare disease is flagged by the AI, clinicians often override it. They trust their own judgment more than an AI that's been wrong before. So there's a human factors layer too. Luna: Right. The AI gives a probability that the patient has Erdheim-Chester disease—say 70 percent. The doctor has never seen it, so they dismiss it. That's not just an AI problem; it's an education problem. Lucas: And the study showed that when the AI flagged a rare disease, clinicians accepted the diagnosis only about 40 percent of the time. Compare that to common diseases, where acceptance was over 85 percent. So we have both an AI model that's less confident in rare diseases and a human system that's skeptical. Luna: Which means the solution isn't just better algorithms. It's also training for doctors—and maybe changing how AI presents its findings. Instead of a binary 'positive/negative,' what if it said 'this pattern is unusual; here are three rare conditions that match; consider consulting a specialist'? Lucas: Exactly. And some companies are doing that. There's a startup called DxLogic that builds AI specifically for undiagnosed disease—they call it the 'diagnostic odyssey.' Their model doesn't just output a disease name; it outputs a differential and suggests the next test. Early data shows they catch rare conditions about three times faster than standard care. Luna: That's the kind of approach that treats the AI as a collaborator, not an oracle. And it respects the reality that rare disease diagnosis is inherently uncertain. Lucas: There's also a regulatory angle. The European Union's new AI Act, which is starting to phase in, classifies medical AI as 'high risk' and requires continuous monitoring of performance across demographic and clinical subgroups. That means if your model drops in accuracy for rare conditions post-deployment, you have to report it and potentially retrain. Luna: Which is a stronger requirement than the FDA's current draft guidance. So we might see a divergence where the EU pushes for rare-disease performance and the U.S. follows. That could create market pressure on developers to improve. Lucas: I think that's likely. And it's not just about regulation—it's about patient advocacy. Rare disease communities are incredibly organized. Groups like the National Organization for Rare Disorders—NORD—have started publishing 'AI report cards' that grade diagnostic tools on how well they handle rare conditions. That's consumer pressure. Luna: That's smart. If you're a hospital choosing which AI system to buy, and one has a rare-disease accuracy of 80 percent versus another at 45 percent, you have a concrete differentiator. Lucas: Yeah. So the agenda is clear: more representative training data, better human ai collaboration design, regulatory mandates, and consumer-driven accountability. The question is whether the market moves fast enough for the 300 million people who are currently invisible to these systems. Luna: And whether we'll look back in five years and say we fixed it—or that we built a two-tier diagnostic system: excellent for the common, guesswork for the rare. Lucas: That's the risk. And it's a choice we're making right now, every time a dataset is assembled and a model is trained. So I hope the pressure keeps building.