Latest / DX Today | No-Hype Podcast & News About AI & DX / Microsoft AI Hit 83% on Cases Where Doctors Got 36%

Transcript

Microsoft AI Hit 83% on Cases Where Doctors Got 36% · DX Today | No-Hype Podcast & News About AI & DX

• 0:00
  SPEAKER_01: Welcome to the DX Today Podcast, your daily deep dive into the AI ecosystem.
• 0:05
  SPEAKER_01: I'm Chris, and joining me as always is Laura.
• 0:07
  SPEAKER_00: Hey Chris, and hello everyone.
• 0:09
  SPEAKER_00: Today we are talking about a medical AI study that landed in the journal Science on Thursday, and it is one of those results that sounds almost too good to be true at first.
• 0:18
  SPEAKER_01: So set the scene for me, because I saw the headline that an AI system from Microsoft beat doctors at diagnosis.
• 0:25
  SPEAKER_01: And my first reaction was that we have heard that kind of claim many times before.
• 0:30
  SPEAKER_00: Totally fair reaction.
• 0:31
  SPEAKER_00: The system is called the Microsoft AI Diagnostic Orchestrator.
• 0:35
  SPEAKER_00: And on a set of really hard medical cases, it got the right diagnosis 83% of the time, while the doctors it was compared against got 36%.
• 0:44
  SPEAKER_01: 83 versus 36 is a huge gap.
• 0:47
  SPEAKER_01: More than double.
• 0:48
  SPEAKER_01: So before we get excited, I want to know exactly which doctors and exactly which cases we are talking about here.
• 0:53
  SPEAKER_00: The cases are the famous case records from the New England Journal of Medicine.
• 0:58
  SPEAKER_00: More than 300 of them.
• 0:59
  SPEAKER_00: And these are the puzzles that get published precisely because they stumped people and took a long road to the answer.
• 1:05
  SPEAKER_01: So these are not your everyday sore throat and sprained ankle visits.
• 1:09
  SPEAKER_01: These are the zebras, the rare and confusing presentations that even experienced physicians might see only once or twice in a whole career.
• 1:17
  SPEAKER_00: Exactly.
• 1:18
  SPEAKER_00: And the doctors were primary care physicians and general internists working on their own.
• 1:22
  SPEAKER_00: Though this time they were allowed to use textbooks and search the internet, which matters a lot for reasons we will get to in a minute.
• 1:29
  SPEAKER_01: Okay, and what does it actually mean that the system is an orchestrator?
• 1:32
  SPEAKER_01: Because that word suggests something more than just typing symptoms into a chatbot and reading whatever comes back.
• 1:38
  SPEAKER_00: Right, it is a group of AI agents that act like a virtual panel of doctors.
• 1:42
  SPEAKER_00: One proposes hypotheses, another picks tests, another plays the skeptic, and they debate each other before the panel commits to a next step or a final diagnosis.
• 1:51
  SPEAKER_01: That is a cool design, and I like that it mirrors how a hospital case conference really works, with people arguing over a whiteboard rather than one genius staring at a chart and announcing the answer.
• 2:02
  SPEAKER_00: And the key idea is sequential diagnosis.
• 2:05
  SPEAKER_00: The system does not get the whole file at once.
• 2:07
  SPEAKER_00: It starts with a short summary, then has to ask history questions, choose physical exam findings, order tests one at a time, and pay for each one.
• 2:16
  SPEAKER_01: Wait, it pays for the tests?
• 2:18
  SPEAKER_01: Walk me through that, because I think a lot of listeners would assume a computer gets every lab result for free and can simply order everything it wants.
• 2:26
  SPEAKER_00: That is the clever part of the benchmark.
• 2:28
  SPEAKER_00: Every test carries a realistic price.
• 2:31
  SPEAKER_00: So a model that orders a full body scan and every blood panel on day one gets punished, and the researchers track both accuracy and the total cost of the workup.
• 2:40
  SPEAKER_01: So it is measuring judgment and not just knowledge, which honestly is what separates a great clinician from someone who has memorized a textbook but still sends every patient for every test available.
• 2:50
  SPEAKER_00: Harsha Nori, who leads AI at Microsoft AI Health, said the team cares about how judiciously and carefully the system does the workup.
• 2:59
  SPEAKER_00: And Dominic King, a vice president there, said multiple choice exams no longer tell us much at all.
• 3:04
  SPEAKER_01: I remember that point because the models have basically maxed out the medical licensing style questions.
• 3:09
  SPEAKER_01: So another headline about a chat bot passing a board exam really does not move anyone in medicine anymore.
• 3:14
  SPEAKER_00: King put it bluntly, saying that AI has become very good at getting a 100% pass rate on those.
• 3:20
  SPEAKER_00: So the whole point here was to test the messy, step-by-step work that real diagnosis actually requires.
• 3:26
  SPEAKER_01: Now let me push back a little because I will recall an earlier version of this work last summer in 2025, and the critics had a field day with the way the doctors were handicapped in that first comparison.
• 3:36
  SPEAKER_00: You remember correctly.
• 3:38
  SPEAKER_00: The preprint in July of 2025 claimed about 85% for the AI versus roughly 20% for 21 physicians, but those doctors could not consult colleagues, books, or the internet, which is not how anyone practices medicine.
• 3:52
  SPEAKER_01: Right.
• 3:53
  SPEAKER_01: So in that version, the AI had the world's medical literature baked into it, while the humans were working from memory alone.
• 4:00
  SPEAKER_01: And that felt like comparing a student with an open book to a student with nothing.
• 4:04
  SPEAKER_00: This peer-reviewed version answers part of that criticism because the physicians now had reference tools, and their score went up to 36%.
• 4:13
  SPEAKER_00: But the system still came out more than twice as accurate on the hardest cases.
• 4:17
  SPEAKER_01: That is a real improvement in the study design, but I notice it is still generalists only.
• 4:23
  SPEAKER_01: What about specialists or a team of doctors working together the way a teaching hospital would handle a truly baffling case?
• 4:30
  SPEAKER_00: Not compared, and that is a genuine limitation.
• 4:33
  SPEAKER_00: A rheumatologist or an infectious disease specialist might do far better on the cases in their own lane.
• 4:39
  SPEAKER_00: And a team conference of humans might close a lot of that gap as well.
• 4:42
  SPEAKER_01: So the honest headline is that the system beats a solo generalist with internet access on very rare puzzles, in a simulation which is impressive, but quite a bit narrower than the phrase beats doctors suggests.
• 4:54
  SPEAKER_00: Yes, and there is a second nuance that I find even more interesting.
• 4:58
  SPEAKER_00: The researchers ran the orchestrator on top of models from several companies, including OpenAI, Google, Anthropic Meta, and DeepSeek, so they could separate the scaffold from the brain underneath.
• 5:10
  SPEAKER_01: Oh, that is the part I wanted to dig into.
• 5:11
  SPEAKER_01: Because if you swap the underlying model and the results change dramatically, then the question becomes how much credit belongs to Microsoft and how much belongs to whoever built the model?
• 5:21
  SPEAKER_00: Here are the numbers.
• 5:22
  SPEAKER_00: The newest flagship model from OpenAI scored 72% on the complex cases all by itself.
• 5:29
  SPEAKER_00: And with the orchestrator wrapped around it, that climbed to nearly 87% at slightly lower cost, too.
• 5:35
  SPEAKER_01: So the model alone already gets 72, which is double the doctor's on its own.
• 5:40
  SPEAKER_01: That makes me think most of the magic is simply that the frontier models have become very good at medicine.
• 5:46
  SPEAKER_01: And the orchestrator adds the final 15 points.
• 5:49
  SPEAKER_00: That is exactly what TechCrunch pointed out, that much of the gain may come from stronger underlying models.
• 5:55
  SPEAKER_00: And for the previous generation flagship from the same company, the orchestrator added only about four points of accuracy, though it saved roughly$3,000 per case.
• 6:04
  SPEAKER_01: $3,000 a case is not nothing, though.
• 6:07
  SPEAKER_01: If you think about a hospital doing thousands of diagnostic workups a year, trimming unnecessary imaging in labs at that scale could be the most important result in the entire paper.
• 6:17
  SPEAKER_00: I agree, and the cost angle is underrated.
• 6:19
  SPEAKER_00: Then there is the most striking result of all.
• 6:22
  SPEAKER_00: A small, cheap Google model went from about 46% alone to nearly 69% with the orchestrator, which is a gain of more than 20 points.
• 6:30
  SPEAKER_01: So the orchestrator helps the weaker models the most, which actually makes intuitive sense because structure and checklists help a less capable thinker more than they help someone who already has great instincts.
• 6:42
  SPEAKER_00: Right.
• 6:42
  SPEAKER_00: And the researchers said the gains naturally shrink as models get close to the theoretical ceiling.
• 6:48
  SPEAKER_00: That has a practical consequence because a clinic could run a cheaper model inside this kind of scaffold and get surprisingly strong performance for a fraction of the price.
• 6:57
  SPEAKER_01: That is a really interesting second-order effect, especially for rural hospitals or health systems in lower-income countries that cannot afford premium model pricing at scale, but still desperately need help with difficult cases.
• 7:08
  SPEAKER_00: But here's the result that should make everyone cautious.
• 7:11
  SPEAKER_00: On emergency department cases drawn from a large database of real admissions, the two strongest models showed no meaningful gain from the orchestrator at all, even though Microsoft says the systems still outperform physicians there.
• 7:24
  SPEAKER_01: Interesting.
• 7:25
  SPEAKER_01: So on the rare puzzles, the orchestration helps, but on the bread and butter emergency room cases, it adds very little, which suggests the benefit depends heavily on what kind of medicine you are actually practicing.
• 7:36
  SPEAKER_00: Exactly.
• 7:37
  SPEAKER_00: Emergency medicine is about speed, triage, and ruling out the dangerous things first.
• 7:42
  SPEAKER_00: While the journal case records reward a long and careful hunt for an unusual answer, those are really different skills, and a panel of debating agents fits one better.
• 7:50
  SPEAKER_01: Now I have to raise the obvious concern, which is that this was conducted and reported by Microsoft's own researchers, and Microsoft sells clinical software.
• 7:59
  SPEAKER_01: Peer review in science is a strong signal, but it is not the same as independent replication.
• 8:05
  SPEAKER_00: Completely fair, and it is still a simulation, not live patients.
• 8:09
  SPEAKER_00: Real patients are not tidy case write-ups.
• 8:12
  SPEAKER_00: They forget things, they describe pain in vague ways, their records are scattered across three hospital systems, and sometimes the key detail never gets written down.
• 8:21
  SPEAKER_01: In the published cases, come with a known answer, which is great for scoring, but in the real world, a lot of diagnosis happens under uncertainty, where nobody ever finds out for sure whether the label was right.
• 8:33
  SPEAKER_00: Which is why the next step matters so much.
• 8:35
  SPEAKER_00: Microsoft is partnering with Beth Israel Deaconess Medical Center in Boston to test the system in emergency and urgent care, comparing its recommendations with the choices doctors make for consenting patients in real time.
• 8:48
  SPEAKER_01: And I assume the AI is not actually treating anyone in that study.
• 8:51
  SPEAKER_01: It is more of a shadow mode, where the system makes its call quietly in the background, and the researchers compare it later against what the human team decided.
• 9:00
  SPEAKER_00: That is the design as described, and they also want to look at triage in emergency department waiting rooms.
• 9:05
  SPEAKER_00: Dominic King said he expects it to be the largest study of its kind, which is a bold claim we should hold them to.
• 9:12
  SPEAKER_01: And Google is running a similar playbook, right?
• 9:14
  SPEAKER_01: I believe their diagnostic system also went through Beth Israel.
• 9:18
  SPEAKER_01: So it feels like the big labs are now racing toward the same hospitals for clinical validation.
• 9:23
  SPEAKER_00: Yes, Google recently published a Beth Israel study of its diagnostic AI called Amy and the Lancet.
• 9:30
  SPEAKER_00: So the competition is moved from benchmark leaderboards into real clinical trials, and honestly, that is exactly where this competition should be happening.
• 9:38
  SPEAKER_01: Let us talk about the big vision, because Mustafa Suleiman, who runs Microsoft AI and co-founded DeepMind, keeps using the phrase medical superintelligence, and that is the kind of language that makes many doctors roll their eyes immediately.
• 9:50
  SPEAKER_00: He called this result a step toward medical superintelligence, and he predicted a coding style breakthrough for medical records in 2027.
• 9:58
  SPEAKER_00: He also said healthcare models could become 10 times more capable within 24 to 36 months.
• 10:04
  SPEAKER_01: A coding style breakthrough is a vivid phrase.
• 10:07
  SPEAKER_01: I take it to mean that what happened to software engineering, where agents suddenly started doing real work end-to-end, could happen to clinical documentation and chart review next.
• 10:17
  SPEAKER_00: That is my reading too, and Microsoft is laying the groundwork.
• 10:21
  SPEAKER_00: It already sells dictation and radiology tools to clinicians, and it is working with the Mayo Clinic on a healthcare foundation model that it plans to release sometime next year.
• 10:30
  SPEAKER_01: So if I am a practicing primary care doctor listening right now, what should I actually take away from this?
• 10:36
  SPEAKER_01: Beyond the alarming idea that a computer scored 83 while people like me scored 36?
• 10:41
  SPEAKER_00: I would say the takeaway is that these tools are becoming excellent second opinions for the weird cases.
• 10:47
  SPEAKER_00: The patient who has seen five doctors without an answer.
• 10:50
  SPEAKER_00: That is precisely where a tireless panel of agents, checking a huge literature, could save lives.
• 10:56
  SPEAKER_01: I like that framing because it positions the AA as a consultant you call in when you are stuck, not as a replacement for the person who knows the patient, notices the subtle details, and has to deliver difficult news with care.
• 11:09
  SPEAKER_00: And there are real questions left for regulators and hospitals.
• 11:13
  SPEAKER_00: Who is liable when the panel is wrong?
• 11:15
  SPEAKER_00: How do you audit its reasoning?
• 11:16
  SPEAKER_00: And how do you stop busy clinicians from simply rubber stamping whatever the machine recommends at the end of a long shift?
• 11:22
  SPEAKER_01: Automation bias is the phrase people use for that, and it is a real risk.
• 11:26
  SPEAKER_01: The better the system gets, the more tempting it becomes to stop thinking.
• 11:31
  SPEAKER_01: Which is ironic because the whole design rewards careful, deliberate reasoning.
• 11:36
  SPEAKER_00: So my verdict is cautiously excited.
• 11:39
  SPEAKER_00: The science is meaningfully better than last year, the cost results are underrated, and the gains on cheaper models are a big deal, but the live hospital trial is where this either proves itself or quietly fades.
• 11:50
  SPEAKER_01: That is a fair place to land.
• 11:52
  SPEAKER_01: Impressive simulation, honest caveats, and a real world test coming soon.
• 11:56
  SPEAKER_01: So we will be watching that Boston trial closely and reporting back as soon as the first results come out.
• 12:00
  SPEAKER_01: That's all for today's episode of the DX Today podcast.
• 12:03
  SPEAKER_01: Thanks for listening, and we'll see you next time.