#54 - The Cancer Exam That AI Failed
What happens when the models everyone keeps calling "doctor-level" have to actually take the test? We put that claim under pressure with a study that lands uncomfortably close to real life: six popular large language models were handed 137 multiple-choice questions on colorectal cancer and forced into a strict zero-shot format — no examples, no reasoning shown, just the letter. Performance came in around chance, or below. In other words, the models often sound certain while effectively guessing.
We break down why colorectal cancer is such a revealing stress test for medical AI. Guidelines…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.