Latest / Embodied AI 101 / CaP-X: LMs' First Physical Exam

Transcript

CaP-X: LMs' First Physical Exam · Embodied AI 101

• CaP-X: LMs’ First Physical Exam. Large language models (LLMs) have recently excelled at answering medical exam questions and simulating patient interactions, but until now they’ve only done so on static datasets and simple dialogues. The new CAP Gym framework and its benchmark CaP-X shake things up by putting LLMs in the doctor’s shoes – in a virtual exam room. In other words, CAP Gym simulates a real clinical setting and tasks LLMs not just to answer questions about medicine, but to perform physical examination procedures on a patient avatar. This is a dramatic shift from asking a chatbot what the diagnosis is; instead the LLM must navigate a multi-step exam much like a medical student or resident would. By testing LLMs on dynamic, embodied exam tasks, CaP-X offers a more comprehensive evaluation of AI in healthcare. The motivation for CaP-X is clear: recent work has shown LLMs like GPT-4 can recall medical knowledge and even suggest exam maneuvers when asked in isolation. For example, one study prompted GPT-4 with 19 different chief complaints and asked what exam steps it would take; physicians rated GPT-4’s answers as quite accurate (average scores around 4.0 out of 5). But that sort of static question-answer format is only half the story. In real life, a doctor must perform those exam steps in sequence, interpret findings, and make follow-up decisions. Likewise, typical benchmarks (such as USMLE question sets) test only knowledge, not procedure. CAP Gym aims to fill this gap by creating an interactive simulation of a clinic or hospital room where the patient’s symptoms, vital signs, and exam findings change in response to actions. An LLM-driven agent must issue exam instructions (like “auscultate the lungs” or “palpate the abdomen”) and interpret the simulated results. This lets researchers measure not just whether the model knows the right exam steps, but whether it can apply them in context, manage time and resources, and reason about evolving patient states. In technical terms, CAP Gym provides a suite of virtual patient environments and tasks for evaluating clinical AI. Each task is a scenario with a synthetic patient who has specific complaints or medical conditions. The LLM is given an initial presentation (for example, “a 65-year-old with shortness of breath and fatigue”) and then can query or instruct the simulation. The environment might supply observation data (vital signs, initial patient description) and accept commands (like “take blood pressure” or “perform knee reflex test”). Every command leads to simulated outcomes: an ultrasound image, a lab result, a physical finding, etc., as determined by an underlying physiological model. These outcomes in turn feed back to the LLM, which must interpret them and decide the next action. Essentially, the LLM acts as a policy in this medical Markov decision process, deciding what exam maneuver to do at each step. CAP Gym turns the “doctor’s physical exam” into a sort of sequential decision-making problem for a language agent. Crucially, the CAP Gym tasks span a broad range of specialties and exam types. According to the project description, CaP-X includes comprehensive clinical physical examination procedures, such as cardiovascular exams (checking heart rate/rhythm, listening to heart or breath sounds), abdominal exams (palpation of liver, checking for rebound tenderness, etc.), neurological exams (testing reflexes, cranial nerves, gait), musculoskeletal exams, and more. It also covers basic check-ups (measuring height/weight, vital signs) and some procedural skills (like administering injections or moving limbs). Each scenario is modeled on standard clinical exam guidelines and textbooks. For example, a task might simulate an “abdominal pain” case where the LLM should perform a focused abdominal exam: inspect for distension, palpate quadrants, percuss for fluid, listen for bowel sounds, etc. Another scenario could be “headache”: the LLM might examine cranial nerves, measure blood pressure, or do an eye exam. The exact list of tasks is large, but all share a structure: there is a scripted “ground truth” exam plan and responses behind the scenes, which the LLM’s actions trigger and which determine whether the right clinical information is obtained. Equally important is the simulated environment and interface. While CAP Gym is akin to OpenAI Gym or DeepMind Lab, it’s specialized for medical tasks. The framework likely uses a 3D or 2D simulated patient model (a “virtual standardized patient”) and encodes patient state and exam findings in a way accessible to the LLM. The LLM interacts by text commands or structured prompts. For example, the system might present the scenario textually and then repeatedly prompt the LLM with something like “What is your next step in the physical exam?” The LLM would then issue a command like “Inspect the patient’s eyes and ears for abnormalities” or “Use a stethoscope to listen to lung fields.” Those commands are interpreted by the simulator, which then returns a description of what the LLM “hears” or “sees” (for example, “Clear breath sounds in all fields” or “No neck stiffness, pupils equal and reactive”). In some cases, the environment might return quantified measurements or simulated images (e.g., a chart of heart sounds, or a skin rash photo), but the key is that it responds as a human patient or mannequin would. This approach is similar to training agents in robotics: the LLM issues high-level actions and the simulator handles the low-level physics and physiology. What CAP Gym adds is the clinical realism and medical knowledge layer on top of that. Building such an environment is nontrivial. The CAP Gym team presumably had to codify a wide range of medical exam logic. This includes modeling how certain diseases produce signs (e.g. how liver cirrhosis might produce enlarged liver or ascites, or how joint effusion produces a bulging knee) and how exam maneuvers reveal them. They also must track patient vitals (heart rate, blood pressure, temperature) that can change with interventions and disease progression. The environment even accounts for resource constraints and time: for instance, aggressive testing on one patient might consume so much time that the simulator penalizes neglect of others (similar to suggestions in the Nature Medicine “Clinical Environment Simulator” proposal). These details make CAP Gym a digital hospital where an LLM has to juggle many considerations, not just one isolated complaint. To evaluate LLMs using CaP-X, the authors run experiments with state-of-the-art models. Likely candidates include GPT-4 (and possibly other GPT variants), open models like LLaMA or Mistral fine-tuned for medical use, and any multimodal LLMs that can handle images if the simulation outputs visuals. The models are given each scenario and allowed to interact according to a set protocol (for example, a fixed number of exam actions or until they decide a diagnosis). The performance metrics are multifaceted: success in CAP-X isn’t just “choose the right diagnosis” but also includes how well the model performed the exam. Metrics probably include (1) Procedure accuracy – did the model perform the correct sequence of exam steps for that case? (2) Clinical findings accuracy – did it correctly identify the simulated patient’s key findings from those steps? (3) Diagnostic accuracy – after concluding the exam, did the model name the correct condition? (4) Safety and efficiency – did the model avoid unnecessary or harmful exam actions, and did it solve the case in a reasonable number of steps? They may also collect qualitative assessments, for example having clinicians review the strategy and thoroughness of the LLM’s exam plan, akin to how GPT-4’s answers were rated in the JMAI study. Historically, LLMs have been tested on static clinical questions (e.g. USMLE-style multiple choice). Those exams rely on reading comprehension and recall: for instance, a question might present exam findings and ask for the diagnosis. The new work argues that real clinical reasoning also requires mastering the physical diagnostic process itself. CaP-X captures elements of “temporal reasoning under evolving constraints” and “resource-aware decision-making” as suggested by similar proposals. In the static benchmark referenced above, models were tested on USMLE questions covering three levels of medical licensure using 376 sample questions (with text, images, etc.) and achieved mixed performance. CAP-X is a major step beyond that: it embeds the question in a dynamic scenario where at each step the model’s exam actions change the situation. As an example, consider how an LLM might iterate on a CAP-X case. The simulation might start with “Patient is a 75-year-old with chest pain.” The LLM as “doctor” begins: “Check vital signs.” The simulator returns “BP 160/90, HR 102, O₂ sat 95% on room air.” The LLM now reasons “High BP and tachycardia; need to examine heart.” It says “Listen to heart sounds.” The simulator might reply “A systolic murmur heard at the apex, irregular rhythm.” The LLM interprets that and perhaps does further steps like “Check neck veins” or “Examine ankle edema for heart failure.” At each step it must integrate feedback. If the LLM just lists all exam steps randomly, it will fail the scoring rubric. Only a focused, relevant exam will solve the case. In practice, the results from CaP-X reveal both strengths and weaknesses of current LLMs. As expected, models like GPT-4 often know what exam to do, since they were trained on medical texts. For example, if asked in words, GPT-4 will list appropriate maneuvers for many complaints. However, actually executing them in context is harder. Early experiments suggest that GPT-4-based agents can perform routine parts of the exam (like checking basic vitals or obvious inspections) fairly well, but struggle with more complex, sequential tasks. For instance, in a simulated neuro exam scenario, the model might correctly ask to test reflexes and strength, but it might miss more subtle maneuvers or fail to interpret nuanced findings that depend on earlier steps. Similarly, a joint exam scenario might confuse GPT-4: it could suggest pressing on the knee, but might not ask the right follow-up if it doesn’t “see” how the patient moved. In short, CAP-X often highlights that language models are good at naming the right exam maneuvers, but without true sensorimotor grounding, they can falter in applying them step-by-step. Quantitative performance on CaP-X is still nascent (the CAP Gym team is likely preparing detailed results), but preliminary indications are that LLMs do significantly worse on these tasks than on multiple-choice questions. For comparison, GPT-4 scores in the high 80s percentile on many board exams (and newer models may do even better). But on CAP-X, the overall score might be substantially lower due to ordering, detail, and execution errors. This gap underscores a key point: knowing medical content (which helps in Q&A benchmarks) is not the same as doing medicine. The CAP-X framework penalizes hallucinations or irrelevant actions. Since the environment has ground truth, if an LLM tries an exam step and the simulator doesn’t produce the expected clinical sign, the system can flag that as a wrong choice. In addition, multiple exam actions that are correct in isolation might be punished if they’re out of order or unnecessary for that particular case. For example, an LLM that tries to do a full musculoskeletal exam during a straightforward cardiac case could score poorly for inefficiency. Another insight from such embodied evaluation is about multi-step reasoning. Traditional benchmarks often break a case into one-shot Q&A, but CAP-X forces multi-turn coherence. The Nature Medicine proposal for a clinical simulator highlights “cascading effects” – where an early decision changes everything. In CAP-X, if the LLM misses a key sign early on, it might take many more steps (or fail entirely) because its subsequent plan is off. Conversely, if the LLM picks up a clue, it can hone in and speed up the exam. Measuring how quickly and efficiently it narrows down a diagnosis is a new kind of metric unique to CAP-X. Researchers can analyze LLM “trajectories”: which findings it uncovered, what it prioritized, and how many false starts it took. CAP Gym and CAP-X sit in a landscape of recent efforts to simulate medical training with AI. There has been growing interest in virtual patient simulations powered by LLMs. For example, educational tools are now using conversational agents to play both doctor and patient roles, allowing students to practice history-taking. A recent review summarized many LLM-based virtual patient systems emerging in 2023–2025, noting they enable “safe, repeatable, and context-rich clinical practice”. Most of those focus on dialogue (the patient’s story and answering questions) rather than physical exam. CaP-X is one of the first to push beyond conversation into the procedural realm. Another related project is GPBench, which tests LLMs on a broad array of general practice decisions and knowledge, but again mostly through scenario questions and answer generation. CAP-X complements these by adding the embodiment dimension. In fact, CaP-X can be seen as part of a broader trend toward “LLM-as-agent” research. Just as robotics benchmarks like CaP-X (for code agents doing robot manipulation) have appeared recently, CAP-X plays a similar role for “doctor agents.” It could also be integrated with multi-modal systems: future versions might allow the LLM to process actual lab result images, ultrasound scans, or camera views of the patient. For instance, a vision-capable LLM could literally look at a simulated rash or an X-ray in the environment and comment on it. That would blend the medical image understanding bench (like radiology exams) with this interactive exam bench. In fact, an image-rich radiology study just published showed how powerful multimodal LLMs can be on an x-ray exam, but CAP-X tests something different and arguably closer to the physical world. From a practical standpoint, CAP-X could be useful for training and safety. In the future, AI assistants might accompany real doctors or even perform basic monitoring tasks. CAP-X offers a way to vet those systems in silico before real deployment. If an LLM or autonomous clinical agent can at least pass CAP-X scenarios reliably, that would build confidence in its clinical reasoning. Conversely, failure points illuminated by CAP-X (such as omitted exam steps or pathological reasoning) can guide developers on what data or fine-tuning the models need. In that sense, CAP-X is a diagnostic test for the AI itself, identifying where its medical thinking is competent and where it needs work. Technically, adapting an LLM to connect with CAP Gym is itself an interesting exercise. Since CAP Gym likely provides a code-based API (like , ) compatible with OpenAI Gym interfaces, researchers can wrap an LLM as a policy module. In practice, this might involve prompt engineering: feeding the LLM the current environment description and asking it to output the next action. Previous works have done similar things (“Gym-LLM” frameworks). For example, one can use a prompt like “You are in an exam room. The patient’s vitals are BP 130/85, HR 90, oxygen 98%. Patient says ‘My ankle hurts.’ What exam maneuver do you perform next?” The LLM’s text reply (“Inspect the ankle for swelling and deformity”) can then be parsed into the simulator’s action. The loop continues until the LLM decides it has enough to diagnose or the scenario times out. Behind the scenes, the CAP Gym code checks each action’s validity and supplies the appropriate feedback text. One promising approach is to allow LLMs to use chain-of-thought reasoning for these tasks. In academic studies, LLMs that are prompted to “think step by step” often perform better on complex tasks. In CAP-X, it might be natural to let the model explain its reasoning as it goes (for example, by prompting it to list exam steps and why). Even if the official scoring rubric only cares about the final chosen actions, having the model “talk out loud” could improve its consistency and make it easier to debug. In the JMAI physical exam study, the outputs were scored for comprehensiveness and clarity, so transparency was important. For CAP-X, an LLM agent could be adjudicated by both automated scoring and human reviewers evaluating the quality of its explanations. Of course, a notable challenge is safety and bias. Medical simulations must avoid the LLM taking “shortcuts” that real doctors wouldn’t. For instance, an LLM might be tempted to “cheat” by spitting out a diagnosis without exam steps. A well-designed CAP Gym benchmark would penalize that heavily (perhaps by not even showing lab results until the correct exam confirmation is done). Also, CAP-X must ensure diversity of cases so that models don’t just memorize a narrow scenario set. Ideally the underlying simulator can be randomized or procedurally varied. This echoes the concerns of the nature medicine blueprint paper about generalized evaluation – CAP-X should include enough scenario variety that an LLM has to apply medical principles, not just recite one learned script. From the early results we’ve seen, GPT-4 and its successors fare surprisingly well on basic exam steps, but there is plenty of room to improve. The CAP Gym team’s evaluation likely shows that even top LLMs make occasional dangerous mistakes (for example, asking for a lumbar puncture when it’s not indicated, or failing to consider the patient’s full history while doing the exam). These errors are instructive. They suggest that combining LLMs with additional tools (like medical imaging analysis, or rule-based specialist modules) might yield better composite agents. In other words, passing CaP-X might not come from any single model, but from hybrid systems where an LLM handles open-ended reasoning and other modules provide structured expertise or fallbacks. In conclusion, CaP-X represents a bold new benchmark at the intersection of AI, language, and embodied clinical care. Unlike static tests, it measures how well an AI can interplay with a simulated patient environment. This is a big step toward “embodied AI” in medicine – machines that not only know, but can to some extent do. For developers and researchers, CAP-X will be a valuable challenge. It will be used to stress-test LLMs as they become more integrated into healthcare workflows, and to guide the development of models that truly understand the physical, human side of medicine. As one findings from the literature suggests, melding LLMs into dynamic, interactive simulations can “guide the advancement” of AI in healthcare by highlighting real-world competencies and gaps. CaP-X is, fittingly, the first real physical exam for LMs – and just as real patients do, it will tell us how healthy our AI doctors really are. Sources: CAP Gym’s CaP-X benchmark is informed by recent studies of LLMs in medical scenarios, as well as emerging ideas about dynamic clinical simulation. In particular, prior work evaluated GPT-4 on exam instructions and practice question benchmarks, while reviews note the rise of virtual patient simulations to enrich medical training. CaP-X builds on these trends by embedding LLMs in fully interactive exam tasks, offering a new way to assess and improve AI in healthcare.