Latest / The Augur Dispatch / Medical AI: a good answer is not a finished job

Transcript

Medical AI: a good answer is not a finished job · The Augur Dispatch

• 0:00
  SPEAKER_00: A medical AI can write the right plan and still leave the work undone.
• 0:04
  SPEAKER_00: Stanford's researchers found agents that describe an action in a clinical note without placing the order.
• 0:10
  SPEAKER_00: In a separate administrative benchmark, the strongest tested agent finished only about a third of complete tasks.
• 0:17
  SPEAKER_00: For a buyer, that changes what counts as a successful demonstration.
• 0:21
  SPEAKER_00: Ask to see the completed request in the receiving system.
• 0:24
  SPEAKER_00: Then count the staff time needed to get it there.
• 0:27
  SPEAKER_00: This is the Augur dispatch for Tuesday, October 20th, 2026.
• 0:33
  SPEAKER_00: Picture an authorization coordinator near the end of the afternoon.
• 0:36
  SPEAKER_00: The draft looks good, the supporting information is there, and the software says the request is ready.
• 0:42
  SPEAKER_00: Someone still has to establish whether the file reached the payer and whether the confirmation made it back into the record.
• 0:49
  SPEAKER_00: That person isn't an inconvenient detail in the automation business case.
• 0:53
  SPEAKER_00: Their remaining workload is part of its cost, and Auger's view is that buyers should price the complete assisted process before claiming they've removed a job.
• 1:03
  SPEAKER_00: Stanford's earlier benchmark, Medhelm, brought 29 clinicians into the design of a broad map of medical work.
• 1:11
  SPEAKER_00: It covers activities such as clinical notes, patient communication, and administration.
• 1:16
  SPEAKER_00: That is a better starting point than assuming a medical exam represents everything a health system needs.
• 1:22
  SPEAKER_00: But Medhelm mainly gregs responses.
• 1:25
  SPEAKER_00: The researchers say explicitly that scoring what an AI writes at one step doesn't establish whether a later step happens at all.
• 1:33
  SPEAKER_00: The low administration score in Stanford's historical summary isn't the current leaderboard ceiling.
• 1:38
  SPEAKER_00: The live table already shows a much higher relative score for GPT-5.
• 1:43
  SPEAKER_00: A relative win rate compares models within an evaluation.
• 1:47
  SPEAKER_00: It isn't the share of hospital requests that an agent completes correctly.
• 1:51
  SPEAKER_00: That distinction matters more than choosing the more flattering number.
• 1:55
  SPEAKER_00: A procurement team could replace an old low score with a new high score and still be measuring the wrong thing for the contract.
• 2:03
  SPEAKER_00: Stanford therefore built tests that let agents operate software, then inspect what changed.
• 2:09
  SPEAKER_00: Those tests get closer to the work a buyer is paying for, although they still aren't a hospital deployment.
• 2:15
  SPEAKER_00: Health Admin Bench covers authorization requests, appeals, and medical equipment orders in simulated systems.
• 2:22
  SPEAKER_00: Researchers spent more than a hundred hours observing administrators at Stanford Healthcare before building the environments, and the patient information used in the benchmark is synthetic.
• 2:33
  SPEAKER_00: The best tested agent, Claude Opus 4.6 in its native computer use system, completed about 36% of whole tasks.
• 2:41
  SPEAKER_00: That number is specific to this evaluation, not a prediction of how every hospital pilot will perform.
• 2:47
  SPEAKER_00: The sharper comparison comes from GPT 5.4.
• 2:51
  SPEAKER_00: It completed about 83% of individual steps, yet only about 27% of whole tasks.
• 2:58
  SPEAKER_00: The same agent can look capable when you count small actions, and unreliable when you ask whether the job finished.
• 3:05
  SPEAKER_00: Researchers describe information getting lost between systems, files that weren't uploaded or downloaded, and agents running out of allowed steps instead of returning to repair an earlier mistake.
• 3:17
  SPEAKER_00: Return to that coordinator's desk.
• 3:19
  SPEAKER_00: A polished draft may save useful time.
• 3:22
  SPEAKER_00: A missing attachment can send the case back into the queue, where someone must discover the omission and repeat part of the process.
• 3:29
  SPEAKER_00: The software has produced something.
• 3:31
  SPEAKER_00: Whether the team has less work is a separate measurement.
• 3:35
  SPEAKER_00: A completion report would be a particularly unfortunate place to discover another unfinished task.
• 3:41
  SPEAKER_00: The clinical evidence adds a different warning.
• 3:44
  SPEAKER_00: Physician Bench gives agents longer tasks inside a simulated patient record system.
• 3:49
  SPEAKER_00: Its best tested model completed 46% on a single attempt, and the research includes the cases where an agent wrote the intended action without executing it.
• 3:59
  SPEAKER_00: Don't compare that 46% directly with the administrative benchmark's 36%.
• 4:04
  SPEAKER_00: The environments, tasks, models, and grading differ.
• 4:08
  SPEAKER_00: The shared purchasing lesson is to inspect the recorded result rather than treating the agent's explanation as proof of completion.
• 4:16
  SPEAKER_00: The strongest objection to Augur's caution is that useful assistance doesn't require full autonomy.
• 4:22
  SPEAKER_00: A system might draft a request well enough to reduce total effort even when a person still submits it and checks the outcome.
• 4:29
  SPEAKER_00: That objection is right.
• 4:31
  SPEAKER_00: These studies don't establish that every assisted workflow loses money, and they don't tell you the economics of your team's particular arrangement.
• 4:39
  SPEAKER_00: They also leave out some production obstacles, including changing interfaces, authentication steps, and session timeouts.
• 4:46
  SPEAKER_00: Real systems could expose further difficulties, while specialist integration or human recovery could improve performance.
• 4:54
  SPEAKER_00: Neither direction should be assumed away.
• 4:56
  SPEAKER_00: The defensible purchase is the arrangement you have tested, with the remaining human work included.
• 5:02
  SPEAKER_00: A demonstration can be impressive without being a staffing plan.
• 5:07
  SPEAKER_00: A larger model isn't necessarily the cheapest way to repair this problem.
• 5:11
  SPEAKER_00: The surrounding agent software controls how the model uses tools, remembers information, and recovers from mistakes.
• 5:18
  SPEAKER_00: Engineers often call that software a harness or wrapper.
• 5:23
  SPEAKER_00: Cerebris compared two wrappers using the same Claude model on one whiteboard application build.
• 5:28
  SPEAKER_00: Factory Droid finished 11 minutes ahead of Claude Code, at roughly twice the pace, with about half the tool calls.
• 5:36
  SPEAKER_00: That is a vendor's account of one run, not a universal winner.
• 5:40
  SPEAKER_00: It does show why changing the surrounding software deserves a test before a buyer assumes the model alone sets performance.
• 5:47
  SPEAKER_00: An engineering lead reviewing a failed authorization could discover that the model had the right information, but the system never carried it to the next screen.
• 5:56
  SPEAKER_00: Buying more reasoning won't necessarily repair a missing handoff.
• 6:00
  SPEAKER_00: Augur would pilot one narrow administrative workflow under the Revenue Cycle Operations Lead, with required human approval and no unapproved rights to live records.
• 6:10
  SPEAKER_00: Compare completed cases and total checking and repair time against the current process, not just the time spent drafting.
• 6:17
  SPEAKER_00: For a pilot starting today, the proposed review is November 3rd.
• 6:21
  SPEAKER_00: Stop if a material safety or privacy error occurs, the agent exceeds its permissions, or review work consumes the expected benefit.
• 6:30
  SPEAKER_00: Expand only after repeated runs show that quality holds and total staff effort fall.
• 6:36
  SPEAKER_00: The useful advantage would come from learning local exceptions and building reliable recovery.
• 6:41
  SPEAKER_00: A subscription is available to competitors too.
• 6:44
  SPEAKER_00: Knowing why your own queue gets stuck takes more work than entering a purchase order.
• 6:49
  SPEAKER_00: There are a few signals on the clock, distinct from Augur's dated predictions.
• 6:54
  SPEAKER_00: For the healthcare lead, watch improvements in complete workflow execution and recovery, not merely a better response score.
• 7:02
  SPEAKER_00: The administrative benchmark's individual step results show how an encouraging average can coexist with unfinished cases.
• 7:09
  SPEAKER_00: The same caution applies to local hardware.
• 7:12
  SPEAKER_00: Apple has published results for Muse Spark Open on both two and four Mac Studios.
• 7:18
  SPEAKER_00: Two machine performance is measured, not just a promise that the model fits.
• 7:22
  SPEAKER_00: Its table shows about 25 generated word pieces a second with two machines using one parallel approach, versus about 37 with four.
• 7:31
  SPEAKER_00: Those are specific performance tests, not evidence that the model will complete your agent workload.
• 7:37
  SPEAKER_00: Watch its task results separately before turning a hardware demonstration into a procurement commitment.
• 7:43
  SPEAKER_00: OpenAI has also already published scores for its new programming evaluations, including a test of building a database engine and another of migrating older COBOL programs.
• 7:54
  SPEAKER_00: The next useful evidence is replication and preservation of real business rules, not the first appearance of scores.
• 8:01
  SPEAKER_00: Teams using affected Transformers software should validate the latest input type fix against their own applications.
• 8:08
  SPEAKER_00: The written dispatch retains Augur's prediction that a U.S.
• 8:11
  SPEAKER_00: developed open weight model will exceed 39 on the Artificial Analysis Intelligence Index by March 31st, 2027, at 65% odds.
• 8:21
  SPEAKER_00: Augur calls carry explicit hit-or-miss criteria and are tracked on the website so their claims can be checked later.
• 8:28
  SPEAKER_00: Imagine the coordinator leaving on time because the assisted process really did reduce the cue, including the checking.
• 8:35
  SPEAKER_00: That is a worthwhile outcome.
• 8:37
  SPEAKER_00: It doesn't require pretending that every part of the job disappeared.
• 8:41
  SPEAKER_00: Adopt completion checks now, pilot the assisted workflow, and watch the claims that still lack results in your environment.
• 8:48
  SPEAKER_00: Augur would rather see a modest purchase with measured savings than a grand automation commitment that quietly creates another review team.
• 8:56
  SPEAKER_00: The written brief with every source is at AugurDispatch.ai.
• 9:01
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 9:04
  SPEAKER_00: That is the Augur Dispatch for Tuesday, October 20th, 2026.
• 9:09
  SPEAKER_00: Thank you for listening.