Latest / Becker’s Healthcare Digital Health + Health IT / AI chatbots spread medical misinformation: Mount Sinai, UPMC + UChicago Medicine back AI startup in $55M funding round + more

Transcript

AI chatbots spread medical misinformation: Mount Sinai, UPMC + UChicago Medicine back AI startup in $55M funding round + more · Becker’s Healthcare Digital Health + Health IT

• 0:00
  This is Naomi Diaz with the Becker's Digital
• 0:02
  Health podcast. Here's your daily industry news briefing
• 0:05
  for today, August 6.
• 0:07
  First,
• 0:08
  researchers from the Icahn School of Medicine at
• 0:11
  Mount Sinai found that widely used AI chatbots
• 0:14
  designed for medical support
• 0:16
  are vulnerable to repeating and elaborating on false
• 0:19
  health information.
• 0:21
  Published August 2 in communications medicine,
• 0:24
  the study tested how large language models respond
• 0:27
  to fictional patient scenarios containing made up medical
• 0:30
  terms.
• 0:31
  The chat bots often accepted the false information
• 0:34
  without question,
• 0:35
  generating detailed responses about nonexistent
• 0:38
  conditions, symptoms, or treatments.
• 0:42
  Researchers then introduce a simple safeguard,
• 0:45
  a one sentence warning at the start of
• 0:47
  the prompt reminding the AI that the information
• 0:49
  provided may be inaccurate.
• 0:52
  With this added prompt, the rate of errors
• 0:54
  drops significantly,
• 0:55
  suggesting that even minimal guardrails
• 0:58
  can help reduce misinformation
• 1:00
  generated by AI systems.
• 1:02
  The findings highlight the risk of relying on
• 1:04
  AI chat box for medical decision making and
• 1:07
  underscore the importance of prompt engineering,
• 1:10
  system design, and human oversight.
• 1:13
  According to the research team, the results support
• 1:15
  the need to test AI tools using fabricated
• 1:18
  inputs before they are deployed in clinical settings.
• 1:21
  Researchers plan to apply the method to de
• 1:23
  identified patient data and explore additional safety props
• 1:27
  and retrieval tools as AI technology
• 1:30
  continues to evolve.
• 1:32
  Second,
• 1:33
  the venture capital arms of UPMC and UChicago
• 1:36
  Medicine have invested in a health care AI
• 1:39
  startup as a part of a $55,000,000
• 1:41
  series c funding round.
• 1:44
  UPMC
• 1:45
  enterprises and UCM ventures back all Ultramix,
• 1:49
  which says it offers its first FDA cleared
• 1:52
  Medicare reimbursed
• 1:53
  AI technology
• 1:54
  to aid in detecting cardiac amyloidosis
• 1:57
  and heart failure with preserved ejection fraction.
• 2:00
  The company developed the tool in collaboration with
• 2:03
  Mayo Clinic.
• 2:05
  UltraMix other health system partners include university hospitals
• 2:09
  and Northwestern
• 2:10
  Medicine.
• 2:11
  The startup says this platform helps hospitals cut
• 2:13
  unnecessary
• 2:14
  tests, enhance workflows, and start treatments earlier.
• 2:18
  Third,
• 2:19
  health systems are increasing their adoption of Microsoft's
• 2:22
  Ambient AI for clinical documentation.
• 2:26
  Here are four things to know from a
• 2:27
  July 30 Microsoft earnings call.
• 2:30
  First, Microsoft had a breakout physical 2025
• 2:33
  for the use of its Dragon co pilot,
• 2:35
  AKA DAX, which ambiently listens to patient visits
• 2:38
  and drafts clinical notes for the EHR, said
• 2:41
  Microsoft chair and CEO, Sato Nadala.
• 2:44
  Second,
• 2:45
  health care customers employ the tool to document
• 2:47
  over 13,000,000
• 2:48
  patient encounters in the most recent quarter, up
• 2:50
  nearly seven times year over year.
• 2:53
  Third,
• 2:54
  at Mercy, more than 1,000 physicians are now
• 2:57
  using Dragon Co pilot.
• 2:59
  Four, mister Nadella said of Mercy, quote, they
• 3:02
  have saved more than a 100,000
• 3:05
  to date and plan to expand to all
• 3:07
  5,000 providers.
• 3:08
  As one physician put it, the best thing
• 3:11
  to happen to my practice in ten years,
• 3:13
  end quote.
• 3:14
  If you would like the latest health IT
• 3:16
  and digital health industry news delivered to your
• 3:18
  inbox every morning,
• 3:20
  subscribe to the Becker CIO and health IT
• 3:22
  review e newsletter through our website at ww.beckershospitalreview.com.