Latest / Becker’s Healthcare Podcast / Trust Is Not a Feature: The New Rules of Clinical AI Governance and Validation
Transcript
- 0:00- Well, hello, everyone. This is Erica Spicer Mason
- 0:02- with Becker's Healthcare.
- 0:04- Thank you so much for tuning in to
- 0:05- the Becker's Healthcare podcast series.
- 0:08- Today, we're going to talk about clinical
- 0:10- AI governance.
- 0:12- And joining me for this conversation today, we
- 0:14- have doctor Sheila Bond, the head of clinical
- 0:17- AI at Wolters Kluwer Health.
- 0:19- Doctor Bond, welcome. Thank you so much for
- 0:21- being here today.
- 0:22- Thank you. Pleasure to be here. Well, doctor
- 0:25- Bond, it's so great to have you with
- 0:27- us today. And I thought to get us
- 0:28- started, it would be great if you could
- 0:30- just share a a little bit more about
- 0:32- yourself, your path in health care, and how
- 0:34- does your work as head of clinical AI
- 0:37- at Wolters Kluwer
- 0:38- look day to day? What does it involve?
- 0:41- Sure.
- 0:42- It's been I mean, one, it's just such
- 0:44- an exciting era
- 0:46- for medicine.
- 0:47- I've been in medicine for
- 0:49- twenty two years now, so taking care of
- 0:52- patients either at Mass General or Brigham
- 0:54- for that time period.
- 0:56- And
- 0:57- for the past,
- 0:59- fourteen years or so, alongside of that, I've
- 1:01- been in clinical decision support, usually at the
- 1:04- intersection of content and technology.
- 1:07- So I have really thought deeply
- 1:09- about how the words that we create in
- 1:12- medicine, whether that's through an evidence based medicine
- 1:14- reference
- 1:15- or tool,
- 1:17- or how we teach students on the words,
- 1:20- making sure that the information that we give
- 1:22- to the next generation is the right information
- 1:25- for patient care and always encouraging them to
- 1:28- think critically.
- 1:29- So this has been just a really exciting
- 1:31- era for me. I think it's just a
- 1:33- shift in how we're gonna create medical information
- 1:35- for generations to come. So
- 1:38- I feel really honored to, you know, simply
- 1:41- shepherd that in. And the teams that I
- 1:43- work with are
- 1:45- amazing in terms of the dedication and passion
- 1:48- that they have for this mission.
- 1:50- As the head of clinical AI, my role
- 1:53- is really to
- 1:55- help them create the the vision of what's
- 1:57- next for how we deliver information,
- 2:00- whether that's through technology or through the words
- 2:02- that we write on the page.
- 2:04- It's also getting clinicians and engineers
- 2:07- together
- 2:07- to create that vision that takes both of
- 2:09- their combined
- 2:11- expertise
- 2:11- to figure out how to use generative AI
- 2:14- to remain trustworthy
- 2:16- and, to create not only the system
- 2:19- that creates information and presents it in a
- 2:22- new way, but also how we validate it,
- 2:24- how we govern it, and how we hold
- 2:26- ourselves accountable.
- 2:27- And it's, you know, exciting from start to
- 2:30- finish, and it probably taps into every element,
- 2:33- of every skill that I've accumulated over the
- 2:36- years.
- 2:37- Yeah. Doctor Bond, what I appreciate the overview,
- 2:40- by the way, and what an exciting role
- 2:42- you've stepped into as you mentioned. You know,
- 2:44- the fact that it's combining so much of
- 2:46- what you've already done in health care and
- 2:48- is navigating a lot of uncertainty ahead. It
- 2:50- sounds like a very exciting place to be.
- 2:53- And I wanna go a little bit deeper
- 2:54- on something that you mentioned earlier in your
- 2:56- response, and that was this
- 2:58- real dedication that I know your teams at
- 3:00- Walter's Cloura Health have when it comes to
- 3:03- clinical AI and making
- 3:05- sure that the content that is surfaced at
- 3:07- the point of care is trusted.
- 3:10- Trust comes up constantly in conversations when we're
- 3:13- talking about clinical AI, but it can definitely
- 3:15- mean different things to different people, different stakeholders
- 3:19- in the care journey.
- 3:20- So from your point of view, when a
- 3:22- clinician or a health system says that they
- 3:24- trust a tool, what do you think that's
- 3:26- really resting on?
- 3:28- That's that's a great question. I think the
- 3:29- era has really come back around to trust
- 3:32- and what it means specifically. So that's one
- 3:34- of the questions that I hear
- 3:36- a lot these
- 3:37- days. It's not a feeling. Right? And I
- 3:39- think the way we have defined it is
- 3:41- one,
- 3:42- there's governance.
- 3:43- There are people who oversee the system, how
- 3:46- it's designed,
- 3:47- and know how it's designed so they can,
- 3:50- a, create what needs to be created, and
- 3:53- b, measure outcomes
- 3:54- and measure how it functions and display that
- 3:57- to others. So the fundamental tenants of trust
- 4:00- for us are, one, having a governance system.
- 4:03- Two, at this point in time, you know,
- 4:05- generative AI is still early,
- 4:08- in its history.
- 4:09- So we make sure that, it is grounded
- 4:12- in content that is worthy of our trust,
- 4:15- already. And I think that's one thing that
- 4:17- people forget is
- 4:18- that there has to be a source of
- 4:20- truth. Right? Generative AI is very good at
- 4:22- assembling words in the right order and having
- 4:24- them sound great, but making sure that they
- 4:27- are actually tied to a source
- 4:29- that is trustworthy itself is important.
- 4:32- And then the validation and the data that
- 4:34- we produce
- 4:35- is also, I think, really core to trust.
- 4:37- Right? Does the system perform as expected? How
- 4:40- do we know? How do we measure that?
- 4:41- And are we willing to share it with
- 4:43- you?
- 4:44- And then ultimately,
- 4:45- you know, who is
- 4:47- accountable for this? When something goes wrong, who
- 4:50- do you call? How do you discuss it?
- 4:51- How do people act, in that circumstance?
- 4:54- You know, it is a new technology, so
- 4:56- it won't be perfect, but we do have
- 4:58- ways and means of introducing things into medicine
- 5:01- when they're new. And I think those are
- 5:03- the central tenets. Right? That there is accountability,
- 5:05- governance,
- 5:07- measurements, and the ability to act and improve.
- 5:10- And I think the other thing that is
- 5:12- really important
- 5:14- is that, you know, technology took hold of
- 5:16- this early on, and a lot of the
- 5:18- measurements that you see out there are really
- 5:20- coming from the technical world.
- 5:22- But for us in health care, right, the
- 5:24- end user
- 5:25- is in health care. And usually,
- 5:28- well, the information that is acted upon is
- 5:30- for a patient. So I think we all
- 5:32- collectively as a community who is bringing this
- 5:34- together needs to remember that that's the end
- 5:37- game in this, and the measurements are not
- 5:39- just technical measurements of progress
- 5:41- for an AI system, but the measurements that
- 5:44- we wanna put forth, really demonstrate that this
- 5:48- is enough to act upon and the right
- 5:50- guardrails and safety mechanisms are in place for
- 5:53- use for with the patient
- 5:55- as the endpoint in mind.
- 5:58- It's a really helpful framework, doctor Bond, and
- 6:00- I appreciate you elaborating on those key pillars,
- 6:04- that kind of are building this this concept
- 6:07- of trust,
- 6:08- and not just a concept, but something that
- 6:10- is measurable, actionable.
- 6:13- And I wanna go a little bit deeper
- 6:14- too on governance. You had mentioned governance a
- 6:17- few times now, and this is another term
- 6:19- that can be used kind of broadly just
- 6:21- like the concept of trust. So
- 6:24- inside an organization,
- 6:25- what does AI governance look like when it's
- 6:27- working well?
- 6:29- I'd love to know,
- 6:30- you know, from your where you stand, who
- 6:32- tends to own it, and how do decisions
- 6:34- actually get made in a way that's effective?
- 6:37- That's a wonderful question as well.
- 6:40- And it really forces you and I think
- 6:42- a healthy way to break down what quality
- 6:44- is so that other people can understand it,
- 6:47- see it, and measure it. So it's a
- 6:48- good forcing function.
- 6:50- Our governance is really cross
- 6:52- disciplinary. I think, our clinical and our clinical
- 6:55- AI team set the standard for what good
- 6:58- is.
- 6:59- Everybody, right, needs to understand that so that
- 7:01- we can measure it. That includes our product
- 7:03- teams and our technology teams, And all of
- 7:06- us, right, are responsible for understanding the industry
- 7:09- out there. So technology does need to understand
- 7:11- a bit of health care and health care
- 7:12- does need to understand
- 7:14- a bit of technology.
- 7:16- So I think for the most part, you
- 7:17- know, in terms of the organization working well,
- 7:20- it's all of us collectively
- 7:22- understanding the same thing, having the same vision,
- 7:25- and predefining what good is and how we're
- 7:27- going to measure that, building the system, and
- 7:30- then building the tools to make those measurements,
- 7:33- looking at them all together, right, and understanding,
- 7:36- whether we've met our goals or not. And
- 7:39- if we have or if we haven't,
- 7:41- is there anything unexpected that we didn't foresee
- 7:44- that has come up? And then I think
- 7:46- one of the most challenging things is sort
- 7:48- of shifting modes. It's shifting from sort of
- 7:51- creating
- 7:51- into measuring.
- 7:53- And then before release, you know, technology is
- 7:55- known for fast release cycles. But once you
- 7:58- get into the elements
- 8:00- of
- 8:01- understanding and managing risk, it's a different framework.
- 8:04- Meaning, you know, you you work until that
- 8:07- risk is gone or otherwise mitigated or settled
- 8:10- in some other way.
- 8:11- And I think that's,
- 8:13- sort of a mark of the era is
- 8:15- a, you know, we all have to be
- 8:16- able to share a vision, communicate to one
- 8:19- another because there's no one
- 8:21- domain, so to speak, that's gonna be able
- 8:23- to do this themselves. It's gonna take an
- 8:25- alliance between health care and technology.
- 8:29- So it is that ability to, you know,
- 8:31- break something down so that,
- 8:33- folks you don't usually work with can understand
- 8:35- it, to measure it, to measure it to
- 8:37- a predefinition,
- 8:38- and then,
- 8:40- to look at it and say, have we
- 8:41- done enough? And if we haven't, you no
- 8:43- longer work to a time. Right? You work
- 8:45- until that risk is gone. Mhmm.
- 8:48- Yeah. Yeah. Doctor Bond, I appreciate you, defining
- 8:51- governance.
- 8:52- You know, we we are fresh off of
- 8:54- our IT and revenue cycle conference at Becker's,
- 8:56- and I have to say that the way
- 8:58- that you just described it aligns so closely
- 9:01- with what I know a lot of organizations
- 9:03- are are striving for.
- 9:05- But to your point, it is challenging to
- 9:07- put into action, and also there's no one
- 9:10- way to do governance. So it seems to
- 9:13- certainly be a challenge for most organizations.
- 9:16- And I also wanna follow-up with
- 9:18- on this idea of risk here
- 9:21- and talk a little bit more about validation.
- 9:23- I know validation tends to get framed as
- 9:25- something that happens before a new tool or
- 9:28- technology goes live.
- 9:30- But how do you think about it once
- 9:32- a tool is already in clinical use? What's
- 9:35- hardest to measure at that stage?
- 9:38- Great question as well. I think that speaks
- 9:40- to sort of the accountability
- 9:42- pillar
- 9:43- that I talked about earlier.
- 9:46- It's not only, right, what you do before
- 9:48- you design your system and how you measure
- 9:50- it when it's at the gate,
- 9:52- but, you know, that post monitoring surveillance
- 9:55- is very important.
- 9:57- So looking at, you know, whatever your system
- 10:00- is designed to do, right, looking at how
- 10:02- it fares in the real world, at this
- 10:04- point in time, I think for most clinical
- 10:06- AI systems, it's understanding the types of questions
- 10:08- that will come in, what your system's output
- 10:11- is.
- 10:12- And we're always really sure that we've got
- 10:14- expert eyes on the sampling of what is
- 10:16- out there in the industry
- 10:18- just because of how, precious
- 10:21- it is. The hardest thing, of course, is
- 10:24- that, you know, this is a technology that
- 10:26- isn't going to be perfect for a while.
- 10:29- So measuring how,
- 10:31- you do in the real world isn't an
- 10:33- easy matter. And even so with some of
- 10:35- our traditional medical references, right, you can write
- 10:38- something to perfection in that state,
- 10:41- but still someone might not not understand it
- 10:43- well enough. Right? So the hardest part is,
- 10:45- you know, the gap between,
- 10:47- you know, doing everything you possibly can to
- 10:49- ensure that, you're delivering
- 10:52- what should be worthy of trust.
- 10:55- But then, you know, the actions that someone
- 10:57- takes, or how someone interprets that information
- 11:00- is,
- 11:01- you know, always the thing one of the
- 11:03- things that, will keep you up at night.
- 11:05- That's sad. Right? I think, you know, when
- 11:07- you're talking about governance,
- 11:09- it probably, you know, there's the industry level
- 11:12- governance and then there is also,
- 11:14- you know, what we are responsible for as
- 11:16- a community of medicine and as an industry,
- 11:19- which I think is to educate our users
- 11:21- on what AI literacy is, and how to
- 11:24- use these tools
- 11:26- both individually and then, you know, that's now
- 11:28- part of medical school education as well, coming
- 11:32- more and more
- 11:33- AI literate. So I think those things go
- 11:35- hand in hand with the governance of the
- 11:37- system.
- 11:38- And as always, it's just really heartening to
- 11:40- be part of the community of medicine and
- 11:43- see that they do respond to that call.
- 11:45- Right? I've been all over the world talking
- 11:47- about clinical AI.
- 11:48- And whether it's, you know, in Europe or
- 11:51- Asia or
- 11:53- The United States,
- 11:54- I think everyone is starting to speak the
- 11:57- same language and taking on, the responsibility that
- 12:00- the community of medicine does. So it's been
- 12:01- really heartening to see folks step up in
- 12:04- the way that they usually do and heartening
- 12:05- to see that, you know, across the world,
- 12:08- there's the layer of of medicine and how
- 12:10- much people care no matter what region they're
- 12:12- from that we get this right. So
- 12:15- I appreciate the global perspective there, doctor Bond.
- 12:18- And so interesting too to hear how medical
- 12:21- curricula are already changing to support AI in
- 12:25- clinical use.
- 12:26- And kind of in that vein, I'm curious
- 12:28- whether we're talking about clinical AI use or
- 12:31- governance or even validation.
- 12:34- What questions are you hearing from clinical and
- 12:37- IT leaders now that maybe you weren't hearing
- 12:39- even just one or two years ago?
- 12:42- I think the questions are more pointed and
- 12:45- precise now. I think we're sort of beyond
- 12:47- the wow factor,
- 12:49- and, we're beyond the usability factor and sort
- 12:52- of the fun factor and the novelty factor
- 12:54- of this, and we're into
- 12:56- how are we actually gonna integrate it, and
- 12:58- what does it take to get it right,
- 13:00- and what does it take to scale.
- 13:03- So, you know, as you talked about at
- 13:05- the beginning of the interview and trust, people
- 13:07- are now asking
- 13:08- not just for the who and the how
- 13:10- or the system design, but really the specific
- 13:12- data and starting to drill down into what
- 13:15- really matters.
- 13:16- So I've been one, you know, we've been
- 13:18- working on that foundationally for so long. I'm
- 13:21- really thrilled to hear those questions coming back
- 13:24- from the industry itself and from, you know,
- 13:27- academic medical centers
- 13:29- as to, you know, what really the foundation
- 13:31- should like should look like, and what should
- 13:34- you measure to make sure that,
- 13:36- you know, whatever we build now is gonna
- 13:38- be the foundation for the future.
- 13:40- So I'm really happy to hear that people
- 13:43- are educating
- 13:44- themselves,
- 13:45- looking at the data, reviewing the data. Right?
- 13:48- There are, you know, chats on LinkedIn
- 13:50- about,
- 13:51- benchmarks in medicine and how best we should
- 13:54- measure things.
- 13:55- So I'm just very glad to see that
- 13:58- those questions are coming up and people are
- 14:00- asking for,
- 14:02- what precisely
- 14:03- one should measure. And in that regard, you
- 14:05- know, I think
- 14:07- for us, one of the most important things
- 14:10- is to make sure and,
- 14:12- this is really interesting because,
- 14:15- as I said, I was just in Asia
- 14:16- and Singapore and in China, and, you know,
- 14:19- we were all coming to the same point
- 14:21- about, you know, what of all of this
- 14:22- should be measured
- 14:24- as we're starting to grow and to scale.
- 14:27- And making sure
- 14:29- that,
- 14:30- when this technology is creating words that will
- 14:33- impact patient care,
- 14:34- that there is,
- 14:36- no unsupported leaps. Right? And how do we
- 14:38- start to quantify
- 14:40- inference, right, is sort of this point of
- 14:43- intersection that we were at across, you know,
- 14:45- three different countries. So it was very interesting.
- 14:47- It's not like a benchmark or a safety
- 14:49- score or
- 14:50- something gimmicky.
- 14:51- But the intersection that we were at was,
- 14:53- you know, for years, we have,
- 14:56- and decades really, in the field of evidence
- 14:58- based medicine made sure that, we have a
- 15:01- system for
- 15:03- analyzing and appraising the medical literature to make
- 15:06- sure it's the right information to act on
- 15:08- for patient care.
- 15:09- And we have that strong foundation. So we
- 15:11- now don't want to
- 15:13- create a new foundation based on ease of
- 15:15- use
- 15:16- and or just information that is plausible or
- 15:19- unsupported, and that technology can so easily do
- 15:22- that.
- 15:23- What we came around to, I think, in
- 15:24- discussion was
- 15:26- the ability to quantify
- 15:28- and measure whether your system is taking a
- 15:30- leap and making a recommendation for patient care
- 15:33- or are you being disciplined? And, you know,
- 15:35- even though it might not be as
- 15:37- personally pleasing, right, to to get a more,
- 15:41- challenging answer to your question, that may be
- 15:43- the right thing to do, and that's what
- 15:45- we wanna measure.
- 15:47- Yeah. Well, it it's been great hearing doctor
- 15:49- Bond about this increased
- 15:51- curiosity and more targeted questions that you're hearing
- 15:54- from clinicians and leaders alike over the over
- 15:56- just the last two years, really.
- 15:59- But as you look ahead,
- 16:00- you know, when you consider where clinical AI
- 16:03- may be heading and even just three to
- 16:05- five years from now, is there anything on
- 16:07- your mind that we haven't yet touched on
- 16:09- that you wanted to highlight or any final
- 16:11- thoughts you'd like to leave our listeners with?
- 16:14- Sure. Probably two things. I think first, I
- 16:17- can't underscore enough. I said it probably throughout
- 16:20- this interview, but
- 16:21- right now, we're foundation building again. Right? And,
- 16:25- making sure that we get this foundation right
- 16:28- is just mission critical for us because I
- 16:30- think it really will be the foundation
- 16:33- for the health care information that we use
- 16:36- in EHRs, in ambient systems,
- 16:38- and for many, many, many things to grow.
- 16:41- So don't rush into it. You know, make
- 16:43- sure that foundation is worthy
- 16:45- of the decisions that it's going to support.
- 16:48- That's number one. And number two is
- 16:52- the whole reason to do that is that
- 16:54- this opportunity for health care is absolutely enormous.
- 16:57- Right? When you think
- 16:58- about the power of generative AI, it's not
- 17:00- really to summarize and retrieve things. It really
- 17:03- is to transform things so that people can
- 17:06- understand them better.
- 17:07- And throughout my career in medicine and twenty
- 17:10- two years, that really is the most important
- 17:13- thing is to make sure that your patients
- 17:15- understand. Right? Every patient journey is unique.
- 17:19- And I remember starting in medicine, it was
- 17:21- entirely different field.
- 17:24- You used to take care of your patient
- 17:26- in the outpatient setting, in the inpatient setting.
- 17:29- Sometimes you would meet them in the emergency
- 17:30- department
- 17:31- and walk them up to the room. And
- 17:33- you knew who they were holistically as a
- 17:35- person, so you knew how to connect with
- 17:37- them or how to
- 17:38- communicate with them. And, you know, in those
- 17:41- little moments, you might tell them what would
- 17:43- happen next. And that was such a huge
- 17:45- part of taking care of a person. And
- 17:47- now as medicine has changed and grown, right,
- 17:50- someone else does those things. Someone else takes
- 17:52- a patient to the ED. Someone else walks
- 17:54- them from point a to point b.
- 17:57- And that ability to explain is lost
- 18:00- in some ways, or that moment to moment
- 18:02- care where you're helping someone understand what is
- 18:05- next is lost.
- 18:06- And I think the really enormous opportunity that
- 18:09- we have
- 18:10- for health care workflow
- 18:12- is to bring that back in. Right? So
- 18:15- folks these days talk about workflow like it's
- 18:17- an efficiency thing or like workflow as a
- 18:19- destination and everything in it is fixed.
- 18:22- But I think the worst thing we could
- 18:23- do for health care in many ways is
- 18:25- to stuff more information into that old workflow.
- 18:28- Right? That workflow was designed for the past.
- 18:31- And if I'm to sort of reenvision a
- 18:33- new new workflow, right, at the center of
- 18:35- it is probably using this technology so that
- 18:38- other people can understand,
- 18:40- what's happening to them again and for its
- 18:43- transformative potential.
- 18:45- So, you know, the reason we invest so
- 18:47- much in our foundation building
- 18:49- is so that we can realize that. Right?
- 18:51- So that we can take what is trusted
- 18:53- information, so that we can take
- 18:55- information that we know doesn't have any unsupported
- 18:57- or crazy leaps in it and then change
- 18:59- it. Right? So that someone else can understand
- 19:01- in a way that is helpful to for
- 19:03- them.
- 19:04- And to me, that's sort of the ultimate.
- 19:07- You know, when you think about medical
- 19:09- information, it's sort of locked in writing, and
- 19:11- that's a really elite form of communication
- 19:14- that takes a lot of years, right, to
- 19:16- understand. And for medical information,
- 19:18- even more so. Right? So if we can
- 19:20- use our brains and our expertise and everything
- 19:22- we learned, instead of unlock that, I think
- 19:24- the future is
- 19:26- is bright.
- 19:28- Well, doctor Bond, it's been such a pleasure
- 19:31- talking to you. Your passion for your work
- 19:33- comes comes through so clearly just as we're
- 19:36- speaking to one another. So I wanna thank
- 19:38- you for for bringing that to our conversation
- 19:41- and also
- 19:42- helping our listeners
- 19:43- see what is possible when that foundation
- 19:46- is
- 19:47- gotten right from the beginning. So,
- 19:49- again, appreciate your time today and sharing your
- 19:51- thought leadership with us. Very much appreciated.
- 19:54- Thank you. Always a pleasure to be here.
- 19:56- Wonderful questions. And as I said, I could
- 19:57- talk about this forever. So thank you.
- 19:59- Maybe we'll do a part two.
- 20:03- Well, thank you again, doctor Bond. And, of
- 20:04- course, we also wanna thank our podcast sponsor
- 20:07- today, Wolters Kluwer.
- 20:09- Listeners, be sure to tune in to more
- 20:10- podcasts from Becker's Healthcare by visiting our podcast
- 20:13- page at beckershospitalreview.com.