Latest / Health Ethics Podcast / How Will This Benefit Patients? AI Integration into Clinical Decision Making

Transcript

How Will This Benefit Patients? AI Integration into Clinical Decision Making · Health Ethics Podcast

0:08
Welcome to Health Ethics, a series of conversations with leading experts about ethical considerations
0:15
associated with health and healthcare, both locally and globally. I am Brian Pilkington,
0:20
Professor of Bioethics in the Department of Medical Sciences at the Hackensack Meridan School
0:25
of Medicine. This series, like its predecessor, which focused on the COVID
0:29
nineteen pandemic, relies on the idea that challenging ethical issues are best addressed by
0:34
folks from diverse backgrounds practically reasoning together, and complements the Health Ethics series with
0:41
its panel of leading experts from medicine, nursing, politics, economics, philosophy
0:47
and the law. Thank you and enjoy the episode Welcome to the Health Ethics
1:00
Podcast. I am Brian Pilkington, Professor of Bioethics at the Hackensack Maritean School
1:04
Medicine, and I have the great pleasure today of speaking with doctor Charles Binkley.
1:08
Doctor Binkley is a surgeon bioethicist, currently Director of Bioethics Central Region at
1:15
Hackensack Maritian Health. Doctor Binkley is also Associate Professor of Surgery at the Hackensack
1:19
Meridian School Medicine and holds the honorary title of Bioethics Fellow at the MARCLA Center
1:25
for Applied Ethics in Silicon Valley. His academic research is focused on ethical issues
1:30
impacting clinicians and patients when artificial intelligence systems are used to support clinical decisions.
1:37
And the best part about today's conversation is doctor Binkley has written a new book
1:42
and so I'm very excited to speak with him about the book. The book
1:45
is Encoding Bioethics, AI and Clinical Decision Making, which he wrote with Tyler
1:51
Loftis. Doctor Binkley, thank you for being with me today on the Health
1:56
Ethics podcast. You've written a new book which I'm super excited to read.
2:00
I know it'll be out soon. Tell us how did you get interested in
2:04
artificial intelligence in AI ethics? Yeah, thank you, Brian, and thank
2:07
you so much for having me on your podcast. I'm an avid listener and
2:13
I'm just absolutely flattered that you would invite me to be back on again.
2:17
So. I'm a liver in pancrea surgeon by training, and when I first
2:23
started in practice, I would see patients with newly diagnosed pancreatic cancer that met
2:28
all the criteria that we had to have a favorable prognosis. They had a
2:31
small tumor, there did not have metastasis, their tumor markers were low.
2:37
It looked like surgery was the absolute best treatment, and it was. And
2:42
so I would take these patients to the operating room. I would do surgery,
2:46
they would get a chemotherapy afterwards, sometimes radiation. By the book,
2:51
they would have the best possible prognosis, but then they would die within the
2:55
first couple of years because the cancer recurred. And this is something that has
3:00
caused cancer surgeons to scratch their heads and to sort of wonder, you know,
3:07
are there better ways that we can predict who will and who won't benefit,
3:10
because we're talking about a lot of human suffering that goes in to try
3:14
to achieve a benefit. We're talking about people going through major surgery, chemotherapy,
3:19
radiation in order to try to extend their survival, and then we failed
3:22
them, and then that doesn't happen. And so I kept asking myself,
3:25
We've got to be able, and many of us did. We've got to
3:29
be able to predict this better. We've got to be able to find indicies
3:34
that would be better prognosticators that would tell us who would benefit and who wouldn't
3:38
so that we can really enter into truly shared decision making with patients. And
3:44
people had looked at genes and gene expression and various tumor markers and nothing really
3:49
panned out. And so when I started to think about artificial intelligence within this
3:53
context of clinical uncertainty, and then all the other instances of clinical uncertainty that
3:58
I had faced in my career, clinical uncertainty about the sick patient whose septic,
4:01
or the patient with horrible rip roar Q pancreatitis, like are they going
4:06
to survive? Or patients in terms of a functional recovery, what is the
4:12
likelihood that this patient will recover function? And I thought, well, maybe
4:15
artificial intelligence, maybe there are factors that we're not accounting for. So I
4:20
can think about you know, tumor size, margin status and lymph node status,
4:25
you know, metastasity just or no, and those those are the factors
4:29
that I know, Those are the prognostic factors that I know that are very
4:30
well proven. But what if there are other things that have a huge influence
4:34
that we don't even pay attention to. And that was the idea behind you
4:41
know, maybe artificial intelligence will be better at predicting, helping us predict things
4:46
informing our decisions. You know, I think about when we talked about risks
4:50
and benefits, most surgeons kind of give a risk benefit profile based on their
4:57
own experience, and it may not really be very well tailor to that patient
5:00
and it may not necessarily take into account all the things that it should.
5:03
And so that's really an opportunity that where I see that artificial intelligence could potentially,
5:10
you know, make our predictions more objective. And then there's also the
5:14
idea of bias. You know, there's lots and lots of good literature showing
5:18
that physicians are biased in terms of what they offered what they don't offer patients.
5:23
And many times the bias is along socioeconomic lines, which becomes along racial
5:28
lines, and so the idea of removing that bias. I think about bias
5:31
also in terms of disabilities. You know, we have there are lots of
5:36
patients that with disabilities who are presumed not to have a high quality of life,
5:44
and that can very much bias things that they're offered or withheld based on
5:50
physician biases. So my idea is that artificial intelligence may be able to address
5:56
clinical uncertainty, reduce bias, and make our predictions more objective. Touch on
6:00
a variety of AI ethics issues, many of what you've spoken eloquently on in
6:05
all the context, and also what you've written about sort of explainability concerns,
6:09
considerations of bias. If I asked you the tough question what's the most pressing
6:13
topic in AI ethics, what would you say and why? Yeah, it's
6:16
a really great question, and so I would I would say, you know,
6:20
AI ethics is such a broad term. It that depending on the application
6:27
of AI, the broad application of AID, they're going to be very discrete
6:30
ethical concerns. I think when it comes to using AI for clinical decision making,
6:35
I think the overall biggest ethical concern that I have is how clinicians will
6:41
be able to integrate the prediction into their own clinical decision making, neither overutilizing
6:46
it and being prone to automation bias nor rejecting it summarily. Is so you
6:51
know, this is a fad and this isn't sticking around it. Actually,
6:55
if we prove that these models are beneficial, that I think we're ethically obligated
6:59
to it them. I think we're ethically obligated to at least consider the prediction.
7:03
So right now, when I order a cat scan, or when I
7:06
order a red blood cell count or any other test and I get it back,
7:11
I know how to integrate that. That's why I went to medical school.
7:14
That's what I've been doing for the last twenty something years is learning how
7:17
to integrate clinical information into decision making. But I'm going to suggest to you
7:24
that an AI model is very different from a blood test or from a cat
7:28
scan, and so knowing how to integrate that into clinical decision making, I
7:31
think is the point of greatest vulnerability for these models. When these models are
7:36
being trained, they're being validated, they're in a laboratory setting that they're not
7:42
prone to human frailty, they're not prone to human fallibility. And so when
7:48
we think about this is how this was intentioned, and we can validate it,
7:51
and we can even prove it to be beneficial, but then when we
7:55
deploy it, we lose control of that because human beings are going to is
8:00
going to be an information source that human beings can't not know. So if
8:03
we make a prediction and a clinician knows that prediction, the clinician has to
8:07
act on that in some way. I don't think it's justified just ignore it.
8:11
So I think that for clinicians that's the biggest thing, and maybe even
8:16
for medical students, you know, I think really helping medical students realize how
8:20
to integrate these models into their clinical decision making really at this level. And
8:28
then I think about for the patient, the perspective of the patient, So
8:31
what does AI mean for the patient? And there was the Pew Trust did
8:37
a study a couple of years ago, then they published it and they found
8:43
that sixty percent of people would not want AI to be involved in their care.
8:48
And when you drill down on that, it's really just independent. They
8:50
want to make sure there's a human that's part of the equation. They want
8:54
to make sure that a human is still part of the decision making, even
8:58
if they think, even if they think that AI may actually be more accurate,
9:03
they still want a human there. And I think it's because of the
9:07
ethics of the human brings. But vis a v my professional commitment. I
9:11
have an ethical obligation to the patient that the machine does not have. And
9:16
so I believe that that's largely why patients value having the human in the loop
9:20
because they can trust it. They can trust it in ways that they cannot
9:24
really trust a machine. Now, I think there are also a couple of
9:26
other groups to think about what's the most pressing issue. So I think for
9:31
our developers and our programmers, it's really thinking about how these models are going
9:35
to be used, not how they're intended, but thinking about all the ways
9:39
of which they may be used harmfully. I think about a model that predicts
9:45
patients at high risk for HIV so that you can put them on pre exposure
9:48
prophylaxis. Sounds great, but imagine a nefarious government getting that information, or
9:54
that information potentially getting the hands of a payer who doesn't want to ensure people
10:01
who are at high risk for HIV. So you can see how these models
10:05
can be and I think that it's it's up to our programmers and our developers
10:09
to think carefully about not everything that can be built needs to be built.
10:13
And then I think about the people who are making the decisions about whether to
10:18
use these models, whether to deploy these models. Those decisions are largely not
10:22
going to be made by clindicians exclusively. They're going to be made by health
10:26
systems, and part of a health system's consideration is whether there's an operational Is
10:33
there a business benefit to this model? And I think that's completely legit.
10:37
I think it's completely legit to consider is there an increased efficiency, is there
10:43
a decrease waste? What will this do? That actually has a business case.
10:48
But I think in order for the business case to have credibility, the
10:52
moral imperative is that you use it. To incorporate ethics into this. I
10:56
think it's not going to work. I think you're you're not going to get
11:00
a return on investment if you don't, if you don't make sure that all
11:05
the things that patients value, all the things that clinicians value, are built
11:09
into these models. And so in some ways, the moral imperative is also
11:13
the business case, because the people who are making the decisions want to return
11:18
on their investment, and the best way to get the return of the investment
11:20
is to engender trust and to teach people how to use these Let's pick up
11:24
the business model line for a bit you thought about because I haven't read the
11:28
book yet, and for those listening, everyone should buy the book when it's
11:31
out. The book is Encoding Bioethics, AI and Clinical decision Making by Charles
11:35
Binkley and Tyler loftis so business case, Charles, that might be the case
11:41
that there are different clusters of algorithms which different health systems could pick up and
11:46
some of our employ and might there be sort of a motivation to get the
11:52
cheaper cluster of algorithms which might give certain kinds of outputs as opposed to the
11:58
one that took a lot more sources with community engagement, polling and population responses
12:05
and patient preferences, are there different packages? Yeah, again a really great
12:09
question. And so I sort of link this to AI governance and decisions about
12:18
what models right now, there's you know, I was on a call recently
12:20
where a hospital leader said, I'm only going to We're only going to buy
12:26
models that are FDA approved. Well, most these models are not going to
12:28
be FDA approved, right They don't require FDA approval. Some people are going
12:33
to seek them. Some actually meet the criteria, but most actually have an
12:35
exemption. Most algorithms, most clinical decisions of poor systems are actually exempt from
12:41
FDA approval. It's not necessary. And so there are vendors out there who
12:46
are building models for all sorts of things, and there's not a huge amount
12:50
of guidance that well, there's not a lot of regulation, but there's also
12:54
not a lot of guidance for individual health systems to decide and to waigh the
13:00
models relative to each other, like you know this this one was trained on
13:03
a better data set, or or this one is able to be more valid,
13:07
or or this one actually has greater user interface or user user acceptability.
13:13
So they are all kinds. Nobody knows exactly how to compare these things right
13:18
now, and so I think people trying to do the best thing best they
13:22
can. I've worked within our own network on coming up with a with a
13:26
questionnaire that we're going to ask vengers to complete, and for me, it
13:31
helps me stratify the risk, like how what is the risk of harm?
13:37
And ifine, I define lack of benefit as a harm, right if we
13:41
buy if we buy a model that actually doesn't confer a clinical benefit, then
13:46
I consider that a harm and I think we have to think about that in
13:48
advance. So those are the sorts of things that I think about from a
13:52
business perspective. Other sort of tack on there is that I think there is
13:58
the risk that I will take large hospitals that are well funded with lots of
14:05
resources and make them even better. And again it's a little bit like that
14:09
lack of benefit is a harm. The hospitals that can't afford these models,
14:13
the hospitals that have less resources, fewer resources, the hospitals that don't have
14:18
sophisticated IT structures. I'm afraid they're going to be left out on the cold.
14:22
I'm afraid that they're going to continue. It's going to continue to create
14:26
a disparity in terms of quality of healthcare. Really interesting ethical theory in the
14:31
background here as well. I mean the points about sort of not conferring a
14:33
benefit being a harm, I mean that seems like a high bar. So
14:39
could you say a little bit more to unpack. I mean, I like
14:41
it. I think that this is a sort of a patient centered, strong
14:46
ethical focus. But I imagine in making the business case and conversations with health
14:50
systems, is that sort of line well received. Do folks think about harm
14:54
and benefit in that kind of way or is this an approach which you know
15:00
to be encoded, you know, into these algorithms such that we can really
15:03
produce the most ethical results when prompted, they do that. That's what I
15:09
would say, when prompted they do. When you talk about these things,
15:13
again, we have a very long tradition of medical ethics, bioethics. I
15:18
think when we're able to to sort of, you know, think about the
15:24
grander, the beauty, the elegance of an AI model, but then to
15:28
think about what are the normative expectations of the relationship into which it enters,
15:33
you know, what are the expectations of the of the clinician patient relationship,
15:37
and how does that model fit into it. So I think I think that
15:41
becomes in some ways the equalizer of thinking about, you know, how do
15:46
you integrate this model into a relationship and and not necessarily you know, thinking
15:54
about the loftiness of it, but thinking about kind of the basic human connection
15:58
and then how do you how do you how do you how do you think
16:02
about the AI model and that context, and so that that's sort of I
16:06
think. When you do that, then you start to think about the guiding
16:10
principles of harm and benefit. And when when folks are prompted, I think
16:17
there's this idea that that of risk and so what is risk? And so
16:22
I risk is a risk of something right, and so it could be a
16:26
risk of We don't typically think about risk as a benefit, but you know
16:30
the risk of some benefit. But we typically think about risk as the risk
16:33
of some kind of harm. And harms can occur in so many different ways
16:37
into so many different people. So when I think about harm for AI models,
16:42
I think about harm to the to the health system. If the AI
16:48
model isn't deployed well and patients lose trust. I think about harm to the
16:52
physician patient relationship or the clinician patient relationship, and I think about harm to
16:56
the patient, think about harm the clinician and world distress if the clinician doesn't
17:02
really know how to utilize the tool, or if the clinician disagrees with the
17:07
prediction, but the prediction is very valid, you can't really you can't really
17:11
reason with it. It's not explainable, as you mentioned earlier. So I
17:17
think that the harms and the benefit equation comes out when we start thinking about
17:26
what are the ethics, What are the ethics besides just you know, the
17:30
operational benefit, How will this benefit patients and is it a benefit that patients
17:37
desire, that patients want? I also think again tacking onto this. I
17:42
think this is the rule for participatory AI. I think we absolutely have to
17:47
include our patients and our families when we're thinking about these models, when we're
17:52
thinking about building these models and integrating them. I think it is absolutely imperative
17:56
because the user is the clinician, but the recipient is the patient, and
18:03
I think that that's an important distinction. The patient doesn't necessarily need to know
18:08
how the clinician integrated it and to their decision making, but they have to
18:14
be able to know it's there, and they have to they have to believe
18:17
that it's valuable, and we have to prove that it's valuable. What that
18:21
kind of sort of consideration of harm and benefit brings with it a burden on
18:26
whoever is adopting these systems, right, It's a higher ethical bar. One
18:32
picky practice question. Then I've got a big book question, because it's fun
18:34
to talk with people about new books. The picky practice question is, and
18:38
this is what some of our medical students raise all the time. And I
18:41
know you thought about the way when I show up to you to have,
18:45
you know, surgery of some kind, you can explain how things work in
18:49
the body to me, which I don't understand. You use non jargon and
18:52
all that sort of stuff, and it's great, and I say, yes,
18:55
Charles, please, you know, help me. Given the way some
18:57
of the information generated by these systems is either sort of not the pushball,
19:04
but is offered to physicians, can you explain that to me in the same
19:10
kind of way? And if not, how do you get around that?
19:12
Right? You can tell me the liver's doing this, the heart's doing that.
19:15
Sorry for my ludit approach this, But any help on sort of the
19:19
practical implications of the explainability problem in a minute before the big book question?
19:23
Yeah, again, a really wonderful question. So when I think about the
19:27
explainability issue, I think about disagreements. So if the model is beta prediction
19:33
that the clinician disagrees with, what is the basis for rejecting it? If
19:37
you just simply say I disagree with this, I think this is wrong.
19:41
The idea is that the model is going to be better than we are in
19:44
some ways, right, and so it's going to be able to make a
19:47
more accurate prediction. And so if we simply reject it because it's not ours,
19:51
then I don't know that that's the right, that's that's right. And
19:55
I think about mortality predictors, and I always talk about mortality predictors so that
19:57
I think there's lots of value laden considerations. But for instance, you know,
20:03
somebody, a twenty nine year old comes in with a femur fracture and
20:07
the mortality that their mortality prediction is, you know, eighty nine percent in
20:12
six months, and the clinicians like, this is a healthy twenty nine year
20:15
old with a femur fracture. This doesn't make any sense, and so they
20:19
disregard it. Well, then you find out that the reason for the femur
20:22
fractures of the is that the patient races motorcycles without helmets or you know,
20:27
something that's extraordinarily risky. But they rejected this out of They rejected this sort
20:33
of out of hand because they're like, oh, this doesn't make any sense.
20:37
So I think we have to be able to really carefully considered. I
20:40
think we're at an inflection point in terms of we know how to treat diseases
20:45
once they occur. We don't really understand what to do with a prediction.
20:48
So for another instances, a sepsis model, you know, if somebody is
20:53
predicted to be at high risk percepsis, do you start them on antibiotics?
20:57
Do you do you give them fluids? Do you monit to them carefully?
21:00
Like? What's the intervention? And we haven't really proven yet what intervention mitigates
21:06
the eventuality if the prediction is made. I think about that with post operative
21:08
wound infections. I'm working with a group on a model that would predict the
21:12
likelihood of post operative wound infections based on you know, do you take a
21:15
picture of the wound with your cell phone and uh? And then what do
21:18
you do if the patient's predicted to be at high risk? Do you open
21:22
the wound early? Do you start them on antibiotics? So I think these
21:25
are some of the unanswered questions about the explainability, particularly if there's disagreements,
21:30
that's helpful, that's helpful. The big book question what's not I mean it's
21:33
inappropriate, but what's not in the book? Right? What would you if
21:37
you if you could do it all over again? What was omitted? You
21:41
know? I think about this and it's actually maybe the segue into the next
21:45
book. I think about so a couple of things. One is I present
21:51
a very idealized view of the physician patient relationship. And I don't know if
21:56
that is the current model. I don't know if that's the future model.
22:02
I think it's been the past model. So I worry a little bit about
22:04
that omitting other considerations and the other thing that I wrote this exclusively from the
22:11
perspective of there will be a clinician in the loop. There will be a
22:15
clinician with an ethical obligation. That's kind of the premise is that you know
22:19
the physician. The clinicians ethical obligations essentially extend to this model if this model
22:27
is entering to the relationship of the AI model. But what if there's not
22:33
somebody with an ethical obligation or a moral obligation this in the loop. For
22:37
what I've really been thinking a lot about lately is how do we ensure how
22:42
do we build trust? Do we still need do we still need trust?
22:47
Right? Has trust moved from from the idea that I need to be able
22:55
to confide in you. I need to be able to know that you prioritize
23:00
my well being? Have we lost that? I mean, do patients even
23:04
assume that anymore? And is it a trade off? If a patient says,
23:08
look, you know this thing is going to be much more accurate,
23:11
it's more convenient, it's not grumpy, it doesn't it doesn't you know,
23:15
it had great bedside manner. And I'll take all those things, right,
23:21
I'll accept all those things. And I mean, the other thing that I
23:25
think about is that as the models keep getting better and better, and they
23:27
are getting better and better right now, most things those models still needs us,
23:33
but it's getting better and better. We're not evolving that fast, and
23:37
so as the models get better and better, it may come to a point
23:40
where they are better than we are, and we have to realize that.
23:44
Again, I think there is an obligation if the model performs better than humans
23:49
with greater benefit and less harm, and patients want it. Honestly, I
23:55
think we're a bit obligated to provide it, even if that means that we're
24:00
not in the loop. A potentially humbling experience and a great answer and a
24:06
good tease for the next book. So, Charles, thank you so much
24:08
for being with me today. It's been great to chat about this. Absolutely,
24:11
it's absolutely my pleasure. Brian, thank you so much for having me
24:15
on. And yes, please go out and buy the book. Thanks everyone,
24:18
for listening. I've been speaking with doctor Charles Binkley, author along with
24:23
Tyler Loftus of Encoding Bioethics AI in Clinical Decision Making, which is available I
24:30
believe shortly from uc Press again Encoding Bioethics AI and Clinical Decision Making. This
24:47
podcast series is a production of the Health Ethics Lab, and I am grateful
24:51
to colleagues in the Department of Medical Sciences, the IHS Library, and the
24:53
Hackensack I'm ready in the School of Medicine for making this collaboration possible. To
24:59
learn more about our work or to engage with the series content, reach out
25:03
to me Brian Pilkington at Brian dot Pilkington at hm h N dot org.