Latest / Health Ethics Podcast / When AI and Doctors Disagree: Rethinking Medical Decision-Making

Transcript

When AI and Doctors Disagree: Rethinking Medical Decision-Making · Health Ethics Podcast

• 0:09
  Speaker 1: Welcome to Health Ethics, a series of conversations with leading
• 0:13
  Speaker 1: experts about ethical considerations associated with health and healthcare, both
• 0:17
  Speaker 1: locally and globally. I am Brian Pilkington, Professor of Bioethics
• 0:21
  Speaker 1: in the Department of Medical Sciences at the Hackensack Meridian
• 0:24
  Speaker 1: School of Medicine. This series, like its predecessor, which focused
• 0:29
  Speaker 1: on the COVID-19 pandemic, relies on the idea that challenging
• 0:32
  Speaker 1: ethical issues are best addressed by folks from diverse backgrounds
• 0:37
  Speaker 1: practically reasoning together. and complements the Health Ethics Series with
• 0:42
  Speaker 1: its panel of leading experts from medicine, nursing, politics, economics, philosophy,
• 0:47
  Speaker 1: and the law. Thank you, and enjoy the episode.
• 0:59
  Speaker 2: It is my great pleasure to be speaking with Georgia Ponce,
• 1:02
  Speaker 2: who is a postdoctoral researcher at Delft University of Technology
• 1:06
  Speaker 2: in the Netherlands.
• 1:08
  Speaker 3: In her research, she mostly.
• 1:09
  Speaker 2: Inquires into topics that fall in the intersection between ethics
• 1:14
  Speaker 2: and epistemology of artificial intelligence, with a special interest in
• 1:18
  Speaker 2: medicine and healthcare as areas of application. She has particularly.
• 1:22
  Speaker 1: Focused on tackling forms.
• 1:24
  Speaker 2: Of injustice emerging due to the integration of AI systems
• 1:28
  Speaker 2: in medical practice and how these systems influence the patient-physician relationship.
• 1:34
  Speaker 2: More generally, she's interested in topics related to bioethics, epistemology,
• 1:38
  Speaker 2: and philosophy of science and their crossovers in connection with
• 1:41
  Speaker 2: novel challenges raised by AI systems. Georgia, thank you so
• 1:46
  Speaker 2: much for joining me today on the Health Ethics Podcast.
• 1:49
  Speaker 2: You've written a ton on disagreement and related topics on trust,
• 1:53
  Speaker 2: on mistrust, even on distrustworthiness. But let's start with your
• 1:57
  Speaker 2: recent analysis of the relationship between human physicians and AI.
• 2:01
  Speaker 2: And this will impress an A job and a great paper. Actually,
• 2:05
  Speaker 2: before we jump in there, we should take a step back.
• 2:07
  Speaker 2: What got you interested in questions of AI and disagreement?
• 2:12
  Speaker 4: Yeah, so, Ivan, thanks for having me. So, well, disagreement
• 2:18
  Speaker 4: is becoming more and more relevant since AI systems are
• 2:22
  Speaker 4: increasingly being introduced in medical practice. So I think what
• 2:26
  Speaker 4: got me interested into the topic, well, is twofold. So
• 2:30
  Speaker 4: on the one hand, there is a theoretical perspective. So
• 2:34
  Speaker 4: I've been interested for a while in questions revolving around
• 2:39
  Speaker 4: epistemic authority and the status, epistemic status that we can
• 2:43
  Speaker 4: attribute to AI systems. So when, if and when we're
• 2:46
  Speaker 4: justified or not justified in attributing epistemic authority to AI.
• 2:51
  Speaker 4: And if we look at what the philosophical debate on
• 2:54
  Speaker 4: the epistemology of disagreement has to offer as a starting point,
• 3:00
  Speaker 4: There we can see that most of the discussions start
• 3:02
  Speaker 4: from disagreements between epistemic peers. So basically individuals who have
• 3:07
  Speaker 4: more or less the same level of expertise in the
• 3:10
  Speaker 4: matter at hand or have access to the same amount
• 3:13
  Speaker 4: of evidence. And I think these debates are hard to
• 3:17
  Speaker 4: translate in contexts in which we have so different entities
• 3:24
  Speaker 4: between which the disagreements occur, like human clinicians and AI systems.
• 3:29
  Speaker 4: So I think it is theoretically, from a theoretical perspective,
• 3:32
  Speaker 4: very interesting. And there is a whole lot of work
• 3:36
  Speaker 4: to be done in this space. And also, of course,
• 3:41
  Speaker 4: there is the practical relevance. So how we deal with
• 3:44
  Speaker 4: disagreements has normative consequences and has also a bearing on
• 3:50
  Speaker 4: collaborations among professionals, as well as on patient-physician relationships. So
• 3:56
  Speaker 4: depending on how we deal with disagreements, while these impacts
• 4:00
  Speaker 4: are possibilities to exchange and provide reasons for decision-making, for example.
• 4:05
  Speaker 4: So these are the two dimensions that got me first
• 4:07
  Speaker 4: interested in the topic.
• 4:09
  Speaker 5: Really helpful background for a variety of reasons. One, conceptually
• 4:12
  Speaker 5: it works. And also among our host of listeners on
• 4:15
  Speaker 5: the podcast, we have a lot of medical students, future physicians.
• 4:18
  Speaker 5: And so sort of seeing how expert researchers got thinking
• 4:21
  Speaker 5: about problems in the backgrounds is sort of a useful
• 4:24
  Speaker 5: meta point for them too. So Let's turn to your
• 4:28
  Speaker 5: recent paper in A-Job. In it, you describe models for
• 4:31
  Speaker 5: helping us navigate the terrain of disagreement between clinicians and AI.
• 4:36
  Speaker 5: So for those who haven't read the paper yet, I
• 4:38
  Speaker 5: assume everyone who listens is going to go and read it.
• 4:40
  Speaker 5: Can you briefly outline the argument or just frame the
• 4:43
  Speaker 5: general idea for us?
• 4:47
  Speaker 2: Sure.
• 4:47
  Speaker 4: So usually in debates on AI clinician disagreement, the starting
• 4:52
  Speaker 4: point are cases in diagnostics. So relatively simple cases in
• 4:56
  Speaker 4: which let's say we have a CT scan, an AI
• 4:59
  Speaker 4: system detects an abnormality and the clinician in charge thinks
• 5:04
  Speaker 4: that this scan looks normal. So the question is, how
• 5:07
  Speaker 4: do we go about these kinds of disagreements where there
• 5:10
  Speaker 4: is a sort of misalignment between the AI's interpretation or
• 5:13
  Speaker 4: output and the clinician's preferred course of action? Now, while
• 5:18
  Speaker 4: looking at these cases, we noticed that this can lead
• 5:22
  Speaker 4: to take a quite binary approach on how to deal
• 5:26
  Speaker 4: with this situation. So as if the available options that
• 5:31
  Speaker 4: we have were to either follow the AI system or
• 5:35
  Speaker 4: to disregard it completely. And the idea behind our paper
• 5:39
  Speaker 4: is to challenge this and to show that to do
• 5:42
  Speaker 4: justice to the complexity of medical decision-making, which always occurs
• 5:46
  Speaker 4: under conditions of uncertainty and disagreements are a big part
• 5:49
  Speaker 4: of it, we need a more nuanced approach. So what
• 5:54
  Speaker 4: we basically did was to try and reconstruct what is
• 5:57
  Speaker 4: out there in the literature, and we could flesh out
• 5:59
  Speaker 4: three main accounts on how to deal with disagreements. And
• 6:03
  Speaker 4: we argued that none of them can satisfactorily deal with
• 6:07
  Speaker 4: cases that are a bit more complex. So we considered
• 6:11
  Speaker 4: a case in treatment, treatment recommendation case, and we showed that,
• 6:16
  Speaker 4: or this was our claim, that none of these cases
• 6:19
  Speaker 4: can satisfactorily, or this account can satisfactorily, an interpretation of disagreement,
• 6:24
  Speaker 4: a complete interpretation of disagreement. So again, we wanted to
• 6:28
  Speaker 4: paint a more complex picture in which we also consider
• 6:31
  Speaker 4: the possibility that maybe both the AI inclination might be
• 6:35
  Speaker 4: wrong or both might be partially right. And the disagreement
• 6:38
  Speaker 4: is not necessarily something that always needs to be resolved.
• 6:43
  Speaker 3: Great summary.
• 6:43
  Speaker 5: And again, I think folks will go and read the paper.
• 6:46
  Speaker 5: And the point about uncertainty is super helpful. So our listeners,
• 6:50
  Speaker 5: if they follow along regularly, will have just heard last week,
• 6:54
  Speaker 5: Alex John London sort of waxing philosophically about some concerns
• 6:58
  Speaker 5: of research ethics and AI. And the importance of uncertainty,
• 7:01
  Speaker 5: the concept of uncertainty played a large role in our
• 7:04
  Speaker 5: conversation last week and in our analysis. So you mentioned
• 7:08
  Speaker 5: sort of three different approaches. And in the paper, the
• 7:11
  Speaker 5: deference approach, the overruling approach, and a second opinion approach. Again,
• 7:16
  Speaker 5: for folks who haven't read it yet, could you just
• 7:19
  Speaker 5: briefly go through each of those so we have an
• 7:21
  Speaker 5: idea of how the argument progresses?
• 7:24
  Speaker 4: Yeah, sure. So the first one is, as you said,
• 7:27
  Speaker 4: the deference approach, the first one that we tackle. And
• 7:30
  Speaker 4: here the main idea is to say that, well, since
• 7:32
  Speaker 4: AI systems are what we could consider powerful knowledge-generating entities, because, well,
• 7:37
  Speaker 4: because they are trained on the knowledge of many experts
• 7:40
  Speaker 4: and they are often very accurate, in cases of disagreements,
• 7:43
  Speaker 4: we should defer to the AI systems output. Of course,
• 7:47
  Speaker 4: there are stronger and milder versions of this argument in
• 7:51
  Speaker 4: the literature, but basically the bottom line is that based
• 7:55
  Speaker 4: on purely epistemic reasons, so reasons of accuracy, The AI
• 7:59
  Speaker 4: is considered the epistemic authority and all things being equal
• 8:04
  Speaker 4: on this ground, we are justified or even required to
• 8:07
  Speaker 4: defer to its output in cases of disagreement. The next
• 8:11
  Speaker 4: one is the overruling approach, which basically argues the opposite
• 8:16
  Speaker 4: of the deferring approach. So we can find this approach
• 8:20
  Speaker 4: in a lot of AI guidelines that adopt a human
• 8:22
  Speaker 4: in the loop approach. basically stating that humans should always
• 8:26
  Speaker 4: be able to eventually override the AI systems when necessary.
• 8:31
  Speaker 4: And this can sound attractive, of course, because it allows
• 8:35
  Speaker 4: us to eradicate responsibility gaps to a certain extent. So
• 8:38
  Speaker 4: if clinicians remain in control, they can provide patients with reasons.
• 8:43
  Speaker 4: And this can also, as a consequence, lead to higher
• 8:46
  Speaker 4: patient acceptance. But as we point out in the paper,
• 8:50
  Speaker 4: also this approach has some issues. But it's the complementary one,
• 8:54
  Speaker 4: let's say, to the deference approach. And the third one
• 8:58
  Speaker 4: is the second opinion approach. It has been offered in
• 9:02
  Speaker 4: a paper that has been published a couple of years
• 9:04
  Speaker 4: ago by Kempt and Nagel. And what these authors do
• 9:08
  Speaker 4: is they offer a so-called rule of disagreement to deal
• 9:11
  Speaker 4: with the cases of disagreements that I mentioned before. So
• 9:15
  Speaker 4: they say if the diagnosis of an AI system is
• 9:18
  Speaker 4: not aligned with that of a human clinician, then the
• 9:21
  Speaker 4: best way to go about this disagreement is to require
• 9:24
  Speaker 4: a second opinion from another clinician, who is then supposed
• 9:29
  Speaker 4: to settle the disagreement. So ultimately, what we want to
• 9:33
  Speaker 4: do in the paper is to argue against some of
• 9:34
  Speaker 4: the common assumptions of this account, particularly the fact that
• 9:38
  Speaker 4: disagreements need to be always resolved by epistemic authority, be
• 9:42
  Speaker 4: it epistemic authority of the AI system if we follow
• 9:45
  Speaker 4: the deference approach, or of a clinician if we follow
• 9:50
  Speaker 4: the overruling approach, or a second possible human clinician if
• 9:54
  Speaker 4: we follow the second opinion approach. But we actually wanted
• 9:59
  Speaker 4: to point out that there are cases in which disagreements
• 10:03
  Speaker 4: can be unproblematic or can have even some value and
• 10:09
  Speaker 4: we do not need to overcome them entirely.
• 10:12
  Speaker 5: Yeah, so that's great and helpful. And I may want
• 10:15
  Speaker 5: to return to the human in the loop feature because
• 10:17
  Speaker 5: that seems to be a powerful rhetorical point lots of
• 10:19
  Speaker 5: folks make. They say, oh, well, I would feel, I've heard,
• 10:22
  Speaker 5: I will feel comfortable if, and it usually follows with
• 10:26
  Speaker 5: some version of the human in the loop. I'm always
• 10:30
  Speaker 5: curious what human people would want in the loop. Is
• 10:33
  Speaker 5: it the medical expert? Is it the disagreement expert? Is
• 10:38
  Speaker 5: it someone who knows the AI systems? But let's put
• 10:40
  Speaker 5: that to the side because it, you just sort of
• 10:43
  Speaker 5: teased it perfectly well. And it's a really, really nice
• 10:45
  Speaker 5: point in the paper that disagreement can be productive. And
• 10:48
  Speaker 5: so it connects up with folks who know the philosophical
• 10:50
  Speaker 5: literature on sort of democratic theory and process of reasonable
• 10:53
  Speaker 5: disagreement and all that. And so disagreement on your view
• 10:57
  Speaker 5: can force a revision of views and lead to more
• 11:00
  Speaker 5: accurate outcomes. So could you say more about those kinds
• 11:04
  Speaker 5: of cases, what you're thinking about and sort of when,
• 11:07
  Speaker 5: you know, like the AI and physician might, both be
• 11:10
  Speaker 5: right or how together they can come to a more
• 11:12
  Speaker 5: accurate outcome.
• 11:14
  Speaker 2: Yes.
• 11:15
  Speaker 4: So we wanted to make this a centerpiece of our paper.
• 11:19
  Speaker 4: So what we wanted to argue against, as I mentioned before,
• 11:22
  Speaker 4: is to argue against absolutes. So whether following the AI
• 11:26
  Speaker 4: or rejecting it with nothing in between. And we think
• 11:28
  Speaker 4: that disagreement can be productive and valuable in many respects.
• 11:32
  Speaker 4: So I mentioned before that we discuss a case in
• 11:35
  Speaker 4: the paper And maybe I can just briefly illustrate that
• 11:39
  Speaker 4: so that it might be also easier to follow the argument.
• 11:42
  Speaker 4: So we consider a case of pulmonary embolism, so a
• 11:47
  Speaker 4: treatment recommendation. So we want basically to consider a hypothetical
• 11:52
  Speaker 4: AI system that has been introduced in medical care and
• 11:56
  Speaker 4: has been used for, let's say, five years and reliably.
• 12:01
  Speaker 4: And we also have the recommendation from a human clinician. Now,
• 12:05
  Speaker 4: we know that pulmonary embolism can be quite risky, right?
• 12:09
  Speaker 4: So we imagine the situation of this patient who comes
• 12:13
  Speaker 4: to the ER with a CT scan that shows a
• 12:17
  Speaker 4: small pulmonary embolism. Now, of course, acute pulmonary embolism has
• 12:22
  Speaker 4: a very high mortality if not treated. But in this
• 12:25
  Speaker 4: particular case, evidence is mixed. And there are not very
• 12:29
  Speaker 4: clear guidelines on how to go about this with treatment.
• 12:33
  Speaker 4: So there are a lot of options to choose from
• 12:36
  Speaker 4: that are open to physicians' discretion. So one could go,
• 12:40
  Speaker 4: for example, with anticoagulation treatment, let's say block thinners, or
• 12:45
  Speaker 4: in more serious cases with thrombolytic treatment or with no
• 12:48
  Speaker 4: treatment at all and monitoring of the patient. So now
• 12:52
  Speaker 4: in the particular case that we analyze in the paper,
• 12:56
  Speaker 4: We assume that the AI system suggests full-course anticoagulation treatment,
• 13:00
  Speaker 4: while the clinician suggests closed monitoring and no treatment. How
• 13:07
  Speaker 4: do we go about this? It's a bit more nuanced
• 13:10
  Speaker 4: case than the ones in diagnostics that are often considered
• 13:14
  Speaker 4: in the literature. If we consider this case according to
• 13:18
  Speaker 4: the deference or overruling approach, then we would either go
• 13:23
  Speaker 4: with what the eye is suggesting or with what the
• 13:25
  Speaker 4: physician is suggesting, right? However, we see the value of disagreement,
• 13:29
  Speaker 4: and here I come back to your question, Brian, in
• 13:31
  Speaker 4: the fact that having this disagreement could open up new
• 13:36
  Speaker 4: possibilities for the physician in the sense that they could,
• 13:39
  Speaker 4: for example, include the eye system's output into their reasoning
• 13:43
  Speaker 4: and without discarding it completely, but also without insisting on
• 13:47
  Speaker 4: remaining of their original opinion. So for example, in this
• 13:52
  Speaker 4: particular case, the clinician could think that a short course
• 13:55
  Speaker 4: anticoagulation treatment would be the best possible course of action
• 14:00
  Speaker 4: for this particular patient. And so probably what we argue
• 14:05
  Speaker 4: is that probably the validity of this option would not
• 14:07
  Speaker 4: have been immediately clear had the disagreement not emerged. So
• 14:12
  Speaker 4: this shows that disagreements, for example, about evidentiary sources can
• 14:17
  Speaker 4: be productive and can also signal reasonable uncertainty and make
• 14:22
  Speaker 4: us behave somehow more cautiously. And it can also, in
• 14:26
  Speaker 4: trying to navigate it, it can also foster our critical
• 14:29
  Speaker 4: thinking abilities. And it is unavoidable, as I was saying before.
• 14:34
  Speaker 4: And since there is, again, no objective baseline for the
• 14:38
  Speaker 4: acceptabilities of acceptability of which risks in medicine. So that's
• 14:42
  Speaker 4: why we wanted to point out through this case that
• 14:45
  Speaker 4: the picture is more complex and that there is indeed
• 14:48
  Speaker 4: some value to disagreement.
• 14:50
  Speaker 5: The case is, it's a super helpful illustration. I think
• 14:53
  Speaker 5: sort of powerfully shows that disagreement can be productive. So
• 14:57
  Speaker 5: let's think a little bit more with you about disagreement.
• 15:00
  Speaker 5: Are there prerequisites for resolving disagreement and anticipating that disagreement
• 15:05
  Speaker 5: might be resolved?
• 15:06
  Speaker 3: Because of aren't.
• 15:08
  Speaker 5: And though it's a nice thing if some is being productive,
• 15:11
  Speaker 5: many aren't. So in person-to-person interactions, right, so disagreements lots
• 15:15
  Speaker 5: of us are more familiar with, we might think that
• 15:17
  Speaker 5: there are certain persons or persons with certain character traits
• 15:21
  Speaker 5: which would more easily work together or more easily resolve disagreement, right?
• 15:24
  Speaker 5: So we may have certain kinds of structures, and maybe
• 15:27
  Speaker 5: they're bad, maybe there may be sort of ethical problems
• 15:29
  Speaker 5: with some of these structures, but maybe it's just some.
• 15:31
  Speaker 3: Kind of hierarchical deference, right?
• 15:33
  Speaker 5: So for the students listening, The surgeon about to operate
• 15:36
  Speaker 5: has just been operating longer or has a more impressive title,
• 15:40
  Speaker 5: is a more senior clinician, or appeals to experience. That
• 15:43
  Speaker 5: particular surgeon, when there's some interoperative disagreement, they've been doing
• 15:49
  Speaker 5: more of that kind of surgery or care. Or there's
• 15:52
  Speaker 5: a special relationship, right? So Georgia, you're my doctor. And
• 15:55
  Speaker 5: even though someone else might have been around longer, we
• 15:57
  Speaker 5: have that trusting relationship. So I can see how this
• 16:02
  Speaker 5: works with person-to-person interactions. that doesn't seem to translate to
• 16:05
  Speaker 5: AI human disagreement. So can you help us think through that?
• 16:10
  Speaker 2: Yeah.
• 16:11
  Speaker 4: Yeah, no, this is a great question. And it's indeed correct.
• 16:14
  Speaker 4: And also why disagreement, I think, between so fundamentally different
• 16:19
  Speaker 4: entities like AIs and human clinicians is particularly challenging. So
• 16:24
  Speaker 4: one could try and come up with a sort of
• 16:27
  Speaker 4: rule to follow in cases of AI clinician AI disagreement,
• 16:31
  Speaker 4: and this is in fact what the second opinion approach proposes,
• 16:35
  Speaker 4: so a rule of disagreement. However, as I mentioned before,
• 16:38
  Speaker 4: also this comes with some downsides. For example, in terms
• 16:42
  Speaker 4: of practical feasibility, if we think about the pulmonary embolism
• 16:47
  Speaker 4: case that I explained earlier, well, there we want to
• 16:50
  Speaker 4: act quickly. So asking for a second opinion, for example,
• 16:54
  Speaker 4: would not always be feasible. And also, it's not really
• 16:58
  Speaker 4: clear whether, for example, a second opinion could help us
• 17:01
  Speaker 4: settle the disagreement. It could even add more depth to
• 17:04
  Speaker 4: the disagreement itself by suggesting, for example, a completely different
• 17:08
  Speaker 4: course of action. So that's why we argue that dealing
• 17:12
  Speaker 4: with disagreements in these cases requires ad hoc considerations and
• 17:18
  Speaker 4: contextualization case by case. So even if it might be
• 17:22
  Speaker 4: possibly desirable to have a general rule to follow, I
• 17:25
  Speaker 4: think there are downsides to it. Ultimately, for this reason,
• 17:28
  Speaker 4: we want to refrain from providing a sort of general
• 17:31
  Speaker 4: heuristic on how to resolve disagreement, but rather really highlight
• 17:36
  Speaker 4: the complexity of AI-supported decision-making. So we just need to,
• 17:42
  Speaker 4: I think, accept also to a certain extent that AI
• 17:44
  Speaker 4: systems provide new information to the clinical decision-making process And
• 17:49
  Speaker 4: depending on the context, physician knowledge, patient preferences, it must
• 17:54
  Speaker 4: be taken into account. But I don't think we can
• 17:57
  Speaker 4: go with a standardized rule because it would not do
• 17:59
  Speaker 4: justice to this complexity.
• 18:01
  Speaker 3: That's really helpful.
• 18:02
  Speaker 5: And so for the future physicians, the physicians and training
• 18:06
  Speaker 5: health professionals in training who listen to the podcast, as
• 18:09
  Speaker 5: they think and look forward about their impending relationships, I mean,
• 18:13
  Speaker 5: that's too robust, but about their impending interactions with AI,
• 18:17
  Speaker 5: and they reasonably expect they're going to disagree. And in
• 18:21
  Speaker 5: lots of instances, not understand sort of some of the
• 18:23
  Speaker 5: features of the output of the algorithm. As they train
• 18:27
  Speaker 5: to care for patients, how should they be thinking about
• 18:30
  Speaker 5: AI in the future?
• 18:30
  Speaker 3: Do you have any recommendations?
• 18:32
  Speaker 5: And even as folks who teach them, right, from medical educators,
• 18:36
  Speaker 5: are there character traits in future physicians that we should
• 18:40
  Speaker 5: seek to inculcate? Like, Is there any advice you have
• 18:43
  Speaker 5: for folks on helping to train future physicians so that
• 18:46
  Speaker 5: they're good fits for resolving disagreement and using AI to
• 18:52
  Speaker 5: its fullest potential for patient betterment?
• 18:55
  Speaker 4: Yeah, so I can think of two complementary ways of
• 18:58
  Speaker 4: going about this. So I think going about situations of
• 19:03
  Speaker 4: disagreement within AI system with an open mind might be
• 19:05
  Speaker 4: an important feature to cultivate in terms of epistemic humility.
• 19:10
  Speaker 4: So to allow for that kind of critical space where
• 19:14
  Speaker 4: we acknowledge the fact that AI systems might contribute to
• 19:16
  Speaker 4: a perspective that was just not envisioned at the beginning,
• 19:19
  Speaker 4: for example. So instead of insisting on sticking on one's
• 19:22
  Speaker 4: own original position and attitude of epistemic humility, in recognizing
• 19:28
  Speaker 4: one's own possible limitations might be important. On the other hand, though,
• 19:33
  Speaker 4: we know that AI systems are everything but epistemically humble.
• 19:37
  Speaker 4: So they provide us with information very often without mechanisms
• 19:42
  Speaker 4: of uncertainty quantification or the acknowledgement of errors in place. So, um,
• 19:50
  Speaker 4: this might lead us to very dangerous over-reliance. And so
• 19:55
  Speaker 4: this is, I think, a risk that is particularly present
• 19:58
  Speaker 4: for young professionals who are likely in the process of
• 20:01
  Speaker 4: building their confidence on the job and might feel like
• 20:04
  Speaker 4: they have to defer to what an advanced technology company
• 20:08
  Speaker 4: is telling them to do, even if it contradicts their
• 20:10
  Speaker 4: own judgment. So I think hitting the right spot between
• 20:13
  Speaker 4: these two tendencies is difficult, but important. So we should
• 20:19
  Speaker 4: focus more and more research efforts on how to facilitate
• 20:23
  Speaker 4: this and also how to make it conceptually clear. So, yeah.
• 20:28
  Speaker 5: I mean, that's super helpful and sort of some kind of,
• 20:32
  Speaker 5: I don't know, Aristotelian mean between the sort of critical
• 20:36
  Speaker 5: appraisal of these outputs, but also, I mean, I really
• 20:39
  Speaker 5: appreciate the point about epistemic humility, and that might even
• 20:42
  Speaker 5: be what guides future physicians and interacting with patients when
• 20:46
  Speaker 5: they can't fully explain certain kinds of things. So that's helpful.
• 20:50
  Speaker 5: Can I push you to recommendations for medical educators or
• 20:56
  Speaker 5: anyone involved in health professions education about training for a
• 21:00
  Speaker 5: future with a significant.
• 21:01
  Speaker 3: Involvement in medical AI?
• 21:03
  Speaker 5: So when I teach students, Are there character traits I
• 21:05
  Speaker 5: should be thinking about? Any advice given your expertise and
• 21:09
  Speaker 5: disagreement in AI that I should have in mind when
• 21:12
  Speaker 5: I teach my students?
• 21:13
  Speaker 4: So I think what is really important is to provide
• 21:16
  Speaker 4: students with a very realistic outlook on what these systems
• 21:20
  Speaker 4: can and cannot do. So, and look beyond absolute. So
• 21:24
  Speaker 4: very often in public perception, AI is depicted as either
• 21:27
  Speaker 4: the way to go that will replace doctors, make them obsolete,
• 21:30
  Speaker 4: or something that will destroy us. So I don't think
• 21:33
  Speaker 4: that these views are appropriate or realistic. And views that
• 21:39
  Speaker 4: in general antagonize clinician NAI systems, I think, fuel rhetoric
• 21:44
  Speaker 4: that can be counterproductive. And the aim of our paper
• 21:49
  Speaker 4: was exactly to bring more facet to this issue and
• 21:52
  Speaker 4: show its complexity. And I think for medical educators and
• 21:55
  Speaker 4: other medical professionals, it is important to be offered ways
• 21:58
  Speaker 4: to gain the necessary literacy to interact with AI systems
• 22:03
  Speaker 4: in productive ways in their medical practice. And it is also,
• 22:08
  Speaker 4: I think, worth noting that AI systems outputs almost always
• 22:11
  Speaker 4: need to be contextualized against other relevant information and values
• 22:17
  Speaker 4: and that standalone uses of AI systems are important. the
• 22:22
  Speaker 4: exception rather than the rule. So having a more realistic
• 22:26
  Speaker 4: idea of what AI can and cannot do can help
• 22:29
  Speaker 4: also manage expectations and indeed promote a more productive collaboration.
• 22:36
  Speaker 4: So as I see it, one of the biggest challenges
• 22:39
  Speaker 4: to preserve one's own critical scrutiny and avoid over-reliance on
• 22:44
  Speaker 4: these systems. And for this, we need to strive towards
• 22:49
  Speaker 4: creating epistemic environments that preserve this critical scrutiny and do
• 22:54
  Speaker 4: not put clinicians in a moral corner if they have
• 22:57
  Speaker 4: the feeling they have to stick to what the AI
• 22:59
  Speaker 4: is recommending them to do.
• 23:03
  Speaker 5: Super helpful and dovetails nicely. I mean, this is very self-interested,
• 23:07
  Speaker 5: but I've been pushing students to just read more and
• 23:10
  Speaker 5: study more philosophy because I think that will help them
• 23:12
  Speaker 5: with that kind of critical engagement. So we'll We'll throw
• 23:15
  Speaker 5: this in the bucket of points in favor of philosophy
• 23:18
  Speaker 5: for medical education. So really, really helpful. Occasionally, especially when
• 23:25
  Speaker 5: you're teaching in health professions contexts, folks are concerned about
• 23:30
  Speaker 5: the future of the profession. And so that's a huge question.
• 23:34
  Speaker 5: And this is utterly unfair to you. I'm asking you
• 23:36
  Speaker 5: to take out a crystal ball and prognosticate. But one
• 23:39
  Speaker 5: less talked about way to resolve disagreement is to simply
• 23:42
  Speaker 5: choose one side. Maybe this is sort of an extreme
• 23:44
  Speaker 5: version of your deference option. And so physicians and other
• 23:48
  Speaker 5: health professionals might just defer regularly to an AI, right?
• 23:52
  Speaker 5: And maybe you've answered this, but let's just suppose lots
• 23:55
  Speaker 5: of physicians do this. They just keep deferring to the
• 23:58
  Speaker 5: relevant algorithm. Do you see a world in which physicians
• 24:04
  Speaker 5: or other health professions begin to disappear? Might they just
• 24:08
  Speaker 5: have to reimagine their work in different ways? How do
• 24:13
  Speaker 5: you think about humans in the future in some of
• 24:16
  Speaker 5: these task spaces where given the kind of collaboration that
• 24:19
  Speaker 5: might occur, and even if it needs to be contextualized,
• 24:22
  Speaker 5: the AI outputs are just, it's just, it's so good.
• 24:25
  Speaker 3: We maybe don't need people anymore.
• 24:27
  Speaker 5: Help us think through a case where given the, a
• 24:30
  Speaker 5: lot of deference or the overreliance, right?
• 24:33
  Speaker 3: To borrow the term you used.
• 24:35
  Speaker 5: The overreliance on AI leads to whether it's de-skilling or
• 24:39
  Speaker 5: it just AI does better than physicians. Do we have
• 24:42
  Speaker 5: physicians in the future?
• 24:45
  Speaker 4: Yeah, so I would find a scenario in which we
• 24:48
  Speaker 4: stick to the deference approach consistently quite concerning, obviously. Not
• 24:53
  Speaker 4: really because I realistically think that AI systems might replace
• 24:57
  Speaker 4: medical professionals, but rather... Yeah, for the risk of what
• 25:01
  Speaker 4: you were also referring to as automation bias. So this
• 25:04
  Speaker 4: sort of a critical over-reliance on AI in which we
• 25:09
  Speaker 4: suspend our critical scrutiny. So, and this would be problematic
• 25:13
  Speaker 4: because if we could follow an AI output that ends
• 25:16
  Speaker 4: up being harmful for patients, for example. And another concern
• 25:21
  Speaker 4: might be that clinicians might become less sensitive to other
• 25:24
  Speaker 4: sources of evidence that exceed what the AI system is
• 25:28
  Speaker 4: providing us with. For example, patient's testimony of their lived
• 25:32
  Speaker 4: experience of illness that might not be aligned with the
• 25:35
  Speaker 4: AI system's output. So we need to create epistemic environments
• 25:39
  Speaker 4: in which clinicians still remain sensitive to these other important
• 25:44
  Speaker 4: forms of evidence. And I also think, and this might
• 25:48
  Speaker 4: be a challenge in the future, there are also other
• 25:51
  Speaker 4: forms of knowledge that are not easily quantifiable but very important,
• 25:59
  Speaker 4: for example, clinicians' tacit knowledge or their trained gut feelings.
• 26:04
  Speaker 4: And we need to make space in medical interactions with
• 26:07
  Speaker 4: AI for these forms of knowledge so that we can
• 26:11
  Speaker 4: have good patient-physician relationships. And also a concern would be
• 26:17
  Speaker 4: to try and remain answerable to patients. So if we
• 26:21
  Speaker 4: had to justify our decision-making to patients, for example, in
• 26:25
  Speaker 4: situations of emergency, by only referring to the eye system's
• 26:28
  Speaker 4: output in a sort of computer said so fashion, this
• 26:33
  Speaker 4: would fundamentally change the physician-patient relationship and also the communicative
• 26:39
  Speaker 4: practice of exchanging reasons that underlies it. So this is
• 26:45
  Speaker 4: something that we definitely need to put effort into preserving.
• 26:49
  Speaker 4: And so I want to clarify that the deference approach
• 26:53
  Speaker 4: that we refer to in our paper the authors are
• 26:57
  Speaker 4: equally concerned with the ethical consequences of deferring to AI systems.
• 27:01
  Speaker 4: For example, the risk of falling back to paternalistic patterns
• 27:04
  Speaker 4: of care. But our point is that even only epistemically speaking,
• 27:10
  Speaker 4: so if we only consider the accuracy of the AI system,
• 27:14
  Speaker 4: also in that case, it would not probably be the
• 27:16
  Speaker 4: best thing to do to just follow through with what
• 27:18
  Speaker 4: the AI system is recommending us. So it's even this
• 27:21
  Speaker 4: more circumscribed claim that we would like to challenge in
• 27:25
  Speaker 4: our paper because accuracy is important, but it is just
• 27:29
  Speaker 4: one datum that needs to be contextualized against other important
• 27:33
  Speaker 4: considerations related to patient's values, for example.
• 27:39
  Speaker 3: Excellent. Really helpful.
• 27:41
  Speaker 5: And I'm more of, well, it's good to know or
• 27:46
  Speaker 5: to have a paper and a conversation which highlights the
• 27:48
  Speaker 5: complexity of this landscape because it's so often just, you know,
• 27:51
  Speaker 5: as you noted before, sort of pushed into the two binaries.
• 27:56
  Speaker 3: And so it's really helpful and good.
• 27:57
  Speaker 5: Especially for the future physicians and health professionals who listen
• 28:02
  Speaker 5: to this podcast. Now, they may still have jobs and
• 28:04
  Speaker 5: opportunities to care for patients in the future. Before I
• 28:08
  Speaker 5: let you go, because I see we're running short on time,
• 28:11
  Speaker 5: for folks who listen to this, obviously, they're going to
• 28:13
  Speaker 5: go and read the paper if they haven't. But what's
• 28:17
  Speaker 5: next for you? Is there a big talk, paper, book,
• 28:19
  Speaker 5: What's coming out? What can we turn listeners to if
• 28:22
  Speaker 5: they're interested in learning more from you?
• 28:25
  Speaker 4: So I have some ongoing research on topics related to
• 28:29
  Speaker 4: this paper. But so earlier this year, I co-edited a
• 28:33
  Speaker 4: volume that was published in Cynthia's library at the beginning
• 28:38
  Speaker 4: of the year. So still quite fresh on the philosophy
• 28:41
  Speaker 4: of science for machine learning. And also there we have
• 28:44
  Speaker 4: also interesting content on machine learning in medicine. So I
• 28:48
  Speaker 4: think that would be probably interesting for our listeners.
• 28:53
  Speaker 3: That's great.
• 28:54
  Speaker 5: And we'll get all the information, links for citations and
• 28:56
  Speaker 5: everything else on our show notes page. So folks who
• 29:00
  Speaker 5: are listening, you can pick it up there. Georgia, thanks
• 29:04
  Speaker 5: so much. This has been terrific.
• 29:05
  Speaker 4: Thanks a lot for having me.
• 29:06
  Speaker 1: This series is a production of the Health Ethics Lab,
• 29:09
  Speaker 1: and I am grateful to colleagues in the Department of
• 29:11
  Speaker 1: Medical Sciences, the IHS Library, and the Hackensack Meridian School
• 29:15
  Speaker 1: of Medicine for making this collaboration possible.
• 29:18
  Speaker 3: To learn more about our work or to engage with
• 29:21
  Speaker 3: the
• 29:21
  Speaker 1: Series' content, reach out to me, Brian Pilkington, at brian.pilkington
• 29:26
  Speaker 1: at hmhn.org.