Latest / Code & Cure / #51 - The Autonomy Illusion

Transcript

#51 - The Autonomy Illusion · Code & Cure

• 0:00
  SPEAKER_02: If an algorithm always knows your next best move, are you still the one making decisions?
• 0:16
  SPEAKER_00: Hello and welcome back to Coding Cure, where we discuss decoding health in the age of AI.
• 0:21
  SPEAKER_00: My name is Vasant Sarathi.
• 0:23
  SPEAKER_00: I'm a cognitive scientist and AI researcher.
• 0:26
  SPEAKER_00: I'm with Laura Hagobian.
• 0:28
  SPEAKER_02: I'm an emergency medicine physician.
• 0:31
  SPEAKER_02: And uh last week's topic was like so exciting.
• 0:35
  SPEAKER_02: We were celebrating so many great things about AI.
• 0:38
  SPEAKER_02: And I feel like this week's topic is like the opposite.
• 0:40
  SPEAKER_02: It's like, let's take, let's take it down a notch.
• 0:43
  SPEAKER_02: Let's take it down a notch a little bit.
• 0:44
  SPEAKER_02: Because today we're going to be talking about this concept of like autonomy, artificial autonomy, how algorithms can actually be paternalistic and and how AI may actually reduce our autonomy, which is, I don't know, it's kind of like scary to think about.
• 1:01
  SPEAKER_00: Yeah.
• 1:01
  SPEAKER_00: And I think some of us feel it, right?
• 1:03
  SPEAKER_00: When we use AI systems for when we ask Chat GPT questions, we're offloading some of that thinking onto it.
• 1:08
  SPEAKER_02: And, you know, and we ask it to make like decisions for us sometimes, right?
• 1:13
  SPEAKER_02: Or we'll we'll ask it a question and it'll make like a clear recommendation.
• 1:17
  SPEAKER_02: And sometimes I just do it, especially because it sounds so so confident in what that what that decision is.
• 1:23
  SPEAKER_00: And scarily enough, sometimes you might say, okay, I don't want to just make a decision.
• 1:27
  SPEAKER_00: Let's talk about it.
• 1:28
  SPEAKER_00: And you have a conversation with it, and then you arrive at the decision.
• 1:31
  SPEAKER_00: And it's it it might, you know, you have to realize that you feel like you've gone through this process of deliberation and thinking with a really smart person uh through the problems, and you arrive at a decision, and you're struck you're even more confident of the decision, and you've used an AI system to help you get there.
• 1:48
  SPEAKER_00: And is that, you know, sufficiently you should you be sufficiently more confident than you were before?
• 1:55
  SPEAKER_02: It's interesting because like sometimes I'll debate things with you know another human.
• 1:59
  SPEAKER_02: Oh, what's the best path forward?
• 2:00
  SPEAKER_02: Here, like a friend or whatever.
• 2:02
  SPEAKER_02: Here are some options, right?
• 2:03
  SPEAKER_02: A friend or a family member.
• 2:05
  SPEAKER_02: And I don't know that I would see AI as as being the same way, right?
• 2:10
  SPEAKER_02: But at the same time, these LLMs are like always available at 11 p.m.
• 2:16
  SPEAKER_02: or 2 a.m.
• 2:16
  SPEAKER_00: or whatever.
• 2:17
  SPEAKER_00: And they put things you don't know, right?
• 2:19
  SPEAKER_00: At least like there's topics, maybe you ask it a question about things you are uncertain about.
• 2:24
  SPEAKER_02: Right.
• 2:25
  SPEAKER_02: And it comes back with a sure answer, right?
• 2:28
  SPEAKER_00: Because or it comes back with what seems like if forward progress in your judgment process, right?
• 2:34
  SPEAKER_00: Right.
• 2:34
  SPEAKER_00: Yeah.
• 2:34
  SPEAKER_02: And if you're talking to a friend or family member, it may there still may be uncertainty.
• 2:39
  SPEAKER_01: Yeah.
• 2:40
  SPEAKER_02: And you know, sometimes the path isn't linear forward.
• 2:44
  SPEAKER_01: Yeah.
• 2:44
  SPEAKER_02: And that's normal, right?
• 2:46
  SPEAKER_02: That's actually I actually think that's a good thing.
• 2:48
  SPEAKER_02: There's not always a straight and clear answer to every question, even though the LLMs sort of make you feel like there is.
• 2:56
  SPEAKER_00: Yeah.
• 2:56
  SPEAKER_00: I mean, so this comes down to decision making, right?
• 2:59
  SPEAKER_00: With the help of AI systems and their role in your decision making.
• 3:03
  SPEAKER_00: And, you know, we've talked about this in the in many different settings, you know, a human in the loop and things like that.
• 3:09
  SPEAKER_00: But this is more direct.
• 3:11
  SPEAKER_00: This is more about your own personal decision making.
• 3:14
  SPEAKER_00: And it and it shows up in the in in in in healthcare and medical applications where you have to make decisions about what you want to do next, right?
• 3:20
  SPEAKER_02: Right.
• 3:20
  SPEAKER_02: I mean, it shows up everywhere, right?
• 3:22
  SPEAKER_02: And it the question is like, are we as providers, are we as patients giving up some of this autonomy by allowing AI to do some of these tasks for us?
• 3:32
  SPEAKER_02: Yes.
• 3:32
  SPEAKER_02: Are we like let like I want to be in control, right?
• 3:38
  SPEAKER_00: I mean, we've given up control in so many different aspects of our lives, right?
• 3:41
  SPEAKER_00: You know, we've we have so many machines in our lives that do different things.
• 3:45
  SPEAKER_02: I guess that I'm like, there are certain things maybe I don't care to be in control of, like uh, you know, certain operational tasks where I'm like, okay, the machine can do that.
• 3:54
  SPEAKER_02: But if something's like a big life decision, I want to feel like I have the practice and skills to come up with an answer.
• 4:02
  SPEAKER_02: And then I've like made the decision myself.
• 4:05
  SPEAKER_01: Yeah.
• 4:06
  SPEAKER_01: Yeah.
• 4:06
  SPEAKER_01: Yeah.
• 4:07
  SPEAKER_02: That that something else hasn't made the decision for me.
• 4:09
  SPEAKER_02: And I I think sometimes with AI, it makes you feel like you've made the decision yourself, even though it's like nudged you in some sort of direction.
• 4:18
  SPEAKER_00: Yeah, yeah, yeah.
• 4:19
  SPEAKER_00: For sure.
• 4:19
  SPEAKER_00: For sure.
• 4:20
  SPEAKER_00: And I think, you know, we found we found this article that kind of unpacks some of the ideas here around uh what they're calling AI paternalism.
• 4:30
  SPEAKER_00: And I think it's very fascinating to dig deeper into that.
• 4:33
  SPEAKER_02: Yeah.
• 4:33
  SPEAKER_02: I mean, we can go through some of these concepts.
• 4:36
  SPEAKER_00: Um It's a bit theoretical, just to warn uh warn you guys.
• 4:40
  SPEAKER_00: But you know, it's it's it's a little theoretical, but at the same time, I think it's important because it takes the thinking out from the level that we are all conversing about when we talk about AI as being, oh, you know, we can hallucinate and get things wrong, or or worse yet, it's magical, it's amazing.
• 4:54
  SPEAKER_00: Like that level of conversation is fine.
• 4:56
  SPEAKER_00: But I think we need to understand the next level of conversation, which is about what it's actually doing, what what are the dynamics that are happening behind the scenes when a human and an AI are together are making a decision.
• 5:07
  SPEAKER_02: Yeah.
• 5:08
  SPEAKER_02: I and they actually start to unpack hey, what are the aspects of autonomy that we should be thinking about?
• 5:15
  SPEAKER_02: What makes up autonomy?
• 5:16
  SPEAKER_02: And one of those is understanding, right?
• 5:18
  SPEAKER_02: You have to understand what's going on in the realm around you in order to make like an informed decision later on about what to do.
• 5:30
  SPEAKER_02: And so you can think of lots of ways that these LLMs can enhance your understanding, right?
• 5:37
  SPEAKER_02: Um, and they will do a great job of like, you know, ingesting a bunch of papers, summarizing them, compiling them, oh, making it, you know, uh something easy for a patient to read, decreasing the reading level, um, making it more practical, whatever.
• 5:54
  SPEAKER_02: So you can think of ways where it could enhance your understanding.
• 5:57
  SPEAKER_02: But then, like you were just saying, we know that AI can spread misinformation, that it can hallucinate, that um, we've talked about models drifting over time.
• 6:09
  SPEAKER_02: Maybe something that used to apply doesn't apply anymore and it's it's being applied.
• 6:13
  SPEAKER_02: Or maybe it was trained on a subset of data that was biased or different from how you want to apply it now.
• 6:19
  SPEAKER_00: And so even though or it doesn't have all of the context in which to make the right call, right?
• 6:24
  SPEAKER_02: Yeah, absolutely.
• 6:25
  SPEAKER_02: And so it could enhance your understanding, but it could also you have you ever gone down to like the I'm gonna give an example.
• 6:35
  SPEAKER_02: Okay, this is not really medical, but like on Google, if I've typed something in before, like I typed in the name of a person, I want to look them up, and I look at the AI summary, and it and I let's just say I chose, you know, Jim Collins, and I look up and Jim Collins, there's like a little AI summary about this person, and it actually, when I when I went deeper, it had taken two Jim Collinses and combined them and combined them.
• 7:00
  SPEAKER_02: What?
• 7:00
  SPEAKER_02: Yeah.
• 7:01
  SPEAKER_02: Uh and so it's like, well, you kind of have to go down to the original source to get like the true information.
• 7:06
  SPEAKER_02: And I was confused because I was like, wait, it said this person was like born this year and died that year, doesn't make sense.
• 7:12
  SPEAKER_02: And so I like went Yeah, yeah.
• 7:13
  SPEAKER_00: And that was that was a clear example.
• 7:14
  SPEAKER_00: So you were able it sort of raised your spidey senses to realize something was off, but that may not happen, right?
• 7:21
  SPEAKER_00: It may be much more subtle than that.
• 7:23
  SPEAKER_02: Right, exactly.
• 7:24
  SPEAKER_02: And so you might not know that what you're getting is misinformation, especially if it's a a a complex topic that you're trying to sort out.
• 7:32
  SPEAKER_02: And so while it can improve your understanding, if it's pulling from like, you know, the the right studies and the right evidence base and we know it's applicable or whatever, it can also decrease your understanding.
• 7:47
  SPEAKER_02: Right.
• 7:47
  SPEAKER_02: If it's pulling from, you know, wrong information, things that have been debunked, um, or if it's hallucinating or whatever it is out there.
• 7:56
  SPEAKER_02: And all of those things can actually, if you're not able to understand something correctly because it's not provided good information, then it's going to actually decrease your autonomy because you don't have the understanding to make the correct decision.
• 8:11
  SPEAKER_02: Yeah.
• 8:11
  SPEAKER_02: And in fact, worse yet, it it decreases the autonomy because you have the wrong understanding.
• 8:15
  SPEAKER_02: Yeah, right.
• 8:16
  SPEAKER_02: It decreases the autonomy.
• 8:17
  SPEAKER_00: Yeah.
• 8:19
  SPEAKER_02: There's other aspects to this too.
• 8:22
  SPEAKER_02: So one of the other things they go into is this idea of competency, of how, you know, we actually need like practice to make decisions.
• 8:31
  SPEAKER_02: Yeah.
• 8:31
  SPEAKER_02: The more decisions you make, like over time, you know, the better you're gonna get at doing something like that.
• 8:37
  SPEAKER_02: Um, and so when we can obviously have support from these systems in making decisions, but over time, if it's making all these decisions for us, then we become less good at making decisions.
• 8:51
  SPEAKER_02: We had an episode about how um endoscopists who were looking for polyps on colonoscopies actually de-skilled over time.
• 9:01
  SPEAKER_02: They got worse at finding the polyps when they were used after they started using this AI tool.
• 9:08
  SPEAKER_02: And so not only have they de-skilled, but now they may overrely on these AI systems and become, you know, less competent because they have this bias to just like automate it.
• 9:21
  SPEAKER_02: And so you offload the task onto the AI and you're not having to think it through.
• 9:26
  SPEAKER_00: Yeah, and de-skilling doesn't have to be like something very technical or specific that you've learned to do and now you're doing it worse.
• 9:34
  SPEAKER_00: De-skilling is more general than that.
• 9:36
  SPEAKER_00: What we're saying here is I think de-skilling could be de-skilling of your general problem-solving decision-making skills.
• 9:43
  SPEAKER_00: So it's possible that that same and I'd be curious about this actually.
• 9:47
  SPEAKER_00: I don't know if this is uh true.
• 9:49
  SPEAKER_00: To what degree does using AI for making decisions, like otherwise unimportant decisions in your life, affect your um affect your job in some fashion where you do use AI to make those technical decisions?
• 10:02
  SPEAKER_00: Like, are you because you're not practicing your problem solving skills all the time, and because you're uh sort of deferring to AI for your decisions, maybe it's affecting your ability to make decisions in other places, right?
• 10:14
  SPEAKER_00: And so, like For sure.
• 10:15
  SPEAKER_00: Yeah.
• 10:15
  SPEAKER_02: So like are you paralyzed?
• 10:16
  SPEAKER_02: Do you feel like you have to ask?
• 10:17
  SPEAKER_02: There are people out there who are like, oh no, I need to ask Chat GPT before I make this minor decision.
• 10:22
  SPEAKER_02: Right, right?
• 10:22
  SPEAKER_02: Right, right.
• 10:23
  SPEAKER_02: I feel like that's a good idea.
• 10:24
  SPEAKER_02: That's not a good thing.
• 10:25
  SPEAKER_00: I know, I know.
• 10:25
  SPEAKER_00: And you're I mean people are aware of it, and you know, they're it's hard to address that.
• 10:29
  SPEAKER_00: And I this is the problem, right?
• 10:31
  SPEAKER_00: It's the it's the sub it's like a soft, slow takeaway of your of your autonomy, of your cognitive skills, and it's different from like other things we have in our lives that help us, right?
• 10:42
  SPEAKER_00: I I always give this example of Google Maps because I think that to me, I'm from the generation where um I did start with physical maps when I was younger.
• 10:53
  SPEAKER_00: And at some point, GPS came along, and at some point, you know, that became even more easy to use through Google Maps.
• 10:59
  SPEAKER_00: And all of a sudden, now everywhere I go, I put that address in and I just literally followed the map.
• 11:05
  SPEAKER_00: Now, what's interesting about this is that I've have I de-skilled my navigation skills?
• 11:10
  SPEAKER_00: Probably.
• 11:11
  SPEAKER_00: I mean, you know, part of navigation requires you to look around, right?
• 11:15
  SPEAKER_00: Look around and see look for the landmarks, look for um, you know, the nearest gas station that you're gonna turn left on before or whatever, right?
• 11:22
  SPEAKER_00: That those are those indications that you are going in the right direction.
• 11:25
  SPEAKER_00: Maybe pay attention to where the sun is, you know, subconsciously.
• 11:28
  SPEAKER_00: Um, but that is gone now because now all you're doing is looking at your phone and looking at the blue line on your Google map and looking at the little car and telling and it tells you turn left next.
• 11:39
  SPEAKER_00: You're not paying attention to what's at that left turn anymore.
• 11:43
  SPEAKER_02: Um this is like a particularly funny example because we are the same generation and I do not use Google Maps most of the time or Waze or any of them.
• 11:51
  SPEAKER_02: If I'm if I'm driving for the most part and know where I'm going, I don't even turn it on.
• 11:56
  SPEAKER_02: I just go.
• 11:56
  SPEAKER_02: If I'm worried about traffic, sure.
• 11:58
  SPEAKER_02: Yeah, I'll turn it on.
• 11:59
  SPEAKER_02: But otherwise, I'm like, okay, I do the landmark thing.
• 12:04
  SPEAKER_00: Yeah, I definitely find myself being anxious and uh interested in getting to my destination, and so I'll just put it in just in case.
• 12:13
  SPEAKER_02: So and what has that done to your understanding and your competency?
• 12:17
  SPEAKER_00: Right, yeah, right?
• 12:18
  SPEAKER_00: Right, right.
• 12:18
  SPEAKER_00: And the thing is, it's what's interesting also is that I, you know, I put it in only after I've done the route a bunch of times do I stop worrying about the map.
• 12:28
  SPEAKER_00: And that takes a lot of effort.
• 12:29
  SPEAKER_00: And even then, I put it in sometimes because, and if it's a fair enough distance, because what if there's a a closure in a road and I have to take an alternative route?
• 12:38
  SPEAKER_00: Well, Google Maps computes that alternative route for me immediately, right?
• 12:42
  SPEAKER_00: I don't have to like go through you know local roads to figure out how to get back onto the, you know, get the fun.
• 12:49
  SPEAKER_00: Right, right.
• 12:49
  SPEAKER_00: But I mean, all of this is to say that that's an example of a de-skilling process that I think is happening here in other aspects of our lives too, with AI usage.
• 12:58
  SPEAKER_02: And I think it transitions well to this third aspect of autonomy they talk about in this paper, which is like voluntariness.
• 13:04
  SPEAKER_02: Like, are you I mean, you're voluntarily opening up Google Maps, but like what's kind of going on in the background there, right?
• 13:13
  SPEAKER_02: Um, because there are certain aspects of AI that can, like you said, elaborate on your options.
• 13:21
  SPEAKER_02: Maybe if you don't want to turn right because you see a traffic jam and you turn left, like it's gonna help you understand what your options are, how what other ways you could go.
• 13:29
  SPEAKER_02: Right.
• 13:30
  SPEAKER_02: It um, but at the same time, it can figure you out.
• 13:35
  SPEAKER_02: It can know, like, oh, this this is your vulnerability, or we know what your fears are.
• 13:41
  SPEAKER_02: We're gonna like target your your weaknesses.
• 13:44
  SPEAKER_02: Um, and so it can like nudge you, it can sort of limit what your perceived options are.
• 13:52
  SPEAKER_02: It can in in some ways with the algorithms like co coerce you into potentially doing something and you may not even notice.
• 13:58
  SPEAKER_02: Yeah, yeah, yeah.
• 13:59
  SPEAKER_02: And so we think that it's voluntary to ask the LLM this question, but it can nudge you in a direction where you kind of like, yes, depend on it, and then you just sort of do what it says.
• 14:14
  SPEAKER_02: And so you feel like you've made the decision, but like, was it really voluntary?
• 14:18
  SPEAKER_02: Like, I'm not, I'm not sure.
• 14:20
  SPEAKER_02: This is less applicable to Google Maps because you're like, well, is it turning left or right?
• 14:24
  SPEAKER_02: It doesn't really matter that much.
• 14:26
  SPEAKER_02: But it becomes more important for like a healthcare decision.
• 14:30
  SPEAKER_02: Did you voluntarily make that decision?
• 14:32
  SPEAKER_02: Right.
• 14:33
  SPEAKER_02: And what was that process like?
• 14:35
  SPEAKER_01: Right, right, right, right.
• 14:37
  SPEAKER_01: Yeah.
• 14:37
  SPEAKER_00: I mean, that's the the voluntariness relates to this notion of agency, right?
• 14:42
  SPEAKER_00: And and that's kind of the idea that you as a human have the agency to do make those decisions.
• 14:47
  SPEAKER_02: And you may Yeah, that you can act on your beliefs.
• 14:50
  SPEAKER_00: Right, yeah.
• 14:51
  SPEAKER_02: That you can act on your own values, that you've done your own reasoning.
• 14:54
  SPEAKER_02: Right.
• 14:57
  SPEAKER_02: It's crazy to think that we're maybe letting AI take that away from us.
• 15:02
  SPEAKER_02: Like when you sit back and think about it, I'm like, oh, this is not good.
• 15:07
  SPEAKER_00: Yeah, yeah.
• 15:08
  SPEAKER_00: I mean, that's the thing, right?
• 15:10
  SPEAKER_00: These are pieces of these are technology tools that are helping us think in a certain way, help help us do our life do our things in our lives better and faster and more effectively.
• 15:20
  SPEAKER_00: And as a result, we're giving up certain um human capabilities, right?
• 15:27
  SPEAKER_00: And it's I think more me, I think it really affects us right now because it's all all of these things that it's automating is like mental skills, which we believe to be quintessentially human in so many ways.
• 15:37
  SPEAKER_00: Like it's not the same as a car driving us places when we could have walked.
• 15:43
  SPEAKER_00: Maybe we have weaker legs as a human race because we don't walk as much or something.
• 15:46
  SPEAKER_00: That's probably true.
• 15:48
  SPEAKER_00: But but that's my point.
• 15:48
  SPEAKER_00: So that we don't seem to care much about that loss of de-skilling, so to speak.
• 15:53
  SPEAKER_00: It's not de-skilling, but you know, loss of some capability.
• 15:56
  SPEAKER_00: But for some reason, this is different because it goes at our thinking skills, and we are more uncomfortable with having that being taken away if we notice it even.
• 16:06
  SPEAKER_02: Well, that's the I think that's part of the other problem is like, do you even know it's happening?
• 16:10
  SPEAKER_02: I do think with every technological advance that's happened, you're like, oh, like this isn't it.
• 16:15
  SPEAKER_02: There are people with it may not be great for, like, you know, the the Xerox machine, is that taking away from you know, printing pro printing press and other things?
• 16:23
  SPEAKER_02: Or what whatever technology, like machines that have been created to weave, is that taking away from the weavers?
• 16:29
  SPEAKER_02: Right, like there's so many things like industrial revolution-wise, where um a new technology was created and it and it can affect people in a negative way.
• 16:38
  SPEAKER_02: But I kind of agree with you that this feels different because it's like it's almost like covert.
• 16:44
  SPEAKER_02: Like you don't even know necessarily that you're being affected, that it's personalizing it to you, that it's nudging you.
• 16:50
  SPEAKER_02: Like you feel like you've made the decision, and you probably don't notice that you're de-skilling along the way either.
• 16:57
  SPEAKER_02: You might notice like six months later when you're like, oh, I used to be able to do that so easily, and it's not I'm not so great at that anymore.
• 17:03
  SPEAKER_00: Yeah, yeah, yeah, yeah, yeah.
• 17:04
  SPEAKER_00: I mean, I think we're seeing this a lot with um, especially in the space that I work in, which is computer programming and software development and AI uh research, where people primary task that we do on a daily basis is writing code.
• 17:20
  SPEAKER_00: And writing code because we are writing programs that test our AI theories and you know build models and so on.
• 17:26
  SPEAKER_00: But all of that is now heavily influenced by AI-based coding agents, that is tools that are helping us code.
• 17:35
  SPEAKER_00: And AI coders are pretty good.
• 17:36
  SPEAKER_00: Like they uh AI has ingested enough of the world's programs across different languages, across different use cases, that it's basically able to generate programs for any algorithm.
• 17:47
  SPEAKER_00: And it's very effective in that sense, and it's heavily used uh across the industry now.
• 17:53
  SPEAKER_00: Uh and it's interesting because I use it daily.
• 17:55
  SPEAKER_00: Um, I use AI to help me write a lot of code.
• 17:58
  SPEAKER_00: Uh, but what's interesting is I do notice it from time to time where maybe there's a temporary like issue with the AI model that I'm using that I can't use it.
• 18:06
  SPEAKER_00: And all of a sudden I'm stuck.
• 18:07
  SPEAKER_00: I'm like, okay, how do what do we do?
• 18:09
  SPEAKER_00: What do I do next?
• 18:10
  SPEAKER_02: Well, this is what I worry about too is like, is there a knowledge drain that's going to happen?
• 18:14
  SPEAKER_02: Are we like all so dependent?
• 18:16
  SPEAKER_02: Like we're so dependent that we're all just getting dumber.
• 18:18
  SPEAKER_02: And then you're you have like kind of this holistic loss of expertise in society because we've just like let AI do it.
• 18:26
  SPEAKER_01: Yeah.
• 18:26
  SPEAKER_02: And what's going to happen in one or two generations?
• 18:29
  SPEAKER_02: I don't know.
• 18:30
  SPEAKER_02: This is where I go into a little spiral, like, oh gosh, I don't know.
• 18:33
  SPEAKER_02: This is a good thing.
• 18:34
  SPEAKER_00: Yeah.
• 18:35
  SPEAKER_00: And then speaking, going back to the coding example, I mean, that's happening right now, right?
• 18:38
  SPEAKER_00: You have a lot of um uh companies choosing to have uh AI coding assistants uh help senior developers, and that way they're not hiring as many junior software engineers.
• 18:50
  SPEAKER_00: And at the moment, it seems like that's a very effective way to do things because the senior developers already know how to code, they already understand um what the core business problem that they're solving is, and all they need to do is somebody to help quickly implement and iterate on what their ideas are, which is what an automatic, like an AI-based system can do.
• 19:09
  SPEAKER_00: The problem is as the senior engineers get older, who is going to be the new generation of senior engineers?
• 19:15
  SPEAKER_02: There's no one who's been trained up, right?
• 19:18
  SPEAKER_02: Right.
• 19:18
  SPEAKER_00: You are you gonna make the AIs the senior engineers?
• 19:21
  SPEAKER_00: Yes.
• 19:21
  SPEAKER_00: So that problem exists in that space, and I think people are finding different ways of addressing it and working, you know, finding different ways of kind of I I think the tool will adapt to some degree to the human use cases, and there's gonna be some world in which both are being used.
• 19:34
  SPEAKER_00: But you know, I think that is just one example.
• 19:37
  SPEAKER_00: But in other domains, like I think healthcare, it's different.
• 19:40
  SPEAKER_00: I think it's even more complicated.
• 19:41
  SPEAKER_00: It's not as cut and dry as a pro writing a program.
• 19:44
  SPEAKER_02: Um absolutely not.
• 19:46
  SPEAKER_02: And and we've talked about before how much uncertainty there is in in medicine and in healthcare.
• 19:52
  SPEAKER_02: And uh back in like the 1950s, there was this, there was sort of like this paternalism in medicine where doctor knows me.
• 20:00
  SPEAKER_02: Yeah, exactly.
• 20:01
  SPEAKER_00: But that was your doctor, right?
• 20:02
  SPEAKER_00: Your doctor was a person.
• 20:04
  SPEAKER_00: So at least at some level, you had a person who kind of cared about your family and knew you.
• 20:10
  SPEAKER_00: And fine, there was paternalism, that was not ideal.
• 20:12
  SPEAKER_00: They made the decision for you.
• 20:14
  SPEAKER_00: They made the decision for you, and that then evolved through the course of decades to a more um personalized form of paternalism where you now have informed consent where they're required to get tell you everything you need to understand the situation you're in.
• 20:29
  SPEAKER_02: Yeah, here are the risks, here are the benefits, here are all the alternatives, right?
• 20:34
  SPEAKER_00: Yes, and and they help you with the decision making in the sense that they give you the alternatives, and they might even say, you know, I you might ask them, you know, what would you do, doc, in my position?
• 20:43
  SPEAKER_00: And they might tell you, oh, I would do this and this, and they would give you a little bit more.
• 20:47
  SPEAKER_00: But that's again, it's personalized in a way that the doctor has built, a doctor and you have built enough trust with each other where you are now comfortable asking the doc doctor for their perspective on it, right?
• 20:58
  SPEAKER_00: It's a different situation there where uh so but that's the world we we are in.
• 21:02
  SPEAKER_00: But now with AI and healthcare advice and diagnostics provided by AI, the problem is I think we're going back to a world pre-1950s, whatever, pre-the-old paternalism, where now it's not only that the AI knows best or the AI doctor knows best, it's that the AI doctor makes you believe that the decision was yours when it wasn't, right?
• 21:26
  SPEAKER_00: It's this uh communication because for example, an AI system, when it's providing you any kind of advice, it's uh it has many choices it can make.
• 21:34
  SPEAKER_00: I'm not saying it's making these choices intentionally, I'm just saying that there's different things it can do.
• 21:38
  SPEAKER_00: It can refuse or withhold information, it can filter information to you, it can hedge things, it can say things and then you know, sort of add caveats, it can nudge in a certain direction, it can even control and police your emotional tone by telling you to feel a certain way or not.
• 21:55
  SPEAKER_00: It can reframe things for you and it can sort of uh begin to sort of talk down to you, talk up.
• 22:00
  SPEAKER_00: Do whatever, right?
• 22:01
  SPEAKER_00: All of these things have a certain degree of influence on you, some of which is overt.
• 22:06
  SPEAKER_00: Like it might just say, No, I can't tell you that.
• 22:08
  SPEAKER_00: That's a form of paternalism, right?
• 22:09
  SPEAKER_00: Because it says, You're asking for information, but I'm not going to give you that information because I don't believe you should have it.
• 22:15
  SPEAKER_00: That's a form of paternalism.
• 22:16
  SPEAKER_00: That's very overt.
• 22:17
  SPEAKER_00: But it's can be very covert too, where it's nudging you in a particular direction.
• 22:21
  SPEAKER_00: It's saying, Hey, here's the things I'm telling you.
• 22:24
  SPEAKER_00: And you, as a human listener, will obviously make a decision a certain way.
• 22:29
  SPEAKER_00: But it was the whole time you're thinking it's your choice, but it wasn't.
• 22:32
  SPEAKER_02: It was nudging you in that direction based on how it was filtering the information.
• 22:37
  SPEAKER_00: Right.
• 22:37
  SPEAKER_00: So it's almost like you have you feel like you have the final say, but you really don't.
• 22:42
  SPEAKER_00: And I think that that's a form of paternalism that's even worse than what it was when you know the doctor just told you what to do.
• 22:49
  SPEAKER_02: Yeah, because it's like it's it's there and it's tricking you into believing it's not.
• 22:54
  SPEAKER_01: Yeah, yeah, right, right.
• 22:55
  SPEAKER_02: Which is scary.
• 22:56
  SPEAKER_02: I think, you know, I I like the concept of how we how we provide risks, benefits, alternatives, and guidance without a hundred percent making the decision for someone.
• 23:09
  SPEAKER_02: Now there are times where it's like, hey, you have strep throat.
• 23:12
  SPEAKER_02: We can either treat you with uh pills or a shot.
• 23:15
  SPEAKER_02: Which one do you want?
• 23:17
  SPEAKER_02: But you need treatment, right?
• 23:18
  SPEAKER_02: Like we're not gonna like just send you home without treatment because we know that you're at risk for rheumatic heart disease or whatever.
• 23:25
  SPEAKER_02: So we have to treat the strep throat, but like here are your two options or three options for the antibiotics.
• 23:31
  SPEAKER_02: Choose one.
• 23:31
  SPEAKER_02: And any of those are fine.
• 23:33
  SPEAKER_02: Right.
• 23:33
  SPEAKER_02: That's like an example of something where, hey, there's a clear recommendation we need to treat this to prevent problems later on.
• 23:40
  SPEAKER_02: And then you can choose, you know, if you want the shot or the pill.
• 23:45
  SPEAKER_02: That that makes sense to me.
• 23:47
  SPEAKER_02: But I think when you have when you're talking to AI about some of these things and it's kind of pointing you in a direction, one, it may not may not really have the expertise, right?
• 23:59
  SPEAKER_02: Right.
• 24:00
  SPEAKER_02: And two, it's like making you think that you're making the decision when it's actually doing it for you.
• 24:08
  SPEAKER_00: Exactly.
• 24:09
  SPEAKER_01: Exactly.
• 24:10
  SPEAKER_00: And that's the the angle of paternalism that I think it takes out the voluntariness a little bit, going back to your three points, right?
• 24:15
  SPEAKER_02: Right.
• 24:16
  SPEAKER_02: It can it can impact your understanding by filtering out the information.
• 24:20
  SPEAKER_02: Right.
• 24:21
  SPEAKER_02: And um you over-rely on it because you use it to make decisions all the time.
• 24:27
  SPEAKER_02: And then now it's sort of coerced coerced you uh in a way, whether that's something that you were able to tell or not, it kind of takes away that voluntary nature when it has this sort of like potential hidden intents in it.
• 24:44
  SPEAKER_00: Right, right.
• 24:45
  SPEAKER_00: No, that's exactly right.
• 24:46
  SPEAKER_00: And I think, you know, going back to the the idea that you were bringing up earlier, we have those sort of three uh angles, right?
• 24:52
  SPEAKER_00: So we talked about understanding, we talked about voluntariness, and there's a third one, right?
• 24:56
  SPEAKER_02: Yeah, competency, which we talked about a bit too, which like you know, the the the decision support is helpful, but we don't want to over-rely on it or de-skill because of that.
• 25:06
  SPEAKER_00: Yes, yes.
• 25:07
  SPEAKER_00: And and I just want to note that there's a lot of people doing research on many different pieces of this across the board and trying to address these issues.
• 25:13
  SPEAKER_00: But while that research is happening, uh AI tools are out there.
• 25:17
  SPEAKER_02: Yeah, and I don't want to give up my agency.
• 25:19
  SPEAKER_02: I don't want my patients giving up their their agency either, right?
• 25:23
  SPEAKER_01: Yeah.
• 25:24
  SPEAKER_02: We this is one of the papers that just kind of scared me the most because like you take a step back, like in the moment, it feels really good to use AI to answer some of these questions that you might have, whatever, whatever the topic is, health, healthcare related or not.
• 25:40
  SPEAKER_02: It it's there, it gives you lots of information, it feels very sure.
• 25:44
  SPEAKER_02: You can go back and forth with it, whatever.
• 25:46
  SPEAKER_02: But then you take a step back and you're like, hey, what am I doing?
• 25:50
  SPEAKER_02: What am I doing in general?
• 25:52
  SPEAKER_02: How often am I using this?
• 25:53
  SPEAKER_02: How often am I using this to help me make decisions?
• 25:56
  SPEAKER_02: All the time.
• 25:57
  SPEAKER_02: Right?
• 25:57
  SPEAKER_02: And then you're like, hey, I just gave up my decision-making ability.
• 26:03
  SPEAKER_02: I just gave up my agency.
• 26:05
  SPEAKER_02: I'm not sure if this is actually aligned with my authentic values.
• 26:10
  SPEAKER_02: I just lost an opportunity to talk to my friend or family member about what I should do next.
• 26:16
  SPEAKER_02: And that's that's the area where I get nervous and scared because it's not the single interaction that's problematic.
• 26:24
  SPEAKER_02: It's like taking all of those over time.
• 26:27
  SPEAKER_02: And now you've you've lost out on relationships that could help you in life in general and to decrease your loneliness.
• 26:36
  SPEAKER_02: Like you've lost out on like, who are you?
• 26:39
  SPEAKER_02: What is your authentic self?
• 26:40
  SPEAKER_02: If AI is making those decisions for you, do you even know who you are anymore?
• 26:45
  SPEAKER_02: And you've lost out on your ability to make decisions when you're outsourcing it to someone or something else all the time.
• 26:53
  SPEAKER_02: Absolutely.
• 26:53
  SPEAKER_00: And I think that's a great point, great place to end here.
• 26:56
  SPEAKER_02: On this, like dun, dun, dun, doom, you think is a great place to end?
• 27:00
  SPEAKER_00: I think so.
• 27:01
  SPEAKER_00: I think so.
• 27:02
  SPEAKER_00: I I think I think we'll I think in this podcast, our goal has been to kind of mix it around, right?
• 27:06
  SPEAKER_00: Have use cases of AI where you know there's excitement and promise, and use cases of AI where there's potential issues.
• 27:12
  SPEAKER_00: And then there's just to general this one specifically, where we're talking about AI use as it impacts all of these use cases simultaneously, right?
• 27:20
  SPEAKER_00: And there is some doom and gloom.
• 27:22
  SPEAKER_00: I I think there is some excitement.
• 27:24
  SPEAKER_00: That's the nature of new technology.
• 27:26
  SPEAKER_02: Yeah.
• 27:26
  SPEAKER_02: And I think we'd be remiss if we didn't cover all the different aspects of it and take a step back sometimes and say, hey, there are some not-so great things.
• 27:35
  SPEAKER_02: Like, what are you what are you outsourcing to AI that maybe you could just like not today or tomorrow?
• 27:40
  SPEAKER_02: Right.
• 27:40
  SPEAKER_02: Take, take a step back and be like, hey, I don't, I don't need to use Google Maps for that.
• 27:44
  SPEAKER_02: Or I'm gonna make the decision by asking my friend or family member instead of asking a chat bot and see how that makes you feel.
• 27:51
  SPEAKER_00: Yeah, and frankly, and this is the kind of doom and gloom stuff that we're talking about here, which is different from just like the general existential threats that people talk about AI taking over the world and those kinds of things, which is not a helpful to me.
• 28:02
  SPEAKER_00: That's not a helpful framing.
• 28:03
  SPEAKER_00: It's too broad, too out there, whatever.
• 28:05
  SPEAKER_00: This is a real thing that's happening, right?
• 28:07
  SPEAKER_00: And I think it's what we people need to talk about this.
• 28:10
  SPEAKER_02: And we can choose to make steps in a direction that we say, I'm gonna take back my agency, I'm gonna take back my authentic self by making these small decisions every day that push me in the direction of like, I want to make sure that I'm competent, that I understand things correctly by going to the source, or that the decision I make, you know, it's hard to tell some of the time if it's voluntary.
• 28:36
  SPEAKER_02: But if you talk to a friend or family member instead of a chatbot, you may not be getting all these, you know, background nudges that would push you in one direction or another.
• 28:46
  SPEAKER_02: So I think it is good to take a step back and to end with a little bit of doom and gloom because maybe it will make us change a little bit about how we approach these things, especially when it comes to healthcare.
• 28:58
  SPEAKER_00: Thank you for joining us.
• 28:59
  SPEAKER_02: We'll see you next time on Code and Cure.