Latest / Code & Cure / #61 - Congrats, You Added A Human in the Loop. Now What?

Transcript

#61 - Congrats, You Added A Human in the Loop. Now What? · Code & Cure

• 0:00
  SPEAKER_02: Putting a human in the loop doesn't automatically make medical AI safe.
• 0:05
  SPEAKER_02: The real question is, what does meaningful oversight actually look like?
• 0:18
  SPEAKER_00: Hello and welcome back to Code and Cure, where we discuss decoding health in the age of AI.
• 0:24
  SPEAKER_00: And my name is Vasant Sarathi.
• 0:26
  SPEAKER_00: I'm an AI researcher and cognitive scientist.
• 0:29
  SPEAKER_00: And I'm here with Laura Hagopian.
• 0:31
  SPEAKER_02: I'm an emergency medicine physician.
• 0:33
  SPEAKER_02: And I think we talk about human in the loop all the time as like a panacea.
• 0:37
  SPEAKER_01: Like, oh, you know, all the problems of the world can be solved if we just have a human in the loop.
• 0:42
  SPEAKER_00: Remember, it's a default option, right?
• 0:44
  SPEAKER_00: Like, and and it's very easy for anybody who criticizes an AI system to say, well, is there a human in the loop?
• 0:49
  SPEAKER_00: Yeah.
• 0:49
  SPEAKER_02: And and and now the human is responsible for verifying everything?
• 0:53
  SPEAKER_02: Question mark?
• 0:54
  SPEAKER_02: Like they're accountable because the AI maybe can't be.
• 0:57
  SPEAKER_00: Yes, yes.
• 0:58
  SPEAKER_00: They're like a fail-safe.
• 0:59
  SPEAKER_00: The humans are the default fail-safe, but are they really a fail-safe, right?
• 1:03
  SPEAKER_00: I mean, what does that mean?
• 1:05
  SPEAKER_02: Exactly.
• 1:05
  SPEAKER_02: And does it fall to like each individual human who is, for example, like from a clinical standpoint, does it fall to like each individual physician or advanced practice provider who's using AI to like verify it each and every time?
• 1:22
  SPEAKER_02: Because that feels like, I don't know, difficult.
• 1:28
  SPEAKER_00: Well, too much.
• 1:29
  SPEAKER_00: Well, not only is it difficult, but what is the expectation, right?
• 1:32
  SPEAKER_00: If you're selling a tool to somebody to use, an AI tool, and you say to them that it does all these things, then and and let's say you you also convince them that it's all tested and everything else, then they're going to rely on it.
• 1:44
  SPEAKER_00: That's the whole totally.
• 1:45
  SPEAKER_00: That's the whole point of them buying the tool in the first place.
• 1:47
  SPEAKER_00: Right.
• 1:48
  SPEAKER_02: It's supposed to make their lives easier, right?
• 1:49
  SPEAKER_00: Right.
• 1:49
  SPEAKER_00: Right.
• 1:50
  SPEAKER_02: And so I don't think you can like delegate it to all the individual humans at the point of care.
• 1:57
  SPEAKER_02: That's that's crazy.
• 1:59
  SPEAKER_00: It's a really hard question, to be honest.
• 2:01
  SPEAKER_00: I I don't I mean, it is, it's some sense, um, kind of makes you feel better when there is a expert human in the loop who is looking at all the AI outputs or whatever, and just even if they're glancing it, there's a sense that, oh, maybe they are expert enough that if there's any red flags, they'll catch them.
• 2:22
  SPEAKER_00: And um, or if there's even like orange flags, they'll be able to like identify them, right?
• 2:27
  SPEAKER_00: So better than just not having anybody at all.
• 2:30
  SPEAKER_00: And maybe that's true, and maybe that gives people a sense of satisf uh comfort at some level, but that packs in a whole bunch of assumptions that what that human is doing and what they're capable of, and what the expectations are for them and what they're supposed to do, and all of that, right?
• 2:46
  SPEAKER_00: And it could be variable.
• 2:47
  SPEAKER_00: Some people could be extremely careful with every single line that the AI output in every single time, while others might trust it more, right?
• 2:54
  SPEAKER_00: And uh even the people who don't trust it uh might begin to trust it after multiple uses, or it might go the other way as well.
• 3:01
  SPEAKER_00: But the point is there's a lot of variation uh there too.
• 3:05
  SPEAKER_02: Totally.
• 3:05
  SPEAKER_02: And I actually think an example might be helpful here because um because it kind of illustrates this point that you're talking about, which is that hey, if we expect the AI to be sort of like doing this job for for us, we we may kind of more automatically accept what it gives us as like a as like a ground truth almost.
• 3:28
  SPEAKER_02: So they did this electronic prescribing study where they did this sort of like decision support to help figure out, okay, like let's make sure that these prescriptions are going through correctly.
• 3:39
  SPEAKER_02: We don't want to like omit any piece of the prescription.
• 3:42
  SPEAKER_02: We want to make sure that it's like correct.
• 3:45
  SPEAKER_02: Um, and so what they did was they found that if they gave providers correct decision support, as in they gave them the correct information with AI, then the number of errors went down.
• 3:59
  SPEAKER_02: That makes sense, right?
• 4:01
  SPEAKER_02: You give them the right answer, people accept it, and then there aren't as many errors.
• 4:06
  SPEAKER_02: But then they did the flip side.
• 4:07
  SPEAKER_02: They said, hey, what would happen if we give incorrect recommendations with the AI on purpose?
• 4:15
  SPEAKER_02: What would happen then?
• 4:16
  SPEAKER_02: And in that case, the providers accepted the incorrect recommendations.
• 4:21
  SPEAKER_02: And so it actually increased the errors.
• 4:24
  SPEAKER_02: Wow.
• 4:25
  SPEAKER_02: Which just goes to show you that there's this like automation bias where you're like, oh, well, the AI has recommended it.
• 4:31
  SPEAKER_02: That sounds good, accept.
• 4:33
  SPEAKER_02: And you just like clicked the accept button.
• 4:35
  SPEAKER_02: And this is not just the case for electronic prescriptions.
• 4:40
  SPEAKER_02: They've studied this across different areas, including like EKG interpretation or mammograms or other places where it's like, hey, we've decided we trust the AI almost more than we trust ourselves, andor this is the AI's job to do.
• 4:57
  SPEAKER_02: I'm gonna like check the box and move on.
• 4:59
  SPEAKER_02: Right.
• 5:00
  SPEAKER_02: And so this is where it's scary because you're like, well, you can't accept, you can't expect, sorry, individuals to come out and like QA check it every time.
• 5:10
  SPEAKER_02: That takes like a lot of thought work and critical thinking into something that you you almost expect to be automated.
• 5:16
  SPEAKER_02: Like I would expect a decision support about a prescription to be automated.
• 5:21
  SPEAKER_00: Well, that's the whole point of the tool.
• 5:22
  SPEAKER_00: Right.
• 5:22
  SPEAKER_00: Yeah.
• 5:23
  SPEAKER_02: So then I don't want to have to think about, oh, is that dosing three or four times a day or uh how many milligrams is it supposed to be for strep throat versus some other illness?
• 5:32
  SPEAKER_02: You know what I mean?
• 5:33
  SPEAKER_02: You expect that to automate it for you.
• 5:35
  SPEAKER_02: Yes.
• 5:36
  SPEAKER_02: And if it's not there, then you're gonna critically think, okay, like I remember this is the dose or this is how many times a day this medication is, or you look it up, yeah, which is fine.
• 5:45
  SPEAKER_02: But when the automation is there, it's easy to just accept it, whether it's right or wrong.
• 5:51
  SPEAKER_00: Yeah, yeah.
• 5:51
  SPEAKER_00: That's exactly the problem, right?
• 5:53
  SPEAKER_00: I mean, at some level.
• 5:54
  SPEAKER_00: And so that's an example of human in the loop and the human just hitting accept, right?
• 5:59
  SPEAKER_02: Exactly.
• 6:00
  SPEAKER_02: And so it's not, that's not enough, right?
• 6:03
  SPEAKER_00: Yeah.
• 6:03
  SPEAKER_02: That that like idea of human in the loop isn't enough.
• 6:06
  SPEAKER_00: Yeah.
• 6:07
  SPEAKER_00: Yeah.
• 6:07
  SPEAKER_02: Um, and and it's like individual human in the loop, but you also have to think about how are we systemically doing this, right?
• 6:13
  SPEAKER_00: Yeah, and actually, this brings me to to two more points, which I think were very interesting.
• 6:17
  SPEAKER_00: One was that it's not like we're saying somehow that humans are better than the AI.
• 6:23
  SPEAKER_00: In some tasks, the AI systems are in fact better and faster than humans.
• 6:28
  SPEAKER_00: Uh, and in some, you know, some tasks, humans are better.
• 6:31
  SPEAKER_00: It's not that it's not that we're putting the human in the loop just because humans are, quote, better than the AI to check everything.
• 6:36
  SPEAKER_00: That's not what we're trying to say either.
• 6:38
  SPEAKER_00: But and it's not just about accuracy, about getting things right.
• 6:42
  SPEAKER_00: I mean, that's those examples of automation bias are great and display accuracy.
• 6:46
  SPEAKER_00: But to your point about the old system at play, it's about the decision making and what comes with that, right?
• 6:51
  SPEAKER_00: There's accountability.
• 6:52
  SPEAKER_00: Um, there is actionability.
• 6:54
  SPEAKER_00: Like, whatever decision you take, you have to be able to go out and do it.
• 6:56
  SPEAKER_00: For sure.
• 6:57
  SPEAKER_00: And then there's reversibility, whether those decisions can be reversed.
• 7:00
  SPEAKER_00: I mean, they're consequences as well to be considered.
• 7:02
  SPEAKER_00: So, like these are um other factors that play into the kind of the role the human plays.
• 7:10
  SPEAKER_02: And then there's this like added layer of critical thinking too.
• 7:15
  SPEAKER_02: Where, you know, if something is automated for me, then I'm probably gonna think less about it.
• 7:22
  SPEAKER_00: Yeah.
• 7:22
  SPEAKER_02: Right?
• 7:22
  SPEAKER_00: Yeah.
• 7:23
  SPEAKER_02: If it's not automated, I'm gonna think more about it.
• 7:25
  SPEAKER_02: I need to, I know, okay, in this situation, I need to use my critical thinking skills to figure this out.
• 7:31
  SPEAKER_02: Versus if something's automated, I'm like, okay, this is all set, just move on.
• 7:34
  SPEAKER_00: Yeah.
• 7:35
  SPEAKER_00: I mean, that's the natural uh sort of state of the brain.
• 7:37
  SPEAKER_00: There's something called epistemic vigilance, which is the idea that for the most part, your brain just kind of works and autopilot, does things quickly because that's beneficial for your survival, that's beneficial for you to be in the world.
• 7:48
  SPEAKER_00: But anytime you notice something weird is when you start sort of are extra vigilant towards it.
• 7:53
  SPEAKER_00: And this happens, you know, you can think about the example being when you first learn to drive, you're like super vigilant about every little detail because you're still learning.
• 8:02
  SPEAKER_00: But once you've learned it, you don't think about it while you're driving on the highway unless something weird is occurring.
• 8:08
  SPEAKER_00: And then at that point, you're like diving deep in and solving that problem, right?
• 8:11
  SPEAKER_00: So, like, I think that's a way to think about this is that as you develop um uh comfort and expertise in a field, you start to like kind of do stuff on autopilot a little bit more, unless things like you know, are are weird at some level.
• 8:26
  SPEAKER_00: Now, the problem with the AI systems is that not everything rises up to that level of weird and could still be wrong, right?
• 8:32
  SPEAKER_00: And so then you miss things easily.
• 8:34
  SPEAKER_02: Yeah.
• 8:34
  SPEAKER_02: And this actually happened to me yesterday while I was working at clinical shift where um I used uh, you know, ambient AI recording to write my notes for me.
• 8:46
  SPEAKER_02: And I had a patient who had come in with low back pain, and I asked a bunch of questions while I was in the room, but some of the questions were about red flag symptoms, red flags being like, hey, I would be worried about these, these would require imaging, et cetera.
• 8:59
  SPEAKER_02: So, like any fevers, any IV drug use, any um urinary incontinence, any numbness in the groin area, that kind of stuff.
• 9:08
  SPEAKER_02: And I definitely asked those questions, and the ambient AI definitely did not put those into my note.
• 9:14
  SPEAKER_00: And interesting.
• 9:16
  SPEAKER_02: And it it was particularly interesting to me because I was reading through my note and sort of double checking it, and I almost missed that it wasn't there.
• 9:23
  SPEAKER_02: Yeah.
• 9:23
  SPEAKER_02: I just sort of assumed that every relevant question I asked would have been put into the note.
• 9:29
  SPEAKER_02: And those red flag questions, multiple, were were not there.
• 9:32
  SPEAKER_02: And I almost just like clicked through and said okay.
• 9:36
  SPEAKER_02: And then I just had something ping in my brain, like, oh, these things were missing, and I definitely asked them.
• 9:41
  SPEAKER_02: That's not good.
• 9:42
  SPEAKER_00: Yeah, and and that's an interesting point too, because you were looking through it.
• 9:47
  SPEAKER_00: You did the, you did the gut check, so to speak, of looking at it, and it's the omissions, the things that it leaves out that's really hard to catch.
• 9:54
  SPEAKER_00: If it says something that's obviously wrong, you're gonna see that and and note it right away.
• 9:58
  SPEAKER_00: But if it leaves something out that requires extra effort for you to think at every step, did it what did it leave out?
• 10:04
  SPEAKER_00: What did it leave out?
• 10:05
  SPEAKER_00: There could be a host of things it could have left out.
• 10:07
  SPEAKER_00: And that's a really big cognitive load on you to kind of think through all the omissions that it that it that and some of these omissions are critical, they're central to everything, right?
• 10:15
  SPEAKER_00: They can make or break the entire case.
• 10:17
  SPEAKER_00: So, as you've noted, so like that's a huge problem with the human in the loop piece, which is humans aren't great at detecting omissions, right?
• 10:25
  SPEAKER_00: And so that's a challenge too.
• 10:27
  SPEAKER_02: And I think there is this idea that the clinician is the human in the loop.
• 10:33
  SPEAKER_02: But I would argue that it can't just be the clinician that's responsible or the human in the loop.
• 10:39
  SPEAKER_02: It's like there's other people involved in the development of these AI tools and they need to be in the loop too, right?
• 10:46
  SPEAKER_02: It's like you have people who are developing these tools.
• 10:49
  SPEAKER_02: So the developers, the AI developers, need to be involved and maybe get clinical input, for example.
• 10:56
  SPEAKER_02: Or we need um, if if a lot, a lot of these things that we've talked about on the podcast have been related to like operational workflows, well, those owners need to be involved in it.
• 11:05
  SPEAKER_02: It's like institutional leadership or departmental leadership, um, like IT departments, uh, safety governance, etc.
• 11:13
  SPEAKER_02: It can't fall to like the individual clinicians seeing patients because a lot of these things are systemic problems.
• 11:20
  SPEAKER_02: And I think that if there's an issue, there's gotta be a way for someone to raise their hand and say, like, hey, let's turn this off, or let's investigate, or let's pause it, or let's deactivate it so we can figure out what sort of correction needs to happen.
• 11:40
  SPEAKER_00: Yeah.
• 11:41
  SPEAKER_00: And I think you mentioned workflow governance.
• 11:43
  SPEAKER_00: I think uh that's that's the key here, right?
• 11:46
  SPEAKER_00: And in this paper, they talk about these sort of four different uh pieces of the equation, and one of which is uh we can go through each one, but the first one is sort of epistemic capacity, which is the idea that do the humans who are in the loop actually understand the technology?
• 12:03
  SPEAKER_00: Do they understand?
• 12:04
  SPEAKER_00: I mean, they don't have to be able to make the technology, but they have to understand what the uh AI model or whatever can do, can't do, and what you know, kind of the limitations are of it and what data that they are affected by those models.
• 12:19
  SPEAKER_00: And so, like, there is a sense that without that understanding, you're not gonna go in informed to be able to handle it, right?
• 12:26
  SPEAKER_00: And so I think that's an interesting piece to me.
• 12:28
  SPEAKER_02: Totally.
• 12:28
  SPEAKER_02: And we've spent many episodes of this podcast talking about how there's not general medical AI, right?
• 12:33
  SPEAKER_02: So you have to understand, okay, this tool is meant to do this task.
• 12:38
  SPEAKER_02: It's not gonna, if it's supposed to do task A, it doesn't mean it can do task B or C.
• 12:43
  SPEAKER_02: We shouldn't ask it to do that.
• 12:44
  SPEAKER_02: So you have to understand, like, hey, this is the task at hand that it's trying to solve, and this is how it could fail, or this is the limitation of it, so that people understand what it can and cannot be used for.
• 12:57
  SPEAKER_02: How is it validated?
• 12:59
  SPEAKER_02: Um, you know, how does it perform?
• 13:02
  SPEAKER_02: Where can I expect it to fail?
• 13:04
  SPEAKER_02: And then in addition to that, we need to make sure people are trained on that, right?
• 13:09
  SPEAKER_02: It's like you we we introduce a new tool to someone, we're like, hey, just use it.
• 13:15
  SPEAKER_00: Yeah.
• 13:16
  SPEAKER_02: Whereas maybe you need to have more education about this is what this tool does, this is how you should and shouldn't use it.
• 13:23
  SPEAKER_02: Here's what you do if there's a problem.
• 13:25
  SPEAKER_00: Right, right, right.
• 13:26
  SPEAKER_00: I think that's really, really that's one part of the system, right?
• 13:29
  SPEAKER_00: Um, I I thought the other part of the system that was really interesting was sort of the second point, which was around cognitive space.
• 13:35
  SPEAKER_00: And this is the idea that when people are making decisions, they should be allowed the space and time to be able to make the right decision.
• 13:43
  SPEAKER_00: And the problem is, of course, a lot of these decisions are time constraints, or there's like deadlines, or there's like a lot of time pressure.
• 13:50
  SPEAKER_00: Um, and so what I liked about this was the thinking about how you might have to design the interface of these tools appropriately so as to enable or ensure that the humans get the appropriate cognitive space to think through the evaluation, right?
• 14:08
  SPEAKER_02: Totally.
• 14:08
  SPEAKER_02: I think the user experience of the clinician is really important here.
• 14:11
  SPEAKER_02: And just like for an example from the medical record, you know, oftentimes it will flag like a vital sign as red if it's abnormal.
• 14:18
  SPEAKER_02: Like, oh, this heart rate of 120, that's abnormal.
• 14:21
  SPEAKER_02: Let's flag it.
• 14:22
  SPEAKER_02: So here, if you have a AI tool, maybe it flags something that's uncertain.
• 14:26
  SPEAKER_02: Oh, this needs to be reviewed, or this is an area like that the AI is good at, and this is an area that it's not good at.
• 14:33
  SPEAKER_02: Or um, I don't know, just different, different ways that you make it so that we don't automatically accept what's put in front of us unless it's like very certain.
• 14:43
  SPEAKER_00: Yeah.
• 14:43
  SPEAKER_00: And there's a larger point here, which is in general, what's what AI is doing, and a lot of companies are building AI are doing, um, is that they are um making things, the experience frictionless.
• 14:56
  SPEAKER_00: The whole idea is to make the experience faster and faster, more easy, more fluid, and so on.
• 15:00
  SPEAKER_00: But friction is not a bad thing.
• 15:03
  SPEAKER_02: Especially here, like you want to have you don't want people to get like alarm fatigue, but you also want to make sure hey, we need people to review rather than passively accepting XYZ as truth.
• 15:18
  SPEAKER_02: And we need to flag these areas that need to be reviewed.
• 15:21
  SPEAKER_02: And that belongs in like the operational workflows and the interface design.
• 15:27
  SPEAKER_00: Yes, exactly.
• 15:28
  SPEAKER_00: So that was sort of the the first two points were very interesting, had to do with technology and how your understanding of it.
• 15:34
  SPEAKER_00: That's what your mind and brain epistemic capacity can handle.
• 15:37
  SPEAKER_00: The second was about the space to be able to make those decisions, and the third point was about decisional authority, which to me is an interesting one too, because you know, people sell these AI systems to you and they tell you it can do all these amazing things, and you begin to think that wait a minute, it can actually do these things better than me.
• 15:54
  SPEAKER_00: Who am I to argue against it?
• 15:56
  SPEAKER_02: This happens to me all the time, too, when there's like an automated reading on something.
• 15:59
  SPEAKER_00: Yeah.
• 16:00
  SPEAKER_02: Um, I I had an EKG yesterday where, you know, the the machine read it as sinus tachycardia with irregularity or something.
• 16:08
  SPEAKER_02: And I was like, no, this looks like AFib.
• 16:10
  SPEAKER_02: But then I was second guessing myself.
• 16:12
  SPEAKER_02: Like, is this AFib?
• 16:14
  SPEAKER_02: Why is the machine reading it differently?
• 16:15
  SPEAKER_02: I'm scouring around for P waves, etc.
• 16:18
  SPEAKER_02: So I I think it's very easy as a clinician to hesitate when this automated reading comes out that you you're like, should I trust this more than than myself?
• 16:26
  SPEAKER_02: And then you're like, well, what when push comes to shove, and there's like, say, there's a legal case or medical legal case or whatever, is is my company gonna back me when I disagreed with the reading.
• 16:39
  SPEAKER_00: Yeah, yeah, exactly.
• 16:40
  SPEAKER_00: No, exactly.
• 16:41
  SPEAKER_00: And in fact, um, the point here is that people who are in the loop, humans in the loop, or the physicians looking at these AI results, they should be allowed the authority to be able to disagree with it, as long as they can provide some kind of justification for their own point of view.
• 16:56
  SPEAKER_00: That that that that is okay to be to say the AI is wrong.
• 16:59
  SPEAKER_00: Right.
• 17:00
  SPEAKER_00: And in my case, it was wrong.
• 17:01
  SPEAKER_02: No, but the thing is, like it was AFib, right?
• 17:03
  SPEAKER_02: It wasn't sinus tachycardia with irregularity.
• 17:06
  SPEAKER_02: I don't even know what that means, but um it it was like, hey, I should be able to document that there are no P waves, that it's irregularly irregular, and that is why it is atrial fibrillation, right?
• 17:17
  SPEAKER_02: I have to give reasoning for it, but then I should be protected.
• 17:20
  SPEAKER_00: Yeah.
• 17:20
  SPEAKER_02: It shouldn't be like, oh, you disagreed with the AI, you, the clinician, are wrong.
• 17:24
  SPEAKER_02: It should be like, oh, the clinician disagreed with AI and the institution backs it up because there's a clear reason.
• 17:30
  SPEAKER_00: Yeah.
• 17:30
  SPEAKER_00: And I think this is also where on the AI side, there should be more effort towards explainability and interpretability and justification for its actions.
• 17:40
  SPEAKER_00: Because it's one, it's almost as if who has the burden to justify that decision?
• 17:45
  SPEAKER_00: If the AI just spits out an answer with no explanation, then and and you as a doctor is looking at it and saying, Oh, I'm not sure.
• 17:51
  SPEAKER_00: I guess I'm I gotta second guess myself.
• 17:53
  SPEAKER_00: You're working super extra hard to justify your decision uh and also see if it makes sense against the AI's decision.
• 17:59
  SPEAKER_00: So the burden's entirely on you, which seems ridiculous.
• 18:02
  SPEAKER_00: That does seem ridiculous.
• 18:03
  SPEAKER_00: So it almost it almost pushes for, in my view, a more explainable system or at least an interpretable system that says, hey, it's this, I think because of A, B, and C, it's this.
• 18:13
  SPEAKER_00: Then at least you can say, actually, your B is wrong.
• 18:16
  SPEAKER_00: And I'm okay with my decision being different.
• 18:18
  SPEAKER_00: And that gives you the ability to make to disagree with it, right?
• 18:21
  SPEAKER_00: Right.
• 18:22
  SPEAKER_00: Totally.
• 18:22
  SPEAKER_00: Yeah.
• 18:23
  SPEAKER_00: I I think this is such an important point.
• 18:25
  SPEAKER_00: Such an important point because it's it's it's it's as we as a society we're getting closer and closer to the point where we're just beginning to accept whatever the AI systems are doing, and it's a huge problem.
• 18:34
  SPEAKER_00: This happens in my field too.
• 18:36
  SPEAKER_00: It happens with coding agents, right?
• 18:38
  SPEAKER_00: They're providing you with with uh incredible coding capabilities and doing experiments for you and things.
• 18:43
  SPEAKER_00: And it's very convincing.
• 18:45
  SPEAKER_00: And saying no is really hard.
• 18:47
  SPEAKER_00: Saying I disagree with you can be hard, you know.
• 18:50
  SPEAKER_00: And um, so I I get it.
• 18:52
  SPEAKER_00: I think that's an important point.
• 18:54
  SPEAKER_02: And then the fourth point on here was around intervention effectiveness, where it's like, well, if it's not doing a good job, can we control the AI?
• 19:04
  SPEAKER_02: Can we say, like, stop, turn it off?
• 19:07
  SPEAKER_02: Can we deactivate it?
• 19:08
  SPEAKER_02: Can we pause it?
• 19:09
  SPEAKER_02: Um, this could happen in lots of different fashions, including with like a model update where it was fine and then the model got updated and now it's not fine anymore.
• 19:18
  SPEAKER_02: It's like, can we can we stop, can we stop it from having um downstream effects?
• 19:23
  SPEAKER_02: Like, is there a good way to govern this?
• 19:26
  SPEAKER_02: And I think this is important in general because you know, we may reject at some time what the AI says, and it's like, well, we should be able to override it if there's a good reason for that, or pause it, or make sure we're making the decisions we want to make as clinicians based on the evidence that we're seeing.
• 19:42
  SPEAKER_02: And I think this is this is key.
• 19:46
  SPEAKER_02: Um, because if you just implement it and you can't press pause, you could have tons of bad downstream effects if it goes awry.
• 19:54
  SPEAKER_00: Yeah.
• 19:54
  SPEAKER_00: And actually, this makes the case for what's called regression testing, which uh basically is a form of testing where it's a software testing technique where you create these bunch of tests, you release their software that it passes all the tests, and then you make some updates to the software and you do regression testing.
• 20:11
  SPEAKER_00: So you make sure that the new updates don't break things that were working before.
• 20:15
  SPEAKER_00: Your new updates might actually improve it in some ways, but we don't want it to break things that were there before.
• 20:19
  SPEAKER_00: And so you do regression testing, which means there's a burden here on the AI builders to make sure it's regression tested, but even on the clinical companies that are making those specific products to make sure that their use cases are also regression tested.
• 20:32
  SPEAKER_02: Yeah, exactly.
• 20:34
  SPEAKER_02: And anytime a model update might be happening, that would need to happen.
• 20:38
  SPEAKER_02: So that's it.
• 20:39
  SPEAKER_02: I mean, though that was the four areas that the paper talked about.
• 20:42
  SPEAKER_02: Epistemic capacity, which is like, hey, you know, can we understand what the model is designed to do, how it was validated, who it's supposed to work in?
• 20:52
  SPEAKER_02: Cognitive space, as in like, hey, do we have the time to actually go through this?
• 20:59
  SPEAKER_02: And do we have the user experience that supports our active involvement as clinicians?
• 21:04
  SPEAKER_02: Decisional authority, which is like, hey, can we say no to it?
• 21:08
  SPEAKER_02: Can we say no to it?
• 21:09
  SPEAKER_02: Right?
• 21:10
  SPEAKER_02: Because we don't want oversight to be symbolic.
• 21:12
  SPEAKER_02: We don't want to just like check the box, like, oh, a human looked at it.
• 21:16
  SPEAKER_00: Right.
• 21:16
  SPEAKER_02: We don't want that.
• 21:17
  SPEAKER_02: We need to be able to diverge from it if it's appropriate.
• 21:21
  SPEAKER_02: And then this whole concept of being able to pause, to deactivate, to correct things so that there are not poor downstream effects.
• 21:31
  SPEAKER_00: Yeah.
• 21:32
  SPEAKER_00: No, exactly.
• 21:32
  SPEAKER_00: I think this is kind of this paper was definitely a more kind of uh conceptual paper in some ways, but very useful to take a little bit more of a deep dive into what a human in the loop actually should involve.
• 21:44
  SPEAKER_02: And it's more than just like one human in the loop, right?
• 21:48
  SPEAKER_02: It's like all of the systems in place and not just a clinician in the loop, but also an AI developer in the loop, a UX department in the loop, a leadership, leadership in the loop, um, you know, clinical and safety governance in the loop, et cetera.
• 22:01
  SPEAKER_02: This isn't just like a single person's job to do.
• 22:05
  SPEAKER_02: And I think that's important to point out is that you have to build this around your clinicians and that other people are on the hook too for creating a systematic way for um making sure that these tools are being used in an appropriate fashion.
• 22:23
  SPEAKER_00: Yeah, and with that, we will end today.
• 22:25
  SPEAKER_00: Thank you for joining us.
• 22:26
  SPEAKER_02: We'll see you next time.