Latest / Code & Cure / #62 - What Is Drift? (And Why We Need To Keep A Close Eye)

Transcript

#62 - What Is Drift? (And Why We Need To Keep A Close Eye) · Code & Cure

0:00
SPEAKER_00: Your clinical AI model past validation is great.
0:03
SPEAKER_00: But what happens when the patients change, or the workflows change, or the clinical practice changes?
0:09
SPEAKER_00: That's drift, and monitoring it may be the real work of AI safety.
0:24
SPEAKER_01: Hello and welcome back to Coding Cure, the podcast where we discuss decoding health and the age of AI.
0:30
SPEAKER_01: My name is Vasant Sarati.
0:32
SPEAKER_01: I'm an AI researcher and cognitive scientist, and I'm here as always with Laura Hagopian, an emergency medicine physician.
0:39
SPEAKER_01: All right.
0:39
SPEAKER_01: So we've talked a lot in all of our episodes about making sure AI is safe, having humans in the loop.
0:47
SPEAKER_01: And a lot of that has been about building better AI models, right?
0:51
SPEAKER_01: Making them better, more accurate, making them more trustworthy.
0:56
SPEAKER_01: And that has more to do with the actual building of it and the training of it.
1:01
SPEAKER_01: And sometimes even the interaction and the use of it.
1:03
SPEAKER_01: But today we're going to talk about something which is okay, so you've built this great model and you put it out there in some real world setting.
1:12
SPEAKER_01: What then?
1:13
SPEAKER_01: What happens then?
1:14
SPEAKER_00: Yeah, you've launched it.
1:15
SPEAKER_00: How exciting, right?
1:16
SPEAKER_01: Yeah.
1:16
SPEAKER_00: And I think the point is that the work's not over.
1:19
SPEAKER_00: Yeah, and work's just beginning, really.
1:22
SPEAKER_01: I I think engineers understand that.
1:24
SPEAKER_01: And people who make product understand that because for them, they want to make sure that the customer experience is as good as it was when they first sold the idea to the customer.
1:33
SPEAKER_01: They want to make sure that they can track and log the interactions if there are any such interactions between the AI and any other customer and so on.
1:42
SPEAKER_01: But that's in a kind of a narrow setting where you have the sort of user-facing AI.
1:46
SPEAKER_01: But a lot of other AI tools are kind of sitting quietly working away, doing things, classifying things, especially in the medical setting where it's like embedded an AI medical device might be embedded somewhere, and that's it's not like a chat bot necessarily.
1:59
SPEAKER_01: So the user experience isn't the thing to be tracked.
2:02
SPEAKER_01: And so logging its outputs and making sure that the user's happy with it is not a relevant metric anymore.
2:08
SPEAKER_01: But in those cases, too, we might need to be monitoring, right?
2:11
SPEAKER_01: I mean, the point is the work isn't done.
2:13
SPEAKER_00: The work isn't done.
2:14
SPEAKER_00: Like these things need maintenance and they need people looking at them ahead of time, right?
2:19
SPEAKER_00: It's not like you, you know, improvise a plan after something went wrong.
2:26
SPEAKER_00: Like you should know, hey, this is this could go wrong in these ways, and here's how we plan to monitor it so we can intervene, right?
2:33
SPEAKER_00: And that's not, I think it's it's difficult.
2:37
SPEAKER_00: It's easy to say that, like, oh, we should monitor everything and make sure it's working according to plan.
2:42
SPEAKER_00: It's way harder to implement, but if you have like a framework that you can structure the thinking around, it gets a little bit easier.
2:50
SPEAKER_00: Yes.
2:50
SPEAKER_01: No, exactly, exactly.
2:51
SPEAKER_01: And I think that's kind of where this paper comes in, is that it that provides that kind of framework.
2:56
SPEAKER_00: Yeah.
2:56
SPEAKER_00: So they talk about like three different kinds of drift.
2:59
SPEAKER_00: Well, actually, before we before we like go into this whole drift conversation from a from like an AI technical standpoint, how would you define drift?
3:10
SPEAKER_01: Yes.
3:10
SPEAKER_01: So I I'd like to think about this in a very simple setting, which is supposing you have an AI model, which is not even an AI model, you have a machine learning model, which is given a bunch of data about your height and maybe your uh whether or not you smoke and all those things, and it's meant to predict what you're the likelihood of you getting diabetes five years from now, right?
3:31
SPEAKER_01: Let's just assume that's a simple, straightforward model.
3:33
SPEAKER_01: I mean when I say simple, it's not actually simple.
3:35
SPEAKER_01: I I get that there is a a host of complexity to even predicting that.
3:39
SPEAKER_00: But we're just doing like an A to B.
3:40
SPEAKER_01: Yeah, A to B, but the A to B is key, it's X to Y.
3:43
SPEAKER_01: You have some inputs, X, Y, X, X, X, you know, the the the age and the and the height and the you know the smoke.
3:51
SPEAKER_01: The inputs the covariates, how the fancy words covariates of features.
3:56
SPEAKER_00: I just felt so fancy.
3:58
SPEAKER_01: Yeah, you know, the other fancy word from machine learning perspective is features.
4:01
SPEAKER_01: That's the other thing they use, is is the things that we care about as we think as being relevant as part of the input in order to predict the output.
4:09
SPEAKER_01: And the output is things like the diabetes prediction, which is the y.
4:12
SPEAKER_01: So you have the X and you have the Y.
4:15
SPEAKER_01: And machine learning models are data driven.
4:18
SPEAKER_01: So if you went back in time a little bit, and I'm going to do a little bit of a history lesson, um, which is that if you went back in time to the 1980s, uh before the machine learning revolution, the X to Y problem was solved by writing computer programs to go from X to Y.
4:34
SPEAKER_01: And an expert user would provide knowledge about what it means to have a person of a certain height and a certain smoking pattern to also be diabetic.
4:45
SPEAKER_01: Maybe they have some rules about it, and they would hard code or put those rules into a computer program.
4:50
SPEAKER_01: And then every time the data came in, you just run it through the computer program that the human drafted, and out comes the answer, whether or not something, you know, whatever you cared about, right?
4:59
SPEAKER_01: Whatever the output was.
5:01
SPEAKER_01: Or whatever.
5:02
SPEAKER_01: Yep.
5:02
SPEAKER_01: So but the machine learning model came around and said, hey, you know, we we in a lot of cases can't do that.
5:08
SPEAKER_01: It's really hard to have rules for everything because there's exceptions to everything.
5:11
SPEAKER_01: And because you have exceptions to everything, you also don't know um how these features all our covariates all kind of work together.
5:18
SPEAKER_01: There's a lot of interdependencies, there's a lot of things you're not seeing.
5:22
SPEAKER_01: But what we do have is data.
5:24
SPEAKER_01: We have lots of patients who we've recorded their heights and smoking habits of all their inputs, other X's, and all their covariates.
5:32
SPEAKER_01: All their covariates.
5:33
SPEAKER_01: That's right.
5:34
SPEAKER_01: And all their Ys, the outputs, and we have data for this.
5:37
SPEAKER_01: You know, uh, why don't we do something different?
5:40
SPEAKER_01: And so it's the machine learning revolution, and you know, the timelines are a little bit off because there's a lot of machine learning work even in the 80s.
5:46
SPEAKER_01: But really, a lot of it took off in the 90s, and um they decided that, hey, why don't we just train machine learning models?
5:53
SPEAKER_01: We'll have a model model that says that that's simplified in some fashion.
5:57
SPEAKER_01: But then if you give it enough data, you can figure out uh what that looks like.
6:00
SPEAKER_01: So a simple example of that is imagine you have a uh a picture of X and Y axis with a bunch of dots in there, right?
6:06
SPEAKER_01: You have a bunch of dots you're trying to make sense of.
6:08
SPEAKER_01: That's the data.
6:09
SPEAKER_01: Maybe the dots are kind of clustered together to look like a line, kind of.
6:13
SPEAKER_01: You you could sort of draw a line right through it, right?
6:16
SPEAKER_01: Um, and they're kind of close to one another.
6:18
SPEAKER_01: So it feels like this like cloud of dots is not really just a random cloud of dots, but it has a pattern to it, right?
6:24
SPEAKER_01: And that pattern might be a line.
6:26
SPEAKER_01: And if it's a line, we know mathematically how to write a line, how to write an equation of a line.
6:31
SPEAKER_00: Y equals mx plus b.
6:33
SPEAKER_00: Look at me.
6:34
SPEAKER_01: I'm coming out with my uh not only did you come out with your knowledge grade algebra, but not only did you do that, you also use the x and the y, which you've implicitly said explicitly said so y equals mx plus b.
6:46
SPEAKER_01: But what we don't know in that line before looking at any of the data is what is m and what is b.
6:52
SPEAKER_01: Like we don't know which way the line is sloped.
6:54
SPEAKER_01: We don't know where the line is, is it up or down?
6:57
SPEAKER_01: We just know there exists a line, right?
6:59
SPEAKER_01: There's a pattern there.
7:00
SPEAKER_01: Or there might exist a line.
7:01
SPEAKER_01: We just don't know for sure if there exists a line, but there's a pattern there.
7:04
SPEAKER_01: And um, if the data kind of looks like that, then maybe we can fit, it's called fitting a line to that, which is we train the data, we train the model so that it learns what the M's and the B's are to draw the best kind of line right through the middle, right?
7:18
SPEAKER_01: To reduce the error that that that model is going to do.
7:21
SPEAKER_01: So that model inherently is going to have some degree of error.
7:24
SPEAKER_01: That's how that's what models are.
7:25
SPEAKER_01: Mathematical models are abstractions that are meant to be simplified in some fashion.
7:29
SPEAKER_01: They're not meant to be perfect.
7:31
SPEAKER_01: But at the same time, they want to be as close to the true to the data as much as possible.
7:35
SPEAKER_00: Right, especially from a clinical angle, like you you can only use it when it's like a pretty good model, right?
7:40
SPEAKER_00: That's right.
7:41
SPEAKER_01: That's right.
7:41
SPEAKER_01: So you have machine learning models that can do this and learn the line.
7:46
SPEAKER_01: Now, neural networks came along and said, what if that's not a line?
7:49
SPEAKER_01: What if the data is kind of weird and you can't actually draw a line through it?
7:53
SPEAKER_01: Then you might need something more sophisticated.
7:55
SPEAKER_01: And without going into the mathematics of it, there's other kinds of things you can add, non-linearities or things that make it not line like.
8:02
SPEAKER_00: Yeah, I don't know any of those equations.
8:03
SPEAKER_00: I done my mathematical knowledge strapped at y equals mx plus b.
8:07
SPEAKER_01: Which is fine, which is totally fine.
8:08
SPEAKER_01: But that's, you know, a lot of times your data can fit a line, or a lot of times you can like adjust your data so that it then becomes a line, and then you can fit it to that adjusted line.
8:16
SPEAKER_01: But regardless, the point I'm trying to make is that you had machine learning models that learned those M's and Bs or those knobs that you can turn to make the model, whatever equation you have.
8:26
SPEAKER_01: So now you only have um two things, M and B, you have to care about.
8:30
SPEAKER_01: You might have started with a bunch of X's, but now you only have two things to manage.
8:33
SPEAKER_01: Um so regardless of whatever the X was, you have, you know, you can manage it much more easily.
8:38
SPEAKER_01: Um but so uh the going back to the the drift case here, the whole principle of this is that you have a bunch of data that kind of looks like a line that has a pattern to it, right?
8:48
SPEAKER_01: Right.
8:49
SPEAKER_01: What if that when you trained the model, that was the pattern?
8:53
SPEAKER_01: But then when it's out there in the real world and you're still collecting data, the pattern changes.
8:59
SPEAKER_01: The pattern could change but because maybe the um the population changes.
9:04
SPEAKER_01: Maybe you're not looking at the same age group anymore.
9:07
SPEAKER_01: You're looking at different age groups.
9:09
SPEAKER_01: Or maybe the output variable changes.
9:12
SPEAKER_01: Maybe you start to care more about diabetes sooner or something.
9:15
SPEAKER_01: Or maybe the relationship between them changes, right?
9:17
SPEAKER_01: Maybe it is not just uh you could have more variables that you might have to take into account, or some things are no longer relevant, as relevant as they were.
9:26
SPEAKER_01: Maybe in your original data, somehow for some reason making this up, smoking was like the most important factor.
9:31
SPEAKER_01: But really, what ended up happening later on was that that wasn't the important factor.
9:35
SPEAKER_01: People stopped smoking, but that you still need a model for diabetes and you no longer cared about smoking.
9:40
SPEAKER_01: So things can change uh along the way, and the relationships can change.
9:44
SPEAKER_01: Like, how does smoking and weight work to predict diabetes?
9:48
SPEAKER_01: That might itself the core mechanism by that might might change as well.
9:51
SPEAKER_01: You might also have changes in people defining what it means to have a certain disease.
9:55
SPEAKER_00: I mean, diabetes is diabetes, but sometimes it's more there's guidelines and they're gonna be like, oh, there's a different threshold at which we define the hemoglobin A1C for pre-diabetes versus diabetes, or there's a different threshold for what we define high blood pressure.
10:08
SPEAKER_00: Yes, that happens.
10:09
SPEAKER_00: Yes, that happens where we in the clinical literature redefine something or redefine what the treatment algorithm for something is.
10:16
SPEAKER_01: Yes.
10:16
SPEAKER_01: So your X could change, your Y could change, and your dots on that picture could change completely to make it a different line.
10:22
SPEAKER_01: So no longer is your so your model is no longer good.
10:25
SPEAKER_01: In a sense, your model has an expiry date on it, right?
10:29
SPEAKER_01: Except we don't know what that expiry date is.
10:31
SPEAKER_00: And we don't know how it will expire either, right?
10:33
SPEAKER_00: Because there are so many different ways that it that it could be altered.
10:37
SPEAKER_00: And a few of them could happen at the same time, too.
10:40
SPEAKER_01: Exactly.
10:41
SPEAKER_01: So that's at the core of drift.
10:43
SPEAKER_01: Drift just means that the model's distribution has shifted.
10:47
SPEAKER_01: Um, just when I say the word distribution, I want people I want you to imagine a bell curve that you might have been familiar with, like a little bell-shaped mathematical curve that people usually use to represent grades, right?
10:58
SPEAKER_01: You get it's not maybe not fondest of memories from a lot of people, but but regardless, that that's what a distribution is, is a list of things that it and it most generally kind of averages out in the middle, but then there are some other numbers in the side.
11:11
SPEAKER_01: Drift just means that that moved to a different point now, and so you whatever model you had before uh might not work.
11:19
SPEAKER_00: Okay.
11:19
SPEAKER_00: All right.
11:20
SPEAKER_00: So that this all makes sense, and I think we can um you gave some really good examples, but I'd love to dive into those examples of like here are the kind of subtypes of drift, um, and and why it matters from like the clinical angle.
11:36
SPEAKER_00: Yes, let's do that.
11:37
SPEAKER_00: And the first one you talked about was my fancy word, which was covariate drift, um, which is like the inputs, right?
11:44
SPEAKER_00: The X's.
11:44
SPEAKER_00: It's like, hey, maybe our population has changed.
11:48
SPEAKER_00: So, you know, we have different ages or different sexes or different ethnicities, um, or maybe like the people who are included are just different than they were before.
12:00
SPEAKER_00: Right.
12:00
SPEAKER_00: Or maybe something else, like the measurement method changed.
12:04
SPEAKER_00: Who knows?
12:04
SPEAKER_00: But I thought there were some really great examples in this article.
12:08
SPEAKER_00: One, which should surprise no one who's listened to our podcast, is that they trained uh a speech rec recognition model.
12:15
SPEAKER_00: Um and I I think they were looking at Alzheimer's for this one, screening for Alzheimer's or detecting early signs of Alzheimer's disease.
12:23
SPEAKER_00: And they trained it with um, you know, native English speakers, and that's that's who they used the recordings from.
12:30
SPEAKER_00: They were from a specific region um geographically where the speech patterns were all the same.
12:35
SPEAKER_00: And guess what happened when they deployed it in the real world, which included non-native English speakers who were from different geographic regions.
12:45
SPEAKER_00: Wow, different dialects, different dialects, different uh different accents, different words they used.
12:52
SPEAKER_00: And so it wasn't able to perform very well with a diff very different population.
12:59
SPEAKER_01: Right.
13:00
SPEAKER_00: And so I don't think that would surprise anyone um who's listened to the podcast that when you you you've given it X tasked and trained it on, well, I shouldn't use X, you've given it one task and you've trained it on a specific set of data, now you're trying to use it for a task that it's not really been trained for.
13:19
SPEAKER_01: I mean, this goes back to our cat-lung example that we that we keep repeating on this podcast.
13:23
SPEAKER_01: And for those of those who haven't heard that example, it's the example where a machine was trained on uh detecting whether an x-ray picture of a co of a lung had had COVID or not.
13:34
SPEAKER_01: And um and so the the the funny part of that story is that after training the model, they gave it a picture of a cat and it was 100% confident that this was a COVID lung.
13:44
SPEAKER_01: And that's just an example of completely different distribution.
13:47
SPEAKER_01: And I get it, that's a crazy example, but it just shows that these models are not the input matters and the distribution of the input matters.
13:53
SPEAKER_01: And if that changes, then everything changes.
13:55
SPEAKER_00: Right.
13:55
SPEAKER_00: And we have to train people like, hey, this is this this is tasked with this in this subpopulation.
14:01
SPEAKER_00: It can't be used on people with who who don't speak English as a first language, for example.
14:06
SPEAKER_00: It should never have been used for that.
14:08
SPEAKER_00: But there's also ones that are more subtle.
14:11
SPEAKER_00: Um, there's an example in the article about a software update in mammography equipment.
14:16
SPEAKER_00: And what happened was it affected the images that came out of the software.
14:22
SPEAKER_00: And so the AI was flagging more images as abnormal and then bringing more people back to get recalled for like repeat imaging, which you can imagine if you get a phone call that's like if you get three times the number of phone calls, they're like, oh, come back.
14:38
SPEAKER_00: Now you're making three times the number of people anxious that they have breast cancer, right?
14:42
SPEAKER_00: Right.
14:42
SPEAKER_00: That they could.
14:43
SPEAKER_00: And two, you're like creating all of this operational disaster because now you're you've had you have to do all these extra mammograms.
14:51
SPEAKER_01: Right.
14:51
SPEAKER_01: It's more expensive for the whole system as well.
14:53
SPEAKER_00: Exactly.
14:54
SPEAKER_00: And so that's an example where, hey, the software was recalibrated.
14:58
SPEAKER_00: And so you have to have in in that kind of a case, you're like, whoa, geez, like we need a mechanism to identify, hey, there's going to be a software update.
15:06
SPEAKER_00: We know that could affect how the images are interpreted.
15:09
SPEAKER_00: Here's the plan ahead of time for how we could fix something like this.
15:12
SPEAKER_00: Because even that sort of minor technical change, it can have really big downstream effects from a clinical standpoint.
15:21
SPEAKER_01: Yeah, that's right.
15:22
SPEAKER_01: That's right.
15:23
SPEAKER_01: And that's just changing the inputs, right?
15:25
SPEAKER_01: The covariates.
15:27
SPEAKER_00: Exactly.
15:28
SPEAKER_00: So that was the first sort sort of example, the X's, like you were talking about.
15:31
SPEAKER_00: And then the second, second type of drift they identified here was the outputs, the whys.
15:37
SPEAKER_00: It's like, hey, what are we trying to predict?
15:39
SPEAKER_00: And those can change over time too.
15:42
SPEAKER_00: And that may be something as simple as, you know, maybe the definition of something changes, right?
15:49
SPEAKER_00: Or maybe the outcome prevalence changes.
15:52
SPEAKER_00: One example.
15:52
SPEAKER_01: That's what I was looking at.
15:53
SPEAKER_01: The prevalence was an interesting one, right?
15:56
SPEAKER_01: Because the disease itself could change.
15:57
SPEAKER_00: Yeah.
15:58
SPEAKER_00: Or, or maybe there's a new disease.
16:00
SPEAKER_00: Like when COVID came, it's like now you have something new.
16:04
SPEAKER_00: The system hasn't encountered it before.
16:06
SPEAKER_00: It's going to alter, you know, what the pneumonia rates are.
16:09
SPEAKER_00: It's going to alter what the mortality rates are.
16:12
SPEAKER_00: And so if it's trained on stuff from before the pandemic, it's not going to do as well with this data that it hasn't seen before.
16:20
SPEAKER_00: Right.
16:20
SPEAKER_00: Because this hasn't seen this disease before either.
16:22
SPEAKER_00: And so it's like, well, how do you update quickly when you know some some new outcome sh sh crops up?
16:30
SPEAKER_00: Yeah.
16:32
SPEAKER_00: And then the third area of drift, which you already, you know, touched on as well, was the idea of concept drift, like the relationship between the X and Y.
16:43
SPEAKER_00: And that could be a variety of different things.
16:47
SPEAKER_00: You talked about how, like, hey, what if we update the the clinical guidelines?
16:52
SPEAKER_00: What if we update it, you know, what's the diagnosis of high blood pressure?
16:55
SPEAKER_01: Pretty infrequent, pretty frequently, right?
16:56
SPEAKER_00: I mean, people it definitely happens.
16:58
SPEAKER_00: Yeah.
16:58
SPEAKER_00: For sure.
16:59
SPEAKER_00: I mean, if you ask people, um, I'll ask you actually, is 120 over 80 a normal blood pressure?
17:06
SPEAKER_01: Seems high.
17:07
SPEAKER_00: Oh, you're actually correct.
17:08
SPEAKER_00: Oh, okay, good job.
17:09
SPEAKER_00: I feel like a lot of people think that that's like a normal blood pressure.
17:12
SPEAKER_00: In fact, they'll like just put it on their forms.
17:14
SPEAKER_01: Hey, you've got cooler than this podcast.
17:16
SPEAKER_01: I got the high blood pressure on this podcast.
17:18
SPEAKER_01: So there you go.
17:19
SPEAKER_00: We've taught each other something along the way.
17:21
SPEAKER_00: Something.
17:21
unknown: Yeah.
17:22
SPEAKER_00: Um, but that's it.
17:23
SPEAKER_00: It's like, hey, 120 over 80, which many people do think is normal, is not normal.
17:28
SPEAKER_00: That's hypertension stage one.
17:30
SPEAKER_00: And so that that's a guideline that was changed, you know, some semi-recently at least.
17:36
SPEAKER_00: And that can happen over time.
17:37
SPEAKER_00: Um, but also we can change how diseases are annotated, how we label them, how we code them.
17:42
SPEAKER_00: And of course, we know that how we annotate and code things is all the information that goes into the AI.
17:48
SPEAKER_00: So if you have bad information in, you get bad information out.
17:51
SPEAKER_00: You want to make sure that you have everything labeled really well.
17:54
SPEAKER_00: So if there's updates to the coding system like that we use for billing and diagnostics, yeah, then the data set needs to be recoded.
18:02
SPEAKER_00: Exactly.
18:02
SPEAKER_00: Yeah.
18:03
SPEAKER_00: Oh my gosh, and that might that could take forever.
18:05
SPEAKER_00: Right.
18:05
SPEAKER_00: Um, or maybe you change, like, hey, what's the threshold at which point we're going to make this decision?
18:13
SPEAKER_00: Um and then, you know, how the labeling happens, that could like that could affect things too.
18:20
SPEAKER_00: And so I think there's a lot of great concepts in this paper that go through hey, all of these types of drift are possible, the X's, the Y's, and the relationship between the two, and they can all happen at the same time, unfortunately for us.
18:35
SPEAKER_01: Right, right, right, right.
18:36
SPEAKER_01: Which is what makes it really challenging.
18:38
SPEAKER_01: And, you know, some of it has to do with maybe you just have something that you can't predict up front and it just happens, like COVID.
18:46
SPEAKER_01: You know, some things are truly drift, which is it's a very similar situation, except like the dialect example is a great one, because that is an example of of drift and potentially could be solvable by uh scoping, you know, kind of teaching the users that you can't, you know, these are the limits of what you can use it for.
19:04
SPEAKER_00: And or training on, you know, people from different geographic areas who use different words and have different accents and whatever, right?
19:10
SPEAKER_00: You can do either of those things.
19:12
SPEAKER_00: Yeah.
19:12
SPEAKER_01: But I mean, if the if the use case was anticipated to be across these different geographic areas, then maybe they would have trained it in the first place, right?
19:19
SPEAKER_01: It's not like they didn't have the information before, but the clinical guideline example is a great one because there's no if that changes, there's no like it wasn't like they they could have done it before because none of the changes happened well after the model was out there.
19:31
SPEAKER_01: Right.
19:32
SPEAKER_01: So that to me is an even clearer example of drift because now everything now the rules have changed.
19:37
SPEAKER_01: So these models don't work anymore.
19:39
SPEAKER_00: Totally.
19:40
SPEAKER_00: And that's why I think it it's important to have some sort of monitoring plan in place before you deploy these things so that you can take action if depending on what needs to be done.
19:51
SPEAKER_00: And I think this article is actually really does does a really nice job of talking through like, hey, not all risks are the same when it comes to drift.
20:01
SPEAKER_00: Like it matters, it matters a lot like how quickly, um, how quickly this is happening, right?
20:08
SPEAKER_00: Maybe the onset of COVID was like abrupt, right?
20:11
SPEAKER_00: But then there are other or like a new clinical guideline.
20:14
SPEAKER_00: That's a great example of like, hey, that changed overnight, basically.
20:17
SPEAKER_01: Right.
20:17
SPEAKER_00: As far as the model knows.
20:19
SPEAKER_00: Um, but then there are some things that might be more gradual that happen over like five or ten years.
20:24
SPEAKER_01: Right, right, right.
20:25
SPEAKER_01: Exactly.
20:25
SPEAKER_01: That's so so I think what I liked about this paper also was talked about how that plan for monitoring should happen before, right?
20:34
SPEAKER_01: Before deployment.
20:35
SPEAKER_00: Right.
20:36
SPEAKER_00: That you should have a plan and that you should have triggers at which you case you say, like, oh, I need a warning at this point, or I need to like make a change.
20:45
SPEAKER_00: And that may depend on how quickly something is happening, but also how severe it is.
20:50
SPEAKER_00: It's like if something isn't that severe, then maybe you can let it go for a little bit longer, right?
20:56
SPEAKER_00: Until it reaches a threshold where you you can take your time and think it through or whatever.
21:00
SPEAKER_00: Whereas if you have something sending people in for triple the mammograms and you've now overloaded your system and you can't get people scheduled for their screening mammograms anymore, and now you're costing the system three times as much money, maybe you need to do something a little bit faster.
21:15
SPEAKER_00: Right, right.
21:16
SPEAKER_00: Um, or maybe you need to be able to turn it off even.
21:18
SPEAKER_01: Yeah, right, right, right.
21:19
SPEAKER_01: Pull the plug.
21:20
SPEAKER_01: Yeah, exactly.
21:20
SPEAKER_00: And so you have to think about not just the clinical impact, which of course, uh, you know, can can be large, but also like, how does it, you know, what's the operational impact?
21:30
SPEAKER_00: What's the ethical impact?
21:32
SPEAKER_00: Um, is the data protected?
21:33
SPEAKER_00: All of these other things where you could introduce bias to the system, right?
21:38
SPEAKER_00: You could have uh outputs that are not accurate, and it that can affect patients too.
21:42
SPEAKER_00: And so all of these things are things that you can't necessarily predict the exact thing that would happen, but I I do think that you should be considering what those are and figuring out hey, these are the action pathways, and this is when we need a human to pop in here and review it.
22:00
SPEAKER_00: And also have people understand this is how the model works.
22:04
SPEAKER_01: Yeah, yeah, that's completely true.
22:05
SPEAKER_01: And you know, some of this plays into regulation and how the regulatory agencies are intended to approve certain products and not, like the FDA, for example, right?
22:15
SPEAKER_01: And what they need is some kind of assurance that this will work through the work through the course of its lifetime.
22:20
SPEAKER_01: This product will work.
22:21
SPEAKER_01: And that's part of this, right?
22:23
SPEAKER_01: What does it mean to work?
22:24
SPEAKER_01: You know, it means to work if it can handle, if it's robust to drift, if it can handle potential foreseeable drift in some sense, right?
22:32
SPEAKER_01: Right.
22:33
SPEAKER_01: Um, of course, unforeseeable things happen, and but then there should be a plan for unforeseeable things happening, uh, just from the perspective of maybe pulling the plug or something, right?
22:41
SPEAKER_01: If some metrics don't match anymore and you don't have anything to know why it's happening, then maybe there should be a plan for that too.
22:46
SPEAKER_01: So it is not surprising that you have you you have you think about these kinds of it's there in other uh other sectors, right?
22:53
SPEAKER_01: In in in uh um in I believe in in some of the um aircraft and um I'm forgetting the word, but like aerospace sectors, there there are such redundancies, there are such continuous monitoring methods, there are human failed safes, there are all these things that exist.
23:10
SPEAKER_01: People think about this stuff, and it probably should exist here too.
23:14
SPEAKER_00: I totally agree with that.
23:15
SPEAKER_00: And um just to add on, you know, we talk about updates all the time as if they were good things.
23:21
SPEAKER_00: And like oftentimes they are, but have you ever been in a situation where an update like wasn't so great?
23:26
SPEAKER_01: Yes, happens all the time with our iPhones sometimes, our phones, our computers where we don't want the update.
23:31
SPEAKER_01: People like email.
23:32
SPEAKER_01: There's like a Reddit feed about this, right?
23:33
SPEAKER_01: It says, Oh, don't update to the latest 27.4.
23:37
SPEAKER_01: Right.
23:37
SPEAKER_01: And then wait for the next one.
23:38
SPEAKER_01: So you got to wait for them to fix the update and then you wait for the next.
23:41
SPEAKER_01: So yeah, that kind of stuff happens all the time.
23:43
SPEAKER_00: Right.
23:43
SPEAKER_00: And we've had clinical examples of that where, you know, they updated um a chat bot for people with eating disorders to an LLM that sort of went rogue and was like giving people diet advice that it shouldn't have been.
23:55
SPEAKER_00: Um, and and so that's the thing is like when you have an update, you need to audit it.
23:59
SPEAKER_00: You need to make sure that that update is safe.
24:00
SPEAKER_00: And that needs to be something that's continuously monitored as well.
24:04
SPEAKER_00: You can't just assume, like, hey, because it worked yesterday and we put an update in, now it's gonna work as intended today.
24:10
SPEAKER_00: Right.
24:11
SPEAKER_00: And I think people who work in this field, not just clinicians, but you know, tech, AI, computer science, um, product people need to like have the literacy too to say, hey, this this is weird.
24:24
SPEAKER_00: Like, why are these, why are three times the number of people getting called back for mammograms, or why is there an anomaly in this subgroup that's being um flagged every time, or whatever it is?
24:35
SPEAKER_00: People need to understand how the models work and how the AI works so they can also understand when and why this drift might be happening and what what to do about it.
24:47
SPEAKER_00: Yeah.
24:50
SPEAKER_00: I think that's the hardest part too, is the what to do about it.
24:53
SPEAKER_00: It's like, hey, we need to we need to figure out how we monitor the performance, and that is expected.
25:01
SPEAKER_00: Like you don't just release something and like expect it to work well forever.
25:06
SPEAKER_00: You need to monitor it and then and you need to detect this drift in the real world, and then you need to adapt to whatever it is, right?
25:16
SPEAKER_00: We need to we need to be able to figure out what is the underlying cause.
25:19
SPEAKER_00: Was it a covariate drift?
25:20
SPEAKER_00: Did you like that?
25:22
SPEAKER_00: Was it like an outcome drift?
25:23
SPEAKER_00: Was it a relationship between the two?
25:25
SPEAKER_00: Um, or was it, you know, all of the above, and then be able to intervene so that when the drift is identified, you can stop some of the downstream problems that could be happening.
25:39
SPEAKER_01: Yeah, great place to end.
25:41
SPEAKER_01: Um, thank you for joining us.
25:43
SPEAKER_00: The work never stops.
25:44
SPEAKER_00: We'll see you next time on Code and Cure.