Latest / Code & Cure / #7 - Predicting No-Shows: The Surprising Science Behind Missed Appointments

Transcript

#7 - Predicting No-Shows: The Surprising Science Behind Missed Appointments · Code & Cure

• 0:01
  Speaker 1: Can we actually predict who might miss their doctor's appointment?
• 0:04
  Speaker 1: Turns out yes, and these might not be the reasons you're expecting.
• 0:18
  Speaker 2: Hello and welcome back.
• 0:19
  Speaker 2: My name is Vasant Sarathy and I am here today with Laura Hagopian.
• 0:25
  Speaker 2: Hey Laura, how's it going today?
• 0:27
  Speaker 1: I'm doing well.
• 0:27
  Speaker 1: How are you?
• 0:28
  Speaker 2: I'm good, I'm good.
• 0:29
  Speaker 2: What are we talking about?
• 0:31
  Speaker 2: What's on the docket for today?
• 0:33
  Speaker 1: Well, we are talking about missed appointments and late cancellations and whether or not those could be predicted, Because if we know, you know who is going to no-show or why people might not show up to their appointments, we can potentially do something about it.
• 0:51
  Speaker 2: Right, right, right, and that happens frequently and it's probably a really important issue to tackle too, because it probably affects their health.
• 0:58
  Speaker 1: Yeah, I mean in studies.
• 1:00
  Speaker 1: The range of appointments missed in primary care practices is anywhere in the eight to 24% range.
• 1:07
  Speaker 1: Wow that's like that could be almost a quarter of patients not not coming to their appointment or canceling within 24 hours.
• 1:14
  Speaker 1: Yeah, yeah, which is like very interesting to hear because I feel like it's so hard to get a doctor's appointment.
• 1:18
  Speaker 1: So I'm I'm curious to dig into this and understand better.
• 1:24
  Speaker 1: You know what is happening that people aren't showing up or that they're canceling with very little notice for their appointments.
• 1:30
  Speaker 1: Yeah, and we found at.
• 1:45
  Speaker 1: What I think was interesting in this study was not only do they look at the stuff that would be normally available in your electronic health record things like demographic data you know their sex, their age, their race, their language but they also looked at things like their healthcare utilization how long it had been since they had a scheduled their appointment, who their clinician was, how long it had been since they had a scheduled their appointment, who their clinician was, how long their clinician had been practicing.
• 2:09
  Speaker 1: But they also looked at geosocial and environmental context.
• 2:15
  Speaker 1: So they looked at things like oh, how far is this person from their doctor's office?
• 2:21
  Speaker 1: And they looked at their area deprivation index and they looked at things like geez, was it raining that day?
• 2:29
  Speaker 1: What was the temperature?
• 2:30
  Speaker 1: Was there any snowfall?
• 2:32
  Speaker 1: And so when you think about something like this holistically, this makes a lot of sense, right?
• 2:36
  Speaker 1: It's not just the information about the patient, but about all the things that are surrounding them that could affect whether or not they show up for an appointment.
• 2:44
  Speaker 2: Yeah, and this sort of, to me at least, reminds me of the Florence Nightingale discussions we had before Folks can refer to, I think podcast.
• 2:54
  Speaker 2: It was number three, but we talked about how other factors are really relevant and potentially could be more important than what you think of.
• 3:01
  Speaker 2: Is the reason why people cancel these appointments.
• 3:05
  Speaker 1: Like might people cancel because it's a really hot or a really cold day?
• 3:08
  Speaker 1: The answer is the answer is yes actually.
• 3:10
  Speaker 1: So it's, it's interesting.
• 3:12
  Speaker 1: And they used, you know, machine learning to try to figure out what the relationships might be or what are the things that could actually predict who's going to not show up for their appointment.
• 3:23
  Speaker 1: So I'd love to hear more from you about the machine learning models they used.
• 3:28
  Speaker 1: And I read this fancy term called gradient boost and I want to know what it means.
• 3:34
  Speaker 2: Yeah, I think this paper was great because folks who are new to machine learning or folks who are picking up machine learning to try to learn how to do it, this is actually kind of a nice paper because it takes you all the way from data to models to types of things to evaluate, all the way through, and kind of shows you how this sort of analysis should be done, and so I really enjoyed reading this paper.
• 3:59
  Speaker 2: And it's not like it's using the most fanciest, latest tool.
• 4:04
  Speaker 2: It's not like a chat GPT-LLM type of machine learning here.
• 4:07
  Speaker 2: This is the kind of stuff that you know folks in universities are learning when they take their intro to machine learning class.
• 4:16
  Speaker 2: Now, of course, the versions that they're learning in the intro to machine learning class are a little bit more simple than what you know is more sophisticated here in this paper.
• 4:23
  Speaker 2: But it's those basic ideas and I think it's worthwhile taking a step back.
• 4:27
  Speaker 2: We've talked in this in our podcast here about AI right generally, but before language models and before all of the big neural net revolutions, there was more basic machine learning systems, and machine learning at its very core went from AI that used to just be computer programs that humans wrote to take some data in and produce some useful insight out and those programs were authored by hand to machine learning models in the 90s that took a bunch of data and learned patterns in those data and were able to do something useful with them.
• 5:06
  Speaker 2: And in this particular paper, we're focusing more on predictive analytics and the idea here is that you have all of these different pieces of data right.
• 5:14
  Speaker 2: You just mentioned all these different factors, everything from the person's information to healthcare utilization, to geo, you know kind of other kinds of information, geo-environmental information.
• 5:24
  Speaker 2: All of that is relevant information potentially.
• 5:27
  Speaker 2: But you also know what you're trying to do is do some prediction with that and your prediction could be.
• 5:32
  Speaker 2: In this particular case it was one of three things it could be either the person is a no-show.
• 5:37
  Speaker 2: The person is, you know, going to cancel the appointment late under 24 hours I think is what they picked or is going to complete the appointment late under 24 hours I think is what they picked or is going to complete the appointment and show up right.
• 5:47
  Speaker 2: So this sort of task in the machine learning world at some level is called a classification task.
• 5:54
  Speaker 2: You're taking a bunch of information and you're deciding whether it fits into one of these three categories no show, completed late.
• 6:00
  Speaker 2: And a similar task would be like identifying whether an email is spam or not spam.
• 6:06
  Speaker 2: It's two classes spam or not spam and the email has all this text and other information that you're providing it.
• 6:11
  Speaker 2: So that's roughly the broad subtask that this paper is working on, and if you think about the data that's coming in, I think it's worthwhile.
• 6:23
  Speaker 2: Whenever you think about machine learning models, it's worthwhile to always think about data first, then the model it ends up being.
• 6:29
  Speaker 2: Actually, you know, 80 to 90% of the work in building a good machine learning model actually just isn't working with data and that's really messy.
• 6:37
  Speaker 2: It's you're getting your hands dirty working with really.
• 6:41
  Speaker 1: So dirty, yeah, dirty in the that's funny.
• 6:44
  Speaker 1: Dirty really.
• 6:46
  Speaker 1: So.
• 6:46
  Speaker 1: So dirty, yeah, dirty in the that's funny dirty in the compute computational sense.
• 6:48
  Speaker 1: I mean, you're talking to someone who you know was hands deep in bodily fluids all the time.
• 6:50
  Speaker 2: So when you talk about getting dirty with data yeah, well, um, but so dirty in that sense, in the sense that it's not nice clean programs, it's.
• 6:58
  Speaker 2: You're working and trying to clean up all the data to make it make it sort of palatable for the, for the machine learning models.
• 7:03
  Speaker 2: So let's talk a second about data.
• 7:04
  Speaker 2: So you have all the stuff about person's gender, maybe where they live, and all of that stuff, and there's different types of it.
• 7:10
  Speaker 2: So there's types of data that's categorical, that has specific answers to a question, that has specific categories for each entry.
• 7:20
  Speaker 2: There could be data that is just numerical you could have the temperature of the day and that's just a number and there's other kinds of, you know, particular kinds of data itself that goes into each of these fields.
• 7:31
  Speaker 2: So if you think of the data as a giant spreadsheet, each column is a different factor or feature, and we talked about different categories of factors before, but each one of those is a separate column and there's a separate column right at the end of this spreadsheet that is assigned for the prediction, which could be no show, late or completed, right.
• 7:54
  Speaker 2: So that's three categories, that's one column, that's the final answer.
• 7:58
  Speaker 2: But we have all these other columns of data and every row you can think of as a different patient.
• 8:02
  Speaker 2: And so you have all this information now in these different rows.
• 8:05
  Speaker 2: You have thousands upon hundreds of thousands of rows.
• 8:08
  Speaker 1: I don't actually yeah, there were over a million visits and they used most of them to train the model, but they think they tested the model on about 20%.
• 8:16
  Speaker 2: Yeah, and that is typically what people do is just to hold out about 20% of the data and then they train the model on the remaining so that the holdout data isn't so the model can be really tested properly and you're not just like testing on stuff that the model is already kind of memorized and so that 80% of the data is still a large amount of data that they have.
• 8:35
  Speaker 2: But this data is very heterogeneous.
• 8:37
  Speaker 2: Like I said before, there's different types of data, but they're all just big giant table right, and so there's that.
• 8:42
  Speaker 2: Then there's a separate issue of quality of data.
• 8:45
  Speaker 2: There might be missing elements here and there.
• 8:46
  Speaker 2: Maybe some information is dropped out.
• 8:48
  Speaker 2: That is like hard engineering work to try to get all of that right, because the models are only going to do as well as the data is.
• 8:55
  Speaker 2: So if the data is messy and has all these problems with them, the model isn't going to perform as well either.
• 9:01
  Speaker 1: Yeah, that makes sense.
• 9:03
  Speaker 2: And so if you want to think about the data we talked about, all these different factors there looks like in this paper at least, they use a couple dozen, maybe more factors, which is also called features, which I'll use interchangeably here.
• 9:18
  Speaker 2: But but the machine learning models are also dependent.
• 9:22
  Speaker 2: The performance of these machine learning models are also dependent on how many of those there are.
• 9:25
  Speaker 2: So there's all these factors, so let's get into the models themselves.
• 9:28
  Speaker 2: So broadly, the very simplest way to think about machine learning models is something called linear regression, and linear regression is….
• 9:35
  Speaker 1: I remember that from math class.
• 9:36
  Speaker 1: There you go.
• 9:37
  Speaker 2: But even if you haven't taken any math class, it's kind of easy to visualize.
• 9:45
  Speaker 2: Because if you had a chart, imagine a chart in front of you with a whole bunch of dots on it If you had a chart that had a bunch of dots that roughly look like a straight line, then you can draw a line right through those dots and say, okay, this line kind of represents my dots right.
• 9:56
  Speaker 2: And the tighter they are and closer they are to the line, the better the line fits those dots, the easier it is for me to draw the line Right.
• 10:03
  Speaker 2: But if you had it scattered everywhere, then you're not going to be able to draw a line as easily and it's more complicated.
• 10:07
  Speaker 2: But maybe it's not a line, maybe those dots are kind of a weird S shape, and then with your pencil you could draw an S shape covering most of the dots and then you know that you're pretty close to the right answer.
• 10:17
  Speaker 2: So that's machine.
• 10:19
  Speaker 2: Charting is essentially a big piece of it is doing that kind of work.
• 10:22
  Speaker 2: It's for the machine to automatically figure out what the slope, what the angle of that line is and where that line should live in that chart.
• 10:30
  Speaker 2: And instead of you having to figure it out.
• 10:33
  Speaker 2: It does it by just looking at the data and using some math.
• 10:36
  Speaker 1: But there's all these factors here.
• 10:38
  Speaker 1: It's not just like one factor that you're charting right, yeah, or they could interact with each other.
• 10:42
  Speaker 2: So now this is where the notion of dimensions come in.
• 10:46
  Speaker 2: And when I asked you to imagine a chart, that's just two dimensions, right, you have the vertical part of the chart and the horizontal part of the chart, and your dots, that kind of, are represented in that middle of that piece of paper, but you have multiple dozen factors, so it's a multiple dozen dimensional chart.
• 11:05
  Speaker 2: You can't imagine that in your head, nobody can.
• 11:08
  Speaker 1: I don't know what you're talking about.
• 11:09
  Speaker 2: Yeah, it's a wrinkle in time, right, but the thing is, you don't have to imagine that in your head.
• 11:14
  Speaker 2: That's why you have the math right, the math works out.
• 11:16
  Speaker 2: But that's essentially what they're trying to do is fit a line or whatever higher dimensional version of that is on those dots, now what they.
• 11:27
  Speaker 2: So linear regression just gives you a line.
• 11:30
  Speaker 2: But here what we want is we want it to say hey, this is a no-show, it is a cancellation, or it's a late, late cancellation, or, if it's a completion, right, we don't.
• 11:40
  Speaker 2: We don't want a line, we want three, one of three answers we want the output we want to figure out what's going to happen before it actually does.
• 11:46
  Speaker 2: We want to predict the future, but we don't want to predict.
• 11:48
  Speaker 2: It's not like predicting the cost of a house, where we have another number that we can put on that line.
• 11:54
  Speaker 2: It's not just a number, it's one of these three outcomes, right?
• 11:57
  Speaker 2: So what we want is something that tells us one of these three answers, and that's slightly different.
• 12:02
  Speaker 2: It requires a little different math, and that's one of the models that they use is called logistic regression, which is the same idea as linear regression, except that you also then have to decide, based on the line, whether or not it's one of these three categories.
• 12:14
  Speaker 2: The easiest way to look at it would be the line might tell you where on the chart it is, but you might have a threshold.
• 12:21
  Speaker 2: You might say you might cut it off and say above this number it's a yes and below this number it's a no, or above this number it's a no-show, between these numbers it's a cancellation.
• 12:34
  Speaker 2: So you might have to decide what the threshold is that makes it work.
• 12:36
  Speaker 2: Now the machine learning work for logistic regression is around that this paper also had other machine learning models.
• 12:42
  Speaker 2: I also want to just take a step back and throw more jargon at you.
• 12:46
  Speaker 1: Oh, thanks.
• 12:47
  Speaker 2: Yeah, that's what I was looking for, but a umbrella term for all of this stuff here that we're talking about is called supervised learning, and supervised just exactly means that you give it the right answer, you tell it.
• 12:58
  Speaker 2: Here's the thing that I want you to take in, and here's the type of output I want you to give me and I want you to take in, and here's the type of output I want you to give me, and I give you a whole bunch of right answers.
• 13:04
  Speaker 2: You know problems and right answers, data, and so that's called supervised learning.
• 13:08
  Speaker 1: Like I'm the supervisor or the data is the supervisor, the data is the supervisor in a sense.
• 13:12
  Speaker 1: Yeah.
• 13:14
  Speaker 2: And the machine updates how much to bend that line based on as it processes each piece of data, it updates that information and it tries to reduce the error in how it predicts the line and the way.
• 13:27
  Speaker 2: So that's logistic regression.
• 13:28
  Speaker 2: There's other models here and they do it in different ways.
• 13:30
  Speaker 2: There's this whole set of models called decision trees, which is like asking 20 questions.
• 13:35
  Speaker 2: So you, instead of going through and drawing a line, what you try to do is look at each of the factors and you decide on a threshold and you say if it's above, if the person is above 36, maybe they will be a no-show.
• 13:47
  Speaker 1: And then you imagine building out a whole 20 question, except for each of this, for each of these things, like what was their prior, like their prior no-show rate, uh is high, so they're likely to go down this tree and now I know if they had a late cancellation before then I'm going to keep going down the they're likely to no shower or cancel or whatever it is.
• 14:06
  Speaker 2: Right, and if they're above a certain age and they live near Paris and they do this, you can imagine going down that tree in that fashion and that's called a decision tree.
• 14:13
  Speaker 2: It's very powerful model actually.
• 14:15
  Speaker 2: It works really well for many cases and it allows interpretation because after the fact, if you get an answer, you can sort of trace the path that it took to come at that answer in a sense, and understand like what it took into account as it came up with the answer.
• 14:28
  Speaker 1: So it's not necessarily causal, but it's like explainable.
• 14:31
  Speaker 2: Like, we understand why this happened.
• 14:33
  Speaker 1: It's not some sort of black box out there.
• 14:35
  Speaker 1: We can figure out why we think this person is going to be a no-show, for example.
• 14:40
  Speaker 2: And just to be clear, even logistic regression is very explainable in the sense that for every factor, you assign a weight on how much you should be placing on that.
• 14:48
  Speaker 2: At the end of the training process, the machine has learned I need to weigh this factor so much, I need to weigh this other factor so much, I need to weigh this other factor so much so the model, but just by definition, will tell you how much it weighs every single factor.
• 15:01
  Speaker 2: So in a sense it's explainable.
• 15:03
  Speaker 2: And so, going back to decision trees, so that's like asking 20 questions.
• 15:08
  Speaker 2: Now imagine you had a jury of people, each of them asking 20 different questions, variations of these same 20 questions for different pieces of the data, and then at the end, taking a vote.
• 15:18
  Speaker 2: That's what random forests are.
• 15:20
  Speaker 2: The word forest comes from the fact that you have many trees.
• 15:24
  Speaker 2: Get it.
• 15:25
  Speaker 1: Yes, yes, I get it.
• 15:26
  Speaker 1: Haha, that was a dad joke.
• 15:29
  Speaker 2: Yeah, but that's it.
• 15:35
  Speaker 1: So you could ask the questions in different orders and then get slightly different results, and then you pull it all together.
• 15:38
  Speaker 2: And then again you learn what questions to ask, when to ask them, and so on, and so that's random forests and they're really powerful.
• 15:43
  Speaker 2: Gradient boost also works.
• 15:45
  Speaker 2: It works kind of differently.
• 15:47
  Speaker 2: Think of it as a series of coaches, where the first person kind of tells you roughly what to do in a very loose way, and you kind of do it, you make all these mistakes, and the second person looks at the mistakes you made and teaches you something different.
• 16:00
  Speaker 2: So in each iteration it's getting the errors that you made to make it slightly better, to make the model slightly better.
• 16:08
  Speaker 2: It's a little bit different in the way it works, but it's essentially that's.
• 16:12
  Speaker 2: The core idea is that you utilize the.
• 16:15
  Speaker 2: You kind of start kind of rough and then you kind of get finer and finer and more and more correct as you go on.
• 16:21
  Speaker 2: But you start with something that's kind of rough, if that makes sense, and so that's so those.
• 16:27
  Speaker 2: But gradient boost to some degree is doesn't care about the underlying model, although we do talk about it in terms of decision trees.
• 16:33
  Speaker 2: It could be applied to other models too, but that's kind of roughly the broad world of, and then you have neural networks.
• 16:41
  Speaker 2: Sorry, I didn't mention that because I think they did work with.
• 16:44
  Speaker 2: I think there's a.
• 16:49
  Speaker 2: I've thrown a lot at you.
• 16:50
  Speaker 2: Right now I don't know what you're talking about, but I think one thing to keep in mind as you process all of this is why do we need so many models Like?
• 17:00
  Speaker 2: Why do we need all of these different things?
• 17:02
  Speaker 2: Well, some of the things to keep in mind when machine learning engineers think about these things is, like I said before, the nature of the data.
• 17:08
  Speaker 2: How many lines of data do you have?
• 17:10
  Speaker 2: How many patients do you have?
• 17:11
  Speaker 2: How many lines of data do you have?
• 17:12
  Speaker 2: How many patients do you have?
• 17:13
  Speaker 2: How many features do you have?
• 17:14
  Speaker 2: Imagine I gave you an image and told you tell me if it's a cat or not.
• 17:16
  Speaker 2: If you think about the actual image, it's a bunch of pixels, and if it's a high resolution image, it's millions of pixels, where each pixel is red, green or blue.
• 17:26
  Speaker 2: Right, that's kind of how images are made up.
• 17:28
  Speaker 2: That's a million factors.
• 17:31
  Speaker 2: That's way too many for any decision tree or any of these other models to work.
• 17:35
  Speaker 2: Neural networks really shine in those scenarios, but because of that you lose interpretability, because they're all giant black boxes, so you don't have the benefit, whereas decision trees work really well in these settings where you have tabular data.
• 17:48
  Speaker 2: It doesn't matter how many data points you have, but you have a reasonable number of factors.
• 17:53
  Speaker 2: But you have something that is very heterogeneous.
• 17:56
  Speaker 2: If you think about the image again, it's all the same, right.
• 17:58
  Speaker 2: Every pixel is kind of the same.
• 17:59
  Speaker 2: It's not heterogeneous.
• 18:01
  Speaker 2: Every single pixel is either red, green or blue, whereas decision trees are better equipped to handle more heterogeneous data.
• 18:07
  Speaker 2: The reason gradient boost is doing really well in this model and even random forest is that there might be inherent interaction between the factors.
• 18:16
  Speaker 1: Yeah, of course they're not independent.
• 18:18
  Speaker 1: Yeah.
• 18:19
  Speaker 2: In a linear model they are all kind of separate.
• 18:21
  Speaker 2: So, like in a logistic regression model, you can say that remember I talked about putting it to a line.
• 18:26
  Speaker 2: But what if you can't put it to a line?
• 18:28
  Speaker 2: What if it's some complex shape that it actually has to capture?
• 18:30
  Speaker 2: Then you have interactions between these factors and logistic regression isn't as good anymore.
• 18:35
  Speaker 1: I mean that makes sense to me, right.
• 18:36
  Speaker 1: If it's like a hot day and I live far from the clinic and I have to walk there because my area deprivation index is, you know, a certain number, then I'm just going to be less likely to go.
• 18:47
  Speaker 1: So it does make sense that if we can capture the interactions between the factors, we can kind of understand it all better.
• 18:53
  Speaker 2: Yeah, and I think that's a way to think about why there exists so many different types of models is they solve slightly different problems with the data.
• 19:01
  Speaker 2: They're better suited for certain kinds of data, and often the trade-off is interpretability, and so you know that depends on the application.
• 19:11
  Speaker 1: Right, and the interpretability here is key, because the whole point is to understand why people don't show up or why people cancel late, so that you can intervene on that right, so that you can do something about it.
• 19:23
  Speaker 2: Yeah, no, exactly.
• 19:24
  Speaker 2: And I do want to take a second to talk also about how one judges these models.
• 19:31
  Speaker 2: How do we know a model is doing well versus not well?
• 19:34
  Speaker 2: And there's all these different metrics that people use.
• 19:37
  Speaker 2: People often use the term accuracy, Like how accurate is it?
• 19:40
  Speaker 2: Did it get it right?
• 19:41
  Speaker 2: But there's more to that term accuracy than meets the eye.
• 19:44
  Speaker 2: Right, you could have the model predict a no-show and in fact the person was a no-show.
• 19:51
  Speaker 2: Or you could have the model predict something else and the person was a no-show.
• 19:53
  Speaker 2: Or you could have the model predict something else and the person was a no-show.
• 19:55
  Speaker 2: Or you could have the model predict a no-show and the person was something else.
• 19:59
  Speaker 2: right, there's like different ways of going wrong, in other words.
• 20:02
  Speaker 2: So machine learning engineers use the terms true positives and true negatives to describe scenarios where the machine got it right, where the prediction and the underlying truth was the same.
• 20:17
  Speaker 2: True positive means machine predicted true and the underlying truth is also true.
• 20:25
  Speaker 2: I think an easy way to envision this is actually not with our example here, but with the spam email the machine, spam detector.
• 20:31
  Speaker 2: The machine predicted spam and it actually was spam.
• 20:33
  Speaker 2: The machine, spam detector.
• 20:34
  Speaker 2: The machine predicted spam and it actually was spam.
• 20:36
  Speaker 2: Versus the machine.
• 20:36
  Speaker 2: And that's a true positive and a true negative.
• 20:38
  Speaker 2: Is machine predicted not spam and it really was not spam.
• 20:41
  Speaker 2: But there's also false positives, which is the machine predicted spam but it actually wasn't, and the other way around.
• 20:49
  Speaker 2: So what that means all of that is to say that there is other metrics like that that the machine learning engineers use to build on these core ideas.
• 20:57
  Speaker 2: They use things like precision and recall and all of these ideas and again, all of these ideas.
• 21:04
  Speaker 2: They might seem really complex, but at the very core of it, all they're trying to do is decide if whatever the machine is predicting, first of all, is that right, all the spams that it's predicted?
• 21:14
  Speaker 2: Are they really all spams and all the spams that exist out there?
• 21:18
  Speaker 2: Is it really catching all of them?
• 21:20
  Speaker 2: And they're after these sorts of metrics and they use fancy terms like position and recall to capture these ideas.
• 21:26
  Speaker 2: But that's the very core of it.
• 21:28
  Speaker 2: But they don't stop just there.
• 21:30
  Speaker 2: Remember I mentioned before that we're not just drawing a line, we're trying to like decide where to put the threshold to decide if something qualifies as spam versus not spam, and you know the sort of when, where to draw the line.
• 21:44
  Speaker 2: That line is arbitrarily drawn at some level.
• 21:47
  Speaker 2: And what you know in this paper they've done a good job of is to say, okay, what if we just pick various lines?
• 21:52
  Speaker 2: We just drew all kinds of lines and we, we calculated for each of those lines, we just we evaluated how well the model did, and then we draw a graph out of that and say, okay, I've tried all of these different thresholds, all of these different ways to cut up my data.
• 22:19
  Speaker 2: Does the model do well regardless?
• 22:20
  Speaker 2: And so that's one of the big, I think one of the nice pieces here in this paper as well, which is a standard practice, but I think it's nice to note, because what it does for you, it tells you that can the model actually discriminate between those three cases, in our case the no-show, the cancellation and the completion?
• 22:31
  Speaker 2: How well can it actually discriminate between those three cases, in our case the no-show, the cancellation and the completion?
• 22:33
  Speaker 2: How well can it actually discriminate between them?
• 22:36
  Speaker 2: And so you need to be able to think about these other kinds of mathematical measures, and that's again.
• 22:44
  Speaker 2: These papers are filled with mathematics for that reason, because they want to get at these very core ideas.
• 22:49
  Speaker 2: How accurate is it?
• 22:51
  Speaker 2: Does it catch everything I want to catch and can it really tell apart these different groups Right?
• 22:59
  Speaker 1: And I think they did.
• 23:01
  Speaker 1: Actually, when you look at the factors, they really were able to figure out what the most influential factors were.
• 23:09
  Speaker 1: The number one most influential factor drumroll please was actually the scheduled lead time.
• 23:17
  Speaker 1: So if you schedule an appointment and it was more than 60 days out, you were much less likely to show up for it, which I mean, I don't know, I've definitely missed things that were pretty far out of my calendar, and you can imagine perhaps that if someone had such a non acute issue, maybe it went away by then or they just, you know, it fell off their radar as being something important or whatever it was.
• 23:41
  Speaker 1: So schedule lead time was the number one factor, but there were a lot of other factors they found at sort of the population level, like how long a provider had been practicing, what was the age of the patient?
• 23:53
  Speaker 1: Was the visit with the patient's own PCP?
• 23:57
  Speaker 1: I think that one's interesting because I really like seeing my own PCP.
• 24:00
  Speaker 1: I don't like seeing other people in the practice, right?
• 24:05
  Speaker 1: Other things that were important factors here were whether or not someone had gone to their other appointments, right?
• 24:12
  Speaker 1: Were they a no-show in the past or had they canceled in the past because they were more likely to then again.
• 24:18
  Speaker 1: And then the geosocial context turned out to be important, right when they lived, what their area deprivation index was, how far it was from their home to the clinic, and, like I mentioned before, environmental factors like the temperature played a role as well, and so they were able to see, okay, at the population level.
• 24:34
  Speaker 1: There are all these factors that are at play, but I think one of the things that was very interesting to me is not only could they see that across the whole population of the more than 100,000 patients they looked at, but they were actually able to drill down and say oh at.
• 24:54
  Speaker 1: But they were actually able to drill down and say oh, vasant, like you didn't go to your appointment or you were a late cancellation, and we know why for you and your reason wasn't the lead time, because your lead time was short, your reason was the temperature and the distance from the clinic or whatever it was.
• 25:06
  Speaker 1: They were able to, like, drill down into individuals and figure out which of those features, which of those factors, were the most important.
• 25:14
  Speaker 2: Yeah, and that is something that you know is called the Shapley value, which is a mathematical term, and you know it's one thing that I love about this is that it's getting at the, it's breaking apart the features.
• 25:29
  Speaker 2: We talked about all the factors, but identifying which ones played the role to make the model predict what it predicted is what Shapley calculations are all about and that's what they use to determine what you just said and what I like about the Shapley value is that the way it does it is again going back to our spreadsheet and having all of the factors to be different columns.
• 25:49
  Speaker 2: What they essentially do is try to see how accurate the model is.
• 25:53
  Speaker 2: If I just took out a certain column and they play around with that, I'm sort of oversimplifying, but the basic idea is that you take columns out and then you see what happens.
• 26:02
  Speaker 2: And then you take other columns out and you see what happens.
• 26:04
  Speaker 2: You take groups of columns out and see what happens and you sort of average out all of that and then you realize, okay, based on all of these, these are the factors that really contributed to the globally to the model.
• 26:17
  Speaker 2: That means I'm not talking about specific patients here, but broadly looking at all of the data that the model has looked at.
• 26:23
  Speaker 2: It's making its predictions highly influenced by some of these.
• 26:28
  Speaker 2: These are the following factors that it's making its decisions on, you know, highly influenced by these.
• 26:32
  Speaker 2: So that's at the global level.
• 26:35
  Speaker 2: But, like you said, the same Shapley value can also be used for an individual patient.
• 26:40
  Speaker 2: So if I come in with a new piece of data, I have this trained model ready to go.
• 26:43
  Speaker 2: It's sitting on my table and I come in and I put a new line, new row in that spreadsheet for a new patient.
• 26:51
  Speaker 2: I can do a Shapley type calculation there too, where I literally change a column.
• 26:58
  Speaker 2: This is how they do it.
• 26:59
  Speaker 2: Actually.
• 26:59
  Speaker 2: They take one of the columns that they're interested in.
• 27:02
  Speaker 2: They do this for every single column, but they pick one column and then they just randomly replace it by some other value from some other piece of data from their existing data set and then they make variations, perturbations of that data point and then see what happens to the accuracy and the prediction.
• 27:20
  Speaker 2: Does the prediction change, how much does the prediction change by?
• 27:23
  Speaker 2: And so that's kind of how they're able to do that.
• 27:26
  Speaker 2: But that's right.
• 27:27
  Speaker 2: But I do want to note that it is not the case that the model is saying hey, you showed up late because you don't go out in hot days or whatever.
• 27:40
  Speaker 2: Right, whatever the factor was, that was deemed.
• 27:42
  Speaker 1: Temperature yeah, yeah.
• 27:43
  Speaker 2: Temperature right.
• 27:43
  Speaker 2: I'm not saying that that is the factor, but just for hypothetical sake.
• 27:46
  Speaker 2: Yeah, but that's not what the model is saying.
• 27:49
  Speaker 2: The model is not saying it.
• 27:50
  Speaker 2: That's not a causal model.
• 27:51
  Speaker 2: It's not a causal prediction no-transcript.
• 28:22
  Speaker 2: The cause no, and it's a.
• 28:24
  Speaker 2: It's a factor in relation to the trained model that was trained on all the other data that was there before.
• 28:29
  Speaker 1: Which, like, makes sense, right, the the fact, some of the factors, like having a no-show in the past.
• 28:36
  Speaker 1: It doesn't cause you to have a no-show now, right, but it does maybe explain, if you've had no-shows before, that you might be more likely to have one again.
• 28:46
  Speaker 2: Yeah, and the purpose of the Shapley values is to explain the model predictions so you understand the model better.
• 28:54
  Speaker 2: It is not meant to serve as a explainer of the human behavior involved.
• 29:00
  Speaker 1: Yeah, I mean that makes sense.
• 29:02
  Speaker 1: But at the same time my mind is going to okay, like, what do we do with this information?
• 29:06
  Speaker 1: What do we do with this population level information?
• 29:08
  Speaker 1: What do we do with this individual level information?
• 29:11
  Speaker 1: And so there are certain things like age was found to be important in the model.
• 29:15
  Speaker 1: Okay, we can't change their age, right.
• 29:17
  Speaker 1: But there are other things like the lead time to appointment, where, okay, maybe it's not causal, but like, what if there's?
• 29:25
  Speaker 1: Maybe, what if it is a little bit right?
• 29:27
  Speaker 1: And what can we do about it?
• 29:29
  Speaker 1: Because that's always my question is how can we reduce the no-shows, how can we reduce the late cancellation?
• 29:34
  Speaker 1: So what if, in this case, a clinic tried out and said, hey, we're going to make sure we book all our appointments for this subpopulation within 60 days and see if that changes our no-show or late cancellation rate?
• 29:51
  Speaker 1: Or let's see if we can get people in with their own PCPs better rather than having cross-coverage.
• 30:02
  Speaker 1: Or maybe, when you think about clinic operations, perhaps we can't predict the hot day or whatever.
• 30:12
  Speaker 1: But there are situations where you could say, hmm, I'm expecting a lot of no-shows or late cancellations, maybe we can squeeze an extra patient in here.
• 30:22
  Speaker 1: So, even though they're not causal, it's like maybe they're actionable or you could experiment with pieces of this and try to figure out is there something we can optimize here?
• 30:33
  Speaker 1: Can we get you know?
• 30:35
  Speaker 1: One of the things we didn't talk too much about was like could we get you know an interpreter of someone who doesn't speak English is less likely to show up, or could we get transportation for someone who who's you know, has a hard time getting into clinic?
• 30:46
  Speaker 1: All of these things are associated with it.
• 30:48
  Speaker 1: So, even though it doesn't prove it's causal, in some cases, like in my mind, I'm like well, yeah, but it could be, yeah yeah, yeah, no, absolutely.
• 30:56
  Speaker 2: I think that it serves as at least ideas, right, ideas for potential interventions, and it's funny because it's meant to be a predictive model, but maybe it's a descriptive model of what's happening and at some weird level and that's kind of what you're suggesting at these correlations could mean something.
• 31:13
  Speaker 2: We don't know yet, but it's worthwhile investigating them.
• 31:15
  Speaker 1: Right, yeah, so I think this is a great paper because it really does show, you can predict for the most part who's going to miss or not show up, who's going to be a late cancellation.
• 31:28
  Speaker 1: And these machine learning models really help, especially when we give them lots of cleaned up, diverse data sets that involve more than just what's in the electronic health record and they don't quite explain why.
• 31:42
  Speaker 1: But you know, we could try to take some of that information and apply it at the population level, at the patient level, at the clinic level, to intervene and try to reduce the number of no-shows and late cancellations.
• 31:58
  Speaker 2: Yeah, I'd be very curious to see what the authors here move forward with.
• 32:03
  Speaker 2: Do they build something from it and work with that and actually do those next steps that you just talked about?
• 32:08
  Speaker 1: Yeah Well, thanks so much for joining us.
• 32:10
  Speaker 1: Thank you.
• 32:11
  Speaker 1: See you next time.