Latest / Code & Cure / #41 - If You Cannot Trace The Data, Do Not Trust The Model

Transcript

#41 - If You Cannot Trace The Data, Do Not Trust The Model · Code & Cure

• 0:00
  SPEAKER_01: What if the model guiding a clinical decision wasn't wrong because of the math, but because no one really understood the data behind it?
• 0:19
  SPEAKER_00: Hello and welcome back to Code and Cure, the podcast where we discuss decoding health in the age of AI.
• 0:25
  SPEAKER_00: My name is Vassant Sarathi.
• 0:27
  SPEAKER_00: I'm a cognitive scientist and AI researcher, and I'm with Alara Hagopian.
• 0:31
  SPEAKER_01: I'm an emergency medicine physician.
• 0:34
  SPEAKER_01: And today we're going to talk about this concept of data provenance and the idea that if you train models on crappy data, then you get crappy things out of them.
• 0:48
  SPEAKER_00: Yeah, this is the whole garbage in, garbage out world, right?
• 0:50
  SPEAKER_00: That's always been said about computers in general, you know, in general.
• 0:53
  SPEAKER_01: Yeah, exactly.
• 0:55
  SPEAKER_01: And when you think about applying this in a clin with a clinical lens, it's like, okay, well, the garbage out then means like you're making clinical decisions for someone based on garbage, which is like, ugh, not good.
• 1:10
  SPEAKER_00: Yeah, yeah, yeah, yeah.
• 1:11
  SPEAKER_00: I mean, ultimately a lot of these um machine learning AI models are meant to make our lives easier um by allowing machines to do the things that we um, you know, routinely do and speed it up and scale it up and so on.
• 1:25
  SPEAKER_00: But if they're wrong, then that's obviously a problem.
• 1:28
  SPEAKER_01: And it's not necessarily a problem with the model itself, right?
• 1:30
  SPEAKER_01: It's like the problem is in the training data.
• 1:32
  SPEAKER_01: So, you know, you have tons and tons and tons of data available, and you're like, okay, well, let's like crunch these numbers, let's find the patterns, let's predict something, let's uh make decisions based on them.
• 1:43
  SPEAKER_01: But you have to make sure all that data that you're putting into the system is actually like good and correct.
• 1:50
  SPEAKER_00: That's right.
• 1:51
  SPEAKER_01: And if you don't, then you end up with a a big problem.
• 1:54
  SPEAKER_00: Yeah, yeah.
• 1:54
  SPEAKER_00: And there's many dimensions to what makes a data piece of data good, correct, a reliable, all these words, right?
• 2:02
  SPEAKER_00: And you know, when you think about clinical data, you're thinking about concretely what you're thinking about when you see data is a file on a computer with numbers and other things on it.
• 2:14
  SPEAKER_00: And usually data like this has records.
• 2:17
  SPEAKER_00: So you have maybe a big table that has a whole bunch of different rows where each row provides information about a patient, about their various physiological measures, maybe, or some other things, and then some kind of label that says this patient has a certain disease or not.
• 2:33
  SPEAKER_00: And that's a very generic uh type of data set, which um, you know, you would have, you would train a machine learning model to predict that label, that thing, that disease or whatever, based on the other factors that were provided to it.
• 2:47
  SPEAKER_00: But all of those rows in that table came from somewhere.
• 2:52
  SPEAKER_01: Hopefully from someone, right?
• 2:53
  SPEAKER_01: Hopefully they were made up.
• 2:54
  SPEAKER_01: It's like real.
• 2:55
  SPEAKER_00: They were real human.
• 2:57
  SPEAKER_00: Because that's the idea.
• 2:57
  SPEAKER_00: You want because ultimately, again, the point is the machine learning model needs examples.
• 3:02
  SPEAKER_00: Uh, and by an example, I mean it needs input and output examples to learn how to go from the input to output.
• 3:08
  SPEAKER_00: What's the pattern, you know?
• 3:10
  SPEAKER_00: And so the that's the basis of all of data.
• 3:14
  SPEAKER_00: And what makes a good data set has many different meanings to it.
• 3:18
  SPEAKER_00: One meaning is that whatever data set you have is in some sense finite.
• 3:24
  SPEAKER_00: Maybe you have 220 ex uh rows, maybe you have two million rows, but the data set has a finite number of things in it.
• 3:32
  SPEAKER_00: And so, based on that finite number of things, you're gonna have to predict in the future other things that are not in the data set, right?
• 3:37
  SPEAKER_00: That's the model's job to generalize what's in the data set, learn the patterns, and then apply it to a new example.
• 3:43
  SPEAKER_00: And that new example uh needs to be the kind of data that you've looked at in your 20 or 2 million size data set.
• 3:51
  SPEAKER_01: And we've talked about this before, right?
• 3:53
  SPEAKER_01: Like if you have this huge data set, you might train on a chunk of it.
• 3:56
  SPEAKER_00: Yeah.
• 3:57
  SPEAKER_01: And then you would test on some part of the remainder that you haven't trained on.
• 4:01
  SPEAKER_01: Like you set it aside from the beginning.
• 4:03
  SPEAKER_01: Yeah.
• 4:04
  SPEAKER_01: Um, and then you're like, okay, well, I want to test, test on the rest of this set and make sure it's like functioning as we intended it to function.
• 4:12
  SPEAKER_00: Yes.
• 4:12
  SPEAKER_00: But even in that case, once you have decided the model works really well and tested it on that what's what's called a holdout data set, uh holdout part of the data set.
• 4:23
  SPEAKER_00: Um, once you've done that, you have you you say you come out and say, Look, I have this box, I have this model, it works.
• 4:28
  SPEAKER_00: Then you go out there and put it out in the real world.
• 4:31
  SPEAKER_00: And now you're getting real information coming in.
• 4:33
  SPEAKER_00: And you have to, you have to be sure that that information has the same shape and the same behavior as both your original training set and the thing that you left out.
• 4:42
  SPEAKER_00: All of that stuff was done before you put this model out in the world.
• 4:45
  SPEAKER_01: And of course, like, and not surprisingly, if you've trained on data that was maybe some from some one region of the world, like from Asia, it may not apply that well, say in the United States.
• 4:58
  SPEAKER_01: Yes, yes.
• 4:58
  SPEAKER_01: And that's that's something you always have to keep in the back of your mind, which is you know, we need to apply it in the same situation, or it may not be generalizable to every population.
• 5:11
  SPEAKER_01: And so you have to be careful, careful when you start to apply it.
• 5:14
  SPEAKER_00: Machine learning researchers use these uh use jargon words like uh out of distribution to suggest something uh as a data point that is not something within the learning space that the machine was meant to learn, something that's outside of it.
• 5:27
  SPEAKER_00: And the and and the classic example of this is the uh there was a study that was done with COVID lungs, and the data was X-ray images of COVID lungs, and the um the label was either person has COVID or the person doesn't have COVID.
• 5:43
  SPEAKER_00: And the images were shown, and you know, it trained on that data set, and then someone decided to show it a image, which is also, you know, it's the same format, right?
• 5:52
  SPEAKER_00: An image is just a computer file.
• 5:54
  SPEAKER_00: So somebody gave it an image of a cat, and it was 100% confident that this was a COVID lung.
• 6:00
  SPEAKER_00: Now that sounds ridiculous to a human, right?
• 6:02
  SPEAKER_00: You look at a picture of a X-ray image of a lung and you look at anything to do with it, but that's not what the the the the uh model learned.
• 6:12
  SPEAKER_00: The model learned given these pixels and these arrangements, this is the prediction I can make.
• 6:17
  SPEAKER_00: This is the pattern I see.
• 6:18
  SPEAKER_00: So the pattern that it's seeing in the pixels of the cat is based on that.
• 6:23
  SPEAKER_00: But obviously, the cat is out of distribution.
• 6:25
  SPEAKER_00: That is not the type of pattern you care about.
• 6:27
  SPEAKER_00: If you are interested in the task of detecting cats, then you wouldn't be giving it COVID lungs to train on.
• 6:33
  SPEAKER_00: And so there's a mismatch between what was in the data and what was out there.
• 6:37
  SPEAKER_00: That's like one set of issues with data, right?
• 6:39
  SPEAKER_00: There's the issue of having the machine learning model learn something very specific and then put it somewhere else where it's not meant to be.
• 6:46
  SPEAKER_00: But that's different from even if you had a situation where you had these COVID lungs images and then you had another image of a lung, you could still have data problems if that original image or that original data set was unreliable in some other way.
• 7:04
  SPEAKER_01: Well, that's that's actually what I was gonna ask you.
• 7:06
  SPEAKER_01: I was gonna ask, like, you were talking about, okay, maybe there's 20 rows in a table versus two million rows in the table.
• 7:12
  SPEAKER_01: Well, like, how do you even get two million rows?
• 7:14
  SPEAKER_01: You mean you may not even be able to get that much healthcare data, especially because uh, you know, it's really hard, it you have to de-identify everything and you have to make sure that like a patient can't be re-identified, all this stuff.
• 7:27
  SPEAKER_01: So I'm curious, like, do you do people ever sort of simulate or fabricate or I don't know, like in in in any way make up data to help create these training sets?
• 7:43
  SPEAKER_00: Yes.
• 7:43
  SPEAKER_00: So even in this world where we have the right domain, we're not testing cats on on COVID lungs, even in this world, you have uh a bunch of images of COVID lungs.
• 7:53
  SPEAKER_00: Um, and maybe you don't have enough images for machine learning model.
• 7:57
  SPEAKER_00: These machine learning models need a certain amount of data to be able to learn uh what they need to learn, but just maybe you don't have enough.
• 8:04
  SPEAKER_00: And this happens actually a lot in um not in images necessarily, but in the space of language, where maybe you have examples of some um small culture of folks and there's just not enough data in their specific uh specialized language.
• 8:20
  SPEAKER_00: And so that's an example where you have um, you know, it's called a low resource language where you just don't have as much coverage.
• 8:28
  SPEAKER_00: There's a lot of English data out there, but there might not be a lot of this specific language out there.
• 8:33
  SPEAKER_00: And so what people end up doing is finding ways to augment the data.
• 8:36
  SPEAKER_00: And they do that sometimes with the help of AI systems to create new data items like the ones that exist in order to kind of um bulk up the data set so that the machine learning model can learn uh the right or learn as much as they can.
• 8:50
  SPEAKER_00: Now, this is a measure to take when you don't have enough data and there's nothing else you can do except hopefully try to collect more data, but potentially if you don't have it, then you have to find some other way to train the model.
• 9:02
  SPEAKER_01: Okay, but is that like I I mean, I can like understand why it needs to be done, but is that bad?
• 9:07
  SPEAKER_01: Like, are you training it on like fake data?
• 9:10
  SPEAKER_00: Yeah, I mean, again, it comes down to the distribution of data.
• 9:13
  SPEAKER_00: If that small, tiny 20 uh item data set has a certain behavior, certain pattern to it, and your augmenting process creates, you know, 200,000 more items of it of the same type of process, then the question is how good is that?
• 9:30
  SPEAKER_00: The augmenting process might be fine if, in fact, in the real world, that 20 samples was really good.
• 9:36
  SPEAKER_00: Was really representative.
• 9:37
  SPEAKER_00: Right.
• 9:37
  SPEAKER_00: Was really representative.
• 9:39
  SPEAKER_00: And so again, this goes back to other data issues where you have a data set that even if you had forget synthetic data or uh augmented data or artificial data, if you had real data, if you had 200,000 real data items, there could still be problems if all of the uh patients in the data set were white males, then what you have is a data set that is essentially like the lung cat situation, where it's great at predicting um, you know, for for white males, but it might not be great for predicting for black females, right?
• 10:11
  SPEAKER_00: And so you've now biased the data set has some certain biases to it.
• 10:15
  SPEAKER_00: And that's another piece of the data that needs to be quote unquote good, right?
• 10:20
  SPEAKER_00: There has to be, it has to be representative, real world data, but it also has to have coverage.
• 10:24
  SPEAKER_00: It needs to be uh, you know, uh there are limitations of what it actually captures.
• 10:29
  SPEAKER_00: Ultimately, we're not the data set doesn't contain every single human in the world.
• 10:32
  SPEAKER_00: Right, of course not.
• 10:33
  SPEAKER_00: So ultimately, we're making some guesses as to the balance and we try to our best to balance the data.
• 10:38
  SPEAKER_00: It needs to be accurate.
• 10:39
  SPEAKER_00: That is, it needs to actually be, you know, even in this case of synthetic generation, if it's producing medical lines of data that are medical lines but actually don't make sense, then there's a problem there too.
• 10:51
  SPEAKER_00: Like there, maybe there's some medical measure that doesn't correlate with another.
• 10:56
  SPEAKER_00: Um, but because the data was synthetically generated and because the system didn't know to look for that, it ended up being correlated, which would be weird for a doctor to see that, right?
• 11:05
  SPEAKER_00: But the system doesn't know that.
• 11:06
  SPEAKER_00: So that's an issue with the artificially generating data as well.
• 11:09
  SPEAKER_01: Yeah.
• 11:10
  SPEAKER_01: So it's interesting because what they did in this paper is they said, okay, we're gonna look at some of the common data sets that get used in a lot of clinical papers.
• 11:19
  SPEAKER_01: Like they're used in review articles, they're used in clinical prediction studies.
• 11:24
  SPEAKER_01: These are like very commonly used data sets and they're about common conditions, stroke, diabetes.
• 11:30
  SPEAKER_01: Yes.
• 11:30
  SPEAKER_01: And let's like figure out if they have good data prop providence.
• 11:36
  SPEAKER_01: Let's figure out if it's clear, we understand where it's coming from, if it's authentic.
• 11:40
  SPEAKER_01: And they actually went through a checklist um for two data sets that were used in a bunch of papers and prediction models.
• 11:49
  SPEAKER_01: And they tried to figure out, hey, like, is are these are these good?
• 11:54
  SPEAKER_01: And what were what were some of the things they looked at?
• 11:56
  SPEAKER_00: Yeah.
• 11:56
  SPEAKER_00: So they looked at, you know, things like um the sources of the data, how they were developed, um, were um, you know, how they were evaluated, you know, was there randomized trials?
• 12:07
  SPEAKER_00: Was it what were the cohorts?
• 12:09
  SPEAKER_00: Um, you know, how did they uh did they register the data?
• 12:12
  SPEAKER_00: How representative is the data?
• 12:13
  SPEAKER_00: Um, and then specific things about the data set collection process.
• 12:17
  SPEAKER_00: When was it collected?
• 12:18
  SPEAKER_00: What was the start and end?
• 12:19
  SPEAKER_00: Who were the participants?
• 12:21
  SPEAKER_00: What was the demographics?
• 12:23
  SPEAKER_00: Um, what was the setting of the studies?
• 12:25
  SPEAKER_00: Uh, what kind of quality controls did they do up front um to make sure that the data was good?
• 12:30
  SPEAKER_00: How much did they, you know, did they actually test if the um the data set was going to be uh predictive of the of the sample of people and a whole host of other things like that?
• 12:42
  SPEAKER_00: And um, what's the size of the study and so on and so on?
• 12:46
  SPEAKER_00: If there was any missing data, what how did how was that handled?
• 12:49
  SPEAKER_00: What what would what were the you know the missing spots filled with?
• 12:52
  SPEAKER_00: Um or were they left empty?
• 12:54
  SPEAKER_00: And so there's a whole host of questions.
• 12:55
  SPEAKER_00: And data provenance is just a fancy way of saying I need to understand the origin and the context around that data data set.
• 13:03
  SPEAKER_01: Like, do we think it's reliable essentially?
• 13:06
  SPEAKER_01: And the stuff that you were just reading off, it it feels like any data set should have.
• 13:12
  SPEAKER_01: It's like, where did the data come from?
• 13:13
  SPEAKER_01: Like, when was it collected?
• 13:16
  SPEAKER_01: Um, what are we predicting?
• 13:19
  SPEAKER_01: All of those things seem like normal things you would expect to have in a data set.
• 13:25
  SPEAKER_00: Yes, and machine learning research usually begins by like stating a problem that you're um that you're trying to solve as a machine learning problem.
• 13:35
  SPEAKER_00: And machine learning problems have a very specific format, which is you state what the shape of the input is, and then you shape you, you you you try to state what the shape of the output is, and then you try to come up with some ideas for models for how the input can be uh representative of the output.
• 13:49
  SPEAKER_00: And then you that's kind of the rough idea of design of the of the models, but you often start with this broad issue and you um you try to build a data set and you try to build one that makes sense.
• 14:02
  SPEAKER_00: Now, a lot of research um also depends on other people's uh data sets, right?
• 14:07
  SPEAKER_00: I mean, that's that's normal.
• 14:08
  SPEAKER_00: You you have others or experts who are building data sets and they're writing papers about the data sets that they're building and its utility.
• 14:15
  SPEAKER_00: And those data set papers actually are very useful because what they also do is they um describe the provenance of the data set and they do some baselines that that is they they train some models on that data set and tell you how those models behaved.
• 14:30
  SPEAKER_00: So that tell those are really good data set papers, and then other researchers can use those data sets for their own models and cite those papers, right?
• 14:38
  SPEAKER_00: So that's typically how you know that's ideally how this is done.
• 14:42
  SPEAKER_00: Now, there are websites, um, like for instance, the one used in this paper called Kaggle.
• 14:47
  SPEAKER_00: Um, and Kaggle is a very popular source of data.
• 14:51
  SPEAKER_00: Now, it's anybody can upload a data set into Kaggle.
• 14:54
  SPEAKER_00: Like I can just upload one.
• 14:55
  SPEAKER_00: Yes, and I can say this is a diabetes data set.
• 14:58
  SPEAKER_00: You can.
• 14:59
  SPEAKER_00: And, you know, and so and some data set and and and what was mainly used for is to help uh new and upcoming machine learning um, you know, students practice their skills on on various types of data sets.
• 15:12
  SPEAKER_00: But it's also now sort of it hosts both kind of artificial data sets, but also real ones, and they're kind of overlapping.
• 15:18
  SPEAKER_00: And and so some of the and so the two data sets they used in um this paper, one for stroke and one for diabetes, are uh on Kaggle and they are heavily used.
• 15:30
  SPEAKER_00: I mean, the last I checked, there was like 290,000 downloads and increasing, like it's going up.
• 15:35
  SPEAKER_01: That's right.
• 15:36
  SPEAKER_01: Yeah.
• 15:36
  SPEAKER_00: That's right.
• 15:37
  SPEAKER_00: And people can comment on them and you know, and so on.
• 15:40
  SPEAKER_00: But and so this study, they looked into those data sets and looked at, okay, what was there?
• 15:46
  SPEAKER_00: What questions did they answer the questions on the data provenance, all the things that I just mentioned?
• 15:50
  SPEAKER_01: No, they answered zero of zero of them.
• 15:53
  SPEAKER_01: There's this whole checklist that they went through and they were like, oh, do we know the source of the data?
• 15:58
  SPEAKER_01: Do we know the date it was collected?
• 16:01
  SPEAKER_01: Do we know the setting it was collected?
• 16:03
  SPEAKER_01: Um, do we know any sort of pre-processing or quality checking?
• 16:07
  SPEAKER_01: You know, do we know the outcome that's being predicted?
• 16:10
  SPEAKER_01: Do we know how missing data was handled?
• 16:12
  SPEAKER_01: I could go on and on, but there was like a nine-item checklist that they went through.
• 16:16
  SPEAKER_01: Zero.
• 16:16
  SPEAKER_01: Like zero of those items were there.
• 16:19
  SPEAKER_01: So it's like we don't know the data provenance.
• 16:21
  SPEAKER_01: And in in one of the data sets, they actually said, hey, this is for educational purposes only.
• 16:27
  SPEAKER_01: It should not be used for research.
• 16:30
  SPEAKER_01: It should not be used for for um economical purposes either.
• 16:34
  SPEAKER_01: And so it's like, well, if it's just for people trying to mess around and build a model, that's one thing, it's not supposed to be used for clinical prediction.
• 16:44
  SPEAKER_01: That's correct.
• 16:44
  SPEAKER_00: And it is being used for clinical prediction.
• 16:46
  SPEAKER_00: Well, that's the next step, right?
• 16:47
  SPEAKER_00: So the uh the there's a whole bunch of papers that the authors in this in this paper identified that used those data sets.
• 16:54
  SPEAKER_01: Exactly.
• 16:54
  SPEAKER_01: And so now if we're making decisions based on data that we don't know where it came from and we don't know if it's good.
• 17:01
  SPEAKER_00: Yes.
• 17:02
  SPEAKER_00: It's not a good thing.
• 17:03
  SPEAKER_00: And I think it was really shocking to me a couple of things, right?
• 17:06
  SPEAKER_00: One was the data set had some inconsistencies, like medical inconsistencies.
• 17:09
  SPEAKER_01: Yeah, and we can we can talk about that because there were definitely some irregularities when they started to look at the data.
• 17:15
  SPEAKER_01: It was like just kind of weird patterns.
• 17:17
  SPEAKER_01: So, first of all, when you have like a real world data set, it's always missing a ton of information, right?
• 17:23
  SPEAKER_01: It's like, oh wow, super messy.
• 17:25
  SPEAKER_01: It's messy.
• 17:26
  SPEAKER_01: It's and and and it's like that's frustrating.
• 17:28
  SPEAKER_01: I find that frustrating, but that's like something that I would predict to happen.
• 17:33
  SPEAKER_01: Like if you're like, oh, I wanna have, I want to make sure that we have weights on everyone and heights on everyone so we can calculate their BMI.
• 17:40
  SPEAKER_01: And you have like 50,000 people in your data set, there's no way you're gonna have heights and weights and BMIs on every single person.
• 17:46
  SPEAKER_01: There's just like no way.
• 17:48
  SPEAKER_01: Um, and so pretty much no data points were missing in this data.
• 17:53
  SPEAKER_00: That's incredible.
• 17:54
  SPEAKER_00: Yeah.
• 17:54
  SPEAKER_01: I mean, it it's like it makes me think that it's not real.
• 17:58
  SPEAKER_00: Or they handle the missing data without telling you how.
• 18:01
  SPEAKER_01: Yes, exactly.
• 18:02
  SPEAKER_01: So, you know, we would have to have like a reason.
• 18:05
  SPEAKER_01: If they were omitting a person because they didn't have data, that actually changes your data set significantly because there might be some sort of pattern.
• 18:13
  SPEAKER_01: Yep.
• 18:14
  SPEAKER_00: There's another variable, another reason for why those that that's exactly.
• 18:17
  SPEAKER_01: Like maybe we don't have a weight on someone because there was no scale in the clinic in a low uh income neighborhood, or there was, you know, we didn't have the heights on, you know, 10,000 people.
• 18:27
  SPEAKER_01: So we just got rid of those.
• 18:29
  SPEAKER_01: That that data, like that doesn't does does not compute.
• 18:32
  SPEAKER_01: You're you're changing uh what your your output's gonna be by doing that.
• 18:37
  SPEAKER_00: Right.
• 18:38
  SPEAKER_01: But then there was some other weird stuff um in this data where, for example, okay, so blood glucose is measured in the moment, like you prick your finger and you can check it.
• 18:51
  SPEAKER_01: Um A1C, a hemoglobin A1C is a kind of a measure of blood glucose over the course of three months.
• 18:58
  SPEAKER_00: Okay.
• 18:59
  SPEAKER_01: And so in your mind, would you expect those to be correlated?
• 19:04
  SPEAKER_00: Yeah.
• 19:04
  SPEAKER_01: Yeah, exactly.
• 19:06
  SPEAKER_01: And in this paper, there was like not a normal association.
• 19:11
  SPEAKER_01: You would expect a strong association between those things, and actually between body mass index too.
• 19:16
  SPEAKER_01: And what they found was not a strong association in the data set, which doesn't make sense.
• 19:22
  SPEAKER_01: You would expect that, like, if one measure of glucose is high, the other measure of glucose would be high too.
• 19:28
  SPEAKER_01: If one is low, the other would be low too.
• 19:30
  SPEAKER_01: So it's like makes you scratch your head, like, ugh, like that that feels weird and it makes me wonder if there's a problem with the data.
• 19:39
  SPEAKER_01: Right.
• 19:39
  SPEAKER_01: And then the other thing that they called out was um that for hemoglobin A1C and for blood glucose, there's like a there's a wide variety of values that could be.
• 19:55
  SPEAKER_01: Like you could have for hemoglobin A1C, you could have a value of 5.1, 5.4, 5.8, 6.7, 7.3, 8.9, 9.2.
• 20:06
  SPEAKER_01: I could keep going, but like uh what I'm trying to show is that you could have a ton of different values for it.
• 20:13
  SPEAKER_00: Yeah.
• 20:14
  SPEAKER_01: And in these data sets, for both A1C and blood glucose, there were just 18 discrete results.
• 20:20
  SPEAKER_01: When you're talking about 100,000 people, like that doesn't even make sense.
• 20:24
  SPEAKER_00: Right.
• 20:24
  SPEAKER_00: Why why was the how are they clustered into those eight discrete groups?
• 20:27
  SPEAKER_01: 18, 18.
• 20:28
  SPEAKER_01: It's about 18, it's still 18, but like it doesn't, it doesn't actually make sense.
• 20:32
  SPEAKER_01: Like if your average blood glucose is 118 and mine is 121, and somebody else's, like it could there could be a huge range there.
• 20:42
  SPEAKER_00: For 100,000 people, you could have you might have some overlaps, but for sure you could have overlaps.
• 20:46
  SPEAKER_00: You're not going to just have 18 groups.
• 20:48
  SPEAKER_00: That doesn't make any sense, right?
• 20:49
  SPEAKER_01: Yeah.
• 20:50
  SPEAKER_01: And so it's like you start to like look under the hood of this and you're like, well, I I I don't know where the data came from.
• 20:56
  SPEAKER_01: Yeah.
• 20:57
  SPEAKER_01: And now that I'm looking at the data, it's weird.
• 21:00
  SPEAKER_01: Like, there's no missing data, there's not associations where I expect them to be.
• 21:06
  SPEAKER_01: And there's like this weird finding of like only 18 discrete results for blood glucose and A1C.
• 21:13
  SPEAKER_01: Doesn't, it doesn't make sense.
• 21:15
  SPEAKER_01: And so then it's like, well, if that's what's being used to create models that then do clinical prediction and get used in clinical practice, like that's that's maybe not good.
• 21:27
  SPEAKER_00: No, and it was to be honest, honest, it was very it was shocking to me to to read that that data set was used extensively in so many research papers.
• 21:36
  SPEAKER_00: Research papers that well, I think what was most amusing, not amusing necessarily, but scary was some of those research papers did say, make claims about the origin of that data.
• 21:49
  SPEAKER_00: And they were all different.
• 21:50
  SPEAKER_00: And you know, one said it came from a prestigious um institute, another said it came from something else, and one of them said it came from Bangladesh.
• 21:58
  SPEAKER_00: And so there was how can they all All be there was like four or five different places that that same data set was attributed as being the source.
• 22:06
  SPEAKER_00: Like first of all, how uh it's not possible to have four different ones.
• 22:10
  SPEAKER_00: This is one source.
• 22:11
  SPEAKER_00: But but also uh you know, those research papers, there was some degree of um sort of exaggeration or something that uh had the authors make up the source of the data, right?
• 22:26
  SPEAKER_00: Which is interesting because there is uh, you know, I understand to some degree in an academic setting you have research pressures to get papers out.
• 22:33
  SPEAKER_01: And uh this concept of like fast churn, right?
• 22:37
  SPEAKER_00: Yes, yes, exactly.
• 22:38
  SPEAKER_00: And so the academics and their students are trying to get papers written, they go to the nearest resource, they go on Kaggle, they look for diabetes data set, they find one that has 290,000 downloads and um hasn't been retracted.
• 22:51
  SPEAKER_00: And so they think, okay, maybe this thing is meaningful.
• 22:54
  SPEAKER_00: Let's use it and let's write a paper.
• 22:56
  SPEAKER_00: Oh, we came up with a new way to do better on this.
• 22:58
  SPEAKER_00: The current best model on this has 85% accuracy.
• 23:02
  SPEAKER_00: We're gonna do some, you know, play around with our model and look, we're getting 95% accuracy.
• 23:07
  SPEAKER_00: Oh my god, this is great.
• 23:09
  SPEAKER_00: It's doing so well on this diabetes data set, it has all these other implications.
• 23:13
  SPEAKER_00: We can now use it in this setting, in that setting, and so on and so forth.
• 23:16
  SPEAKER_00: That's not that uncommon for people as a process to follow to write a paper on uh a new model for this.
• 23:26
  SPEAKER_01: And the idea, your idea here that I'm I'm hearing, anyways, is that people want to publish quickly.
• 23:32
  SPEAKER_01: Yes, and uh and easily.
• 23:35
  SPEAKER_01: Uh, and it's not really about necessarily like building our understanding of health or predicting or improving health outcomes.
• 23:45
  SPEAKER_01: Like the goal is to get something published.
• 23:48
  SPEAKER_01: And so if there's a data set available, let's just publish it.
• 23:51
  SPEAKER_01: But then on my side, I'm like, oh gosh, like now they're using this to like predict stroke, say that we can diagnose stroke, say that we can reduce the incidence of diabetes.
• 24:02
  SPEAKER_01: But I'm not sure that any of that can actually be true if the data behind it isn't valid.
• 24:08
  SPEAKER_00: That's right.
• 24:09
  SPEAKER_00: That's right.
• 24:10
  SPEAKER_00: And and they found that, you know, not only the not only were there a lot of papers that were written that used that data set, but then those papers also claimed real-world utility of their models.
• 24:23
  SPEAKER_00: So they took it one step further as well.
• 24:25
  SPEAKER_00: And and that step means that somebody reading the paper who was not aware of this is going to think, oh, wait a minute, we have this model here.
• 24:32
  SPEAKER_00: We can use it in our setting, in our hospital setting, in our clinical setting.
• 24:36
  SPEAKER_00: And that that's that's a that's directly being deployed.
• 24:39
  SPEAKER_00: So you have a really bad model that's being directly deployed.
• 24:43
  SPEAKER_01: And so now I'm like, whoa, like let's take a step back because uh, you know, before we had the ability to do this with LLMs and machine learning and these huge data sets, this wasn't something you maybe had to think about as much.
• 25:00
  SPEAKER_01: I'm not saying, I mean, data providence has always been an issue, but here it's like everyone can just grab these data sets from anywhere and just do whatever research they want on them, right?
• 25:09
  SPEAKER_01: And it's like, well, I think we need to take a step back and have everyone think about the data provenance problem before they use these giant data sets.
• 25:17
  SPEAKER_01: And that includes like journals and publishers, it includes the data repositories like Kaggle, it includes the the researchers and the clinicians, everyone kind of coming together and saying, Hey, before I use this data, like I need to know why was it collected?
• 25:34
  SPEAKER_01: Who collected it?
• 25:35
  SPEAKER_01: Where was that done?
• 25:36
  SPEAKER_01: Um, when was that done?
• 25:38
  SPEAKER_01: And who funded the data collection?
• 25:41
  SPEAKER_01: And um all of that should be like part of the data set.
• 25:46
  SPEAKER_00: Yeah.
• 25:47
  SPEAKER_01: Right.
• 25:47
  SPEAKER_01: And if it's not, then maybe like a journal would have to go back and say, hey, I need I need this information.
• 25:54
  SPEAKER_01: You need to get give me this information before I can publish a study on this.
• 26:00
  SPEAKER_00: Right, right.
• 26:00
  SPEAKER_00: And the challenge, of course, is you have to make sure that the um that that that information itself is accurate, right?
• 26:06
  SPEAKER_00: And so I think that encouraging the data set makers to themselves, you know, write up what they've done.
• 26:13
  SPEAKER_00: Now there's a that there is a uh an issue, which is some of the data might be confidential, in which case, what do you do then?
• 26:19
  SPEAKER_01: Yeah, you how do you de-identify this huge amount of data?
• 26:22
  SPEAKER_01: But that's something that has been done before.
• 26:25
  SPEAKER_01: It's just like another step.
• 26:26
  SPEAKER_00: Yes.
• 26:27
  SPEAKER_01: Right.
• 26:27
  SPEAKER_01: And you were talking about how when people are trying to learn, they might download a data set from Kaggle and it doesn't really matter, like if it's poor provenance, because that's not the goal.
• 26:38
  SPEAKER_01: But if the goal is to use it for research and for clinical decision making, at that point, I think you have to mandate some sort of provenance reporting and make sure it is de-identified in some way.
• 26:52
  SPEAKER_01: Um, and and that piece is really important for something that you're going to actually make decisions off of.
• 26:59
  SPEAKER_00: Yes, agreed.
• 27:00
  SPEAKER_00: And I know I went back on Kaggle to look at this particular data set, one of these data sets, and somebody posted a comment there saying, citing the paper that we're just talking about, saying this just the data set should be not used at all, which is good, which is but you know, there's sort of a full circle.
• 27:15
  SPEAKER_00: People have looked at this paper, identified the issue.
• 27:17
  SPEAKER_00: But uh even after this paper was written, there was, you know, the uh people are still downloading the data set.
• 27:22
  SPEAKER_00: So, you know, I think that you know, this the solution is multifaceted, which is you need the researchers to be um to be explaining the provenance of their data.
• 27:32
  SPEAKER_00: Um, you need the journal uh reviewers to look for that kind of information.
• 27:37
  SPEAKER_00: You need um people who deploy these systems to understand where the models and the data came from for each system so that they can be sure that things that were not meant for, uh that were only meant for educational purposes or things that were only meant for whatever were not just like immediately deployed.
• 27:54
  SPEAKER_00: Now, this is easier said than done because everyone has the research churn pressures, incentives to publish, but there's also incentives to deploy AI and all these new machine learning, fancy machine learning tools quickly so that you can show that your um organization has is up is up to date and and and is and is you know keeping up with the trends.
• 28:12
  SPEAKER_00: So there's all of these other business and practical pressures that come into play.
• 28:17
  SPEAKER_00: And you know, I think it's I'm glad we're talking about this because it is an issue and it's not a fully resolved issue.
• 28:23
  SPEAKER_00: And I think that you know it it comes down to the individual researcher who's trying to solve a particular domain problem, understands their world, understands that domain really well.
• 28:32
  SPEAKER_01: Yeah, absolutely.
• 28:33
  SPEAKER_01: And I think um we can't assume that everything that is published at this point doesn't have any errors.
• 28:43
  SPEAKER_01: Like I think part of this peer, like we're seeing all these papers pass through a peer review process, but I think there's this, okay, we need to take a step back and think about what was the data it was trained on, where is it coming from, and making sure that that is high quality at all these different levels, right?
• 29:01
  SPEAKER_01: And making sure that there are good data sets available to our researchers and clinicians.
• 29:06
  SPEAKER_01: And you can't just say, oh, well, this was published in 80 other papers, I'm gonna use it in mine.
• 29:10
  SPEAKER_01: You actually have to go back and look at it.
• 29:12
  SPEAKER_01: And if that data isn't there, isn't available, or doesn't meet these findings, like, oh, we don't know if it's synthetic or simulated, we don't know if any data was removed, we we don't know who collected it, then maybe it's time to step back and say, okay, we need to find a different data set.
• 29:26
  SPEAKER_00: Yeah.
• 29:28
  SPEAKER_01: All right.
• 29:28
  SPEAKER_01: Well, I think that we can wrap up here.
• 29:31
  SPEAKER_01: Um, thanks for chatting with me about data providence today.
• 29:35
  SPEAKER_00: Thank you for joining us.