Latest / This Is Why with Dr. Busti / Internal and External Validity in Research: Are the Results Real?

Transcript

Internal and External Validity in Research: Are the Results Real? · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Hey, this is Dr.
• 0:01
  SPEAKER_00: Bustaye, and we're in the Study Design Development Interpretation Module Playlist, and this lecture is going to focus on the internal and external validity of a study.
• 0:17
  SPEAKER_00: Why?
• 0:18
  SPEAKER_00: Well, because junk in is basically junk out, isn't it?
• 0:21
  SPEAKER_00: And so for those of those of you doing research, and we as we move through those steps and understand the processes of clinical research, we need to set ourselves up to be more efficient but also positioned to improve the quality of the research, which naturally leads to a greater acceptance for publication and by clinicians to use the information, right?
• 0:45
  SPEAKER_00: Because when we have quality information, right, that improves the overall internal validity.
• 0:53
  SPEAKER_00: And that matters so that we can extrapolate it.
• 0:56
  SPEAKER_00: Hey guys, I'm Dr.
• 0:57
  SPEAKER_00: Bustaye, and welcome to This Is Why, a show that helps healthcare professionals transform their knowledge from what to why, because when you understand why you do what you do, even in the context of the best available evidence, sometimes it's not great evidence, but it's the best available evidence.
• 1:15
  SPEAKER_00: It changes how, and that changes everything about the way we make decisions every day in the patients that we serve.
• 1:23
  SPEAKER_00: Now, these lectures that we put together for you in a module are as part of a series on clinical research.
• 1:29
  SPEAKER_00: Okay, they are in a sequence if that's helpful for you.
• 1:33
  SPEAKER_00: You can find those by exploring the full collection or playlist at YouTube, our podcast channels, and also at thisiswhy.
• 1:47
  SPEAKER_00: So we are here at this spot in the module series where we've already covered in topics, and if you want to go back to look at those, those are available at all those channels.
• 1:57
  SPEAKER_00: But hypothesis testing, the study design, which is obviously going to impact a lot of how we get to the truth, which leads to this conversation here, right?
• 2:09
  SPEAKER_00: We also then cover topics in bias, the types of bias, the impact that bias has, and ways that we can control it.
• 2:17
  SPEAKER_00: And all of that leads down to this subject here, okay?
• 2:23
  SPEAKER_00: What is the information that was produced by the study?
• 2:27
  SPEAKER_00: Is it internally valid?
• 2:29
  SPEAKER_00: And if it's if it is, can we then externally apply it?
• 2:34
  SPEAKER_00: Okay, so that's internal and external validity.
• 2:37
  SPEAKER_00: And again, thinking about our steps in research, we're going to be focusing our attention at the development of the study concept as a researcher and the protocol.
• 2:49
  SPEAKER_00: So we're controlling this and considering the internal validity and ways to strengthen it.
• 2:57
  SPEAKER_00: And so that again, for the researcher that's before the study, for that clinician, we want to know what the researcher, investigator who published this information or is presenting it should have done.
• 3:10
  SPEAKER_00: And maybe they did, maybe they didn't.
• 3:13
  SPEAKER_00: Right?
• 3:13
  SPEAKER_00: But those are the that's that transparency and that accountability that when we do it correctly, and the more so that we do it up front, the more likely we are going to produce quality research.
• 3:25
  SPEAKER_00: Now, let's get on the same page of terminology.
• 3:30
  SPEAKER_00: What is internal validity and what is external validity?
• 3:33
  SPEAKER_00: I mean, the names at some level help us to get oriented around that, but um we need to understand the dependent and independent variables, okay, because that's and that's terminology that's used in the EBM literature, biostatistics, epidemiology literature.
• 3:49
  SPEAKER_00: But the independent variable is typically the intervention, the thing that we did, the thing that was manipulated, okay, um, in the research.
• 4:00
  SPEAKER_00: So it's usually that intervention itself.
• 4:03
  SPEAKER_00: Many, many times it's a medication, but it can obviously be a procedure, medical device, implementation, any of those things, right?
• 4:10
  SPEAKER_00: Something that we as the researchers have done.
• 4:14
  SPEAKER_00: The dependent variable is that variable that's the response to the interventions, the outcome.
• 4:21
  SPEAKER_00: Okay?
• 4:22
  SPEAKER_00: Um, and these are typically the endpoints of the study, all right?
• 4:27
  SPEAKER_00: And so that's why having your terminology is correct because that leads us to that internal external validity, where the internal validity is did that independent variable, the intervention, the thing that we did, the thing that we applied to the subjects and the patients, what impact and how much of it is responsible for the change?
• 4:50
  SPEAKER_00: We want it to be pure, right?
• 4:53
  SPEAKER_00: And if you go back and remember in our study design topic, we talked about those explanatory clinical trials versus pragmatic clinical trials, and we think about concepts that you see in the in the reports of a per protocol result or analysis versus intention to treat, right?
• 5:14
  SPEAKER_00: The per protocol is going to be tighter, more relevant to the effect efficacy of that intervention.
• 5:21
  SPEAKER_00: We want to know what happens when we apply the thing, right?
• 5:26
  SPEAKER_00: So when we produce these results from a study, the results are there, they are what they are, but they're not the true population results.
• 5:34
  SPEAKER_00: We want to get as close to that as we can.
• 5:37
  SPEAKER_00: So, how internally valid is that information?
• 5:41
  SPEAKER_00: There's a lot of variables that go into determining that internal validity.
• 5:46
  SPEAKER_00: And it's not always going to be agreed upon from one clinician to another or one researcher to another.
• 5:52
  SPEAKER_00: And that is why a lot of times evidence-based medicine is not easy.
• 5:57
  SPEAKER_00: Sometimes it's also not cookie-cutter.
• 6:00
  SPEAKER_00: We don't always agree.
• 6:02
  SPEAKER_00: You'll see that in evidence-based guidelines all the time.
• 6:05
  SPEAKER_00: Depending on who's on the committee, and the changing out of the committee influences the recommendations that are made.
• 6:12
  SPEAKER_00: And we've seen that happen a lot, especially in like the dyslipidemia literature.
• 6:17
  SPEAKER_00: Oh my goodness.
• 6:18
  SPEAKER_00: Like lots of debates and arguments over the bias that someone brings into and the way that they want that data to they only wanted certain types of internally valid information.
• 6:28
  SPEAKER_00: So they had certain structure.
• 6:29
  SPEAKER_00: Nothing wrong with that as long as you define it up front.
• 6:32
  SPEAKER_00: But you many times can lose access to other studies, and that's what happened with the hyperlipidemia or dyslipidemia literature over the time and the guidelines as you follow them throughout the decades.
• 6:45
  SPEAKER_00: There was a lot of pivots around that.
• 6:48
  SPEAKER_00: Okay, external validity.
• 6:50
  SPEAKER_00: All right, this is taking the about the generalizability or the applicability of the result to the population that you are treating.
• 6:59
  SPEAKER_00: So if something is internally invalid, you should probably not be trying to extrapolate it externally.
• 7:09
  SPEAKER_00: Junk in, chunk out, right?
• 7:10
  SPEAKER_00: I mean, it's not going to change magically in your external uh application of it if it's internally invalid.
• 7:18
  SPEAKER_00: So that's why we want to really get down to that because then think about what are the threats to internal validity.
• 7:26
  SPEAKER_00: Okay?
• 7:27
  SPEAKER_00: So there's a number of them.
• 7:29
  SPEAKER_00: History, okay?
• 7:30
  SPEAKER_00: So an event that occurs outside of the study, you have no control over that, okay, that may have had an effect on observed outcome.
• 7:39
  SPEAKER_00: I mean, things like pandemics, how about the evolution of social media or AI, depending again, it depends on what you're measuring and you're studying.
• 7:48
  SPEAKER_00: But there are things that have happened in time points in history that have some impact on people.
• 7:54
  SPEAKER_00: So if you're doing in the psychiatry world or psychology analysis, maybe screen times and social media and all that stuff that's gotten lots of attention over the years and the impact that it has on mental health.
• 8:10
  SPEAKER_00: Okay, well, that that is a historical shift.
• 8:14
  SPEAKER_00: AI was a historical shift.
• 8:16
  SPEAKER_00: Pandemics are a historical shift that are outside that could influence the behaviors of people, but also what happened inside the study, especially if people were lost or influenced by something else.
• 8:29
  SPEAKER_00: So you have to put it in the context of the study.
• 8:31
  SPEAKER_00: How about maturation?
• 8:33
  SPEAKER_00: The effect of the dependent variable, again, remember the outcome, the endpoint that we're measuring, it was due to an event or process that occurred as an individual as a function of their time.
• 8:47
  SPEAKER_00: So if you're studying people for years, okay, maybe they arrived at a certain endpoint because of their age and maturation alone.
• 8:56
  SPEAKER_00: It had nothing to do with the intervention that was done.
• 8:59
  SPEAKER_00: Okay.
• 9:01
  SPEAKER_00: Testing.
• 9:02
  SPEAKER_00: A person can learn something from doing something the first time, and then if they're allowed to repeat a study or a test, or they get additional training, then your next result may be modified or influenced by that.
• 9:18
  SPEAKER_00: Okay?
• 9:19
  SPEAKER_00: What if the instrument has gone through a calibration or a change in the way you measure or use the tool that was done within one particular study?
• 9:30
  SPEAKER_00: And so that can change the observer's scores, okay, or the participant scores.
• 9:36
  SPEAKER_00: There's also statistical regression, okay, groups selected on the basis of extreme scores that tend to move toward the actual population mean.
• 9:48
  SPEAKER_00: How about selection?
• 9:50
  SPEAKER_00: These characteristics are equivalency that may account for different observed biases, and that's why they produce baseline characteristics and try to prove to you that randomization worked and what it looks like.
• 10:03
  SPEAKER_00: And then mortality.
• 10:05
  SPEAKER_00: What happens when patients are no longer in a study?
• 10:08
  SPEAKER_00: What happens to the overall effect of the results?
• 10:11
  SPEAKER_00: Okay, so they drop their falling out of the study.
• 10:14
  SPEAKER_00: So what is the impact on the magnitude of effect if you create some selection bias if they're falling out because of mortality?
• 10:23
  SPEAKER_00: Threats to external validity, okay, again, that interaction of testing and the treatment.
• 10:28
  SPEAKER_00: So results from studies with repeated testing may not be applicable to studies where there's no retesting.
• 10:36
  SPEAKER_00: Again, that maturation and growth and development from the experience itself.
• 10:41
  SPEAKER_00: So that's where that interaction with a selection of the treatment that was used that may affect the patients and their what their how they respond to the next phase.
• 10:52
  SPEAKER_00: Okay, so there's an interaction between that.
• 10:55
  SPEAKER_00: The interaction of the setting and the treatment.
• 10:58
  SPEAKER_00: Okay, some unique characteristics in the environment.
• 11:02
  SPEAKER_00: Uh, where the study is taking place that may gain, may gain or reduce the generalizability of the results to the general population.
• 11:10
  SPEAKER_00: Okay.
• 11:11
  SPEAKER_00: I mean, most of this stuff seems straightforward, but it's something you've got to think about.
• 11:14
  SPEAKER_00: About reactive arrangements.
• 11:16
  SPEAKER_00: The reaction of a patient may change if they know they're being observed.
• 11:20
  SPEAKER_00: That's kind of the Hawthorne effect, right?
• 11:22
  SPEAKER_00: That was a type of bias.
• 11:24
  SPEAKER_00: Multiple treatment intervention interference.
• 11:27
  SPEAKER_00: So they're getting one test and another test, and maybe there's an intersection between those two interventions, and there's some additive or synergistic effect on the subject that may not be the same as if that one intervention was not present.
• 11:42
  SPEAKER_00: So again, that's why it's so important that when you think about a study intervention or independent variable that was used in a clinical trial, even the way it was dosed, let's say, for example, you know, steroids and meningitis, the dosing of dexamethasone in patients where you are concerned about bacterial meningitis and maybe have low immunization coverage and those things, that is that is influenced by the study.
• 12:07
  SPEAKER_00: How frequently it is administered, the root of administration, all of that points to the way it was valid.
• 12:15
  SPEAKER_00: So you have to apply that same method of application intervention of the independent variable to achieve those desired results if you're looking to achieve them.
• 12:25
  SPEAKER_00: Okay?
• 12:26
  SPEAKER_00: So core concepts is that internal validity has to do with the degree of confidence.
• 12:33
  SPEAKER_00: How much do you believe that the outcome of this study was actually due to the intervention that was applied?
• 12:41
  SPEAKER_00: So that the intervention resulted in the outcome.
• 12:44
  SPEAKER_00: You have that per-protocol analysis and intention to treat analysis that many times we we try to think about.
• 12:50
  SPEAKER_00: And then there's the external validity.
• 12:53
  SPEAKER_00: Once you've determined something is internally valid, that has to do now with your ability to apply it and generalize it to the patients that you're serving in your population.
• 13:03
  SPEAKER_00: Does it fit?
• 13:05
  SPEAKER_00: Does it apply?
• 13:07
  SPEAKER_00: You know, the um the big New England, you know, I talked about meningitis, the big New England study and with dexamethasone was adults.
• 13:17
  SPEAKER_00: So does that apply to pediatrics?
• 13:19
  SPEAKER_00: Okay.
• 13:20
  SPEAKER_00: That stuff is important, right?
• 13:22
  SPEAKER_00: So it's the extrapolation, it's the generalizability of that, the external validity of the use of that information in certain populations.
• 13:30
  SPEAKER_00: So I hope that you found this to be helpful in this particular series on This Is Why.
• 13:35
  SPEAKER_00: I appreciate you watching.
• 13:37
  SPEAKER_00: If you want access to some of those other topics that I mentioned, again go back to the playlist, the collection.
• 13:42
  SPEAKER_00: Those are available at YouTube, Podcasts, and at thisiswhy.
• 13:56
  SPEAKER_00: Okay, which I help to start and develop.
• 13:58
  SPEAKER_00: But that's where if you practice applying the content for real questions, it's one thing to interact and hear something and go, I understand that, I got that, no big deal.
• 14:07
  SPEAKER_00: But it's a difference in applying it to a multiple choice question, isn't it?
• 14:11
  SPEAKER_00: And many times our boards, whether you're nursing, pharmacy, medicine, it doesn't matter.
• 14:15
  SPEAKER_00: We've all I've taken all those exams, uh, being a nurse and a pharmacist and a physician.
• 14:20
  SPEAKER_00: I've I've had those questions show up.
• 14:22
  SPEAKER_00: And so you need to be able to practice applying it.
• 14:25
  SPEAKER_00: If you're in a residency program or training program, and your program wants tools like this or a program that's structured, uh, we can put that together for you.
• 14:34
  SPEAKER_00: We do that over that high yield meta review.
• 14:35
  SPEAKER_00: So thanks so much for joining.
• 14:37
  SPEAKER_00: Give us a like if you we would really appreciate that.
• 14:40
  SPEAKER_00: Um at YouTube in particular, helps us to keep supporting what we're doing.
• 14:44
  SPEAKER_00: Thanks so much, and we'll see you next time.