Latest / This Is Why with Dr. Busti / Hypothesis Testing Explained for Clinical Research

Transcript

Hypothesis Testing Explained for Clinical Research · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Hey, it's Dr.
• 0:01
  SPEAKER_00: Bus Eye, and we are in the Study Design, Development, and Interpretation Module Playlist Series.
• 0:08
  SPEAKER_00: This lecture is going to focus on hypothesis testing.
• 0:14
  SPEAKER_00: Now, when we think about the process of doing clinical research, at least as it's broadly described by the NIH, we move through several steps, right?
• 0:24
  SPEAKER_00: We develop the study concept, we secure the funding if needed, we prepare the study protocol, protect our participants, conduct the study, close the study, right, analyze the data, then we communicate.
• 0:37
  SPEAKER_00: But everything really starts at the study concept stage, and that's where our focus is.
• 0:43
  SPEAKER_00: And one of the most first fundamental things that we need to do is defining that hypothesis.
• 0:51
  SPEAKER_00: And the hypothesis is the core question that you're trying or the researcher is trying to answer.
• 0:58
  SPEAKER_00: And that question becomes incredibly important because it determines not only the null and alternative hypothesis that you describe in the protocol, but also the statistical analysis you will eventually use to analyze the data, or if you're reading it, that they should have done.
• 1:16
  SPEAKER_00: In other words, the question you asked at the beginning drives the entire process of how the study is designed, analyzed, and interpreted.
• 1:26
  SPEAKER_00: And with that, I'm Dr.
• 1:27
  SPEAKER_00: Bustide, your faculty and host on This Is Why.
• 1:30
  SPEAKER_00: This is a show that helps healthcare professionals transform medical knowledge from what to why, because when you know why you do what you do in the context of the best available evidence, it changes the how.
• 1:44
  SPEAKER_00: And that changes everything about the patient decisions that we make every day, guys.
• 1:48
  SPEAKER_00: Okay, now each of the lectures is part of a larger part of a sequence or collection or playlist, right?
• 1:57
  SPEAKER_00: Remember, the collection is the big umbrella playlists, individual groups of topics.
• 2:03
  SPEAKER_00: Okay, so if you're not sure or you want to understand in the context of the other topics, go check out the full playlist.
• 2:12
  SPEAKER_00: Okay, that's where we build on some of these concepts, sometimes in a step-by-step fashion, also individual topics for your use.
• 2:19
  SPEAKER_00: Within this module or playlist, we're gonna cover several core areas.
• 2:25
  SPEAKER_00: We're gonna be focusing today, though this lecture right here on hypothesis testing, but don't ignore the other components in this series of the playlist.
• 2:34
  SPEAKER_00: Study design, types of bias that can be introduced or that you should be looking for, and threats to both internal and external validity.
• 2:45
  SPEAKER_00: Now, all of these steps and pieces work together to help you critically evaluate clinical research.
• 2:53
  SPEAKER_00: Okay, if we zoom out and look at the steps in clinical research, we can see where hypothesis testing fits into the larger process.
• 3:07
  SPEAKER_00: You begin by developing the study concept, okay.
• 3:12
  SPEAKER_00: Then you move through, obviously, funding, you develop the protocol, you show them how you're going to protect the patients and the participants, you then start the study, you monitor it, you then close it, you analyze it.
• 3:27
  SPEAKER_00: Obviously, you then interpret the data, and then finally you communicate those results.
• 3:33
  SPEAKER_00: But everything begins at the study concept level.
• 3:38
  SPEAKER_00: Okay?
• 3:39
  SPEAKER_00: And within that stage, the hypothesis, one of the first things that must be clearly defined.
• 3:48
  SPEAKER_00: When researchers are developing the study concept and protocol, several things must then be also considered.
• 3:55
  SPEAKER_00: So obviously, hypothesis testing is there, that's obvious.
• 3:58
  SPEAKER_00: That's where we're at, that's where we're focusing this topic about.
• 4:02
  SPEAKER_00: But you also have to still keep in the back of your mind, right?
• 4:06
  SPEAKER_00: Study design, potential sources of bias, threats to validity, both internal and external.
• 4:14
  SPEAKER_00: What basic statistical analysis and application are you going to use?
• 4:19
  SPEAKER_00: And if the study is a diagnostic in nature, then you also need to understand the properties of the diagnostic test, right?
• 4:26
  SPEAKER_00: And throughout the entire process, the reporting standards need to always be in the back of your mind and followed, even at the protocol development level.
• 4:37
  SPEAKER_00: So there's an important take-home point here.
• 4:39
  SPEAKER_00: For the researcher, you must know what the desired endpoint is before you start the study.
• 4:44
  SPEAKER_00: As a clinician, remember, guys, you're you should understand what the how this process works so that you can properly appraise and apply the results.
• 4:55
  SPEAKER_00: So the endpoint or outcome, guys, represents the answer to the research question, and you've got to know what that is, right?
• 5:06
  SPEAKER_00: The outcome is commonly referred to the dependent variable.
• 5:12
  SPEAKER_00: Just keep that straight.
• 5:14
  SPEAKER_00: The dependent variable is typically the outcome.
• 5:18
  SPEAKER_00: The exposure refers to the factor that may influence that outcome and is often referred to as the independent variable.
• 5:29
  SPEAKER_00: Now, in randomized controlled trials or RCTs, the exposure is usually the treatment that's being tested, isn't it?
• 5:37
  SPEAKER_00: In a case-controlled study, it might be a risk factor that is being evaluated.
• 5:45
  SPEAKER_00: Okay.
• 5:46
  SPEAKER_00: The relationship between the exposure and outcome forms, guys, the basis of the hypothesis.
• 5:56
  SPEAKER_00: Okay.
• 5:57
  SPEAKER_00: So what is hypothesis testing?
• 6:02
  SPEAKER_00: Well, at its core, hypothesis testing is about comparing two competing possibilities, not to make this more complicated than it is.
• 6:12
  SPEAKER_00: One side, here we see, is the null hypothesis.
• 6:19
  SPEAKER_00: On the other side is the high alternative hypothesis, and usually designated with this H0 or H1.
• 6:27
  SPEAKER_00: The null hypothesis essentially states that there is, guys, there is no different present.
• 6:37
  SPEAKER_00: So if you conduct a study, collect the data, analyze it appropriately, the null hypothesis assumes that there is no meaningful difference between the groups being compared.
• 6:53
  SPEAKER_00: Now, when a study fails to find a difference, we say, another way of saying it, that we have failed to reject the null hypothesis.
• 7:07
  SPEAKER_00: In everyday language, many people interpret that as accepting the null hypothesis.
• 7:14
  SPEAKER_00: So failing to reject the null hypothesis is accepting it.
• 7:20
  SPEAKER_00: I know.
• 7:21
  SPEAKER_00: The terminology can sound a little confusing at first, but the concept is actually straightforward.
• 7:29
  SPEAKER_00: No difference was demonstrated.
• 7:33
  SPEAKER_00: Okay?
• 7:33
  SPEAKER_00: So these interpretations basically say the same thing, different ways.
• 7:41
  SPEAKER_00: I bring that up, but I'm pausing you for a second to stress an important point for board exams.
• 7:47
  SPEAKER_00: Board exams like to throw these kinds of phrases into answers or questions that make your head sometimes spin.
• 7:59
  SPEAKER_00: Okay?
• 8:00
  SPEAKER_00: But the failure to find a difference can be influenced by a lot of things.
• 8:05
  SPEAKER_00: And one of those is not having enough power.
• 8:12
  SPEAKER_00: We're going to come back to that.
• 8:13
  SPEAKER_00: We'll dive into that later.
• 8:15
  SPEAKER_00: But failing to find a difference, the very first thing that you want to know is was there enough power in this study?
• 8:23
  SPEAKER_00: Did they include enough patients and subjects to find a difference if it was supposed to be there?
• 8:30
  SPEAKER_00: Alright, now let's move on to the alternative hypothesis, which proposes that there is a difference.
• 8:38
  SPEAKER_00: Okay, so if the results support a difference, then listen, the null hypothesis is rejected.
• 8:47
  SPEAKER_00: Another way to think about hypothesis testing is through a decision table, right, that compares the reality of a conclusion reached in a study versus what really exists.
• 9:12
  SPEAKER_00: So over here is our decision.
• 9:15
  SPEAKER_00: This is what the study result is, said it was, but this over here is the actual reality.
• 9:25
  SPEAKER_00: What is the truth?
• 9:27
  SPEAKER_00: Okay.
• 9:29
  SPEAKER_00: So if we accept the null hypothesis, and in reality, okay, the null hypothesis is true, there is no difference, then we have made a correct decision.
• 9:48
  SPEAKER_00: But if we accept the null hypothesis, okay, when a difference actually exists, that means that the null hypothesis would be false.
• 10:01
  SPEAKER_00: So we say the study gets done, hey, there's no difference here.
• 10:06
  SPEAKER_00: Oh no, in reality there is.
• 10:09
  SPEAKER_00: Then you are prone to have made a type 2 error.
• 10:14
  SPEAKER_00: Okay?
• 10:16
  SPEAKER_00: Now this error is represented by beta.
• 10:20
  SPEAKER_00: And beta is important because it relates to statistical power, which I was just commenting on.
• 10:27
  SPEAKER_00: Now power is defined as 1 minus beta, and it represents the probability that we will correctly detect a true difference when one exists.
• 10:40
  SPEAKER_00: That's why you want to have enough patience.
• 10:42
  SPEAKER_00: And this is why it's so important that power be considered in the designing of a study protocol before you even start.
• 10:51
  SPEAKER_00: How many patients do you have to enroll to find that difference if it exists?
• 10:58
  SPEAKER_00: So if the sample size is too small, the study may fail to detect a real difference.
• 11:04
  SPEAKER_00: Now let's consider the other possibility.
• 11:06
  SPEAKER_00: What if we reject the null hypothesis and conclude that a difference exists, but the reality is, the reality is that it's true, there's no difference.
• 11:21
  SPEAKER_00: Well then we've made a type one error.
• 11:24
  SPEAKER_00: And that kind of error is represented by alpha.
• 11:27
  SPEAKER_00: So keep beta and alpha straight.
• 11:31
  SPEAKER_00: Type 1 error is alpha, type 2 is beta.
• 11:35
  SPEAKER_00: Alpha represents the probability, guys, of falsely concluding that a difference exists in your study, but in reality it doesn't.
• 11:46
  SPEAKER_00: Now alpha is typically set ahead of time, most commonly at 0.05, which means that we're willing to accept a 5% chance of making a type 1 error.
• 12:02
  SPEAKER_00: Now another important concept in hypothesis testing is the difference between a one-tell versus a two-telled analysis.
• 12:14
  SPEAKER_00: In a one-tailed or one-sided process, this specifies a direction of the effect.
• 12:25
  SPEAKER_00: In other words, right, it assumes that the intervention that we're making, the exposure, will influence the outcome in a particular direction, whether it's positive or negative.
• 12:40
  SPEAKER_00: Now some researchers would argue that that approach or mentality in a study protocol is not appropriate in many clinical studies.
• 12:51
  SPEAKER_00: Why is that?
• 12:52
  SPEAKER_00: Well, because if you already assume the direction of the outcome with your intervention, it raises the question then why are you doing the study in the first place?
• 13:04
  SPEAKER_00: Good scientific research should remain open to the possibility of the hypothesis that you have could be wrong.
• 13:17
  SPEAKER_00: And so for this reason, many researchers prefer a two-tailed or two-sided analysis or hypothesis.
• 13:28
  SPEAKER_00: And in this approach, the study remains open.
• 13:31
  SPEAKER_00: You as a researcher remain open to the possibility that the intervention that you are making and you're exposing patients to could either improve or worsen the outcomes.
• 13:46
  SPEAKER_00: So the direction of the effect is not predetermined with a two-tailed.
• 13:52
  SPEAKER_00: Instead, you let the data and the truth determine the answer.
• 13:59
  SPEAKER_00: And this approach tends to be viewed as more objective and transparent.
• 14:07
  SPEAKER_00: Okay.
• 14:09
  SPEAKER_00: So what difference does this stuff make?
• 14:13
  SPEAKER_00: Okay.
• 14:16
  SPEAKER_00: Well, alpha, most studies set the alpha at 0.05.
• 14:23
  SPEAKER_00: We've talked about that.
• 14:24
  SPEAKER_00: Repetition makes it stick.
• 14:27
  SPEAKER_00: But remember that 0.05 alpha represents a 5% chance that you will be making a type 1 error.
• 14:36
  SPEAKER_00: So in a one-tailed test, the entire alpha is placed on one side of the probability distribution.
• 14:47
  SPEAKER_00: And I'm going to show you a graph here in a minute.
• 14:50
  SPEAKER_00: But this means that the all of the statistical significance threshold lies in a single or one-sided direction.
• 15:01
  SPEAKER_00: In a two-tailed, right, that alpha of 0.05 is actually split into two.
• 15:10
  SPEAKER_00: They're shared.
• 15:11
  SPEAKER_00: So 0.025 is on one side, 0.025 is on the other side of the distribution.
• 15:18
  SPEAKER_00: And doing that makes it harder on the researcher to reach statistical significance because the threshold is now divided across two directions.
• 15:34
  SPEAKER_00: Doing that reduces the likelihood of falsely declaring a difference when one truly exists.
• 15:41
  SPEAKER_00: And I just want to show you what that looks like on the distribution curve, right?
• 15:46
  SPEAKER_00: So a one-tailed, this is a one-tailed, where all of the alpha is distributed to one direction.
• 15:55
  SPEAKER_00: So you're assuming the outcome is going in that direction, again, positive or negative.
• 16:03
  SPEAKER_00: But this is problematic because, again, you already have a pre-perceived idea of where it's going.
• 16:14
  SPEAKER_00: Whereas that two-tailed, you take that 0.05 and now you cut it in half, and you see that the space that you have to find statistical significance when it really exists is now even harder on you.
• 16:38
  SPEAKER_00: So when is a one-tail test becomes problematic is when we are using the data after the data has already been collected.
• 16:48
  SPEAKER_00: This is why the protocol, setting it up up front, going through the IRB, having somebody else understand it, and what you're doing is protecting the integrity.
• 17:02
  SPEAKER_00: But when you do what is called a post hoc analysis, this is often referred to as data dredging.
• 17:11
  SPEAKER_00: When people have already done a study, they're now manipulating their environment.
• 17:16
  SPEAKER_00: This is using what are they doing?
• 17:20
  SPEAKER_00: They're trying to find statistical difference somewhere in their thing so that they can have something to publish.
• 17:27
  SPEAKER_00: And that's why establishing the hypothesis and the statistical approach before the study, okay, in what we call a priori approach or mentality, is so important.
• 17:41
  SPEAKER_00: Now, let's kind of bring this concept to light with a real example.
• 17:46
  SPEAKER_00: And the Hope trial is a great example of this.
• 17:50
  SPEAKER_00: This was a landmark clinical trial published in the New England Journal of Medicine back in the year 2000.
• 17:56
  SPEAKER_00: You might say, well, that's a long time ago, but it's actually still very relevant and has still one of the foundational landmark studies for why we do what we do in patients with higher risk of heart failure.
• 18:10
  SPEAKER_00: So the study evaluated whether ACE inhibitor, ramapril, could reduce cardiovascular outcomes in high-risk patients.
• 18:21
  SPEAKER_00: Okay?
• 18:22
  SPEAKER_00: And we see that because it says that.
• 18:25
  SPEAKER_00: Where are we?
• 18:27
  SPEAKER_00: Well, if you come down here, you see that we're right before the methods.
• 18:31
  SPEAKER_00: This is usually at the very end of the intro, where they actually give you what is the purpose or the primary outcome, many times the hypothesis, of the study.
• 18:47
  SPEAKER_00: You need to remember, you need to know what is the uh the exposure, ramapriel, and what is the outcome.
• 18:55
  SPEAKER_00: Well, they're looking at a composite of death from cardiovascular causes, myocardial infarction, stroke, right?
• 19:03
  SPEAKER_00: That's their primary.
• 19:06
  SPEAKER_00: Okay.
• 19:07
  SPEAKER_00: If you dive a little bit deeper, you can see that initially they didn't give a direction.
• 19:13
  SPEAKER_00: They weren't saying, well, we want to know if ramapril is worse.
• 19:17
  SPEAKER_00: We want to see that it's better.
• 19:19
  SPEAKER_00: They didn't say that.
• 19:20
  SPEAKER_00: So when you dive deeper into the methodology section, which we'll get into in a minute, a little bit more detail, you see that they actually use the words two-sided alpha.
• 19:32
  SPEAKER_00: That's good.
• 19:34
  SPEAKER_00: That's as a reader, clinician, going, wow, they made it harder on themselves at some level to find a difference, and they were open to the fact that it could be no different, right?
• 19:49
  SPEAKER_00: The null is correct.
• 19:50
  SPEAKER_00: It could be that the rampill is worse or that it's better.
• 19:55
  SPEAKER_00: I love that in a researcher.
• 19:57
  SPEAKER_00: And again, studies in New England, higher quality many times, right?
• 20:02
  SPEAKER_00: Well done.
• 20:03
  SPEAKER_00: That's a good example of a two-sided alpha being applied that made it harder for them to look for that difference.
• 20:12
  SPEAKER_00: So their null hypothesis was the risk, okay, applying it to this topic, the risk of death from cardiovascular cause MI stroke is not significantly affected by the use of ramipril in a high-risk population.
• 20:27
  SPEAKER_00: Again, they're defining the context.
• 20:30
  SPEAKER_00: That's important for you.
• 20:33
  SPEAKER_00: Okay.
• 20:34
  SPEAKER_00: The alternative hypothesis is there is a difference, but the effect is not being made here by them in the intro or their methods.
• 20:46
  SPEAKER_00: So they were open to that fact.
• 20:49
  SPEAKER_00: And that is how when you write a protocol, they want to see this written out, the IRB.
• 20:57
  SPEAKER_00: Good researchers should be able to articulate that.
• 21:01
  SPEAKER_00: Should be able to write that out very clearly.
• 21:04
  SPEAKER_00: And it should be published in the paper very clearly so that I, or you, as the clinician reading the paper, should not be confused.
• 21:12
  SPEAKER_00: Because this member drives the statistical analysis.
• 21:17
  SPEAKER_00: So as we look in here, did they stick with it?
• 21:20
  SPEAKER_00: Did they actually do what they said they were going to do?
• 21:24
  SPEAKER_00: Well, interestingly, when you look at it, you can see that this is a I zoomed in or I added the statistical analysis here, because remember that drives statistical analysis.
• 21:34
  SPEAKER_00: Well, remember they did a power analysis and they did it, they made their power 90%.
• 21:40
  SPEAKER_00: They made it even more, made sure they could find the difference if it was really there.
• 21:46
  SPEAKER_00: And so they said they needed 9,000 patients would be required to have a study power of 90%.
• 21:53
  SPEAKER_00: So remember, if they didn't achieve adequate power, they may be making a type 2 error if they found no difference.
• 22:00
  SPEAKER_00: But did they find a difference?
• 22:01
  SPEAKER_00: Well, let's first look at the number of patients.
• 22:04
  SPEAKER_00: Well, when you add those two up, that's more than 9,000.
• 22:10
  SPEAKER_00: Okay?
• 22:11
  SPEAKER_00: That's good.
• 22:12
  SPEAKER_00: So they enrolled more than 9,000 patients.
• 22:15
  SPEAKER_00: And then you come down here and you see their p-value, which is at statistical significance or not, you see that it's less than 0.001.
• 22:26
  SPEAKER_00: So they found statistical significance, but also clinical relevance.
• 22:31
  SPEAKER_00: We know that this p-value is accurate because the 95 the relative risk and the 95% confidence rule does not pass through or include 1.0, which would be relevant in a relative risk.
• 22:45
  SPEAKER_00: So that is statistically significant.
• 22:47
  SPEAKER_00: It is also clinically relevant, and they made it harder on themselves by making the p value or the alpha value have to be divided out.
• 22:59
  SPEAKER_00: So did these individuals, this study, did they make a type 2 error?
• 23:04
  SPEAKER_00: Well, no, because they found a difference.
• 23:15
  SPEAKER_00: Well, that's not the case.
• 23:16
  SPEAKER_00: Remember, they achieved power.
• 23:19
  SPEAKER_00: Did they make so then we have to go, did they make a type 1 error?
• 23:23
  SPEAKER_00: They say there's a difference, but in reality there's not.
• 23:26
  SPEAKER_00: Well, remember, they achieved power.
• 23:28
  SPEAKER_00: They gave themselves a two two-sided alpha, and their p-value was way below that.
• 23:34
  SPEAKER_00: They clearly.
• 23:36
  SPEAKER_00: Yeah, I mean, is it possible?
• 23:38
  SPEAKER_00: Sure.
• 23:39
  SPEAKER_00: Right?
• 23:39
  SPEAKER_00: That p-value of 0.001, right?
• 23:43
  SPEAKER_00: That's their chance that they have random error.
• 23:47
  SPEAKER_00: So you you take the p-value, you move the decimal place over two places.
• 23:52
  SPEAKER_00: So a p-value of 0.001 means that there's a 0.1% chance that their findings are due to random error.
• 24:02
  SPEAKER_00: So to me, those are real.
• 24:06
  SPEAKER_00: Those are real.
• 24:07
  SPEAKER_00: And other clinicians agree, guideline committees agree, it changed clinical practice.
• 24:14
  SPEAKER_00: Right?
• 24:15
  SPEAKER_00: So, in summary, right, wrapping this whole topic up, first, hypotheses should be determined a priori before the study begins.
• 24:26
  SPEAKER_00: Okay.
• 24:26
  SPEAKER_00: Second, a two-telled hypothesis approach is generally preferred because it remains open to either direction or effect.
• 24:36
  SPEAKER_00: Type 1 errors occur when a study says that there is a difference, but in reality there is not.
• 24:44
  SPEAKER_00: Remember, that's also known as alpha.
• 24:46
  SPEAKER_00: Type 2 error, when the study says, hey, there's no difference, but in reality there is beta, right?
• 24:54
  SPEAKER_00: Usually from insufficient power.
• 24:58
  SPEAKER_00: A few fast facts.
• 25:00
  SPEAKER_00: Remember, alpha is that risk for making a type 1 error most of the time.
• 25:05
  SPEAKER_00: It's set at 0.05 or 5% chance of making a type 1 error.
• 25:12
  SPEAKER_00: Beta is at risk of making type 2.
• 25:15
  SPEAKER_00: Remember, a beta is usually set at about 0.2.
• 25:18
  SPEAKER_00: We'll dive into that later in the biostatistics section.
• 25:21
  SPEAKER_00: That means that there's 80% chance of not making a type 2 error.
• 25:28
  SPEAKER_00: That's why we put a power to be at that level.
• 25:32
  SPEAKER_00: So these concepts of alpha, beta, power, they're all they're all actual foundational to understanding clinical research and design and your ability to interpret this information.
• 25:42
  SPEAKER_00: I thank you guys so much for watching.
• 25:44
  SPEAKER_00: I hope that you found this to be very helpful.
• 25:47
  SPEAKER_00: Please jump over to thisiswhy.health or the playlist on YouTube, okay, or in the in the uh channels that we have.
• 25:55
  SPEAKER_00: Continue exploring aspects of the playlist.
• 25:57
  SPEAKER_00: Try to walk through the process of evidence-based medicine.
• 26:00
  SPEAKER_00: We do offer it in a step-by-step fashion that we feel like best aligns with kind of building, but you can use them individually.
• 26:07
  SPEAKER_00: And for those of you that are preparing for clinical practice or board style learning, high yield med reviews has cubanks, practice questions on this subject, can help you prepare for those sections on your exams.
• 26:23
  SPEAKER_00: Okay.
• 26:23
  SPEAKER_00: This is why.health members, there's a coupon there for you at a discount, right, for joining.
• 26:31
  SPEAKER_00: So thank you so much for being here.
• 26:34
  SPEAKER_00: Thank you for investing in your knowledge and understanding the evidence and applying it appropriately.
• 26:39
  SPEAKER_00: We'll see you on the next episode.
• 26:40
  SPEAKER_00: Take care, bye.