Latest / This Is Why with Dr. Busti / Alpha Value Explained (Type I Error)| Biostatistics Series

Transcript

Alpha Value Explained (Type I Error)| Biostatistics Series · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: All right, guys, we are in the Biostatistics Series on Statistical Analysis Plans or SAP.
• 0:07
  SPEAKER_00: And in this episode, we're going to focus our discussion on the alpha value, why it's set a priori, what it means to you as you interpret the results.
• 0:18
  SPEAKER_00: Now, whether you're participating in research or reading the clinical literature, this is a concept that you need to understand in its proper context.
• 0:25
  SPEAKER_00: And I'm going to make it easy for you.
• 0:27
  SPEAKER_00: And with that, I'm Dr.
• 0:28
  SPEAKER_00: Bustaye, your faculty and host on This Is Why, a show that is set out to help healthcare professionals transform medical knowledge from what to why.
• 0:39
  SPEAKER_00: Because when we know why we do what we do, it changes the how.
• 0:43
  SPEAKER_00: And that changes everything about patient-oriented decisions in the context of the evidence that we should be making every day.
• 0:50
  SPEAKER_00: Alright, now with that, putting us in the context of the overall outline of this biostatistic series, there's their part one, which is before the study starts, which is what the researcher is supposed to do.
• 1:02
  SPEAKER_00: That is our statistical analysis plan, that's where we are at.
• 1:05
  SPEAKER_00: There's also part two, which is once you complete the study, doing that statistical analysis, known as summary statistics.
• 1:13
  SPEAKER_00: And then part three really is designated for the diagnostic test.
• 1:17
  SPEAKER_00: When we think about part one, we are focused mainly in the area of uh the applicability of the alpha that leads also and influences the power analysis calculation, which is in another separate lecture.
• 1:32
  SPEAKER_00: But this goes through a process.
• 1:35
  SPEAKER_00: Each of these topics stand alone as individual concepts that you need to know.
• 1:40
  SPEAKER_00: However, there is some element of a foundation that is built upon.
• 1:43
  SPEAKER_00: So if you need that, please check out our collections or playlists at the YouTube channel, our podcast channels, or at thisiswhy.health.
• 1:53
  SPEAKER_00: As we finish up this section, we'll move into when you're done, sort of the statistical interpretation, those summary statistics that go back and have to do with this orientation.
• 2:03
  SPEAKER_00: And alpha is critical to that.
• 2:06
  SPEAKER_00: And we break those down in descriptive versus inferential statistics, but we'll see that alpha is really reserved when we plan to do inferential statistical analysis.
• 2:17
  SPEAKER_00: And then the last one again is diagnostic test, where we focus mostly on the test itself.
• 2:22
  SPEAKER_00: However, alpha can apply in this situation when you are comparing more than one diagnostic test, maybe to its uh to another test that would be done for obviously a diagnostic study.
• 2:37
  SPEAKER_00: So, first things first, alpha values.
• 2:41
  SPEAKER_00: They sure they should be determined a priori.
• 2:44
  SPEAKER_00: That means before the study begins.
• 2:47
  SPEAKER_00: So a good clinical researcher will do that.
• 2:50
  SPEAKER_00: Many times the IRB and the protocol development team will require that before you get started.
• 2:56
  SPEAKER_00: And the reason that has to do is it matters is because you're trying to define at some level where that level of significance, if you're going to find a difference, what is it gonna take to get there so that you can actually try to achieve that if it's truly present?
• 3:12
  SPEAKER_00: Right?
• 3:13
  SPEAKER_00: Now, alpha is also something that is known to be the acceptable level of probability or risk of making a type 1 error.
• 3:23
  SPEAKER_00: Now remember, a type 1 error is you say at the end of a study there is a difference, but in reality there isn't.
• 3:30
  SPEAKER_00: And the most common way that we kind of represent that is this kind of two by two table, where I'll try to get you oriented here on the left side, and those of you on our podcast channels, again, this is a two by two with labels kind of for the rows and columns.
• 3:46
  SPEAKER_00: And on the left that represent the rows, that is your decision, right?
• 3:50
  SPEAKER_00: So this goes back to our topic on hypothesis.
• 3:54
  SPEAKER_00: You have a null hypothesis and you have an alternative hypothesis.
• 3:57
  SPEAKER_00: Are you going to accept or reject the null or uh the hypothesis?
• 4:02
  SPEAKER_00: And then, of course, there is the that's the decision that you have to make in the study, but then there's also the reality.
• 4:08
  SPEAKER_00: What is the truth?
• 4:10
  SPEAKER_00: Remember, we're trying to discern the truth if it's there.
• 4:13
  SPEAKER_00: All right?
• 4:13
  SPEAKER_00: And so the way you work this table is if you make the decision to accept the null hypothesis and in fact the null, i.e.
• 4:22
  SPEAKER_00: there is no difference between the interventions or exposures or the two groups that you evaluated, then you have made obviously the correct decision.
• 4:31
  SPEAKER_00: However, if you accept the null, you have failed to reject it.
• 4:36
  SPEAKER_00: That means there's no difference found, right?
• 4:39
  SPEAKER_00: But in fact the null hypothesis is false, then you have made a type two error.
• 4:44
  SPEAKER_00: This really dives directly into power, which is another component of this that should be determined a priori.
• 4:52
  SPEAKER_00: Okay, again, separate lecture, focus on that if you need to uh dive deeper in that.
• 4:56
  SPEAKER_00: But it's important to recognize what it is and what is not.
• 4:59
  SPEAKER_00: Now, alpha is over here where you choose to reject the null.
• 5:03
  SPEAKER_00: You're saying, because remember, a null means no difference.
• 5:06
  SPEAKER_00: Okay, you reject the null and you say there is a difference.
• 5:11
  SPEAKER_00: You have rejected that there is no difference.
• 5:14
  SPEAKER_00: I know that sounds really weird to say and to maybe wrap your mind around it, but you've got to be used to looking at that comment from different angles.
• 5:23
  SPEAKER_00: Board exams love to throw those things at you.
• 5:26
  SPEAKER_00: But let's say you reject it, but in fact, the null hypothesis is true, and that's the thing that we don't want to make, right?
• 5:34
  SPEAKER_00: That type 1 error or alpha.
• 5:37
  SPEAKER_00: When we reject the null and the null is actually false, right, there is a difference, then that is when we've made the correct decision.
• 5:46
  SPEAKER_00: So that's what we're really driving down into in this discussion is trying to avoid saying that something is different when in fact it isn't.
• 5:56
  SPEAKER_00: Now, again, the researcher is supposed to define that in advance appropriately, but you as a reader clinician interpreting the literature, having to evaluate it to determine if there's internal validity and something meaningful here, you should be looking for that in the methodology section and putting in its context into a one-sided or it's two-sided or one-tail, two-tail analysis.
• 6:22
  SPEAKER_00: Okay, so when is it going to be used?
• 6:25
  SPEAKER_00: When should alpha, because alpha is not going to show up everywhere.
• 6:28
  SPEAKER_00: Okay, so if the study design is going to compare groups or tests of association where you want to try to make a claim for clinical significance, right, that p-value, and that's desired, that's when you are going to need to define in advance, preferred, the that alpha value that you're setting, okay?
• 6:53
  SPEAKER_00: Because that helps us to understand what is the level of risk that we're making a type 1 error if we find a difference.
• 7:00
  SPEAKER_00: Now, studies that are testing hypothesis also that need inferential statistics.
• 7:06
  SPEAKER_00: Remember that summary statistics, part two that I mentioned at the beginning, that's where we're um you have descriptive analysis and you have inferential statistics.
• 7:16
  SPEAKER_00: And inferential statistics being the generation of p-values, 95% confidence intervals, odds ratios, risk ratios, relative risk, all those kinds of things, right?
• 7:25
  SPEAKER_00: When you're doing that, that is when you need to consider establishing as a researcher that alpha value.
• 7:33
  SPEAKER_00: When is it not to be used?
• 7:34
  SPEAKER_00: Okay.
• 7:35
  SPEAKER_00: So descriptive studies where you're describing the data, mean, media, modes, those kind of things, right?
• 7:41
  SPEAKER_00: Exploratory or pilot studies, you're just laying the groundwork, the foundation, qualitative analysis.
• 7:47
  SPEAKER_00: Remember that human experience.
• 7:49
  SPEAKER_00: And again, if you're not familiar with these study designs, I have a whole lecture designed specifically for that topic.
• 7:56
  SPEAKER_00: Help you to get oriented to the various types, when to use them, when not to use them, what are the pros, what are the cons, what's their purpose, what do we not achieve with certain types of studies.
• 8:05
  SPEAKER_00: All right.
• 8:06
  SPEAKER_00: This is another example of that.
• 8:07
  SPEAKER_00: But it also includes diagnostic studies.
• 8:10
  SPEAKER_00: Okay?
• 8:11
  SPEAKER_00: And where this shows up is when you're comparing two different diagnostic tests.
• 8:19
  SPEAKER_00: Okay.
• 8:19
  SPEAKER_00: Now, if you're just looking at the diagnostic test itself, right, and you're not comparing those tests to another, one test to another, then that's that's where this would apply.
• 8:32
  SPEAKER_00: Because you're you're looking at in those situations where you're only looking at the individual tests and how it performs, those are things like the sensitivity, specificity, the um receiver operator curves, area under the curve, and likelihood ratios and those kind of things.
• 8:48
  SPEAKER_00: All right.
• 8:49
  SPEAKER_00: All right.
• 8:51
  SPEAKER_00: So alpha, all right, and the sample size become really important to the power of a study.
• 8:59
  SPEAKER_00: And that's why I talk about power at some level in the context of this particular focus topic on alpha.
• 9:06
  SPEAKER_00: You can't just fully look at one of these things by themselves.
• 9:10
  SPEAKER_00: They do interact, okay, they do play a role in influencing each other.
• 9:14
  SPEAKER_00: And most researchers will not typically uh uh select an alpha that is going to be greater than 0.05 or 5%.
• 9:26
  SPEAKER_00: Okay.
• 9:27
  SPEAKER_00: I mean, again, that's the risk of making a type 1 error, right?
• 9:30
  SPEAKER_00: And they won't go less than two standard deviations from the mean, right?
• 9:35
  SPEAKER_00: That's what we typically see people do.
• 9:37
  SPEAKER_00: So we normally accept this alpha of being less than 0.05 as being at minimum the target.
• 9:46
  SPEAKER_00: Okay?
• 9:47
  SPEAKER_00: Now remember, this is re-influenced in part by the wording of your primary objective, which then really is your null and alternative hypothesis.
• 9:59
  SPEAKER_00: Okay?
• 10:00
  SPEAKER_00: So I again in that lecture on the hypotheses that you've generated for the study, it's the question you're asking, which is so important for helping you to decide what kind of data are you dealing with.
• 10:12
  SPEAKER_00: But remember, it can be one or two-tailed and go in either direction.
• 10:15
  SPEAKER_00: And remember, one-tailed or one-sided basically specifies a direction between the intervention that is being made by us as the researcher or the researcher who did the study and the outcome.
• 10:28
  SPEAKER_00: Okay?
• 10:29
  SPEAKER_00: That effect can be positive or it can be negative, positive or negative in its association.
• 10:36
  SPEAKER_00: Okay?
• 10:37
  SPEAKER_00: Now many believe that it's not appropriate to typically do a one-sided, and that is not the norm accepted in the literature.
• 10:45
  SPEAKER_00: Why is that?
• 10:46
  SPEAKER_00: Well, in part because you are assuming a direction.
• 10:49
  SPEAKER_00: That means you already have a presumption of the result.
• 10:52
  SPEAKER_00: Are you truly objective?
• 10:54
  SPEAKER_00: Are you willing to do a two-tailed that leaves open that chance that your results, right, and that outcome could go either direction.
• 11:05
  SPEAKER_00: You might have a direction that you think it's going to go, you hope it's going to go, maybe it's based on some preliminary evidence, pilot study, or case reports.
• 11:15
  SPEAKER_00: You have an idea, but to be transparent, objective, and to make the study more robust and valid, ideally, most papers are published in a two-tell.
• 11:27
  SPEAKER_00: That becomes really important, that angle, that context, that perspective, not only relates to affile value, which I'm going to show you here in a minute graphically, but becomes very relevant in also your power analysis and the calculation that you would do to determine the sample size that you need to avoid making a type 2 error.
• 11:48
  SPEAKER_00: Remember, you want to find a difference if it really exists.
• 11:51
  SPEAKER_00: And you need appropriate number of people in order to do that.
• 11:56
  SPEAKER_00: Okay?
• 11:56
  SPEAKER_00: So what difference does that make on the hypothesis of one-sided or two-sided?
• 12:00
  SPEAKER_00: Well, it has to do with the distribution of that alpha value that you have predefined.
• 12:07
  SPEAKER_00: That alpha value of 0.05 at minimum that we most accept or 5% is getting distributed either in one direction or it's being divided, okay, if it's two-sided, which again is the most common, accepted one in the literature for well-done studies.
• 12:24
  SPEAKER_00: Now you're dividing it out.
• 12:26
  SPEAKER_00: And I want to show you what that looks like graphically because the area under the curve there, where that point, that what's called z-score down here, okay, that we get from tables, that z-score is that point of basically there's a difference, right?
• 12:42
  SPEAKER_00: And so if this is this is representative of a one-tailed or one-sided alpha value, and that value would be 1.645, which becomes very relevant in the calculation of power analysis.
• 12:57
  SPEAKER_00: That's why it matters.
• 12:59
  SPEAKER_00: Okay?
• 13:01
  SPEAKER_00: So the 5% over there, that's the area.
• 13:04
  SPEAKER_00: This space in here is the space that there is a difference, right?
• 13:11
  SPEAKER_00: That is going to be the space where we reject the null hypothesis.
• 13:15
  SPEAKER_00: And the null hypothesis says there's no difference, right?
• 13:18
  SPEAKER_00: And it's also the risk of making a type 1 error.
• 13:20
  SPEAKER_00: So the smaller that area is, the less chance we have made a type 1 error.
• 13:28
  SPEAKER_00: The harder it is to also find a difference.
• 13:31
  SPEAKER_00: And you see that because of the area that's left when you divide it by 2.
• 13:36
  SPEAKER_00: So it's 0.025 on one side, 0.025 on the other side when you use a two-tailed or two-sided approach with your hypothesis or your objective.
• 13:47
  SPEAKER_00: Okay?
• 13:48
  SPEAKER_00: So this is where orientation is so important in the individual words that are used not only in the protocol by the researcher, but also by you reading the paper, helps you to stay oriented to what your results of significance actually mean.
• 14:03
  SPEAKER_00: And if the researcher chooses to say in the methodology they did a two-sided, but then they take a one-sided approach after the study was done, that's a red flag.
• 14:15
  SPEAKER_00: That means they're looking for something, right?
• 14:17
  SPEAKER_00: They couldn't find a difference and they want to get published.
• 14:21
  SPEAKER_00: That kind of stuff is irritating, it's frustrating, we lose our credibility as clinicians, researchers when we do that kind of stuff, right?
• 14:28
  SPEAKER_00: And you lose respect for the paper and its integrity when that stuff is done.
• 14:32
  SPEAKER_00: But you can see now that the with the Z-score is a different number, 1.96.
• 14:38
  SPEAKER_00: And again, that that number is going to go into the power analysis.
• 14:42
  SPEAKER_00: And so, again, if you're not familiar with power analysis and the components that make up the calculation, not that you need to memorize all that, but it's important to have that context.
• 14:51
  SPEAKER_00: I'm going to show you in that lecture how easy you it is to calculate power or to figure out where you're at in orientation to a study that you're reading.
• 15:01
  SPEAKER_00: It is not that difficult.
• 15:02
  SPEAKER_00: Okay.
• 15:04
  SPEAKER_00: So when using a one-tailed or one-sided t-test analysis or using alpha that's one-sided in a post hoc analysis, i.e., you finished a study and now you're doing additional analysis that you didn't a priori mention in your paper or protocol, that raises that red flag.
• 15:26
  SPEAKER_00: Okay, we call that data dredging when you're trying to dive into the information and find something positive to draw attention to.
• 15:37
  SPEAKER_00: And so postdoc analysis raised the risk of bias in particular.
• 15:41
  SPEAKER_00: Okay.
• 15:42
  SPEAKER_00: Now, wrapping this up, okay, uh the take-home points that are relevant is that alpha is that threshold that the clinician or the researcher sets.
• 15:53
  SPEAKER_00: You can set it too as a reader of what you think it would should be or that you would like it to see for it to have been.
• 16:01
  SPEAKER_00: But the reality is it's supposed to be set a priority.
• 16:04
  SPEAKER_00: And that p-value, when it's determined at the level of the inferential statistical analysis, is what we generate to see if that difference exists, right?
• 16:15
  SPEAKER_00: Um, and again, alpha is the chance of making a type 1 error.
• 16:18
  SPEAKER_00: We are most comfortable setting that at minimum of 0.05 or 5%.
• 16:24
  SPEAKER_00: Again, one-tailed would be all 5%, that area over on one side.
• 16:29
  SPEAKER_00: Most common in the literature and the most accepted, respected way to do it is a two-tailed hypothesis.
• 16:36
  SPEAKER_00: Okay.
• 16:37
  SPEAKER_00: All right, I hope that was helpful.
• 16:39
  SPEAKER_00: You can access the full statistical analysis plan playlist at thisiswhy.health as well as other evidence-based medicine topics and collections that include playlists alongside other relevant areas in this whole thing.
• 16:55
  SPEAKER_00: If you're doing, if you're learning how to do research, this is a great approach from learning how to set up a protocol from the beginning and then how to analyze it after you're done.
• 17:03
  SPEAKER_00: If you're a clinician researcher or clinician and you don't care about doing research, then it teaches you the process of how to interpret and what that stuff means so that you can appropriately apply it to your patients.
• 17:16
  SPEAKER_00: That is the process of EBM.
• 17:17
  SPEAKER_00: It's step five.
• 17:19
  SPEAKER_00: Step four and five.
• 17:20
  SPEAKER_00: Apply and then assess your own clinical practice.
• 17:23
  SPEAKER_00: Okay, now if you need extra help on practicing applying this content for board exams, all right, that's why I started high yield med reviews.
• 17:32
  SPEAKER_00: Okay.
• 17:32
  SPEAKER_00: We have biostatistics cubanks over there for you.
• 17:37
  SPEAKER_00: Those members at thisiswhy.health get a discount that's free.
• 17:42
  SPEAKER_00: That's an investment, right?
• 17:44
  SPEAKER_00: But most board exams, nursing, pharmacy, and medicine, I've taken all of them, being that I've trained in all those areas, you will get statistical related questions on your boards and of course your coursework.
• 17:55
  SPEAKER_00: Why is that?
• 17:56
  SPEAKER_00: Because you're expected to invest in your license for the rest of your life.
• 18:01
  SPEAKER_00: It's called lifelong learning.
• 18:03
  SPEAKER_00: Things change, new evidence comes out.
• 18:05
  SPEAKER_00: You need to be able to read it.
• 18:06
  SPEAKER_00: So I hope this topic helped you to wrap your mind around alpha, and we'll see you next time.
• 18:10
  SPEAKER_00: Take care.