Latest / This Is Why with Dr. Busti / Types of Bias in Clinical Research & Impact on Results

Transcript

Types of Bias in Clinical Research & Impact on Results · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Hey, it's Dr.
• 0:01
  SPEAKER_00: Bustaye, and we are in the Study Design Development and Interpretation Module or Playlist.
• 0:07
  SPEAKER_00: And this lecture is going to focus specifically on the type of bias in clinical research.
• 0:14
  SPEAKER_00: And as we move through, guys, and understand the steps of clinical research, we set ourselves up to be more efficient, but also positioned to improve the quality of the research that's produced, which naturally then leads to a greater acceptance of publication and dissemination of our results.
• 0:33
  SPEAKER_00: Okay, but the key here, guys, is quality of research.
• 0:37
  SPEAKER_00: The researcher should be asking themselves these questions before they start, just as much as the reader should look at the study and ask themselves what the researcher should have done to control for the risk of their own as well as outside bias that could impact what?
• 0:57
  SPEAKER_00: The internal validity of the study.
• 1:00
  SPEAKER_00: Because if the results are internally compromised, then the external validity or application of the general public is also compromised.
• 1:11
  SPEAKER_00: And so let's make sure we know how to prevent and/or consider that.
• 1:16
  SPEAKER_00: Hey guys, I'm Dr.
• 1:18
  SPEAKER_00: Bustaye, and thank you so much for joining This Is Why, a show that helps healthcare professionals transform their medical knowledge from what to why, because when you know why you do what you do, in the context of the best available evidence, it changes the how, and that changes everything about the patient care decisions that we make every day, doesn't it?
• 1:39
  SPEAKER_00: Now, as you've heard from me before, each lecture in this series or module is part of a larger learning sequence.
• 1:49
  SPEAKER_00: Okay, and so if you need access to that, go explore the full collection or playlist that's available at thisiswhy.health or on YouTube or the podcast channel.
• 1:57
  SPEAKER_00: Okay?
• 1:58
  SPEAKER_00: Now these end-of-topics, including this one here on BIOS, can stand alone.
• 2:04
  SPEAKER_00: But if you need a step-by-step process, that's what these playlists and these modules are put together for, because some topics are built upon.
• 2:13
  SPEAKER_00: Okay?
• 2:14
  SPEAKER_00: Alright, so here we are.
• 2:16
  SPEAKER_00: We've covered in the other parts of this module hypothesis testing, getting that null and alternative hypothesis, study design, making sure you know why one study was done over another, their pros and the cons.
• 2:29
  SPEAKER_00: And we're on types of bias, and then they'll be following this that internal and external validity, because that's where it all kind of comes together.
• 2:38
  SPEAKER_00: Okay.
• 2:39
  SPEAKER_00: Now, when we think about the steps in clinical research, again, we go through that study concept followed by the securing the funding if that's relevant, preparing a study protocol.
• 2:51
  SPEAKER_00: So most of where we're anchoring our time are in the that first one and that third step before we get started.
• 2:59
  SPEAKER_00: But as a clinician who's reading the paper, I should still understand this process.
• 3:06
  SPEAKER_00: So some of you that may be listening go, I don't I don't do research and I don't care.
• 3:10
  SPEAKER_00: I don't want to do it.
• 3:11
  SPEAKER_00: In fact, I don't like it.
• 3:13
  SPEAKER_00: I get that.
• 3:14
  SPEAKER_00: But you still are expected to do lifelong learning and understand what the researcher should have done or did do so that you can determine if their results are internally valid.
• 3:26
  SPEAKER_00: The researcher would hope to be motivated to maximize the internal validity of their results.
• 3:34
  SPEAKER_00: So our goal here in this particular topic, when we're developing study protocols and following like spirit, and we already are anticipating the consort if we're doing a randomized control trial or some other publication criteria or set of guidelines that a journal might require of us, even for a case report, okay, we need to think about our own bias that we bring into the process and also biases that are externally brought into this process, okay, so that we can mitigate that up front or try our best to control and reduce it.
• 4:10
  SPEAKER_00: We might have to acknowledge it, but at least we're making aware of that.
• 4:14
  SPEAKER_00: Now, all of this matters because the thing that we measure, the result that we get, is influenced by these outside influences, ours and these other ones, right?
• 4:26
  SPEAKER_00: And so there's this equation that's commonly used, it's the equation of error, and it just kind of helps to create a framework of that concept, right?
• 4:37
  SPEAKER_00: So you have the measurement of the result, the thing that you are doing.
• 4:40
  SPEAKER_00: Well, there's somewhere out there is the real truth.
• 4:45
  SPEAKER_00: Okay?
• 4:45
  SPEAKER_00: That's the real value.
• 4:48
  SPEAKER_00: It would be ideal if you could figure it out, but most studies don't, right?
• 4:52
  SPEAKER_00: We try to get close to that, right?
• 4:56
  SPEAKER_00: But but we sometimes, depending on the size of the study, we move, we are farther away or closer to the population or the true result mean.
• 5:04
  SPEAKER_00: Okay.
• 5:05
  SPEAKER_00: Then there's some introduction of bias.
• 5:08
  SPEAKER_00: And then there's the random error.
• 5:10
  SPEAKER_00: Okay, so what are those?
• 5:11
  SPEAKER_00: Well, bias is that systematic error, and these are going to be influenced by the researcher and the even the reader.
• 5:18
  SPEAKER_00: What is your bias?
• 5:19
  SPEAKER_00: Maybe you have an affiliation, or maybe you've done some clinical research, or maybe you have a particular experience with certain medicines or treatments or interventions, and you already come to the paper with positive or negative perspectives on what you've read in the abstract or the title.
• 5:36
  SPEAKER_00: Okay?
• 5:37
  SPEAKER_00: Random error.
• 5:38
  SPEAKER_00: Uh, this can be influenced by the sample size.
• 5:41
  SPEAKER_00: Uh looking at your confidence interval.
• 5:44
  SPEAKER_00: How wide are they?
• 5:44
  SPEAKER_00: Remember, we don't typically want really wide confidence intervals because somewhere in that confidence interval is the true result.
• 5:53
  SPEAKER_00: So the wider it is, the more variability it is.
• 5:56
  SPEAKER_00: We like tight 95% confidence intervals, and we like small p-values because that's the risk of the like the the smaller the p-value, the less likely that my results are random.
• 6:11
  SPEAKER_00: I don't want them to be random.
• 6:12
  SPEAKER_00: I want them to be definitive.
• 6:15
  SPEAKER_00: Okay?
• 6:16
  SPEAKER_00: So one of the big things that contributes to getting to the truth is a use of a good study design that controls for confounders, that recognizes the potential areas of bias, and preemptively, prospectively controls for those.
• 6:34
  SPEAKER_00: Even if you're doing a retrospective study, you can still in your design factor those things in and look for those things and try to accommodate and/or to do statistical modifications in your analysis to control for those confounders, making sure you get adequate sample size.
• 6:53
  SPEAKER_00: And then for the reader, again, this is down here, right?
• 6:56
  SPEAKER_00: These are the clinicians, these are the researchers on top who are deciding these things.
• 7:01
  SPEAKER_00: Using a critical appraisal tool or CAT, okay, remember that is just a systematic approach to how you review a paper, right?
• 7:10
  SPEAKER_00: Recognizing your own bias, knowing how to interpret statistical metrics.
• 7:15
  SPEAKER_00: And again, if you need help with those, yeah, provide lectures that dive into more of the understanding of those things so that you can apply them appropriately and know what they mean to you.
• 7:28
  SPEAKER_00: Okay.
• 7:29
  SPEAKER_00: This all matters because we're trying to get to ideally this scenario, okay?
• 7:36
  SPEAKER_00: This is just that equation, kind of in a different through a different lens and perspective.
• 7:42
  SPEAKER_00: So let's look at that, okay?
• 7:44
  SPEAKER_00: So we have over here random error, okay?
• 7:48
  SPEAKER_00: Low random error, high random error.
• 7:51
  SPEAKER_00: Okay.
• 7:52
  SPEAKER_00: And then up here is our bias.
• 7:55
  SPEAKER_00: Okay.
• 7:56
  SPEAKER_00: So we have low and high.
• 7:58
  SPEAKER_00: So we want low bias and we want low random error.
• 8:03
  SPEAKER_00: Okay?
• 8:04
  SPEAKER_00: And so that allows us to get to the desired truth closest and mimic it.
• 8:10
  SPEAKER_00: Okay?
• 8:11
  SPEAKER_00: That's that true result, is in the center of the bullseye that we all want to hit.
• 8:16
  SPEAKER_00: Okay?
• 8:17
  SPEAKER_00: But you can see what happens when our bias is high, we might still have low risk for random error, but we're not hitting the target.
• 8:26
  SPEAKER_00: Okay?
• 8:27
  SPEAKER_00: Same thing when we have random error, our results don't mean anything because they're hitting all over the place.
• 8:33
  SPEAKER_00: You see?
• 8:35
  SPEAKER_00: Just randomly showing up all over the place.
• 8:38
  SPEAKER_00: Whereas if our bias, our our random is high and our bias is high, we're never getting to the, we're never getting true result.
• 8:47
  SPEAKER_00: We're having a publication bias, if you will, at some level.
• 8:50
  SPEAKER_00: Almost an inverted funnel plot looking effect, you know, where the weight of the data is all in one particular area.
• 8:57
  SPEAKER_00: And so there are different types of bias that we factor in.
• 9:01
  SPEAKER_00: And I just want to go through some of those and highlight the key kind of element to consider in that.
• 9:07
  SPEAKER_00: Many of these things are common questions on, again, your board exams, but things to consider selection allocation bias, where patients or subjects are not being randomly assigned to a group.
• 9:20
  SPEAKER_00: If I hand pick you and I know what background or intervention you receive or exposure, or where I I've seen you and I know what you do, well, I know what kind of result I'm likely going to get.
• 9:32
  SPEAKER_00: Okay?
• 9:33
  SPEAKER_00: So that's where randomization becomes so, so valuable.
• 9:38
  SPEAKER_00: Sampling bias.
• 9:40
  SPEAKER_00: Um, when you basically have patients that do not reflect or represent the general population that you're caring for.
• 9:49
  SPEAKER_00: You miss the group, right?
• 9:52
  SPEAKER_00: Pediatric versus adult, maybe.
• 9:55
  SPEAKER_00: Uh gender, right?
• 9:57
  SPEAKER_00: Pregnant, non-pregnant.
• 9:59
  SPEAKER_00: You know, all the certain disease, no, just is it representative of the population?
• 10:03
  SPEAKER_00: So you might read a paper that's very well done.
• 10:06
  SPEAKER_00: Appropriate intervention, appropriate outcome, but wrong patient population.
• 10:12
  SPEAKER_00: It's not gonna, it's not worth your time to read.
• 10:14
  SPEAKER_00: Okay.
• 10:15
  SPEAKER_00: Now, if it's the only paper that's out there and it's the only information available, then you can try to use that and extrapolate it.
• 10:24
  SPEAKER_00: We've got to be careful with that, right?
• 10:26
  SPEAKER_00: Confounding bias.
• 10:27
  SPEAKER_00: So that's that's when one factor or variable results in the influence of another variable's outcome.
• 10:34
  SPEAKER_00: Okay, and the in the study design lecture that I do, available also at uh the on the channel and the podcast channel, and at this is why.
• 10:56
  SPEAKER_00: Okay, so not uh controlling for that bias, lead time bias.
• 11:01
  SPEAKER_00: Well, this is important when we're screening for a diagnosis earlier than we would have normally.
• 11:07
  SPEAKER_00: Okay, so what ends up happening is you identify these patients that are at risk or they would not have normally presented, right?
• 11:15
  SPEAKER_00: You start to treat them, they live a certain duration, it makes it look like they've lived longer, you that intervention prolonged their lifespan, but in reality, if that patient was not in the study, they would not have presented because they were asymptomatic.
• 11:36
  SPEAKER_00: Or they weren't thinking about it until many times patients present, like I see them in them sometimes in the emergency department, like something's really wrong.
• 11:46
  SPEAKER_00: And I do a CT scan or something, and voila, there's a big mass or something, right?
• 11:53
  SPEAKER_00: So I find them when they're symptomatic, but if you go looking for them, it that lead in time can prolong the appearance of their survival and falsely make it look longer than normal.
• 12:06
  SPEAKER_00: Recall bias is a big one, especially when you're asking people surveys, retrospective analysis, going back and looking at exposures and trying to collect data.
• 12:16
  SPEAKER_00: You know, how much time back do I have to think and remember?
• 12:20
  SPEAKER_00: Most people don't accurately describe things, how much they eat, how much they drink, how much they exercise.
• 12:25
  SPEAKER_00: We all exaggerate that, okay, at some level.
• 12:28
  SPEAKER_00: So it can be misleading and misclassified, and that can lead to that differential misclassification.
• 12:35
  SPEAKER_00: Okay.
• 12:36
  SPEAKER_00: Procedure bias occurs when the patients in one group or another are not treated the same.
• 12:42
  SPEAKER_00: Okay.
• 12:43
  SPEAKER_00: And so do they get both, basically.
• 12:46
  SPEAKER_00: Even sham surgeries have been studied.
• 12:49
  SPEAKER_00: Okay.
• 12:50
  SPEAKER_00: Lost a follow-up.
• 12:52
  SPEAKER_00: Why did that person fall out of the study?
• 12:54
  SPEAKER_00: What happened to them?
• 12:56
  SPEAKER_00: What about them?
• 12:57
  SPEAKER_00: What is there something about those people that drop out of a study and do they tell a story?
• 13:02
  SPEAKER_00: Are they dropping out because the intervention was a problematic side effect?
• 13:06
  SPEAKER_00: Did they die from it and you didn't know?
• 13:09
  SPEAKER_00: That's so important that you are able to capture that information through death certificates and other surrounding information and data to validate that you weren't causing harm versus like, yeah, we don't know where they went.
• 13:22
  SPEAKER_00: Okay.
• 13:24
  SPEAKER_00: The Hawthorne effect is when patients change obviously their behavior in the context of being that they're known that they're being watched.
• 13:32
  SPEAKER_00: They know they're in a study.
• 13:33
  SPEAKER_00: So is that the real world effect?
• 13:36
  SPEAKER_00: The observer bias, right?
• 13:38
  SPEAKER_00: That we see that a lot sometimes in qualitative uh studies where the um interpretation is thrown off.
• 13:47
  SPEAKER_00: That leads us to this concept of differential and non-differential bias that you might also kind of hear uh potentially be tested on.
• 13:55
  SPEAKER_00: What it comes down to is understanding the role or influence on systematic error and bias on the impact on the groups.
• 14:04
  SPEAKER_00: So differential bias, okay, there's an unequal, okay, systematic error or bias that impacts the both groups, but it does it differently.
• 14:15
  SPEAKER_00: Okay.
• 14:16
  SPEAKER_00: Whereas the non-differential bias, there's equal systematic error and bias affecting both groups, okay, and it's it's the same.
• 14:27
  SPEAKER_00: Okay.
• 14:28
  SPEAKER_00: Um, why does that matter?
• 14:30
  SPEAKER_00: Well, that has to do with the disproportionate amount of subjects that are being misclassified compared to another.
• 14:37
  SPEAKER_00: And what that ends up ultimately doing in the differential, when we have unequal systematic error and bias, is that it pushes, it can push the results toward or away from the null hypothesis.
• 14:50
  SPEAKER_00: Remember, the null hypothesis is there's no difference, okay?
• 14:54
  SPEAKER_00: Um, and it's very unpredictable.
• 14:57
  SPEAKER_00: Whereas a non-differential bias, there's equal impacting both groups, okay?
• 15:04
  SPEAKER_00: But so there's if you have a misclassification of a disease or exposure, it's happening equally in both groups.
• 15:10
  SPEAKER_00: But that typically moves the results toward the null hypothesis.
• 15:14
  SPEAKER_00: And again, the null hypothesis would say there's no difference.
• 15:18
  SPEAKER_00: Okay?
• 15:19
  SPEAKER_00: Um, and so just kind of keep that terminology a little bit in mind.
• 15:24
  SPEAKER_00: Now, ways to reduce bias, most commonly in the big picture, and again, common high-yield core testable items, multi-center trial versus single center, being randomized versus non-randomized, okay, uh comparing to a placebo uh or active treatment, obviously.
• 15:44
  SPEAKER_00: Blinded studies that have double dummy treatment options so that the patient and the researcher both don't know what the what group the patient's in.
• 15:55
  SPEAKER_00: Crossover studies are very powerful.
• 15:57
  SPEAKER_00: Okay, we talked about that in a study design lecture, um, where the patients serve as their own control.
• 16:05
  SPEAKER_00: They take a little longer to do, they require a period of washout, but when patients serve as their own control, right, then the confounders are the same.
• 16:16
  SPEAKER_00: They're there.
• 16:17
  SPEAKER_00: They're already you know that's what part of randomization is trying to make sure these groups look relatively the same, but they're still independent of each other.
• 16:28
  SPEAKER_00: Um crossover studies are really good for that.
• 16:31
  SPEAKER_00: Now, obviously, in my lectures, what I try to do is bring some relevance to it.
• 16:36
  SPEAKER_00: Does some of the stuff that I talk about translate actually to be true to what I'm saying?
• 16:43
  SPEAKER_00: Is there evidence to support some of the things of controlling for bias?
• 16:47
  SPEAKER_00: And of course the answer is yes.
• 16:49
  SPEAKER_00: So I'm not gonna uh feed you guys stuff that's irrelevant, right?
• 16:53
  SPEAKER_00: The goal is to make sure that you understand why we're doing what we're doing.
• 16:57
  SPEAKER_00: So when you look at, and some of these papers have been around like we've done better over the past several decades to improve it.
• 17:05
  SPEAKER_00: That's why publication guidelines have been written, even protocol guidelines like spirit guidelines have been written.
• 17:12
  SPEAKER_00: All of these tools are addressing methodological quality problems.
• 17:20
  SPEAKER_00: Cats, critical appraisal tools, help us as readers and lifelong learners to appropriately assess papers and expect good information to come out.
• 17:32
  SPEAKER_00: Okay, this is a big, big, important paper that came out of by uh uh Doug Altman, uh who's a pioneer in this in this topic area, but basically he was looking at concealment, right?
• 17:47
  SPEAKER_00: Those who were like, I knew what group you were in or not.
• 17:50
  SPEAKER_00: So if you lost concealment, I knew what group you were in.
• 17:54
  SPEAKER_00: So blinding, essentially.
• 17:56
  SPEAKER_00: Okay, and I highlighted the main results here, but you can see that the odds ratios to just get to the point here were uh were exaggerated by 41% when concealment was inadequate.
• 18:11
  SPEAKER_00: Okay, 41% and 30% when there was unclear concealment.
• 18:17
  SPEAKER_00: Okay, and that was after adjusting for other aspects of quality.
• 18:21
  SPEAKER_00: The trials that were not double blinded also yielded larger estimates with odds ratios being exaggerated by 17%.
• 18:31
  SPEAKER_00: Okay, so again, that points to why would that happen?
• 18:35
  SPEAKER_00: Because we in our behavior changes whether we're the patient or the researcher in what we do when we know what we're getting and who's getting what.
• 18:47
  SPEAKER_00: Okay, um, and so you can see a different the impact of bias and various study characteristics like case-controlled studies.
• 18:54
  SPEAKER_00: Clearly, um, there is an impact for bias, right?
• 18:58
  SPEAKER_00: Differential reference, different reference tests can be not controlling for that.
• 19:04
  SPEAKER_00: Uh no description test, right?
• 19:07
  SPEAKER_00: No description population, and the impact that that has on the overall bias.
• 19:13
  SPEAKER_00: Okay, so that's what this table is trying to show you that I keep using over and over.
• 19:19
  SPEAKER_00: The reason that people love the gold standard RCT is because the potential for bias or systematic error is traditionally less because we can prospectively, preemptively control for those confounders and improve the internal validity of the paper.
• 19:34
  SPEAKER_00: But as you go down into observational studies, they're still analytic.
• 19:38
  SPEAKER_00: But as you go down and get to case reports, like that risk of bias starts rising.
• 19:44
  SPEAKER_00: And then when you get to qualitative studies, even though you can do them really well, they are subject to a higher risk of bias if they don't follow strict protocols and standards.
• 19:57
  SPEAKER_00: Um, and that again leads to that study design pyramid of why do we see this pyramid?
• 20:03
  SPEAKER_00: Why do we teach it?
• 20:04
  SPEAKER_00: Why do we show it to you?
• 20:06
  SPEAKER_00: Because when studies are done correctly with good methods, and they're controlling for bias, they produce and usually have more internally valid information.
• 20:17
  SPEAKER_00: You still need to know if it if that internal validity of the data that was reported is still externally relevant, right, clinically.
• 20:27
  SPEAKER_00: Um and so there is bias in the literature.
• 20:30
  SPEAKER_00: So can you imagine that you know many times publications get published because there's a positive finding?
• 20:37
  SPEAKER_00: But we should be publishing papers that also have a negative finding or nothing.
• 20:43
  SPEAKER_00: Yes, it's not as exciting, but we should give that, especially if it's well done, if there's no nothing there to be seen, then we should stop looking at it.
• 20:53
  SPEAKER_00: Okay?
• 20:55
  SPEAKER_00: So many times we see publication bias.
• 20:58
  SPEAKER_00: I talked about that earlier, because only positive studies get published.
• 21:02
  SPEAKER_00: Um, and so there are ways to do that.
• 21:04
  SPEAKER_00: Inverted funnel plots are very helpful in seeing that visual of the distribution of only positive looking papers.
• 21:14
  SPEAKER_00: And that's why clinical registries like clinicaltrials.gov and other international groups have required people to preemptively, prospectively say, hey, here's our protocol, we're registering our study, we're doing it, and it's expected to end this date.
• 21:30
  SPEAKER_00: And so, therefore, you can anticipate within three to six months or 12 months after that that the results will be published and presented at a meeting andor in the journal.
• 21:39
  SPEAKER_00: And if it doesn't, and you never see it come out, why not?
• 21:45
  SPEAKER_00: Okay?
• 21:46
  SPEAKER_00: So it's that transparency in that.
• 21:49
  SPEAKER_00: And so, is there any evidence of publication bias?
• 21:52
  SPEAKER_00: Again, papers that have been done over the years looked at this, and you can see they did analysis of institutional review.
• 22:00
  SPEAKER_00: Approved protocols.
• 22:02
  SPEAKER_00: And when they went in there and they looked at those protocols, and those that had significant findings were more likely to be published.
• 22:14
  SPEAKER_00: Versus those that didn't, right?
• 22:16
  SPEAKER_00: So there was a statistical significant association between the level of significance in the results and the ability to get it published.
• 22:25
  SPEAKER_00: And that's even a little bit of a knock on the journals and the editors, in my opinion, because we should embrace negative studies if the study was done right.
• 22:36
  SPEAKER_00: But right, we don't want studies that are high risk for type two errors, where you say there's no difference, but in reality there is.
• 22:44
  SPEAKER_00: Maybe because you had an underpowered study.
• 22:46
  SPEAKER_00: So it's terrible.
• 22:46
  SPEAKER_00: That's junk published.
• 22:48
  SPEAKER_00: We're not going to publish junk.
• 22:50
  SPEAKER_00: Right?
• 22:50
  SPEAKER_00: So this is where the quality of the research is very important to the information that we actually receive.
• 22:59
  SPEAKER_00: Alright, wrapping things up, bias can be introduced at many, many levels.
• 23:04
  SPEAKER_00: Okay.
• 23:05
  SPEAKER_00: It's best to control it prospectively or what people call a priori.
• 23:10
  SPEAKER_00: Okay.
• 23:11
  SPEAKER_00: This leads us to the involvement of the good study design.
• 23:14
  SPEAKER_00: If you remember the equation for error, that truth and one of the biggest things is study design in doing that.
• 23:24
  SPEAKER_00: Failing to control for these influences can also influence our perception and the results of the overall effect size, and that can jade us, right?
• 23:36
  SPEAKER_00: Jade the perspective on the information.
• 23:38
  SPEAKER_00: So I hope that you found this to be helpful.
• 23:41
  SPEAKER_00: Okay.
• 23:41
  SPEAKER_00: If you did, check out our collection playlist, not only on the YouTube channel, but at thisiswhy.
• 23:49
  SPEAKER_00: We have full collections and playlists.
• 23:51
  SPEAKER_00: Try to organize them in a user-friendly format, okay, so that we walk you through evidence-based medicine concepts in a step-by-step fashion.
• 24:00
  SPEAKER_00: The goal is to make sense of it.
• 24:02
  SPEAKER_00: They do stand alone.
• 24:04
  SPEAKER_00: If you are preparing for board exams, okay, over at thisiswide.health, we give you a coupon.
• 24:11
  SPEAKER_00: We can do practice questions on this subject matter.
• 24:14
  SPEAKER_00: Okay.
• 24:15
  SPEAKER_00: Because you need to be able to apply the core concept.
• 24:18
  SPEAKER_00: Does it make sense talking about it?
• 24:20
  SPEAKER_00: Maybe it makes sense in doing a journal club, or maybe you're the researcher.
• 24:23
  SPEAKER_00: But at the end of the day, you're going to take exams for those of you that are still in that level of training.
• 24:28
  SPEAKER_00: If you're a program director or a clinician and you want to use this for training, we offer this and we offer different services.
• 24:35
  SPEAKER_00: If you need reports, you need practice tests, all those things, we offer that service over that high yield med review.
• 24:41
  SPEAKER_00: So go check that out.
• 24:43
  SPEAKER_00: So thanks again for joining.
• 24:44
  SPEAKER_00: Appreciate you guys.
• 24:45
  SPEAKER_00: If you like this, give us a like.
• 24:47
  SPEAKER_00: That really helps us a lot on our channels.
• 24:50
  SPEAKER_00: And we'll see you next time.
• 24:52
  SPEAKER_00: Take care.