Latest / This Is Why with Dr. Busti / How a Systematic Review & Meta-Analysis is Done: Key Steps Explained

Transcript

How a Systematic Review & Meta-Analysis is Done: Key Steps Explained · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Okay, guys, not all systematic reviews and meta-analysis should change your clinical practice, even if they're published in a high-impact journal.
• 0:10
  SPEAKER_00: This is why understanding how a systematic review and meta-analysis is performed can be just as important as understanding its conclusions.
• 0:20
  SPEAKER_00: Every step from asking the right clinical questions, selecting the studies, to assessing the risk and evaluating for things like heterogeneity can strengthen or weaken the validity of the final results.
• 0:34
  SPEAKER_00: And as clinicians, we often do consider systematic reviews and meta-analysis to represent that highest level of evidence-based methodology.
• 0:45
  SPEAKER_00: But that only holds true when they're conducted using a rigorous, transparent, and evidence-based approach.
• 0:55
  SPEAKER_00: Okay, and if that process is flawed, the conclusions can also be flawed or even more problematic, misleading, regardless of how impressive the statistics appear.
• 1:07
  SPEAKER_00: And on that note, I'm Dr.
• 1:08
  SPEAKER_00: Bustaye, your host on This Is Why.
• 1:11
  SPEAKER_00: So welcome back to our review series on systematic reviews and meta-analysis.
• 1:17
  SPEAKER_00: Okay, it's part of our evidence-based medicine, biostatistics, clinical literature evaluation.
• 1:22
  SPEAKER_00: And this is why my goal is to help healthcare professionals transform their medical knowledge from what to why, because when you know why you do what you do in the proper context of the evidence, it changes the how.
• 1:35
  SPEAKER_00: And that changes everything about the patient-oriented decisions that we make on a daily basis.
• 1:41
  SPEAKER_00: Now, in this series, there are five main sort of topics or lectures.
• 1:47
  SPEAKER_00: And if you're coming in at this one first, I highly encourage you to stop here and go back and watch the foundation.
• 1:58
  SPEAKER_00: What is a systematic review versus a meta-analysis?
• 2:02
  SPEAKER_00: What is a Cochrane review?
• 2:03
  SPEAKER_00: What makes them similar, what makes them difference?
• 2:06
  SPEAKER_00: All right, what are the pros or cons?
• 2:08
  SPEAKER_00: It establishes a foundation because I'm going to build on that foundation in this particular lecture.
• 2:15
  SPEAKER_00: So if you're the one learning how to do a systematic review meta-analysis and you're going to conduct it, well, there are key steps that have to be done in order to do that the right way so that you prepare for proper reporting.
• 2:31
  SPEAKER_00: That's your publication, which we dive into in later lectures.
• 2:35
  SPEAKER_00: But more importantly, we get to the meta-analysis.
• 2:38
  SPEAKER_00: That quantitative or statistical assessment of the data to get to that pooled estimate.
• 2:47
  SPEAKER_00: Okay.
• 2:48
  SPEAKER_00: And then you've got to learn how to re how to report those results properly, how to interpret a forest plot.
• 2:55
  SPEAKER_00: What is a grade assessment?
• 2:56
  SPEAKER_00: Why do we use it?
• 2:57
  SPEAKER_00: When do we use it?
• 2:59
  SPEAKER_00: How do we publish this paper?
• 3:01
  SPEAKER_00: What is the criteria?
• 3:02
  SPEAKER_00: There's guidelines out there.
• 3:04
  SPEAKER_00: And then for those of us reading these, if we're not the researcher, just fine.
• 3:09
  SPEAKER_00: We need to make sure that our research colleagues did it correctly.
• 3:12
  SPEAKER_00: Because at the end of the day, I'd got to figure out what am I going to do with it.
• 3:15
  SPEAKER_00: And so there are critical appraisal tools that can guide us.
• 3:19
  SPEAKER_00: And again, if you've been a part of the series, this is a diagram that shows the two, right?
• 3:24
  SPEAKER_00: Systematic review and meta-analysis can exist on their own, but most commonly, if you're doing a systematic review, you might as well move on and do the meta-analysis in most cases.
• 3:37
  SPEAKER_00: I mean, it's very helpful if you can.
• 3:40
  SPEAKER_00: Vice versa, there's what's the point?
• 3:41
  SPEAKER_00: If you're going to do a meta-analysis, you have to do a systematic review.
• 3:45
  SPEAKER_00: Okay?
• 3:46
  SPEAKER_00: Knowing what's coming into the data.
• 3:49
  SPEAKER_00: All right.
• 3:49
  SPEAKER_00: And that's where this Venn diagram sort of thing looks like.
• 3:52
  SPEAKER_00: And this is what many groups around the world that are known experts that are contributors to these are doing.
• 4:00
  SPEAKER_00: Okay.
• 4:01
  SPEAKER_00: Now, let's break down those steps of what to do from the beginning to the end.
• 4:08
  SPEAKER_00: And then as we as I review these, I'm not going to stop and go into detail about every detail about that.
• 4:16
  SPEAKER_00: I'm laying the groundwork for what's coming in the rest of the lecture where we will dive deeper into some of those areas.
• 4:24
  SPEAKER_00: But the first thing is you got to define the question.
• 4:26
  SPEAKER_00: It needs to be clear.
• 4:28
  SPEAKER_00: And we follow the same structure of evidence-based medicine.
• 4:31
  SPEAKER_00: And if you're not sure about the five A's, right, and the process of doing evidence-based medicine and how to how to ask a clinical question, then I offer lectures on that inside of the playlist, whether you're on YouTube or at on our podcast channels at Spotify, Amazon, Apple Podcasts, and such, right?
• 4:51
  SPEAKER_00: So you define the question, pico picots.
• 4:55
  SPEAKER_00: Okay.
• 4:55
  SPEAKER_00: That's first.
• 4:57
  SPEAKER_00: Standard.
• 4:58
  SPEAKER_00: You register or publish the protocol.
• 5:02
  SPEAKER_00: A priori.
• 5:03
  SPEAKER_00: That's pre-specified production and codification of what you're going to do.
• 5:09
  SPEAKER_00: So that people know, hey, someone else is working on this prop uh project.
• 5:14
  SPEAKER_00: Okay.
• 5:16
  SPEAKER_00: Set the eligibility criteria, just like a clinical trial.
• 5:20
  SPEAKER_00: What papers and studies are you going to include and exclude?
• 5:24
  SPEAKER_00: That's very important.
• 5:26
  SPEAKER_00: And why are you doing that?
• 5:27
  SPEAKER_00: And then you run a comprehensive search.
• 5:29
  SPEAKER_00: We're definitely going to dive into that one.
• 5:31
  SPEAKER_00: Okay.
• 5:31
  SPEAKER_00: Because when I that word comprehensive, it's comprehensive.
• 5:36
  SPEAKER_00: If you do it the right way, it's very comprehensive.
• 5:39
  SPEAKER_00: And then you select the studies.
• 5:41
  SPEAKER_00: Okay.
• 5:42
  SPEAKER_00: Usually you're going to be using two independent reviewers.
• 5:47
  SPEAKER_00: Okay.
• 5:48
  SPEAKER_00: Separate.
• 5:50
  SPEAKER_00: And they're looking at the titles, abstracts, and then eventually get into the full data.
• 5:54
  SPEAKER_00: And then you extract that data and you assess it for risk of bias.
• 6:01
  SPEAKER_00: Again, I'm going to show you what that looks like.
• 6:04
  SPEAKER_00: I'm going to show you, introduce you to grade and some of these tools that we use, depending on the type of studies, randomized controlled trials versus non-randomized controlled trials.
• 6:13
  SPEAKER_00: Okay, there are established verified, validated tools out there that are recommended.
• 6:20
  SPEAKER_00: Then you synthesize.
• 6:21
  SPEAKER_00: If you're doing the meta-analysis part, you're going to synthesize.
• 6:30
  SPEAKER_00: So you're doing narrative synthesis, which is a systematic review, and then the quantitative analysis, which is that statistical pooling of the data and creating a pooled estimate, which is that diamond at the bottom of the forest plot that combines all the data.
• 6:50
  SPEAKER_00: Then you have to say, well, should you have combined the data?
• 6:53
  SPEAKER_00: Now, again, most of this you should know up front if the data is heterogeneous or not, but there are tests of heterogeneity that are done.
• 7:02
  SPEAKER_00: And I will again dive into some of those here today.
• 7:07
  SPEAKER_00: But in order to generate that pooled estimate, you have to look at the type of data you're dealing with, the the uh um what type of uh effect measure you're trying to uh produce and some nuances about the data because that influences the method and the models that are used and applied for generating that.
• 7:30
  SPEAKER_00: Okay, and again, I'll I'm gonna I'm gonna just trust me, I'm gonna dive into it, but I'm trying to make sure that I'm laying that groundwork here.
• 7:39
  SPEAKER_00: And then you present the findings, right, and you try to offer some degree of interpretation.
• 7:45
  SPEAKER_00: And when we compare this to the process of evidence-based medicine, those five A's, there's a mirrored process, okay, and same similar steps.
• 7:56
  SPEAKER_00: So step one is asking that question, the PICO.
• 7:59
  SPEAKER_00: Step two is finding the best available evidence that meets the inclusion exclusion criteria using two independent at minimum investigators.
• 8:10
  SPEAKER_00: Then you appraise in a systematic and reproducible way, okay, again by usually independent reviewers and investigators, and then you synthesize that information, and that's the QFAS sort of format.
• 8:28
  SPEAKER_00: Now, when you define a question, okay, stepping up the very beginning, Pico, PC, it needs to be clear.
• 8:36
  SPEAKER_00: And you should tell us what it is.
• 8:39
  SPEAKER_00: You should codify it in your protocol because that's the way that I it makes it reproducible at some level.
• 8:46
  SPEAKER_00: I should be able to look at that and know exactly how to set up my own search strategy.
• 8:52
  SPEAKER_00: And it should align and mirror what yours was.
• 8:56
  SPEAKER_00: Some people stop at PICO.
• 8:58
  SPEAKER_00: I tend to uh favor and encourage people to think about PICOTS, where T is that time of the onset of the effect or outcome that we're looking at.
• 9:10
  SPEAKER_00: Is this acute, chronic, is it in the next one hour, is it three to six months, five years from now?
• 9:17
  SPEAKER_00: Those things matter.
• 9:18
  SPEAKER_00: And they also influence then the setting because obviously the clinical environment also influences the application.
• 9:25
  SPEAKER_00: And so that's why I encourage that level of codification.
• 9:31
  unknown: Okay.
• 9:31
  SPEAKER_00: Now when we start thinking about that next step of, well, what do we include?
• 9:36
  SPEAKER_00: Okay, well, you want to include all available relevant data, whether it's positive or negative, because what you're trying to do is reduce publication bias, which is something I addressed or started to cover uh in the first lecture on this series.
• 9:52
  SPEAKER_00: Okay, we want to remove that as much as possible because we want the truth.
• 9:58
  SPEAKER_00: Okay.
• 9:59
  SPEAKER_00: Um, the next is that you define those inclusion and exclusion criteria in a pre-specified or a priori way.
• 10:08
  SPEAKER_00: Okay, that means up front before you do it.
• 10:13
  SPEAKER_00: No different than when you do a randomized control.
• 10:16
  SPEAKER_00: So the randomized control trial methodology mirrors this.
• 10:20
  SPEAKER_00: Okay, again, so that it's transparent and reproducible.
• 10:24
  SPEAKER_00: And in some cases, if I'm looking at data, I might even contact a clinical trial investigator if needed for clarification and or for additional information.
• 10:35
  SPEAKER_00: But then you got to go find it.
• 10:36
  SPEAKER_00: You can codify all that stuff, that's great.
• 10:38
  SPEAKER_00: Inclusion, exclusion criteria, that's awesome.
• 10:41
  SPEAKER_00: But you gotta go get it.
• 10:42
  SPEAKER_00: And when I remember I used the word early on in this lecture about comprehensive search, right?
• 10:48
  SPEAKER_00: Remember that's part of the five A's also of evidence-based medicine.
• 10:52
  SPEAKER_00: Acquire, you gotta go create a very robust search strategy, and it's an exhaustive uh review of many search engines and access to uh data.
• 11:05
  SPEAKER_00: So Medline, obviously, MBase, uh Central, which is the Cochrane Center Register for Controlled Trials.
• 11:14
  SPEAKER_00: Uh you have reference lists in the articles, right?
• 11:18
  SPEAKER_00: The citations that people publish or can be buried in there, published abstracts, even uh scientific meetings, okay, uh doctoral thesis, okay.
• 11:31
  SPEAKER_00: I mean, that's deep.
• 11:33
  SPEAKER_00: Digging into that stuff because sometimes they're in the library at the institution.
• 11:37
  SPEAKER_00: They're not always necessarily available as easily.
• 11:42
  SPEAKER_00: Trial registries, right, like clinicaltrials.gov, personal contact with those experts, unpublished studies to regulatory groups, okay, like the FDA.
• 11:53
  SPEAKER_00: And when possible, build a search in collaboration with an information specialist, like a medical librarian, okay, because they can help you fine-tune and really strengthen uh the search depending on which tool that you're using, especially if you are early on in doing search strategies.
• 12:15
  SPEAKER_00: They can be an excellent uh resource and point you to some of those standards.
• 12:20
  SPEAKER_00: Okay, so you you have asked the questions, you've codified the protocol, you've started your search, you're now acquiring the information, and now you need to begin assessing the quality and the risk of bias in those trials and studies that you're using.
• 12:39
  SPEAKER_00: And there's a number of tools out there.
• 12:42
  SPEAKER_00: Cochrane has their handbook for systematic reviews that is, you know, way up there, okay, very stringent, um, a standard for sure.
• 12:51
  SPEAKER_00: And then there's preferred scales or tools uh that are used to assess the quality of studies reported in the literature.
• 12:58
  SPEAKER_00: You can think for randomized controlled trials, yes, you have your the Cochrane's uh risk of bias two, ROB2, uh that tool.
• 13:07
  SPEAKER_00: If you have a non-randomized uh study, that's the Robbins I, that's for interventions.
• 13:14
  SPEAKER_00: That's the it's not one, it's an I, and for interventions.
• 13:17
  SPEAKER_00: Diagnostic accuracy studies, you have your Quatus II.
• 13:21
  SPEAKER_00: Historical um tools that were sometimes used, or JDAT scale, okay, the um Downs Black checklist, um, those kind of things, they've largely been replaced a little bit more with AMSTAR2 and Robus, R-O-B-I-S.
• 13:36
  SPEAKER_00: Okay.
• 13:37
  SPEAKER_00: But scoring quality with a single number is largely kind of no longer really a recommended, like reaching a certain threshold where we become dogmatic.
• 13:49
  SPEAKER_00: Um, a yes or no.
• 13:50
  SPEAKER_00: And if it doesn't cross that line, then doesn't all of it's irrelevant.
• 13:53
  SPEAKER_00: Okay.
• 13:54
  SPEAKER_00: Historically, okay, this is what they look like.
• 13:58
  SPEAKER_00: So you'd have the papers lined up, you can see on top the the author's name and the year, and then there on the on the on the over here on the on the right side of this again example, you see that the variables, the random sequencing, the allocation concealment, and you're grading it, right?
• 14:16
  SPEAKER_00: There's there's these symbols, and you sort of step back away and you can sort of visualize what is the distribution of risk for bias just visually.
• 14:29
  SPEAKER_00: I mean, it's super helpful.
• 14:30
  SPEAKER_00: Obviously, you want to see a bunch of green, you don't want to see much red.
• 14:34
  SPEAKER_00: And in this image, on this example, and I realize some of you are listening to this on our podcast channels, but it's just an example of the ROB1 tool, uh, the risk of bias assessment one tool.
• 14:47
  SPEAKER_00: Okay, and then you sort of get this breakdown of the distribution, and so this again gives us because no study's perfect, right?
• 14:55
  SPEAKER_00: All these papers are inherently have something wrong with them at some point, right?
• 15:01
  SPEAKER_00: So when you combine them together, you're wanting to know what is the general feel of this data?
• 15:08
  SPEAKER_00: And these kind of visuals, again, historically of what they look like.
• 15:11
  SPEAKER_00: This is the more updated version of that risk of uh bias assessment, ROB2.
• 15:18
  SPEAKER_00: Um, and and you again they usually give you a legend at the bottom over here that breaks down the areas of bias that you are grading for each individual study.
• 15:31
  SPEAKER_00: So this looks like a forest plot.
• 15:34
  SPEAKER_00: Okay, it mirrors it, but it's broken down by those studies that are low risk for bias.
• 15:40
  SPEAKER_00: If you look over here on the left hand side, these are all the low risk for bias, some concerns for bias, and then high risk for bias.
• 15:50
  SPEAKER_00: And then there's the totality of that evidence combined together, and you get again a visual compared by study by study.
• 16:00
  SPEAKER_00: And I think again, it helps us to just take that high-level view and go, should these studies be, you know, are is there a concern here?
• 16:10
  SPEAKER_00: Is there's concern about the validity of the outcome as we do that?
• 16:15
  SPEAKER_00: And so again, now we've collected the data, or we've we've asked the question, we've done our search, we've acquired it, we've done an assessment of the data itself, and now we're getting to that point where we need to do analysis.
• 16:30
  SPEAKER_00: Okay, so again, in our pre-specified portion of the protocol that we submit or we should write up or publish online, we are defining certain steps.
• 16:43
  SPEAKER_00: Okay, and that's what I'm stepping into now because we're getting to how do you take that data, combine that data, and do the statistical or quantitative assessment.
• 16:54
  SPEAKER_00: And those are the determining the effect uh measure.
• 16:58
  SPEAKER_00: What that's gonna be based on the type of data that we're dealing with.
• 17:02
  SPEAKER_00: Then you do the choice of the uh choice of model, and you're gonna do is the assumptions made about the effects.
• 17:11
  SPEAKER_00: Okay, are they true?
• 17:13
  SPEAKER_00: Okay, um, and then the choice of method of the method that we utilize to calculate and do this uh study.
• 17:22
  SPEAKER_00: And guess what?
• 17:22
  SPEAKER_00: In the forest plot, because I know some of you are like, what is he talking about?
• 17:26
  SPEAKER_00: In the forest plot, I'm leading to it, and I'm gonna show you an example of it.
• 17:31
  SPEAKER_00: They list this stuff for you.
• 17:35
  SPEAKER_00: Some of you may have never seen it before, it's been there, but you blew right past it.
• 17:41
  SPEAKER_00: But the forest plot is rich with information in a very simple codified format that allude to this, tell you what they did.
• 17:55
  SPEAKER_00: So I want to make sure you're oriented to why they did it's there on the forest plot and what it means.
• 18:04
  SPEAKER_00: That's where we're headed.
• 18:06
  SPEAKER_00: And then there's the actual calculation of that summary statistic, okay, where we get that pooled estimate, we generate the 95% confidence intervals, and that creates the big diamond or at the bottom, right?
• 18:22
  SPEAKER_00: Then we go into another step and say, okay, well, should we have combined this data?
• 18:29
  SPEAKER_00: All right, and that's where we do tests of uh heterogeneity, and we sometimes do sensitivity assessments, okay, to make sure, should we have done this?
• 18:40
  SPEAKER_00: Now, I'll tell you that the tests of heterogeneity are not done, you don't figure out which method and model to use after you get all the data.
• 18:50
  SPEAKER_00: That's like data dredging in a randomized controlled trial.
• 18:53
  SPEAKER_00: Those are things that are codified in the pre-specified a priori in the protocol.
• 18:59
  SPEAKER_00: You should already know at some level what this data looks like because remember, you've done an assessment of it before you got to the part where you're now crunching the actual numbers.
• 19:10
  SPEAKER_00: But you don't want to kind of go back and forth and keep adjusting, like, oh, we're gonna get we're gonna change the protocol, we're gonna get rid of that paper because we want we want the result to look a certain way.
• 19:19
  SPEAKER_00: That's bias.
• 19:21
  SPEAKER_00: Okay.
• 19:23
  SPEAKER_00: Now, I want to say that same thing for those of you that may like different wording or visuals.
• 19:33
  SPEAKER_00: That slide that I just went over and information is summarized again here, but just in a different way.
• 19:41
  SPEAKER_00: Different words, but say the same thing.
• 19:45
  SPEAKER_00: Okay, so if that's helpful to you, great.
• 19:48
  SPEAKER_00: If it's not, I understand, right?
• 19:51
  SPEAKER_00: But remember, we're in that pre-specified steps of the protocol, the effect measure, the data type, that's it.
• 19:56
  SPEAKER_00: Odds ratio ratios, is it a relative risk?
• 19:59
  SPEAKER_00: Right?
• 20:00
  SPEAKER_00: Model, is it is it one true effect?
• 20:03
  SPEAKER_00: That's a fixed model, or do we have concerns that there are random truths?
• 20:09
  SPEAKER_00: They're all true, but there's a family of truths, and they're random.
• 20:13
  SPEAKER_00: Okay.
• 20:14
  SPEAKER_00: Then we pick the method of the analysis, and that's again usually chosen by the data type that drives that.
• 20:23
  SPEAKER_00: And I'm gonna walk you through that.
• 20:26
  SPEAKER_00: Going to walk you through, I'm just trying to simplify it here for you.
• 20:29
  SPEAKER_00: And then the analysis is the result, which is that pooled estimate, the diamond.
• 20:34
  SPEAKER_00: And the width of the diamond represents our 95% confidence interval.
• 20:38
  SPEAKER_00: We're gonna check for the heterogeneity, and we're gonna look for a sensitivity.
• 20:42
  SPEAKER_00: Should we have included these studies, combine them appropriately?
• 20:45
  SPEAKER_00: And then you get this example label, odds ratio, comma, uh, mantle hazel, uh, mh, comma, random model.
• 20:57
  SPEAKER_00: They're telling you the method in the model and the type of measurement of the effect right there.
• 21:04
  SPEAKER_00: And we're gonna see that line right here, okay.
• 21:10
  SPEAKER_00: We're gonna see that kind of information on a forest plot.
• 21:14
  SPEAKER_00: That's why I'm covering it.
• 21:16
  SPEAKER_00: Okay, it's things that you as a researcher, if you're publishing one of these, need to give us, and the journal editors should require of you.
• 21:27
  SPEAKER_00: But for those of us reading the paper, we need to understand what you were doing and why you did it.
• 21:33
  SPEAKER_00: And does it make sense?
• 21:34
  SPEAKER_00: No different than the looking at the statistical analysis.
• 21:37
  SPEAKER_00: Did you use the right statistical test to analyze this data?
• 21:44
  SPEAKER_00: So the big, big picture, like high level view, a meta-analysis is a weighted average of this data from these studies.
• 21:56
  SPEAKER_00: Just think of it like that.
• 21:58
  SPEAKER_00: What happens if I combine?
• 22:00
  SPEAKER_00: All this data, what do I get?
• 22:02
  SPEAKER_00: But remember, you're including studies that may look and be of different sizes, different degrees of influence, and so there is a weighted average.
• 22:14
  SPEAKER_00: Okay, in order to calculate this, you got to make two big decisions.
• 22:19
  SPEAKER_00: Again, this goes up in that pre-a priority, pre-specified protocol, the a priori defined factors.
• 22:26
  SPEAKER_00: Okay.
• 22:27
  SPEAKER_00: First is the model is what you assume about the truth.
• 22:35
  SPEAKER_00: Okay.
• 22:36
  SPEAKER_00: Do all of those studies share one truth or true effect, or does each study have its own truth and they are random?
• 22:50
  SPEAKER_00: So one is fixed or random.
• 22:54
  SPEAKER_00: Okay, just remember that.
• 22:57
  SPEAKER_00: Then we have to think about the method.
• 23:00
  SPEAKER_00: That's the math part for weighing out and combining these studies.
• 23:07
  SPEAKER_00: And the right one that you pick, and there's three most common ones, is based on the type of data that I'm gonna walk you through.
• 23:15
  SPEAKER_00: So that's the big picture, guys.
• 23:17
  SPEAKER_00: And again, this is done a priori.
• 23:20
  SPEAKER_00: You should already have defined these things.
• 23:22
  SPEAKER_00: Think of it like a GPA in college.
• 23:27
  SPEAKER_00: Every course counts.
• 23:32
  SPEAKER_00: True.
• 23:33
  SPEAKER_00: But remember, courses with more credit hours, well, guess what?
• 23:38
  SPEAKER_00: They count more.
• 23:40
  SPEAKER_00: There's a weight associated on the overall GPA.
• 23:46
  SPEAKER_00: And a meta-analysis, more precise studies count more.
• 23:52
  SPEAKER_00: The model decides what we're averaging toward, and the method decides how the weights get calculated in that pooled estimate, guys.
• 24:06
  SPEAKER_00: And that is the big picture.
• 24:11
  SPEAKER_00: Now let's step into the individual components.
• 24:14
  SPEAKER_00: Okay, so stay with me.
• 24:16
  SPEAKER_00: If you need to rewind, check it, listen to that some of that stuff again.
• 24:20
  SPEAKER_00: I get it.
• 24:21
  SPEAKER_00: When I first learned this, I was like, what is he, you know, what are we talking about here?
• 24:27
  SPEAKER_00: Just stay with me.
• 24:28
  SPEAKER_00: Okay, it builds.
• 24:30
  SPEAKER_00: But again, we're just trying to do the analysis on that pooled estimate correctly.
• 24:35
  SPEAKER_00: So we have to get oriented on what we're dealing with.
• 24:39
  SPEAKER_00: So effect measure.
• 24:41
  SPEAKER_00: It's based on the type of data.
• 24:44
  SPEAKER_00: So first, what happens is each study result is converted into the same effect measure, right?
• 24:53
  SPEAKER_00: So for yes and no outcomes like um nominal data, you either are or you're not, right?
• 25:02
  SPEAKER_00: Those are typically ratios.
• 25:04
  SPEAKER_00: You either achieve the goal or you didn't, you're dead or alive.
• 25:08
  SPEAKER_00: So you have risk ratio, you have odds ratio, you have risk difference.
• 25:14
  SPEAKER_00: Whereas continuous variables, right?
• 25:18
  SPEAKER_00: You get a mean with a difference.
• 25:21
  SPEAKER_00: You also can sometimes generate a standard mean difference, and then obviously there's time to event outcomes, those hazards ratios.
• 25:30
  SPEAKER_00: We're familiar with those terms.
• 25:32
  SPEAKER_00: Well, those are just the types of data endpoints from the studies, aren't isn't it?
• 25:39
  SPEAKER_00: But we got to be on the same page because you can only average things that are measured the same way.
• 25:46
  SPEAKER_00: Okay, that's why that first part is you have to define what are your what is the type of data you're dealing with.
• 25:53
  SPEAKER_00: The next is you choose, and most commonly this is what people do, is choose the model.
• 25:59
  SPEAKER_00: Okay, again, it's the assumption about the effect.
• 26:03
  SPEAKER_00: Is it true?
• 26:04
  SPEAKER_00: So we have fixed uh effect model and we have random effect model.
• 26:10
  SPEAKER_00: Now just stop and think about the name of that fixed effect, because it's a fixed effect, it has one true or common true effect, okay?
• 26:26
  SPEAKER_00: And the differences between studies, well, it's just due to chance random effects model says what it says, right?
• 26:38
  SPEAKER_00: It assumes true effects differ across studies, right?
• 26:44
  SPEAKER_00: It's a kind of a family of true effects that estimate their average, and this is where the tau squared comes in.
• 26:56
  SPEAKER_00: You'll see this reported in the papers, but it's trying to understand what is the degree of variability.
• 27:05
  SPEAKER_00: You know, if there's different truths, they're all truths, they're just a family of them, then how do they vary?
• 27:12
  SPEAKER_00: And that's what tau squared result is.
• 27:14
  SPEAKER_00: And again, we'll see those later.
• 27:18
  SPEAKER_00: Now, as a result, the confidence interval is wider when there is heterogeneity that exists, right?
• 27:27
  SPEAKER_00: So if there's different randoms of truths, then there's going to be some degree of heterogeneity there.
• 27:34
  SPEAKER_00: And smaller studies get relatively sometimes more weight.
• 27:38
  SPEAKER_00: And if those smaller studies unfortunately are biased, the pooled estimate can get pulled then towards them.
• 27:48
  SPEAKER_00: Okay.
• 27:49
  SPEAKER_00: And so that's why it's important to know which model they used and what were the assumptions that they made about the studies' findings or the effects.
• 28:02
  SPEAKER_00: So when there is very little heterogeneity, that means very little difference between them, okay, fixed effect and random effects models give similar pulled estimates.
• 28:17
  SPEAKER_00: Okay, listen.
• 28:19
  SPEAKER_00: If the studies, I mean, if you just stop and think about that, random, if they're not as random and they're very close, well, that's looking like a fixed true one result, isn't it?
• 28:34
  SPEAKER_00: Or effect.
• 28:36
  SPEAKER_00: And so guess what?
• 28:37
  SPEAKER_00: Their pulled estimates start to look the same.
• 28:40
  SPEAKER_00: When heterogeneity is present, and the the random effects estimate is typically more conservative, and what ends up happening is the the confidence interval is a little bit wider.
• 28:51
  SPEAKER_00: But here's the thing, guys.
• 28:54
  SPEAKER_00: We are not supposed to be choosing the model, fixed or random effects, solely because the i squared, which we'll talk about here in a minute, so a test of heterogeneity, cross a certain threshold.
• 29:14
  SPEAKER_00: Okay, so while there may be some differences, and if it's real tight, they may show similar pooled effects.
• 29:21
  SPEAKER_00: But if there's heterogeneity present, then random effects can typically be a more conservative output.
• 29:31
  SPEAKER_00: Okay, and now Cochrane and Revman settings, uh, especially when you're using the random effects model, has additional fine-tuning settings.
• 29:41
  SPEAKER_00: And that's all I wanted you to know.
• 29:44
  SPEAKER_00: If you see or read about this, and you actually do these and you're using a random effects model in your statistical assessment, RevMans has some defaults that they use, and that's what's provided here.
• 30:01
  SPEAKER_00: I'm not going to spend much time talking about it because it sometimes causes more confusion, except for those who were actually doing the analysis.
• 30:12
  SPEAKER_00: Okay.
• 30:16
  SPEAKER_00: So, in summary, when we're determining the model for effect, the protocol should specify the model choice based on whether you expect the studies to differ clinically.
• 30:29
  SPEAKER_00: The model should not be chosen after seeing the i squared or the results.
• 30:37
  SPEAKER_00: Right?
• 30:37
  SPEAKER_00: And tau squared if it's zero.
• 30:40
  SPEAKER_00: Okay.
• 30:41
  SPEAKER_00: If tau squared is zero, then both models give the same results because the assumption there is that there's one really true effect.
• 30:52
  SPEAKER_00: It's not really random, is it?
• 30:56
  SPEAKER_00: So I I hope the words and the terminology there are helpful, but we're talking about the model of the effect that we are making assumptions about.
• 31:08
  SPEAKER_00: Now we need to step into the method.
• 31:12
  SPEAKER_00: How the studies are then weighted when you're in when you're combining them.
• 31:18
  SPEAKER_00: Okay.
• 31:21
  SPEAKER_00: Now, this is about the analysis part.
• 31:24
  SPEAKER_00: So when you're talking about the choice of the method for determining that average effect, the goal, guys, remember of a meta-analysis is to combine the studies, those individual study results, to create that effect estimate into a weighted average.
• 31:42
  SPEAKER_00: And that's what gives us that pooled estimate.
• 31:46
  SPEAKER_00: Okay, so again, if this is new terminology to you, you may have to stop.
• 31:51
  SPEAKER_00: Almost need like a dictionary here.
• 31:53
  SPEAKER_00: But the pooled estimate is that that diamond at the bottom in the forest plot.
• 31:57
  SPEAKER_00: But remember, each study is going to contribute a weight, and I'm going to show you that in a real example here in a minute.
• 32:05
  SPEAKER_00: Okay.
• 32:07
  SPEAKER_00: Now, there are three main methods that help us to produce the weighted average, but they differ in which data they handle best.
• 32:19
  SPEAKER_00: Okay, so it just comes down to what kind of data are you dealing with, and you just pick the right test.
• 32:25
  SPEAKER_00: Okay.
• 32:26
  SPEAKER_00: And those methods, not models of the effect, the methods of determining that weighted average is the mantle handle method, PIDO, and then you have the inverse variance method.
• 32:41
  SPEAKER_00: Okay, those are the three.
• 32:43
  SPEAKER_00: And I've summarized those in this table.
• 32:46
  SPEAKER_00: And again, for those of you on our podcast channels, I know you can't see this, but at YouTube it's available.
• 32:54
  SPEAKER_00: I just put each row as a method.
• 32:58
  SPEAKER_00: Okay, and on the right side, there's two columns.
• 33:01
  SPEAKER_00: The right is just really the description.
• 33:04
  SPEAKER_00: So when you think about MH, and you'll see it abbreviated in the results on a force plot m.
• 33:12
  SPEAKER_00: Okay, that's the one, the first one that we're talking about.
• 33:15
  SPEAKER_00: That is default for dichotomous data outcomes.
• 33:21
  SPEAKER_00: Again, yes or no.
• 33:23
  SPEAKER_00: You did or you didn't.
• 33:24
  SPEAKER_00: You achieved it, the goal or you didn't.
• 33:27
  SPEAKER_00: You died or you didn't.
• 33:28
  SPEAKER_00: Right?
• 33:28
  SPEAKER_00: So you get risk ratio, you have a numerator and a denominator.
• 33:32
  SPEAKER_00: Odds ratios.
• 33:34
  SPEAKER_00: And it performs better than inverse variants when there's events are few or considered rare.
• 33:40
  SPEAKER_00: Okay, and it can be used in both the fixed effect model or random effect model.
• 33:48
  SPEAKER_00: Pito pools mainly odds ratios.
• 33:53
  SPEAKER_00: Okay, and it's best for very rare events when effects are not large and the group size are balanced.
• 34:02
  SPEAKER_00: Now the bias when effect, uh the bias when effects are large and the groups are unbalanced.
• 34:09
  SPEAKER_00: Okay, again, these are just little nuances.
• 34:12
  SPEAKER_00: And then the last one is inverse.
• 34:15
  SPEAKER_00: Uh, the inverse variance method, where the weights of each study by its inverse of its variance.
• 34:22
  SPEAKER_00: The standard approach for like a continuous outcomes is widely used for generic pooling.
• 34:30
  SPEAKER_00: Okay.
• 34:31
  SPEAKER_00: Now, small studies can have large variants and therefore less weight.
• 34:39
  SPEAKER_00: Okay.
• 34:40
  SPEAKER_00: Now, it's important that you don't listen, guys.
• 34:45
  SPEAKER_00: I know it's a lot to kind of comprehend here, but you don't pick a method because you like it.
• 34:53
  SPEAKER_00: The diet the type of data is primarily picking it for you.
• 34:58
  SPEAKER_00: Right?
• 34:59
  SPEAKER_00: So continuous data points to inverse variance, yes and no data points, right?
• 35:07
  SPEAKER_00: They they're pointing to the mantle heinsel, right?
• 35:10
  SPEAKER_00: In very rare events, balanced the arms uh with balanced arms and data are pointing to PETO.
• 35:18
  SPEAKER_00: So those are the little key words or nuances, especially from a board exam perspective, um, or writing a protocol that are guiding you.
• 35:28
  SPEAKER_00: Okay.
• 35:29
  SPEAKER_00: Now the next um step is the analysis.
• 35:34
  SPEAKER_00: So you crunch the data, you pick the method, you pick the model, you determine the effect measure, and you've done it.
• 35:40
  SPEAKER_00: You have calculated the summary statistic, and you get that pooled estimate with the 95% confidence interval, and you have shown a diamond.
• 35:48
  SPEAKER_00: So what is that?
• 35:50
  SPEAKER_00: Well, that's shown here on this example.
• 35:52
  SPEAKER_00: This is an example of a forest plot on specific data for uh uh acute ischemic stroke.
• 36:02
  SPEAKER_00: Okay.
• 36:03
  SPEAKER_00: Um, and then and you can see that we have five trials over here on the far left.
• 36:09
  SPEAKER_00: There's five.
• 36:10
  SPEAKER_00: Okay, and that that becomes really important when we talk about degrees of freedom.
• 36:14
  SPEAKER_00: I'll get to in a minute.
• 36:16
  SPEAKER_00: Um, and then you see the uh sort of the data, the endovascular therapy patients versus standard therapy patients, those who experienced the event versus the total number of people.
• 36:28
  SPEAKER_00: And then if you go further, you see the total number of or the weighted average.
• 36:33
  SPEAKER_00: That's key.
• 36:34
  SPEAKER_00: The weight you're you're seeing what percent of influence does that study, individual study, have on the the overall pooled estimate.
• 36:49
  SPEAKER_00: And because at the bottom you see the pooled estimate, it's down here, should be 100%.
• 36:54
  SPEAKER_00: So each study contributes, but it's weighted.
• 36:58
  SPEAKER_00: Again, going back to the GPA example.
• 37:02
  SPEAKER_00: But then look here.
• 37:04
  SPEAKER_00: You see something else, you see the type of data you're dealing with.
• 37:08
  SPEAKER_00: What was the effect measure?
• 37:11
  SPEAKER_00: They said odds ratio, so it's dichotomous.
• 37:16
  SPEAKER_00: Yes or no?
• 37:17
  SPEAKER_00: You had the event or you didn't.
• 37:19
  SPEAKER_00: That makes sense.
• 37:21
  SPEAKER_00: But you see, there's MH.
• 37:24
  SPEAKER_00: They're telling you that method that they used and what assumptions they had.
• 37:30
  SPEAKER_00: They said there's a family of truths, they're random.
• 37:34
  SPEAKER_00: That was their assumption.
• 37:36
  SPEAKER_00: So that was the model that they applied.
• 37:40
  SPEAKER_00: Okay, and then you get the individual odds ratios with their 95% confidence intervals, and then over on the far right is the forest plot, the visual, the thing that everybody always goes to.
• 37:54
  SPEAKER_00: But again, you see that same information right there.
• 37:57
  SPEAKER_00: It's orienting you, everything I just explained.
• 38:02
  SPEAKER_00: They're telling you in a very concise, codified way, and we'll break this down further, how they set it up, and then what each individual study looks like.
• 38:15
  SPEAKER_00: And you can see the weight is influenced by the size of that square.
• 38:20
  SPEAKER_00: It's called a blob.
• 38:23
  SPEAKER_00: And you can see their confidence intervals, the lines around them, and their contribution to the pooled estimate right here.
• 38:30
  SPEAKER_00: And then it even codifies for you what is favoring the intervention versus standard of care.
• 38:36
  SPEAKER_00: And you got to get oriented to all this stuff before you just jump to a conclusion.
• 38:42
  SPEAKER_00: But I want you to take note of another thing on this piece of data.
• 38:47
  SPEAKER_00: It's all right here.
• 38:50
  SPEAKER_00: Should this data have been combined?
• 38:55
  SPEAKER_00: Okay, and that's where we start doing tests of heterogeneity.
• 39:00
  SPEAKER_00: There's your tau.
• 39:02
  SPEAKER_00: Remember when tau is zero?
• 39:04
  SPEAKER_00: What did I say earlier?
• 39:07
  SPEAKER_00: Well, the model of random effects or fixed effects model, the result should roughly be the same, especially when the i squared is zero.
• 39:21
  SPEAKER_00: Again, these are the tests of heterogeneity.
• 39:23
  SPEAKER_00: So that leads me to that last portion, okay, which is determining the validity of the overall effect.
• 39:34
  SPEAKER_00: Should you have combined this data, and what do these tests tell us as we interpret that pooled estimate?
• 39:45
  SPEAKER_00: So I broke this down into another table.
• 39:49
  SPEAKER_00: There's uh three rows, two columns.
• 39:52
  SPEAKER_00: On the left, right, we have the statistics, chi-squared test of homogeneity.
• 39:58
  SPEAKER_00: We have our I squared test, which again we're both on that image I just showed you.
• 40:03
  SPEAKER_00: And then there's the tau squared.
• 40:05
  SPEAKER_00: So all three of those were listed on that image that I just showed you.
• 40:13
  SPEAKER_00: So the question at the top, the title says, is it reasonable to combine?
• 40:19
  SPEAKER_00: Should we have combined the data?
• 40:22
  SPEAKER_00: And so a chi-squared test is also known as Cochrane's Q test.
• 40:27
  SPEAKER_00: You might see that or Q sometimes by itself.
• 40:31
  SPEAKER_00: And that tests where the results differ more than expected by chance alone.
• 40:36
  SPEAKER_00: Okay, so low power with few studies.
• 40:42
  SPEAKER_00: That marker is where a p-value is less than 0.10 or less is often used.
• 40:50
  SPEAKER_00: What about I squared?
• 40:52
  SPEAKER_00: Okay, well, this is about the percentage of variability in the effect estimates due to the heterogeneity rather than chance.
• 41:02
  SPEAKER_00: Okay, so imprecise uh when studies are few, right?
• 41:07
  SPEAKER_00: It's not very helpful.
• 41:09
  SPEAKER_00: We have five.
• 41:11
  SPEAKER_00: The Cochrane Handbook breaks the I squared down not into absolutes, okay, but at percentages that help us to go, hmm, the variability of these effect estimates, you know, is it important?
• 41:28
  SPEAKER_00: There's there's moderate heterogeneity, there's substantial, or there's considerable, right?
• 41:34
  SPEAKER_00: And that doesn't mean you totally disregard the information.
• 41:38
  SPEAKER_00: You just have to go, should it have been combined to begin with?
• 41:42
  SPEAKER_00: But this is part of that process.
• 41:44
  SPEAKER_00: And then the tau squared, again, if it's zero, there's no detectable difference really between the study variation.
• 41:52
  SPEAKER_00: Remember, we do that kind of in that random effects model to see where are the families of truth and how are they different, what's the variance?
• 42:02
  SPEAKER_00: And you can visually see that back on the forest plot.
• 42:05
  SPEAKER_00: They they look like they're lining up, almost like they're showing one true effect, like a fixed effects model.
• 42:15
  SPEAKER_00: Okay.
• 42:18
  SPEAKER_00: So the core concept about heterogeneity is that a high I squared is reason to look harder.
• 42:24
  unknown: Okay.
• 42:25
  SPEAKER_00: Subgroups, clinical differences, what is the risk of bias, but it does not result in an automatic reason to discard the entire review.
• 42:35
  SPEAKER_00: And again, that's why we've kind of moved away from a number.
• 42:39
  SPEAKER_00: Oh, I'm not listening to that.
• 42:41
  SPEAKER_00: This is no longer good.
• 42:42
  SPEAKER_00: You just black and white.
• 42:45
  SPEAKER_00: There's nothing as perfect.
• 42:47
  SPEAKER_00: So I hope that that made sense, guys, at some level.
• 42:51
  SPEAKER_00: And again, sometimes you have to digest this by reading them, looking at them, sometimes listening to this multiple times, but using a systematic approach for each step improves the overall quality of these systematic reviews and meta-analysis.
• 43:06
  SPEAKER_00: And knowing the correct steps and the processes that are defined by a researcher who's supposed to be doing this and how way it's supposed to be reported allows us as readers and clinicians to critically appraise and make an overall interpretation because at the end of the day, what am I going to do with this information?
• 43:26
  SPEAKER_00: Okay.
• 43:27
  SPEAKER_00: And going back to that simplified sort of way of looking at it, where people struggle the most on the analysis part.
• 43:38
  SPEAKER_00: Remember, there's that pre-specified.
• 43:41
  SPEAKER_00: What is the measure of effect?
• 43:43
  SPEAKER_00: What is going to be the model?
• 43:44
  SPEAKER_00: What is the method for calculating this statistic or pulled estimate?
• 43:49
  SPEAKER_00: And then there's the actual analysis.
• 43:52
  SPEAKER_00: And then there's checking, should you combine it?
• 43:55
  SPEAKER_00: And then there's the codification on the results that we looked at.
• 43:59
  SPEAKER_00: And remember that meta-analysis is really just trying to create the weighted average on that pulled estimate.
• 44:08
  SPEAKER_00: Okay, and you got to make two major decisions about the truth of the data you're dealing with.
• 44:12
  SPEAKER_00: What model?
• 44:14
  SPEAKER_00: Is it one true effect, or is it a random or family of truths?
• 44:19
  SPEAKER_00: And then what's the variance between them?
• 44:22
  SPEAKER_00: Okay, and then the method is the math part of how do you distribute that weight from one study to the other.
• 44:32
  SPEAKER_00: So that's a lot, guys.
• 44:34
  SPEAKER_00: I know.
• 44:35
  SPEAKER_00: So I hope it was helpful to you in not only understanding the steps involved in conducting a systematic review meta-analysis, but also explaining why those steps matter when you're critically appraising the medical literature.
• 44:47
  SPEAKER_00: Okay, you know, remember the quality of the evidence depends on the quality of the process and knowing how to recognize a well-conducted systematic review meta-analysis will make you more confident consumer of the research and better at applying evidence-based medicine to your everyday practice.
• 45:03
  SPEAKER_00: So, thank you so much for joining me on This Is Why.
• 45:06
  SPEAKER_00: Okay, again, this episode is part of a series.
• 45:09
  SPEAKER_00: Okay, and we're going to continue to build upon some of these concepts and focus uh on some deeper uh level of information that helps us to further understand and how to apply this um information to practice.
• 45:22
  SPEAKER_00: And we'll see you next time.