Latest / This Is Why with Dr. Busti / Systematic Review vs Meta-Analysis: What's the Difference?

Transcript

Systematic Review vs Meta-Analysis: What's the Difference? · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Hi guys, welcome to this first episode in the module Systematic Reviews and Meta-Analysis, which is part of a larger evidence-based medicine, biostatistics, and clinical literature evaluation series.
• 0:11
  SPEAKER_00: Now, the goal of this first episode is to answer a few practical but very relevant questions, such as what is a systematic review?
• 0:21
  SPEAKER_00: What is a meta-analysis?
• 0:22
  SPEAKER_00: What about a Cochrane review?
• 0:24
  SPEAKER_00: How are they similar?
• 0:25
  SPEAKER_00: How are they different?
• 0:27
  SPEAKER_00: How is a systematic review different from a review article?
• 0:31
  SPEAKER_00: And can a simple meta-analysis add something valuable to the literature that a single randomized controlled trial often cannot?
• 0:42
  SPEAKER_00: My goal is not only to make this simple but also clinically relevant.
• 0:45
  SPEAKER_00: Because let's face it, guys, none of us have time to waste.
• 0:50
  SPEAKER_00: And on that note, I'm Dr.
• 0:51
  SPEAKER_00: Bustaye, your faculty and host on This Is Why, a show that helps healthcare professionals transform their medical knowledge from what to why.
• 1:00
  SPEAKER_00: Because when we know why we do what we do in the proper context of that evidence, it changes how, and that changes everything about patient-oriented decisions that we are going to make every day.
• 1:13
  SPEAKER_00: Now, this is an outline or sort of a table of contents for the series.
• 1:19
  SPEAKER_00: I recommend, depending on where you are interacting with this, hopefully at this one, this is the first one because we're going to talk a little bit about the foundation.
• 1:28
  SPEAKER_00: What is a systematic review versus amount analysis?
• 1:32
  SPEAKER_00: What are the differences?
• 1:33
  SPEAKER_00: Okay.
• 1:34
  SPEAKER_00: That establishes the foundation necessary so that we can move to the second one, all right, which are what are the steps involved in doing a meta-analysis, a systematic review and meta-analysis, and how do you start to analyze the data to pool together that pooled estimate from the data from all these clinical trials.
• 1:55
  SPEAKER_00: And then we move on to how do you report the results?
• 1:58
  SPEAKER_00: What is it, how do you interpret a forest plot?
• 2:01
  SPEAKER_00: And then using things and tools like grade assessment to look for things like bias and assessment of the overall paper's internal validity of the results and the conclusions that can be made so that we can try to extrapolate that to the real world clinical practice.
• 2:17
  SPEAKER_00: And then lastly, we'll look at critical appraisal tools, things about how should the paper have been written.
• 2:23
  SPEAKER_00: So if you're the researcher doing a systematic review meta-analysis, you need to pay attention to this so that you not only build a protocol that fits that, but so that you can report it in the proper way.
• 2:35
  SPEAKER_00: For us as reading, as clinicians reading your papers of the researcher, we want to make sure that you did your job correctly so that what I'm reading at the end of all this is something that I can use and apply, okay, or may lead to the next step.
• 2:54
  SPEAKER_00: So having said that, I want to break down the systematic review and meta-analysis in kind of this table format.
• 3:01
  SPEAKER_00: And for those of you listening to us on our podcast, there is a table, it's basically two columns.
• 3:07
  SPEAKER_00: On the left, a systematic review, on the right is a meta-analysis.
• 3:11
  SPEAKER_00: And what I would like to do is go over first some of the high points about a systematic review.
• 3:17
  SPEAKER_00: Okay?
• 3:18
  SPEAKER_00: Now, this is an overview of the literature, and it's specifically done in a manner that is, listen, systematic and should be, if well written and published correctly, reproducible.
• 3:32
  SPEAKER_00: Those are important key words that really begin to separate that conversation about what is a review article.
• 3:41
  SPEAKER_00: All right.
• 3:42
  SPEAKER_00: Next is the process by which the set of studies that we're using to do the analysis is usually evidence-based, but it's considered qualitative.
• 3:53
  SPEAKER_00: We're doing a review of the information.
• 3:56
  SPEAKER_00: Okay, we're not necessarily doing statistical or mathematical crunching.
• 4:00
  SPEAKER_00: That's what the meta-analysis is for in just a minute.
• 4:03
  SPEAKER_00: So it gives us something called a narrative synthesis.
• 4:06
  SPEAKER_00: So we use these words, at least over time, when as I've studied this subject, I think about them in big sort of buckets or categories.
• 4:15
  SPEAKER_00: So systematic review is a qualitative assessment of really evidence, studies.
• 4:22
  SPEAKER_00: And it provides a narrative synthesis that is done in a systematic and reproducible way.
• 4:29
  SPEAKER_00: Now everything has limitations, right?
• 4:32
  SPEAKER_00: So I can choose which studies to include, not include.
• 4:36
  SPEAKER_00: So there's a potential risk of bias by me.
• 4:40
  SPEAKER_00: All right, it also does not provide that single pooled estimate to help me to go.
• 4:46
  SPEAKER_00: What is all of this data, when it's all comes together, what is the magnitude of the effect of that data combined?
• 4:55
  SPEAKER_00: And that's what allow, that's what leads us to the meta-analysis, which in most cases, if you're doing a meta-analysis, you should have, in most cases, have done a systematic review.
• 5:07
  SPEAKER_00: Okay.
• 5:08
  SPEAKER_00: Sometimes a systematic review can just stop right there at the narrative synthesis and not go on.
• 5:15
  SPEAKER_00: But goodness, it certainly would be helpful to continue on if at all possible.
• 5:20
  SPEAKER_00: So this is where we take the data and we start to pool it.
• 5:24
  SPEAKER_00: We do statistical pooling and analysis that results in a quantitative.
• 5:30
  SPEAKER_00: Okay, remember, systematic review is qualitative.
• 5:33
  SPEAKER_00: This is a quantitative overall assessment.
• 5:37
  SPEAKER_00: We're combining data from multiple studies and we're trying to create that overall magnitude of an effect.
• 5:44
  SPEAKER_00: And that's what gives us this pooled uh estimate, is what it's called.
• 5:49
  SPEAKER_00: Okay, it's it's a it's a diamond at the bottom.
• 5:52
  SPEAKER_00: You'll see it when we get to the forest plot in future lectures where I break down how to read and interpret the individual components of a meta-analysis, and including the obviously specifically the forest plot.
• 6:05
  SPEAKER_00: There's a lot of data information there.
• 6:07
  SPEAKER_00: And if you don't have the foundation going into reading a forest plot, it doesn't make as much sense.
• 6:13
  SPEAKER_00: And that's why we're doing this particular topic first.
• 6:17
  SPEAKER_00: All right.
• 6:18
  SPEAKER_00: Now, doing this can improve our precision and statistical power because we're including multiple sets of data than one study can do by itself.
• 6:30
  SPEAKER_00: And then we get that estimation of the average of the magnitude of effect.
• 6:34
  SPEAKER_00: And each study is going to contribute something different.
• 6:37
  SPEAKER_00: We're going to see that in the study blob, or the magnitude or the weight that that particular study has on the overall point estimate.
• 6:50
  SPEAKER_00: All right.
• 6:50
  SPEAKER_00: Now I want to ask this question: how is a systematic review different from a standard review article?
• 6:58
  SPEAKER_00: And this is, I've already started laying the groundwork for this, but it's really important.
• 7:03
  SPEAKER_00: Okay, so remember a review article is especially those traditional ones that most people are used to seeing.
• 7:10
  SPEAKER_00: It's over an overview of a disease or a topic.
• 7:15
  SPEAKER_00: And the search methods that sometimes go into putting that article together are not pre-specified in a protocol.
• 7:24
  SPEAKER_00: There's not inclusion exclusion criteria codified by the author beforehand.
• 7:29
  SPEAKER_00: And it's not always reproducible.
• 7:32
  SPEAKER_00: My review article on that topic may look different than somebody else's.
• 7:36
  SPEAKER_00: Okay?
• 7:37
  SPEAKER_00: And that's okay.
• 7:38
  SPEAKER_00: They have they serve a purpose, but it's very different than a systematic review.
• 7:43
  SPEAKER_00: And I reason I'm stressing that point is because people many times will get them confused.
• 7:49
  SPEAKER_00: So there's no inclusion-exclusion criteria, and they obviously are not going to be pooling data to create a pooled estimate of the effect.
• 7:59
  SPEAKER_00: So a systematic review, by contrast, right, it starts with a very focused PICO or PICONTS question, has a very well-defined, codified, pre-specified protocol, results usually in a very exhaustive search.
• 8:19
  SPEAKER_00: And I'm talking exhausted, even into doctoral thesis, sometimes emailing investigators and asking for data if you want to do it the best in the right way.
• 8:30
  SPEAKER_00: There's explicit inclusion-exclusion criteria, and it's a very structured overall appraisal or protocol that allows for somebody else to hopefully be able to reproduce it.
• 8:43
  SPEAKER_00: Now I like to look at Venn diagrams.
• 8:45
  SPEAKER_00: We did that with evidence-based medicine.
• 8:47
  SPEAKER_00: When you start to think about the contribution, the overlap, the what are these things, and what are some of these other things that we read or we see as medical professionals, right?
• 8:59
  SPEAKER_00: From groups like AHRQ, the Cochrane Collaboration, NICE, groups like Oxford Center for Evidence-Based Medicine and stuff where I did my training.
• 9:10
  SPEAKER_00: These people are pooling together, and many of them are doing a lot of these systematic reviews and meta-analysis.
• 9:18
  SPEAKER_00: Okay.
• 9:18
  SPEAKER_00: But this Venn diagram hopefully summarizes what I just said.
• 9:23
  SPEAKER_00: So on the left side of the Venn diagram, there's two circles overlapping for those of you again joining us on our podcast.
• 9:31
  SPEAKER_00: We have systematic review on the left, and we have the meta-analysis on the right, and there's overlap.
• 9:36
  SPEAKER_00: Okay, so they can exist by themselves, stand alone, but again, in most cases.
• 9:42
  SPEAKER_00: So I the reason I stress that point also is that I've heard and seen, especially younger clinicians and researchers who may not fully have done these things or don't know the literature at the same level, uh, sometimes assume that a systematic review is a meta-analysis and it's not.
• 10:04
  SPEAKER_00: Okay.
• 10:05
  SPEAKER_00: It has to be combined.
• 10:07
  SPEAKER_00: And then there's this Cochrane review, which seems to hold this higher level of influence and sort of power inside of this conversation.
• 10:18
  SPEAKER_00: And that really comes down to the rigor uh that a Cochrane review undergoes.
• 10:25
  SPEAKER_00: So what is a Cochrane review?
• 10:27
  SPEAKER_00: Okay, we've talked about systematic review, we talked about meta-analysis, but what is a Cochrane review and why do we use that word like if it's really something different?
• 10:38
  SPEAKER_00: Well, it's the group, right?
• 10:40
  SPEAKER_00: Formerly the Cochrane Collaboration opened back in the early 90s, then formed this international collaboration back in the again, early 90s, and there's these global uh network of providers and experts that form review groups, and they they implement certain topics in certain areas, they have method groups, they also have geographical centers, okay.
• 11:08
  SPEAKER_00: And the primary purpose is to do systematic reviews of human health care and health policy, okay, widely regarded as money, again, the most rigorous forms of evidence and synthesis that's available.
• 11:24
  SPEAKER_00: I mean, you talk about holding to a standard, it's very time-consuming and difficult to do these in general.
• 11:32
  SPEAKER_00: You add that extra level of the uh requirements by the Cochrane handbook, and now you've just increased the workload significantly.
• 11:42
  SPEAKER_00: But why does that matter?
• 11:46
  SPEAKER_00: Because the results, whatever it is, they'll tell you if it's bad.
• 11:50
  SPEAKER_00: Not because they didn't do their job, that group, but because the data is just not great.
• 11:57
  SPEAKER_00: They're gonna tell you that, and guess what?
• 11:58
  SPEAKER_00: It's real.
• 11:59
  SPEAKER_00: And if they and if they if something good or whatever, you know that it has followed a standard.
• 12:06
  SPEAKER_00: And so they address a very clearly formulated question using a very set of stringent guidelines to investigate the effects of these interventions for things like prevention, treatment, uh, even rehabilitation, and the accuracy of diagnostic tests.
• 12:23
  SPEAKER_00: Okay.
• 12:24
  SPEAKER_00: Also, these obviously are peer-reviewed, just like other forms of literature.
• 12:28
  SPEAKER_00: They're dynamic.
• 12:29
  SPEAKER_00: That means that as new data emerges and enough of it that warrants a change, they'll go back and add that data back in and rerun the both quantitative and qualitative assessments and reproduce an updated result.
• 12:49
  SPEAKER_00: And that's fantastic.
• 12:50
  SPEAKER_00: That's what's great about some of these groups.
• 12:52
  SPEAKER_00: And then, of course, they obviously are going to summarize it for the general public so we know what to do with it.
• 12:58
  SPEAKER_00: All right, other contributors outside of Cochrane.
• 13:02
  SPEAKER_00: Again, and many of these people that are involved in these are also sometimes involved in those collaborative working groups.
• 13:10
  SPEAKER_00: You got in the United States, the AHRQ, there's also these evidence-based practice centers like Johns Hopkins, where I train, they have one, right?
• 13:19
  SPEAKER_00: Uh, worldwide, there's numerous partnerships with all trials and the Campbell Collaboration, World Health Organization.
• 13:27
  SPEAKER_00: Okay, these are again, they help make up many of these groups.
• 13:32
  SPEAKER_00: Over in Canada, you have Health Evidence, McMaster University, and the National Collaborative Centers for Methods and Tools.
• 13:40
  SPEAKER_00: A lot of the foundation of evidence-based medicine also came from there, just like Oxford.
• 13:45
  SPEAKER_00: And then, of course, you have the University of Oxford Center for Evidence-based medicine, which has a been around for decades, uh, and a uh, again, another contributor to the overall movement of this paradigm shift that occurred where Dave Sackett and all these group people of the early fathers of evidence-based medicine really seemed to codify the process.
• 14:09
  SPEAKER_00: Um, and having trained there, it was a it was a great experience and one of the most challenging experiences, but it taught me a lot about the value of and importance of quality evidence, and that's what they're all about.
• 14:23
  SPEAKER_00: All right, let's dive in a little bit more about a meta-analysis because while it sounds great, right, they have good and bad, just like anything else.
• 14:33
  SPEAKER_00: It's it's data, it's literature, it's it's an inherently flawed process.
• 14:37
  SPEAKER_00: It's not perfect because we're not studying the whole population, are we?
• 14:42
  SPEAKER_00: All right, so what are some of the potential advantages?
• 14:44
  SPEAKER_00: Well, we're gonna increase that sample size, and in doing so, we increase the power.
• 14:50
  SPEAKER_00: And overall, as you increase power, you reduce the risk of a type 2 error.
• 14:56
  SPEAKER_00: Remember, type 2 error, if you don't remember from previous topics that I've covered also on our channel, okay.
• 15:02
  SPEAKER_00: That's where you where you're saying there's there's no difference, but in fact, there really is.
• 15:07
  SPEAKER_00: And one of the most common reasons that people or researchers make a type 2 error is not having a sufficient sample size or the power, usually, to find that difference when it in fact is there.
• 15:21
  SPEAKER_00: Okay, it can reveal clinically important differences in treatments that smaller studies sometimes are unable to detect.
• 15:31
  SPEAKER_00: Okay.
• 15:32
  SPEAKER_00: It allows us to investigate the degree of heterogeneity between the studies.
• 15:38
  SPEAKER_00: Because I mean, face it, you're pulling data from multiple studies.
• 15:44
  SPEAKER_00: Well, all those studies are not perfectly matched and look alike.
• 15:49
  SPEAKER_00: And if they look too different, then that heterogeneity of the data can influence the overall conclusions that you arrive at.
• 16:00
  SPEAKER_00: And that's important as part of our assessment.
• 16:02
  SPEAKER_00: We're gonna see that in both the second part of this series in the lecture.
• 16:06
  SPEAKER_00: I do on the steps that are involved, because that's the the method and the model where you calculate that pulled estimate.
• 16:13
  SPEAKER_00: But it's also where you start to look at the validity and the heterogeneity of that data and whether or not you should have combined them to begin with.
• 16:22
  SPEAKER_00: And so there are tests of heterogeneity that help us to make that assessment.
• 16:29
  SPEAKER_00: We can generate or refine a hypothesis so that we can go on and do a randomized controlled trial.
• 16:35
  SPEAKER_00: And I'm going to show you a real-world scenario that allowed that to happen because of a systematic review meta-analysis.
• 16:45
  SPEAKER_00: Okay?
• 16:45
  SPEAKER_00: Now, it can resolve also apparent conflict among small trials by increasing overall precision.
• 16:54
  SPEAKER_00: Now, the bad.
• 16:56
  SPEAKER_00: Well, I mean, heterogeneity, you're combining data from different backgrounds.
• 17:00
  SPEAKER_00: And if it's too different and heterogeneous, then it's obviously going to influence that point estimate overall conclusion that you're able to come up with.
• 17:11
  SPEAKER_00: You could also have a selection bias, and that's where your search strategy, your inclusion-exclusion criteria that are supposed to be pre-specified in your protocol, are supposed to be published and transparent.
• 17:26
  SPEAKER_00: So that we, reading it, can critique did you do it right?
• 17:31
  SPEAKER_00: Did you include the right types of data?
• 17:34
  SPEAKER_00: Why does all this matter?
• 17:35
  SPEAKER_00: Because we're looking for uh the accuracy and the validity of that information, but we're also looking for can I extrapolate it?
• 17:43
  SPEAKER_00: Because garbage in, guys, is garbage out.
• 17:48
  SPEAKER_00: I don't care.
• 17:49
  SPEAKER_00: You can you can pool data, you can do a beautiful, perfect Cochrane review, systematic review meta-analysis, and it's just garbage.
• 17:58
  SPEAKER_00: And Cochrane will tell you that.
• 18:00
  SPEAKER_00: And if they tell you that, it's for real, right?
• 18:04
  SPEAKER_00: But we need to know that because it also allows us to recognize where's the opportunity or the gap in the overall literature that we can now work to fill.
• 18:18
  SPEAKER_00: And so we use things like uh, again, looking for uh sensitivity analysis, heterogeneity.
• 18:24
  SPEAKER_00: We will also do funnel plots to look for the presence of publication bias.
• 18:30
  SPEAKER_00: And we know that happens, right?
• 18:31
  SPEAKER_00: That's why you need to reach out to maybe the pharmaceutical company if they're willing to give it to you.
• 18:35
  SPEAKER_00: Data that was submitted that maybe isn't public to the FDA, uh, researchers who have access to additional data that maybe they didn't publish for whatever reason.
• 18:45
  SPEAKER_00: We want the truth, not just what's published, which might have been selectively reserved for only positive findings.
• 18:54
  SPEAKER_00: And so, does a systematic review meta-analysis actually change the standards of practice?
• 19:02
  SPEAKER_00: So, this is the real-world example here, guys.
• 19:05
  SPEAKER_00: Follow me.
• 19:06
  SPEAKER_00: So in the early 1980s, okay, except and way back, IV streptokinase was used for acute myocardial infarction.
• 19:19
  SPEAKER_00: But at that time it was still undergoing debate.
• 19:23
  SPEAKER_00: There were some trials that had conflicting findings and results.
• 19:28
  SPEAKER_00: Well, in 1982, a a pooled analysis of randomized controlled trials suggested that hey, this drug in this clinical scenario of SC segment elevation seems to lower mortality.
• 19:44
  SPEAKER_00: Then you have additional studies that come out as a result of that.
• 19:49
  SPEAKER_00: Again, it precipitated like this maybe we should look at it.
• 19:53
  SPEAKER_00: And you have the GISI and the ISIS II trials published in the later in the 80s that confirmed the benefit.
• 20:02
  SPEAKER_00: It changed practice, but you needed a randomized controlled trial done appropriately to answer that.
• 20:08
  SPEAKER_00: And we know the ISIS 2 is huge.
• 20:11
  SPEAKER_00: It wasn't just for streptokinase in the context of MI, but also the use of aspirin.
• 20:17
  SPEAKER_00: And so the overall lesson here is that with careful pooling of data, we can expose a signal before a single trial.
• 20:29
  SPEAKER_00: And it can be large enough to settle that question, right?
• 20:33
  SPEAKER_00: And so that's why we consider doing these, and that is the value of them if they are done right.
• 20:40
  SPEAKER_00: So wrapping this up, systematic reviews are a qualitative assessment.
• 20:45
  SPEAKER_00: They provide a narrative summary of the literature, but they are done in a systematic, reproducible way, and that is really important.
• 20:58
  SPEAKER_00: It's also confined to the evidence.
• 21:00
  SPEAKER_00: It's not just what are your thoughts, editorials, you know, expert opinions, you're combining all that stuff.
• 21:08
  SPEAKER_00: That's a traditional and historical review article.
• 21:12
  SPEAKER_00: This is about combining and summarizing the synthesis of the evidence.
• 21:18
  SPEAKER_00: Whereas a meta-analysis, remember, guys, is that quantitative assessment of the literature, specifically that evidence, and it's the statistical mathematical step that pools that data and provides that overall estimate of the effect.
• 21:37
  SPEAKER_00: And that is super important.
• 21:39
  SPEAKER_00: And what I do in the next sets of lectures is to break those components down a deeper so that you understand some of the more necessary details of not only getting the paper published and doing the study and meta-analysis, systematic review, but also how do you interpret as clinicians.
• 22:01
  SPEAKER_00: And so you can access the full uh systematic review and meta-analysis playlist at thisiswhy.
• 22:12
  SPEAKER_00: They're also found on our podcast channels, Spotify, Apple Podcasts, Amazon, okay.
• 22:18
  SPEAKER_00: And these include other relevant playlists if that's helpful for you to you guys.
• 22:22
  SPEAKER_00: I'm Dr.
• 22:23
  SPEAKER_00: Busteye, and thank you so much for joining this episode.
• 22:26
  SPEAKER_00: And if you're enjoying the content, please you know drop a comment, share your experience, add an add a piece of knowledge that we can all gain from because we all benefit from it.
• 22:35
  SPEAKER_00: And remember, if you're preparing for you know board exams, these concepts have to not only be practiced and applied clinically, but you're gonna see them on your boards.
• 22:47
  SPEAKER_00: And this is why members get a discount code at review course companies like High Yield Meta Reviews, where you can get a discount for access to practice questions if that's helpful to you.
• 22:59
  SPEAKER_00: So thanks again for joining us.
• 23:01
  SPEAKER_00: Thanks for learning why.
• 23:02
  SPEAKER_00: Never get up on doing that.
• 23:04
  SPEAKER_00: It's important as it changes the way we practice and implement the evidence in the real world.
• 23:10
  SPEAKER_00: Take care, everybody.