Latest / This Is Why with Dr. Busti / Understanding GRADE in Meta-Analysis: How to Judge Quality of Evidence

Transcript

Understanding GRADE in Meta-Analysis: How to Judge Quality of Evidence · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: Well, many clinicians assume that if a systematic review or meta-analysis finds statistically significant result, that the evidence must then be strong.
• 0:10
  SPEAKER_00: Well, that certainly can be the case, but don't assume that it is all of the time.
• 0:16
  SPEAKER_00: This is why the quality of evidence matters just as much as the results themselves.
• 0:22
  SPEAKER_00: Two studies can report the exact same finding, yet your confidence in those conclusions can be completely different depending on the quality of the evidence behind them.
• 0:35
  SPEAKER_00: And I'm going to show you a good example of that.
• 0:38
  SPEAKER_00: That's exactly why the Great Assessment was developed.
• 0:42
  SPEAKER_00: Great assessment provides a systematic, transparent way to evaluate how much confidence we should place in the evidence before changing clinical practice.
• 0:55
  SPEAKER_00: Because making the right decision isn't just about knowing the evidence and what it says, it's about knowing how much you should trust it.
• 1:05
  SPEAKER_00: And with that, I'm Dr.
• 1:06
  SPEAKER_00: Bustaye, your host on This Is Why, where my goal is to help healthcare professionals transform their medical knowledge from what to why, because when you know why you do what you do in the proper context of the evidence, it changes the how, and that changes everything about the patient-oriented decisions that we make every day.
• 1:24
  SPEAKER_00: So here we are in our series, and depending on when you're coming in, if this is the first uh lecture of this series, I strongly encourage and advise you, while it can stand alone by itself, that you really go back and go to the order where we cover what is a systematic review, meta-analysis, what is a cocking review, how is it different than a review article?
• 1:48
  SPEAKER_00: What are the key steps of not only conducting a systematic review, but also then moving on to the meta-analysis?
• 1:55
  SPEAKER_00: And then when you get those statistical assessments, the point estimate in a meta-analysis, and you generate a forest plot, what does it mean?
• 2:04
  SPEAKER_00: How do you get oriented to it?
• 2:06
  SPEAKER_00: What are its parts?
• 2:08
  SPEAKER_00: We are on the grade assessment in this particular lecture where we're talking about how to assess the risk for bias and the quality of the recommendations that come out of the overall meta-analysis.
• 2:23
  SPEAKER_00: And then the last topic is really about how do you report and publish a meta-analysis systematic review and meta-analysis and then how do you appraise it.
• 2:31
  SPEAKER_00: There are tools that give us direction and guidance that we should, as readers, understand so that we are knowing that it was reported correctly, but also how do we interpret that information.
• 2:43
  SPEAKER_00: And the first thing I want to talk about is GRADE.
• 2:46
  SPEAKER_00: Okay.
• 2:48
  SPEAKER_00: This has been a process of evolution that has had a significant impact, in my personal opinion, and even in my own career, I've seen changes.
• 2:58
  SPEAKER_00: And I'm going to give you a very real example from a Cochrane review where the implementation of GRADE drastically changed a recommendation.
• 3:10
  SPEAKER_00: Okay.
• 3:11
  SPEAKER_00: So it began in the uh began back in 2000.
• 3:15
  SPEAKER_00: And what are the goals?
• 3:16
  SPEAKER_00: First is to reduce confusion among variations in grading of the evidence and recommendations that have been evolving at that point.
• 3:27
  SPEAKER_00: Okay.
• 3:28
  SPEAKER_00: Also, it helps to rate the quality of the evidence.
• 3:33
  SPEAKER_00: Okay, because at the end of the day, we need to make a recommendation.
• 3:36
  SPEAKER_00: And so it helps us to do that.
• 3:38
  SPEAKER_00: The great thing about GRADE is it has become more universal.
• 3:43
  SPEAKER_00: It's now used by Cochrane, the World Health Organization, NICE in the UK, professional societies, and many of the guideline groups worldwide.
• 3:53
  SPEAKER_00: So it's just a tool that you need to know and will probably continue to evolve and be integrated into the things that you read.
• 4:03
  SPEAKER_00: Okay.
• 4:04
  SPEAKER_00: So Gray rates a certainty, a confidence in the estimate of effect in each important outcome as being either high, moderate, low, or very low.
• 4:18
  SPEAKER_00: Okay.
• 4:18
  SPEAKER_00: Now, as a starting point, randomized controlled trials obviously start at a high degree of certainty.
• 4:27
  SPEAKER_00: When we go to non-randomized trials, there are other tools.
• 4:33
  SPEAKER_00: Robin's eye is one of those.
• 4:35
  SPEAKER_00: That's the non-randomized studies of interventions.
• 4:39
  SPEAKER_00: And it's a tool that is used specifically for those types of studies.
• 4:43
  SPEAKER_00: And they could start out high, okay, but then they are rated down usually because of the risk of bias, and they end up typically being a little bit lower.
• 4:54
  SPEAKER_00: So what are things that are going to rate it down?
• 4:56
  SPEAKER_00: Okay, the quality of evidence.
• 4:57
  SPEAKER_00: Well, there's obviously a risk for bias, there's inconsistency, there's indirectness, there's also imprecision, probably the presence of publication bias, things that typically rate up the evidence.
• 5:10
  SPEAKER_00: And we we see this more in our non-randomized studies.
• 5:13
  SPEAKER_00: And those things include like a large magnitude of effect, all right, dose response gradients, those kind of things can help us.
• 5:23
  SPEAKER_00: But you're looking at the totality and you're trying to take all these things, ups and downs, and pull it all together across the whole body of evidence.
• 5:34
  SPEAKER_00: And so there's software that can help you.
• 5:37
  SPEAKER_00: There's the Grade Pro GDP, right?
• 5:41
  SPEAKER_00: The output clinicians should look for is the summary of the of the findings table.
• 5:48
  SPEAKER_00: Basically, you're not just looking at a p-value, you're looking at the summary because you really want to get, you know, get from the evidence to the recommendation.
• 5:57
  SPEAKER_00: And so grade also makes a recommendation as being strong or conditional.
• 6:02
  SPEAKER_00: Okay, and that that's helpful.
• 6:04
  SPEAKER_00: Because again, at some level, you're trying to pull all that together, just like a meta-analysis is statistically trying to pull the data together to come up with a point estimate.
• 6:14
  SPEAKER_00: We're also trying to do the same thing about the quality of the evidence that we're dealing with.
• 6:20
  SPEAKER_00: All right, now this is based on a the balance of benefits and harms, the certainty of that evidence, the patient values and preferences, and also resources.
• 6:32
  SPEAKER_00: Okay, and so this follows the evidence to decision framework.
• 6:36
  SPEAKER_00: And so I just wanted to show you an example of kind of how this might look like in an example meta-analysis or systematic review.
• 6:47
  SPEAKER_00: And this is looking at heprin versus placebo and non-ST segment elevation acute coronary syndrome.
• 6:55
  SPEAKER_00: And the reason I'm showing you this one is because it was one of the examples where this group from Cochrane had previously done this analysis and after several years repeated it utilizing the grade tool and their recommendations changed.
• 7:15
  SPEAKER_00: Nothing else really changed in the data, their recommendations changed.
• 7:19
  SPEAKER_00: So this was an example of how understanding the quality of the evidence made an impact on how we, especially if someone just reads the abstract, what we walk away thinking and feeling.
• 7:34
  SPEAKER_00: And you can see here there's this these visuals of the quality of the evidence being low.
• 7:42
  SPEAKER_00: Again, typically independent investigators and researchers utilizing these tools.
• 7:48
  SPEAKER_00: Um, and so let's look at the story of heparin versus placebo in uh acute coronary syndrome.
• 7:55
  SPEAKER_00: And you know, does it really matter?
• 7:57
  SPEAKER_00: Does grade impact?
• 7:58
  SPEAKER_00: Well, obviously it does because I told you it did.
• 8:00
  SPEAKER_00: But let me just walk you through a little bit of the history.
• 8:03
  SPEAKER_00: So initially, Cochrane Review did this uh analysis on heparin versus placebo in acute coronary syndrome.
• 8:11
  SPEAKER_00: Okay.
• 8:12
  SPEAKER_00: Two Cochrane reviews of heparin versus placebo looked at, and when they define heparin, they don't they didn't include all, they defined it as including low moleculate heparins with unfractionated heparin.
• 8:26
  SPEAKER_00: So you have to go in that inclusion exclusion criteria to see how they did it.
• 8:30
  SPEAKER_00: I personally disagree uh with the protocols that were done because they largely excluded low-vonox or anoxaparin.
• 8:40
  SPEAKER_00: Um, and so when you use the same trial sets, they did it in 2008 and then again in 2014.
• 8:46
  SPEAKER_00: And 2014 is when they applied the risk of bias and then the grade certainty assessment, and that changed their overall confidence in the overall result that was reported.
• 9:02
  SPEAKER_00: So, what I've shown you here is a screenshot, and generally, again, Cochrane reviews when when one gets updated, they remove the old one.
• 9:09
  SPEAKER_00: Well, thankfully I have a you know this, but but when you look at it, um, and these are the main results, you'll see that in 2014, I just want to read the I want to read the main results here too, you just kind of word for word.
• 9:23
  SPEAKER_00: Compared to placebo, and again, I'm not suggesting you read abstract uh only, but I've read this, I've analyzed the paper uh or the review, and you should do the same.
• 9:34
  SPEAKER_00: But compared to placebo, just listen to this, patients treated with heparins, all of them included except an oxyparine, had similar risk of mortality, revascularization, recurrent angina, major bleeding, and thrombocytopenia.
• 9:51
  SPEAKER_00: However, those treated with heparins had decreased risk of MI and higher incidence of minor bleeding.
• 10:00
  SPEAKER_00: So there is, yeah, there was no significant effect compared to placebo on mortality revascularization, but there was a decreased risk of MI.
• 10:10
  SPEAKER_00: And then there was a higher incidence of minor bleeding.
• 10:13
  SPEAKER_00: So then in 2014, they even say uh there were no new studies included in this update.
• 10:21
  SPEAKER_00: I mean, just flat out said it.
• 10:23
  SPEAKER_00: Okay.
• 10:24
  SPEAKER_00: Same numbers, same statistics, same everything.
• 10:27
  SPEAKER_00: Nothing has really changed.
• 10:30
  SPEAKER_00: But their conclusions and the tone of the recommendation changed.
• 10:35
  SPEAKER_00: Compared with placebo, patients treated uh with heparins had similar risk, and again, mortality, revascularization, recurrent anginot thrombocytopenia.
• 10:42
  SPEAKER_00: However, those treated with heparins had decreased myocardial function and a higher incidence of bleeding.
• 10:48
  SPEAKER_00: So that didn't change.
• 10:50
  SPEAKER_00: But they added this last sentence.
• 10:52
  SPEAKER_00: Overall, the evidence assessed in this review was classified as low quality according to the great approach.
• 11:00
  SPEAKER_00: The results presented in this review must therefore be interpreted with caution.
• 11:08
  SPEAKER_00: I mean, that's massive, guys.
• 11:12
  SPEAKER_00: And the reason that is so relevant is because when you go back, I mean, how many, how many years have transpired and if since the 2008, and if somebody just read the 2008, there was a very large misconception that Heparin offered really no value, which is not true, right?
• 11:40
  SPEAKER_00: And it was largely based off of the conclusions of that review.
• 11:46
  SPEAKER_00: Okay, but when you add in the quality assessment, you clearly see that man, this is not great data, and quite honestly, they didn't include all of the data.
• 11:58
  SPEAKER_00: Why?
• 11:59
  SPEAKER_00: I don't know, but they didn't.
• 12:01
  SPEAKER_00: Okay.
• 12:02
  SPEAKER_00: So, what are the core concepts of this?
• 12:04
  SPEAKER_00: Well, GRADE helps us to standardize that assessment, makes it more objective, okay, uh, related to the certainty of the evidence.
• 12:14
  SPEAKER_00: It can clearly change the final interpretation of results.
• 12:18
  SPEAKER_00: We just saw a perfect example of that.
• 12:21
  SPEAKER_00: Okay, it many times gets missed because unless you have the other ones to compare with to see the difference, like I just showed you, it's hard to determine that.
• 12:31
  SPEAKER_00: And then lastly, a statistically significant pool decimate with low or very low certainty should not automatically change practice.
• 12:42
  SPEAKER_00: And unfortunately, I'm concerned that that's what happened.
• 12:46
  SPEAKER_00: So, how do you read a great summary of findings table, right?
• 12:50
  SPEAKER_00: Well, you want to look at the outcome, what happened to the patients, you want to understand the relative effect or absolute effect, what were the number of patients in the trials, okay?
• 12:59
  SPEAKER_00: The certainty, was it high, moderate, low, very low?
• 13:03
  SPEAKER_00: Okay.
• 13:04
  SPEAKER_00: Why uh was the certainty rated down?
• 13:07
  SPEAKER_00: Look in the many times it's in the footnotes, okay?
• 13:11
  SPEAKER_00: And then put it in the perspective, what is the absolute effect?
• 13:15
  SPEAKER_00: Okay, because that's obviously the thing that's gonna make the most impact on patient decisions.
• 13:20
  SPEAKER_00: Well, I hope this episode helped you to connect why evaluating the quality of the evidence is just as important as interpreting the results themselves.
• 13:30
  SPEAKER_00: Remember, evidence-based medicine isn't about finding statistically significant information or a study.
• 13:37
  SPEAKER_00: It's about determining whether the evidence that we have is trustworthy enough to influence patient care decisions.
• 13:44
  SPEAKER_00: Tools like GRADE help us to place research in its proper clinical context so that we can make better informed decisions for our patients.
• 13:54
  SPEAKER_00: So, thanks so much for joining me on This Is Why.
• 13:57
  SPEAKER_00: This is part of a series on systematic review and meta-analysis.
• 14:00
  SPEAKER_00: And again, I recommend you go through the other parts to really build that solid foundation so that you can make a more complete and appropriate assessment.
• 14:11
  SPEAKER_00: And we also have playlists over on YouTube and at thisiswhy.health.
• 14:16
  SPEAKER_00: And again, I if you felt like this was helpful, leave a comment, share your experience, add to.
• 14:20
  SPEAKER_00: We all learn from each other.
• 14:22
  SPEAKER_00: If you haven't done so, subscribe to the channel.
• 14:24
  SPEAKER_00: We really appreciate your support, especially with likes.
• 14:27
  SPEAKER_00: That helps us to continue to provide and deliver this kind of content for you.
• 14:31
  SPEAKER_00: So thanks so much for joining, and I'll see you next time.