Latest / This Is Why with Dr. Busti / How to Report & Evaluate a Systematic Review: PRISMA & AMSTAR-2

Transcript

How to Report & Evaluate a Systematic Review: PRISMA & AMSTAR-2 · This Is Why with Dr. Busti

• 0:00
  SPEAKER_00: You know, many clinicians assume that if a systematic review or meta-analysis is published in a reputable journal, it must be trustworthy.
• 0:07
  SPEAKER_00: This is why reporting guidelines and critical appraisal tools matter, because a published review is only as reliable as the methods used to produce and report it.
• 0:19
  SPEAKER_00: If those methods aren't transparent, complete, and reproducible, it's difficult to know whether the conclusion should influence patient care.
• 0:28
  SPEAKER_00: Tools like Prisma help authors report a systematic review completely, and tools like AMSTAR2 help readers judge whether the methods are trustworthy.
• 0:40
  SPEAKER_00: They provide a structured framework to determine whether the evidence was gathered, analyzed, and reported in a way that deserves our confidence.
• 0:53
  SPEAKER_00: Given that, I'm Dr.
• 0:54
  SPEAKER_00: Busai, your host on This Is Why.
• 0:56
  SPEAKER_00: Welcome back again to the series on systematic reviews and meta-analysis, which is part of a larger discussion that we're doing on evidence-based medicine, biostatistics, clinical literature, evaluation.
• 1:14
  SPEAKER_00: Because when you know why you do what you do in the proper context of the evidence, it changes the how.
• 1:20
  SPEAKER_00: And that's what most of us are trying to make sure that we do right.
• 1:24
  SPEAKER_00: And that changes everything about the patient-oriented decisions that we're going to make on a daily basis.
• 1:31
  SPEAKER_00: Okay?
• 1:31
  SPEAKER_00: So when we approach reading a paper and you open that systematic review, there are a series of questions that you should ask yourself.
• 1:42
  SPEAKER_00: Now, if you're not sure what a systematic review is, a meta-analysis, Cochrane review, review article, how they're the different different, what their pros and cons are, all that stuff.
• 1:51
  SPEAKER_00: I cover those parts and dive deep into how they're done, how to interpret that, what is a test of heterogeneity, what is things like tile squared, and you know, what method and model for the point estimate is done.
• 2:05
  SPEAKER_00: I cover all those and other topics if you need them.
• 2:07
  SPEAKER_00: Well, we're talking about actually getting the paper or at the time that you're going to publish it.
• 2:12
  SPEAKER_00: So you're either the researcher publishing it or you're the person reading it.
• 2:16
  SPEAKER_00: But either way, I personally believe that we should know both.
• 2:21
  SPEAKER_00: Because if I want I want to see if a paper was published correctly, it should be seeing the right parts, but then I'm also going to critically appraise it.
• 2:29
  SPEAKER_00: In fact, the publishing reporting guidelines help me to make sure that the paper was done right and the study was done right.
• 2:39
  SPEAKER_00: It's part of the appraisal process for me.
• 2:41
  SPEAKER_00: Okay.
• 2:42
  SPEAKER_00: And so there's a series of questions that we want to ask.
• 2:45
  SPEAKER_00: Were they transparent?
• 2:47
  SPEAKER_00: Did they register and publish the protocol?
• 2:50
  SPEAKER_00: Yes, for a systematic review and meta-analysis, you there are definite websites that can do that.
• 2:56
  SPEAKER_00: So you let people know that this is being done, because it's a lot of work.
• 3:01
  SPEAKER_00: Okay.
• 3:01
  SPEAKER_00: I don't want to go do something that necessarily somebody else is going to do unless two meta-analysis agree, and then it helps, but again, right, it helps us understand what's being done, just like clinical trial registries.
• 3:14
  SPEAKER_00: Um, is it current?
• 3:16
  SPEAKER_00: Check the search uh dates on it.
• 3:20
  SPEAKER_00: Is the question rather explicit in following the structure of the format of a clinical question that EBM supports, PICO or PCOTS?
• 3:32
  SPEAKER_00: Okay, T and S add in the time to the outcome and the setting, which is what I would advocate.
• 3:39
  SPEAKER_00: Was the reporting aligned with the Prisma statement?
• 3:45
  SPEAKER_00: Again, these are the reporting guidelines on how a systematic review is to be published in the medical literature.
• 3:53
  SPEAKER_00: So journals follow these criteria, authors should follow these criteria, you should know it up front if you're gonna do one, so you don't waste your time collecting and organizing the information in the wrong way.
• 4:05
  SPEAKER_00: Okay?
• 4:06
  SPEAKER_00: Was the search strategy to acquire the evidence comprehensive enough?
• 4:13
  SPEAKER_00: Again, in the steps of how to do one of these, I go into more details, but we're talking pretty extensive.
• 4:20
  SPEAKER_00: It's a lot of work and can take weeks to months, sometimes even longer.
• 4:26
  SPEAKER_00: Okay.
• 4:27
  SPEAKER_00: Could another team read your protocol and reproduce the methods and what was written?
• 4:36
  SPEAKER_00: That's how transparent it is.
• 4:38
  SPEAKER_00: It's no different than a clinical trial protocol.
• 4:41
  SPEAKER_00: And then was there an assessment of the risk for bias?
• 4:45
  SPEAKER_00: And there's like a number of tools, again, that I dive into on grade and assessment tools and the steps that are done.
• 4:54
  SPEAKER_00: If you're not familiar with that, then please check those parts out in my other lectures.
• 4:59
  SPEAKER_00: Okay.
• 5:00
  SPEAKER_00: But these are some of the things that you want to ask and look at.
• 5:04
  SPEAKER_00: And when we look at the QFAS QFAS, right, it mirrors the process of how we answer a clinical question according to the principles of evidence-based medicine, the five A's.
• 5:18
  SPEAKER_00: Okay.
• 5:19
  SPEAKER_00: But what's typically unique in the systematic review is the systematic, reproducible, transparent approach within the specific use of studies.
• 5:33
  SPEAKER_00: Okay, and you have two independent reviewers at minimum making some assessments of various parts, okay?
• 5:43
  SPEAKER_00: And and so you see that model and that parallel, or you should, in order for it to be done.
• 5:52
  SPEAKER_00: And then was the synthesis correct?
• 5:54
  SPEAKER_00: And was it a priority defined in the method and the model for determining the point estimate?
• 6:01
  SPEAKER_00: Do you understand what the measure of effect was?
• 6:04
  SPEAKER_00: And did then they report it that way?
• 6:07
  SPEAKER_00: All right, no different than a clinical trial.
• 6:10
  SPEAKER_00: Same process.
• 6:12
  SPEAKER_00: Now there's a number of reporting guidelines out there based on the type of study or paper that you're doing.
• 6:22
  SPEAKER_00: Okay, and I've listed some of those there and some of the references.
• 6:25
  SPEAKER_00: You can search for them on the internet and things like that.
• 6:28
  SPEAKER_00: But but many of these are validated tools.
• 6:32
  SPEAKER_00: And Prisma is clearly the one that's linked to systematic reviews and meta-analysis.
• 6:38
  SPEAKER_00: And so that's a great kind of board question that shows up, but it's also a tool for you to use.
• 6:44
  SPEAKER_00: What about critical appraisal tools?
• 6:47
  SPEAKER_00: Now, some of you who've listened to some of my other lectures in the evidence-based medicine series where I talk about the assessment of the literature, I don't like the word critical appraisal tool, because critical can sometimes be viewed as negative.
• 7:04
  SPEAKER_00: I would rather view it as a systematic assessment tool, that I'm able to read a paper in a systematic, reproducible method, no matter what the topic is or where it's published.
• 7:19
  SPEAKER_00: Okay.
• 7:20
  SPEAKER_00: And I'm not approaching it with a negative mindset, as in seems like journal clubs are about tearing apart papers.
• 7:28
  SPEAKER_00: What's wrong with them?
• 7:29
  SPEAKER_00: And what did the author do wrong?
• 7:31
  SPEAKER_00: And what are the research?
• 7:32
  SPEAKER_00: I mean, if you've ever conduct clinical research, which I have, um, and I'm sure many of you listen, it is not easy.
• 7:39
  SPEAKER_00: It is very difficult to do.
• 7:41
  SPEAKER_00: So give some grace and to our colleagues who are working hard to try to bring good evidence to the literature.
• 7:49
  SPEAKER_00: Okay.
• 7:50
  SPEAKER_00: Now, I will say that there's a lot of stuff published, and there's a lot of journals out there that don't always have good information.
• 7:58
  SPEAKER_00: But that's why you need to know how to appraise it appropriately.
• 8:02
  SPEAKER_00: And these are tools that are out there, and the one for systematic review of meta-analysis, uh, AMSTAR2 is probably the most common.
• 8:10
  SPEAKER_00: So we have how do you report, and then how do you appraise the paper.
• 8:17
  SPEAKER_00: And there are CATS design, many of them validated tools that are out there.
• 8:23
  SPEAKER_00: So let's dive in a little bit to Prisma.
• 8:26
  SPEAKER_00: Okay, so this is just a snapshot of what the Prisma uh website looks like.
• 8:31
  SPEAKER_00: You can see down at the very bottom there are hyperlinks to the actual checklist that you can download and use.
• 8:37
  SPEAKER_00: So for those of you doing journal clubs, or this is new for you, you're not sure, then those are tools that are freely available.
• 8:45
  SPEAKER_00: Uh very nice service.
• 8:47
  SPEAKER_00: There's also an expanded checklist.
• 8:49
  SPEAKER_00: There's a flow diagram, and what the flow diagram is trying to help you to see is how is the paper, how was the flow of the studies acquired and handled.
• 9:00
  SPEAKER_00: Um, and then they give some information about their own analysis and how they came up with it and where these things come from.
• 9:07
  SPEAKER_00: But this is what the checklist looks like.
• 9:10
  SPEAKER_00: And I realize it's a small font, and you sometimes may not be able to see it depending on on the screen that you're looking at it on.
• 9:17
  SPEAKER_00: But what it does is it basically walks you through the structure of the paper, no different than a consort statement does for a randomized controlled trial.
• 9:27
  SPEAKER_00: You have the title, you have the abstract, you have the introduction, you have the methods, and then there's subparts to those.
• 9:33
  SPEAKER_00: And what you're doing is you're you're trying to, it's asking you questions and it's pointing you to, again, a very systematic, transparent, objective way of reading the paper so that you don't miss things.
• 9:47
  SPEAKER_00: And it may sound childish or even trivial, depending on your level of experience and training.
• 9:52
  SPEAKER_00: And when I was at Oxford getting my master's degree in evidence-based healthcare, I can tell you they made us do these, yes, at Oxford.
• 10:01
  SPEAKER_00: And I thought at first, this is absurd.
• 10:04
  SPEAKER_00: I know how to read a paper.
• 10:05
  SPEAKER_00: Um, but I absolutely agreed with the program's effort to try to get us to exposure to these tools because what they really establish is very similar to the soap note when we work up a patient, isn't it?
• 10:24
  SPEAKER_00: It's a framework, it's a structure, it's reproducible, it's systematic so that we don't miss or forget to do something.
• 10:32
  SPEAKER_00: It offers a more complete and comprehensive picture.
• 10:36
  SPEAKER_00: This is that sort of algorithm or the flow diagram that you'll commonly see where they do the identification of the how do you find the papers you're acquiring the studies, how are those studies then handled by the independent reviewers from a screening standpoint?
• 10:53
  SPEAKER_00: How did they get excluded?
• 10:55
  SPEAKER_00: What number?
• 10:56
  SPEAKER_00: Right?
• 10:57
  SPEAKER_00: All and you should see this no different than you should see that in a well-published randomized controlled trial, isn't it?
• 11:06
  SPEAKER_00: It's very similar.
• 11:07
  SPEAKER_00: Um and so these go on even to the results and the discussion and then for other information.
• 11:13
  SPEAKER_00: So you these tools will evolve.
• 11:16
  SPEAKER_00: Okay, it can change any day, it doesn't matter.
• 11:20
  SPEAKER_00: It doesn't don't get focused on the number of the year.
• 11:24
  SPEAKER_00: It's the process, it's the concept behind the approach to reading the literature that matters.
• 11:32
  SPEAKER_00: What about the appraisal side?
• 11:34
  SPEAKER_00: Okay, so Prisma was the reporting guideline, AMSTAR two, and who knows, it'll be AMSTA 3 and 4 or whatever, right?
• 11:44
  SPEAKER_00: It it again, it's a systematic approach.
• 11:48
  SPEAKER_00: And so uh show you some of the examples here of what the tool looks like, and it has like literally like it walks you through it, check, check.
• 11:58
  SPEAKER_00: I mean, I had to fill these out.
• 12:01
  SPEAKER_00: And it did take a little bit of time, but again, the process of reading through this and checking to find out what was there, was not there, helped me to begin to formulate from a little bit of a different perspective my assessment on the quality of this paper that I'm reading and what am I gonna do with it.
• 12:27
  SPEAKER_00: Um and they would make us defend that, right?
• 12:30
  SPEAKER_00: So it's it's a it's a critical appraisal tool.
• 12:34
  SPEAKER_00: It's about 16 items, okay.
• 12:37
  SPEAKER_00: Um, and several of those cover various domains like protocol, the justification, uh, for why you included excluded studies, okay.
• 12:48
  SPEAKER_00: Um, some of the analysis, publication bias, was it done?
• 12:52
  SPEAKER_00: Okay.
• 12:53
  SPEAKER_00: Um, and so these tools, again, overall assessing trustworthiness of the inf of the study, is it valid, is it quality, and was it reported correctly?
• 13:06
  SPEAKER_00: And so, in summarizing this, reporting guidelines Prisma 2020, um, was the work described completely?
• 13:15
  SPEAKER_00: AMSTAR II, Robus, those are for determining the trustworthiness.
• 13:20
  SPEAKER_00: Okay, and at the bedside, even a well-reported review can have be the wrong evidence for the patient.
• 13:27
  SPEAKER_00: And that's why the PICO should match, your PICO should match the paper's PICO.
• 13:34
  SPEAKER_00: Okay.
• 13:34
  SPEAKER_00: So I hope you found this uh episode and topic to be helpful in the overall topic and pointing you to the right information, especially for those of you starting out to either do these or read these papers.
• 13:47
  SPEAKER_00: Okay, I would encourage you to not bypass this.
• 13:51
  SPEAKER_00: I know it seems, like I said, trivial, maybe even childish to some degree, but the exercise will be worth it to you.
• 13:59
  SPEAKER_00: Okay.
• 14:00
  SPEAKER_00: So remember that reading the conclusions of a systematic review, guys, or meta-analysis is only one part of the critical appraisal.
• 14:09
  SPEAKER_00: As clinicians, we also need to determine whether that review was conducted and reported in a transparent, systematic, and reproducible way before allowing it to influence our day-to-day patient decisions.
• 14:24
  SPEAKER_00: So thank you so much for joining me.
• 14:26
  SPEAKER_00: I hope that you enjoyed it.
• 14:28
  SPEAKER_00: If you did, please give us a like, uh post a comment.
• 14:31
  SPEAKER_00: Those things help support the channel.
• 14:34
  SPEAKER_00: Uh feel free to share this with your friends, colleagues, those of you that are doing research.
• 14:38
  SPEAKER_00: Um, we offer those playlists there to create a structured uh curriculum if you if your program needs that.
• 14:46
  SPEAKER_00: Um, and many programs do use it.
• 14:49
  SPEAKER_00: So uh thanks again for joining me.
• 14:51
  SPEAKER_00: Uh, appreciate all that you're doing, and we'll see you next time.