Latest / The Ask Mike Reinold Show / Drowning in Evidence: How to Read a Meta-Analysis Without Getting Fooled

Transcript

Drowning in Evidence: How to Read a Meta-Analysis Without Getting Fooled · The Ask Mike Reinold Show

• 0:50
  SPEAKER_00: On this episode of The Ask Mike Reinhold Show, we have a journal article review.
• 0:53
  SPEAKER_00: We actually go over an editorial talking about maybe the fact that meta-analyses aren't as good as we once thought.
• 1:01
  SPEAKER_00: The Ask Mike Reinhold Show.
• 1:05
  SPEAKER_00: Helping people feel better, move better, perform better.
• 1:18
  SPEAKER_00: Before we get to the podcast, I wanted to make sure you knew about my free online course on the introduction to performance therapy and training.
• 1:26
  SPEAKER_00: If you want to learn how to get started optimizing and enhancing performance, this is the course for you.
• 1:31
  SPEAKER_00: Head to micrenault.com/slash performance.
• 1:34
  SPEAKER_00: Sign up today.
• 1:35
  SPEAKER_00: Welcome back, everybody.
• 1:36
  SPEAKER_00: The latest episode of the Ask Mike Reynolds Show.
• 1:38
  SPEAKER_00: We are here with another journal article review episode.
• 1:42
  SPEAKER_00: I'm joined with Lenny McCrina, Dave Tilly, Brenda Gates, Noah Sprinkle, and Anthony Videtto.
• 1:47
  SPEAKER_00: We're here with another journal article.
• 1:49
  SPEAKER_00: And this one's a little different.
• 1:51
  SPEAKER_00: And, you know, I think that's okay, right?
• 1:53
  SPEAKER_00: We kind of said we're going to review some journal articles that make a big impact on our clinical, you know, day-to-day, I guess.
• 2:00
  SPEAKER_00: And, you know, I think a lot of people assume that means like research studies where we look at some findings of things that come out.
• 2:06
  SPEAKER_00: But we have an interesting one today.
• 2:08
  SPEAKER_00: And it's actually an editorial.
• 2:09
  SPEAKER_00: So it's not a research study that we're going to draw conclusions from, but an editorial.
• 2:14
  SPEAKER_00: And the reason why we picked this one for this episode is I think we all agree, right?
• 2:19
  SPEAKER_00: Like when we start talking about this, we kind of agree to the concept here.
• 2:22
  SPEAKER_00: But I think this is a very important editorial uh piece and an opinion for our profession right now, because we're going down a road that is uh, you know, misleading, I will say.
• 2:32
  SPEAKER_00: So let's start with this.
• 2:33
  SPEAKER_00: The article we're talking about, and the links will be in the show notes so you can check it out, is Drowning in Evidence, the meta-analysis boom and rigor it demands.
• 2:41
  SPEAKER_00: And this is from the Scandinavian Journal of Medicine and Science and Sport from 2026.
• 2:46
  SPEAKER_00: And it's from Franco and Pelazieri and Ian Schreer, who are two well-established and excellent researchers and academic people.
• 2:56
  SPEAKER_00: Um, they know what they're talking about.
• 2:58
  SPEAKER_00: So when when they put together an editorial with their opinion, and trust me, this is a strong opinion piece from them.
• 3:04
  SPEAKER_00: When they put that together, though, I think that's something that we need to listen to a little bit.
• 3:09
  SPEAKER_00: So um, so just real quick, it's an editorial, it's not long, so you should definitely check it out.
• 3:14
  SPEAKER_00: Um, but what they did uh essentially was this they start off by saying, just because a study is a meta-analysis does not mean that it's a great study.
• 3:25
  SPEAKER_00: And gosh, I remember putting this on Twitter back before it was Twitter, back it was X, and then everybody yelled at me again, which is part of why I got off Twitter.
• 3:33
  SPEAKER_00: But um uh I remember putting on there, um, putting a bunch of crappy studies together does not make one good study, right?
• 3:40
  SPEAKER_00: It just makes a big crappy study.
• 3:42
  SPEAKER_00: And you know, I guess maybe, you know, not everybody wanted to hear that.
• 3:44
  SPEAKER_00: But that's what I think sometimes with these meta-analyses.
• 3:47
  SPEAKER_00: So what they kind of determine here was this is we're seeing a rise in meta-analyses.
• 3:52
  SPEAKER_00: If you look at the percentage of meta-analyses that are being conducted, it's actually crazy the percentage that are being done in sports medicine.
• 4:01
  SPEAKER_00: It's very large.
• 4:02
  SPEAKER_00: Sports-related meta-analyses now make up roughly 16% of all meta-analyses.
• 4:07
  SPEAKER_00: And think about that.
• 4:08
  SPEAKER_00: Sports is not 16% of medicine.
• 4:11
  SPEAKER_00: So, for some reason in our field, we're we're really shifting towards that.
• 4:15
  SPEAKER_00: And we can talk about why, because I do think there's some reasons why, but I'll let everybody else chime in on that.
• 4:19
  SPEAKER_00: But essentially, we're see we're seeing this.
• 4:21
  SPEAKER_00: And what's happening is I think a lot of people are saying since it's a meta-analysis, this thing must be perfect.
• 4:26
  SPEAKER_00: So, what they said is you have to break this down.
• 4:30
  SPEAKER_00: You have to look at a meta-analysis and determine if it's a good study.
• 4:34
  SPEAKER_00: And what they did is they kind of came up with like a four-step system that they recommend you do.
• 4:38
  SPEAKER_00: And by the way, this isn't necessarily on how to perform a meta-analysis, this is on how to interpret it as well.
• 4:44
  SPEAKER_00: So it's for all of us because you should read these meta-analyses before you determine if you think this is something you should apply to your clinical practice.
• 4:52
  SPEAKER_00: So the first thing they they kind of came up with was this framework.
• 4:55
  SPEAKER_00: And what they they come is these four steps.
• 4:57
  SPEAKER_00: They call it identify, appraise, synthesize, and then interpret.
• 5:01
  SPEAKER_00: And they gave, you know, kind of steps for each one of those.
• 5:04
  SPEAKER_00: And I really urge you to check that out because you should read each one of those things.
• 5:08
  SPEAKER_00: But what they're saying basically is did they identify a good research question, right?
• 5:12
  SPEAKER_00: Was it performed with proper methodology?
• 5:15
  SPEAKER_00: And they gave some really good examples of how you can perform a sloppy meta-analysis versus a good meta-analysis.
• 5:21
  SPEAKER_00: You should then appraise it and not just appraise it for the outcomes necessarily or more importantly, the quality, right?
• 5:28
  SPEAKER_00: Which is something that everybody does.
• 5:30
  SPEAKER_00: But they bring up the point that we often leave out the risk of bias and they they make a strong point towards that.
• 5:36
  SPEAKER_00: So you should check that out.
• 5:37
  SPEAKER_00: And then for synthesize, they bring up a couple of examples.
• 5:40
  SPEAKER_00: But the one I thought was really cool was they talk about the concept of vote counting, right?
• 5:44
  SPEAKER_00: So they say six out of seven studies showed that the Nordic hamstring exercise is good at preventing an injury, right?
• 5:50
  SPEAKER_00: So it sounds like, wow, you, you know, that's the majority of studies, but you don't know the, you know, the population of each one and you don't know the percentage of those there.
• 5:59
  SPEAKER_00: So it's kind of like a kind of a sly way of doing it.
• 6:02
  SPEAKER_00: And then obviously putting that all together for interpretation.
• 6:05
  SPEAKER_00: So again, what's the clinical take-home for us here?
• 6:08
  SPEAKER_00: Kind of to summarize this right here, is just because it's a meta-analysis doesn't mean it's a great research study.
• 6:14
  SPEAKER_00: And everybody thinks these meta-analyses now are like like scripture, right?
• 6:17
  SPEAKER_00: That if a meta-analysis says something, you can you can you can go for it.
• 6:21
  SPEAKER_00: That's the way to do it.
• 6:22
  SPEAKER_00: And we're starting to see even people on social media kind of like uh like the rehab, the rehab bros.
• 6:27
  SPEAKER_00: I think I just came up with that.
• 6:28
  SPEAKER_00: You know, like you know, we say finance bros, it's like rehab bros.
• 6:31
  SPEAKER_00: It's like right, like those those young new rehab guys on social media, that's like they want to be the contrarian, they want to say nothing works, or they want to confirm their bias with a study, right?
• 6:42
  SPEAKER_00: And you kind of put that together.
• 6:43
  SPEAKER_00: So uh I'll leave it at that.
• 6:45
  SPEAKER_00: But I think if you're you know truly into research and you want to make sure that you're reading stuff on your own, you should check out this article and follow their their four-step process so that way you can identify a good meta-analyses versus a bad one because there are a lot of bad ones out there right now.
• 7:02
  SPEAKER_00: Um, so what do you guys think?
• 7:04
  SPEAKER_00: We've we've been talking about this a lot in the background, and we've been talking about how there's so many studies out there.
• 7:09
  SPEAKER_00: What how do you guys handle this?
• 7:10
  SPEAKER_00: What are your thoughts?
• 7:11
  SPEAKER_00: What'd you think of this article?
• 7:12
  SPEAKER_00: Dave, you want to start?
• 7:13
  SPEAKER_02: Yeah, I just jump in because fresh of mine because I gave an in-service to the students a few weeks ago, and this was on my mind a lot, right?
• 7:18
  SPEAKER_02: Because um I treat a lot of people for back pain, for example, and you know, did a huge PubMed review last year.
• 7:23
  SPEAKER_02: And what this reflected for me is what is your system for analyzing evidence and applying stuff in the clinic?
• 7:31
  SPEAKER_02: When someone sits in front of you with a certain problem, back pain, knee pain, whatever, ACL stuff, Lenny talks a lot about graphs and like quads versus patellers, like what is your system for gathering information and delivering that to a person who's in front of you who your job is to educate them on what is available out there?
• 7:46
  SPEAKER_02: Like that's that's kind of part of our profession, right?
• 7:49
  SPEAKER_02: So if your system for educating somebody on quad tenant graph versus pateller graft, LET or not is Instagram, Facebook, and an occasional Google search, right?
• 7:58
  SPEAKER_02: Like that's not a very good system for analyzing high-level evidence, right?
• 8:01
  SPEAKER_02: Like the hard thing about our profession, especially in our situation where a lot of our time is with clinic and we don't get like, you know, we don't get paid lunches, paid hours like for research and stuff like that.
• 8:10
  SPEAKER_02: Should just be your work you do on your own.
• 8:12
  SPEAKER_02: Like your system, unfortunately, has to be waking up before a clinic, getting two articles, drinking coffee, going through in-depth, reading the full study, looking at the methodology, and and then taking an evaluation of does this apply?
• 8:25
  SPEAKER_02: Is the study good?
• 8:26
  SPEAKER_02: Do they have good internal, external validity?
• 8:28
  SPEAKER_02: Does this apply to the per people that I see in front of me?
• 8:30
  SPEAKER_02: And how confident do I feel that this is something I can educate somebody about and use in the clinic, right?
• 8:35
  SPEAKER_02: So I think the sentence in here that I stuck out to me is like, if the studies are so biased that the true effect could differ substantially from our estimate in either direction, how can one support the claim that the intervention works?
• 8:46
  SPEAKER_02: Right.
• 8:46
  SPEAKER_02: And that's kind of what I was alluding to, which is if I if I look at a bunch of stuff for spiny fractures and someone says, you know, um, hamstring stretching is good for back pain, and that's the headline article.
• 8:56
  SPEAKER_02: And I just read that on Google, but I didn't look at the fact that it was seven kids in the middle of Switzerland who play handball and they're from eight to 12 years old, right?
• 9:05
  SPEAKER_02: How does that apply to the person that I'm treating?
• 9:06
  SPEAKER_02: It's a college gymnast with back pain.
• 9:08
  SPEAKER_02: You know, like it's it's kind of tough.
• 9:10
  SPEAKER_02: So that was what stuck out to me is like unfortunately, you need to have a system in place where you're willing to do the work to read articles and analyze their methodology before you take conclusion.
• 9:19
  SPEAKER_02: And in a meta-analysis, you have to look at their methods inside the analysis to say, did the studies, like you said, go inside?
• 9:25
  SPEAKER_02: Is it worth it?
• 9:25
  SPEAKER_02: Is it worth my time?
• 9:26
  SPEAKER_02: Was it high quality?
• 9:27
  SPEAKER_02: If six of the seven studies in the meta-analysis are mixed populations with mixed interventions, how how in the world can I apply that to the person in front of me?
• 9:36
  SPEAKER_00: Agreed.
• 9:37
  SPEAKER_00: And you know, you brought up like you brought up a couple things on meta-analyses that I think's you know, really interesting with like how like, you know, you talked about six, seven patients, right?
• 9:48
  SPEAKER_00: You brought that up.
• 9:49
  SPEAKER_00: I want to be very clear about that.
• 9:51
  SPEAKER_00: But more importantly for me, look it up right now.
• 9:54
  SPEAKER_00: I want everybody to go look this up right now.
• 9:56
  SPEAKER_00: Do a lit review right now on the effectiveness of manual therapy for shoulder pain.
• 10:01
  SPEAKER_00: And to me, this just this highlights the problem with systematic reviews, like these meta-analyses kind of things.
• 10:08
  SPEAKER_00: You're gonna find that the patient population was 18 to 65 years old, then they had shoulder pain and they had manual therapy.
• 10:15
  SPEAKER_00: They don't define what manual therapy is, what shoulder pain is.
• 10:19
  SPEAKER_00: And my gosh, if an 18 and a 65-year-old have the same kind of shoulder pain, that's crazy.
• 10:23
  SPEAKER_00: But who performed a manual therapy, blah, blah, blah, blah, blah.
• 10:25
  SPEAKER_00: We'll go down that list all freaking day.
• 10:28
  SPEAKER_00: But you'll find the rehab bros.
• 10:30
  SPEAKER_00: I'm gonna trademark that on Instagram basically saying, like, see, manual therapy doesn't work for shoulder pain, right?
• 10:36
  SPEAKER_00: So, you know, it's hard to draw conclusions based on that.
• 10:39
  SPEAKER_00: So to Dave's point, it's hard to draw conclusions if if maybe it's not the right population, but it's also hard to draw conclusions when we have such diluted crap out there, right?
• 10:48
  SPEAKER_00: Of course nothing's gonna work.
• 10:50
  SPEAKER_00: So, guess what the outcome is?
• 10:52
  SPEAKER_00: Well, guess what their conclusion is?
• 10:53
  SPEAKER_00: Well, well, the you know, we we um, you know, the the results are uh are questionable, right?
• 10:59
  SPEAKER_00: That's what they'll say.
• 11:00
  SPEAKER_00: Or, you know, we we're not a hundred percent sure.
• 11:02
  SPEAKER_00: And people say, well, see, there's no evidence that it works.
• 11:05
  SPEAKER_00: So you got to be really careful.
• 11:06
  SPEAKER_00: Like you can't put together a bunch of crappy articles, right?
• 11:09
  SPEAKER_00: I guess that's that main point.
• 11:10
  SPEAKER_00: But what else?
• 11:10
  SPEAKER_00: Who else wants to jump in on this one?
• 11:12
  SPEAKER_00: This is a this is a good topic, but who's got some some more for me?
• 11:15
  SPEAKER_00: Anthony, you want to jump in?
• 11:16
  SPEAKER_01: Yeah, um, I guess I'll add that sometimes when I'm going to the literature or looking for research, I might find that I'm a little bit more lazy than most people.
• 11:28
  SPEAKER_01: And I find myself hovering towards meta-analyses a lot because I think, oh, well, look, these people, these authors are synthesizing a lot of data for me on the topic.
• 11:37
  SPEAKER_01: And I can appreciate that because maybe I don't have a lot of time to try to find an answer to a question I have that I want to present to a patient that day, let's say.
• 11:46
  SPEAKER_01: And so I think I just kind of take meta-analyses as kind of like descripture, like you said, because you know, these authors did all the synthesis for me.
• 11:54
  SPEAKER_01: I'm just trusting that they did a good job at, you know, including good studies and, you know, kind of getting rid of the bad studies.
• 12:00
  SPEAKER_01: So I guess this is kind of maybe a little bit more eye-opening for me that I'm probably need to do a better job at like actually diving into, you know, the studies that were included in these papers, but then have this framework that this paper is kind of talking about in terms of how am I identifying a good meta-analysis versus a bad meta-analysis.
• 12:17
  SPEAKER_01: So as someone who might just kind of be looking for a quick answer, I probably need to dive into this stuff a little bit more personally.
• 12:23
  SPEAKER_01: Um, but I feel like a lot of people out there might relate to that kind of um outcome as well.
• 12:29
  SPEAKER_00: Yeah, I mean, if if if if you feel that way, I bet you tons of people do as well.
• 12:33
  SPEAKER_00: And I I think you're right.
• 12:34
  SPEAKER_00: I think we all want that, right?
• 12:36
  SPEAKER_00: We all look for that, right?
• 12:37
  SPEAKER_00: And then, you know, then we're sometimes we're disappointed.
• 12:40
  SPEAKER_00: But um, can we get serious?
• 12:42
  SPEAKER_00: Can we talk?
• 12:43
  SPEAKER_00: You want to what's the dirty secret of meta-analyses?
• 12:47
  SPEAKER_00: Do we know this?
• 12:48
  SPEAKER_00: Do you guys know?
• 12:49
  SPEAKER_00: All right, so let's start with this.
• 12:50
  SPEAKER_00: There's two dirty secrets.
• 12:51
  SPEAKER_00: Two dirty secrets.
• 12:52
  SPEAKER_00: This is this is a this is a tell all podcast episode.
• 12:55
  SPEAKER_00: Dirty secret number one.
• 12:57
  SPEAKER_00: Do you know why so many are being conducted?
• 13:00
  SPEAKER_00: What do you guys think?
• 13:01
  SPEAKER_00: Anyone jump in?
• 13:02
  SPEAKER_02: They get published faster.
• 13:03
  SPEAKER_02: They're like bigger.
• 13:05
  SPEAKER_00: Yep, they get published faster.
• 13:06
  SPEAKER_00: They're easier.
• 13:07
  SPEAKER_00: Easier to conduct.
• 13:08
  SPEAKER_00: Yeah, so you're you're seeing a lot of fellows, residents, research assistants, uh, academic institutes.
• 13:14
  SPEAKER_00: They you could sit in front of a computer and you don't need a lab, you don't need uh IRB, right?
• 13:20
  SPEAKER_00: You don't need to go in there, and you can pump these things out, right?
• 13:23
  SPEAKER_00: So you you can get so many publications.
• 13:24
  SPEAKER_00: So that's part why we're doing it.
• 13:26
  SPEAKER_00: Okay, what's the flip to that?
• 13:27
  SPEAKER_00: Why are the journals publishing so many?
• 13:30
  SPEAKER_00: Dirty secret number two.
• 13:32
  SPEAKER_00: I'm gonna whisper a little ASMR for that.
• 13:35
  SPEAKER_00: People read them more.
• 13:37
  SPEAKER_00: People read them more, they download them more, and it helps with the ranking of the journal.
• 13:41
  SPEAKER_00: Comes becomes popular.
• 13:43
  SPEAKER_00: So authors want to do them, journals want to publish them, and then rehab bros on Instagram want to say nothing works, right?
• 13:49
  SPEAKER_00: And that's that's that's the dirty cycle of meta-analyses right now.
• 13:53
  SPEAKER_00: So look, we can do better.
• 13:54
  SPEAKER_00: I don't expect everybody in the world to go out there and just do these huge lit reviews all the time.
• 13:58
  SPEAKER_00: But you know what?
• 13:59
  SPEAKER_00: We're in a point now where we can do better, right?
• 14:02
  SPEAKER_00: We have websites like open evidence, right?
• 14:04
  SPEAKER_00: If you're not using open evidence right now, which is like an AI uh tool that's specific to medical journals to help you to go through and search your stuff, you should be, right?
• 14:13
  SPEAKER_00: But we can do better now.
• 14:14
  SPEAKER_00: And the other thing is just follow the right people online, right?
• 14:18
  SPEAKER_00: You should not be getting your research updates and in your con ed from Instagram.
• 14:22
  SPEAKER_00: And trust me, everybody here on this podcast, we teach via social media.
• 14:26
  SPEAKER_00: But hopefully you know that we also teach and we also publish and we also conduct research and we're actually really clinicians, right?
• 14:32
  SPEAKER_00: I think that's why people like us and trust us, but just realize that not everybody that you're learning from may be in that boat.
• 14:38
  SPEAKER_00: So be careful with these meta-analyses traps, right?
• 14:41
  SPEAKER_00: And thank you so much to these authors for publishing this editorial because now it's not just us saying that, right?
• 14:46
  SPEAKER_00: But you actually have quality researchers that are well respected in the field that throw this editorial out there because the world needs to see it.
• 14:54
  SPEAKER_00: Just because it's a meta-analysis doesn't mean it's quality.
• 14:57
  SPEAKER_00: So we have to be very careful with the interpretation of that.
• 15:00
  SPEAKER_00: Okay, awesome.
• 15:01
  SPEAKER_00: So good review.
• 15:02
  SPEAKER_00: Thanks, guys.
• 15:02
  SPEAKER_00: Appreciate everybody's input on that one.
• 15:04
  SPEAKER_00: And thank you so much for that.
• 15:05
  SPEAKER_00: You can check out the article in the uh show links on the website.
• 15:09
  SPEAKER_00: You really should because I do want you to read it.
• 15:11
  SPEAKER_00: It's quick, it's an editorial, so you can you can fill it in uh some small gaps in your schedule.
• 15:15
  SPEAKER_00: But check it out and be sure to subscribe so you can get future updates of our next episode.
• 15:20
  SPEAKER_00: See you on the next episode.