Latest / The Psychology Undergrad Podcast / Open Science & the Replication Crisis: p-Hacking, Preregistration & Reform
Transcript
- 0:00Welcome to the psychology undergrad. So for today's
- 0:03deep dive into the source material, we're unpacking
- 0:06a massive shockwave that hit the field of psychology.
- 0:09Oh, it was a total earthquake for the discipline.
- 0:12Yeah, exactly. We're talking about the replicability
- 0:14crisis and, you know, looking at how the scientific
- 0:17community is actually trying to clean up its
- 0:19act. It's a huge topic and honestly a really
- 0:22important one if you're navigating these courses
- 0:24right now. Right. And to kick things off, there's
- 0:26this fascinating study. from the source text
- 0:29by Simone Schnall and her colleagues. This came
- 0:32out in 2008. Oh, the hand washing one. Yes, the
- 0:35hand washing study. So they basically claimed
- 0:37that just washing your hands actually leads people
- 0:40to view moral transgressions as less wrong. Which
- 0:44is such a wild finding when you first hear it.
- 0:46It is. But it also, it perfectly mirrors these
- 0:50deeply rooted religious concepts. Like washing
- 0:53away your sins or baptism. Right, this idea of
- 0:55a physical act of cleansing being linked to a
- 0:58psychological feeling of moral purity. Exactly.
- 1:01You literally wash your hands and suddenly your
- 1:03brain goes, OK, I'm clean. Maybe I can forgive
- 1:05that person for doing something bad. We even
- 1:07say things like having blood on your hands. It
- 1:09makes total sense intuitively. And that study
- 1:12got massive attention. It was everywhere. But
- 1:15here is the plot twist. A few years later, this
- 1:19massive collaborative effort called the Many
- 1:22Labs Replication Project comes along. Yep. And
- 1:25they tried to reproduce this exact same study
- 1:28following all the original steps. and they completely
- 1:31failed. The effect just completely vanished.
- 1:33Totally gone. Hand washing did absolutely nothing
- 1:36to change moral judgments. You know, it's funny
- 1:39because hearing that a famous study failed to
- 1:41replicate, it can feel like a disaster. Like,
- 1:44wait, is everything we're learning wrong? Yeah,
- 1:46that's exactly how I felt when I first read about
- 1:48this. But it actually perfectly illustrates the
- 1:50self -correcting nature of science. That's kind
- 1:52of our core theme today. What happens when science
- 1:55fails to repeat itself, and how do we fix it?
- 2:00Right, because it wasn't just this one hand -washing
- 2:02study. We have to look at the sheer global scale
- 2:04of this problem. It was systemic. Which brings
- 2:08us to the Reproducibility Project. This was published
- 2:11by Arts and Colleagues in 2015, and the stats
- 2:14are just mind -boggling. Oh, they really are.
- 2:16So you had 270 psychologists from around the
- 2:20world teaming up, and their mission was to retest
- 2:23100 previously published psychological experiments.
- 2:26Just to see if the original findings would actually
- 2:28hold up. Exactly. This is the aha. data moment.
- 2:32Out of those original 100 studies, 97 of them
- 2:35claimed to have statistically significant effects.
- 2:38But when they ran them again? When they ran them
- 2:40again, depending on the metric, only about 36
- 2:43to 39 of the replications yielded significant
- 2:46results. Our textbook infographic puts it at
- 2:4939. Wow. From 97 successful studies down to 39.
- 2:54And it gets worse. Even in those 39 studies that
- 2:57did replicate, the effect sizes were on average
- 3:00cut in half. So the effects were way weaker than
- 3:02originally reported. Right. So I have to push
- 3:04back here for a second because if you're listening
- 3:06to this, the immediate instinct is to panic.
- 3:08If less than half of these studies hold up, does
- 3:10that mean all these psychology findings are just
- 3:12fake? That is the big question. Like, is this
- 3:15like trying to bake a cake with a vague recipe
- 3:18from a blog? And when it inevitably collapses,
- 3:21you blame the flour instead of realizing you
- 3:23just had different ovens. That baking analogy
- 3:25is actually perfect, because a failure to replicate
- 3:28doesn't automatically mean the original study
- 3:30was bogus or fake. OK, so it could just be the
- 3:33oven. Exactly. In science, we're dealing with
- 3:36different populations, different moderating variables,
- 3:38or differences in statistical power. Okay, wait,
- 3:41let's break that down for our hypothetical exam.
- 3:43What's a moderating variable in this context?
- 3:46So, a moderating variable is a contextual factor
- 3:49that changes the relationship between two things.
- 3:51Like time or culture? Yeah. Exactly. If the original
- 3:55study tested college students in California in
- 3:571998 and the replication tested, I don't know,
- 4:00retirees in Japan in 2015, you might get totally
- 4:03different results. Because human behavior is
- 4:06messy and context dependent. Right. It's not
- 4:09chemistry. But you also mentioned statistical
- 4:11power, which is huge here. Yeah. What does statistical
- 4:15power actually mean? Because it sounds like a
- 4:18superhero ability. It basically means whether
- 4:20your study has a large enough sample size so
- 4:23your results aren't just a random fluke. Ah,
- 4:25okay. Think about flipping a coin. If you flip
- 4:28a coin three times, you could easily get three
- 4:30heads in a row just by chance. Right. But if
- 4:32you flip it 3 ,000 times, you're going to get
- 4:34a 50 -50 split. Exactly. A lot of older psychology
- 4:38studies had really low statistical power. They
- 4:42tested small groups, so their findings might
- 4:44have just been the equivalent of getting three
- 4:46heads in a row. And when the replication teams
- 4:48tested larger groups, that random fluke just
- 4:51disappeared. Yep. The variants vanished. OK.
- 4:54So some of it is natural scientific. variance.
- 4:56But normal variance doesn't explain a drop from
- 4:5997 successful studies to 39. No it doesn't. There
- 5:03has to be a darker alternative at play. And there
- 5:05is. A lot of researchers point to systemic publication
- 5:09bias. The shiny sexy results. Exactly. Academic
- 5:12journals heavily reward surprising counterintuitive
- 5:15findings. A study saying washing your hands makes
- 5:17you judge people less gets you on the evening
- 5:20news. It gets you tenure. Right. But if you run
- 5:22a rigorous study that just proves a cool theory
- 5:25is wrong, journals historically didn't want to
- 5:28publish it. Which leads to the file drawer problem.
- 5:30Yes. You find no connection between two variables,
- 5:33so you just shove the data into a file drawer
- 5:35and it never sees the light of day. Which creates
- 5:38this immense pressure to find a significant result.
- 5:41Because your career depends on it. And that brings
- 5:44us to the core of the crisis. Questionable research
- 5:48practices. or QRPs. These are the big ones to
- 5:51know for sure. Yeah. If we stick to the morality
- 5:54theme from the hand washing study, these are
- 5:56basically the moral failings, the sins of research.
- 6:00I like that framing. Let's review and define
- 6:02them so the listener can easily remember them.
- 6:05Sin number one, selective deletion of outliers.
- 6:08Also known as cherry picking. Right. So you collect
- 6:11data. The trend isn't really there, but you have
- 6:13a few extreme data points that ruin the trend
- 6:16you wanted. So you just quietly discard those
- 6:19specific data points to artificially inflate
- 6:21the relationship. You only report what supports
- 6:24your hypothesis. Okay, here is my memory aid
- 6:27for cherry picking. It's like taking a family
- 6:30photo and photoshopping out the cousin who's
- 6:32picking his nose. just so you can claim you have
- 6:34a flawless family. That is exactly what it is.
- 6:37You aren't fabricating the people who are there,
- 6:39but you're distorting reality to fit your narrative.
- 6:42Exactly. But what if you cherry pick and your
- 6:45data still doesn't work? That brings us to sin
- 6:47number two, arcing. H -A -R -K, coined by Kerr
- 6:52in 1998. Right, hypothesizing after the results
- 6:55are known. This one is insidious because it attacks
- 6:58the whole foundational logic of science. Because
- 7:00you're supposed to predict things beforehand,
- 7:02right? Yes. Science relies on an a priori hypothesis.
- 7:07You state what will happen, then you test it.
- 7:10But our king is when you collect data, sit through
- 7:12it, find a random relationship, and then write
- 7:15your paper claiming you predicted it all along.
- 7:17Right. The analogy for this one, this is like
- 7:19shooting an arrow into a blank wall and then
- 7:21painting the bullseye around it so you look like
- 7:23a genius. Which completely destroys falsifiability.
- 7:26Because you can never be wrong. Exactly. If you
- 7:29wait to see the data to make your prediction,
- 7:32you remove the risk of failure, so you aren't
- 7:34really doing science. That makes total sense.
- 7:37Okay, sin number three, which is probably the
- 7:38most famous, p -hacking. documented by head and
- 7:42colleagues in 2015. P -hacking is huge. So real
- 7:45quick, to demystify the p -value. In psychology,
- 7:48the standard is a p -value of .05, right? Meaning
- 7:51there's less than a 5 % chance your results are
- 7:54just dumb luck. Right, if you're under .05, it's
- 7:57statistically significant. But the problem is
- 7:59when the math spits out, like at 0 .06. You are
- 8:03agonizingly close to being published. So what
- 8:05do researchers do? They look at that .06 and
- 8:08decide to tweak the parameters. They might say,
- 8:10let's recruit 10 more people and see if it drops.
- 8:12And if it drops to .055, they add five more people.
- 8:16Exactly. They keep testing, running the math
- 8:18over and over until it slips under .05, and then
- 8:22they stop testing. Oh, wow. So my memory aid
- 8:24here, it's like looking at your poker hand and
- 8:26then deciding how many cards are allowed in the
- 8:28deck so you can guarantee a win. You're manipulating
- 8:30the test mechanics until you win. But you know,
- 8:33researchers probably convince themselves they're
- 8:35just massaging the data. Yeah, they rationalize
- 8:37it. But then there's a line they cross into something
- 8:40way darker. Sin number four, outright fraud.
- 8:45Yeah, this is the extreme moral failing. It's
- 8:49beyond just a questionable practice. The textbook
- 8:52brings up Dietrich Staple, right? Right. Staple
- 8:54didn't just tweak his data or paint bullseyes
- 8:57around arrows. He completely fabricated data
- 9:00out of thin air. That's just wild. He generated
- 9:03entire spreadsheets for experiments that never
- 9:05even happened. So that isn't a questionable practice.
- 9:08That's just outright lying. It is, and it does
- 9:10profound damage to the integrity of the whole
- 9:12discipline, which is why the field needed a way
- 9:14to catch this stuff. Enter the replication index.
- 9:17Yes. Created by Ulrich Schimek in 2014, it acts
- 9:20as a statistical doping test. A doping test for
- 9:24scientists. I love that. But wait, how does it
- 9:27actually catch them? It goes back to those p
- 9:29-values. If an effect is real, you should see
- 9:32a natural spread of p -values across a researcher's
- 9:35studies, some at .01, some at .001. But Shimmick's
- 9:39index looks at a researcher's entire body of
- 9:42work. If dozens of their studies miraculously
- 9:45have p -values of .048 or .049... Just barely
- 9:50scraping under the threshold. Exactly. The math
- 9:53screams that it didn't happen naturally. It's
- 9:56statistically impossible without human manipulation.
- 9:58So it flags them for cheating the system. Yep.
- 10:01OK, so reviewing all these sins, cherry picking,
- 10:04arcing, packing, outright fraud, it feels pretty
- 10:09bleak. You can definitely feel that way. But
- 10:10this crisis doesn't mean science is broken. It
- 10:13means it's fixing itself. So let's talk about
- 10:15the path to redemption. The structural solutions.
- 10:18Right. How do we achieve salvation from these
- 10:20sins? The textbook lists a few big remedies to
- 10:23increase rigor. First off, we need enough statistical
- 10:25power. No more tiny underpowered studies. Bigger
- 10:29samples, clearer data. Second, publishing null
- 10:31findings alongside significant ones. Kill the
- 10:34file drawer problem. Exactly. If a massive study
- 10:36finds no effect, we need to know that just as
- 10:39much as if it did. And third, researchers have
- 10:41to describe their designs in exhausting detail.
- 10:43Which goes back to the baking analogy. You need
- 10:46to know the exact oven temperature and brand
- 10:48of flour so you can replicate it perfectly. And
- 10:52fourth, journals actually publishing high quality
- 10:54replications like Brandt and Collies talked about
- 10:57in 2014. We have to reward the people doing the
- 11:00vital work of verifying old data. Yes. And all
- 11:04of this is part of the open science movement,
- 11:06which is just huge. It's a massive cultural shift
- 11:09for the field. We have the Center for Open Science
- 11:12and the Toppy Guidelines. The transparency and
- 11:15openness promotion guidelines. Which have been
- 11:16adopted by over 500 journals. And 50 orgs. The
- 11:20U .S. National Science Foundation. The Canadian
- 11:23Tri -Council. Yeah. It's everywhere. It really
- 11:25is. But my favorite part is how they're implementing
- 11:27it. Journals like Psychological Science are issuing
- 11:30digital badges. It's essentially the gamification
- 11:33of scientific integrity. It's totally like earning
- 11:35scout badges for moral scientific behavior. Like
- 11:38you get a badge for sharing your raw data, another
- 11:40one for sharing your materials. But the most
- 11:42important one is the pre -registration badge.
- 11:45Yes. Walk us through why pre -registration is
- 11:48the ultimate antidote here. So... When you pre
- 11:51-register a study, you lock in your hypothesis
- 11:54and your exact mathematical analysis plan before
- 11:57you touch the data. You post it publicly. Exactly.
- 12:00You put it on a public server before you recruit
- 12:02a single participant. Which means it becomes
- 12:04literally impossible to ARC. Right. You can't
- 12:07paint the bullseye around the arrow after the
- 12:09fact because you already declared where the bullseye
- 12:11is. And it just shuts down packing too. Completely.
- 12:14You can't endlessly add participants because
- 12:16your pre -registration explicitly states, I will
- 12:19stop at exactly 200 participants. Wow. So it
- 12:23just removes the temptation to sin by making
- 12:25everything radically transparent. It does. It
- 12:28forces the scientific process into the light.
- 12:30OK. So summarizing this study session for you,
- 12:33the listener, why does all this matter? That's
- 12:35the key takeaway. Understanding the replicability
- 12:38crisis doesn't mean you should mistrust science.
- 12:41It just means you know how the sausage is made,
- 12:44which makes you a much smarter consumer of headlines.
- 12:47Right. When you see a flashy headline, you know
- 12:49what questions to ask. Exactly. Was it pre -registered?
- 12:53Did they have good statistical power? Has it
- 12:55been replicated? Science is actively fixing its
- 12:58own blind spots. Which leads to a final, pretty
- 13:01provocative thought to chew on. Oh, lay it on
- 13:04us. If we are moving into this era of radical
- 13:07openness, where you get badges for being totally
- 13:10transparent, what happens to the pressure to
- 13:12publish only exciting results? That's a great
- 13:15question. Right. Like if a scientist pre -registers
- 13:18a massive, expensive study and finds absolutely
- 13:21nothing, will our culture finally learn to celebrate
- 13:25that null result as a triumph of truth? Wow.
- 13:28Will we actually applaud the scientists who prove
- 13:31the emperor has no clothes? I guess we'll have
- 13:33to wait and see. It's definitely something to
- 13:34think about. Well, thank you for joining this
- 13:36deep dive into the course materials with us here
- 13:38on The Psychology Undergrad. Catch you next time.