Latest / The Psychology Undergrad Podcast / Quasi-Experimental Design: Research Without Random Assignment
Transcript
- 0:00Welcome to the Psychology undergrad. Hey, everyone.
- 0:02It is great to be back. Yeah, pull up a chair,
- 0:04grab your coffee or, you know, your energy drinks.
- 0:07Definitely energy drinks today. Right. So you're
- 0:10sitting right here in the study group with us.
- 0:12And today we are tackling the beast that is Chapter
- 0:1511. Oh, yeah. The methodology notes. Yep. the
- 0:19messy, chaotic, completely fascinating world
- 0:22of quasi -experimental research? Because, like,
- 0:26we have this mission today, right? We need to
- 0:28figure out how psychologists actually measure
- 0:31if an intervention works. Right, like a new therapy
- 0:34or a massive school program. Exactly, but how
- 0:37they measure it out in the messy real world,
- 0:40you know, where perfect sterile lab conditions
- 0:43just totally do not exist. Yeah, the real world
- 0:45doesn't really let you put people into perfectly
- 0:47matched, isolated little boxes. No, it definitely
- 0:50doesn't. It's entirely unpredictable. It's just
- 0:52full of these hidden underlying variables that
- 0:55can completely contaminate your data if you aren't
- 0:57paying attention. Okay, let's unpack this because
- 0:59I feel like when you You or I see a study in
- 1:02the news. We always just assume it's a true experiment,
- 1:04right? Oh, absolutely. Everyone does. But the
- 1:07notes highlight this specific category, the quasi
- 1:10-experimental stuff. And I mean, if I'm just
- 1:13breaking down the word quasi basically means
- 1:15resembling, right? Yeah, resembling or almost.
- 1:18So it looks like a traditional experiment, but
- 1:21it's fundamentally missing a puzzle piece. It
- 1:24is missing one very specific, highly critical
- 1:27ingredient. to get why it's missing, we should
- 1:32probably look at what it actually has first.
- 1:34OK, yeah, lay it on me. So in a true rigorous
- 1:38experiment, you manipulate an independent variable.
- 1:41and you randomly assign participants to conditions.
- 1:44Right, the gold standard. Exactly. Now, in a
- 1:46quasi -experiment, you still manipulate that
- 1:48independent variable, which is actually a massive
- 1:51advantage over simple observational research
- 1:53because it completely eliminates the directionality
- 1:56problem. Oh, right, knowing what caused what.
- 1:58Exactly. By introducing the intervention yourself,
- 2:01you know exactly what came first in the timeline.
- 2:03You lock down the cause and effect direction.
- 2:05If I give you a brand new study drug and then
- 2:08a month later your memory gets way better, I
- 2:10know the drug came first. Yes. The improved memory
- 2:12didn't travel back in time and caused me to hand
- 2:15you a pill. Right. The timeline is totally clear.
- 2:18But here's the critical catch. Quasi -experiments
- 2:21do not use random assignment. There it is. You
- 2:25aren't randomly sorting people into a treatment
- 2:27group and a control group. And because that sorting
- 2:31isn't random, you just cannot assume your two
- 2:33groups are perfectly equal before you start.
- 2:38neutralized the confounding variables. Exactly.
- 2:41All those hidden differences between people.
- 2:43So its internal validity basically sits right
- 2:46in this gray area between just observing correlations
- 2:48and a true lab experiment. I was actually thinking
- 2:51about this when I was reviewing my notes last
- 2:53night. I think the best way I can describe quasi
- 2:55experiment to you is that it's essentially the
- 2:57diet soda of research methods. The diet soda.
- 3:00OK, I like that. Right. Because it resembles
- 3:02the real thing. It gives you a taste of causality.
- 3:04You know, comes in a similar can. but it is fundamentally
- 3:07missing the sugar. And the sugar is random assignment.
- 3:11Yes. Without that sugar, you don't get that absolute
- 3:14certainty. You're just left with this artificial
- 3:16aftertaste of doubt. That is actually a great
- 3:19analogy. The sugar of random assignment is what
- 3:21gives an experiment its mathematical power. But
- 3:24what's fascinating here is why researchers ever
- 3:27settle for the diet soda in the first place.
- 3:29Right. If it leaves room for doubt, why not just
- 3:32demand true experiments for everything? Well,
- 3:34it comes down to field settings, you know, schools,
- 3:36hospitals, communities. In those places, random
- 3:39assignment is often physically impossible or
- 3:42honestly, morally questionable. Oh, like the
- 3:45ethics of it. Yeah. Imagine testing a new educational
- 3:48intervention that helps struggling kids learn
- 3:51to read. Morally, researchers have a really hard
- 3:54time forcing half of those kids into a control
- 3:56group. Where they intentionally get zero help.
- 3:59Right, intentionally denying them a life -altering
- 4:01intervention. just to satisfy a need for perfect
- 4:04data. Yeah, you can't just look a parent in the
- 4:06eye and say, sorry, your kid was randomly assigned
- 4:09to the control group so they don't get to learn
- 4:11to read. Exactly. That crosses a major ethical
- 4:13line. So they use quasi -experimental designs
- 4:16as a necessary compromise. Which brings up a
- 4:19really interesting problem from the textbook.
- 4:22If we can't randomly assign kids to get the reading
- 4:24program, we have to test it on existing groups.
- 4:27The non -equivalent groups design. Yes. It's
- 4:30a between -subject but without random assignment.
- 4:34The book uses this example of teaching fractions
- 4:37to third graders. Right, the classic classroom
- 4:40scenario. Yeah, so a school wants to test a new
- 4:42math curriculum. They take Mrs. Williams's class,
- 4:45she uses the new method. Down the hall, Mr. Jones's
- 4:49class acts as the control. So they just compare
- 4:51math scores at the end of the year. Right. But
- 4:54wait, if they are both third grade classes at
- 4:56the exact same school, aren't they basically
- 4:58identical? Like, why does the book call them
- 5:00nonequivalent? They're all just eight -year -olds
- 5:02learning fractions. Well, they appear equivalent
- 5:04on the surface. But without random assignment,
- 5:06hidden differences are totally baked into those
- 5:09classrooms from day one. Oh, like how they get
- 5:11assigned to the classes. Exactly. Think about
- 5:14it. It's rarely a blind lottery. Parents who
- 5:17are super involved might specifically lobby the
- 5:20principal to put their high -achieving kid in
- 5:23Mrs. Williams's class because she has a great
- 5:26reputation. Oh, wow. So the parents who show
- 5:28up to the PTA and hire tutors are pulling strings.
- 5:32Yeah. Or on the administrative side, if Mr. Jones
- 5:35is known as this strict disciplinarian, the principal
- 5:38might dump all the kids with behavioral challenges
- 5:41into his room to keep order. Which completely
- 5:45ruins the certainty. If Miss Williams's class
- 5:47crushes the test, we have no way to prove that
- 5:50the new method worked or if she just had all
- 5:52the motivated kids. Right. The baseline motivation,
- 5:55the teaching style, those are massive confounding
- 5:58variables. Now, researchers do try to mitigate
- 6:00this through matching. Ratching. Yeah, they might
- 6:02try to find two teachers of the same age or make
- 6:05sure both classrooms have similar past test scores.
- 6:08But without true random assignment, you can never
- 6:10completely eliminate the confounds. You just
- 6:13can't measure everything. So it's still diet
- 6:14soda. Still diet soda. So if comparing two classrooms
- 6:17is ruined by hidden variables, it makes sense
- 6:20that they try a different angle. What if we just
- 6:22look at one single group and measure them before
- 6:25and after a treatment? Ah, the pre -test, post
- 6:28-test design. Yes, the second big design. You
- 6:31measure the dependent variable twice. Like, a
- 6:33school does a massive anti -drug education program.
- 6:37Week one, they survey That's a pre -test. Right.
- 6:40Week two, they do the program. Week three, they
- 6:42survey again. The post -test. If attitudes are
- 6:45more negative toward drugs, boom, massive success.
- 6:48Well, our immediate instinct is to say yes. But
- 6:51the sources wave a massive red flag here. A simple
- 6:55pre -test post -test is incredibly vulnerable
- 6:58to these four illusions. They trick our brains
- 7:00into thinking a treatment worked. Alternative
- 7:03explanations. Yeah. And to help you remember
- 7:04these, I came up with a mnemonic. HMRS. Okay,
- 7:08let's hear it. Picture a heavy mutineering rogue
- 7:11ship. It looks like it's carrying your data to
- 7:13the truth, but it's actually dragging your internal
- 7:15validity to the bottom of the ocean. A heavy
- 7:18mutineering rogue ship. I like it. History, maturation,
- 7:22regression, spontaneous remission. Let's start
- 7:25with the H history. So history here means significant
- 7:29outside events that happen between your pre -test
- 7:31and post -test, events that have nothing to do
- 7:34with your intervention. Right. Going back to
- 7:36the anti -drug program, what if during week two,
- 7:40a hugely popular celebrity tragically dies of
- 7:43a drug overdose and it's all over social media?
- 7:46every teenager would be talking about it. The
- 7:48tragedy of it would naturally make them hate
- 7:50drugs more. Exactly. So in week three, the scores
- 7:53improved, but it was driven entirely by the historical
- 7:56event, not the school's program. Wow, yeah. OK,
- 7:59next on the sinking ship is M for maturation.
- 8:02This one is just natural growth, right? Yeah,
- 8:04human beings aren't static. If an anti -drug
- 8:06curriculum takes a full year, you're testing
- 8:08seventh graders than eighth graders. Adolescent
- 8:10brains are actively developing. They just naturally
- 8:12get better reasoning skills. Exactly. They get
- 8:14less impulsive simply due to biological maturation.
- 8:17That natural growth might be the only reason
- 8:20scores improved. OK, then we hit the R, which
- 8:22is regression to the mean. This one is a total
- 8:26statistical mind bender. It really is. It's the
- 8:29fact that if someone scores at an absolute extreme,
- 8:32their next score will naturally drift back toward
- 8:35their average. The book has that great bowler
- 8:37example. Right. Imagine a bowler who averages
- 8:39150. One night, they bowl a 220. That is an extreme
- 8:44outlying score for them. So statistically, what's
- 8:47guaranteed to happen next game? They're almost
- 8:49certainly going to score lower. They regress
- 8:52toward their normal 150. It's like statistical
- 8:54gravity. But how does that ruin a study? Well,
- 8:57Imagine testing a math tutoring program, but
- 9:00you only select the 10 students who scored the
- 9:03absolute lowest on a placement test. Oh, I see.
- 9:06Even if the tutoring is useless, regression to
- 9:08the mean basically guarantees those extreme low
- 9:11scores will bounce back up a bit by pure chance
- 9:14on the next test. And the researcher takes the
- 9:16credit. Exactly. Which brings us to a spontaneous
- 9:19remission. This is when problems just naturally
- 9:21improve over time without any treatment. And
- 9:24I want to tie this to morality for a second because
- 9:26it really gets into how we think. The source
- 9:29mentions the common cold. It's like if you pray
- 9:31for a cold to go away or you eat a bowl of chicken
- 9:34soup. If a hundred sick people eat soup or pray
- 9:38and feel better in a week, was it the intervention
- 9:41or just time? The viral infection just ran its
- 9:44course. Right. But human nature desperately wants
- 9:47to give credit to the soup or the prayer. We
- 9:50want it to be the hero. We look for causality
- 9:53everywhere. Yeah. And morally, think about the
- 9:55implications. If you take credit for natural
- 9:58healing, aren't you essentially taking credit
- 10:00for spontaneous remission? We really are. And
- 10:03the data on this is striking. Posternak and Miller,
- 10:05in 2001, they looked at treatments for depression.
- 10:08Oh, the weightless study? Yes. They isolated
- 10:11data from weightless control groups, people suffering
- 10:14from depression who hadn't received a single
- 10:16therapy session yet. Just waiting for treatment.
- 10:18Exactly. And those completely untreated participants
- 10:20saw their symptoms improve by 10 % to 15 % before
- 10:24they got any intervention. 10 % to 15 % just
- 10:26by existing and letting time pass. So if a flawed
- 10:30pre -test, post -test study shows a 12 % improvement,
- 10:34They haven't cured anything. They just watched
- 10:36spontaneous remission happen. Here's where it
- 10:38gets really interesting. This exact vulnerability
- 10:42triggered a massive crisis in clinical psychology.
- 10:45The Hans Eysenck article from 1952. Oh, it was
- 10:48an intellectual earthquake for the field. Right,
- 10:50because early clinical psychology relied on these
- 10:53pre -test, post -test designs. Eysenck looked
- 10:55at 24 studies showing two -thirds of therapy
- 10:58patients improved. Looks like a victory. Going
- 11:01a little deeper. Yeah, he compared it to archival
- 11:03data from state hospitals and insurance records
- 11:05for patients who got zero. And they recovered
- 11:09at the exact same rate. Two -thirds spontaneously
- 11:12went into remission without any therapy at all.
- 11:15So I have to ask, was Isinc calling cytotherapy
- 11:18a total scam? Like, was he saying it was fraud?
- 11:21This raises an important question about scientific
- 11:23criticism. He didn't say it was ineffective.
- 11:25He said there was no valid evidence that it was
- 11:28effective. Ah, huge distinction. It's everything.
- 11:31He was laying down a moral challenge. It is profoundly
- 11:34unethical to charge suffering people for treatments
- 11:37that haven't been rigorously proven by true experiments.
- 11:40He was demanding the sugar. No more diet soda,
- 11:43pre -test, post -test stuff. Exactly. And to
- 11:46the field's credit, they answered his challenge.
- 11:48By 1980, Smith, Glass, and Miller summarized
- 11:52hundreds of true experiments with random assignment.
- 11:55So they eliminated spontaneous remission. Yes.
- 11:58And they proved psychotherapy is highly effective.
- 12:0080 % of treatment participants improved more
- 12:03than the average control. That is an amazing
- 12:05redemption arc. But it brings us back to the
- 12:07dilemma. If we can't do a true experiment, how
- 12:10do we fix the quasi experiment? Well, there are
- 12:12a couple solutions. First is the interrupted
- 12:14time series design. Instead of just two measurements,
- 12:17you take a long sequence of measurements before
- 12:19and after the treatment. Like the Cook and Campbell
- 12:21factory study in 1979, they reduced work shifts
- 12:25from 10 hours to eight hours. Yeah, and they
- 12:27didn't just measure productivity the week before
- 12:29and after. They tracked it long term. Exactly.
- 12:33The notes have that graph example of college
- 12:35absences. A professor takes public attendance
- 12:37to reduce skipping. If you just measure week
- 12:40seven and week eight, absences might drop from
- 12:42eight down to three. Boom, victory. But if you
- 12:45use an interrupted time series. and track all
- 12:4814 weeks, you might see absences naturally bounce
- 12:51between three and eight every single week anyway.
- 12:54The analogy I love for this is it's the difference
- 12:56between looking at two isolated snapshots versus
- 12:59watching a whole continuous movie. Oh, that's
- 13:02really good. Snapshots give you zero context.
- 13:04But the movie lets you see if the intervention
- 13:06actually shifted the plot or if it was just a
- 13:08random blip. The context is vital. Now, the second
- 13:11solution is combination designs. Merging non
- 13:14-equivalent groups with pre -test, post -test.
- 13:16OK, so with the anti -drug program, instead of
- 13:18one school, you use two similar schools. Right.
- 13:22You pre -test both. School A gets the program.
- 13:24School B gets nothing. Then post -test both.
- 13:27So if it's just history or maturation like that
- 13:30celebrity overdose, both schools change equally,
- 13:33right? Exactly. The baseline shifts for everyone.
- 13:35So if school A changes more, you likely have
- 13:37a real effect. It's still not perfect, though.
- 13:40What if a student overdose happens at only one
- 13:42school? Right. Local history is still a threat.
- 13:45Without random assignment, confounds always exist.
- 13:48But it's vastly superior to relying on a single
- 13:51group. So what does this all mean for you as
- 13:53you're listening to this? Why geek out over methodology?
- 13:57It's because quasi -experiments are this messy
- 13:59vital bridge between the perfect lab and the
- 14:03chaotic real world. If we connect this to the
- 14:05bigger picture, As a consumer of information,
- 14:08you have to constantly question headlines. Yeah,
- 14:10when you see X, here's Y, you need to ask, was
- 14:12this a true experiment or just a pre -test, post
- 14:15-test illusion? Did they prove causality or just
- 14:17observe spontaneous remission? I want to leave
- 14:20you with a final thought to mull over. Think
- 14:22about the multi -billion dollar wellness industry
- 14:24today, from crystal healing to specialized detox
- 14:28teas. Oh, wow. How much of their proof relies
- 14:31entirely on spontaneous remission and regression
- 14:33to the mean? You know, you feel awful, you buy
- 14:36a detox tea, time passes, you naturally heal,
- 14:39and you credit the tea. Are we still making the
- 14:41exact same moral mistakes Eysen called out in
- 14:441952? Exactly. Keep that in mind next time you
- 14:48see an ad for a miracle cure. Thanks for joining
- 14:51our Sobe session on the Psychology undergrad.
- 14:53See you next time.