Latest / The Psychology Undergrad Podcast / Standardized Testing Explained: How Formal Assessments Really Work
Transcript
- 0:00Welcome to the psychology undergrad. Grab a seat,
- 0:03pull up a chair to our study table. Yeah, welcome.
- 0:06Get comfortable because We've got a lot of ground
- 0:08to cover today. We really do. So if you're joining
- 0:10us, we are doing a deep dive into chapter 12
- 0:13of our educational psych textbook. It's the standardized
- 0:18and other formal assessments chapter. Right.
- 0:20The one everyone is kind of dreading for the
- 0:22upcoming exam. Exactly. But our mission here
- 0:25today isn't just to memorize a bunch of dry testing
- 0:29terms. We really want to understand the moral
- 0:33and psychological weight behind how society measures
- 0:36human minds. Yeah, because it's not just numbers
- 0:38on a page. I mean, we're unpacking the actual
- 0:40mechanics of testing, the high stakes accountability
- 0:43forced onto schools. Right, the literal math
- 0:45of the bell curve. And the really dark ethical
- 0:48dilemmas of bias and cheating that come out of
- 0:51it. And to do this without getting lost in all
- 0:53the academic jargon, I'm going to try to use
- 0:55common mental models and memory tricks. Which
- 0:58is so helpful, honestly, because usually when
- 1:00you get an x -ray for a broken arm, the doctor
- 1:02just like... points for a jagged white line on
- 1:04the screen and says, there it is. Yeah, it's
- 1:06totally binary. Right. The bone is broken or
- 1:09it isn't. The diagnosis is clean. The engineering
- 1:11is precise. But when society tries to take an
- 1:15X -ray of the human mind, when we try to measure
- 1:18intelligence or academic achievement, that diagnostic
- 1:22machine is just fundamentally murky. It is the
- 1:24absolute definition of diagnostic muddy waters.
- 1:27And I mean, the stakes could not be higher here.
- 1:30No, they really couldn't. We are talking about
- 1:33measurements that dictate school funding, property
- 1:37values, whether a teacher keeps their job. literal
- 1:41child's future. Yeah, it's intense. So let's
- 1:44start at the very foundation of this. Before
- 1:46we can even judge the ethics of testing, we have
- 1:49to understand the two fundamentally different
- 1:51ways a test can be graded. Right, the two main
- 1:53pillars. Exactly. Criterion reference tests and
- 1:57norm reference tests. So let's look at criterion
- 1:59reference testing first. This model is entirely
- 2:04about measuring performance against a specific
- 2:08fixed standard. That standard is the criterion.
- 2:11Right. A great real world example from the reading
- 2:13is the physical exam for newly hired firefighters
- 2:16in Massachusetts. Oh yeah, the ladder thing.
- 2:19Yeah. To get the job, a candidate has to physically
- 2:22carry a heavy hose, climb a ladder, and like
- 2:25simulate a rescue. You either have the physical
- 2:27strength to carry that hose up the ladder or
- 2:30you don't. Exactly. It doesn't matter how fast
- 2:31the candidate next to you did it. You just have
- 2:33to cross the bar. And that same logic applies
- 2:35to the classroom, right? Like a criterion reference
- 2:38test in fourth grade math just wants to know,
- 2:41can this specific child successfully add two
- 2:43fractions, yes or no? Right. The standard is
- 2:46fixed. But then you have the other pillar, which
- 2:48is norm -referenced standardized tests. The tricky
- 2:51one. Yeah. These do not measure you against a
- 2:53fixed bar. They report your performance relative
- 2:56to everyone else. So if you get a score placing
- 2:58you in the, say, 72nd percentile in reading.
- 3:01Which sounds like a C minus usually. Right. But
- 3:04it does not mean you got 72 % of the questions
- 3:07correct. It means you outperformed 72 % of the
- 3:10students in the norm group who took the test
- 3:12when it was being developed. OK. I have a mental
- 3:15model for this that I jotted down while studying.
- 3:17Oh, let's hear it. Think about the last time
- 3:19you were evaluated at a job. Were you judged
- 3:21on whether you hit your own personal sales quota?
- 3:24Right. Or were you judged on whether you build
- 3:26more hours than the person sitting in the cubicle
- 3:29next to you? That's a huge difference. It changes
- 3:31the entire psychology of an environment. So to
- 3:33remember this for the exam, criterion is the
- 3:37climb. Like, did you cross the bar? And norm
- 3:41is the neighbors. Did you beat the guy next to
- 3:43you? Criterion is the climb. Norm is the neighbors.
- 3:45I like that a lot. That's highly effective. And
- 3:48if we... pull back and look at the bigger picture.
- 3:52There is a really profound ethical implication
- 3:54built into the math of these two systems. Because
- 3:57one system inherently requires someone to fail.
- 3:59Precisely. Think about the morality baked into
- 4:01that grading structure. Norm reference grading
- 4:04by mathematical definition requires losers. Someone
- 4:08has to be at the bottom. Right. For a student
- 4:10to be celebrated in the 99th percentile, another
- 4:13student is mathematically forced into the first
- 4:15percentile. Even if the entire class works incredibly
- 4:19hard and learns a massive amount of material.
- 4:21Somebody still has to be the first percentile.
- 4:23That's kind of messed up. It is. A certain percentage
- 4:26is structurally doomed to remain at the bottom
- 4:28of the curve. Whereas criterion reference testing
- 4:31holds this ethical position that every single
- 4:35student can theoretically succeed. Because if
- 4:37the standard is just knowing how to add fractions,
- 4:41then literally 100 % of the class can cross that
- 4:44bar. Everyone can make the climb. Exactly. Which
- 4:47leads us to a really fascinating historical pivot
- 4:49in the textbook. If the ethical goal of a society
- 4:52is for everyone to succeed, how does the government
- 4:54actually enforce that? And in the U .S., that
- 4:56attempt came in the form of No Child Left Behind,
- 4:59NCLB. Yep. The controversial one. Very. This
- 5:03is where the shift toward criterion reference
- 5:04testing became federal law. Under this act, states
- 5:08were forced to define exactly what proficient
- 5:10meant and they set this staggering mandate. It
- 5:13really was staggering. They mandated that 100
- 5:16% of students were expected to reach proficiency
- 5:18by the 2013 to 2014 school year. Every single
- 5:22child. Every child. And schools were judged heavily
- 5:25on whether they were making what the law called
- 5:27adequate yearly progress, or AYP, toward that
- 5:30absolute 100 % goal. But, and I know you notice
- 5:33this in the reading too, There's a glaring mathematical
- 5:36paradox in how this was enforced. Yes. Oh my
- 5:39gosh. Exhibit 15 in the textbook. I have to point
- 5:41this out because it's mathematically and ethically
- 5:43wild. Let's look at two hypothetical schools
- 5:46from the AYP data. School A and School C. OK.
- 5:50Lay it out. So School A is this under -resourced
- 5:53school starting way behind the proficiency line.
- 5:56To hit that 100 % goal by 2014, the government
- 5:59required them to improve at a rate of 6 % every
- 6:02single year. Which is a ma - year -over -year
- 6:05leap in education. Huge. Meanwhile, school C,
- 6:08which is an already wealthy, high -performing
- 6:10school, only had to improve by 1 % a year. So,
- 6:13I mean, isn't it a moral failing to demand the
- 6:17steepest, most grueling climb from the most under
- 6:20-resourced schools? Especially when they were
- 6:22threatening those struggling schools with severe
- 6:24sanctions like literal state takeovers or firing
- 6:28the whole staff if they failed to make that 6
- 6:30% jump. It sounds totally backwards. It sounds
- 6:33like you're demanding a sprint up a mountain
- 6:34from the people carrying the heaviest backpacks.
- 6:37Exactly. But to understand why the law was written
- 6:39that way, we have to look at the mechanics of
- 6:42the subgroup's rule. And this is a classic case
- 6:45of good moral intentions, just practically backfiring.
- 6:48OK, so unpack the subgroup's rule for us. How
- 6:51does that work? So. Historically, a school principal
- 6:53could point to their data and say, look, our
- 6:56average reading score is fantastic. But that
- 6:58overall average was acting as a rug. Oh, they
- 7:01were sweeping kids under it. Exactly. They were
- 7:03sweeping their most vulnerable kids under it.
- 7:06The school's special education students or minority
- 7:08students or low income students might be completely
- 7:11failing. But because the majority of the school
- 7:14is doing well. Those struggling students were
- 7:16just buried in the average. That's awful. It
- 7:19was. So the architects of No Child Left Behind
- 7:22mandated that every single demographic subgroup
- 7:25in a school had to independently meet AYP. OK,
- 7:29ethically, I mean, that makes complete sense.
- 7:31You can't hide your most vulnerable kids in the
- 7:33math anymore. Right. The intention was to force
- 7:35visibility. But the practical application was
- 7:39devastating. Because of how the math played out.
- 7:41Yeah. Let's look at the statistics. A large diverse
- 7:44urban school naturally has many different subgroups.
- 7:47You have different ethnicities, English language
- 7:48learners, varying income levels. To pass AYP,
- 7:52that diverse school might have to hit the target
- 7:54across like eight different subgroups. And if
- 7:57just one subgroup misses the mark, the entire
- 7:59school is labeled as failing. So it's basically
- 8:01like forcing the diverse school to flip a coin
- 8:04eight times and demanding they get heads every
- 8:06single time. Yes. While a small wealthy racially
- 8:10homogenous school might only have one or two
- 8:13subgroups. They only have to flip the coin twice
- 8:16So they were mathematically shielded from failing.
- 8:18Completely. The absolute finish line of AYP essentially
- 8:22punish the schools doing the hardest work simply
- 8:25because they serve more diverse populations.
- 8:27That is so deeply frustrating. So if demanding
- 8:31that everyone reach the exact same absolute finish
- 8:34line is mathematically punishing to diverse schools,
- 8:38what do we do? How do we measure success fairly?
- 8:41Right. How do you reward a teacher who takes
- 8:43a student from miles behind the starting line
- 8:45to just like inches behind it. Yeah, because
- 8:47that teacher did an amazing job. Well, that question
- 8:50sparked the development of value added, or growth
- 8:52models. Instead of looking at an absolute finish
- 8:54line, growth models track how much a student
- 8:56improves over the course of a single year, regardless
- 8:59of where they started in September. Okay, I have
- 9:01some hard numbers from the book to show why this
- 9:03matters. There's this landmark data out of Dallas,
- 9:05Texas. Oh, the Dallas study, yeah. Researchers
- 9:07tracked low -achieving fourth graders over a
- 9:10three -year period. Of the kids who are assigned
- 9:12highly effective teachers for all three years,
- 9:1590 % passed the seventh grade math test. 90 %?
- 9:19Yeah. That is a staggering success rate for students
- 9:22who started out labeled as low achieving. It's
- 9:24incredible. But wait for it. For the kids who
- 9:26are assigned to ineffective teachers for those
- 9:28three years, only 42 % passed. Wow. The teacher
- 9:31is the defining variable. So to keep our mental
- 9:34models going for the exam, proficiency is the
- 9:37snapshot, growth is the movie. I love that. Yeah,
- 9:41like, proficiency just takes a static picture
- 9:43of where you are on a random Tuesday in April.
- 9:46But growth watches the movie of how far you traveled
- 9:49over the year. And growth models allow us to
- 9:52see what researchers call school effects. You
- 9:55could have a school that looks like a total failure
- 9:57on paper because its absolute proficiency snapshot
- 10:00is low. Right. But when you look at the growth
- 10:02movie, you realize those teachers are performing
- 10:04absolute miracles. They are accelerating learning
- 10:08at incredible rates. It's just that their students
- 10:10started exceptionally far behind. So the math
- 10:13actually captures their effort. Yeah. Growth
- 10:15models try to calculate this by taking a student's
- 10:18past test scores. and mathematically estimating
- 10:21where they should be a year later. If the student
- 10:24exceeds that expectation, the teacher added value.
- 10:27But, and there's always, but with this stuff,
- 10:30there is a massive debate about using this growth
- 10:32data to hand out teacher bonuses or like fire
- 10:36people. Oh, a huge controversy. Because if you're
- 10:39calculating a teacher's value based on a classroom
- 10:41of just 25 kids, the sample size is incredibly
- 10:45small. Too small to be statistically reliable.
- 10:47Right. A few kids having a bad morning or a few
- 10:50lucky guesses creates statistical noise that
- 10:53can make a teacher look dramatically better or
- 10:56worse than they actually are. Which brings us
- 10:58to the underlying architecture of all these measurements.
- 11:00The math of the mind. Exactly. If we are going
- 11:03to judge human worth and distribute funding,
- 11:05and close schools based on numbers, we really
- 11:08need to understand how psychometricians actually
- 11:10crunch those numbers to define what normal is.
- 11:14We have to look at the statistical tools, mostly
- 11:16central tendency and variability. So when people
- 11:19talk about central tendency, they usually just
- 11:21mean the mean, right, the average, but the average
- 11:24can be incredibly deceptive. Oh, totally. Think
- 11:27about it like this. Imagine a 20 -point pop quiz
- 11:30in a small class. Nine students do amazingly
- 11:34well. They score 19s and 20s. But one student,
- 11:37maybe they are dealing with something really
- 11:39heavy at home, they completely bomb it and get
- 11:41a four. If you just calculate the mean, that
- 11:44single extreme outlier drags the whole class
- 11:48average down to around 17. Which creates a completely
- 11:52false narrative. If an administrator looks at
- 11:54that average, they assume the whole class is
- 11:56struggling to master the material. when in reality,
- 11:5990 % of the room mastered it perfectly. That
- 12:02is why the median is often a much truer, fairer
- 12:05middle. The median is literally just the middle
- 12:07score. If you line all 10 of those test scores
- 12:09up from lowest to highest, the median in that
- 12:12class is still a 19. Right, it doesn't get dragged
- 12:14down. It is completely resistant to that one
- 12:17extreme outlier. And once you establish where
- 12:19the true middle is, you need to know how spread
- 12:21out the rest of the scores are. That's variability.
- 12:24And the most vital tool for measuring variability
- 12:26is standard deviation. Standard deviation simply
- 12:30tells you, on average, how far away all the individual
- 12:33scores are from the middle. And if you graph
- 12:35that out for most human traits, you get the famous
- 12:38bill curve. Yes, the normal distribution. The
- 12:40vast majority of people clump together in the
- 12:42middle, which creates the high peak of the bell,
- 12:44and very few people fall at the extreme high
- 12:47or extreme low ends. Right. the tails. Exactly.
- 12:50So on a standard IQ test, the mean is set at
- 12:52100, and the standard deviation is 15. The laws
- 12:55of statistics dictate that about 68 % of all
- 12:58people will always fall within one standard deviation
- 13:01of the mean. So roughly 68 % of humanity scores
- 13:04between an 85 and a 115? Yep. And to make these
- 13:07curves easier to digest, test makers translate
- 13:10raw scores into standard scores. Like z scores?
- 13:13Right. A z score tells you exactly how many standard
- 13:15deviations you are from the middle. So the z
- 13:17score of positive two means you are two standard
- 13:20deviations above average. But having a negative
- 13:23z -score sounds terrible to parents. It really
- 13:25does. Yeah. So phygometricians created t -scores,
- 13:28which basically scale everything up so there
- 13:30are no negative numbers. The mean is just set
- 13:33at 50. Oh, that makes sense. They also used stenines,
- 13:36which chopped the bell curve into nine broad,
- 13:40easy -to -understand slices. But this actually
- 13:44brings us to a metric that is highly controversial.
- 13:47Great equivalent score. Yes. This is where statistical
- 13:50communication completely breaks down and causes
- 13:53real harm. OK, let's look at how this plays out
- 13:55in the real world. Say you have a fourth grade
- 13:58student named James. James takes his fourth grade
- 14:00standardized reading test, and the score report
- 14:03gets mailed home to his parents. Printed right
- 14:06there on the paper is a grade equivalent score
- 14:08of 6 .0. And any parent reading that is going
- 14:11to immediately think, my child is a genius. Exactly.
- 14:13They think, James is reading at a sixth grade
- 14:15level. He should be bumped up to do sixth grade
- 14:18work. But that is a fundamental misinterpretation
- 14:21of the math. James has not been tested on sixth
- 14:24grade vocabulary. He hasn't read sixth grade
- 14:26text. A 6 .0 just means that James aced his fourth
- 14:29grade test the exact same way an average sixth
- 14:32grader would have aced that exact same fourth
- 14:34grade test. It's just a developmental trend line
- 14:36extrapolated by the testing company. It is not
- 14:39a measure of actual capability, which raises
- 14:42a serious ethical question. It really does. Why
- 14:45do testing companies and school districts continue
- 14:48to use a metric that they know for its statistical
- 14:50fact? consistently misleads parents. Is it morally
- 14:54questionable to use a school that gives families
- 14:57a completely inflated, inaccurate sense of their
- 15:00child's achievement? I mean, when an assessment
- 15:02tool is designed in a way that creates mass illusion
- 15:04rather than clarity, it fails its primary purpose.
- 15:07Yeah. And that illusion of objectivity is what
- 15:10leads to the absolute darkest corners of standardized
- 15:13testing. When these flawed numbers are given
- 15:15the power to dictate funding, teacher livelihoods,
- 15:18and a student's ability to graduate, the Hence
- 15:20pressure just warps the entire system. Let's
- 15:23talk about bias, because this is huge for the
- 15:25exam. We like to think of standardized tests
- 15:28as this objective x -ray. But what if the x -ray
- 15:31machine is calibrated to only accurately read
- 15:35certain demographics? That's a great way to phrase
- 15:37it. The textbook provides a glaring example of
- 15:40this regarding cultural bias in vocabulary. There
- 15:43was a multiple choice question asking students
- 15:45to identify the meaning of the word field in
- 15:47the context of a sentence like, my dad's field
- 15:50is computer graphics. Right. Children of professionals,
- 15:54like lawyers, doctors, engineers, they grow up
- 15:56hearing the word field used abstractly all the
- 15:59time. Like a field of vision or field of study.
- 16:01Exactly. But working class children whose parents
- 16:04might be cashiers or mechanics, they may have
- 16:07only ever been exposed to the word field in its
- 16:09literal physical sense. Right, like a plowed
- 16:11field on a farm or a baseball field. Yeah. So
- 16:14the working class kid gets the question wrong.
- 16:17But did they lack reading comprehension? No.
- 16:19Not at all. They just lacked the cultural exposure
- 16:22to the way wealthy people use that specific word.
- 16:25The test is essentially punishing them for their
- 16:28socioeconomic class, all under the guise of measuring
- 16:31their cognitive ability. And this inherent bias
- 16:34shows up in the predictive accuracy of tests
- 16:36like the SMP, too. Statistically, the SAT overpredicts
- 16:41college success for Black and Latino students.
- 16:43Which is really concerning. It is. It means they
- 16:45often achieve lower freshman grades than the
- 16:48test guess they would. Conversely, it underpredicts
- 16:51for females who consistently achieve higher college
- 16:54grades than their where SAT scores suggest. So
- 16:57the measurement tool itself is fundamentally
- 16:58unstable across different demographics. Exactly.
- 17:01And it goes even deeper than just bad questions.
- 17:04The testing environment itself triggers psychological
- 17:06barriers. Let's look at stereotype threat. Oh,
- 17:09this concept is fascinating. The mechanics of
- 17:11it are literally mind blowing. If you take a
- 17:14marginalized group, say women taking an advanced
- 17:17math test, and you simply remind them of their
- 17:19gender right before the high stakes test. Like
- 17:21just having them fill out a demographic bubble
- 17:24sheet. Yes. Just checking that box causes their
- 17:28performance to drop. Because you have activated
- 17:30a psychological burden. Exactly. It's like their
- 17:33brain is suddenly forced to run a heavy background
- 17:36app. They are using so much cognitive energy
- 17:38worrying about confirming the negative societal
- 17:41stereotype that women are bad at math that it
- 17:44literally drains their working memory capacity.
- 17:46That is why they physically have less brainpower
- 17:49available to focus on the actual math problems
- 17:52in front of them. The anxiety of the test environment
- 17:54alters their cognitive capacity in real time.
- 17:57And just as the pressure alters the student,
- 18:00it alters the school environment itself through
- 18:02curricular narrowing. Oh, right. By the early
- 18:052000s. data showed schools were drastically reducing
- 18:08the time spent on art, science, and social studies.
- 18:11Because they weren't on the state tests. Exactly.
- 18:14Administrators were aligning their entire buildings
- 18:16to test prep. There's this one principal mentioned,
- 18:19Dr. Mucci, who forced even art and music teachers
- 18:22to drill math and reading skills. When you optimize
- 18:25a complex system entirely for a single metric,
- 18:28everything else withers. Which brings us to the
- 18:30ultimate, maybe the most tragic consequence of
- 18:33this high -stakes pressure cooker. Systemic cheating.
- 18:37Yeah, this part of the reading was intense. We
- 18:39aren't talking about a kid glancing at their
- 18:41neighbor's scantron. Because of the existential
- 18:43threat of no child left behind sanctions, otherwise
- 18:47ethical, dedicated educators were driven to extreme
- 18:51actions. You really have to understand the psychology
- 18:54of survival here. Teachers were facing the reality
- 18:57that if scores didn't rise, their school would
- 19:00be taken over by the state. and their colleagues
- 19:02would be fired. It's a huge threat. That level
- 19:04of threat creates massive cognitive dissonance.
- 19:07It led to teachers staying after hours to systematically
- 19:10erase and change student answers on test booklets.
- 19:14Administrators were giving unapproved extra time.
- 19:17Teachers were handing out actual test items as
- 19:19practice worksheets the day before the exam.
- 19:21It was completely compromising their morals.
- 19:23Yeah. An independent task force in Texas had
- 19:26to be mobilized just to investigate the sheer
- 19:28volume of tests and proprieties committed by
- 19:31school staff. It forces us to confront a terrifying
- 19:35reality about how we manage education. You have
- 19:38a system that was theoretically designed to measure
- 19:40truth, right? The objective reality of how much
- 19:43a student has learned. Right. But because the
- 19:45punishments attached to that truth are so existentially
- 19:48threatening, the system ends up incentivizing
- 19:51mass deception. The very teachers tasked with
- 19:55molding the morality of the next generation are
- 19:57pushed into rationalizing fraud just to keep
- 20:00their school doors open a system designed to
- 20:03measure truth that instead incentivizes lying.
- 20:06That is incredibly heavy, but it's vital to understand
- 20:09for the exam. Definitely. So let's quickly recap
- 20:11the ground we've covered today at the study table.
- 20:14We explored the fundamental difference between
- 20:15criterion -referenced grading. Measuring against
- 20:18a fixed standard making the climb. Right, and
- 20:20norm -referenced grading, which is grading on
- 20:22a curve against your neighbors. Then we examined
- 20:25the brutal math of adequate yearly progress.
- 20:27And how demanding absolute proficiency ironically
- 20:30punished the most diverse schools with the subgroup
- 20:32rules. Exactly. We looked at the mechanics of
- 20:35value -added growth models, measuring the distance
- 20:38traveled rather than the finish line. The movie
- 20:40versus the snapshot. Yep. We laid out the statistical
- 20:43foundations of the Bell curve, standard deviations,
- 20:46and why great equivalent scores are a deceptive
- 20:48vanity metric. Poor James and his 6 .0 reading
- 20:51score. Seriously. And finally, we confronted
- 20:55the psychological damage of testing, cultural
- 20:57bias in the questions, the cognitive drain of
- 21:00stereotype threat, and the systemic cheating
- 21:03born out of high stakes panic. As we wrap up
- 21:05this deep dive, I want to leave you with one
- 21:07final thought to mull over before the exam. Go
- 21:09for it. If our standardized tests are not the
- 21:12perfect clean x -rays we want them to be. If
- 21:15they are, in reality, culturally -bound snapshots
- 21:18of developed ability rather than fixed innate
- 21:20capacity, and they inherently force our schools
- 21:23into a state of narrow high -stress panic, what
- 21:26happens to a society that optimizes its entire
- 21:29educational system purely for a test rather than
- 21:33for the human being taking it? That is the multi
- 21:37-billion dollar question right there. Well, thank
- 21:39you for joining us on this deep dive study session
- 21:42for the Psychology Undergrad. See you next time.