Latest / Professor Peterson’s Psychology Classroom / Don't Trust the Dashboard: How Numbers Mislead in Substance-Use Prevention

Transcript

Don't Trust the Dashboard: How Numbers Mislead in Substance-Use Prevention · Professor Peterson’s Psychology Classroom

0:05
Welcome to Professor Peterson's Psychology Classroom,
0:09
connecting the science of human behavior to everyday life.
0:13
Here's Dr. Curtis Peterson.
0:17
Just a disclaimer, this podcast was created
0:21
using AI tools. With that being said, all content are
0:25
the responsibility of me, Dr. Curtis Peterson,
0:29
as I have reviewed and validated all of the content.
0:34
You know, you make decisions every day based on data. I mean,
0:38
think about it. At work, you're probably looking at performance
0:42
metrics or like quarterly reports, right? And even in
0:45
your personal life, maybe you're tracking your sleep score on your watch or just counting
0:48
your daily steps. Yeah, exactly. We all do it, right? You pull
0:52
up a dashboard, you see a hard number, And you
0:56
just naturally assume you're looking at the absolute truth. But how
1:00
often do you actually stop and question the quality or the context or just the
1:04
fundamental meaning of that data? Honestly,
1:08
almost never. It's— it is incredibly tempting
1:11
to treat numbers as objective reality. We crave that certainty,
1:15
you know, like a clean bar graph or a green checkmark on a
1:19
screen. Yeah, it feels so definitive. It does, but numbers are just representations
1:23
of reality, and they are based on very specific, completely subjective
1:28
decisions that someone made about, well, what to measure, how to
1:32
measure it, and what to just entirely ignore. Which is wild
1:35
when you really think about it. And that brings us to our deep dive
1:39
today. We are unpacking applied research and evaluation,
1:44
specifically looking at it in the realm of substance use prevention.
1:47
Yes. Now, before anyone assumes this is going to be like a dry
1:50
statistics lecture, let me clarify the mission here.
1:54
The goal today is not to turn you into a data scientist.
1:57
Thankfully. Right? No complex math today, I promise.
2:01
The actual goal is to develop a skill called evaluative
2:05
thinking. Which I have to say is an invaluable tool no matter
2:09
what field you are in. Evaluative thinking is really just the
2:12
disciplined habit of connecting claims to evidence.
2:16
Okay, let's unpack this because in complex human services like substance
2:20
use prevention, evaluative thinking isn't just a
2:23
nice-to-have, right? It is essential. Completely essential
2:27
because it means looking at a stated outcome and instead of just blindly
2:31
accepting it, asking, what alternative explanations might
2:36
exist here? Right. Like, you can't just look at a final binary outcome,
2:39
like whether a teenager used a substance or didn't. And declare a
2:43
whole program a success or a failure based on that one thing. No, you really
2:46
can't. And to understand why a simple binary outcome is so insufficient,
2:50
we have to look through a specific framework known as the Ecological Identity Model,
2:54
or EIM for short. EIM. Got it. So this model
2:58
argues that human behavior does not happen in a vacuum. If we
3:02
actually want to evaluate an intervention, we have to measure the entire ecology
3:06
surrounding that person. The whole environment. Exactly. The framework breaks
3:10
this down into several specific components, which are identity appraisal,
3:14
cross-pressure, relational safety, and perceived ecological cost.
3:18
Oh, okay. Those are some heavy terms. Let's try to ground
3:22
them in a real-world scenario so we can picture it. Imagine a 16-year-old
3:26
walking into a clinic for like a routine checkup, and they get handed a substance
3:30
use screening questionnaire. How do those 4 components
3:34
actually play out right there in the waiting room? Well, let's take them one by
3:37
one. So first, identity appraisal. That's really about how that teenager views
3:41
themselves in this specific clinical setting. Like how they think the clinic sees them.
3:45
Yeah, exactly. Do they feel like they're being seen as a patient
3:48
who deserves care, or do they feel like they are being categorized
3:52
as, you know, a problem kid or a potential addict? Oh, wow.
3:55
Yeah, that changes everything. And what was the second one? Cross-pressure?
3:59
Right, cross-pressure. This is the tension between different expectations.
4:03
The clinic is handing them a form asking for total honesty, but maybe
4:08
the teenager's parents are sitting 3 feet away. Right. Or maybe their peer group has
4:12
totally different norms about vaping or drinking. They are essentially
4:15
caught between 2 worlds. Yeah, that makes so much sense.
4:19
What about relational safety? That was the third one. That simply asks
4:23
whether the teenager trusts the person handing them the form. Do they actually believe
4:27
the nurse or the doctor is going to support them, or do they fear judgment?
4:31
Or even punitive action. Which leads right into the last one,
4:34
right? The perceived ecological cost? Yes, and this is the big
4:38
one. The teenager is actively calculating what they
4:41
stand to lose by participating honestly. Like, what are the actual consequences?
4:45
Exactly. If they admit on this piece of paper to smoking marijuana,
4:49
will the clinic tell their school? Are they going to get kicked off the basketball
4:52
team? Will they lose their parents' trust? So if a clinic
4:56
only measures the final behavior, like just the checked box on
5:00
the form, they are missing a massive, totally invisible ecosystem of
5:03
fear, trust, and social pressure. Precisely.
5:07
To evaluate if a program is actually working, we have to map out how
5:10
it is supposed to function within that complex ecology. And from
5:14
the source material, that mapping process is usually done using something
5:17
called a logic model, right? Yeah, it sounds
5:20
like terrible corporate jargon, I know, but a logic model is simply a
5:24
visual representation of how a program intends to create change. Like a
5:28
roadmap. Right. It forces you to articulate your inputs,
5:31
your activities, your outputs, and your outcomes. And I'll
5:35
tell you, the single most common trap in all of evaluation is confusing outputs with
5:39
outcomes. Oh, I see this happen everywhere, not just in healthcare.
5:43
Let's really define the difference here. Outputs are just the literal
5:46
things a program produced, right? Yes, just the stuff.
5:50
So, for example, a clinic's output might be that 400
5:54
students attended an educational assembly, or 85%
5:59
of patients were successfully screened, or 60 referrals
6:02
were handed out. Exactly. It's just counting the widgets you
6:06
manufactured. Right. Whereas an outcome is entirely different.
6:10
An outcome is a documented change in actual knowledge or
6:14
behavior, health status, relational connection, something real.
6:19
Yes. Having high outputs might be a necessary prerequisite
6:22
for good impact, but it is absolutely not proof of impact.
6:26
I always think of it like baking a cake. If I invite you over and
6:29
I say, hey, my recipe was a massive success because I used exactly
6:33
3 eggs, 2 cups of flour, and I baked it for precisely 30
6:37
minutes, I am just measuring my outputs. Right. You're just listing
6:40
the steps. Exactly. I followed the operational steps, but the true
6:44
outcome is whether the cake actually tastes good. I mean, I could follow
6:48
the recipe perfectly and still serve you something completely inedible.
6:51
That analogy holds up perfectly because if a clinic proudly reports
6:55
a massive increase in the output of, say, therapy referrals made,
6:59
that data point tells us literally nothing about the outcome.
7:03
Because we don't know what happened next. Exactly.
7:06
Did those adolescents actually schedule an appointment? Did they feel supported?
7:10
Did their coping skills actually improve? The output just
7:14
tells us the staff did the paperwork. That's such a good point. And this logic
7:18
also applies to another concept in evaluation that we found in the sources: the
7:22
tension between technical fidelity and essential form.
7:26
Yes, this is huge. So, let's say a clinic has a highly structured
7:30
prevention program with a strict step-by-step manual. A clinician could read
7:35
the script from that manual word for word, checking off every single box. Right,
7:38
and that would be considered high technical fidelity. They produced
7:42
the output perfectly. But what if they read it in this robotic,
7:46
totally judgmental tone while glaring at the teenager?
7:49
Well, then they have completely destroyed the actual goal of the intervention.
7:53
This is why we have to separate essential functions from adaptable forms.
7:58
Okay, break that down for us. The essential function of that scripted
8:02
intervention is not the specific vocabulary on the page.
8:05
The essential function is building a nonjudgmental
8:09
relational connection with the youth. I see.
8:12
So delivering a program perfectly by the book might actually ruin the essential
8:16
function if the delivery feels unnatural or, say, culturally inappropriate
8:21
to the actual patient sitting in the room. Exactly.
8:24
Sometimes adapting the language to fit the specific culture or setting,
8:28
like changing the outward adaptable form, is literally
8:31
the only way to preserve the underlying essential function.
8:35
Wow. Yeah, a really sophisticated evaluation doesn't
8:39
just ask if the script was read out loud. It measures whether that crucial relational
8:43
mechanism remained intact. But wait, if we agree that these deeper
8:47
outcomes and these relational mechanisms are what actually matter, we immediately
8:51
run into a massive structural problem. Which is? How in the
8:55
world do we measure things we cannot physically see? I mean, I cannot put
8:59
a tape measure around a teenager's level of trust. I can't draw blood
9:03
to test for identity threat. You've just hit on the core challenge of construct
9:08
validity. Things like trust or stigma or a sense
9:12
of belonging, those are constructs. Constructs, okay. They're complex ideas
9:16
that cannot be observed directly. Because of this, researchers have
9:20
to create what are called operational definitions. Right. They basically have
9:24
to design a specific measurable procedure to represent that invisible
9:28
construct. That might be a 5-point survey scale or an observation
9:32
of body language or even just a piece of administrative data. OK,
9:36
I am going to push back on this a little bit, or at least point
9:39
out a huge flaw in that system. Go for it. Any time you try to
9:42
turn a complex, messy human emotion into a clean, simple data point,
9:47
you are narrowing it. You are forcing this rich,
9:50
3-dimensional experience into a 2-dimensional box. Oh,
9:53
absolutely. Operationalization always narrows
9:56
a concept. Always. And this forces us to really examine the difference between
10:00
reliability and validity. Okay, what's the difference there?
10:03
Reliability is simply about consistency. If you give the same
10:06
teenager the same survey twice in one week, do you get similar scores?
10:10
If so, your measure is reliable. It's consistent. Right.
10:14
Validity, on the other hand, is about accuracy for a specific context.
10:18
Does this measure actually represent the invisible construct we think it does?
10:22
Oh, I've got a great physical example for this. Let's say the bathroom scale in
10:26
my house is broken, right? And it reliably weighs me at exactly
10:30
5 pounds lighter every single morning. I think we all wish we had that
10:34
scale. But mathematically, it is a highly reliable instrument.
10:37
It is incredibly consistent, but it is completely invalid
10:41
for telling me my actual body weight. Exactly. So translating
10:45
that to a healthcare setting, let's say a clinic decides to
10:50
define the construct of low patient engagement simply by counting the number of
10:53
missed appointments? Aren't they just reliably measuring the wrong thing?
10:58
If we connect this to the bigger picture of the ecological identity model,
11:02
your critique is spot on. Defining low engagement purely as
11:05
a missed appointment is a classic validity failure.
11:08
Because it assumes too much. Right. A missed appointment does not automatically equal
11:12
disinterest or resistance. The patient might just lack reliable
11:16
transportation. Or the clinic's online scheduling portal might
11:20
be malfunctioning. Or if we return to our EIM lens, the patient
11:24
might be experiencing an intense fear of disclosure and punitive consequences. Yes.
11:28
By defining engagement simply as physical attendance, the clinic is
11:32
totally misreading the situation. They think they have a motivation problem,
11:36
but they actually have an ecological barrier problem. Wow.
11:40
The dashboard literally says unengaged, but the actual human reality
11:44
is terrified. Exactly. Furthermore, the EIM
11:48
framework highlights that the act of measuring something actually changes
11:52
the ecology itself. Wait, really? How? Well,
11:55
when a youth is handed a screening survey, the measurement process is never neutral.
12:00
Depending on who is asking, how the questions are framed,
12:03
and what privacy guarantees are given, that survey can feel like profound
12:08
care and belonging. Oh, I see. Or it can feel like
12:11
dangerous adversarial surveillance. It is almost like the observer effect
12:16
in quantum physics, but applied to human emotions. Just the act of measuring the
12:19
phenomenon alters the phenomenon.
12:22
Exactly. But let's assume, just for the sake of argument, that we actually do
12:27
get valid, accurate measures. What do the numbers actually prove?
12:30
Because human nature makes us really want to believe that if a number goes up
12:33
after we launch a new program, Well, our program obviously
12:37
caused that number to go up. Oh, yeah. That is the illusion of causal inference.
12:42
Quantitative designs are fantastic for showing patterns and correlations.
12:45
Don't get me wrong. Right. But proving that a specific intervention
12:49
caused a specific change is incredibly difficult in
12:52
the real world. There are just so many alternative explanations, which evaluators
12:56
call threats to internal validity. Here's where it gets really interesting.
13:01
Let's walk through a specific example from the source material to see how these threats
13:04
actually work. Let's do it. Imagine a clinic implements
13:08
a new warm handoff program. So instead of just handing a teenager a
13:12
piece of paper with a therapist's phone number, the doctor physically walks the teen
13:16
down the hall and introduces them to a behavioral health specialist right
13:20
there in the office. Okay, great intervention. Right. And over
13:24
the next 3 months, the clinic's completed referral rate jumps from 30% to 48%,
13:29
an 18-point jump. Why shouldn't the clinic director immediately declare
13:33
a massive victory for the Warm Handoff program? Because of all those alternative explanations
13:37
hovering in the background. The first threat is called history.
13:40
History? Yeah. What if during those exact same
13:44
3 months, a brand new, highly publicized, totally free
13:48
behavioral health clinic opened up right across the street? Oh. That external
13:52
historical event could completely explain why more teens are completing referrals,
13:56
and it has absolutely nothing to do with the doctor's Warm Handoff. That makes total
14:00
sense. What about just the natural passage of time? Like,
14:04
teenagers are constantly developing and changing. Yes, that is the maturation
14:08
threat. People change naturally as they grow. Perhaps the
14:11
initial 30% baseline was measured during a highly stressful midterm
14:15
exam season. And the 48%? That could have been measured
14:20
during summer break when stress levels naturally dropped and teens had way more free
14:23
time to actually attend appointments. Wow. Okay. There is also something
14:27
called regression to the mean, which honestly trips people up constantly.
14:30
It really does. Let's say the clinic only instituted the warm handoff
14:34
program because they had an unusually terrible month where referrals dropped to like 10%.
14:38
Right. An outlier. Exactly. Statistically, extreme outliers
14:42
tend to settle back toward the normal average over time, regardless of what
14:46
you do. So the 18-point jump might just be the system returning to its
14:49
normal baseline, not a result of the new intervention at all. Spot on.
14:53
We also have to consider selection bias. Did the highly motivated
14:57
families self-select into the warm handoff pathway,
15:00
making the outcomes look fantastic, while the adolescents
15:05
facing the highest ecological barriers were just entirely excluded
15:08
from the pilot? Or what about instrumentation?
15:11
Maybe the clinic staff just received new software training midway through the 3 months.
15:16
So they just started recording the completed referrals differently.
15:19
Exactly. So that 18-point jump, it is a positive
15:23
signal. It definitely warrants further investigation,
15:26
but you cannot definitively point to it and say, our warm
15:30
handoff absolutely caused this. Right. Quantitative data
15:33
tells us about amount and distribution.
15:37
It tells us the what. But because numbers can obscure
15:40
causality and really hide the human context, we desperately
15:45
need qualitative evidence to understand the why and the how. Yeah,
15:49
if quantitative numbers are so vulnerable to all these optical illusions
15:53
like history and maturation and selection,
15:57
we just have to talk to the actual people behind the data points. We really
16:00
do. Qualitative inquiry, which involves things like in-depth interviews,
16:04
focus groups, and careful observation, it examines meaning
16:08
and lived experience. Right. When a program is succeeding or failing
16:12
and the administrative dashboard cannot tell you why,
16:16
qualitative data is literally the only way to figure out what is
16:19
actually happening on the ground. Which leads us to one of the most powerful concepts
16:23
in evaluative thinking that we found, the idea of missingness.
16:28
Oh, missingness is huge. Missing data is not just a clerical
16:32
error where, you know, someone forgot to fill out the back page of a form.
16:35
Missing data is actually vital information about the system itself.
16:39
Yes. In prevention programs, missingness is almost
16:43
always patterned. You constantly have to ask, who drops out of the
16:47
study? Who refuses to complete the follow-up survey? It's rarely
16:50
random. It's almost never random. Often it
16:54
is the youth dealing with unstable housing or the youth who felt profoundly
16:58
stigmatized during their first visit. Their absence from your clean,
17:02
perfect data set is screaming something incredibly important about the
17:06
ecological cost of participation. Participating in your program. Think about
17:09
looking up online reviews for a restaurant. If you only read the surveys filled
17:13
out by the diners who stayed to finish a 3-course meal and pay the bill,
17:17
well, the restaurant might look like a 5-star establishment. Right, because they stayed.
17:21
Exactly. But you are completely missing the reality of the dozens
17:25
of people who walked out the front door after seeing the exorbitant prices,
17:28
or the people who left immediately because the host was incredibly rude.
17:32
The missing data represents the silent majority. If you don't track who
17:35
is missing and actively investigate why, you are just
17:39
actively deluding yourself. And look at what happens in a
17:43
real-world clinical scenario. Take a teenager from
17:46
the sources. We'll call him Roger. Okay, Roger.
17:50
If the clinic director only looks at the administrative database, Roger is categorized
17:55
as a massive success. The dashboard shows he completed
17:59
a behavioral health visit. He gets a beautiful green checkmark.
18:02
But when researchers utilize qualitative methods, when they actually sit
18:06
down and interview Roger, the reality of that visit is entirely
18:10
different, isn't it? Entirely different. Yeah.
18:13
The interview reveals that Roger only attended the appointment because he was terrified that
18:17
refusing to go would cost him his spot on the varsity sports team.
18:20
Right. He didn't believe the conversation with a clinician was actually confidential,
18:25
so he just stonewalled. He provided whatever answers he thought would end
18:29
the session as quickly as possible. So the quantitative system logged
18:33
a success, but the qualitative reality was a complete system failure.
18:37
Wow. This raises an important question. It requires
18:41
evaluators to practice what is called epistemic justice.
18:45
We have to treat lived experience as a credible, valid form of knowledge
18:50
carrying just as much weight as attendance metrics. Which is hard
18:53
for some institutions to swallow. Very hard. But if an institution
18:58
accepts the database as factual reality but dismisses Rogers'
19:01
description of his fear and distrust as merely, you know,
19:04
subjective complaining, that evaluation is just reproducing institutional
19:09
blind spots. So if quantitative data gives us a false
19:13
sense of security and qualitative data reveals these hidden
19:17
ecological failures, how do program directors actually make decisions?
19:21
Like, how do we weigh those 2 things against each other to figure out
19:24
what to do next? Well, we use comprehensive frameworks that force
19:29
us to look at the entire picture. And one of the most robust ones out
19:31
there is the RE-AIM framework. It stands for Reach,
19:35
Effectiveness, Adoption, Implementation, and Maintenance.
19:39
RE-AIM. Okay, so RE-AIM basically stops you from making
19:42
the critical mistake of judging a program's success only by looking
19:46
at the people who finished it under perfect, controlled conditions.
19:50
Let's walk through how all 5 of these letters apply to a clinic launching a
19:54
new screening protocol. Sounds good. First up is
19:57
reach. Who is actually participating, and more importantly,
20:00
who is excluded? Then we look at effectiveness.
20:04
Does the program actually improve outcomes for the people it reaches?
20:08
And does it cause any unintended negative consequences? But a program only
20:11
works if people actually use it, right? So that brings in adoption.
20:16
Are the frontline workers, like the busy doctors and nurses, actually willing to use
20:20
this new protocol? Exactly. If the Warm Handoff takes 15
20:23
extra minutes per patient, the staff might just abandon it entirely.
20:27
Right. Which leads right into implementation. Is the program delivered
20:31
as intended in the messy, real world, considering all the actual costs
20:35
and resource constraints? And finally, maintenance. Maintenance.
20:39
Meaning, does it last? Yes. 6 months from now, when the
20:43
initial excitement fades and the grant funding runs out, does the clinic keep doing it,
20:47
or do they just revert to their old habits? Okay, let's apply this
20:50
RE-AIM framework to a specific pilot pathway from the source material.
20:54
So, a clinic rolls out a new EIM-informed screening
20:58
program. Initially, the quantitative effectiveness
21:02
data looks fantastic. The rate of private screenings jumps from
21:05
54% to 87%. Which looks great on paper.
21:10
But when the evaluation team actually investigates across the other RE-AIM
21:14
dimensions, they discover severe complications. Uh-oh.
21:17
Yeah. Under effectiveness, they find an unintended negative
21:21
consequence. An automatic online portal notification sent
21:25
to a patient's parents revealed a sensitive appointment category, which caused
21:29
massive family conflict. Oh, wow. So the intervention actually
21:33
breached privacy? The very trust it was trying to build. Exactly. And when looking at
21:37
reach and effectiveness combined, they uncover another major issue.
21:41
The average effectiveness looked great, but when they segmented the data,
21:44
rural youth showed almost no improvement. Why rural
21:48
youth specifically? Because the protocol relied heavily on telehealth sessions,
21:52
and qualitative interviews revealed that these rural youth entirely
21:56
lacked a private space in their homes to actually have a confidential therapy conversation.
22:01
This is a perfect example of a concept we found called multifinality,
22:05
isn't it? Yes, multifinality. Yeah. It means an average effect
22:08
almost always conceals subgroup differences.
22:12
Right. So the exact same intervention can be highly beneficial for one demographic group,
22:16
have literally zero effect on a second group, and actively
22:21
cause harm to a third group. Exactly. So what
22:25
does this all mean? The overall average numbers are up. But the
22:28
program is failing rural youth and causing dangerous privacy breaches through
22:32
the portal. Do you just scrap the whole program and go back to the drawing
22:36
board? No, you don't abandon the effort. But you also definitely don't
22:40
blindly celebrate the higher average. You use this robust mixed methods evaluation
22:44
to adapt. To adapt. Right. Okay. Right. The clinic preserves
22:48
the essential functions like the nonjudgmental inquiry and
22:51
the higher screening rates, but they aggressively redesign the adaptable forms.
22:55
How so? They might partner with local schools to provide secure private
22:59
telehealth rooms for rural students, which directly addresses the reach
23:02
issue. And they immediately overhaul the portal privacy settings
23:06
to prevent further harm. I love that. They use the evaluation
23:09
to actually learn, pivot, and improve rather than just grading
23:13
themselves with a simple pass or fail. Which really brings
23:17
us to the ultimate takeaway of this deep dive: evaluation is
23:21
not a final audit. It's really not. It is not a stamp of
23:25
approval you get at the very end of a project just so you can justify
23:28
your funding. No, it is a continuous dynamic cycle of learning.
23:32
Evaluative thinking is really about making your uncertainty visible.
23:36
Making your uncertainty visible. Yeah. I like that phrasing.
23:40
Yeah. It requires matching your level of confidence to the actual quality
23:44
of your evidence and cultivating a genuine willingness to revise
23:48
your decisions when deeper, more contextual information becomes available.
23:52
Because the ultimate goal isn't to just rigidly
23:56
defend a program's outcomes. Exactly. It is to learn how
24:00
to make the program more trustworthy, equitable, and effective for everyone involved.
24:04
Making your uncertainty visible. That honestly flips the
24:07
entire traditional paradigm of leadership and data analysis on its head.
24:11
As we wrap up today, I want to leave you with a provocative thought to
24:14
apply to your own daily life. I love this. Think about your own job,
24:17
whatever field you were in. If you were evaluated tomorrow
24:21
purely on your most easily measurable outputs, say the raw
24:25
number of emails you send or the lines of code you write
24:29
or the total number of meetings you attend, what vital,
24:33
invisible part of your actual impact would be completely
24:36
missed by that dashboard? And really consider what essential function
24:41
of your professional value would be erased if your performance was measured
24:45
solely by technical fidelity to a job description. Absolutely. The next
24:49
time you look at a data dashboard, whether it is an annual review at work,
24:54
headline in the news, or just the health app on your phone,
24:57
remember that you are only seeing the outputs someone deliberately chose
25:01
to count. Right. The real outcomes, the hidden human reality,
25:05
and the silent message of the missing data— that is where
25:09
the truth actually resides. Thanks for joining us for this deep dive.
25:12
Keep asking the hard questions, and we'll catch you next time.