Latest / The Reading Instruction Show / What is Evidence and Research?

Transcript

What is Evidence and Research? · The Reading Instruction Show

0:02
Greetings, I'm Andy Johnson. This presentation
0:04
is called Understanding Evidence and Research.
0:11
I have a lot of notes and handouts and other
0:14
things related to this at my free sub stack.
0:18
Plug in the reading instruction show and you
0:20
will find it there. It's free, there's no subscription
0:23
charge. Also, a lot of people have asked me about
0:27
teaching writing and writing instruction is one
0:30
of the most under instructed skills in our schools.
0:34
Again, another free substack, how to teach writing,
0:37
you can find it there. All the stuff there is
0:40
based on my book. And of course, like the rest
0:43
of 90 % of America, I have a podcast with tens
0:48
of listeners. Okay, let's start. Context. matters
0:54
to understand things, to understand research,
0:57
to understand data, to understand evidence. Context
1:02
matters. You can't pull things out of context
1:05
and try to understand. So let us look at the
1:09
big picture. The big picture is to enable all
1:12
children to develop their full literacy potential
1:16
so that they can use literacy for real life purposes
1:21
in the real world, not to pass a unit test or
1:26
to get higher scores on this and that. So to
1:30
use literacy to achieve real life goals. And
1:35
literacy includes reading and writing, but also
1:38
speaking, listening, and thinking. So I ask unto
1:42
you, what constitutes evidence of this? Beware
1:48
of the number monkeys who try to describe learning
1:51
only in terms of numbers. If it can't be quantified,
1:56
they tell us it doesn't exist. Reading is not
2:01
sounding out words. It's creating meaning with
2:03
print. And we read for two purposes. We read
2:08
to understand information, and we read to enjoy
2:12
books. That is the end state, the goal. All our
2:18
objectives should lead to that goal. Everything
2:22
we do to teach reading should lead to those two
2:26
goals. And the ability to do these things should
2:31
define students' reading ability. So what is
2:37
evidence of a literacy crisis? We're talking
2:40
about evidence. The crisis, a crisis, any crisis,
2:44
is a time of intense difficulty, trouble, or
2:47
danger. And some insist we have this thing called
2:51
a reading crisis. You can read the media and
2:56
literacy. Thousands of children functionally
2:59
illiterate. Oh, really? Just because you see
3:03
it or hear it doesn't mean it's true. Teachers
3:06
criticize their colleges of ed for not preparing
3:09
them to teach reading. How many teachers? Two
3:13
teachers? Four teachers? How old were they? First
3:16
year? 20th year? 50th year? How many colleges
3:20
of ed? We must look for evidence as we consume
3:26
the media, as we consume various forms of media.
3:31
It's alarming, they tell us. Children are severely
3:35
behind, teachers say. Behind what? Behind who?
3:40
What educators are saying what? Just because
3:43
is it two educators saying it? Journalists, journalists
3:50
are important in a democratic society. I'm not
3:52
getting down on them, but journalists write stories.
3:57
Stories are not evidence. Stories are things
4:02
that people put together. not necessarily for
4:06
accuracy, but for impact, to get clicks on their
4:11
little websites, to get awards at banquets. That's
4:15
what journalists do. There is not legitimate
4:19
evidence that there is a literacy crisis. Evidence,
4:24
legitimate. Paul Thomas talks about it and has
4:28
written about all the various literacy crises
4:32
starting in the early 50s. And there'll be another
4:36
one coming around in 10 to 15 years when this
4:39
one has found to be another boondockle or when
4:42
some publisher needs something else to be sold.
4:47
There will be another one when they find this
4:51
science of reading thing doesn't work. Why? Because
4:56
it's much easier to blame schools and teachers
5:00
and parents than to address social issues, systemic
5:06
poverty, economic inequality, overcrowded schools,
5:13
income inequality, lack of school funding. These
5:16
things impact learning. But it's easier to say
5:21
schools are failing, teachers are doing a bad
5:24
job. Everybody who's been to first grade seems
5:30
to have all the answers to fix public education.
5:35
Teach more phonics, George Bush told us once.
5:40
Higher standards, get tough. Get rid of teachers
5:43
unions. Reward the good teachers. More tests.
5:49
Reward high performing schools. Interesting.
5:52
Simple, simple answers. to complex problems.
5:58
Blaming teachers takes people's attention from
6:02
the real issues and teachers can't fight back.
6:09
This has led to a teacher shortage. Teacher turnover
6:14
is the leading factor in teacher shortages. They
6:19
are leaving. Qualified teachers are leaving teachers
6:23
who love their job, good teachers, for reasons
6:26
other than retirement. There's no shortage of
6:32
teachers. There's a shortage of teachers who
6:35
are willing to put up with baloney. As an example,
6:42
I am retiring in three months and one of the
6:46
reasons I'm retiring is because I would be required
6:50
to go through letters training, what I call re
6:54
-education camp. Just like in a clockwork orange,
6:58
they force that guy's eyes open to watch those
7:01
scenes. I would be forced to watch letters or
7:05
to take letters training. There is not a reading
7:09
crisis, a reading achievement crisis. Now we
7:14
are going to look at data from the National Assessment
7:18
of Educational Progress. This is part of the
7:21
U .S. Department of Education. They take a large
7:24
sample, a representative sample. They give tests
7:28
to students in grades four, eight, and 12 every
7:30
year. It's called the nation's report card. It
7:34
gives us a sense of how we are doing. Rikin,
7:40
Ruby, and Risco. There's no irrefutable evidence
7:44
that we are experiencing a national literacy
7:47
crisis. Crisis would look something like that
7:52
or something like that. It would not look like
7:58
that. These are long -term reading scores in
8:05
fourth grade and in eighth grade, starting in
8:09
71. You can see there's just a gradual release
8:12
of scores, but there's a natural fluctuation,
8:15
and that's called normal, normal. This is the
8:21
latest one I found a couple of days ago on the
8:24
website, and it's hard to find this data. They've
8:27
included a whole bunch of stuff that makes it
8:30
difficult. No statistically significant difference
8:33
from 2013 to 2020. Of course, we had the COVID
8:39
dip. didn't we? And before the dip, we were higher
8:44
than we were statistically significantly higher
8:48
than we were in 92. Same with eighth grade. Here
8:57
is the COVID dip and we expected that. But from
9:01
12 to 20, no significant dip. Reading scores
9:07
fall to new lows, says Sarah Schwartz from Education
9:12
Week. And she's a reporter. She just writes a
9:15
lot of stuff. She is a reporter. She's not an
9:19
academician or a scholarly writer. But here's
9:23
the thing, even if she weren't lying, think about
9:26
it. This is the dates when all the states have
9:30
passed their miracle reading legislation. Mississippi
9:36
in 2013. Here's when the miracle started. And
9:39
look at all these other things, all these other
9:43
miracles starting 2009, 21, 22. Here's Minnesota
9:47
right there in 2023. So let the miracles occur.
9:53
We should just be sitting back enjoying our cup
9:56
of coffee and just watching the miracles unfold
9:59
in front of us. Dramatic pause. except they are
10:06
not. But even if there were a crisis, which there
10:10
is not, the science of reading a fellow's people
10:16
have determined, have claimed that they've isolated
10:21
the causal variable. They say that they know
10:25
both the cause and the solution to a problem
10:28
that does not exist. But let me tell you, my
10:32
friends, Randomly determining causality is not
10:38
science. That's baloney, sometimes called pseudoscience.
10:44
These are all the variables that impact student
10:49
achievement. And you can pause and look at it
10:52
yourself for a good time on a Saturday afternoon.
10:57
Again, we must distinguish between scholarly
11:00
articles, scholarship, and journalists. Journalists
11:06
write down what people tell them. People tell
11:09
them things, they write it down, and they create
11:12
their stories. And everyone wants to be the next
11:16
Emily Hanford. Look how famous she is and how
11:19
much money she's getting and how many speaking
11:22
gigs. Oh, I want to be just like Emily. publish
11:26
stuff. They put numbers to it. They have sensational
11:30
things. They always have a picture of a kid who's
11:35
crying or looking distraught. And look at this
11:38
graph. Oh, that's just horrible. That's horrible.
11:41
Pulling things out of context, putting them in
11:45
ways that create a sensation. It's a straw man
11:50
argument. This is when you create a cartoonish
11:53
version of something and argue against the cartoon.
11:58
Things like balanced literacy. People describe
12:01
it in ways that it is simply not true. Whole
12:05
language, the three QE systems. Some people are
12:08
just horribly upset about what they call Q -ing,
12:13
and they don't even know what it is. They're
12:14
describing something horrible. Public schools,
12:18
DEI has been made a swearing word. Critical race
12:23
theory, same thing by people that have not taken
12:26
the time to understand what it is. I had this
12:31
fellow on my show. I was interviewing him. He
12:34
asked me to, so I did. He had a book which had
12:38
a very interesting premise. The book was called
12:41
Failure Factor, and in this book, he described
12:44
administration. as changing teachers' grades
12:49
to show that they were doing a good job. And
12:53
in Baltimore, apparently, instead of a superintendent,
12:56
they have a CEO, hence the factory motif. But
13:00
Brother Chris here has some very simplistic answers
13:04
to a very complex situation. And he either misunderstood
13:08
or misrepresented information. His answer, of
13:14
course, was just to hold students to higher standards.
13:17
Oh, okay. Just hold them to higher standards.
13:20
Duh. Why didn't I think of that? Based on a lot
13:23
of I -thinkisms. Now notice, here's some of the
13:27
stuff from his book indicating failure. He says,
13:31
well, the budget has stayed consistent, all right,
13:35
and the number of enrollment has gone down significantly.
13:39
Look at all these things. But none of these things
13:43
are indicative of the quality of education occurring
13:46
in classrooms. SAT, that's a self -selected test
13:51
that people going to college take. That's not
13:54
a representation of how schools are doing. AP,
13:58
advanced placement exams, again, a representative
14:01
sample. Attendance, you can have quality teaching
14:05
going on, but that does not impact attendance
14:09
or dropout. or absenteeism, or enrollment. Some
14:14
of these are societal factors. So either he doesn't
14:19
know, he doesn't understand, or he's purposely
14:22
misrepresenting to get his little statues and
14:25
to get book sales. Again, he said, more discipline.
14:30
Hold students to higher standards. Get rid of
14:33
teachers unions. I hear that a lot. And in my
14:37
book, I will be talking about teachers unions
14:39
at some point, not today. Here's the thing I
14:43
read in his book that made me want to throw up
14:46
in my mouth. When a reporter focuses on one beat,
14:50
they become an expert on that beat. Oftentimes
14:54
a beach reporter knows the subject matter better
14:57
than the industry professional they cover, unquote.
15:03
What do you say to something like that? You don't.
15:06
You just vomit in your mouth a little bit. They
15:09
write stories. This is a story written like a
15:13
detective novel. Watch the City has fallen into
15:17
a state of decay. And that was certainly my view
15:20
one September afternoon in 2017, as my photographer
15:25
and I were. You get the idea. This isn't scholarly
15:29
writing, not that you need scholarly writing,
15:32
but he's creating a story for impact. That's
15:36
what they do in television. That's how you get
15:39
those little statues at award ceremonies. Everyone
15:42
wants a statue. I want a statue. I welcome journalists
15:47
into our academic conversations. I welcome you,
15:51
but you do not get a free pass on ignorance.
15:55
In my conversation with Chris and John Stossel,
16:00
I understood that you have to have some background
16:03
knowledge to understand and interpret what you
16:06
see and hear. I often had to correct the presumption
16:12
upon which questions were based before I answered
16:15
questions. They had a shallow and disjointed
16:19
knowledge base. There's nothing wrong with that.
16:23
But you don't realize that. It makes it difficult
16:27
to engage in academic discussion. discourse with
16:30
these people, these people, with a journalist
16:33
sometimes. I don't want to generalize. Reading
16:36
instruction is first and formally an academic
16:40
topic, and it has been, discussion has been going
16:44
on in academic, academia, academic discussions
16:47
for decades. And there's always been people with
16:52
differing ideas. There isn't a reading war. And
16:55
again, You are welcome as anybody is into our
17:00
academic discussion. Anybody can publish in an
17:04
academic journal. It's subjected to blind peer
17:07
review. Anybody could speak at a conference,
17:11
but allow for questions, get alternative views.
17:14
When you say research, cite the research. You
17:19
should not get a pass on ignorance. And that
17:22
is happening. As I mentioned, I was interviewed
17:26
by John Stossel. The interview took about 60
17:29
minutes. He put together a story and my little
17:33
part, I'm guessing was two to four minutes at
17:36
most. And he created a collage with it to create
17:39
the story that he wanted. This is an email I
17:43
got from a producer. We're doing a story on the
17:46
literacy crisis in America. of which there is
17:49
not, I came across your advocacy of whole word
17:53
reading instruction. Where did you come up with
17:56
that? I don't know. Do you know what whole word
18:00
reading instruction is? I've talked about whole
18:03
language, but that's different from whole word.
18:07
But anyway, he said, we would like to get your
18:09
take on the subject. They would like to get my
18:13
take. Imagine that. They didn't want my take
18:17
on anything. They wanted to create a collage.
18:21
Now, the speaking fee for John Stossel, for him
18:24
to do a keynote is $75 ,000. I know because I
18:29
pretended I was someone else and I contacted
18:31
his speaking bureau. Am I a bad man for doing
18:34
that? The speaking fee for me to do a keynote
18:38
is give me a little flattery and a hamburger
18:41
and I will be there. Now, in 1992, somebody determined
18:46
that the NAAP reading scores weren't low enough,
18:49
so we need to invent categories, advanced, proficient,
18:53
and basic. Well, what is basic? We just assume
18:58
that it's something. What is proficient? We just
19:01
assume that it's something. What's average? This
19:06
is what Rinking Ruby and Risco said. The cut
19:10
scores used to define these categories were arbitrarily
19:14
set by a small group of hand -picked politically
19:20
appointed experts against the recommendations
19:24
of the psychometricians who have been contracted
19:27
to advise the development of this approach. You
19:32
hired people? Interesting. Okay, interesting.
19:38
Categories and levels and content are arbitrarily
19:42
defined. citing some more sources there. Now,
19:46
if you go on the website, US Department of Education,
19:50
they report in percentile rankings, 10th, 25th,
19:54
50th, 75th, and they have so much data and little
19:58
data bits, it's designed more to confuse, and
20:02
you can pull out little data bits to create a
20:05
variety of stories, a variety of impressions.
20:09
If you do this, pull it out of context. context
20:13
as needed. This is average right here, so let
20:17
me explain. 1992, this is average, so they decided,
20:23
hmm, I think we'll make basic a little below
20:28
average and proficient a little above average.
20:34
What would that look like in a normal population?
20:37
In a normal population, we have the bell -shaped
20:40
curve where 50 % are below the mean and 50 %
20:45
are above the mean. And in every school, if it's
20:49
a normal population, we're going to have 50 %
20:52
below the mean or below average. Well, if they
20:57
decide that this is basic right here, by definition,
21:02
40 % are gonna be below the basic level. And
21:08
60 % are gonna be below the proficient level.
21:13
And you can say, oh, we have a crisis. Only 40
21:17
% of our students are reading at the proficient
21:20
level. You see how the game is played? That's
21:24
how they play the game. That's not right. Eighth
21:29
grade, same thing. Same thing. Same thing. That's
21:35
not right. Walton said, this is from their website.
21:43
This is from the Center for Educational Statistics.
21:46
Proficiency levels are based on desired expectation.
21:49
These levels are aspirational, not mathematical.
21:54
We would like this to be basic. We would like
21:57
this. there to be used on a trial basis, we need
22:02
evidence to determine if they are reasonable,
22:06
valid, and informative. Hand -picked political
22:13
appointees, arbitrarily decided levels, any time
22:17
you hear about NAP levels of basic and proficient,
22:22
even on state level tests. These should always
22:26
be put in this context and they should always
22:30
be questioned. State criterion reference test.
22:37
You can't compare state to state because each
22:40
state has a different test, but someone establishes
22:44
a criterion. We think third graders should know
22:47
that and sixth graders should know that. A test
22:50
is given every year. And we think this is basic,
22:54
and that is, arbitrarily decide who decides the
22:59
criteria, who decides the content of these tests,
23:03
who decides what's advanced, proficient, and
23:07
basic, and how are these decisions made. I had
23:12
to do a lot of looking here in Minnesota on their
23:15
websites, and I finally found, after a lot of
23:18
phone calls and tiny print on their websites,
23:22
that originally these MCAs were based on the
23:26
Common Core State Standards. They took these
23:30
three -dimensional standards and made a two -dimensional
23:33
test. Of course, you never actually get to see
23:35
what's on the test. And in your real life, how
23:39
often have you had to do these things? But yet
23:42
these are used to define the literacy potential
23:47
of our students. These are, and these standards,
23:51
Common Core, were aspirational, not mathematical.
23:55
To prepare students, we would like them to be
23:58
able to do these things at each level, aspirational.
24:02
And this idea, like they have here in Minnesota,
24:06
that every student will be reading at grade level,
24:09
that's simply absurd. That creates winners and
24:14
losers. Grade level. How do you determine grade
24:18
level? Criterion based, norm reference based,
24:22
aspirational, inspirational, fairy tale based
24:25
measures. Now in Minnesota, a couple of years
24:29
back, they claimed that test scores were falling,
24:32
but no test scores were ever reported. They don't
24:35
give you that. They simply give you percentages.
24:39
And four of these arbitrarily defined categories.
24:44
Those aren't test scores. And you can't use percentages
24:50
to make comparison. This is percentages of students
24:54
at or above proficiency. That's not a comparison
24:58
of scores over time. That's what's called in
25:02
the business baloney. And that's the family friendly
25:06
term for it. So tips for evaluating scholarly
25:10
work. And again, Academic discourse is when you
25:14
use research and reason to make a case for one's
25:17
argument. Argument here isn't the yelling kind
25:20
of argument. It's where you put forth a proposition
25:23
and you make a case for that using reason, using
25:27
reason and research. And disagreement is a natural
25:33
part of the academic conversation. It's how we
25:37
evolve our thinking. There should never be a
25:41
singular type of thinking. This insulated bubble
25:44
that the reading league tries to create for itself,
25:48
that creates cognitive inbreeding and deleterious
25:52
mutations. We need to hear other ideas to evolve
25:57
our thinking. I say the reading league because
26:00
when I post things on their various platforms,
26:03
they block me. Oh, we can't have our people hearing
26:06
that idea. That's horrible. They may believe
26:09
the wrong thing. So they block me. Interesting.
26:15
Now, it's important that we critique each other's
26:19
work. That's what we do in a scholarly, in academia,
26:24
in a scholarly context. Now, I have a book coming
26:28
out this summer, the one you see here. And I
26:30
hope you critique it. I hope you tell me what
26:34
I got wrong. Don't tell me I'm a bad person or
26:37
I'm a horrible person. I already know that. Don't
26:40
tell me I'm not very smart. I already know that.
26:43
Tell me what you disagree with and what I got
26:46
wrong. And here's the thing. If you tell me what
26:49
I get wrong and I'm convinced, I will make another
26:53
video. But don't just say I'm a bad person and
26:56
I got it wrong. That's not academic. Discourse.
27:01
Disagreements based on works published or presented
27:04
in an academic context. That's what academic
27:09
discourse is. Not on personal attributes, not
27:13
on personal emails. This is where we disagree.
27:18
We must have disagreement. That's how we evolve
27:21
our thinking. Academic discourse using reason
27:26
and research. research -based theories. And by
27:29
the way, there never was a reading war. That
27:33
was a term invented by people outside of an academic
27:39
context in the media to sell books or papers,
27:43
whatever. There has always been differing ideas
27:46
for decades since the National Council of Teachers
27:50
of English was formed in 1911. the International
27:55
Literacy Association, then the International
27:57
Reading Association in 1956. People have differing
28:02
ideas based on their theoretical orientations.
28:06
Academic discourse is how we hold ourselves accountable.
28:11
And I've taken Sally Shaywitz to terms for her
28:17
research, for her bad research. but I'm sure
28:20
she's a wonderful human being and she has to
28:22
be very smart, but I have trouble with all the
28:27
flaws and confounding variables in her research.
28:33
How does she get away with that, I wonder? The
28:37
differences we have in academia are based on
28:39
theoretical orientations. Every theory can be
28:42
supported by research data and evidence. Theories
28:46
are neither right nor wrong. They're weak or
28:49
robust. And I'll demonstrate that. Flat earth,
28:52
that's a theory. There may be little bits of
28:55
data. It's a very, very weak theory. And I think
28:59
the simple view of reading is the flat earth
29:02
theory of reading. Yes, there's research to support
29:05
it. but it's weak research. It's a weak theory.
29:10
Theories are based on research. Research study
29:14
is a data dot. That's why you can have separate,
29:18
you can have different research -based theory
29:21
connecting different data dots differently, and
29:24
they're both research -based. It's weak or robust.
29:31
theory. A robust theory connects a lot of data
29:34
dots. A weak theory leaves a lot of data unaccounted
29:37
for. The interactive theory of reading, psycholinguistic,
29:43
neurocognitive theories, account for a lot of
29:47
data dots. The simple view of reading account
29:50
for a few. They ignore a whole bunch of stuff.
29:55
Now here's one way to transform weak into robust
29:59
theories. You can do what the National Reading
30:01
Panel did. They ignored or excluded a whole bunch
30:06
of research. Oh, all that research over there.
30:09
That's not good stuff. Only this kind of research
30:13
is good. Okay. Reading instruction is an academic
30:19
topic. Academic arguments based on theoretical
30:24
differences have been going on for years. It's
30:27
a natural part, and as I said, the term reading
30:32
wars was invented by the media. All right, let's
30:38
take a look at scholars. We are looking at evidence.
30:41
Scholars should not get a free pass on obfuscation.
30:49
Common tricks, using buzzwords. That's a word
30:52
used to elicit an emotional response. rather
30:57
than accurately transmit information. And using
31:02
them in an academic context, that's an academic
31:05
buzzword used to create the illusion of something
31:08
important. Now, take a look at these academic
31:13
buzzwords. They're used all the time to make
31:17
something sound very, very important. Could you
31:20
define each one of these? If I went to a reading
31:26
league conference and sat down and have each
31:30
person write a definition of these, how much
31:34
similarity do you think there would be in these?
31:39
What is evidence -based instruction? Well, you
31:43
know, it's instruction and it's based on evidence.
31:46
Okay, buzzword. And we already talked about the
31:51
straw man argument. Louisa Moats, oh, Louisa,
31:58
Louisa, Louisa. She wrote this for the Thomas
32:03
Fordham Institutes, not peer reviewed. It's,
32:06
but here's the ultimate straw person talking
32:10
about whole language, uses teacher modeling,
32:14
not direct instruction. Instead of teaching children
32:19
how to read and comprehend, they engage in shared
32:22
reading. Students can repeat the language and
32:26
read by osmosis or memorization. Just, Louisa,
32:30
do you not know better? That is not at all what
32:33
whole language is. Do you not know better or
32:36
do you know better and you're doing this anyway?
32:39
Both are frightening propositions, but this is
32:42
a straw man argument. This is a cartoonish version
32:46
of something. Do you have no decency? Don't do
32:50
that. Baloney, is interpreting research or misrepresenting
32:59
decontextualized data. This is why we have to
33:04
critically evaluate all the scholarly work. We
33:08
can't be snookered by important sounding brain
33:11
words. We have to pull them apart. We have to
33:14
examine the propositions and the citations. And
33:18
a cow pie is when you put a whole bunch of academic
33:21
buzzwords together. to create a conveniently
33:25
inaccurate perception of something. Here's an
33:28
example of a cow pie. Goldenberg wrote this in
33:33
reading research quarterly, talking about reading
33:36
wars. Now, a proposition is a basic claim or
33:41
assertion within a sentence. You can say it's
33:43
true or false. The dog is in the house. Three
33:47
propositions. There's a dog. The dog is in something
33:50
and it is the house in which the dog is in. Three
33:54
propositions there. Here's Goldenberg's abstract.
34:00
The author's intent is to begin with the knowledge
34:03
base of the science of reading, build onto it
34:07
research on effective literacy instruction for
34:09
ELLs, and then begin articulating what could
34:12
become a science of reading instruction for students
34:15
who are learning and progressing as readers in
34:17
English while learning to speak and understand.
34:20
It sounds very good. It sounds so wonderful.
34:26
But let's, when we look at that cow pie, we can
34:30
find here's just two of the propositions. There
34:34
is a knowledge base and the science of reading
34:37
has it. The science of reading knowledge base.
34:40
Where is that? Where can I find that? Who put
34:44
the knowledge in the base? Is there a balanced
34:47
literacy knowledge base? Hmm, these are buzzwords
34:50
strongly together. I could do that. The presenter's
34:55
intent in this presentation is to begin with
34:57
the knowledge base of reading research, build
35:00
on to it research related to neuroscience, psycholungus.
35:03
You get the idea. Anyone can create a cow pie.
35:07
It's not that tough. It creates the illusion
35:11
of stuff, but it doesn't transmit information.
35:16
We have to be wary when we see things like this.
35:20
This is a misinterpretation. This is an example
35:23
of baloney. In that paper, he referenced two
35:26
research studies to make a case for structured
35:30
reading instruction. And this is what he said.
35:33
He looked at von et al and Eri et al's research
35:38
studies. They included explicit and systematic
35:41
instruction of targeted knowledge and skills
35:44
and daily lessons that include activities focused
35:47
on the five core skills of phonological awareness,
35:50
phonics, fluency, and comprehension, developing
35:53
oral language, integrated oral language. It sounds
35:56
very good. Stop using my cow pies, the cow says.
36:01
But when we look at that, and I just so happened.
36:06
to look at each of these studies, and I've happened
36:08
to analyze both of them very carefully. These
36:11
are the pretty pretty words that Claude Goldenberg
36:14
uses. Oh, they are so pretty, pretty pretty words,
36:18
but they are conveniently inaccurate because
36:23
in his description none of these words were supported
36:27
by the research he is citing. Yes, some of them
36:33
were used as descriptors in the studies. Their
36:36
description was often inaccurate, incomplete,
36:39
or just plain wrong. But none of the pretty words
36:44
were variables that were isolated and compared
36:47
in these two studies. Why is that important?
36:52
And here's more baloney. Effects were moderate
36:55
measures on English, phonological awareness,
36:59
et cetera, et cetera. OK. Principles of early
37:02
reading instruction. Okay. Isolating variables
37:06
in controlled experimental research, which is
37:10
the science of reading research standards. That's
37:12
not mine. It's used to determine if one thing
37:17
caused the other thing to occur. If structured
37:22
language causes something to occur, but you have
37:25
to start with two groups that are similar and
37:28
exposed to similar conditions. The treatment
37:33
is the variable. That's the only difference that
37:35
should be between the two groups. And if there
37:39
is a difference on some measure between the two
37:42
groups at the end, and if that difference is
37:46
statistically significant, we can say with some
37:49
amount of confidence that the treatment was the
37:53
cause of the variable. That's a very simple overview.
37:57
of controlled experimental research. But in both
38:00
the studies that Goldenberg is citing so confidently,
38:05
they compared students who received individual
38:09
tutoring to students who received instruction
38:13
in small groups. And they said, oh, it was the
38:16
specific type of tutoring that caused the difference.
38:21
When in fact, it had nothing to do with the tutoring.
38:25
One was individual tutoring with individual tutor
38:29
with lots of feedback and a whole bunch of stuff.
38:33
And another was in small group using prepackaged
38:36
remedial stuff. It was designed to give them
38:40
a result that was favorable to the group that
38:44
they wanted to. As an example, this is what it
38:49
was. My research question was, do blue cages
38:53
cause mice to gain weight? And I said, yes, they
38:56
do. Because we compared them, and the blue cage
39:00
mice were heavier, and the yellow cage mice were
39:05
skinnier at the end of this experiment. So if
39:08
you want to lose weight, paint your house yellow.
39:12
And you see the confounding variable here? These
39:16
aren't equal groups. One was fed marshmallows,
39:20
the other was fed salaries. In the studies, Goldenberg
39:24
cited the type of metaphorical food given to
39:29
the metaphorical mice were different. They claimed
39:33
it was the color of cages that caused improved
39:36
reading scores. Anyone with a sixth grade education
39:40
could see that if you read the studies carefully.
39:44
So three questions, which A controlled experimental
39:48
research will not answer. Did Goldenberg not
39:51
read the studies? Did he read the studies and
39:54
not see the confounding variable? Or did he read
39:58
the studies, see the anomaly, and make the citation
40:02
anyway? I would like to know. Understanding research.
40:08
And by the way, if state legislators were really
40:12
serious about evidence -based practice for reading
40:17
and research -based reading instruction, but
40:19
they're not. If they were, every school district
40:23
would have a subscription to the same kind of
40:26
legitimate research database that I do as a university
40:30
professor. And not what works Glaringhouse or
40:34
Eric. That's not legitimate. That's not legitimate.
40:38
How do you expect teachers to make research -based
40:42
decisions and evaluate evidence -based practice
40:45
without research and evidence, and yet you blame
40:50
them and you beat them up. That's not logical.
40:54
That's not science. There's no such thing as
40:58
the research. The research says. The research
41:02
shows. I'm often asked this, show me the research.
41:06
What does the research say? They demand of me
41:10
the research. By the way, when anyone uses research
41:15
in a declarative sentence, they must, in an academic
41:18
setting, cite their research. You can't just
41:21
say, research says there's no such thing as gravity.
41:25
There's no such thing as the research. There's
41:29
no objective God. Oh, almighty research, we prayeth
41:34
unto you, give us truth and enlightenment. There's
41:38
no God who offers this objective view of reality.
41:44
There's no such thing as the holy book called
41:48
the research, where you can look up and say,
41:52
ah, the research says. Maybe they mean Thea research,
41:58
a researcher named Thea research. And they're
42:00
saying Thea research says. Maybe I should call
42:06
my next book the research. So you could say the
42:09
research sets. Research involves science. Science
42:17
uses research. It's a process of asking a question,
42:21
collecting data and using data to answer that
42:25
question. The question up front is important.
42:28
Louisa Motes tends to just collect data and then
42:31
come to a conclusion. That's not research. What's
42:35
the weather? Question. Look at the thermometer.
42:38
Data. It's hot. Data to answer question. That's
42:43
a basic of research. It's a systematic process
42:47
used to collect and analyze data. Systematic
42:50
means there's a fixed plan. based on a set of
42:55
principles that are generally agreed upon by
42:57
the field. And the research method is a systematic
43:02
process used to gather and analyze that data.
43:07
That's important because there is no scientific
43:10
method. There is no the method of science. There
43:13
are methods of science. Controlled experimental
43:21
research, the controlled experimental research
43:23
is only one method of science. We must understand
43:31
that conducting research in the physical sciences
43:35
is a bit different from research in the social
43:39
sciences. With human beings, we cannot control
43:43
all the variables. We cannot put our children
43:45
in giant Petri dishes. and give them similar
43:50
conditions. That's not how it works. And research
43:54
is not research unless and until it has been
43:59
subjected to blind peer review and published
44:02
in an academic journal. Here's the basic blind
44:10
peer review process. This is why it's important.
44:16
You do a study. You write it up. You submit the
44:20
manuscript to an editor. An editor submits it
44:24
to blind peer review. These are experts in their
44:28
field. They don't know who the author of the
44:30
article is. The author doesn't know who the reviewers
44:34
are. They review that and they make a recommendation
44:38
either to reject, accept with revisions, or accept.
44:44
There, and it's usually two to five reviewers,
44:48
their comments go to the editor, who then makes
44:51
a decision based on their comments to accept,
44:58
reject, or accept with revisions, and they work
45:00
with the author. That's the process, checks and
45:04
balances a filter. That's different from the
45:08
process a journalist use. People tell them things,
45:12
they write a story, and they submit it to the
45:14
editor who approves or asks for revision. Much
45:19
different. And the process used to put stuff
45:23
on the internet. You think of stuff, you write
45:26
it, and there you go. It's on the internet. Interesting
45:30
article I recently analyzed. It's not research.
45:35
These publications don't go through blind peer
45:38
review. They just hire reporters to say stop.
45:42
American public media. Data is not research.
45:47
Published reports are not research. White papers
45:51
aren't research. Unpublished theses and dissertations
45:56
do not go through blind peer review. Articles
46:01
describing research are not research. Famous
46:06
researchers aren't research, secretaries of education
46:10
aren't research, and anything on Lexia Learning
46:14
website is not research. These are often used
46:19
as if they were by people outside of academia,
46:24
collecting data, citing white papers, think tanks,
46:29
not research. Not. research. Non -profit does
46:39
not mean non -biased. These are all think tanks
46:43
that have things to say about education, but
46:46
they all have an agenda. The two kinds of research,
46:54
basic and applied, basic science, and applied
46:58
science. Basic is research that is done apart
47:01
from the setting in which it's used, eye movement,
47:04
brain imaging, miscue analysis. Applied research
47:08
is done in the setting in which it will be used
47:10
in the classroom or learning environment. And
47:16
controlled experimental research is used to isolate
47:19
a variable to determine a causal relationship.
47:24
This thing caused that to occur. Now the science
47:31
of reading research standard says only applied
47:37
research using control and experimental groups
47:41
can be used to make decisions about reading policies,
47:45
products, and procedures based on the mistaken
47:49
assumption that you can isolate all the variables
47:52
to determine universal causality and generalize
47:58
that to all of humanity. That's a mistaken assumption.
48:04
Research is not research unless it has been subjected
48:07
to blind peer review. Research is different from
48:10
collecting data. Data is not research. I could
48:19
collect data to show that I am the greatest professor
48:24
in the world, but that wouldn't be research.
48:28
What's a good professor? What's a valid measure
48:31
of a good professor? These are all things that
48:35
I could collect. One of these, oh, look at my
48:38
course evaluation. Oh, look at the number of
48:41
YouTube views. That proves I'm a good professor.
48:47
I highlight this because at our university, someone
48:51
wanted to have metrics, I hate that term, metrics,
48:56
to show that our College of Ed was doing a good
48:59
job with teachers. So to get metrics, they found
49:03
something that could be measured. And they went
49:07
to the scores on the state teacher licensure
49:11
exams that our students took, and they said,
49:14
oh, look at the scores, based on that, we're
49:16
doing a good job. That's not a valid measure.
49:21
And here's the thing, if we spent the three semesters
49:24
we have them, just preparing them to take those
49:27
tests, our scores would be out of this world.
49:31
But we try to prepare them to do this thing called
49:34
teach, teach. Imagine that. A Massachusetts rep
49:41
said, show me the data on queuing. The data,
49:45
the data. Data is not research. Anybody can collect
49:50
data. And here's the thing on queuing. Queuing
49:55
is not a method or a approach. How's that? There's
50:00
some data. The three queuing is not a method.
50:03
It's not something you teach. And we'll talk
50:06
about that later. Methods of science. The scientific
50:12
method. There's no such thing as the method.
50:20
A variety of research methods are used to study.
50:23
And there's two groups, quantitative and qualitative.
50:28
Let us understand the basics. Quantitative research,
50:32
again, seeks to find that causal relationship,
50:36
controls or manipulates reality to isolate a
50:40
variable. You have a measure. You have two groups
50:44
that are similar. You isolate one. The only difference
50:47
is this treatment. And then you have a post -test.
50:50
That is the basics. The difference must be statistically
50:55
significant. That means it's greater than could
50:59
occur by chance. Now, you don't need to understand
51:03
statistics to understand statistical significance.
51:09
statistically significant at the .05 level, that
51:14
means in less than 5 % of the cases, can we say
51:19
that that difference was attributed to chance?
51:24
In 95 % of the cases, we can be confident that
51:28
the difference between the two groups was caused
51:31
by the variable. .05 is good, .01 is the best.
51:37
but 0 .05 is generally regarded as the standard.
51:42
Controlled experimental research seeks to quantify
51:46
reality. But here's the thing. By controlling
51:50
all the variables to study reality, you make
51:54
that bit of reality a bit of non -reality. You
51:59
mess with the environment. It's not how it really
52:03
is if you're controlling all the variables. Qualitative
52:07
research seeks to study reality as it is. There's
52:13
Jane Goodall studying apes in the wild making
52:16
notes. There's a variety of types of qualitative
52:23
research. Here you see a professor in there taking
52:28
notes that helps us understand, that sees the
52:32
people behind the numbers. By using only quantitative
52:37
research, we are describing learning and human
52:40
beings in numbers. But in 2002, the Education
52:45
Science of Reform Act said only quantitative
52:50
research could be used. That's the only kind
52:52
of research that would be funded. That's the
52:55
only kind of knowledge that's valid. So a whole
52:59
bunch of qualitative research that helped us
53:02
understand. that helped us understand the numbers
53:05
was eliminated and slowly went away. It has to
53:10
be experimental or quasi experimental research.
53:14
No qualitative research, no government funding.
53:19
So it's not the use of controlled experimental
53:23
research that's problematic. It's the exclusive
53:26
use. you disallow us to see the people behind
53:30
the numbers by disallowing a whole bunch of good
53:35
qualitative research. And this is what the science
53:39
of reading standards are. Strategies and practices
53:44
that lead to improved outcomes, yes. Strategies
53:49
and practices are supported by experimental or
53:52
quasi -experimental research, establishes causal
53:56
variables and the research has been conducted
54:00
in an authentic learning environment. The science
54:03
of reading says they use strategies and practices
54:06
that have been determined to be effective by
54:10
these standards. Those are the standards and
54:14
I would be just fine with those standards if
54:19
they were consistently applied. Now here's an
54:23
example of baloney. In Minnesota and Massachusetts,
54:29
they recently passed legislation. They assumed
54:32
there was a literacy crisis. They assumed teachers
54:36
were teaching the wrong thing. And they assumed
54:39
the problem was balanced literacy and the answer
54:42
was structured literacy. And the causal relationship
54:46
was determined by anecdotal evidence. That's
54:50
not science, people. That's not science. If you're
54:54
going to be a science of reading, you have to
54:56
be consistently scientific. The science of reading
55:00
should not be given a free pass on non -scientific
55:04
thinking. And these are two examples of things
55:08
that are given a pass. Do these. As research
55:12
show that they lead to improved reading achievement
55:16
or improved teacher ability to teach reading
55:20
effectively? And have they been compared with
55:22
something else? No single study provides irrefutable
55:29
proof of anything. Research is a lot like sighting
55:36
in your deer rifle. I grew up in a small town
55:39
in a deer hunting culture. and every fall we
55:43
would sight in our rifles to see if the rifle
55:46
was shooting straight. The scope often got bumped
55:48
over the winter and we wouldn't take a single
55:51
shot. We would take a lot of shots to give us
55:54
a sense of where the scope was shooting because
55:58
we could have wiggled or there could have been
56:00
something that happened with the table that gave
56:03
us an extreme shot. By taking many shots, we
56:07
get a sense of where the scope is shooting, and
56:10
then we adjust our scope. But even then, if we're
56:14
pretty certain of where it's shooting, even though
56:16
we start walking in the woods, it's the person
56:19
behind the scope that determines where the bullet
56:22
lands and a bunch of other variables. No single
56:27
study. Research is like that. We can't find one
56:30
study and say, ah, that proves that. That's not
56:34
how research works. Research does not prove or
56:39
debunk. And by the way, whole language has not
56:44
been debunked. Yes, there's been people that
56:47
say it's been debunked. In the name of almighty
56:50
research, I debunked you, B debunked you. It
56:55
hasn't been debunked just because people say
56:59
that. Okay? Well, I could debunk things. I could
57:03
debunk the theory of gravity. There it is. It's
57:05
been debunked. I debunked it. And here's my proof.
57:09
You can't just go around debunking things. Research
57:13
answers a specific research question. Research
57:19
proves or disproves a hypothesis. And we have
57:24
to be wary of research that doesn't have a specific
57:28
research question. or purpose, and that is true
57:32
of Louisa Moats's stuff. Can't use the word proven
57:37
with science. Observations are never perfect.
57:43
Interpretations are never perfect. Truth needs
57:48
a context and information is often complete.
57:52
Scientists rarely use that word. Here's an example.
57:56
Phonics instruction is good for increasing phonics
58:00
skills. Absolutely. You teach a bunch of phonics
58:04
and students are going to score higher on phonics
58:08
measures. Absolutely. Absolutely. Phonics instruction
58:13
has shown to be more effective in raising scores
58:16
on measures of phonics than no phonics instruction.
58:19
Absolutely. prove that all students need lots
58:25
of phonics instruction? Something can be technically
58:30
true and conveniently inaccurate at the same
58:33
time. This idea that phonics should be the center
58:37
of reading instruction, that is based on a lot
58:40
of I thinkisms. This does not mean all students
58:44
need heavy doses of phonics. Or students would
58:48
be better readers if they get more phonics. Or
58:52
more phonics is better than less phonics. Or
58:56
phonics is the key variable in learning to read.
59:00
Anything other than phonics instruction should
59:03
be excluded. And phonics should be taught at
59:07
all ages and levels. Phonic skills transfer to
59:12
authentic reading situations. These are all assumptions.
59:17
And these research is based on the assumption
59:21
that reading is sounding out words. Research
59:25
based on sounding out words focuses on sounding
59:29
out words and includes sounding out word measures.
59:34
If you ask sounder out questions, you use sounder
59:37
out measures, you collect sounder out data, you
59:40
are going to make sounder out conclusions. That's
59:44
what science of reading research is. Research
59:50
is not collecting a bunch of data and saying
59:52
stuff. Research must be based on a specific question
59:56
or hypothesis. When you read a research study
59:59
after the abstract, the first thing you should
1:00:02
look for is that research question. And research
1:00:07
always... focuses on a specific group at a specific
1:00:12
location doing a specific task. That's why we
1:00:15
can't use one study. For example, what works
1:00:22
with a first grade student in Blackwater, Arizona
1:00:25
struggling with word recognition may not work
1:00:29
with a fifth grader in Edina, Minnesota struggling
1:00:32
with comprehension. Human beings are not standardized
1:00:38
products. Readers are not standardized products.
1:00:41
Students are not standardized products. Again,
1:00:46
when you read the research, skip the literature,
1:00:52
look for the question. and these are the big
1:00:58
three. All right, I have seen that this is 60
1:01:02
minutes. I am going to stop my presentation here.
1:01:05
I will host this presentation on my sub stack,
1:01:09
the reading instruction show, and I'll post the
1:01:12
rest of this. Those who are still here, I thank
1:01:17
you for listening. I am Andy Johnson.