Latest / Stuff You Should Know / Research Bias: Sort It Out, Science

Transcript

Research Bias: Sort It Out, Science · Stuff You Should Know

0:01
Speaker 1: Welcome to Stuff You Should Know, a production of I
0:04
Speaker 1: Heart Radio. Hey, and welcome to the podcast. I'm Josh Clark,
0:13
Speaker 1: and there's Charles w Chuck, Bryan, and Jerry's here. Jerry's back.
0:16
Speaker 1: Everybody looking well rested and sun kissed and everything, and
0:22
Speaker 1: this is stuff you should know. She's like a beautiful,
0:25
Speaker 1: juicy orange. That's right. That's right, Chuck, that's a really
0:31
Speaker 1: apt description. Ready to be squaws. I wish I could
0:34
Speaker 1: squeeze her, but we're still not squeezing. Uh no, not
0:38
Speaker 1: in this pandemic. Are you crazy? You out of your mind? No,
0:43
Speaker 1: no squeezing. Um Yeah, even Robert, Robert Plant wouldn't let
0:48
Speaker 1: you anywhere near him. Okay, check figure out that joke.
0:53
Speaker 1: Robert the squeeze my my lemon, the lemon song, No
1:01
Speaker 1: until the juice runs down my leg. Yeah, that's the
1:05
Speaker 1: lemon song. Right. No, I don't think so. I think
1:10
Speaker 1: it is. I don't think it is, Okay, I think
1:14
Speaker 1: it's a I don't think it is the lemon song, man.
1:17
Speaker 1: I think it's a whole lot of love. It's whole
1:18
Speaker 1: lot of love, all right. It's maybe the dirtiest thing
1:22
Speaker 1: that was ever said in like a top ten song. Okay, regardless,
1:29
Speaker 1: I'll just let the emails take care of this, I
1:31
Speaker 1: really think. No, the Lemon song is ums I love
1:36
Speaker 1: those lemons, right, No, the Lemon song is all about
1:41
Speaker 1: how how how you have friends, like you want to
1:45
Speaker 1: have friends, and like friends are good to have. Okay, yeah,
1:50
Speaker 1: may be all wrong. No, I think it's a whole
1:52
Speaker 1: lot of love. Yeah it is. I'm a dcent sure buddy.
1:56
Speaker 1: All right, well I encourage you not to google the lyrics.
1:59
Speaker 1: Then well we um we could ask our good friend
2:02
Speaker 1: and um stuff. You should know writer Ed Grabmanowski the
2:05
Speaker 1: grab Stir. Look at that because he is in a
2:09
Speaker 1: band and has been for a while. We've mentioned it before,
2:11
Speaker 1: space Lord, which has just a super cool Zeppelin esque
2:16
Speaker 1: sound to them. Um and they just there and um
2:21
Speaker 1: they just released a new single, which you can find
2:24
Speaker 1: on band camp by searching space Lord. Not the space Lords. No,
2:30
Speaker 1: not Vinny in the space Lords. Yeah space Lord. Just
2:34
Speaker 1: look for space Lord with some cool uh graphics, and
2:38
Speaker 1: that'll you know that's Ed. You'll know it's the Grabster
2:41
Speaker 1: but uh yeah, good stuff. We also have a game
2:46
Speaker 1: out that Trivial Pursuit made. Yeah, we should plug our
2:48
Speaker 1: own stuff every now and then. We just did. Yes,
2:51
Speaker 1: it is a co branded game with Trivial Pursuit from Hasbro,
2:55
Speaker 1: and it is not a trivial pursuit game that you
2:59
Speaker 1: are used to. It is a stuff you should know
3:02
Speaker 1: game that Trivial Pursuit was happy to UH co brandwith.
3:07
Speaker 1: So just what, I don't want his emails that are
3:10
Speaker 1: like this, this this is a Trivial Pursuit. This is some
3:12
Speaker 1: other different game. You're always worried about the emails, aren't you.
3:16
Speaker 1: Don't you just ignore them? Let him roll off, roll
3:19
Speaker 1: off my back, Like, I'm disappointed in you guys for this.
3:23
Speaker 1: I haven't even listened to the episode, but I'm disappointed
3:26
Speaker 1: about that. I just got one of those. Did you
3:27
Speaker 1: see that It just rolls off your back? Yeah, yeah,
3:30
Speaker 1: those are those are always great. I didn't listen, But
3:34
Speaker 1: here's what was wrong. I read that person back. Actually,
3:37
Speaker 1: I was like, we actually kind of did exactly what
3:40
Speaker 1: you hoped we would do. And they're like, oh, sorry
3:42
Speaker 1: for being presumptuous anyway, Oh all is forgiven. So, um,
3:49
Speaker 1: we're talking today about bias Chuck, and I want to
3:53
Speaker 1: set the scene a little bit because you know, um,
3:57
Speaker 1: one of the things that I'm always like harping up
4:00
Speaker 1: about is like the death of expertise, right, And it's
4:04
Speaker 1: a real problem, like this idea that science can't be trusted,
4:07
Speaker 1: that people who go and spend a decade or more
4:10
Speaker 1: learning about a specific thing that they go out and
4:14
Speaker 1: become an expert in or that's their profession, that's their training,
4:19
Speaker 1: um that those people what they have to say is
4:22
Speaker 1: basically meaningless, or that it's it's no better than somebody
4:26
Speaker 1: on the internet's opinion about so that specific subject that
4:29
Speaker 1: that person spent ten or twelve years being trained to
4:32
Speaker 1: be an expert in. Like that kind of stuff to
4:34
Speaker 1: me is like super dangerous, Like it it's it's in
4:38
Speaker 1: a there's an erosion of something, and it's a an
4:40
Speaker 1: erosion of intelligence to start with, but it's also an
4:43
Speaker 1: erosion of just believing in facts and knowing that you're
4:48
Speaker 1: not being taken for a ride or hustled. And is
4:51
Speaker 1: it a huge enormous problem that we're just beginning to
4:55
Speaker 1: wake up too and is still unfolding. It's not like
4:57
Speaker 1: it happened and now we're like real from it. It's
5:00
Speaker 1: still happening in real time, and it is a massive,
5:05
Speaker 1: huge issue one of the biggest issues that that humanity faces,
5:09
Speaker 1: I think because it encompasses so many other large issues
5:14
Speaker 1: like climate change, existential risks, the pandemic, um politics, all
5:20
Speaker 1: of them kind of fall under this this erosion of
5:23
Speaker 1: belief and facts and that there are people out there
5:26
Speaker 1: who know more than you do. Um, it's a big problem. Yeah,
5:30
Speaker 1: imagine being someone who studied and researched something intensely for
5:35
Speaker 1: ten or fifteen years, uh that with when presenting facts
5:40
Speaker 1: to be met with. I don't know about that. That's
5:43
Speaker 1: a response I hear a lot in the South. Yeah,
5:47
Speaker 1: or that they saw something on YouTube that flatly contradicts that,
5:50
Speaker 1: and it's like that it doesn't matter what you just said.
5:53
Speaker 1: Is ridiculous that somebody posted something on YouTube and that
5:57
Speaker 1: that like that has as much weight is what somebody
6:01
Speaker 1: who spent ten or twelve year studying this very thing
6:04
Speaker 1: has to say about it, like knows exactly what they're
6:07
Speaker 1: talking about, has to say about it. It's it's maddening. Yeah,
6:11
Speaker 1: there's there's something about people from the South in general,
6:14
Speaker 1: and I think that are in this group that I
6:17
Speaker 1: have literally heard that response from a lot of different
6:20
Speaker 1: people when I've been like oh no, no no, no, here
6:22
Speaker 1: the facts actually, and then when presented with something that
6:26
Speaker 1: they can't refute, they say, I don't know about that,
6:29
Speaker 1: and like that's it. That's the end of the conversation.
6:31
Speaker 1: That's different than the people I've encountered. The people I
6:33
Speaker 1: encountered like their brow furrows and they start pointing fingers
6:36
Speaker 1: and their their tone goes up, like you are you
6:39
Speaker 1: hanging out at the country club or something. I think
6:42
Speaker 1: it's different types of people that, you know, there's ignorance,
6:45
Speaker 1: and then there's also people that actually think they're better
6:49
Speaker 1: informed that will fire back with YouTube clips. Right. So
6:53
Speaker 1: the reason I brought that up is because and one
6:56
Speaker 1: of the reasons that that is being allowed to exist,
6:59
Speaker 1: that that does exist, I think it's a it's a
7:01
Speaker 1: reaction to something else that's going on simultaneously, which is
7:07
Speaker 1: there are a lot of experts out there who are
7:09
Speaker 1: performing really sloppy science, sometimes outright fraudulent science, and they're
7:18
Speaker 1: frittering away whatever faith the general public or society has
7:23
Speaker 1: in their expertise and in their profession. And there are
7:26
Speaker 1: a ton of scientists out there. I would say the
7:29
Speaker 1: vast majority, by far of scientists are legitimate upstanding, upright
7:35
Speaker 1: dedicants to science, right, That's where they that's where they
7:39
Speaker 1: place there, that's where they hang their hat, that's where
7:41
Speaker 1: their heart is, that's that's what they believe in, and
7:44
Speaker 1: that's what they work to support. But science has like
7:48
Speaker 1: kind of a problem, Chuck in that it's allowing way
7:52
Speaker 1: too much for bias, which is what we're gonna talk about,
7:56
Speaker 1: to creep into science, um and your mind science and
8:01
Speaker 1: basically produce papers that are just useless and trash. And
8:05
Speaker 1: there's a whole lot of reasons for it, but it's
8:07
Speaker 1: a it's something that needs to be addressed if we're
8:10
Speaker 1: ever going to get back on a footing with a
8:13
Speaker 1: faith in experts and expertise and just facts that there
8:16
Speaker 1: are such things as objective facts. Yeah, I mean a
8:19
Speaker 1: lot of times it's financially related, whether it's a lack
8:23
Speaker 1: of funding, a desire for more funding, a desire just
8:27
Speaker 1: to keep your your uh, your lab running and people
8:31
Speaker 1: paid on staff. Which you know, all this stuff is understandable.
8:35
Speaker 1: You want to keep doing this work, but you can't
8:38
Speaker 1: let that get in the way. It's like it's like
8:41
Speaker 1: in Rushmore at the end when Margaret Yang faked the
8:45
Speaker 1: results of that science experiment because she didn't want it
8:49
Speaker 1: to be wrong. You know, I don't remember what um,
8:52
Speaker 1: I don't remember that part was that like a deleted scene? No? No, no,
8:55
Speaker 1: was it in the end when they meet up and
8:57
Speaker 1: he's flying the Uh. I think he's flying the kite
9:00
Speaker 1: with Dirk and she's talking about her science fair project
9:04
Speaker 1: and he was really impressed with it, and she was like,
9:06
Speaker 1: I fake the results. And the reason why was because
9:09
Speaker 1: because she didn't want to be wrong. Uh. And I
9:12
Speaker 1: think a lot of times people will get into a
9:15
Speaker 1: certain body of research or data too because they want
9:19
Speaker 1: to prove a certain thing and if they can't, it
9:22
Speaker 1: might be really hard to live with that. So that
9:24
Speaker 1: weighs into it. Uh, money for personal gain, uh, advancing
9:28
Speaker 1: your career, you know, publisher, parish, that whole thing. Like,
9:31
Speaker 1: we're gonna talk about all this, but there are a
9:33
Speaker 1: lot of reasons that it's been allowed to creep in,
9:37
Speaker 1: But all of it is at the disservice of their
9:40
Speaker 1: the fundamentals of what they base their careers on to
9:43
Speaker 1: begin with. Yeah, it's at the it's at the disservice
9:46
Speaker 1: of science itself, right, because the whole point of science
9:49
Speaker 1: and then the scientific publishing, the whole publishing industry, um
9:53
Speaker 1: is to to basically create a hypothesis, test your hypothesis,
9:58
Speaker 1: and then share the results with the world. And that's
9:59
Speaker 1: I deally what would happen because you're building this body
10:02
Speaker 1: of scientific knowledge. But money and corporate interests and academic
10:07
Speaker 1: publishing have all kind of come in and taken control
10:11
Speaker 1: of this whole thing, and as a result, a lot
10:14
Speaker 1: of the stuff that gets published are trash papers that
10:17
Speaker 1: shouldn't be published. A lot of the really good papers
10:20
Speaker 1: that don't come up with sexy results don't get published.
10:25
Speaker 1: And then, like you said, um, people using science for
10:29
Speaker 1: personal gain. There are a very small chadra of thoroughly
10:36
Speaker 1: evil people who are willing to use their scientific credentials
10:41
Speaker 1: to create doubt in the general public, to to prevent
10:46
Speaker 1: like people from understanding that climate change was real for
10:51
Speaker 1: twenty years, or um, that fossil fuels actually do contribute
10:56
Speaker 1: to to anthropogenic climate change. But what remains focusing on
11:01
Speaker 1: is like bias in the in the sense that people
11:05
Speaker 1: carrying out studies are human beings, and human beings are flawed.
11:11
Speaker 1: We're just flawed, and we bring those flaws to our studies,
11:14
Speaker 1: and that you really have to work hard at rooting
11:19
Speaker 1: those flaws and those biases out to produce a really good,
11:23
Speaker 1: thorough scientific study with good reliable results that can be
11:27
Speaker 1: reproduced by anybody using the same methods. Um and that
11:32
Speaker 1: science is just starting to wake up to the idea
11:35
Speaker 1: that it is really biased and it needs to take
11:38
Speaker 1: these into account in order to to progress forward from
11:41
Speaker 1: the point that it's at right now, which is tenuous,
11:44
Speaker 1: I think, perhaps more tenuous than ever the point that
11:48
Speaker 1: science is at. I think. So science isn't going away.
11:52
Speaker 1: It's not going anywhere. It's probably the greatest of course
11:54
Speaker 1: humans have ever come up with. Right, It's not going anywhere,
11:57
Speaker 1: But it is a terrible position that it's in, and
12:00
Speaker 1: it's going to take some genuine leadership in the scientific
12:04
Speaker 1: community from a bunch of different quarters in a bunch
12:07
Speaker 1: of different fields to basically step up and be like, guys,
12:09
Speaker 1: this is really bad and we need to change it now,
12:13
Speaker 1: and a lot of people need to be called out.
12:15
Speaker 1: In science typically shies away from naming names and calling
12:19
Speaker 1: out by name fraudulent scientists because scientists seem to like
12:24
Speaker 1: to um suppose the best in people, which is not
12:28
Speaker 1: always the case, right, And having said all of this,
12:31
Speaker 1: there could we could root out every bias and and
12:34
Speaker 1: and really clean up the scientific publishing community. And there's
12:40
Speaker 1: still a set a certain set of people in this
12:42
Speaker 1: country and in the world who that wouldn't matter to
12:45
Speaker 1: and would still shut down facts and because it doesn't
12:49
Speaker 1: fit their narrative, so for sure, But the people have
12:52
Speaker 1: always they've always been there, right, and they're always going
12:56
Speaker 1: to be there. There's always it's just countrarians that they
12:59
Speaker 1: are you can call, and free thinkers you can call them, stubborn,
13:01
Speaker 1: you can call purposefully, purposefully ignorant who knows they're always
13:07
Speaker 1: going to exist. The problem that this crisis that science
13:11
Speaker 1: finds itself in right now is that it's allowed that
13:15
Speaker 1: that population to grow and grow, and like people who
13:18
Speaker 1: otherwise didn't never really question science have been allowed to
13:21
Speaker 1: kind of trickle into that fold. And that those are
13:24
Speaker 1: the people that we should be worried about, the ones
13:26
Speaker 1: who would would know better if they believed in science again, right,
13:31
Speaker 1: And our way into this is to talk about different
13:33
Speaker 1: kinds of biases in true stuff. You should know fashion
13:36
Speaker 1: a top ten that is not a top ten. That's
13:39
Speaker 1: exactly right. We ate into at least three in this
13:42
Speaker 1: intro and hopefully shining a light on some of this stuff.
13:45
Speaker 1: People at least be more aware of different biases. And uh, well, yeah,
13:49
Speaker 1: you know. The first one is is good old confirmation bias.
13:52
Speaker 1: I mean, these aren't ranked because confirmation bias would probably
13:55
Speaker 1: be number one as far as people's awareness of it.
13:59
Speaker 1: But there are different examples, um that people use for
14:03
Speaker 1: confirmation bias. And I kind of enjoyed the one from
14:06
Speaker 1: the house Stuff Works article even though it was from
14:09
Speaker 1: N three. After X rays were discovered in Germany, there
14:13
Speaker 1: was a French scientist named Renee Blond Blonde Lot. Yeah,
14:18
Speaker 1: he looked at as the X rays he said, wow,
14:21
Speaker 1: Well who said, ay, I see N rays I've discovered
14:25
Speaker 1: in rays And everyone's like, what's an N ray? He said, well,
14:28
Speaker 1: it's like a corona when electricity discharges from a crystal
14:32
Speaker 1: and you can only see it in your peripheral vision.
14:35
Speaker 1: And American Robert Wood laid the wood and said, I'm
14:40
Speaker 1: gonna come to your lab and check this out and
14:43
Speaker 1: secretly remove the crystals during one of the experiments, and
14:48
Speaker 1: Blonde Lot still saw these N rays and so that's
14:52
Speaker 1: confirmation bias. He wanted to see those in rays. And
14:54
Speaker 1: then later, even though it was disproved, other French scientists
14:58
Speaker 1: supposedly published papers or published papers based on that research
15:02
Speaker 1: because they wanted it to be true. So that's what
15:05
Speaker 1: confirmation biases is when you're starting out with a hypothesis
15:10
Speaker 1: that is going to shape the methodology methodology of your
15:14
Speaker 1: study to to confirm it right. And then it can
15:17
Speaker 1: also occur where you're um. You you're interpreting info um
15:23
Speaker 1: to fit your hypothesis, so you're seeking out stuff that
15:26
Speaker 1: supports your hypothesis, and then the stuff that you is
15:29
Speaker 1: that's just there in front of you, the results, they
15:31
Speaker 1: are there in front of your like, ah, this thing
15:33
Speaker 1: proves that those end rays actually exist, um or this
15:37
Speaker 1: phenomenon cannot be due to anything but end raise. Therefore
15:41
Speaker 1: end Raise exists all of its confirmation bias. And that,
15:43
Speaker 1: like you said, that's number one, because that's not just
15:46
Speaker 1: a scientific bias. I mean, like that is that's like
15:50
Speaker 1: every human uses confirmation bias, and that it's two fold.
15:54
Speaker 1: We we avoid contradictory information because we UM I don't
15:59
Speaker 1: like to be wrong, and we find information that confirms
16:03
Speaker 1: our point of view because we like to be right.
16:04
Speaker 1: That's it's confirmation bias, and it's everywhere among everyone. That's right,
16:10
Speaker 1: Although I will say I know it happens a lot politically,
16:14
Speaker 1: but myself and the people that I congregate with, uh
16:19
Speaker 1: question their own leaders as much as they do leaders
16:23
Speaker 1: from the other parties. Oh, it's good, it's very good
16:27
Speaker 1: to do. And I don't know, there shouldn't be sacred
16:29
Speaker 1: calves and politics. That's a bad jam, well know. And
16:32
Speaker 1: it's like I've always being been like at the forefront
16:35
Speaker 1: of calling out my own parties wrongs and saying no, no, no,
16:40
Speaker 1: that's you need to do better than that, whereas I
16:43
Speaker 1: see a lot of other people in other situations truly
16:46
Speaker 1: bury and ignore those things because they don't you know,
16:50
Speaker 1: I just don't want to face that. Yeah. And it's
16:53
Speaker 1: not like it's not even like I don't want to
16:55
Speaker 1: face it. It just doesn't fit their worldview, so they
17:00
Speaker 1: just don't include it. It It just gets tossed out. But
17:02
Speaker 1: the point is is like it's not an active process necessarily, right,
17:10
Speaker 1: I think we should probably check our first break. I
17:12
Speaker 1: think so too. Chi All right, we'll be right back
17:14
Speaker 1: and talk about sampling bias right after this alright, Chuck,
17:38
Speaker 1: we're back, and we're coming back with UM, something called
17:41
Speaker 1: sampling bias, which is it turns out a sub type
17:45
Speaker 1: of a larger thing called selection bias, and one other
17:48
Speaker 1: thing we should say. We kind of got into it
17:50
Speaker 1: before I could say this. There are different stages in
17:53
Speaker 1: a study where bias can occur. It can happen in
17:57
Speaker 1: like the planning the pre study phase, UM and and uh,
18:02
Speaker 1: it can happen during the actual study, and then it
18:05
Speaker 1: can happen after the study as well. And so when
18:08
Speaker 1: we're talking about any kind of selection bias, including sampling bias,
18:12
Speaker 1: this is pre study bias where when you're actually setting
18:16
Speaker 1: up the study, this bias is where this is where
18:19
Speaker 1: the bias is going. Yeah, and you know what, I
18:20
Speaker 1: think it also bears saying that biases you have to
18:23
Speaker 1: work really hard to avoid it because it's it's almost
18:26
Speaker 1: like a disease that's always trying to get involved. And
18:30
Speaker 1: it's not like just do better, everybody and quit being biased.
18:34
Speaker 1: It's like it's way more complicated than that, because it
18:36
Speaker 1: is always knocking at the door, like you said, in
18:39
Speaker 1: all three phases, trying to sneak in there, and it
18:42
Speaker 1: takes a lot of work in all three phases to
18:44
Speaker 1: avoid it. So it's not as I don't want it
18:47
Speaker 1: to come across this as easy as us just saying
18:49
Speaker 1: like you shouldn't do that, stop it. No, But the
18:54
Speaker 1: first step is to recognizing that there's a lot of
18:56
Speaker 1: bias and different kinds of bias that are just sitting
19:00
Speaker 1: there waiting for a scientist. And then if you start
19:03
Speaker 1: admitting that it's there, you can start being on the
19:05
Speaker 1: lookout for it, and you can start adjusting for it,
19:07
Speaker 1: and then other people who read your papers or here
19:10
Speaker 1: you know, read news articles about your papers, can be
19:12
Speaker 1: on the lookout for that kind of thing. Yeah, so exactly, Uh,
19:16
Speaker 1: sampling bias is your you know, your sample set not
19:20
Speaker 1: being accurate and a good representation of the whole. A
19:23
Speaker 1: lot of times you'll find this um in either studies
19:28
Speaker 1: that are really small scale because you don't have a
19:31
Speaker 1: large sample and you don't have the kind of money
19:34
Speaker 1: like near you. Like maybe you work for university, so
19:36
Speaker 1: you work with university students as your first sample set,
19:39
Speaker 1: who are not indicative of anything, but you know, people
19:43
Speaker 1: eighteen to twenty one years old or so. Now, remember
19:47
Speaker 1: we talked about weird Western educated and realized rich and democrats.
19:52
Speaker 1: That's exactly the thing. It's like, I mean, it's a
19:54
Speaker 1: it's a decent place to start if you don't have
19:57
Speaker 1: much money and you want to get the ball rolling.
20:00
Speaker 1: It's not like, oh, you shouldn't do university studies at
20:02
Speaker 1: all and using students, but those findings definitely don't represent
20:07
Speaker 1: the wide nation and it needs to grow and get
20:10
Speaker 1: more funding if you want to actually have a legitimate
20:13
Speaker 1: claim to something. Another way that sampling bias can come
20:17
Speaker 1: up is from like the group that you're recruiting from.
20:20
Speaker 1: Like if you're doing a strictly online survey, but you're
20:25
Speaker 1: trying to apply your findings to the wider society, that's
20:29
Speaker 1: just not gonna happen because there's so many people who
20:32
Speaker 1: aren't Internet savvy enough to take an Internet survey. Like,
20:35
Speaker 1: by by nature, you are a little savvier than the
20:38
Speaker 1: average person if you're hanging out on the Internet and
20:40
Speaker 1: taking a survey. And then also kind of tangential that
20:44
Speaker 1: I like to tell myself that at least um and
20:47
Speaker 1: then tangential that is something called self selection bias, which
20:50
Speaker 1: is where the people who say, let's say you're doing
20:53
Speaker 1: a study on wellness and you know what eating tuna
20:58
Speaker 1: can do for your health. Um, people who are interested
21:02
Speaker 1: in wellness and health are going to be much more
21:05
Speaker 1: likely to volunteer for that study than people who couldn't
21:09
Speaker 1: care less about health and have no desire whatsoever to
21:11
Speaker 1: further sciences understanding of what makes you healthy. So you
21:15
Speaker 1: would have to go out and find those people and
21:18
Speaker 1: recruit them rather than just relying on the people who
21:21
Speaker 1: volunteered based on the flyer you put up in the
21:23
Speaker 1: student right, or you know, study all financial demographics or
21:28
Speaker 1: pull all financial demographics rather than just and you know,
21:33
Speaker 1: sometimes it's a methodology in which they do try and
21:36
Speaker 1: recruit people steers them in that direction unknowingly that I know.
21:40
Speaker 1: In the article they've talked about the presidential campaign with
21:45
Speaker 1: Roosevelt and alf Landon Republican alf Landon, they were doing
21:49
Speaker 1: polling with like country clubs, rosters and uh, people who
21:54
Speaker 1: drove cars and stuff. At the time that was kind
21:56
Speaker 1: of a luxury, so it was all out of whack.
21:58
Speaker 1: Everyone's like Landing's gonna win in a landslide. It's because
22:01
Speaker 1: you just kind of basically stuck your polling to uh,
22:06
Speaker 1: you know, I don't know about wealthy individuals, but people
22:08
Speaker 1: who are a little more well off. And I think, um,
22:10
Speaker 1: we talked about that in our polling episode, that is
22:14
Speaker 1: that fiasco with polling. I also saw one more two
22:18
Speaker 1: that I want to mention because that has a really
22:19
Speaker 1: great anecdote attached to It's called survivorship bias, where when
22:23
Speaker 1: you're studying something, say like business or something, you're you're
22:28
Speaker 1: probably going to just be looking at the extant businesses,
22:30
Speaker 1: the businesses that have survived twenty years, thirty years, fifty
22:33
Speaker 1: years or something like that, and you're not taking into
22:35
Speaker 1: account all of the failures. So when you put together
22:38
Speaker 1: like a prognosis for business in America, it might have
22:41
Speaker 1: a son your outlook than it should, because all you're
22:44
Speaker 1: looking at are the ones that managed to survive and thrive.
22:47
Speaker 1: And that's survivorship bias. And did you see that The
22:50
Speaker 1: anecdote about the World War Two fighter pilots. It was
22:53
Speaker 1: actually pretty funny because they studied uh planes that had
22:57
Speaker 1: been returned, that had been fired upon managed to get
23:01
Speaker 1: back safely, and they're like, well, let's look at all
23:03
Speaker 1: these different bullet holes and where this plane was hit,
23:07
Speaker 1: and let's beef up all those areas on the body,
23:10
Speaker 1: and a mathematician named Abraham Wald said, uh, no, those
23:14
Speaker 1: are the places where they got shot and did Okay,
23:17
Speaker 1: what you should really do is find these planes that
23:19
Speaker 1: actually went down and beef up those sections of the planet.
23:23
Speaker 1: And that survivorship bias. It's just it's failing to take
23:26
Speaker 1: into account the failures and that have to do with
23:28
Speaker 1: what you're trying to study. What about channeling bias. Channeling
23:34
Speaker 1: bias is another kind of selection bias. Did you get
23:37
Speaker 1: this one? It wasn't the best example of channeling bias. Yeah,
23:40
Speaker 1: I mean that's I got it, all right? Did you
23:45
Speaker 1: not get it? I got it, but it took a
23:47
Speaker 1: lot of work before I finally did. Well. It's basically
23:50
Speaker 1: when let's say you have a patient and they're like,
23:54
Speaker 1: their degree of illness might influence what group they're put into.
23:58
Speaker 1: So if a doctor or if a surgeon was trying
24:01
Speaker 1: to study outcomes of obsticular surgery, they might because they're
24:06
Speaker 1: surgeons and they want to help people out, they may
24:09
Speaker 1: perform that surgery on maybe younger, healthier people who might
24:13
Speaker 1: have better outcomes than someone who is in a different,
24:16
Speaker 1: like higher age group. Right, And the the article is
24:21
Speaker 1: kind of ends it there, and I was like, so
24:24
Speaker 1: what's the problem. And I finally found this example where
24:29
Speaker 1: where it says like, Okay, let's say you're studying a
24:32
Speaker 1: new heartburn medicine or something or something to treat like
24:36
Speaker 1: like gurd and it's new, it's hardcore, it's cutting edge UM.
24:42
Speaker 1: And the people who are likely is to get this
24:45
Speaker 1: new hardcore UM and acid are the ones who are
24:48
Speaker 1: probably in worse shape, right, so, say they're on the
24:51
Speaker 1: verge of going to the e ER. Anyway, well, if
24:54
Speaker 1: you look back at all of the people who have
24:56
Speaker 1: ever been prescribed this new hardcore and acid, you're gonna
24:59
Speaker 1: see like a lot of them ended up in the
25:02
Speaker 1: e R, even though it had nothing to do with
25:04
Speaker 1: this hardcore and acid. And then similarly, the people who
25:08
Speaker 1: have so so gird it's not particularly bad, they'll probably
25:12
Speaker 1: be prescribed the old drug, the standby that everybody knows
25:15
Speaker 1: it's fine, that's going to work. So if you compare
25:17
Speaker 1: the old drug and the new drug, it looks like
25:19
Speaker 1: the old drug is super safe, but the new drug
25:21
Speaker 1: will put you into the e R. Whereas UM, that's channeling.
25:25
Speaker 1: You've channeled different different people with different prognoses into different groups,
25:29
Speaker 1: and they're kind of pitted against each other, um in
25:33
Speaker 1: a in an effort to obscure the truth. If you
25:35
Speaker 1: wanted to really know the genuine health outcomes for that
25:38
Speaker 1: an acid, you would have to give it to people
25:41
Speaker 1: with not so bad gurd and people with really bad
25:43
Speaker 1: gurd and see what happens. See if the e ER
25:46
Speaker 1: visits continue for people who wouldn't otherwise be going to
25:50
Speaker 1: the e R Everyone, right, and not just for you,
25:56
Speaker 1: Like if you're debating surgery and you're like, oh, well
25:58
Speaker 1: it says shows really good outcomes. We're like, well yeah,
26:01
Speaker 1: but who are they operating on right? Right? Yes? So
26:06
Speaker 1: I would like to invite anyone UM who got what
26:10
Speaker 1: I was saying or got channeling because of what I
26:14
Speaker 1: was saying, I invite you to email and let me know.
26:16
Speaker 1: I'm doing like a little bit of surveys here, and
26:18
Speaker 1: I'd like to know if I confuse things more or
26:22
Speaker 1: make it more understandable. Well, but I know it's funny
26:25
Speaker 1: either way. I got that part. I'm just trying to
26:27
Speaker 1: figure out its understanding. But here with your methodology talking
26:30
Speaker 1: about the stuff you should know. No listener who by
26:32
Speaker 1: nature is smarter than your average beyer. Well, I'm not
26:35
Speaker 1: going to publish it. I'm gonna file draw it either way.
26:40
Speaker 1: Oh what a teaser. UH question order bias is the
26:45
Speaker 1: next one that is uh. And this is mainly obviously
26:49
Speaker 1: with UM when you're just doing uh, like polling and
26:53
Speaker 1: stuff or like an online survey or you know, it
26:56
Speaker 1: could be just asking people a set of questions like
26:58
Speaker 1: in a social science research set, and the way you
27:03
Speaker 1: order things can affect the outcome. And this is the
27:06
Speaker 1: thing at all, like everything from the brain's tendency to
27:10
Speaker 1: organize information into patterns to the brain uh, simply paying
27:14
Speaker 1: attention more and being more interested early on. Like I
27:19
Speaker 1: know there was one The General Social Survey was a
27:22
Speaker 1: big long term study of American attitudes and in they
27:26
Speaker 1: were asked to identify the three most important qualities for
27:29
Speaker 1: a child to have, and they were given a list
27:31
Speaker 1: of these qualities. Honesty was just listed higher on the list.
27:36
Speaker 1: When it was it was pick sixty six percent of
27:38
Speaker 1: the time. When it was further down on the list,
27:40
Speaker 1: it's forty eight percent important. And that's simply because people
27:44
Speaker 1: are just reading this list that are like important and
27:47
Speaker 1: then yeah, by the time they got down, you know,
27:50
Speaker 1: three quarters of the way through the list, they started
27:52
Speaker 1: thinking about what they're gonna have for dinner. People get
27:56
Speaker 1: pooped when you're giving them lists of stuff, or you
27:59
Speaker 1: can people and get them all sort of worked up.
28:02
Speaker 1: Like if you have a question like, uh, during the
28:06
Speaker 1: Trump administration, how mad were you at that guy about
28:09
Speaker 1: stuff he did? And you're like super mad? And then
28:13
Speaker 1: you were like, well, how did you feel generally about
28:15
Speaker 1: how your life was affected during his administration? You might
28:20
Speaker 1: say it was awful, Whereas if they hadn't have asked
28:23
Speaker 1: that first question, they were just like, what was your
28:25
Speaker 1: life like from two thousand and I am blocking out
28:28
Speaker 1: the dates. What you might say, well, you know, it
28:33
Speaker 1: wasn't It was okay, man, I had a lot of
28:38
Speaker 1: sandwiches just over those four years though. Um. Yeah, so,
28:43
Speaker 1: and like you said, that's priming, which is a big
28:46
Speaker 1: it's a big thing that you have to worry about
28:47
Speaker 1: when you're doing any kind of survey. UM. So there's
28:50
Speaker 1: some of the ways that you can combat that. You
28:52
Speaker 1: can randomize your question order. Um. Sometimes you'll have a
28:55
Speaker 1: survey where one question is predicated on a previous question.
29:00
Speaker 1: So one thing you might want to do is UM
29:03
Speaker 1: ask that set of questions in a few different ways. UM.
29:07
Speaker 1: So that yeah, so that you can kind of maybe, um,
29:12
Speaker 1: compare the answers to all three, at them up and
29:14
Speaker 1: divide them by three and there's your average answer kind
29:17
Speaker 1: of thing. Um. There's a lot of things you can
29:19
Speaker 1: do to kind of, I guess, um, manipulate to de
29:24
Speaker 1: manipulate your respondent when you're doing a survey like that, manipulate, manipulate.
29:31
Speaker 1: Look it up. You won't find anything on it, but
29:34
Speaker 1: you could look it up still. Oh it's a Roxy
29:37
Speaker 1: music album. Wow, Chuck, Wow, that was great. What's next?
29:47
Speaker 1: So with question order bias, we've entered the during the
29:50
Speaker 1: study kind of bias, like this is this is why
29:53
Speaker 1: you're actually conducting the study, and so is interviewer bias. Um,
29:57
Speaker 1: an interviewer bias. It's kind of like, well, question order
30:02
Speaker 1: biases has to do more with the study design, but
30:04
Speaker 1: it's a bias that emerges during the study interview or
30:08
Speaker 1: bias just straight up is in the middle of the study,
30:11
Speaker 1: and it has to do with the person actually asking
30:14
Speaker 1: the questions in an interview. UM. It can also I
30:18
Speaker 1: think apply to somebody conducting uh, like the like a
30:24
Speaker 1: a clinical trial and a drug. If they know whether
30:27
Speaker 1: somebody is getting placebo or not, it might affect their behavior.
30:31
Speaker 1: But ultimately what it is the the person who's wearing
30:34
Speaker 1: the white lab coat is influencing the outcome of the
30:39
Speaker 1: of the study just through their behavior, through their tone
30:42
Speaker 1: of voice, through the way that they're asking a question.
30:45
Speaker 1: Sometimes it can be really overt and like let's say, um,
30:50
Speaker 1: like a super devout Christian is is, you know, doing
30:53
Speaker 1: a study on whether how many what part of the
30:56
Speaker 1: population believes Jesus saves? And they might be like, you know, Jesus,
31:01
Speaker 1: don't do you think Jesus saves? Is the question? Don't
31:05
Speaker 1: you like? It seems like it? Huh, that kind of
31:07
Speaker 1: thing would be a pretty extreme example, but it's it's
31:10
Speaker 1: sometimes how you understand things is in the absurdities, you know. Yeah,
31:14
Speaker 1: I thought this example in the House of Works article
31:18
Speaker 1: is kind of funny. It was about just like a
31:20
Speaker 1: medical questionnaire where the interviewer knows that the subject has
31:26
Speaker 1: a disease that they're talking about, and they may probe
31:29
Speaker 1: more intensely for the known risk factors. And they gave
31:32
Speaker 1: smoking as an example, and it said so they may
31:35
Speaker 1: say something like, are you sure you've never smoked, never,
31:39
Speaker 1: not even one. Like, if I heard that coming from
31:42
Speaker 1: a researcher, even without knowing a lot about this, would say, what,
31:47
Speaker 1: what kind of a researcher are you? Like it seems
31:49
Speaker 1: like you're looking for an answer you should you should
31:52
Speaker 1: say you are ethically compromised, or even facial expressions or
31:57
Speaker 1: you know, body language. All that stuff weighs in I
31:59
Speaker 1: don't know, tybrow, Like, why don't they just have the
32:03
Speaker 1: robots Alexa or Google or Syria or somebody ask them? Well,
32:07
Speaker 1: that's one good thing about something like an Internet survey
32:10
Speaker 1: is like it's it's just questions, and as long as
32:12
Speaker 1: you design the questions and you randomize their presentation like it,
32:16
Speaker 1: it's gonna be fairly helpful in that respect. But then
32:18
Speaker 1: it's kind of its own pitfalls and pratfalls. You can
32:20
Speaker 1: attract a lot of people who are just taking it
32:22
Speaker 1: to mess with you, and there's a lot of problems
32:25
Speaker 1: with with all of it. But again, if you're aware
32:27
Speaker 1: of all the problems, you can plan for him. And
32:29
Speaker 1: then even if you can't plan for him or control them,
32:32
Speaker 1: you can write about it in the actual study and
32:34
Speaker 1: be like this this study I remember running across studies
32:38
Speaker 1: before where they're basically like, there are you know, there
32:41
Speaker 1: was a kind of bias that we couldn't we couldn't
32:44
Speaker 1: control for, so we can't really we can't really say
32:47
Speaker 1: whether it affected this the outcome or not. And I thought, wow,
32:50
Speaker 1: this is really refreshing and like even daring kind of
32:54
Speaker 1: like I was thrilled. Um, But you don't see that
32:58
Speaker 1: very often. But that, from what I am understand, is
33:00
Speaker 1: the direction that science is going towards now well, and
33:03
Speaker 1: the reason you don't see that. And then something we'll
33:06
Speaker 1: talk about is is what actually ends up getting published.
33:10
Speaker 1: The it may be less likely to get published if
33:12
Speaker 1: they're like, hey, you know what, you know what I'm saying? Yeah,
33:16
Speaker 1: I know. So let's do recall an acquiescence bias because
33:20
Speaker 1: they're very much related, and then we'll take a break.
33:23
Speaker 1: That's our plan. What do you think of it? Everyone
33:27
Speaker 1: says it sounds good to me, all right. So this
33:30
Speaker 1: is also during study, and this is so in the
33:33
Speaker 1: very much the way that an interviewer can influence the outcome,
33:37
Speaker 1: the participant can actually influence the outcome too, especially if
33:41
Speaker 1: they're being asked questions or they're they're being asked to
33:44
Speaker 1: self report. Um, there's a couple of ways that us
33:47
Speaker 1: just being humans can foul up the works on on
33:50
Speaker 1: the findings of a study. The first is recall bias. Yeah.
33:54
Speaker 1: This is when you're obviously you're trying to recall something
33:57
Speaker 1: from the past. And it's amazing what might jump out
34:02
Speaker 1: at you from your past when probed with a certain question,
34:06
Speaker 1: certain correlations that really have nothing to do with it
34:09
Speaker 1: with you. But you may be like, oh, well, you
34:11
Speaker 1: know what, now that I think back, I remember around
34:14
Speaker 1: that time I was I was watching a lot of
34:17
Speaker 1: Dancing with the Stars. I kind of binge that show.
34:20
Speaker 1: So maybe that's why I had homicidal tendencies. I don't
34:26
Speaker 1: think you need to study to prove that. I think
34:29
Speaker 1: that's just intuition, you know. Yeah. Um but yeah, so,
34:32
Speaker 1: and if enough people do that, especially if there's something
34:35
Speaker 1: out kind of in like the zeitgeist about that, how
34:38
Speaker 1: like people who watch too much Dancing with the Stars
34:41
Speaker 1: want to kill other people? Um, like, a number of
34:46
Speaker 1: your participants might recall that thing, whereas other people who
34:50
Speaker 1: don't watch Dancing with the Stars aren't going to recall that.
34:53
Speaker 1: And so in the same way that um, survivorship bias
34:57
Speaker 1: influences that those people who don't have that memory to
35:01
Speaker 1: recall that memory can't possibly be included in the study results,
35:06
Speaker 1: which means that Dancing with the Stars is going to
35:08
Speaker 1: kind of percolate to the top as like a major
35:11
Speaker 1: risk factor in homicidal tendencies. Right, that's not good. You
35:16
Speaker 1: don't want you don't want Dancing with the Stars unfairly canceled.
35:20
Speaker 1: You want to be canceled because it is terrible. I've
35:23
Speaker 1: never seen it. I'm sure it's great if you're not dancing.
35:25
Speaker 1: I haven't either. But watch we're gonna be asked to
35:28
Speaker 1: be on Oh my God, and they'd have to change
35:33
Speaker 1: the name of the show to Dancing with the mid
35:35
Speaker 1: level Internet famous right exactly. Wow, dust off my jazz shoes.
35:43
Speaker 1: It would be us in Chocolate Rain. And you know
35:47
Speaker 1: I love that guy. Tjon Day is his name. Yeah,
35:50
Speaker 1: we actually met him that time. I remember it was great. Man. Um.
35:54
Speaker 1: Another thing, Chuck that people that has to do with
35:56
Speaker 1: recall biases. That, um, like, we just tend to have
36:00
Speaker 1: fault to your memories with stuff that makes us look bad,
36:04
Speaker 1: like say, unhealthy habits. So if if you're doing a
36:09
Speaker 1: study on UM junk food and and health outcomes, and
36:15
Speaker 1: you interview a bunch of people who are in terrible health,
36:18
Speaker 1: and all of them are like, only like cheese. It's
36:21
Speaker 1: like once in a blue moon or something like that
36:24
Speaker 1: in the researcher, right, so once in blue moon cheese.
36:26
Speaker 1: It's um like the results of the study, you're gonna
36:30
Speaker 1: suggest that it takes just a very small amount of cheese.
36:35
Speaker 1: It's to put you in the hospital with long term
36:38
Speaker 1: chronic health conditions. And that's that is a problem with
36:43
Speaker 1: recall bias, Like it is. It's it's the participants affecting
36:46
Speaker 1: it in this case because they just aren't paying attention
36:49
Speaker 1: to aren't really thinking about No, you you've eaten a
36:52
Speaker 1: lot of cheese. It's and it takes a lot of cheese.
36:54
Speaker 1: It's to put you in the hospital, not a very
36:56
Speaker 1: little amount. It's not the best example, but it kind
36:59
Speaker 1: of gets the point cross I think now, is this
37:01
Speaker 1: part of acquiescence bias. No, that was that was the
37:05
Speaker 1: end of recall bias. Acquiescence biases, it's different, but it's
37:08
Speaker 1: certainly related. Both of them kind of fall under an
37:11
Speaker 1: umbrella of participant bias. Yeah, and acquiescence bias. I feel
37:15
Speaker 1: like there's the opposite too. I just don't know they
37:18
Speaker 1: have a name, um, because acquiescence biases is generally like
37:23
Speaker 1: people people want to be agreeable, and they want to
37:26
Speaker 1: answer in the affirmative, and they want to especially they
37:29
Speaker 1: found um, if you are maybe less educated, you might
37:34
Speaker 1: be more willing to just go along with something and
37:36
Speaker 1: say yeah, sure, yeah, yeah, um to maybe appear smarter
37:41
Speaker 1: or just to be more agreeable. I do think it's
37:43
Speaker 1: the opposite can happen too, especially with political um research
37:48
Speaker 1: in social studies and that I think there are also
37:51
Speaker 1: people that are like, oh you're from the what well, yeah, sure,
37:56
Speaker 1: I'd love to be interviewed, and then they go into
37:58
Speaker 1: it with a sort of opposite mental where they're completely
38:01
Speaker 1: disagreeable no matter what anyone says or asked. Yeah, I
38:05
Speaker 1: didn't run across that, but I'm absolutely sure that that
38:08
Speaker 1: is a bias out there. But you can avoid these
38:10
Speaker 1: by doing it more smartly, right, more smartly. Yeah, there's
38:15
Speaker 1: ways that you can, Um, you can frame your questions,
38:19
Speaker 1: like like people don't like to to admit that they
38:21
Speaker 1: didn't actually vote American Democracy, so in instead of saying
38:26
Speaker 1: there was a there was a pew a pew suggestion
38:30
Speaker 1: p um where they said um, rather than saying like,
38:35
Speaker 1: did you vote in the last election. A lot of
38:37
Speaker 1: people who didn't vote are gonna be like, sure, yeah,
38:39
Speaker 1: of course, why why would you ask that um? Instead
38:43
Speaker 1: you would phrase it as like, um, in the two
38:46
Speaker 1: thousand and twelve presidential election, Uh, did things come up
38:51
Speaker 1: that prevented you from voting? Or were you able to vote?
38:54
Speaker 1: And you would probably actually want to train your researcher
38:57
Speaker 1: to use that same intonation to make it seem casual
39:02
Speaker 1: either way, Like you want to give the person a
39:04
Speaker 1: sense of comfort that they're not being judged no matter
39:07
Speaker 1: how they answer. A good way. That's a good way
39:11
Speaker 1: to get away around acquiescence bias. Absolutely, yeah, the old
39:16
Speaker 1: backdoor policy. That's right, where you can squeeze a woman.
39:23
Speaker 1: All right, are we taking a break? I think we're
39:27
Speaker 1: We're mandated by the SEC to do that. After that joke,
39:30
Speaker 1: all right, well, we'll be back and finish up with
39:32
Speaker 1: our final two biases right after this. I also I
39:56
Speaker 1: want to apologize to all the parents who listen with
39:58
Speaker 1: their six year olds these days. My daughter is sick.
40:01
Speaker 1: She doesn't care about what we do. That's great. So
40:05
Speaker 1: it's still just flying overhead, right, I mean she didn't
40:07
Speaker 1: even listen. She like movie crush a little bit. Well,
40:10
Speaker 1: some kids there are six listen and hey, shout out
40:13
Speaker 1: to all of you guys. Listen. No, I'm always whenever
40:15
Speaker 1: I see that, whenever someone writes in says their kid
40:18
Speaker 1: my daughter's age actually listens, I'm like, really, oh, yes,
40:22
Speaker 1: this is my daughter. She loves it, right, Yeah, and
40:26
Speaker 1: she voted in the last election to like, uh, my
40:29
Speaker 1: daughter likes to watch videos of kids playing with toys
40:32
Speaker 1: on YouTube? Kids? Is she into that? Now? Those are
40:36
Speaker 1: the worst videos. I'm starting to get on her now
40:38
Speaker 1: with just in terms of taste. I'm like, hey, you
40:40
Speaker 1: can watch something, but like watch something with a story
40:42
Speaker 1: that's good. It's like, this is garbage. I like it.
40:48
Speaker 1: I can totally see your saying just like that three
40:51
Speaker 1: defiant and happy impression. Alright. Publication bias is one we
40:56
Speaker 1: we kind of poked around it earlier a little bit
40:57
Speaker 1: with the whole publisher parish mentality. Can I add something
41:01
Speaker 1: more to that real quick before we get into publication bias,
41:04
Speaker 1: you you know, mind to what to the to just
41:08
Speaker 1: talking about publication in general. So I don't think that
41:12
Speaker 1: it's fully grasped by most people. It certainly wasn't by
41:16
Speaker 1: me until really diving into this, that the academic publishing
41:21
Speaker 1: industry has a stranglehold on science right now, in a
41:25
Speaker 1: very similar effect that twenty four hour cable news had
41:30
Speaker 1: on like journalism, to where it was like it became
41:35
Speaker 1: this voracious beast that was willing to just spit out
41:38
Speaker 1: money constantly feed it in exchange for yeah, feed it,
41:42
Speaker 1: give me more stories, give me more, give me more pundits,
41:44
Speaker 1: give me like that was the rise of pundits. Pundits
41:47
Speaker 1: didn't really exist prior to that. They just hung out
41:50
Speaker 1: on the editorial pages of newspapers. And then twenty four
41:52
Speaker 1: hour news came along, and there's not possibly enough news stories,
41:56
Speaker 1: like good news stories, to keep going for twenty four hours,
41:59
Speaker 1: so you have to talk about the news stories and
42:00
Speaker 1: analyze them, and then you start getting into who's wrong
42:03
Speaker 1: and all that stuff. The publishing industry is very much
42:06
Speaker 1: like that now, where it's this beast that must be fed,
42:09
Speaker 1: and so there's there can't possibly be that many high
42:13
Speaker 1: quality scientific papers. So scientific papers have just kind of
42:18
Speaker 1: dipped down in quality, and then um, one of the
42:22
Speaker 1: other things that the publishing industry has done is said,
42:26
Speaker 1: we really like like studies that have results. They're called
42:30
Speaker 1: positive results, where like it turned up that that you
42:34
Speaker 1: found uh correlation between something, or the compound you tried
42:39
Speaker 1: on that tumor shrunk the tumor. Like those are what
42:42
Speaker 1: we're interested in, the whole furthering of science with positive
42:45
Speaker 1: and negative outcomes. Just to say this did work, this
42:48
Speaker 1: doesn't work, don't bother trying it. We don't care about
42:52
Speaker 1: that kind of stuff. And that's a huge issue for
42:55
Speaker 1: the scientific community, Like they have to get control of
42:58
Speaker 1: the publishing community again, um if if they're going to
43:01
Speaker 1: come out from under this dark class. Yeah, I mean
43:03
Speaker 1: they found a in two thousand and ten a study
43:06
Speaker 1: about papers in social sciences especially, we're about two and
43:11
Speaker 1: a half I'm sorry, two point three times more likely
43:14
Speaker 1: to show positive results then papers in physical sciences. Even
43:19
Speaker 1: so some some bodies of research or even more apt
43:23
Speaker 1: uh to publish positive results. And that means if you're
43:27
Speaker 1: going you know this going into your profession, and you
43:30
Speaker 1: know this going into your set of research, and it's
43:35
Speaker 1: you know, that's when it becomes sort of put up
43:37
Speaker 1: or shut up time. As far as standing firm on
43:42
Speaker 1: doing good work even if it doesn't get published, right,
43:46
Speaker 1: and so that that confirmation bias can really come in
43:50
Speaker 1: where you start, hopefully inadvertently but certainly not in all cases, inadvertently,
43:56
Speaker 1: start cherry picking data to get a positive outcome where
44:00
Speaker 1: there really wasn't one there before, or you use a
44:03
Speaker 1: kind of a weird statistical method to to suss out
44:06
Speaker 1: the correlation between the variables so that you can have
44:12
Speaker 1: a positive outcome. Because if you're not publishing papers, like
44:16
Speaker 1: your academic career is not progressing and you can actually
44:19
Speaker 1: like lose jobs, so you need to be published. The
44:22
Speaker 1: publishing industry wants your paper, but they just want positive outcomes.
44:26
Speaker 1: So a high quality, well designed, well executed study that
44:31
Speaker 1: found a negative outcome to where they said, well, this
44:34
Speaker 1: compound we tried didn't actually shrink the tumor, that's that's
44:38
Speaker 1: going to be ignored in favor of a low quality
44:41
Speaker 1: paper that found some compound that shrunk a tumor just
44:45
Speaker 1: because they like positive outcomes. It's ridiculous. Yeah, And I
44:48
Speaker 1: mean that kind of goes hand in hand with the
44:49
Speaker 1: last one. You know, there's a lot of overlap with
44:52
Speaker 1: these and a lot that work sort of in concert
44:54
Speaker 1: with one another. And file draw bias is you know,
44:58
Speaker 1: it is what it sounds like. It's like you you
45:00
Speaker 1: got a negative outcome, and whether or not you were
45:03
Speaker 1: being funded by a company that definitely doesn't want that
45:06
Speaker 1: information getting out there, or if it's just as a
45:09
Speaker 1: result of it being less likely to be published because
45:13
Speaker 1: it doesn't have a positive outcome, you just stick it
45:16
Speaker 1: in the file drawer and it goes by by, right,
45:18
Speaker 1: and again, like part of the point of science and
45:20
Speaker 1: scientific publishing is to generate this body of knowledge. So
45:24
Speaker 1: if you're about to do a study, you can search
45:26
Speaker 1: and say, oh, somebody already tried the same exact thing
45:28
Speaker 1: and they found that it doesn't work. I'm gonna not
45:30
Speaker 1: try to reproduce that. I'm just gonna not go with it,
45:33
Speaker 1: um and move on to try something else. It's a
45:36
Speaker 1: huge waste of resources otherwise. And then also you could
45:40
Speaker 1: you can if you aren't publishing that kind of stuff, Um,
45:45
Speaker 1: you're missing out on well, I mean you're missing out
45:48
Speaker 1: on the real data. If the bad data is vile
45:53
Speaker 1: drawered like you're missing out on the truth. You're missing
45:56
Speaker 1: out on the whole picture, right, And also Again, it's
46:02
Speaker 1: not just that the poor negative outcomes they need to
46:04
Speaker 1: be included to. Yes, that's true, but you're also promoting
46:08
Speaker 1: positive outcome studies that actually aren't good studies. There's this
46:13
Speaker 1: thing called the proteus effect where the initial studies, these
46:17
Speaker 1: initial papers on a subject um in sevent cases, a
46:22
Speaker 1: follow up study that seeks to reproduce them can't reproduce them.
46:26
Speaker 1: They don't come to the same finding, the same conclusions,
46:28
Speaker 1: which suggests that a study was really terrible. Um, if
46:32
Speaker 1: it can't be reproduced, or if it's reproduced, somebody comes
46:35
Speaker 1: to a different finding, different conclusion, that's not a good study.
46:39
Speaker 1: So the idea of publishing positive and negative outcomes together
46:43
Speaker 1: would definitely kind of slow that whole crazy twenty four
46:47
Speaker 1: hour news cycle. Positive outcome study. I don't see how
46:51
Speaker 1: it's even legal to bury not bury, but I guess
46:57
Speaker 1: just not even just a file drawer, a study suddy
47:01
Speaker 1: that included like a drug having negative effects like and
47:04
Speaker 1: I know that Congress is stepped up to try and
47:07
Speaker 1: pass laws too. I think there was one UH in
47:11
Speaker 1: two thousand seven requiring researchers to report results of human
47:17
Speaker 1: studies of experimental treatments. Uh, and then they tried to
47:21
Speaker 1: strengthen that in sixteen. Basically this like, you know, even
47:25
Speaker 1: if your drug doesn't come to market, like, we need
47:27
Speaker 1: to have these studies and the results. Like, how, how's
47:31
Speaker 1: it even legal? It seems like you're bearing and it's
47:33
Speaker 1: almost falsification. Well, it is, for sure, because you're also
47:37
Speaker 1: like if you're if you're talking about studies where you
47:40
Speaker 1: have multiple studies on say one drug that's an antidepressant,
47:45
Speaker 1: and all you're doing is publishing the ones that have
47:48
Speaker 1: positive outcomes for that antidepressant, and you're just not publishing
47:52
Speaker 1: the ones that that showed no outcomes or maybe even harm,
47:55
Speaker 1: then yeah, that should be illegal, especially when you're talking
47:58
Speaker 1: about something like an antidepressant or in the biomedical field.
48:02
Speaker 1: But it's certainly unethical for any field of science in particular.
48:06
Speaker 1: Just bury the stuff you don't like that doesn't support
48:08
Speaker 1: your conclusion. It's a kind of a meta form of
48:11
Speaker 1: um confirmation bias, just putting aside the stuff that doesn't
48:15
Speaker 1: fit your hypothesis or your worldview, and um just promoting
48:18
Speaker 1: the stuff that does. I saw one one way around
48:23
Speaker 1: this is the Lancet, you know, the very respected medical journal.
48:26
Speaker 1: I think it's British. The Lancet um has taken to
48:30
Speaker 1: um accepting papers based on the study design and methodology
48:35
Speaker 1: and goals. So when you first plan your study and
48:38
Speaker 1: you have it all together before you ever start, that's
48:41
Speaker 1: when you would apply to have your paper studied and
48:44
Speaker 1: published in the Lancet, and that's when they decide whether
48:47
Speaker 1: it's a high quality enough study to publish or not.
48:50
Speaker 1: So then they're locked into publishing your study, whether your
48:53
Speaker 1: outcome is negative or positive, and has the knock on
48:57
Speaker 1: effect of the Lancet basically being like this is this
49:00
Speaker 1: is a trash study. We would never publish this, don't
49:03
Speaker 1: even bother. So it's saving funds and then the high
49:06
Speaker 1: quality studies are the ones that are going to get published.
49:09
Speaker 1: And then also the positive outcomes and the negative outcomes
49:11
Speaker 1: get get published regardless because they have no idea what
49:15
Speaker 1: outcome is going to be because they accept the paper
49:17
Speaker 1: before the paper, before the study has even been conducted.
49:20
Speaker 1: I saw another thing that said that uh paper would
49:24
Speaker 1: be more likely to get published in the Lancet if
49:26
Speaker 1: it had cool illustrations. That's right, That never hurts. Everybody
49:30
Speaker 1: knows that that's not unethical, especially in color it just
49:33
Speaker 1: put a few of those New Yorker cartoons forget about it.
49:36
Speaker 1: Everybody loves those. Are you got anything else, I've got
49:40
Speaker 1: nothing else. You know, this is a little soapboxy, but
49:42
Speaker 1: this is something that we believe in. It's it's kind
49:44
Speaker 1: of like our episode on the Scientific Method a little bit.
49:49
Speaker 1: Mm hmm. I like it too. Yeah, thanks for doing
49:52
Speaker 1: it with me, man, Thank you for doing it with me.
49:55
Speaker 1: Thank you for squeezing my limits. Sure. Uh, if you
49:59
Speaker 1: want to know more or about scientific bias, there's a lot, fortunately,
50:03
Speaker 1: a lot of sites and great articles dedicated um to
50:07
Speaker 1: routing that stuff out and to make you a smarter
50:09
Speaker 1: consumer of science. And so go check that out and
50:13
Speaker 1: learn more about it. And since I said learn more
50:15
Speaker 1: about it, it means it's time for a listener mail.
50:21
Speaker 1: You know. Sometimes the listener mail dovetails quite nicely with
50:25
Speaker 1: the topic, and that was the case today with our inclusion.
50:30
Speaker 1: Oh yeah, on the Media Bias List, which was pretty exciting. Yeah,
50:35
Speaker 1: what an honor. You know, there's something called the media
50:38
Speaker 1: bias Is it called the media Bias List? I believe so.
50:41
Speaker 1: And it's you know, what it does is it takes
50:44
Speaker 1: news outlets and newspapers and you know, TV and stuff
50:47
Speaker 1: like that, and they just sort of it's a big
50:49
Speaker 1: chart where they're ranked according to like how biased they are,
50:54
Speaker 1: you know, kind of up down, left and right. And
50:56
Speaker 1: they included podcasts this year they did, and we were
50:59
Speaker 1: on the US and it was really kind of cool.
51:01
Speaker 1: We had a bunch of people right in. And this
51:02
Speaker 1: is from Nicholas Bett. Oh, he said, I found this
51:05
Speaker 1: post while I was scrolling through Facebook, uh and waiting
51:08
Speaker 1: for the NFL season to start. Add fonts media? Is
51:12
Speaker 1: it fonts or fontests? We should know? I'm not sure
51:15
Speaker 1: one of the two. It's a it's a watchdog organization
51:19
Speaker 1: known for the media bias chart. Um. They do a
51:22
Speaker 1: media bias chart where they rank every news outlets political bias,
51:25
Speaker 1: and in the recent update they included you guys, and
51:28
Speaker 1: wouldn't you know it the most politically fair piece of
51:30
Speaker 1: media you can possibly consume and all of the known
51:32
Speaker 1: universes stuff you should know. You guys say you're liberal.
51:36
Speaker 1: But until I heard Chuck outright stated, I didn't even
51:39
Speaker 1: know wow that well, I think I think it slips
51:44
Speaker 1: through their son. Well, yeah, we're certainly human beings and
51:49
Speaker 1: we have their own biases, but we definitely try to
51:51
Speaker 1: keep them in check. Is we try to, And I
51:53
Speaker 1: think it's just been confirmed because they're not. Just like,
51:55
Speaker 1: listen to a couple of shows and I want to
51:57
Speaker 1: see these guys seem okay, like they really listen, then
52:00
Speaker 1: they really rank people. Um. They probably saw that too,
52:03
Speaker 1: or perhaps they listen to the North Korea episode where
52:06
Speaker 1: Josh suggested wolf Blitzer apply hot paper clips to a
52:08
Speaker 1: center thighs while writing a nice piece on Trump's Curreyan relations.
52:13
Speaker 1: Hilarious either way, Thank you guys for your fairness and
52:16
Speaker 1: hilarity of all these years. You're both the best. That
52:19
Speaker 1: is from Nicholas bad Oh. Thanks a lot of Nicholas.
52:22
Speaker 1: Thanks to everybody who wrote in to say that they
52:24
Speaker 1: saw that. We appreciate it. Uh. And it was neat
52:26
Speaker 1: to see ourselves right in the middle of the rainbow.
52:28
Speaker 1: Love being in the middle of that rainbow. I do
52:31
Speaker 1: to Chuck. It's nice and warm and cozy in there,
52:33
Speaker 1: isn't it. Yes, Well, if you want to get in
52:35
Speaker 1: touch with this, like Nicholas and the Gang did, you
52:38
Speaker 1: can send us an email to Stuff Podcast at iHeart
52:41
Speaker 1: radio dot com. Stuff you Should Know is a production
52:47
Speaker 1: of iHeart Radio. For more podcasts My heart Radio, visit
52:51
Speaker 1: the iHeart Radio app, Apple podcasts, or wherever you listen
52:54
Speaker 1: to your favorite shows. Two