Latest / Wellness, Actually with Emily Oster & Perry Wilson, MD / Bonus: The Power of Data & Its Limits

Transcript

Bonus: The Power of Data & Its Limits · Wellness, Actually with Emily Oster & Perry Wilson, MD

• 0:00
  Speaker 1: Hello, welcome to a special episode of the Parent Data Podcast.
• 0:04
  Speaker 1: This is not me reading one of my newsletters, but
• 0:07
  Speaker 1: rather the audio from a talk that I gave at
• 0:11
  Speaker 1: south By Southwest Edu conference called the Power of Data
• 0:14
  Speaker 1: and Its Limits. It is a talk all about the
• 0:17
  Speaker 1: importance of data and the key mistakes that we make
• 0:21
  Speaker 1: when we try to analyze data and when we.
• 0:24
  Speaker 2: Try to understand it.
• 0:26
  Speaker 1: The first slide, spoiler alert says data is important but
• 0:29
  Speaker 1: poorly understood, and the rest of the talk is all
• 0:32
  Speaker 1: about that insight and trying to help you understand it
• 0:36
  Speaker 1: a little bit better. So I hope that you get
• 0:40
  Speaker 1: a chance to listen to this and that you enjoy it.
• 0:42
  Speaker 2: Thanks so much, all right, thank you.
• 0:48
  Speaker 3: I am so excited to be here to do something
• 0:51
  Speaker 3: in person with people.
• 0:53
  Speaker 2: So that is just an incredible.
• 0:54
  Speaker 3: From And you know, thank you Greg. That was a
• 0:58
  Speaker 3: very nice introduction. And tell you a little bit sort
• 1:01
  Speaker 3: of of who I am. So my main job, my
• 1:06
  Speaker 3: sort of professional identity is I'm a professor of economics
• 1:09
  Speaker 3: at Round University. And I also have a sort of
• 1:14
  Speaker 3: another part of my life where I do a lot
• 1:15
  Speaker 3: of writing about pregnancy and parenting and COVID and I
• 1:22
  Speaker 3: write a newsletter, and I just do a lot of
• 1:25
  Speaker 3: things in that in that space, and sort of in
• 1:29
  Speaker 3: some ways, when I look at the things I'm doing,
• 1:32
  Speaker 3: they seem sort of disconnected. But actually the piece that
• 1:34
  Speaker 3: brings them all together, the sort of piece of the
• 1:38
  Speaker 3: piece that goes across all parts of this is the
• 1:41
  Speaker 3: importance of data, and the importance of being able to
• 1:46
  Speaker 3: use data thoughtfully and helpfully to answer questions, and the
• 1:52
  Speaker 3: recognition that that usage of data is effectively the same
• 1:59
  Speaker 3: whether your question is should I have an epidural with
• 2:03
  Speaker 3: my pregnancy, or your question is how do people respond
• 2:07
  Speaker 3: to a diabetes diagnosis in terms of their their food consumption?
• 2:12
  Speaker 3: Or if that question is, you know, how should we
• 2:14
  Speaker 3: think about the value of vaccines in the in the
• 2:18
  Speaker 3: COVID pandemic That a knowledge of data and understanding of
• 2:22
  Speaker 3: data crosses all of those all of those areas, and
• 2:28
  Speaker 3: once you sort of recognize that, or to the E
• 2:30
  Speaker 3: sent that I that I recognize that, it makes me
• 2:32
  Speaker 3: think that sort of a key to helping people navigate
• 2:35
  Speaker 3: many of these spaces is understanding understanding data, understanding what
• 2:40
  Speaker 3: it can tell us, and you know, in some cases
• 2:42
  Speaker 3: it what it can tell us, and despite that I
• 2:46
  Speaker 3: find it. In general, people's understanding of data is very poor.
• 2:49
  Speaker 3: So I sometimes data is important, but nobody really nobody
• 2:54
  Speaker 3: really gets it, or people don't always understand it as
• 2:56
  Speaker 3: well as they as they could.
• 2:59
  Speaker 2: And you know, you might say, why does this matter?
• 3:01
  Speaker 3: And one answer is just like, I'm obsessed with data,
• 3:03
  Speaker 3: so of course I think it matters because it's the
• 3:05
  Speaker 3: thing that I do. And we all think the things
• 3:07
  Speaker 3: that we do are very important. So fair enough, But
• 3:10
  Speaker 3: I'll let me tell you why why.
• 3:12
  Speaker 2: I believe that. There's really two reasons. So one is
• 3:18
  Speaker 2: that I think.
• 3:18
  Speaker 3: That actually, for a lot of people, we would make
• 3:21
  Speaker 3: better decisions in our own lives if we had a
• 3:24
  Speaker 3: better understanding of what we can learn, what we can
• 3:26
  Speaker 3: learn from evidence, And so I think there's a sort
• 3:30
  Speaker 3: of personal pitch to like, Okay, you know, invest in
• 3:33
  Speaker 3: this so you can make choices that you are happier with.
• 3:37
  Speaker 3: I think there is also a very key element, and
• 3:41
  Speaker 3: this is in some ways where the kind of education
• 3:45
  Speaker 3: part comes in here. I think if as a society
• 3:48
  Speaker 3: we had a better understanding of some of these some
• 3:51
  Speaker 3: of these issues, it might be easier to make our
• 3:57
  Speaker 3: societal decisions in a better or more informed, and so
• 4:00
  Speaker 3: in particular, you know, during COVID it's became very someways
• 4:03
  Speaker 3: very salient to me. I felt like a lot of
• 4:05
  Speaker 3: the failures, or some of the failures at least that
• 4:07
  Speaker 3: we that we had in the US kind of related
• 4:11
  Speaker 3: to not having the data, the information we needed to
• 4:14
  Speaker 3: make good decisions. And I kept having the feeling if
• 4:18
  Speaker 3: only people understood how important data is, we would have
• 4:22
  Speaker 3: better data, would be more pressure to get better get
• 4:25
  Speaker 3: to get better information, and let us make those decisions,
• 4:27
  Speaker 3: most decisions better. Sometimes when I say that, people are like,
• 4:31
  Speaker 3: that's like a really naive thing to say, you're a
• 4:34
  Speaker 3: person who loves data. Okay, fair enough, maybe it's a
• 4:36
  Speaker 3: little naive, but that's that's my that's my pitch. So
• 4:41
  Speaker 3: in light of that idea, so today I really want
• 4:43
  Speaker 3: to go through kind of four big, big kind of
• 4:46
  Speaker 3: data lessons, and I'll talk a little bit about kind
• 4:49
  Speaker 3: of applications to COVID, but but not exclusively and really
• 4:55
  Speaker 3: sort of these are these are kind of cases where
• 4:57
  Speaker 3: a sort of questions where I think it's possible to
• 5:01
  Speaker 3: understand why they're important.
• 5:03
  Speaker 2: It's not that hard to communicate that. But if we understood.
• 5:07
  Speaker 3: All of these better, we would be able to both
• 5:10
  Speaker 3: communicate data better, to process data better, and to make
• 5:13
  Speaker 3: better choices.
• 5:14
  Speaker 2: So I'm going to talk about the question of where
• 5:16
  Speaker 2: data comes from. I'm going to talk about the question of.
• 5:18
  Speaker 3: Statistical power, the idea that correlation and causation are not linked,
• 5:23
  Speaker 3: and how we can kind of see that. Then I'm
• 5:25
  Speaker 3: going to talk a little bit about base rule. And
• 5:28
  Speaker 3: in all of these cases, I'm going to kind of
• 5:30
  Speaker 3: come in the angle here of like how would I
• 5:33
  Speaker 3: how do I teach this to people? How would you
• 5:35
  Speaker 3: convey this to people in a way that's not like
• 5:37
  Speaker 3: here's the math of what random sampling is, but sort
• 5:40
  Speaker 3: of intuitively, how do we get to these like the
• 5:43
  Speaker 3: understanding of these of these sort of key issues. All right,
• 5:47
  Speaker 3: So let's start with where data comes from? And I'm
• 5:49
  Speaker 3: reminding you that at the end we'll do Q and
• 5:51
  Speaker 3: A and so if you have your little app thing,
• 5:53
  Speaker 3: you should ask questions and then I will try to
• 5:55
  Speaker 3: answer them later. And I'm happy to answer questions about
• 5:57
  Speaker 3: more or less anything that you want.
• 6:01
  Speaker 2: Okay, So let's get started.
• 6:03
  Speaker 3: So where does data come from? So I want to
• 6:09
  Speaker 3: start with this statement. This is this statement I pulled
• 6:13
  Speaker 3: off some CDC website, some academic paper or something. CDC
• 6:17
  Speaker 3: data indicates that forty two point five percent of adult
• 6:20
  Speaker 3: Americans are robs. Right, So the CDC has a lot
• 6:23
  Speaker 3: of statements like this, and it's on a website in
• 6:26
  Speaker 3: which they report out some information about the health of
• 6:29
  Speaker 3: the American population.
• 6:32
  Speaker 2: So the question is like what, like where would you
• 6:34
  Speaker 2: get this?
• 6:35
  Speaker 3: Like where's this? How do we how do we know
• 6:38
  Speaker 3: this piece of information? So in some sense the answer
• 6:42
  Speaker 3: to that seems obvious. Okay, so we must somehow know
• 6:45
  Speaker 3: it from weighing people, presumably people in America based on
• 6:48
  Speaker 3: the UH and presumably adults in America based on this sentence.
• 6:53
  Speaker 3: So we weigh them and we must also.
• 6:54
  Speaker 2: Measure their height.
• 6:55
  Speaker 3: That's how you get to a definition of obesity, and
• 6:58
  Speaker 3: we calculate some number. Okay, So that's like roughly right.
• 7:02
  Speaker 3: The question is, well, like who are you weighing? And
• 7:07
  Speaker 3: sometimes I'll ask my students that and they'll be like, well,
• 7:09
  Speaker 3: I guess you weigh everybody because it's adult Americans.
• 7:12
  Speaker 2: And I said, well, did you get weighed?
• 7:14
  Speaker 3: Like did they come weigh you know, no, no, nobody
• 7:17
  Speaker 3: weigh Okay, Well, can't be everybody?
• 7:19
  Speaker 2: Then at least you personally were not weighed.
• 7:22
  Speaker 3: And in fact, the answer to this question sort of
• 7:24
  Speaker 3: where does a number like this come from is, in
• 7:28
  Speaker 3: this particular case, a CDC survey called the National Health
• 7:32
  Speaker 3: and Nutrition Examination Survey that the CDC runs, and it
• 7:35
  Speaker 3: covers about eleven thousand people every year. So they weigh
• 7:38
  Speaker 3: the eleven thousand people, and they calculate their BMI, and
• 7:41
  Speaker 3: they calculate the share of them that are obese, and
• 7:43
  Speaker 3: then they kind of convert that into a number to
• 7:46
  Speaker 3: be representative of the.
• 7:47
  Speaker 2: Whole US, of the whole US population.
• 7:51
  Speaker 3: So in some sense this makes a lot of sense
• 7:55
  Speaker 3: and sort of statistically like, of course, you can't weigh everybody.
• 7:59
  Speaker 2: That would be like practically impossible, and so.
• 8:02
  Speaker 3: You've got to have some system for using a smaller
• 8:04
  Speaker 3: sample of people to extend up to everybody. But the
• 8:10
  Speaker 3: key is is how you pick that sample. If you
• 8:14
  Speaker 3: want to weigh eleven thousand people and make a statement
• 8:17
  Speaker 3: like this that has some type of validity, you need
• 8:21
  Speaker 3: to be pretty careful about how you pick those people.
• 8:25
  Speaker 2: So, for example, one way.
• 8:27
  Speaker 3: We could pick those people is we could take everybody
• 8:30
  Speaker 3: in a particular location, So we could go to Missouri,
• 8:32
  Speaker 3: or New York, or we could go to you know,
• 8:35
  Speaker 3: you only need eleven thousand people.
• 8:36
  Speaker 2: You could go to like I don't know, like a
• 8:39
  Speaker 2: like a NIXT game.
• 8:40
  Speaker 3: Or a NETS game or a you just go to
• 8:43
  Speaker 3: some stadium and just weigh the people. Is so so convenient,
• 8:47
  Speaker 3: you know, like this actually endings is an enormous amount
• 8:49
  Speaker 3: of work. They have all these like I don't know,
• 8:51
  Speaker 3: mobile clinics and stuff like why don't you just show
• 8:53
  Speaker 3: up at some sports event with eleven thousand people, You
• 8:56
  Speaker 3: weigh them and you're done. That's it very fast, cheap,
• 9:00
  Speaker 3: And of course the problem with that is like, well,
• 9:02
  Speaker 3: that's not going to tell you about everybody at America.
• 9:04
  Speaker 3: That's going to tell you about the people who went
• 9:05
  Speaker 3: to the NETS game on that particular day, and I
• 9:08
  Speaker 3: don't know how those people compare to the overall population.
• 9:13
  Speaker 3: You could go you could have sort of different approach.
• 9:16
  Speaker 3: You weigh everybody who's at the gym. Or what if
• 9:19
  Speaker 3: you only weigh people who showed up to be part
• 9:21
  Speaker 3: of your study? Right, you said, well, come be in
• 9:23
  Speaker 3: my study which is about weighing people. And what kind
• 9:26
  Speaker 3: of people show up for your study about weighing people?
• 9:29
  Speaker 2: Not a random sample of people, you.
• 9:31
  Speaker 3: Know, you could only include people, and this is a
• 9:33
  Speaker 3: little closer to maybe what we get to can only
• 9:36
  Speaker 3: include people who picked up the phone when you called.
• 9:39
  Speaker 3: All of these are ways in which our sampling would
• 9:42
  Speaker 3: be non random. And you can see, you know, particularly
• 9:45
  Speaker 3: with some of these sort of first couple of examples,
• 9:48
  Speaker 3: why that kind of approach, why just doing something convenient is.
• 9:53
  Speaker 2: Actually not going to lead you to the right answer.
• 9:55
  Speaker 3: And it turns out that when you want to use
• 9:58
  Speaker 3: a small sample to speak about a large sample, which
• 10:01
  Speaker 3: is a very very key part of using data and
• 10:04
  Speaker 3: statistical methods, when you want to do that, you must
• 10:09
  Speaker 3: think about sampling people randomly. You must have some way
• 10:14
  Speaker 3: to get a sample that is representative of the population
• 10:17
  Speaker 3: about who you wish to speak. Now, maybe that seems
• 10:22
  Speaker 3: very straightforward, but I would argue that during COVID, we
• 10:25
  Speaker 3: totally fell down, particularly in the US, on that approach.
• 10:30
  Speaker 3: So let's think about what I thought of as a
• 10:33
  Speaker 3: very very basic question that we had about COVID, which
• 10:39
  Speaker 3: is what is the case rate in a given state
• 10:41
  Speaker 3: at a given time? And this turned out to be
• 10:43
  Speaker 3: like tremendously policy relevant, right, many many many, still many
• 10:47
  Speaker 3: many policies.
• 10:50
  Speaker 2: About what you should do or not do, or who
• 10:52
  Speaker 2: can go to.
• 10:52
  Speaker 3: This thing or that thing are based on this case
• 10:55
  Speaker 3: rate number, the number like what's the share of what's
• 10:58
  Speaker 3: the number of people have COVID out of every one
• 11:01
  Speaker 3: hundred thousand people, So it's you know, if it's above this,
• 11:03
  Speaker 3: it's above that.
• 11:04
  Speaker 2: So that was like a very core number.
• 11:06
  Speaker 3: They're kind of like two ways that we that we
• 11:09
  Speaker 3: sort of think about it getting that, and two ways
• 11:12
  Speaker 3: you think of thought about measuring this. One is this
• 11:14
  Speaker 3: case rate of like the number of cases detected per
• 11:17
  Speaker 3: one hundred thousand people. That's something we often report, so
• 11:22
  Speaker 3: we don't have a random sample for doing that, right.
• 11:25
  Speaker 3: So this was a point that the former President Trump
• 11:29
  Speaker 3: made early in the pandemic, which was that if you
• 11:32
  Speaker 3: test more people, you will find more cases. And that
• 11:34
  Speaker 3: has been weaponized as an argument, but it is in
• 11:37
  Speaker 3: fact true that the places that do more testing end
• 11:40
  Speaker 3: up finding more cases because that's the way that works.
• 11:45
  Speaker 2: And so if you measure.
• 11:47
  Speaker 3: Cases in that way, you're gonna get signed of one number,
• 11:49
  Speaker 3: which is sort of non random and depends on the
• 11:51
  Speaker 3: amount of testing. The other way we measure sort of
• 11:55
  Speaker 3: COVID rates is with something called the positivity rate. That's
• 11:57
  Speaker 3: the share of the tests that show up positive. If
• 12:01
  Speaker 3: we don't have a random sample, then that number is
• 12:06
  Speaker 3: also kind of weird because that's like just the people.
• 12:08
  Speaker 2: Who come And so if you're not doing very many tests.
• 12:12
  Speaker 3: The only people who show up maybe people who like
• 12:14
  Speaker 3: almost certainly have it because you're selecting that sample in
• 12:18
  Speaker 3: a funny way. In a world with a random sample,
• 12:22
  Speaker 3: where you were doing this with a random sample, these
• 12:24
  Speaker 3: numbers would be the same, These.
• 12:26
  Speaker 2: Things would line up. They wouldn't be like literally.
• 12:28
  Speaker 3: The same number, but they would capture exactly the same thing.
• 12:31
  Speaker 2: You would just if you.
• 12:32
  Speaker 3: Had like your sample of a million people and you
• 12:34
  Speaker 3: were testing them every week, you would get the same
• 12:38
  Speaker 3: kind of number, no matter the same conclusion, whether you
• 12:42
  Speaker 3: measured it with these case rates or with positivity rates.
• 12:45
  Speaker 3: But in practice, because there was so much variation in
• 12:50
  Speaker 3: how much we were monitoring the pandemic across states in time,
• 12:54
  Speaker 3: we ended up with some stuff that looked like pretty weird.
• 12:58
  Speaker 3: So here's a graph I made at some point of
• 13:01
  Speaker 3: the relationship between the positivity rate, which is here on
• 13:05
  Speaker 3: the vertical axis. This is the share of tests that
• 13:08
  Speaker 3: are positive and on the horizontal access.
• 13:10
  Speaker 2: Is this measure of this daily case rate.
• 13:13
  Speaker 3: Now, again, if this is sort of if we had
• 13:16
  Speaker 3: a random sample, these things would be like there'd be
• 13:19
  Speaker 3: a forty five degree line, right, it would just be
• 13:21
  Speaker 3: like a like on the line because they would exactly
• 13:26
  Speaker 3: line up with each other. In fact, you get things
• 13:29
  Speaker 3: like and I sort of highlighted a couple of things here,
• 13:32
  Speaker 3: So you get things like in South Dakota in November
• 13:36
  Speaker 3: and in Arizona in January. I think these are twenty
• 13:39
  Speaker 3: twenty and twenty twenty one. The case rate's the same.
• 13:43
  Speaker 3: These are both places with a case rate of one
• 13:44
  Speaker 3: hundred and thirty cases detected per one hundred thousand. But
• 13:48
  Speaker 3: the positivity rate, the share of tests that are showing
• 13:50
  Speaker 3: up positive, it's fifty percent in South Dakota at that time,
• 13:54
  Speaker 3: in about fifteen percent in Arizona.
• 13:56
  Speaker 2: And so what that kind of tells you is like
• 13:59
  Speaker 2: it must.
• 13:59
  Speaker 3: Be the case that actually South Dakota has tremendously more
• 14:02
  Speaker 3: COVID at this time than Arizona does, but they must
• 14:05
  Speaker 3: be doing much less testing because half the test they
• 14:07
  Speaker 3: do are showing up positive. We have seen sort of
• 14:10
  Speaker 3: similar things all over that in some sense, like we're
• 14:13
  Speaker 3: seeing this case rate thing that we think of as
• 14:15
  Speaker 3: this like special measure that's measuring something is actually representing
• 14:18
  Speaker 3: totally different things in different locations because we are not
• 14:23
  Speaker 3: randomly sampling people. And so this problem continues to be true,
• 14:28
  Speaker 3: and in fact, you know, I think will in some
• 14:32
  Speaker 3: ways be more true as we go forward, as we
• 14:36
  Speaker 3: do less testing, as we do less monitoring of the pandemic.
• 14:39
  Speaker 3: We're going to come up against these things, these these
• 14:43
  Speaker 3: things as we as we sort of move forward. Okay,
• 14:47
  Speaker 3: so that's my first lesson, which is that we that
• 14:50
  Speaker 3: data comes from the world, but we need to randomly sample,
• 14:57
  Speaker 3: all right.
• 14:58
  Speaker 2: Second second lesson.
• 15:00
  Speaker 3: Statistical power, and this is really a lesson about the
• 15:04
  Speaker 3: ways in which data is you know, potentially limited. All Right,
• 15:08
  Speaker 3: So I want to start with the idea here of
• 15:10
  Speaker 3: a randomized trial. It's probably sort of.
• 15:12
  Speaker 2: Familiar to you guys by this point of the pandemic.
• 15:15
  Speaker 2: We're kind of knows when randomized trial is.
• 15:17
  Speaker 3: But we're imagining we're having a randomized trial, which is
• 15:20
  Speaker 3: a study in which we have a group of people
• 15:22
  Speaker 3: and we randomly give do one thing to one of
• 15:24
  Speaker 3: them set of them, and one thing to the other
• 15:27
  Speaker 3: to the other set and the particular randomized trial I
• 15:29
  Speaker 3: want to I want to imagine is that you you
• 15:34
  Speaker 3: have your high school class and you randomly pick half
• 15:37
  Speaker 3: of them to be forced to be on the cross
• 15:39
  Speaker 3: country running team for the semester.
• 15:42
  Speaker 2: That's like, that's your treatment.
• 15:44
  Speaker 3: And so we're going to divide the kids through some
• 15:46
  Speaker 3: kind of random process and half of them will have
• 15:48
  Speaker 3: to be on the cross country team. And the first
• 15:51
  Speaker 3: outcome we're going to study is their ability to complete
• 15:54
  Speaker 3: a five k in under earth that's five'o like three
• 15:56
  Speaker 3: miles in under thirty five minutes.
• 15:59
  Speaker 2: Okay, now this is not And so.
• 16:03
  Speaker 3: Then the question we're going to ask is is how
• 16:05
  Speaker 3: many people do we need to have, like how many
• 16:08
  Speaker 3: kids do we need to include in our study in
• 16:10
  Speaker 3: order to figure out if our treatment is statistically significant,
• 16:14
  Speaker 3: if our treatment is having a statistical effect. So the
• 16:18
  Speaker 3: kind of key here to thinking about this question is
• 16:21
  Speaker 3: thinking about how large our effect size is. So in
• 16:26
  Speaker 3: a case like this, probably if you took a bunch
• 16:28
  Speaker 3: of high school kids and you made them beyond the
• 16:30
  Speaker 3: class country team, probably like most of them could run
• 16:33
  Speaker 3: a five k, or a lot like a lot of
• 16:35
  Speaker 3: them could end up running a five k within thirty
• 16:37
  Speaker 3: five minutes. That's not like that fast for a high
• 16:39
  Speaker 3: school kid. So my guess is, if you sort of
• 16:42
  Speaker 3: reflect on this, the effect is probably very large. The
• 16:46
  Speaker 3: difference between the kid, the sort of share of kids
• 16:48
  Speaker 3: who can achieve this after a semester of like cross
• 16:50
  Speaker 3: country and a share that's and the share who can
• 16:53
  Speaker 3: achieve it without that, that's probably a.
• 16:55
  Speaker 2: Pretty big a pretty big effect.
• 16:58
  Speaker 3: And what that means is that you actually don't probably
• 17:01
  Speaker 3: don't need that many kids. Probably if you had a
• 17:03
  Speaker 3: couple hundred kids in you know, these two groups, you
• 17:07
  Speaker 3: would be statistically powered, as we'd say sort of formally,
• 17:10
  Speaker 3: you'd be statistically powered to figure out whether there was
• 17:14
  Speaker 3: whether there was an effect here. And the sort of
• 17:16
  Speaker 3: larger is your your expected treatment. The smaller is the
• 17:21
  Speaker 3: kind of sample size that you that you need. But
• 17:24
  Speaker 3: then we think about a sort of another version of
• 17:27
  Speaker 3: this trial, which is the same treatment. But now what
• 17:32
  Speaker 3: we want to study is the impact on people's blood
• 17:35
  Speaker 3: pressure at the age of fifty. So we're going to
• 17:37
  Speaker 3: have these kids, We're going to make them beyond the
• 17:39
  Speaker 3: costs country team. We're going to weigh and wait and
• 17:41
  Speaker 3: wait and wait and wait and wait and wait, and.
• 17:43
  Speaker 2: Then when they're fifty, we're going to find them.
• 17:46
  Speaker 3: You're not going to be able to find them, but
• 17:47
  Speaker 3: let's imagine you can find them. Wait, we're going to
• 17:49
  Speaker 3: find them, and then we're going to measure their their
• 17:52
  Speaker 3: blood pressure. And the question is, like, how many people
• 17:55
  Speaker 3: would you need to include here to be confident in
• 17:59
  Speaker 3: the size of to be confident in whatever effect you're
• 18:03
  Speaker 3: going to find there. And the answer again depends on
• 18:07
  Speaker 3: how big the effect is. But in this case, this
• 18:11
  Speaker 3: effect is going to be really small. Maybe it's the
• 18:13
  Speaker 3: case that running for one semester in high school makes
• 18:16
  Speaker 3: your blood pressure lowered aage fife, it could be, but
• 18:18
  Speaker 3: a lot of other stuff happens between the age of
• 18:20
  Speaker 3: like seventeen and fifty that also affects your health. And
• 18:24
  Speaker 3: so any effect size here is going to be like tiny,
• 18:28
  Speaker 3: so small. And so that means if you want to
• 18:31
  Speaker 3: be able to detect that tiny, tiny effect, you need
• 18:34
  Speaker 3: like I don't know, millions of people, right, You've got
• 18:36
  Speaker 3: to run this experiment on every child in every seventeen
• 18:39
  Speaker 3: year old in America if you actually want to be
• 18:42
  Speaker 3: able to figure out statistically is there is there is
• 18:46
  Speaker 3: there an effect here?
• 18:48
  Speaker 2: And so you know, why is this important?
• 18:50
  Speaker 3: Well, I think that understanding this gives us a sense
• 18:54
  Speaker 3: of where we can expect confidence in the data that
• 18:57
  Speaker 3: we see, and where it's going to be very harder,
• 19:00
  Speaker 3: very much harder, rather very much harder. So this the
• 19:08
  Speaker 3: sort of example in the COVID space that I think
• 19:09
  Speaker 3: is really really important to sort of think about is
• 19:13
  Speaker 3: the case of vaccines. So vaccines are something where we
• 19:15
  Speaker 3: have good randomized frial data. That's how we learn about
• 19:19
  Speaker 3: the efficacy of any kind of any kind of vaccine
• 19:21
  Speaker 3: or medical treatment. Typically when we look at older adults,
• 19:25
  Speaker 3: when we look at like people in their seventies, sixties, seventies, eighties,
• 19:29
  Speaker 3: Actually you don't need an enormously large sample size to
• 19:34
  Speaker 3: figure out the impact of COVID vaccines on hospitalizations or deaths,
• 19:39
  Speaker 3: because that's a fairly not common but that is a
• 19:43
  Speaker 3: somewhat more common outcome in that in that group. So
• 19:48
  Speaker 3: when we look out in the world that data about
• 19:50
  Speaker 3: our vaccines and how confident can we be about their
• 19:53
  Speaker 3: efficacy in that group, and how you know, how tight
• 19:57
  Speaker 3: can we draw our kind of confidence around numbers? We
• 20:00
  Speaker 3: actually can draw it really tightly because that is an
• 20:03
  Speaker 3: outcome that is fairly common.
• 20:05
  Speaker 2: In those in those groups. When we have.
• 20:08
  Speaker 3: Turned into something like looked at for something like, well,
• 20:12
  Speaker 3: how what kind of advocacy are we going to see
• 20:14
  Speaker 3: in kids under under five?
• 20:15
  Speaker 2: One of the things that we run into.
• 20:17
  Speaker 3: There is that this is a group that has just
• 20:20
  Speaker 3: really low risk of serious illness or mortality from COVID.
• 20:24
  Speaker 3: That's extremely lucky on many dimensions. It makes the power
• 20:29
  Speaker 3: really challenging. And so it means when people are looking
• 20:32
  Speaker 3: to those studies and saying, Okay, i'd like to you know,
• 20:35
  Speaker 3: I want to be confident that we're able to detect
• 20:38
  Speaker 3: and impact on hospitalization or an impact on serious illness.
• 20:43
  Speaker 2: Because that effects size is.
• 20:44
  Speaker 3: So small, you need really really big samples to detect that.
• 20:49
  Speaker 3: And because it's hard to get those very big samples,
• 20:51
  Speaker 3: we are sometimes up against really just the limits of
• 20:54
  Speaker 3: what we could conceivably would.
• 20:56
  Speaker 2: Be able to say with these kind of data.
• 21:00
  Speaker 3: And this comes up again a sort of a related
• 21:02
  Speaker 3: point is that when we are trying to detect very
• 21:04
  Speaker 3: rare complications of sort of anything, but in this case
• 21:08
  Speaker 3: of vaccines, the rarer the complication, the larger the sample
• 21:12
  Speaker 3: you need to detect it. And I think that sometimes
• 21:14
  Speaker 3: that sort of missed in some of these conversations. Okay,
• 21:19
  Speaker 3: so that's the second lesson. Okay, So the third point
• 21:25
  Speaker 3: I want to make in some ways is my favorite,
• 21:29
  Speaker 3: and it is that correlation is not causation. So a
• 21:37
  Speaker 3: lot of my academic research is about diet and is
• 21:41
  Speaker 3: about how we both understand the value of particular diet
• 21:47
  Speaker 3: choices and how people make choices about diet.
• 21:49
  Speaker 2: And I just look very interested in diet.
• 21:52
  Speaker 3: And one of the key questions that we sort of
• 21:55
  Speaker 3: ask both as a kind of society when we're developing
• 21:58
  Speaker 3: these kind of dietary guidelines and as individuals is just
• 22:03
  Speaker 3: like what is a good food? What is a healthy food?
• 22:05
  Speaker 3: How should I be eating? What is like the right
• 22:07
  Speaker 3: the right way to diet? So there's a lot of
• 22:11
  Speaker 3: literature on this, a lot a lot of papers that
• 22:13
  Speaker 3: are written about the question of what is the best diet?
• 22:16
  Speaker 3: And there is a very common data approach across these.
• 22:20
  Speaker 2: Across all of these studies.
• 22:23
  Speaker 3: So this approach goes like this, You interview people about
• 22:26
  Speaker 3: their diet, and you ask them what do they eat?
• 22:30
  Speaker 3: And you ask them you know, turns out as like
• 22:32
  Speaker 3: a very active question about how you can figure out
• 22:35
  Speaker 3: what people eat.
• 22:37
  Speaker 2: So one thing.
• 22:39
  Speaker 3: Is you could have them write down everything they eat,
• 22:42
  Speaker 3: but that's actually really hard because once you ask people
• 22:45
  Speaker 3: to write down what they eat, then they eat differently,
• 22:47
  Speaker 3: or they lie or both, and so it's actually really
• 22:50
  Speaker 3: hard to like have sort of care. Like once you're
• 22:52
  Speaker 3: keeping a food diary, you're like, oh, I probably shouldn't
• 22:55
  Speaker 3: have so much ice cream today.
• 22:57
  Speaker 2: Or whatever it is. And so people aren't very good
• 22:59
  Speaker 2: at that.
• 23:00
  Speaker 3: So instead what we use our dietary recall. So you say, okay,
• 23:05
  Speaker 3: we'll think back to last Tuesday and list all the
• 23:08
  Speaker 3: things you ate last Tuesday.
• 23:11
  Speaker 2: But people also not very good.
• 23:12
  Speaker 3: At that, and they also forget a lot of the
• 23:15
  Speaker 3: things they ate or they don't want to tell you.
• 23:18
  Speaker 3: So when you do studies like this, you're kind of
• 23:21
  Speaker 3: like understanding that you're you're going to be sort of incomplete,
• 23:25
  Speaker 3: and so generally they show up with things like, I
• 23:27
  Speaker 3: don't know, like the average person to study will eat
• 23:29
  Speaker 3: like sixteen hundred seventeen hundred calories a day, which of
• 23:32
  Speaker 3: course is you know, kind of substantially lower than the
• 23:35
  Speaker 3: average Americans caloric consumption.
• 23:37
  Speaker 2: At any rate. You ask people about their die and.
• 23:39
  Speaker 3: You're kind of hoping that what they tell you is
• 23:42
  Speaker 3: the sort of approximation of kind of what the truth is.
• 23:45
  Speaker 3: And then you also waive them and you collect typically
• 23:47
  Speaker 3: some other health metrics about there about them, and then
• 23:53
  Speaker 3: these papers correlate. They will look at, like in the
• 23:56
  Speaker 3: data a relationship between the diet choices and people's weight,
• 23:59
  Speaker 3: or help you ask like a very simple question of
• 24:02
  Speaker 3: like when there is when sort of among the people
• 24:05
  Speaker 3: who eat who eat this kind of food, are those
• 24:08
  Speaker 3: people like healthier or less healthy, more or less overweight?
• 24:12
  Speaker 2: Whatever is is your outcome.
• 24:15
  Speaker 3: And typically then when you are going to report out
• 24:19
  Speaker 3: the results of this kind of study, you will look.
• 24:21
  Speaker 2: At whether foods are correlated with.
• 24:22
  Speaker 3: Higher BMI, and if they are, you will sort of
• 24:24
  Speaker 3: say they're unhealthy. So you'll have something like, you know,
• 24:26
  Speaker 3: people who eat too many reasons peanut buttercups are like
• 24:29
  Speaker 3: that's like bad for you because it correlates with a
• 24:32
  Speaker 3: higher weight. So again, very very common set of set
• 24:35
  Speaker 3: of papers. So in sort of thinking about this, at
• 24:40
  Speaker 3: some point I ran a set of like very simple
• 24:43
  Speaker 3: correlate like regressions associating individual foods with weight, and so
• 24:49
  Speaker 3: here is kind of what you get in this sort
• 24:51
  Speaker 3: of standard standard approach. So one thing is there's a
• 24:54
  Speaker 3: lot of different kinds of lettuce that people eat.
• 24:56
  Speaker 2: There's many different lettuce options. So one of the things
• 24:59
  Speaker 2: you find.
• 25:00
  Speaker 3: Is that people who eat iceberg and romain lettuce, those
• 25:03
  Speaker 3: tend to be associated with a higher BMI.
• 25:05
  Speaker 2: So it's like they tend to increase your.
• 25:07
  Speaker 3: Weight, whereas if you choose to eat arugulo or dandelion greens,
• 25:12
  Speaker 3: that will lower your BMI. Also, you have some different
• 25:16
  Speaker 3: options for fat. People who eat margarine tend to have
• 25:19
  Speaker 3: a higher BMI. People eat butter and it's kind of neutral,
• 25:23
  Speaker 3: but if you know a lot of olive oil, that's
• 25:25
  Speaker 3: the lower your BMI. Also, sugar substitutes, I want to
• 25:29
  Speaker 3: be clear. Sugar substitutes are items which contain no calories. Okay,
• 25:32
  Speaker 3: so these are zero calorie items. That's actually also true
• 25:35
  Speaker 3: of lettuce sugar substitutes. What we find is that people
• 25:38
  Speaker 3: who report consuming chemical based sugar substitutes that's like Linda,
• 25:43
  Speaker 3: tend to have a higher BMI, whereas people who consume
• 25:47
  Speaker 3: plant based sugar substitutes like dibia, those people have a
• 25:52
  Speaker 3: lower BMI. Okay, So you sort of look at all
• 25:54
  Speaker 3: of this together and it's like these things cannot actually
• 25:58
  Speaker 3: be causal. Can actually be the case that iceberg lettuce
• 26:03
  Speaker 3: makes you fat? Iceberg lettuce is just water, Okay, it's
• 26:07
  Speaker 3: just like a like a frenchy water item.
• 26:10
  Speaker 2: It doesn't it's not.
• 26:11
  Speaker 3: A thing, right, and and so and at the same
• 26:14
  Speaker 3: and then the same token you know, margar and olive, what
• 26:16
  Speaker 3: kind of similar sugar substitutes are literally all things with no.
• 26:19
  Speaker 2: With no calories.
• 26:20
  Speaker 3: And so it can't be that these items is why
• 26:22
  Speaker 3: I put them up, why I sort of use these here.
• 26:24
  Speaker 3: It kind of can't be that these are really something
• 26:28
  Speaker 3: we'd want to conclude causal relationships from. What's really going
• 26:32
  Speaker 3: on is that the kinds of individuals, the sort of
• 26:37
  Speaker 3: other characteristics of the individuals who consume these different foods
• 26:41
  Speaker 3: are different in other ways. And so in this case,
• 26:44
  Speaker 3: for example, when you look at what kind of individuals
• 26:47
  Speaker 3: are consuming a lot of arugula and dandelion greens tensity
• 26:50
  Speaker 3: people with more education, tensity, people who are higher income,
• 26:53
  Speaker 3: it tends to be people who are doing all kinds
• 26:55
  Speaker 3: of other things that are positive for their health, like
• 27:00
  Speaker 3: exercise or other kinds of dietary choices, and separating out
• 27:05
  Speaker 3: like arugula specifically from you know, generally having a healthy
• 27:10
  Speaker 3: diet or exercising or being a person with access.
• 27:13
  Speaker 2: To medical care.
• 27:14
  Speaker 3: These things are basically almost impossible to do in the data.
• 27:19
  Speaker 3: One of the things that I work on, the last
• 27:23
  Speaker 3: pre pandemic paper I wrote before I started working only
• 27:26
  Speaker 3: on COVID, was about what happens when but the way
• 27:31
  Speaker 3: in which this sort of these kind of things can
• 27:34
  Speaker 3: can occur through information sharing. So one of the things
• 27:39
  Speaker 3: that you see is when we start telling people that
• 27:44
  Speaker 3: something is healthy, the people who adopt that behavior tend
• 27:49
  Speaker 3: to also be the people who are doing all of
• 27:51
  Speaker 3: the other healthy things, and so we get our results
• 27:54
  Speaker 3: are sort of like relationships that we see in the
• 27:56
  Speaker 3: data get more and more and more biased as the
• 28:00
  Speaker 3: result of the messages that we are sending. So if
• 28:03
  Speaker 3: you look back at like the nineteen eighties and you
• 28:05
  Speaker 3: look at the relationship between sugar and obesity, it's actually
• 28:10
  Speaker 3: pretty flat. And part of that is because there was
• 28:13
  Speaker 3: actually this sort of current moment of like sugar as
• 28:17
  Speaker 3: the devil was not.
• 28:19
  Speaker 2: As true in the nineteen eighties.
• 28:21
  Speaker 3: When the like complete breakfast was like a bowl of
• 28:24
  Speaker 3: throasted flakes and a juice and a jam toasting. It
• 28:29
  Speaker 3: was like, this is like when my childhood was like
• 28:31
  Speaker 3: the complete breakfast had a tremendous amount of sugar in it.
• 28:34
  Speaker 3: Now the complete breakfast is like two egg whites and
• 28:36
  Speaker 3: an avocado toast. And so at that point, in this
• 28:41
  Speaker 3: sort of early period, when sugar was not the enemy,
• 28:44
  Speaker 3: you did not see much of a relationship between.
• 28:46
  Speaker 2: Sugar and weight.
• 28:47
  Speaker 3: In the current moment, you see a tremendously large relationship
• 28:51
  Speaker 3: between sugar and weight. And simultaneously you see that the
• 28:55
  Speaker 3: relationship between education and sugar consumption has changed a lot,
• 28:58
  Speaker 3: That the relationship between figurette use in sugar consumption has
• 29:01
  Speaker 3: changed a lot, relationship between exercise and sugar consumption has
• 29:05
  Speaker 3: changed a lot.
• 29:06
  Speaker 2: That as we.
• 29:06
  Speaker 3: Told people don'tate sugar, the people who didn't need to
• 29:10
  Speaker 3: stop eating sugar.
• 29:11
  Speaker 2: Were the people who were also doing all these other things.
• 29:14
  Speaker 3: And that makes these kind of relationships even more challenging
• 29:18
  Speaker 3: to pick up in the data because they are they
• 29:23
  Speaker 3: are sort of effectively a moving like a moving target.
• 29:28
  Speaker 3: I was explaining this research to one of my graduate
• 29:31
  Speaker 3: students the other day, and he was like, are you
• 29:33
  Speaker 3: telling me that we don't know anything about what we
• 29:35
  Speaker 3: should eat? I was like, yeah, basically, yeah, I mean,
• 29:40
  Speaker 3: you know pretty much, that's what it is, builds some
• 29:43
  Speaker 3: most cigarettes.
• 29:46
  Speaker 2: Well yeah, all.
• 29:48
  Speaker 3: Right, so this of course comes up in COVID, and
• 29:53
  Speaker 3: it comes up sort of most I think, in how
• 29:56
  Speaker 3: we think about the relationship between COVID and these on
• 30:01
  Speaker 3: this sort of whole suite of what people call non
• 30:03
  Speaker 3: pharmaceutical interventions. It's a whole suite of things where that
• 30:08
  Speaker 3: we do that kind of aren't vaccines or drugs, but
• 30:11
  Speaker 3: our behaviors. Blackdown's masks, school closures, knowing, door gatherings, all
• 30:17
  Speaker 3: the kinds of sort of like societal restrictions.
• 30:20
  Speaker 2: Part of what's been very challenging.
• 30:22
  Speaker 3: About thinking about which of those we should turn on
• 30:24
  Speaker 3: and off and when we should turn them on and
• 30:26
  Speaker 3: off is that it's very very difficult in the data
• 30:32
  Speaker 3: to separate these things from each other or from any
• 30:36
  Speaker 3: other characteristics of the population.
• 30:39
  Speaker 2: So, like an example, not to like pick on masks,
• 30:43
  Speaker 2: but a good example.
• 30:44
  Speaker 3: Of this is a you know, there's a paper from
• 30:49
  Speaker 3: the CDC a couple of weeks ago in which they
• 30:51
  Speaker 3: correlated mask use with.
• 30:53
  Speaker 2: COVID rates, and they argue that, you know, people who used.
• 30:56
  Speaker 3: Masks were much less likely to have COVID, and that
• 31:01
  Speaker 3: was sort of true in some correlational sense, in the
• 31:04
  Speaker 3: same way that iceberg lettuce is correlated.
• 31:06
  Speaker 2: With being overweight.
• 31:09
  Speaker 3: But what was less true was that you could make
• 31:14
  Speaker 3: a causal claim there. And so in particular in that case,
• 31:17
  Speaker 3: most of the people who use masks were also being
• 31:20
  Speaker 3: tested for their job, whereas most of the people who
• 31:24
  Speaker 3: did not use masks were being tested because they were sick.
• 31:27
  Speaker 3: So there was this very obvious confound which was just like,
• 31:31
  Speaker 3: what are the characteristics.
• 31:32
  Speaker 2: Of these people.
• 31:33
  Speaker 3: There's a bunch of people who need to do surveillance
• 31:35
  Speaker 3: testing for their job and a bunch of people who
• 31:36
  Speaker 3: are coming to your testing site because they're ill, and
• 31:41
  Speaker 3: those things were very highly correlated with mask us.
• 31:44
  Speaker 2: Again, it's not in a case like that, it's not.
• 31:46
  Speaker 3: That that would tell you masks don't work or do work,
• 31:48
  Speaker 3: or it's just like there's no information conveyed, which is
• 31:51
  Speaker 3: in some sense the same point as with the kind
• 31:53
  Speaker 3: of foods.
• 31:54
  Speaker 2: It's not that dandelion greens aren't good for you. They
• 31:56
  Speaker 2: could make good for you.
• 31:58
  Speaker 3: And if you enjoy eating a stringy hemp like substance
• 32:02
  Speaker 3: with a little garlic like, that's fantastic, enjoy yourself. But
• 32:06
  Speaker 3: it isn't obvious from the data that that's true.
• 32:09
  Speaker 2: And I think the same is true of some of what.
• 32:10
  Speaker 3: We see about these these non pharmaceutical interventions. And because
• 32:15
  Speaker 3: that's hard to understand, I think we have generated from
• 32:19
  Speaker 3: the only reason but one of the things that's happened
• 32:21
  Speaker 3: because our data on this is poor, and because we
• 32:23
  Speaker 3: have struggled to really like.
• 32:25
  Speaker 2: Get to anything factual.
• 32:29
  Speaker 3: It is, it has meant that the sort of data
• 32:32
  Speaker 3: hole has been filled with politics. And in a world
• 32:38
  Speaker 3: in which you don't know what the answer is, it's
• 32:40
  Speaker 3: very easy for people to sort of cherry pick pieces
• 32:43
  Speaker 3: of flawed of lawed evidence on both sides and say,
• 32:48
  Speaker 3: you know, this shows that I'm right. Because if there's
• 32:50
  Speaker 3: one thing people like, it is to be right. So
• 32:54
  Speaker 3: so I think this, you know, this is a place
• 32:56
  Speaker 3: where I feel like if we had had had a
• 32:59
  Speaker 3: had a better approach, had a different approach.
• 33:02
  Speaker 2: We might have we might have been able to understand
• 33:05
  Speaker 2: this better.
• 33:07
  Speaker 4: There you go, Oh, okay, So the last thing I
• 33:14
  Speaker 4: want to talk about before we answer questions, and you
• 33:18
  Speaker 4: guys should put put in your questions in the in
• 33:20
  Speaker 4: the thing.
• 33:23
  Speaker 3: Is base rule and here, I'm gonna I'm gonna make
• 33:26
  Speaker 3: you answer a question.
• 33:27
  Speaker 2: So I mean, if you answer the questions. Okay, So
• 33:30
  Speaker 2: we're gonna have a disease. Let's say there's a disease that.
• 33:34
  Speaker 3: Affects one in ten thousand people, and we have a really.
• 33:39
  Speaker 2: Really good test for this disease.
• 33:41
  Speaker 3: This is a test that detects ninety nine point nine
• 33:45
  Speaker 3: percent of cases of the disease and it has a
• 33:49
  Speaker 3: two percent false positive free So so sort of to
• 33:53
  Speaker 3: put that that in your mind means, you know, yeah,
• 33:57
  Speaker 3: ninety nine percent of the time we get right, and
• 33:59
  Speaker 3: then the sort of all chance of a kind of
• 34:01
  Speaker 3: false false positive. And now we imagine we're testing somebody
• 34:06
  Speaker 3: and their test is positive.
• 34:10
  Speaker 2: To what is the chance that they have the disease?
• 34:17
  Speaker 3: Somebody yell in number, say yell a louder Okay, ninety
• 34:25
  Speaker 3: eight percent is a very common answer. Other I'll take
• 34:28
  Speaker 3: some other answers. Fifty good answer, also very common. Points
• 34:42
  Speaker 3: zero zero okay. So we got a good range between
• 34:45
  Speaker 3: ninety eight and points zero zero zero one. And when
• 34:48
  Speaker 3: I do this in class, I usually get about that
• 34:52
  Speaker 3: about that range. Some less people will say one hundred percent,
• 34:54
  Speaker 3: which are quite or zero we usually we get the
• 34:57
  Speaker 3: whole range. But okay, so suppose is like, what is
• 35:01
  Speaker 3: the answer, and the answer is one half of one percent.
• 35:08
  Speaker 2: So think about why this is true, and.
• 35:10
  Speaker 3: It's like, it's actually, I think it is possible.
• 35:13
  Speaker 2: To intuitively get to the point where you get this.
• 35:15
  Speaker 2: Let me walk through it.
• 35:16
  Speaker 3: So we've got our ten thousand, we've got our ten
• 35:19
  Speaker 3: thousand people. Let's assume, to make us a little simpler,
• 35:25
  Speaker 3: that the test is perfect, perfect in the sense of
• 35:29
  Speaker 3: detecting every single case.
• 35:30
  Speaker 2: So rather than ninety nine point nine.
• 35:32
  Speaker 3: It's one hundred percent effective at detecting cases. So we
• 35:35
  Speaker 3: get ten thousand people and one of them has the disease,
• 35:39
  Speaker 3: So the test picks up that person for sure. So
• 35:43
  Speaker 3: for sure we find out that that person has the
• 35:46
  Speaker 3: has the disease. Now there's this two percent false positive rate.
• 35:51
  Speaker 3: Now that number seems small, but there's ten thousand people here,
• 35:57
  Speaker 3: and so two percent of that ten thousand people is
• 36:01
  Speaker 3: two hundred people.
• 36:02
  Speaker 2: So when you run this test, one.
• 36:05
  Speaker 3: Person you're going to pick up who's like a true positive,
• 36:08
  Speaker 3: and you're going to pick up two hundred false positives.
• 36:13
  Speaker 3: And so now you have two hundred and one positive
• 36:17
  Speaker 3: test results.
• 36:17
  Speaker 2: And only one of those people.
• 36:20
  Speaker 3: Has the disease, so it's about a half of about
• 36:23
  Speaker 3: a half of one percent. The intuition here, which is
• 36:32
  Speaker 3: really really challenging, I think unless you sort of like
• 36:37
  Speaker 3: once you work it through, it's easy to see the numbers,
• 36:39
  Speaker 3: but it's very hard to see this intuitively, and the
• 36:42
  Speaker 3: intuition or so the kind of thing that's driving this
• 36:46
  Speaker 3: is that this is rare, that this disease is just uncommon, right,
• 36:52
  Speaker 3: and so most of even with a very limited false
• 36:56
  Speaker 3: positive rate, most people don't have this.
• 36:59
  Speaker 2: So when you like show up, it's very unlikely.
• 37:03
  Speaker 3: So even if you even if the test provides a
• 37:06
  Speaker 3: tremendous amount of information, it started out being super unlikely
• 37:11
  Speaker 3: and so now it's somewhat it's quite a bit more likely. Actually,
• 37:15
  Speaker 3: Like the difference between one and ten thousand and one
• 37:17
  Speaker 3: and two hundred is very very large, but it is
• 37:21
  Speaker 3: still really unlikely because we started at this very unlikely
• 37:25
  Speaker 3: at this very very unlikely place. This comes up in
• 37:33
  Speaker 3: various ways all the time. And the sort of example
• 37:37
  Speaker 3: outside of the COVID context that they came up very
• 37:39
  Speaker 3: recently was was the New York Times published a very
• 37:45
  Speaker 3: big kind of expose effectively on prenatal testing, which was
• 37:50
  Speaker 3: about how there are many new prenatal tests for unusual conditions,
• 37:55
  Speaker 3: for like sort of rare micro deletions or other kind
• 37:59
  Speaker 3: of potential genetic conditions that are really really really uncommon,
• 38:03
  Speaker 3: and there is now much better ways to test.
• 38:06
  Speaker 2: For those conditions.
• 38:07
  Speaker 3: But the thing is that those conditions are really really
• 38:10
  Speaker 3: really uncommon and the tests are not perfect. And so
• 38:14
  Speaker 3: the New York Times story was about the fact that
• 38:18
  Speaker 3: many people test positive on these tests and turn out
• 38:22
  Speaker 3: not to have the conditions, just like in this example.
• 38:27
  Speaker 3: But of course that's true. That is exactly how the
• 38:30
  Speaker 3: tests are designed, right, So that is like completely one
• 38:33
  Speaker 3: hundred percent expected.
• 38:35
  Speaker 2: That is how this should work.
• 38:36
  Speaker 3: If you have a test, have a condition that's very rare,
• 38:40
  Speaker 3: and you have a test with some false positive rate,
• 38:42
  Speaker 3: you will learn a lot from the test, but you
• 38:46
  Speaker 3: won't learn everything. And because it's so rare, it still
• 38:48
  Speaker 3: could be really really likely that actually everything is fine.
• 38:51
  Speaker 2: And what was sort of interesting.
• 38:53
  Speaker 3: About the expose here was that I think for me
• 38:56
  Speaker 3: it revealed not the pitch of the article, which was like,
• 39:00
  Speaker 3: these pre nail tests are terrible and they're ruining people's lives,
• 39:03
  Speaker 3: but in fact, nobody understands base rule, because if you
• 39:06
  Speaker 3: had just kind of understood base rule, you wouldn't have
• 39:11
  Speaker 3: been surprised by any of this. And in fact, instead
• 39:14
  Speaker 3: what was happening was that people and their doctors were
• 39:18
  Speaker 3: kind of reacting to this information in the sort of
• 39:20
  Speaker 3: ninety eight percent ninety eight percent way, or at least
• 39:23
  Speaker 3: in the fifty percent way, rather than in the sort
• 39:25
  Speaker 3: of closer to point one percent way because of this
• 39:29
  Speaker 3: sort of misunderstanding of what we can really learn from these.
• 39:34
  Speaker 2: These kind of tests, and it has real implications.
• 39:36
  Speaker 3: Because one of the implications is that people say, well,
• 39:39
  Speaker 3: actually I don't want that test. I don't want that
• 39:41
  Speaker 3: test because it's not perfect. But that test provides a
• 39:45
  Speaker 3: lot of information. We just need to know how to
• 39:47
  Speaker 3: process the information. And we're running the risk of throwing
• 39:50
  Speaker 3: away pieces of information like that because we are not
• 39:53
  Speaker 3: able to engage with them effectively. There's a kind of
• 39:59
  Speaker 3: just broader picture around Bays Rule that I think is
• 40:03
  Speaker 3: sort of worth making as my kind of final point
• 40:06
  Speaker 3: before we do questions, which is that we talk about
• 40:12
  Speaker 3: we talk about this the kind of key input is
• 40:16
  Speaker 3: what we call our what we call technically call our priors.
• 40:19
  Speaker 3: So in this example here, your prior is the risk
• 40:24
  Speaker 3: before you took the test, So.
• 40:26
  Speaker 2: There's some prior chance that you had it here.
• 40:28
  Speaker 3: It's like one in ten thousand and every time we
• 40:32
  Speaker 3: come into sort of some new information, we come into
• 40:37
  Speaker 3: it with some priors. We come into learning that piece
• 40:40
  Speaker 3: of information with an expectation about how likely is it
• 40:45
  Speaker 3: that this information is true.
• 40:47
  Speaker 2: Or what did I think about what did I think
• 40:48
  Speaker 2: about this.
• 40:49
  Speaker 3: Relationship before I saw this new this new piece of information,
• 40:53
  Speaker 3: or at least that is.
• 40:54
  Speaker 2: How we should do it.
• 40:55
  Speaker 3: So it should be the case when we see a
• 40:57
  Speaker 3: new study about something that we like fire pile away
• 41:00
  Speaker 3: in the back of our mind all of the things
• 41:03
  Speaker 3: that came came before, and we like incorporate this new
• 41:08
  Speaker 3: piece of information in.
• 41:09
  Speaker 2: The context of all of those those things.
• 41:11
  Speaker 3: And what that should mean is if every other thing
• 41:14
  Speaker 3: you have seen in your entire life points you in
• 41:17
  Speaker 3: one direction, and then there's one piece of evidence that
• 41:20
  Speaker 3: comes in that points in another direction. Unless that piece
• 41:23
  Speaker 3: of evidence is like wildly compelling and a billion times
• 41:27
  Speaker 3: better than all this other.
• 41:28
  Speaker 2: Stuff, it shouldn't make you totally flip flop over to
• 41:32
  Speaker 2: everything else to something new.
• 41:35
  Speaker 3: It should move your prior. It should maybe move move
• 41:38
  Speaker 3: you in some in some direction. But we should always
• 41:41
  Speaker 3: have we should always we should be influenced by what
• 41:45
  Speaker 3: has come before. I think that there is often so
• 41:51
  Speaker 3: much of a kind of like newness value, or like
• 41:56
  Speaker 3: what's the latest thing I want to do, the latest
• 41:58
  Speaker 3: thing that we can sometimes forget yet that the latest
• 42:01
  Speaker 3: thing might just sort of like be a false positive,
• 42:05
  Speaker 3: or we shouldn't we shouldn't only listen to the to
• 42:08
  Speaker 3: the newest thing. And this again is something that I
• 42:12
  Speaker 3: think drove in some ways a lot of the sort
• 42:15
  Speaker 3: of lurchy changes that we saw during many aspects of COVID,
• 42:19
  Speaker 3: that we would sort of a new piece of information
• 42:22
  Speaker 3: would come out and would.
• 42:23
  Speaker 2: Be like, okay, now we're like all in on that.
• 42:27
  Speaker 3: So you know, here's one new study from you know,
• 42:31
  Speaker 3: and Lichtenstein that says the following, and now we're going
• 42:34
  Speaker 3: to like all do things totally differently until the next
• 42:36
  Speaker 3: new study comes out the sort of next day, as
• 42:39
  Speaker 3: opposed to saying, look, you know, we're gathering together pieces
• 42:42
  Speaker 3: of information from a lot of places, and we're trying
• 42:45
  Speaker 3: to sort of have them as a kind of moving
• 42:48
  Speaker 3: as a as a moving target and incorporate new pieces
• 42:51
  Speaker 3: of information at any given time and update what we
• 42:54
  Speaker 3: what we know. Instead of that happening, what happened was
• 42:57
  Speaker 3: we went from you know, no mass to all masks,
• 43:00
  Speaker 3: no mask to all masks, or you know, vaccines are
• 43:03
  Speaker 3: one hundred percent protective in all situations like a titanium shield,
• 43:06
  Speaker 3: to like, well it don't work it all and actually
• 43:08
  Speaker 3: they work really well. So sort of like back and
• 43:10
  Speaker 3: forth in these ways that are not helpful for public
• 43:13
  Speaker 3: health messaging and for people's understanding of what's going on.
• 43:18
  Speaker 2: So, you know, think in some ways.
• 43:22
  Speaker 3: The sort of big message that I that I would
• 43:25
  Speaker 3: sort of like to kind of bend here is this
• 43:28
  Speaker 3: idea that if we understood the underpinnings of data better,
• 43:34
  Speaker 3: we would be better at moving it out into varying situations.
• 43:37
  Speaker 2: We'd be better at.
• 43:39
  Speaker 3: The sort of situational fluency of understanding the information that
• 43:43
  Speaker 3: comes in and being able to process it for what
• 43:46
  Speaker 3: it is and see where is it limited? You know,
• 43:49
  Speaker 3: where is it good? Where can I where can I
• 43:51
  Speaker 3: learn from it? And I think that when we teach,
• 43:54
  Speaker 3: we don't do enough, We don't do enough data. So
• 43:59
  Speaker 3: I mean, I think one the other people push for
• 44:02
  Speaker 3: this in different ways than I do. But I think,
• 44:04
  Speaker 3: you know, when I think about what high school kids
• 44:05
  Speaker 3: are learning, I kind of wish they did more data
• 44:08
  Speaker 3: and maybe like a little less trigonometry, or at least
• 44:11
  Speaker 3: we could like have both of them, because these kinds
• 44:14
  Speaker 3: of things come up in how we understand our world,
• 44:18
  Speaker 3: and if people had a deeper knowledge of it, it
• 44:23
  Speaker 3: doesn't have to be infinitely deep.
• 44:24
  Speaker 2: It's not infinitely complicated.
• 44:25
  Speaker 3: I don't think that we need everybody to know exactly
• 44:28
  Speaker 3: how to calculate you know, exactly what the like mathematical
• 44:31
  Speaker 3: formulation of base rule is, or exactly how to calculate
• 44:34
  Speaker 3: the statistical power that you'd need a randomize trial given
• 44:37
  Speaker 3: the effects size and its energy viation or whatever it is.
• 44:40
  Speaker 3: We don't need to know that. But what we need
• 44:41
  Speaker 3: to know are these intuitive messages. And that's something that
• 44:45
  Speaker 3: can be conveyed, and it can be conveyed in ways
• 44:48
  Speaker 3: that are real for people that people can sort of see.
• 44:51
  Speaker 3: And I think that's kind of a key to getting there,
• 44:53
  Speaker 3: is to have people see the ways that this is
• 44:56
  Speaker 3: relevant to the stuff that they're reading every day or
• 44:58
  Speaker 3: they're seeing every day sort of out.
• 45:00
  Speaker 2: In the world. Thanks for listening.
• 45:06
  Speaker 3: If you like what you heard, subscribe to parent Data
• 45:08
  Speaker 3: in your favorite podcast app and rate and review the
• 45:11
  Speaker 3: show in Apple podcasts. You can subscribe to the whole
• 45:14
  Speaker 3: newsletter for free at www dot parentdata dot org.
• 45:19
  Speaker 2: Talk to you soon.