Latest / Quantum Potential / Disrupting Distrust in Political Polling with Josh Clinton
Transcript
- 0:00Because when you just look at the news, you just see people screaming at one another.
- 0:03Yeah.
- 0:03Right?
- 0:034 And the reality is, it's, it's murky, right?
- 0:08The, you know what I love about being in scholarship and being at Vanderbilt is like, we explore the gray, right?
- 0:13Right.
- 0:13There's a, outside the world, you have like bi, it's black versus white, you know, red versus blue.
- 0:17Like all you just, it's very dichotomous.
- 0:19But like the reality of of our world is that, as you know, it's like it's complex, it's nuanced.
- 0:24Today we are thrilled to introduce a true expert in political science and data-driven analysis, Josh Clinton.
- 0:31Josh is the Abby and John Winkelreid Chair and Professor of Political Science at Vanderbilt University, where he serves as co-director of both the Vanderbilt Poll and the Center for the Study of Democratic Institutions.
- 0:42He's also an executive committee member for Vanderbilt's Data Science Institute, bringing a rigorous data perspective to some of the most pressing issues in politics and governance.
- 1:05Welcome to 'Quantum Potential,' a podcast dedicated to curiosity and discovery.
- 1:10My name is Cybele Raver, and I serve as the provost of Vanderbilt University.
- 1:14As a leader in higher education and as a social scientist, I believe that we can transform higher education for the 21st century.
- 1:21At my core, I am wildly curious about other people's work, whether it's in science, the social sciences, the humanities or the arts.
- 1:30This podcast gives us the opportunity to break open new ideas through discovery and creative expression.
- 1:42A two-time Emmy-nominated senior election analyst for NBC News, Josh is responsible for making election night projections— blending deep analytical skill with real-time insights.
- 1:53His impressive academic and professional credentials include a Ph.D. in political science, an M.S. in statistics and a B.A.
- 1:56in economics from Stanford University.
- 2:02He's also been a key figure in the American Association of Public Opinion Research, leading task forces on pre-election polling and contributing to critical analyses of election polling methods.
- 2:12Josh's research covers the dynamics of elections, public opinion, political institutions and lawmaking grounded in advanced statistical methods. With a career that spans rigorous academic scholarship and high-stakes media analysis, Josh brings unparalleled expertise to today's conversation.
- 2:28Josh, thank you so much for being here and welcome.
- 2:30Well, thank you for inviting me and, uh, thank you for that overly ambitious explanation of my, my path.
- 2:35But it's a real pleasure to be here and being at Vanderbilt.
- 2:37I'm looking forward to it.
- 2:38You earned it...
- 2:39I don't know about that.
- 2:40...Professor Clinton.
- 2:40We'll talk about your past and your present, but we, at first, I would just wanna say, you play such an amazing role—not just here at Vanderbilt, but in America— because you are the senior election analyst for NBC News.
- 2:54You are literally the person who projects the winner of our national elections, as well as conducting polls about how and why people vote, both before and after elections.
- 3:05So first, you know, from my perspective, as a person who loves this country and loves democracy, I really want to thank you for everything you do.
- 3:12And second, I have a million questions about how that works, what it's like.
- 3:17Our election last week must have been the culmination of weeks and months of intense work and teaching as you help the rest of our country figure out our choices in voting for our nation's leaders— and as you safeguard, not just voting, but our trust in polling and trust in voting.
- 3:34I'm gonna just pass it over to you, but just wanted to highlight that in a recent talk you gave, uh, for C-SPAN, you talked about how we as Americans love scores and, and following scores— even political scores like statistics about, uh, who's who might win.
- 3:49Uh, we love following the results of polls, and at the same time, our polls can get it so wrong.
- 3:53And you gave this great example from 1936 in the run-up to the presidential election between Franklin Delano Roosevelt and a candidate I'd never heard of before, to be honest, named Landon.
- 4:04Landon.
- 4:04Yeah.
- 4:05Uh, and that the polls had predicted overwhelmingly the likelihood of Landon winning based on the view that it was time for a change.
- 4:11But yet Roosevelt won by a landslide.
- 4:13So I'll just, I'll kick it over to you, Josh, but I just loved that example and how you were really teaching us to think about polls and data.
- 4:20Yeah.
- 4:20Well, it's one of these things that, you know, what I love about my job is that not only do I get to immerse myself in, like, the real world and not just the ivory tower, but then also more importantly, try to convey that to the students at Vanderbilt, but then also citizens around the world.
- 4:33And I think the world that we live in now today, that you can't step three feet without getting hit by some number, right?
- 4:39And some, and both sides have got their own numbers.
- 4:42Totally.
- 4:42And the whole notion of what the truth is and how we understand those numbers is really, really critical, and it's also a lot of it's untested.
- 4:49Like, you know, you say I can 48.7% this.
- 4:53But like understand the numbers, like what that means, where it comes from, the type type of biases that might be there and kind of who's trying to tell you those numbers—it's such a critically important skill to understand, like, where the world is, like, what it looks like descriptively.
- 5:06But then also, like, how we make our way as citizens of the world.
- 5:09And, like, you know, Vanderbilt students trying to teach them how to navigate those numbers so when they're future leaders, and when they're leaders now, frankly, right— how can they get the right decisions that they, you know, data they need to make decisions.
- 5:20And then also, I think, instilling with them, I think one thing about that data that is really important, although a lot of people lose sight of this, is the absolute humility that you need when you work with data, right?
- 5:30I love that.
- 5:30Like, when you do polling, like, you'll be wrong more often than right.
- 5:34I mean, there's, like, there's an unescapable error.
- 5:36Like, even if you do things perfectly, and that's true with most things in life.
- 5:40But, like, you've got this weird disjunction where you can quantify something and it, and it feels really, really certain, right?
- 5:46And really, really precise.
- 5:48But then if you take that and you, you can overinterpret what that means, and we do that all the time as human beings.
- 5:53Like, you're very certain about something without being, having a humility of understanding, well, that number is a result of a bunch of processes, both stated and unstated.
- 6:01Unless you have a full understanding of what that means, like, you're likely to kind of make it very, very wrong, like that 1936 poll. Like, they interviewed, like, 2.4 million people in 1936, 5.5% of the popul ... voting population.
- 6:13Which is crazy.
- 6:13Crazy.
- 6:13Like, most pollsters would kill for a poll that large.
- 6:16Uh, and they got it, like, massively wrong, right?
- 6:18Yeah.
- 6:18And even nowadays, you see pollsters behaving massive polls, and in 2024, like, the polls got it better than 2020, but, you know—close but no cigar.
- 6:25Right?
- 6:25They still didn't really fully pick up the shift towards Trump...
- 6:29Yeah.
- 6:29...that pollsters have been struggling with.
- 6:30Yeah.
- 6:30And so like I love that you're trying to get to that, right?
- 6:32That there's this way in which we use numbers.
- 6:34I certainly, as a leader, I use numbers.
- 6:36I love numbers.
- 6:37I wanna see evidence of the benefit of what we're doing or the benefit of what we've invested in, right?
- 6:43Yeah.
- 6:43But I love that you also use that frame of humility and also, uh, sort of expansion of people's not only understanding of numbers, but empowerment through numbers, right?
- 6:52Like, don't get caught into believing a number just because somebody said it, but come to understand how that number was generated. Push back.
- 7:00And also like on that point that the scoring numbers.
- 7:02Like, like, like we love to know the score, right?
- 7:04And so you always, we're always tracking metrics, right?
- 7:06And so sometimes we get obsessed with metrics that don't really mean anything, right?
- 7:10But like there's something to follow.
- 7:11Like I need to tell a story.
- 7:12And so the election is like one giant story of us following some number that has no real connection to what necessarily is what's gonna happen election day.
- 7:20But every single beave and wob um, kind of like gets blown outta proportion.
- 7:25Like the night before, like, there was the Iowa Seltzer poll, and so, like, depending on what network you were watching, right, they spent like 48 hours talking about, like, one number— which was ludicrous, if you think about it.
- 7:36So, I mean, you think about a conventional, you know, pre-election polling nowadays.
- 7:39And so like the average state-level poll is 800 people.
- 7:42So to get 800 people, a pollster needs to, needs to call 40,000 people.
- 7:46Wow.
- 7:47Right?
- 7:47So right off the bat, you know you got a problem because the 800 people you talk to are abnormal relative to the 39,200 people who did not.
- 7:55Right.
- 7:55So right now you as a pollster.
- 7:56That is crazy.
- 7:57You've got this problem, right?
- 7:58And so I'm, I'm talking to people.
- 8:00I know they're weird because they're talking to a pollster, right?
- 8:02And so, and then you've got the second problem where I need to make them look like...
- 8:06They're generous.
- 8:06...yeah, they're, that's right.
- 8:08They're generous.
- 8:08They're kind of, like, they're, they're willing.
- 8:09They're willing to talk to, you know, which again is great if you're a pollster, but then also, like, if you're trying to capture the electorate, right, where some of the electorate may not want to talk to pollsters, right?
- 8:18Given the nature of our political discourse, then you've got a real problem.
- 8:21And the problem is that then as a, as a pollster, you need to start making decisions about how you make those 800 people look like what you think the electorate's gonna look like— which you don't know, right?
- 8:31Right.
- 8:31Because it hasn't happened yet.
- 8:32Right.
- 8:32And so, right, again, you're basically introducing art in some sense, right?
- 8:37Or judgment or discretion by the part of the pollster to try to make what's a scientific process— like random sampling with no, no non-response.
- 8:44Right.
- 8:45That, you know, science is good on that.
- 8:46But what we actually do in the real world is not really closely tied to science, like, at all.
- 8:51Right.
- 8:51And so...
- 8:51You're shaping but not cooking.
- 8:53Exactly.
- 8:53It's such an interesting...
- 8:54Yeah.
- 8:55...sort of, and, and what are the brakes on that set of decisions?
- 8:58Is it largely because you just, you know, logistically don't have enough time to keep getting the model more and more perfect when you're trying to figure out how to make those 800 people represent...
- 9:10Yeah.
- 9:10...the 49,000, you know, 200 people that, uh, or 39,000, whatever the number was, right?
- 9:16Yeah.
- 9:17Or is it just that you, you know, that there's a rip cord that you pull that says, "this is, you know, this is as far as I'm willing to go" and trying to figure out how to analytically, you know, shift and shape, um, so that you feel like you get the right enough answer.
- 9:32How do you make that judgment?
- 9:34Well, I think everyone makes it differently.
- 9:36And part of the problem is in that there's not transparency, right?
- 9:38And so if you see a number from a pollster, they've done a lot of decision- making that you have no idea what that is.
- 9:44Where they just, like, "Hmm, I wanted this result to show Trump plus two," right?
- 9:49Or did they try to make a best effort to try to do what the reality was?
- 9:52But, like, some of that error is inescapable, right?
- 9:55No one actually knows what the electorate's gonna look like, right?
- 9:57And after 2016 and 2020, a lot of the pollsters said, "Well, we struggle a lot to get Republicans in particular, Trump-supporting Republicans, to take our surveys." And so that, you know, I led the task force that kind of analyzed pre-election polling and said, "Well, this is a real problem for polling.
- 10:12But to be honest, I'm not sure there is a solution and we can see what happens." Yeah.
- 10:17And so what pollsters then try to do is they, you know, you look at your sample and you know, you try to make it look like the demographics of the electorate.
- 10:24But what's that look like?
- 10:25No one has any idea, right?
- 10:26I know what 2020 looked like.
- 10:27Well, And I love that you started addressing that, right, by saying new voters...
- 10:30Yeah.
- 10:31...could potentially look really different in different states.
- 10:34Exactly.
- 10:34Right.
- 10:34So by the virtue of age and you know, massive shifts in birth rates 20 years ago and who's here in the United States today, it may be different than who was here in the United States in voting, you know, 20 years, uh, earlier.
- 10:49And then the candidates themselves, right, with you have concerts and get out the vote efforts.
- 10:53And so are young people gonna show up or like, you know, old people, like are they gonna be crossover voters?
- 10:57And so, like, that's all unknowable, right?
- 11:00And it's all foundationally unknowable.
- 11:02But as a pollster, if you put out one number, you're making a lot of decisions that aren't really being communicated.
- 11:07And so, so I did this work.
- 11:08I, so I just took a poll that was, you know, I commissioned for part of the work I'm doing in my Carnegie project.
- 11:13Um, and just wanted, see, like if I'm try to, if I wanted to be a pre-election pollster, which to be clear, I do not, right?
- 11:18But if I wanted to, like what were the set decisions that I'd have to make to try to make this, to model what the electorate is?
- 11:23And the reality is, is like depending upon the answers that I was willing to make, I could move the top line Harris-Trump margin by eight percentage points.
- 11:31Wow.
- 11:31Right?
- 11:31And so that's, that's decisions that I, as a pollster trying to not bias it, but trying to just do a straight-up read.
- 11:37But, like, how do you account for partisanship?
- 11:39Or how do you account for the demographics?
- 11:41And then you can make reasonable assumptions to make tremendous variation.
- 11:44And now that's above and beyond just the other randomness that's going on.
- 11:48And so now you're talking about, all right, so even its higher media apparatus talking about who's gonna win Michigan, right?
- 11:54Which, best case scenario, it's like a two percentage point...
- 11:58...and you're trying to use a pre, pre-election poll to do that.
- 12:01But the decisions you can make can move that number 8%.
- 12:04And so, like, it's like a surgeon trying to do surgery with a butter knife.
- 12:06Like, you can get close, right?
- 12:08You can cut out an organ, but it's not gonna be pretty, right?
- 12:10It's not.
- 12:11And so in that project, and congratulations by the way, the Carnegie Fellowship is incredibly prestigious.
- 12:16We're thrilled, you know, I'm thrilled as the university's provost that you won that award and that you have that kind of support to do that work.
- 12:23When you're doing that kind of work, how do you sort of unpack the level of complexity around polarization and trust?
- 12:31You know, one of the things that you've pointed out is that if increasingly we become polarized as a nation, we have lower and lower trust in the processes and the information that we exchange to make these really important decisions.
- 12:43No, that's, I mean, that's a great, and that's something that, that keeps me up at night.
- 12:46One of the many things, it's like trying to figure out what that means, right?
- 12:49'Cause we, you know, we got, we know we have this problem with pre-election polling and people who take our polls, right?
- 12:55And that's a generic problem.
- 12:56That's just not pre-election polling, right?
- 12:57But like, as someone who does the Vanderbilt Poll for example, it's something that we struggle with.
- 13:01Like, how do we make sure that we're counting the right people in a state like Tennessee, right?
- 13:06So we can kind of reflect, like, what the public opinion in the small-D Democratic way is, right?
- 13:10I mean, that's absolutely foundational.
- 13:12'Cause polling has a such an important role to play.
- 13:14Because otherwise we're just, like, listening to the voices on Twitter in the streets, which you know, are loud, but not necessarily reflective of the overall population.
- 13:22So, you know, what we try to do is, like, you know ... normal polling is a little bit better situated than pre-election polling because I don't need to make this assumption that ...
- 13:30I don't need to try to figure out what the election may or may not be.
- 13:32Like I know what the population looks like, I know what registered voters look like, but that doesn't really fully get out around the problem that, well, if someone doesn't trust the media and doesn't trust polls, like, you're not having their viewpoint right in your survey.
- 13:46Right.
- 13:46And so then that requires more complex things to have to figure out like, well, do you have any of them?
- 13:51Right in the view?
- 13:52Right?
- 13:52So can you drive down their views a little bit more?
- 13:54But then also kind of more generally, like this thing I struggle with honestly is, like, how much of this is in some sense performative, right?
- 14:01And so like...
- 14:02Interesting.
- 14:02...when people say, you know.
- 14:04And this is like, I mean, I don't know how much to trust, sometimes, the polling that says like, you know, people, they don't trust the election results right, in that states.
- 14:11And so like, that's what my project is looking at.
- 14:13And so it's looking at differences between, like, where you live, because now we're increasingly geographically sorted.
- 14:18And so you're a red state or blue state?
- 14:20And not only do you live in a red state or blue state, but increasingly how you vote is very different between red states and blue states.
- 14:25And so our democracy and the rules of our democracy, because ev..
- 14:28because of federalism, vary very tremendously, but our national elections are, like, wicked tight, right?
- 14:34Using a technical term, right?
- 14:35So like the House is very close, the presidency's real close.
- 14:37The Senate's really close.
- 14:38And so, like, we've got these really, really close elections being decided by people living in politically uncompetitive areas.
- 14:46Voting, using very different voting regimes.
- 14:48Yeah.
- 14:48And so that's not a great recipe for legitimacy.
- 14:51And we're huge, right?
- 14:52And we're huge, right?
- 14:52We're not a small country.
- 14:53And we're not. Huge.
- 14:54And we're, and a variety a diversity of opinions and experiences and, you know, and, and opinions.
- 14:58And so...
- 14:59...of everything.
- 15:00And so how do you put that all together, right, in this messy pot called the United States and kind of get a, a democracy and get legitimacy in the outcome when, you know, people vote differently?
- 15:10And so I can't say that I have the answer.
- 15:12I can't promise I'll have the answer at the end of my investigation.
- 15:15But I hope to get a little bit closer to, kind of, see what are the tensions there, where are some points of commonality.
- 15:20Because when you just look at the news, if, if ... insofar as it is news ...
- 15:24like, you just see people screaming at one another...
- 15:26Yeah.
- 15:26...right?
- 15:26It's, like, left versus right, black versus white.
- 15:29And the reality is, it's, it's murky, right?
- 15:31The, you know, what I love about being in scholarship and being at Vanderbilt is, like, we explore the gray, right?
- 15:35Right.
- 15:36There's a, outside the world, you have, like, it's black versus white, you know, red versus blue.
- 15:39Like all, you just, it's very dichotomous.
- 15:41But, like, the reality of of our world is that, as you know, it's, like, it's complex, it's nuanced.
- 16:03You know, when you're thinking about polarization and being, um, in a very highly local context compared to being in a national context.
- 16:10Yeah.
- 16:10Your description of growing up in Stittville, New York, um, my dad for a while was in Ogdensburg, New York.
- 16:15I lived, you know, in Ithaca, where I traveled through a lot of upstate counties that look nothing like what I grew up in, in terms of 96th and Madison and Manhattan, right?
- 16:24Yeah.
- 16:25And that, that those two could both be in the state of New York.
- 16:28Or that two places, you know, I didn't know before I got here that Middle Tennessee was Middle Tennessee, which is totally different than Knoxville and totally different than Memphis, right?
- 16:38Yeah.
- 16:38So the extent to which, you know, your own experience being in a community and within a household with loving parents who hadn't themselves had the opportunity to be in college.
- 16:49You know, I just think a lot about how willing and open you are to, as you said, be humble in the face of what you know and what you don't know about other people's opinions.
- 16:58And I thought I'd just ask you to talk a little bit about that and how you ended up where you are now.
- 17:03I have no idea.
- 17:04So I mean, like, I'm just, I'm just very lucky and grateful for the opportunities I had.
- 17:08So, I mean, I grew up in Stittville, New York, which is like a thousand people.
- 17:11Uh, you know, my mom was a registered nurse.
- 17:13My dad, like, dropped out of community college to be a land surveyor before he was laid off, uh, in, like, the nineties or whatever.
- 17:19And so that convinced me, I didn't really wanna work for corporate America because, like, that was, like, a searing experience.
- 17:24But, like, I had no idea what college was, like, you know, watching TV.
- 17:26Um, most of my college, uh, went to a community college or joined the family farm or joined the military.
- 17:31Um, I just happened to be good in social studies because I like debate.
- 17:34And so I thought, "Well, maybe you'll be a lawyer then," because'LA Law' and all that stuff, and so, I mean, looking back, like, it's, I'm horrified, especially having just gone through this, like, ginormous and ridiculous process with my son from last year.
- 17:45But like applying for college, like, uh, I applied to Michigan State'cause, like, I did well on my PSAT score.
- 17:50And so they said, "Come take a scholarship," and so my parents got in a car 'cause, like, I think I was on the airplane once, and so we drove across the country to Michigan...
- 17:57Wow.
- 17:57...to take a multiple choice test, which I did not do well, so I didn't get a scholarship there.
- 18:00So there I, I stopped and kind of toured the University of Michigan and, like,"Oh, they're kind of ... Wolverines—like that,"'cause I liked the movie'Red Dawn' when I was a kid.
- 18:07But, but, but anyway but, like, didn't get in there either.
- 18:10Uh, I applied to the Cornell, the state School of Industrial and Labor Relations because, like, a kid in my high school—who was like my idol because I was a freshman, he was a senior on the debate team— and so he went there, and it was a state school and so, like, as a state ...
- 18:21New York state they could afford that part, but, you know, didn't. Got denied early decision.
- 18:25Every point of the way I got kind of denied there, so I still have a little grudge about that.
- 18:29But, you know, chips are good, I guess, on your shoulder.
- 18:31But, but in the last day, like, I also won this Xerox award from the University of Rochester.
- 18:35It's the best social studies one.
- 18:36And my, my, my friend who won the Bausch and Lomb, the science one, he said, "Josh, you applying here?
- 18:40It's like, "Where's that?" No, like, I don't, you know, I'm, ... Wow.
- 18:43...because I was gonna, I'm gonna go to Cornell.
- 18:44Like I, I had the hat, I had, like, I was, like, all in, like, that was my dream school that I was gonna go to and, like, Rochester? Like, no.
- 18:50But then he said, "Well, they waived your application fee." And so the last day, literally the last day I get out a pen—'cause this is the day you had typewriters.
- 18:56Like, I'm aging myself and, like, I had ... all my other applications were, like, with typewritten errors or whatever.
- 19:01But this one I did in blue pen.
- 19:02Wow.
- 19:02Sent it off and kind of got accepted.
- 19:04And it turns out, like, at the end of the day, like, that's the only place I got in besides Michigan State.
- 19:08And so I would've been, I don't know what it would've been done.
- 19:10And so I went there, right?
- 19:12And it was interesting 'cause they, you know, they had a preview weekend there, and I ... remembered being there when everyone else was hearing about the Ivy League admissions and, like, everyone else was getting in other Ivys and said, "We're not going to Rochester." And like my, and I realized that I got denied.
- 19:23I was, like, "Well this is my home now.
- 19:24Like, this is where I'm gonna be." That's right.
- 19:26And in retrospect, it was like the very, very best.
- 19:29But like, to just give you a sense of, like, how crazy I was.
- 19:31Like, I wanted to go to Wash U because they sent me a catalog, and it had a castle in the front.
- 19:35And as a kid growing up, like, I loved castles and dragons, and my parents said, "No, you can't go there.
- 19:38It's too far, and we can't afford it." But like, anyway, that, that was like the level of, of thinking that I was engaged in, right?
- 19:43Totally.
- 19:43Like, totally not, right, profession-wise.
- 19:45I, I didn't have a too, like, I was just, like, insane.
- 19:47And so I ended up at Rochester, thought I was gonna be a lawyer.
- 19:50I, I had no idea what that lawyer was until I talked to one of my friends who basically was in from Manhattan and interned for a lawyer.
- 19:55And so they, their life actually is not great, right,'cause they're on the call all the time.
- 19:59And say, well, and I said,"You don't get to argue?" Like, that's what I loved about it.
- 20:02Like, no, it turns out no, you gotta have the billable hours so that's what you did.
- 20:04So be a political scientist.
- 20:05Well, yeah.
- 20:06So, so like, where's that come from?
- 20:08Like politic, like yeah.
- 20:08Argue with people.
- 20:09Argue with people.
- 20:10That's right.
- 20:10Yeah.
- 20:10Do data stuff.
- 20:11In political science.
- 20:12No, but I took a political philosophy class, and I remember I still, I still have the paper.
- 20:15And he said, like, "You have a talent for abstract analytical thinking.
- 20:18Cultivate it." And so that was my professor Jim Johnson.
- 20:21Amazing.
- 20:21Uh, I, I took as a sophomore and I had this like summer experience, which is something, why I love the Immersion Vanderbilt experience, because that literally changed my life.
- 20:28And so I said, "I've got this $3,000, you know, I'm, last year for summer I worked at a summer camp, like, I'm not sure what I'm gonna do this summer." And he said, "Well, you can just, like, you know, work with me." And so he let me kind of read some obscure philosopher or whatever and kind of, like, go deep and, and I just, and I just loved it.
- 20:44It's not what I do now.
- 20:44But that's just so wild.
- 20:45I mean, listen to the data points you just reeled off, right?
- 20:49The cover of the university brochure, right?
- 20:53I was not a sophisticated thinker.
- 20:54No, but like just for what kids or what any individual has right now to figure out, making a huge decision about whether they can go to college and where to go to college if they can go to college, right?
- 21:06Yeah.
- 21:06So you know that your parents were willing to drive you to take a test, which you subsequently didn't do well enough on.
- 21:12Yeah.
- 21:12And yet you had a friend who convinced you to apply on the last day of the application.
- 21:19I know.
- 21:19And then.
- 21:20That you figured out your career because a professor took on that incredible opportunity and responsibility of mentoring you.
- 21:28Like, what an amazing set of data points that leads you to a Ph.D. from Stanford.
- 21:32It's just wild.
- 21:33No, I mean, just, I mean, I think I every, and I think this is like something I, you know, it's like, it's that, you know, being aware that, you know, wor it, it takes work to get where you are, but it also takes luck, right?
- 21:44And some, you know, and so I think like just being humble and app, you know, 'cause you appreciate like, "I am here, but for like a few decisions that went the other way." And like comp, you know, who knows what would be the outcome.
- 21:55And so I like, so I mean that's, that's good and bad, right?
- 21:58And so, but it's also, I think, you know, I kind of grounds you I think in a really, you know, important way.
- 22:03Like, I have no idea, like, where I was gonna be when I started, you know, college.
- 22:07And even when I started grad school, I started doing something other than what I ended up doing.
- 22:10It's just like...
- 22:11And do you see versions of yourself or versions of who you might have been had not you gone to take those steps in your data for other constituencies or other voters or other groups of people who are, you know, do not have Ph.D.'s from incredibly elite private universities.
- 22:29But who, um, you know, also did well in high school and didn't have a college education, right?
- 22:35Like, do you see yourself sometimes in the data or do you see who your friends are or who your parents, uh, were?
- 22:42Or you know, that way in which, um, you have a clear sort of sanguine understanding of how diverse any data set, uh, or how, how diverse the people are who are represented by any data set.
- 22:57I, I mean, I think that's right.
- 22:58I mean, I think like, you know, we quanti I mean, it gets back to the point of like being overly precise, right?
- 23:02And so like we can quantify something, but that quantification, I think omits the complexity that human beings are, right?
- 23:09Yeah.
- 23:09And the complexity of experiences, the diversity of experiences.
- 23:12And so it, it's always that navigation between trying to quantify something but then also kind of give full voice to the richness of, of the experience.
- 23:21And so like, you know, I, I do too many different projects and so like, I'm, I'm, you know, a colleague ... John Size and I started a project looking at the politics of higher education because that's a hugely politicized topic nowadays.
- 23:32But like the, even the way we talk about that is like, so coarse, right?
- 23:36College-educated.
- 23:37Well that, you know, that doesn't mean... Right, right.
- 23:39...doesn't mean what we think it means, right?
- 23:40Right.
- 23:41The, the, the diversity of what that means is so, is so broad.
- 23:46And so, like, even the way, you know ... so we do survey data and you ask, "Are you, do you have some college or college?" But like, what it means to be college is, you know, like my pathway you, or the pathway some of my, you know, my colleagues took in high school was like, you know, two-year—and that's what my brother did, two years community college and then transferred kind of the, to the local college.
- 24:02Which is a very different experience than, like, a four-year school...
- 24:04Right.
- 24:05...right?
- 24:05or a four-year residential school.
- 24:06Or a four-year residential school or, yeah.
- 24:08Or Right.
- 24:09And you know, my mom had like an, you know, two years associate's nursing degree, right?
- 24:13And so like, like which I guess is that, like, some college like?
- 24:15So it's like, so like we try to reduce things to these boxes sometimes, and life is not always fitting into neat boxes.
- 24:21And so that's something, like, as someone who does statistics and data and, like, that is something that I grapple with every single day.
- 24:28But I also think that's the essential nature of, kind of, being humble, right?
- 24:32And kind of being aware of some of the downsides and some limitations of what you do.
- 24:35Because like I can quantify something, and I can throw it in some giant LLM or, you know, artificial intelligence to solve the world or whatever, but, like, you're making a lot of reductions, right?
- 24:45And a lot of assumptions there, and you're coarsening the world to kind of make that, to make those inferences.
- 24:49And then if you're not cognizant of that, like, you can end up being in a really bad place in terms of getting...
- 24:54Yeah, no, you're totally convincing me that most of the time I focus on the ways in which scholarly work can generate incredible solutions to the problem, right?
- 25:03Yeah.
- 25:03Whatever the problem is.
- 25:05But you, this conversation reminds me that sometimes the tools that we have can actually be driving some of the problem— that the reductive nature of quantification, uh, you know, that this voter is represented by either their political affiliation or by their region, or by their ZIP code, or by whether they've, uh, attained a four-year college degree is, is, you know, in that moment can be potentially really reductive.
- 25:31And it's, and it's super hard, too, 'cause like, like, everyone's, like, "What does the election mean?" Right?
- 25:35I mean, everyone's kind of grappling with that.
- 25:36And so like, I, I, you know, I pity you if you're on social media trying to read all the threads where people scream at each other about like what it does or does not mean.
- 25:43But like, that's like such a foundationally hard thing to do, to figure out, right?
- 25:46Because the reality is it's probably complex.
- 25:49Yeah.
- 25:49Right.
- 25:49And there's, and just because I can predict a behavior using something, does that mean that's causing that behavior?
- 25:55Uh, and like we want certainty, like we want answers.
- 25:57And now I sound like I'm like an anti-science, but like, 'cause I love, you know, this is what I do.
- 26:01Like, I love numbers, I love quantifications, and I love statistics and trying to do that.
- 26:04But I think it's absolutely essential for everything that we do to kind of have some semblance about, like, how sure are you about that?
- 26:12Or if you do wanna make a number, like what are the bounds of that?
- 26:15And not just, like, the statistical bounds, but then also, like, where does that data come from?
- 26:18Yeah.
- 26:18And like, I think sometimes we make the massive mistakes is where we don't account for that.
- 26:23Right?
- 26:23And you are also really great at saying,"Not only should we account for it, but we should be really transparent about it," right?
- 26:29Yeah.
- 26:30You've got this really, really great paper that focuses on the causes of mistrust and how to build it back in terms of surveying more, um, broadly survey data more broadly.
- 26:42Um, I really love this paper because it's written for scholars and scientists, but it's also really, really readable in the sense that it's, you know, makes complete sense, I think, to somebody who isn't in academia.
- 26:54And it's on ... the title's "Protecting the Integrity of Survey Research," Um, and it's in, uh, PNAS, which is super, you know, high-level science, but written for everybody.
- 27:04And I thought it would be great to just have you read a little bit about why survey researchers can arrive at dissimilar conclusions.
- 27:12Um, would you be willing to just read the paragraph that I've highlighted in yellow?
- 27:15Sure.
- 27:19Decades ago, adults in the U.S. could be reached at a particular landline phone number, and most receive very few requests to participate in surveys of any kind.
- 27:27When they're invited to take a survey, many people agreed to do so.
- 27:30These assumptions no longer hold true.
- 27:31Cell phones have replaced landlines, and many people avoid calls from unfamiliar numbers. At the same time, even when reached by phone, some parts of the population are now much less likely to accept invitations to be interviewed than they once were.
- 27:41Where a survey researcher was in earlier times expected to elicit answers for more than 60% of target sample, response rates now reach rarely reach 10%, and nowadays is actually like 2%.
- 27:52So Wow.
- 27:53Consequently it's now more expensive, time-consuming and difficult for survey research to secure the type of representative samples for which the most repeatable surveys have long been known.
- 28:01Declining response rates and changing patterns of non-response are among the factors that affect the quality of the population estimates.
- 28:07It's been reported, for example, that Democrats are more likely than Republicans and independents to be agreed to be interviewed.
- 28:15But I think, like, that's all true, right?
- 28:17And so one solution is like,"Well, I don't trust any polls, therefore I'm gonna give up." But I think, like, you know, apropos of everything we talked about, like, it's really hard to reduce people to a number.
- 28:27But I think that you can't, we can't give up on that, you know? We just have to have the proper understanding of, kind of, how much uncertainty we have.
- 28:33Because like if we don't do that, then like, how do we know, right?
- 28:37So like the thing that polls can do is they can help make sense of, like, the election, for example.
- 28:42Like, I don't know, no one knows, like, why voters voted for the way they did, right?
- 28:46All I know is that someone has supported President Trump or President Harris, right?
- 28:50But the reasons that they support turn out to be really important for like democratic accountability, for democratic legitimacy.
- 28:56I mean, if, 'cause, and if politicians think that voters intended something quite different than what they actually did, like, the downstream consequences of that can be catastrophic.
- 29:04So, and you feel like some they're, they're sending a message.
- 29:07Voters are sending a message, but we don't know what the message is.
- 29:09Yeah.
- 29:09And I think it's incumbent upon political scientists and, kind of, you know, to try to in, in more general, to kind of figure out kind of what that message is because, and that's essential, I think, I mean, politicians also, I mean if they don't figure it out, they'll get voted out, but like, but the meaning of an election, right?
- 29:23The message that that sends, I think, is really important for understanding kind of where our country is.
- 29:27And polls try to give us, you know, the best polls, like what we do in Vanderbilt Polls, like, no one polls in Tennessee because, like, it's not interesting nationally, right?
- 29:35It's not, you know, that interesting even statewide in terms of the elections, right, given how solid Republican it is.
- 29:39But that doesn't mean that people don't have opinions about the policies that are being promulgated and kind of enacted.
- 29:46And a question of like, how well does the state government reflect the opinions of the people, right?
- 29:50Absent a poll, how would you know that, right?
- 29:53Right.
- 29:53People who write letters to the editor— well, that's not, like, normal people.
- 29:56And people like go on social media, protests— well, again, that's like not someone just kind of worried about going about their everyday life.
- 30:02And so the best thing about polls when done well is it gives that elevation of just the normal person.
- 30:23I love that you also spend an equal amount of time not only trying to make the tool better and to solve the problem, but to teach us all about that.
- 30:31So the "normal person argument," I hear you using that as shorthand for what you mean when you say representative sample.
- 30:38Is that right?
- 30:39Yep.
- 30:39So I just thought I'd ask you to explain to an audience, you know, who's listening because they're curious, you know, what you mean by a representative sample.
- 30:48Yeah, so, so again, when I'm for a Vanderbilt Poll, what we do is like, we have a list of registered voters in Tennessee, right?
- 30:54So we know what that looks like, demographically, age, you know, and so like, that gives us the basis for which we can then kind of randomly select voters.
- 31:04And so we can just use like just a random number generator to randomly select voters from Tennessee.
- 31:07Or when we do a poll from Nashville, we have a list of all the addresses in Nashville.
- 31:11And so we can randomly select addresses and mail them all letters to invite them to take surveys and so on the front end, right, I at least know what the population in Tennessee, or what the population of Nashville looks like broadly.
- 31:23And so when I get back my, So if my license plate has a number five in it...
- 31:27Yeah.
- 31:28...and your random generator picks the number five.
- 31:31And it we're on the basis, not of my registration, my voter registration, but my license plate.
- 31:36Yeah.
- 31:36I might get a letter that says, That's right.
- 31:38What is your opinion about oil and gas versus renewable energy?
- 31:42Yeah.
- 31:43And so you, so, and then we get that opinion back and so we know from survey research that the people as we talked about that who take surveys aren't necessarily reflective, right?
- 31:50They're kind of typically more engaged or more likely to be college-educated and older, right?
- 31:55And so then the first task of a survey researcher after they write the question, right, is like, "How do I make this reflect the population I'm interested in?" right?
- 32:02And so this is where I can see well I've got, you know, 40% of my sample got a college degree, but I know that's not true among registered voters in Tennessee.
- 32:10And so that means I need to kind of fight, you know, you know, count the opinions of those without a college degree more.
- 32:15And with those with a college degree slightly, you know, kind of less.
- 32:18And so I can help do that because like I know what the state looks like or I know what the city looks like along these kind of, you know, demographic lines.
- 32:25Now that said, like, it's still possible that I'm not missing.
- 32:29Um, you know, so I think the election polls in 2024, once again, probably did not get, right, supporters of President Trump who are voting but not taking polls.
- 32:37And will a Vanderbilt Poll miss those people?
- 32:39Probably, right.
- 32:40I'm not gonna lie.
- 32:41Yeah.
- 32:41I mean, it's, it, so then the question is like, all right, I know I've got this unavoidable error.
- 32:46Right?
- 32:46And so the, the thing is like that, and that makes me sad, right?
- 32:50But I also wanna kind of figure out kind of like, how do I adjust my results from that?
- 32:53Right?
- 32:53Right.
- 32:53So if I got this shift that means, but like in a, in a poll that we do at Vanderbilt Poll, like I don't whether it's 49 or 50, like it doesn't matter to me, right?
- 33:01'cause like the conclusions are much more, it's almost like qualitative.
- 33:03Like I have quantitative data to make a qualitative conclusion, which is like, is there support bipartisan or not, it's like, you know, is there differences between the parties or not?
- 33:11Is there like more than a majority support or is it evenly split?
- 33:14And so like, they didn't even seem like 49% supporting Governor Lee versus, you know, 51%.
- 33:19I mean, the press sometimes make this up, but that's not a difference, right.
- 33:22That, that's like the same thing.
- 33:23That's about 50%, right?
- 33:25The public.
- 33:25Yeah.
- 33:25But like in a pre-election poll, 49 versus 51, well that's a big deal, right?
- 33:30Right.
- 33:30Proxy is no longer acceptable.
- 33:32Right.
- 33:32That's a exactly right.
- 33:33I need to kind of that precision where when the polling that I like doing, I don't need that precision.
- 33:38Right?
- 33:38So it's kind of like it gives me the estimate and so like, yeah, I could be off by a few percentage points because I don't have people.
- 33:43But like the type of insights that I wanna draw, like people are support this policy or people hate this policy.
- 33:48Right?
- 33:48Right.
- 33:48That's, it's much more qualitative.
- 33:50You know, I think the going in proposition for many people is that you're gonna sample and you're gonna survey to get the answer that you already know, or to get the answer to the question in the way that it benefits you, right?
- 34:02The sense ... mm-hmm.
- 34:03... that somehow, you know, surveys, um, yield something that, uh, we already kind of wanted to see.
- 34:09Yeah.
- 34:10But you have this great way of understanding the difference between a probability versus a non-probability sampling strategy, right?
- 34:17So that you get, uh, and that the randomness of that selection that you just described earlier, um, is another way to make sure that you get an answer that you didn't expect, right.
- 34:27That we sometimes want to know, actually, not sometimes, but a lot of the time we want to know evidence that we didn't already have in hand, right?
- 34:35That we wanna find the counterfactual, we wanna find evidence that we might be wrong.
- 34:40If you're a good person, as I, as I tell my, If you're a strenuous scientist.
- 34:45As I, as I tell my class, like, if, if I'm a good pollster and by a good pollster, I mean bad pollster, I can get any answer you want.
- 34:50And, and that, and I think that this is the grappling, right, and this is why the overall trust in science and trust in polling is so important.
- 34:57Because like, you know, I give examples in my class where I'm like, here are two cla two questions asking about the Green New Deal.
- 35:03Can you tell me which one's pro and which one's con?
- 35:05And they load up the question in different ways.
- 35:07And so like, it is the case that, you know, if you want me to give you a number that supports your side, and I'm a good pollster, which I am, I can get you that number, right?
- 35:16But that's like the illusion of learning, that's not pulling anymore, right?
- 35:19That's not being, uh, motivated by the truth that's trying to almost like use polling...
- 35:24As advertising.
- 35:24...as advertising.
- 35:25Yeah.
- 35:25Which unfortunately is now done, right?
- 35:27There's like push polls and they're, and they're, you know, there's even talking about whether or not pollsters themselves are kind of shaping, trying to release polls to shape the media landscape by showing one candidate versus another candidate.
- 35:37And so like, and that's like not good, right?
- 35:40For those of us who kind of use polling, uh, to try to learn something about democracy.
- 35:44But it's also probably not good writ large.
- 35:47Because like once we lose that ability to kind of, or that trust right?
- 35:52In the work that's being done, then like on what basis do we have to, to evaluate claims about the truth, right?
- 35:57And so like, if I can't trust anything, then it's like then everyone's opinion, then everyone's opinion becomes their their own truth.
- 36:04And then that's really hard to do anything to make any kind of agreement.
- 36:07Right.
- 36:07And so it's, so I think like the having humility about what you're doing and say, "Look, I could screw this thing up.
- 36:13I've got lots of errors." Like, I just think that's like important to be honest, right?
- 36:17Because like, science is so messy, right?
- 36:19And can we, you know, we know that throughout history, science makes mistakes sometimes.
- 36:23But like at the end of the day, like it's incumbent upon— and again, like this is you, you know, we try to do replication, all this stuff, you know, we try to do all this stuff as scientists or try to like, um, protect against that.
- 36:33But foundationally it really.
- 36:36There's, there's limits to what you can do, right?
- 36:37It's basically up to the goodness of the people involved, you know, to do the right thing, right?
- 36:42And so, And I hear you using the word good in two ways.
- 36:44Good as in super skilled, right?
- 36:46Like you're a good pollster or you're a good political scientist, meaning you have the chops, but then the other version of the good, which is like, I really am trying to get closer to an answer that is verifiable.
- 36:57Yeah.
- 36:57Transparent, credible, and replicable, right?
- 37:00Yeah.
- 37:00I mean, you both, you had to be like both skillful and moral in some ways, right?
- 37:04And like Really interesting.
- 37:05So, I don't know.
- 37:05Skillful, moral, and then the thing I wanted to ask you, Josh, is skillful, moral and presumably relatively shielded in either having or expressing your own views personally, right?
- 37:18Yeah.
- 37:18Like you're teaching a class with what, what is it?
- 37:20Vanderbilt's largest class ever.
- 37:23Yep.
- 37:23So you're teaching a class on the U.S. presidential election.
- 37:27Yeah.
- 37:27Surely at least half of the class wanted to know who you supported, right?
- 37:33How did you handle that?
- 37:35I just, I mean, I make it a point so that my goal is that no one knows what I think about, you know, 'cause it's not my goal to tell my students what to think.
- 37:44I do want to equip them with the skills so they know how to think, if that makes sense.
- 37:48But like, not how to think about like, I'm gonna support this team or that team, but kind of like, how do I make reasoned assessments of the data and kind of come to my own conclusions.
- 37:57I think that, I mean, that's the, that's the most critical thing.
- 38:00And 'cause I, you know, people come from different places, people have different experiences in terms of what they do.
- 38:05And college should not be about me telling you what to think, right.
- 38:09It's kind of, it should be kind of me equipping you with tools that you can navigate the world and you can understand the world and you can make the world as you understand it kind of, and, and come to your own kind of conclusions.
- 38:20And so, like, teaching the elections class with 1,100 people, like, that's a terrific challenge, right?
- 38:25Yeah.
- 38:25Yeah.
- 38:25Because like, you know, you want, and like, you, especially in our political climate, like I was frankly unclear how that would, you know,'cause we have like one out of every six Vanderbilt students in, and they're all networked on fizz or whatever crazy social media things they're in.
- 38:36And so like, if that class goes sideways, right?
- 38:39Right.
- 38:39The whole, you know, it's gonna have like large repercussions.
- 38:42It's like, I was kinda like, I was joking with my parents, like my definition of success for this semester if I don't end up on M-S-N-B-C or Fox News for things kind of going, you know, awry in class, which knock on wood, we haven't done that so far.
- 38:53So, I mean, but the, you know.
- 38:53No, it's just, it's so important I think for you to be able to share that.
- 38:59Because I think the kind of notion of being really, really smart and really questioning and really undeclared in your own views and yet hyperdeclared in your commitment to doing the work, to have a viewpoint and to do it in an empirically or, you know, intellectually rigorous way, that's a, that's a lot to try to communicate to students, right?
- 39:25Yeah.
- 39:25And, but it's also important, I mean,'cause it's like, you know, we've been having similar different debates for like hundreds and hundreds of years.
- 39:32Yeah.
- 39:32Right?
- 39:32And so, and like, and people, you know, and I think the most important thing is like, how do you get that skill so you can interact with one another?
- 39:40And so like, I still foundationally think like everyone kind of wants the same, right?
- 39:45Everyone wants the same thing, right?
- 39:47And there's disagreements about how best to achieve that good thing, right?
- 39:51And that's gonna, that, and that's an enduring problem in society.
- 39:54It's always been a problem in society.
- 39:55And so as a, as a person, as a professor, like my job is not to tell you what the good thing is, right?
- 40:01Or kind of what the necessarily the right policy is because you know, I'll leave that to the economists, but they still can't figure that out.
- 40:06But that's a whole other conversation.
- 40:08But like, but it's kind to equip you, right?
- 40:10With the capacity that you can understand some of the arguments that are being put in front of you.
- 40:15And especially in this data-driven world where both sides are mobilizing arguments and data to kind of, you need to have the capacity to kind of understand what that means or what you know, how to interpret that and kind of update and do I change my opinions?
- 40:28Like how much do I weigh this versus that?
- 40:30And I mean that's, you know, watching that journey that students take.
- 40:33I mean that's like, I. I mean, that's part of my, you know, I do all this public-facing and scholarship.
- 40:38But literally the most, the favorite parts of the job is like in the classroom.
- 40:42Like when I'm teaching something and I see just the light bulbs go on.
- 40:45Yeah.
- 40:45Right.
- 40:46Yeah.
- 40:46And like, you know, it's like, that's why like teaching data science and statistics, like every, oh, that's a horrible class.
- 40:50But no, it's a it's a beautiful class.
- 40:52Yeah.
- 40:52Because like, you're teaching someone how to think and how to process data.
- 40:56And when they see, when you see that, they finally understand or hopefully quicker.
- 40:59'cause I think I'm sometimes a good teacher, but, but when you see them, that understanding come across right, then that's like, that, that makes my day.
- 41:05That's a golden moment.
- 41:06Totally.
- 41:06It's a golden moment.
- 41:07'cause then you realize that you've no, I don't know, I don't wanna say changed your life.
- 41:09But, but you have kind of cha you've changed the way, the toolkit that they have for navigating the, this complex world.
- 41:15Yeah.
- 41:15And you've empowered them to kind of go off and kind of make sense of the world and really foundationally, you know, important work.
- 41:21And so, yeah, my papers are good and they're good, you know, and they're cited and blah, blah, blah, blah.
- 41:25But like, at the end of the day, like.
- 41:27You know, it's, it's, it's almost like academics is very humbling.
- 41:31Right?
- 41:31And again, 'cause like everyone I read as in grad school, and I'm sure the same is like no one reads 'em anymore, right?
- 41:36Yeah.
- 41:36I mean it's, you know, it's some exception, but like all, you know, we spend your entire life on trying to hit the top journals, then you realize that, but that's like in such an ephemeral thing.
- 41:44Completely.
- 41:45But the impact you have on a student, right?
- 41:47And so like, working my grad students and having them go off and kind of, you know, taking an assistant professor job at Yale this year, or University of Michigan or kind of seeing my undergrads get into these kind of amazing and seeing their lives literally change before your eyes.
- 41:59Like that is kind of, uh, that's like the most remarkable thing.
- 42:01And it's also, that's super amazing feeling that, you know, whether the paper's ever cited again, is unknown and un as you said, super ephemeral, right?
- 42:10Yeah.
- 42:10And you sort of, if it's missed, no problem, because it wasn't, it wasn't that amazing in the first place.
- 42:17It was great to see the paper come out, but the real work had happened years before, right?
- 42:22And that the real work is actually those graduate students and those undergrads who are now doing exponentially more cool work in the next iteration, and that they're training students who are doing exponentially more cool work in the next iteration, right?
- 42:36Yeah.
- 42:36It's in that way that you've built human capital that's starting to really ripple.
- 42:41Um, and I, I'm really curious about the way that you build human capital nationally with the public, right?
- 42:47You're, I love that phrase that you just said, that you are building a student's toolkit or that you're expanding their toolkit to be able to think differently or think, you know, more sort of rigorously about the things that they really care about but don't necessarily know how to navigate.
- 43:04Do you feel like the public is getting better or more sophisticated or more sharp in its thinking?
- 43:10Is that translational effort landing with a American voting public or the the voting and non-voting public.
- 43:19I think that I, I think that the, there's increasing opportunities for that to happen, is my kind of like nuanced answer to that, but, 'cause, 'cause, but it, but I think it's, it is happening in places, and I think that's really vital.
- 43:30And I think it's largely because of the way the, the media landscape has been disrupted.
- 43:34Now that has good and bad consequences, but like, I remember like some of the polling pieces I wrote this cycle trying to like you know, explain like why polling could get different answers or why the polls were all kind of giving the same answers in ways that were, eh, not super great.
- 43:48Or why everyone was obs, you know, so I had like a million, you know, this is like another piece, like I wrote a paper on early voting and why you should just like take a walk instead of reading it.
- 43:56And it got like a million views on NBC News, right?
- 43:58Because again, people want that certainty, and I was like, you, you're not gonna get that here.
- 44:01Right?
- 44:01Like we, the data's not gonna tell you what uh, it you want to tell, but like, but trying to like, have conversations with like the network news to kind of like have that conversation.
- 44:12They, they weren't really interested in it because like it's, right.
- 44:15It's, it's this great' Because they want the Well, yeah.
- 44:18'cause you want certainty.
- 44:18Right.
- 44:18And so, like I said, like I said, instead of releasing polls, like multiple numbers you have like, you know, you should, with one number, you should have multiple numbers because like that reflects our uncertainty there.
- 44:26But then that muddies it.
- 44:27Yeah.
- 44:28And that gives the impression that we can't really say anything, which is the reality.
- 44:31Right.
- 44:32You can't say anything about Pennsylvania.
- 44:33Right.
- 44:33Well, and maybe some other channel is selling a simpler Exactly.
- 44:35Simpler, simpler explanation.
- 44:36Simpler product.
- 44:37Right.
- 44:37Just go with one number.
- 44:38This is the number here.
- 44:40Exactly.
- 44:40But I, but I have to think right, that, that is shortsighted in the long run.
- 44:45Because that, and you see this across the board, right?
- 44:48People have no trust in American, in institutions writ large.
- 44:51Right?
- 44:52And I think it's because people are always like, that doesn't make any sense, or that's wrong.
- 44:55And I think, I don't know.
- 44:57I like to give people the credit benefit of the doubt.
- 44:58Like you can be honest in the sense that intellectually, that there's, you know, we think this is the case, right?
- 45:06But there's uncertainty about that.
- 45:08And instead of saying this is the truth, right?
- 45:11Well, you know, science is an ever right, as you know, it's an ever evolving achievement towards the truth.
- 45:16But you can never prove anything.
- 45:18You can only rule hypotheses out, right?
- 45:20And so it's like...
- 45:21it's nuanced.
- 45:22Well, and I love that because it's the flip far version of you saying about yourself that you really value your own and or you value humility and the importance of it for yourself.
- 45:33You're also clearly talking about valuing the public's intelligence, right?
- 45:37You're treating the public, you're treating those individuals who are saying, you know, "Oh, is this the number or is it a different number?
- 45:44Which, which single variable should I look at?" And you're saying, "I trust you to actually hold onto two numbers at once.
- 45:51Or to recognize that it might be a useful proxy, but it's not completely right." That, you know, the path forward to building trust in institutions is treating people as if they are, can be thoughtful, um, and can use the skills and the tools that you're giving them.
- 46:08Yeah.
- 46:08And you, you know, meeting them where they are.
- 46:10Right.
- 46:10And then kind of, you know, valuing and say their, they have a useful perspective and not everyone as fortunate as I to have kind of taken this path that I have taken.
- 46:18But that doesn't mean that those other people, but you know,'cause, but for a few chances, like I would've had a very different world.
- 46:24And so it's like you see yourself and you see, you know, and it's not, you know, it's not, that sounds elitist.
- 46:30But it's, it's, it's basically empowering like normal people.
- 46:34Like, it's not just academics who should be making decisions, like, but everyone has a same capacity.
- 46:38And I think if you gave everyone the opportunity, you know, they, they could all be, you know, successful.
- 46:42So they're kinda respecting, right, the intelligence of the, of the people and kind of treating them with respect and, you know, and sometimes there's disagreements and sometimes people see data differently and they have different interpretations and that's, you know.
- 46:54Is your lab pretty heated?
- 46:55Is your research team, um, is there lots of flaring up of, uh, very, uh, different strong opinions?
- 47:02Or do, do all of your students tend to use the same sort of, like, the best outcome here is if you don't know my political views, and you only know that I have really great stats chops.
- 47:15Uh, it's, it's in, it's interesting, right?
- 47:16'cause like there has been a shift, right?
- 47:19In terms of the propensity of people to put their political views out there.
- 47:22And so, like, I'm now recognizing myself as an old person, right?
- 47:25Which is kind of, which is crazy to me.
- 47:27'Cause I look around, I still feel like I'm in grad school and I have no fricking idea what I'm doing.
- 47:30But like, I look around the, the faculty the day, I'm like, oh my gosh.
- 47:32Like I'm, I'm an old person.
- 47:33Like I am like at, you know, I'm far away from the research frontier.
- 47:36But like, but I, this is something I navigate and like, I honestly don't know.
- 47:40Right.
- 47:41Okay.
- 47:41For the record, just for everyone listening, Josh Clinton is not, uh, in my view, uh, categorizable as an old person, but that said, keep going.
- 47:49But, but so like, I know as I talking to one of my students and like, you know, they were taking, um, political views, for example, right?
- 47:55On social media, and I was like.
- 47:58I'm just, you know, it's like, I'm not sure that's a great thing for you to do, but they say, well, I feel that this is important to me to do this.
- 48:05Yes.
- 48:05And it's like, okay, well I'm not telling you not to do that.
- 48:08Right.
- 48:08Here's my perspective.
- 48:09I respect your perspective.
- 48:11'cause people have difference of opinions.
- 48:12Like, I don't do that.
- 48:13But like, you know, you're an independent person, right?
- 48:16You can come to your own assessment of what's going on?
- 48:18And so like, so some people do that, right.
- 48:20And that, and I'm to, you know, totally like you be you, right?
- 48:23It's kind of, people have different impressions and that's, I'm not gonna say silence you if that's what, if you feel important to speak it.
- 48:28I don't think that's what I do.
- 48:30But like I totally respect people kind of who do, and I think it's navigating Yeah.
- 48:34You know, that it's, it's, it's complicated and it's, you know, and I don't know what the right an like, again, like, I don't know, I don't know what the right answer is.
- 48:40Right?
- 48:41And so it's, I let you know.
- 48:42Well it's just interesting 'cause you're in a role of authority where you actually have a lot of sway or impact because your opinion or your, the credibility of your assessment is gonna be put out there on national news, right?
- 48:57And so.
- 48:59In many ways, your neutrality is central to the effectiveness of your ability to do that.
- 49:05Whereas the student is building the capacity to express their views.
- 49:10Learning how to express those views and sharpening that capacity.
- 49:14And maybe the student needs to do that by expressing his or her or their views a lot.
- 49:18Right.
- 49:19And so, yeah.
- 49:19And, and yeah.
- 49:20It may be a difference of position and role.
- 49:23And it's, I'm also so grateful I didn't grow up in social media, right.
- 49:25Where I could make mistakes, right.
- 49:28And do things that I probably ought not have done right.
- 49:30But there's no record, you know?
- 49:32Right.
- 49:33With my son.
- 49:33Like I, you know, he was like doing, you know, not.
- 49:36It's like we had that conversation super early.
- 49:37Like whatever you do, it's permanent.
- 49:39Yeah.
- 49:39Right.
- 49:39And that's really bad because like, you're a kid, you're stupid, right?
- 49:44I mean, you make bad decisions sometimes, right?
- 49:46It's a function of the prefrontal cortex beings developed.
- 49:49It's, you know, it's not your fault, like, but like, but and, but like how do you get that humility, you know, the, you know, make a mistake, but then I can improve or kind of recognize that that was a mistake.
- 49:58And I, you know, so it's, you know, it's complicated.
- 50:01But you know, this is, you know, but this is, this point is something that I navigate 'cause it's like, you know, living in Tennessee, like it is a Republican state, right?
- 50:08And I do polling and I'm from Vanderbilt University and I'm academic.
- 50:11And so people say, well, you know, you're a liberal, you know, democrat.
- 50:15And so, and, and that, and that's disarming from the fact because, you know, regardless of whether that's true or not, right?
- 50:21That what I say then is, is taken as like, well, because you're this Yes.
- 50:26I can't trust what you say about, you know, some, whatever you're saying.
- 50:30And, and particularly poignant for political science, right?
- 50:33'Cause people think political science is politics and they're, it's not quite the same.
- 50:37Yeah.
- 50:37Although sometimes they're.
- 50:39You know, in some parts of the discipline, there is a, a little bit of that.
- 50:41But like, but to the extent that I, I, you can't communicate like across the aisle and communicate truths about Right?
- 50:48Then that's like not great because like if you're only speaking to people who you think agree with you or they think they agree with you, then it's like then you're not really having that full conversation.
- 50:58You're not really having that full impact or kind of Totally.
- 51:01I mean right.
- 51:02You don't, you don't get access to the data for one thing.
- 51:05Yeah.
- 51:05Right.
- 51:05For it to, to the perspective.
- 51:07But also being misattributed in terms of your own, how, how you're viewed may mean, uh, you know, that they, they miss out on the opportunity also, right?
- 51:16Yeah.
- 51:16They've, they've just foreshortened the amount of information they can learn from you because they've made some assumptions about who you are or what you stand for.
- 51:24Yeah.
- 51:24It's really fascinating, the idea that it just limits the amount of information exchange.
- 51:29Yeah.
- 51:29And now, you know, I mean, it makes you, you know, limits your impact.
- 51:35Not that I you know, care that much about like my impact, you know, shaping the world or whatever.
- 51:39But like, but also like the, the interactions that you have and your ability to kind of learn about the world, right?
- 51:44And so, like, I think you just have to, you know, keep an open mind with respect to kind of perspectives and experiences and kind of, you know, engage people again where they are and try to not, you know, let those, you know, especially in the world that we live in.
- 51:57You know, people put on blinders and they make presuppositions about people and then they don't update.
- 52:02Yeah, totally.
- 52:03And I think, I think that's a horrible thing, right?
- 52:04'cause the whole thing about, it's like you need to be willing to kind of engage, change your mind, have conversations.
- 52:09And if we don't, if we're not doing that, then I'm not sure what we're doing.
- 52:12Right?
- 52:12'Cause like, the whole, whole human story is about encountering people and kind of over, you know, interacting with one another and kind of building society and building, building cultures together.
- 52:22Right?
- 52:22And that requires, you know, that kind of interpersonal contact and not just saying, "well I'm not gonna talk to you...
- 52:26Yeah.
- 52:26...'cause like you're wearing a different color hat." Yeah, I mean, I love that idea of continuous updating.
- 52:31It's totally works.
- 52:32And I guess that's where it leads me for the last question I'd love to ask you, Josh, which is, where do you think polling and surveying is going?
- 52:40If it's only 2%, those that you actually speak with, represent such a tiny fraction of those you intend to represent with the numbers that you wanna generate or, or yield.
- 52:51Is there some AI/technologically, you know, enabled future that allows us some cool way of, you know, other than liking or hearting, uh, a livestream on, uh, Instagram.
- 53:07Like what's the thing that will eventually solve this problem?
- 53:12I mean, that's a super great question, and if I knew the answer, I'd probably be building my own company rather than talking with you.
- 53:17But no, but no, it's, but it's like, it's, so I think there's two things.
- 53:21One is that, yeah, it's not great, right?
- 53:23But even said, it's kind of remarkable if you think about it, that I can talk to 800 people and get within like four points of the margin on an election with hundreds of millions of votes.
- 53:33Yeah.
- 53:34Right?
- 53:34And so like, that's kind of like mind-blowing, like even given all the problems we talked about.
- 53:39Right?
- 53:40Right.
- 53:40And so it gets, so that's like a statement about readjusting expectations about what we can plausibly do given the coarseness of the signal and instruments that are available to us.
- 53:48Yes.
- 53:49Right.
- 53:49So I think that, I love that phrase, the coarseness of the signal.
- 53:51Yeah.
- 53:52So I think that we have to do that to some extent, but then, but then technology changes too.
- 53:56And so, like my colleagues and I at Vanderbilt, like, you know, we looked at this like out, you know, artificial intelligence because people, some people I did, well, I don't need to talk to people anymore.
- 54:04I'll just talk to AI, right?
- 54:05And kind of will simulate respondents and see what they think and how well you do.
- 54:09And some people are doing this for trying to predict the election and you know, that can do something.
- 54:13But like, but again, you have to realize what you're doing, right?
- 54:16And so people don't really fully understand AI and LLMs.
- 54:19Because like what that's just doing is like I'm ingesting all the internet content and trying to find associations there.
- 54:24And so again, is the people on Reddit, right, or on YouTube comments, are they really reflective of American public?
- 54:32Right?
- 54:32Probably not.
- 54:33Right?
- 54:33So there's known, there's biases there.
- 54:35And so how do you adjust that?
- 54:36Yeah.
- 54:36And so you can get close, right?
- 54:38If you do AI and you do all this other stuff that people are trying to do.
- 54:41But you know, for the fine-grained precision, it's, it's kind of complicated.
- 54:45The other thing is it kind of bakes in the past, right?
- 54:48And so like if you're trying to predict something, using an internet data, given how quickly the world changes, the only way I can get an updated, you know, forecast if people change their minds is turns out by talking to them, right?
- 54:58Right.
- 54:58I can't sample the internet from the past, Right.
- 55:00And say, well based on this, I suspect this will happen in the future.
- 55:03And so I think there are some promising things that are being done to try to, in some sense also make it more engaging.
- 55:08'Cause like, you know, it's really hard to reach people.
- 55:10So how we shorten that experience and kind of make it more interactive, although I'm not sure AI chatbots solve that.
- 55:15'Cause having, you know, communicated with them on financial websites, they're a disaster or a health care site.
- 55:20That's Totally.
- 55:21Or if I received one more text that told me to respond to the survey, right.
- 55:25There's absolutely no way.
- 55:26Yeah.
- 55:26So I, so I think that's like an ongoing project to kind of, you know, figure that out.
- 55:30And I think some of it will be technological and methodological, but then also I think, I hope, right?
- 55:35Some of that would just be better used to kind of understand like, what can we learn from the data, right?
- 55:40But that's something that's true all, you know, all over, like all these claims that AI can revolutionize the world.
- 55:45Like maybe, but like all these companies rushing in to do these decisions without fully understanding, like the the biases, right?
- 55:52What's behind the curtain?
- 55:53What's behind the curtains?
- 55:54And kind of how does that affect the conclusions that are being made, I think are so foundationally important, right?
- 55:58And like, pulling is like a, an easier version to understand than where we are going in kind of in the future.
- 56:04And so I think that just shows that the increasing that literacy among the public at large about statistics and kind of data.
- 56:10Right?
- 56:11It's only a butter knife, but a butter knife is something you can actually hold in your hand and you know what it looks like and you know how it works.
- 56:16Right?
- 56:16And it can, and it can still do damage.
- 56:17And it can do damage if wielded the wrong way.
- 56:19Right?
- 56:19Right.
- 56:19So a surgeon can use it, but like if, you know, if a bad person has it, like they can also, so like understanding the tools and kind of the limitations of these tools I think is like, is something real important.
- 56:28That's, you know, and that's like every single day teaching these classes at Vanderbilt and seeing and empowering the students for the future, like grad students or undergrads, like, you know.
- 56:35That's, I mean, I get so much joy or that's so much fun than, more fun than respond to review or two about some paper that, you know, will be impactful in my field.
- 56:44But, you know, may not go much, much, much beyond that.
- 56:46And so it's, it's, it always keeps you busy 'cause it's, the world's always changing and there's always something new to kind of talk about and hopefully explain and complexify in appropriate ways that we have an appreciate, you know, appreciative understanding of kind exactly how certain are we about the things we're talking about.
- 57:01'Cause science is not only about numbers, but it's also quantifying our uncertainty.
- 57:05And I don't think we do that enough to really get people the sense.
- 57:08I like that so much as a landing spot, right?
- 57:13That it's not just about estimating a number, it's about quantifying our uncertainty that, you know, A—you're admitting to uncertainty, which I think we as scientists and scholars sometimes find hard to do.
- 57:25Sometimes.
- 57:26I'm not perfect.
- 57:26Well, we're, we're, we're rewarded for being.
- 57:29Yep.
- 57:29Um, both critical and confident, right?
- 57:32Yep.
- 57:32And so the idea that we would be rewarded for or reward each other and be recognized for quantifying un our uncertainty alongside quantifying what we are certain about.
- 57:44Um, and also creatively interested about in, for the future, um, that's a really hopeful and really sanguine, uh, place to land.
- 57:54Any last words, Josh?
- 57:55Any last thoughts in terms of, uh your advice to the United States as we, uh, think about democracy in 2028?
- 58:03Uh, I just say, you know, it's a, it's a process, right?
- 58:05I mean, democracy has always been a process.
- 58:07One of the books I've, I've recently read, um, I've actually, that's a false, I bought it with the intention of reading.
- 58:13It's on my bookshelf hasn't read it yet, but it was, it's about, I think it's like the sunshine, the setting sun or something like that by a historian at Syracuse University.
- 58:19And it's an interesting book that goes through and talks to all the people who wrote the founding fathers, who wrote the Constitution and what they were thinking at the end of their time.
- 58:26Hmm.
- 58:27And like to a person, they all thought the United States was a failure.
- 58:30Wow.
- 58:31Right.
- 58:31Because like all the things, you know, factions and parties, like everything, they didn't want.
- 58:35Manifested, right?
- 58:36Yeah.
- 58:36And it looked, when they were dying, it said, there's no way this country, you know, we are a complete failure.
- 58:40We spent our entire life building this country.
- 58:42And it's coming apart.
- 58:43Just seems like, you know, it's, it's gonna be horrible.
- 58:45And I think, well, geez, like if Madison and Washington and Jefferson and, you know, and you know.
- 58:51They all were kind of like really despondent about the United States.
- 58:54Yeah.
- 58:54Things looked bleak then.
- 58:55And yet, right.
- 58:56And yet here we are.
- 58:58Right.
- 58:58And like, it's not saying things are perfect, right?
- 59:00There's not, there's like problems that need to be solved and whatever.
- 59:03But like if the people design the system at the very outset, and it's a very different world then to be clear, right?
- 59:08But if they were all despondent about it, right?
- 59:10Then, like I'm saying that normalize, its despondency.
- 59:12But like, but like, but it kinda normalize, like this is just like, it's like government is hard, right?
- 59:18Yeah.
- 59:18Politics is hard and like, And what we're doing is really difficult and really exciting.
- 59:22So you have to like, engage with that, with the, with the knowing that, you know, it's with also the humility that it is hard, right?
- 59:28You're gonna fail, you're gonna take steps, you know, that, you know, don't go in the direction that you want or they go in the direction you want for, for a short period of time.
- 59:35And.
- 59:35You know, it's just like, that's what kind of makes a quasi-interesting to see how we bring together these different perspectives to try to make, you know, from many, one and the kind of that's, you know.
- 59:44Yeah.
- 59:45That's an ongoing project and it's always gonna be an ongoing project.
- 59:47There's never gonna be a complete solution.
- 59:50Thank you to our listeners for tuning in.
- 59:52Remember to stay curious and keep exploring new ideas.
- 59:55We'll be back next time with another guest who's pushing the boundaries of discovery.
- 1:00:00Until then, I'm Cybele Raver and this is 'Quantum Potential.' The 'Quantum Potential' podcast comes from Vanderbilt University.
- 1:00:17Your host is C. Cybele Raver.
- 1:00:20Thank you to our guest, Josh Clinton.
- 1:00:23Production by Matriarch Digital Media.
- 1:00:27For more information about Quantum Potential, go to Vanderbilt.edu/QuantumPotential.
- 1:00:40Stay curious.