Latest / Decision Desk HQ Podcast / Episode 6: The State of Polling in 2025, with Elliott Morris & Mary Radcliffe from FiftyPlusOne

Transcript

Episode 6: The State of Polling in 2025, with Elliott Morris & Mary Radcliffe from FiftyPlusOne · Decision Desk HQ Podcast

• 0:00
  SPEAKER_01: We live in a democracy, so what the people think matters.
• 0:02
  SPEAKER_01: That's pretty much it.
• 0:04
  SPEAKER_00: Polls are flawed, we are aware.
• 0:06
  SPEAKER_00: But if you don't have polls, what do you have?
• 0:08
  SPEAKER_00: You're just guessing.
• 0:09
  SPEAKER_01: You're trying to figure out what the electorate is going to look like,
• 0:11
  SPEAKER_01: so I know that was definitely one of the challenges here.
• 0:14
  SPEAKER_01: The fundamental problem for polling,
• 0:16
  SPEAKER_01: the thing that we're trying to solve,
• 0:18
  is that the people who don't pick up the phone can be different than the people
• 0:21
  who
• 0:21
  SPEAKER_01: do.
• 0:21
  We're trying to tell you the set of possible outcomes and how likely those
• 0:24
  possible
• 0:25
  SPEAKER_00: outcomes are,
• 0:26
  SPEAKER_00: not the thing that will happen.
• 0:28
  SPEAKER_00: If we knew the thing that would happen, why would we run the election?
• 0:43
  SPEAKER_02: Welcome to the Decision Desk HQ podcast.
• 0:45
  I'm your host,
• 0:46
  Geoffrey Skelley,
• 0:47
  Chief Elections Analyst at Decision Desk HQ.
• 0:51
  SPEAKER_02: Election Day 2025 is just now entering the rear view mirror.
• 0:55
  It looks like the federal government shutdown may also soon be entering it as
• 0:59
  it's
• 0:59
  SPEAKER_02: expected to end on Wednesday,
• 1:01
  SPEAKER_02: which is the day we're recording here.
• 1:04
  Soit'stimetolookaheadtothenewyearandallthepoliticalandelectoral
• 1:08
  SPEAKER_02: events that are coming up in the 2026 midterm cycle.
• 1:12
  SPEAKER_02: And as always,
• 1:13
  when we're thinking about future elections and sort of the state of affairs
• 1:17
  and
• 1:17
  SPEAKER_02: political life in the United States,
• 1:19
  public opinion surveys are an important tool for trying to understand how
• 1:23
  Americans
• 1:24
  feel and how things are going and what people are planning to do when they
• 1:27
  vote.
• 1:28
  SPEAKER_02: in key primaries and general elections.
• 1:31
  And Decision Desk HQ has polling averages for key questions like
• 1:34
  presidential
• 1:35
  SPEAKER_02: approval and generic ballot.
• 1:38
  SPEAKER_02: So who's leading in the House race for 2026.
• 1:42
  SPEAKER_02: But we're far from alone.
• 1:43
  And there are many other aggregators and collectors of polling data out
• 1:46
  there.
• 1:46
  SPEAKER_02: And the newest entrant is FiftyPlusOne,
• 1:50
  SPEAKER_02: a collaboration involving some of my former colleagues at 538.
• 1:54
  SPEAKER_02: So I'm very excited to welcome two of them
• 1:58
  SPEAKER_02: to talk about FiftyPlusOne's work,
• 2:00
  SPEAKER_02: the importance of polls,
• 2:01
  SPEAKER_02: the challenges involved in aggregating them into averages,
• 2:05
  andtochatalittlebitaboutwhatjusthappenedwithpollinghereinthe2025
• 2:09
  election and what pollsters need to watch out for as we look ahead to 2026.
• 2:14
  SPEAKER_02: So with me,
• 2:15
  SPEAKER_02: I have Mary Radcliffe,
• 2:16
  who's the head of research at FiftyPlusOne and a true data maven when it
• 2:20
  comesto
• 2:21
  SPEAKER_02: tracking down polls and cataloging them.
• 2:23
  So analysts,
• 2:24
  journalists,
• 2:25
  and researchers can use that data now and in the future.
• 2:28
  SPEAKER_02: And also joining me is G.
• 2:30
  SPEAKER_02: Elliott Morris,
• 2:31
  whorunstheStrengthinNumbersSubstackandauthoredabookofthesamename
• 2:35
  SPEAKER_02: about how polls work and their importance in a democracy.
• 2:38
  SPEAKER_02: Elliott is the head of analytics at FiftyPlusOne,
• 2:43
  SPEAKER_02: and his statistical models power the site's averages and forecasts.
• 2:47
  SPEAKER_02: So Elliott and Mary, welcome to the Decision Desk HQ podcast.
• 2:52
  SPEAKER_02: Great to be here.
• 2:52
  SPEAKER_02: Thanks for having us, Geoffrey.
• 2:54
  SPEAKER_01: Thanks, Geoffrey.
• 2:55
  SPEAKER_02: Yeah, well, very exciting to have two of my old 538 colleagues on.
• 3:03
  SPEAKER_02: I pitched this to you guys to come on and you were like, yeah, sure.
• 3:07
  SPEAKER_02: We've done this a few times, just not with me hosting.
• 3:12
  SPEAKER_02: So anyway,
• 3:12
  SPEAKER_02: hey,
• 3:13
  SPEAKER_02: we're going to go over a few things here,
• 3:14
  but I thought a good place to start would be to talk a little bit about
• 3:18
  FiftyPlusOne
• 3:20
  so people understand what you guys are doing and just talk a little bit about
• 3:25
  the
• 3:26
  SPEAKER_02: mission and sort of
• 3:28
  SPEAKER_01: what you're doing with this new site?
• 3:32
  SPEAKER_01: Well,
• 3:33
  SPEAKER_01: I think the people who follow polls,
• 3:35
  SPEAKER_01: maybe it'll be kind of obvious,
• 3:37
  SPEAKER_01: but the service that 538 provided when it existed,
• 3:43
  SPEAKER_01: I think is pretty unmatched in the data ecosystem.
• 3:47
  SPEAKER_01: There are other sites,
• 3:48
  SPEAKER_01: of course,
• 3:48
  SPEAKER_01: you mentioned Decision Desk,
• 3:50
  SPEAKER_01: the New York Times exists as well that collect polling data,
• 3:53
  SPEAKER_01: but I think we do a better job in a couple of
• 3:57
  SPEAKER_01: areas first were faster.
• 3:59
  SPEAKER_01: I mean,
• 4:00
  thespeedwithwhichMarycantrackdownandenternewpollsintoadatabaseis
• 4:05
  SPEAKER_01: really mind boggling.
• 4:08
  538 also is just incredible in its scale of the number of questions that it
• 4:13
  captured.
• 4:14
  So we wanted to replicate both of those things in continuing the original
• 4:18
  idea
• 4:18
  SPEAKER_01: behind the polling aggregation that 538 served.
• 4:22
  SPEAKER_01: And to do that, you need a couple of things.
• 4:23
  SPEAKER_01: First, you need someone to gather the data.
• 4:25
  SPEAKER_01: That's what Mary does.
• 4:25
  SPEAKER_01: You need someone to analyze the data.
• 4:28
  SPEAKER_01: That's what I do.
• 4:28
  SPEAKER_01: And you need someone to make a nice looking website and great charts.
• 4:32
  SPEAKER_01: And that's what our other co-founder, Katie Marriner does.
• 4:36
  SPEAKER_01: Another great 538 alum.
• 4:38
  SPEAKER_01: Yes.
• 4:39
  SPEAKER_00: Yeah,
• 4:39
  SPEAKER_00: I mean,
• 4:39
  SPEAKER_00: I think there's a lot of things that happened inside of 538 that were
• 4:44
  SPEAKER_00: really behind the scenes that people didn't really know about.
• 4:47
  SPEAKER_00: I'll give you one little example from just this fall.
• 4:50
  So a new pollster popped up with a poll in the Virginia governor's race this
• 4:54
  fall.
• 4:55
  SPEAKER_00: Now,
• 4:55
  SPEAKER_00: when new pollsters pop up,
• 4:57
  SPEAKER_00: what 538 used to do is reach out to them,
• 4:59
  SPEAKER_00: talk to them,
• 5:00
  SPEAKER_00: ask questions about their methodology,
• 5:01
  SPEAKER_00: make sure that they met our admittedly very loose standards.
• 5:06
  SPEAKER_00: Now,
• 5:06
  SPEAKER_00: I saw this poll come out because it was aggregated by the New York Times,
• 5:11
  SPEAKER_00: Virginia,
• 5:12
  SPEAKER_00: Governor Tracker,
• 5:13
  SPEAKER_00: and other aggregators.
• 5:14
  SPEAKER_00: And so I reached out to the person who was running the company.
• 5:19
  SPEAKER_00: And I mean, honestly, when I first saw this, I was really skeptical.
• 5:23
  SPEAKER_00: The website was just like a brand new Google site with no name on it,
• 5:26
  SPEAKER_00: no information about who this person was.
• 5:28
  SPEAKER_00: I reached out to them.
• 5:29
  SPEAKER_00: I talked to them.
• 5:30
  SPEAKER_00: They don't meet our admittedly very loose standards.
• 5:33
  SPEAKER_00: So having someone who's out there doing that kind of work,
• 5:37
  SPEAKER_00: really checking,
• 5:38
  SPEAKER_00: like,
• 5:38
  SPEAKER_00: hey,
• 5:39
  SPEAKER_00: is this quality research?
• 5:40
  SPEAKER_00: Is this worth including?
• 5:42
  SPEAKER_00: That was a huge thing that 538 did that served the entire industry.
• 5:47
  SPEAKER_00: We're still doing that at FiftyPlusOne.
• 5:48
  SPEAKER_00: That pollster was aggregated by lots of other people, but not by us.
• 5:52
  SPEAKER_00: So I think that's a really valuable service.
• 5:55
  SPEAKER_00: And frankly,
• 5:56
  SPEAKER_00: for those who follow this sort of industry,
• 5:58
  SPEAKER_00: the polling aggregation,
• 6:00
  SPEAKER_00: I think it's no secret that 538 was the best.
• 6:03
  SPEAKER_00: It always was.
• 6:04
  SPEAKER_00: Sorry.
• 6:07
  SPEAKER_02: It set a standard that others want to live up to now.
• 6:11
  SPEAKER_00: That others aspire to.
• 6:12
  SPEAKER_00: And we are taking that standard forward under a new banner.
• 6:16
  Importantly,
• 6:17
  independent,
• 6:19
  not part of any larger media organization or conglomerate.
• 6:23
  SPEAKER_00: Our goal is really just to focus on the data, right?
• 6:27
  SPEAKER_00: We're not trying to...
• 6:30
  SPEAKER_00: like support a media empire or anything like that, right?
• 6:34
  SPEAKER_00: This is,
• 6:34
  SPEAKER_00: it's all about the data and making sure that we have quality data,
• 6:38
  SPEAKER_00: we have quality aggregation and statistics and moving forward,
• 6:44
  SPEAKER_00: we'll have editorial and journalism based on that independent data.
• 6:49
  SPEAKER_02: No, it's great.
• 6:51
  SPEAKER_02: Obviously,
• 6:52
  SPEAKER_02: as somebody who worked at 538 for a very long time,
• 6:55
  I think having people trying to live up to the standard that it's set is
• 7:01
  exactly
• 7:01
  SPEAKER_02: what we want in this business.
• 7:03
  SPEAKER_02: And I also think it's just generally good to have multiple aggregators,
• 7:09
  SPEAKER_02: even if there are differences in methodology and approach.
• 7:13
  SPEAKER_02: At the end of the day,
• 7:15
  SPEAKER_02: Having more than one person,
• 7:16
  SPEAKER_02: it's sort of actually,
• 7:17
  SPEAKER_02: I mean,
• 7:17
  SPEAKER_02: it's not quite the same,
• 7:18
  SPEAKER_02: but it's like having just one vote count is maybe a little not great.
• 7:22
  SPEAKER_02: So having multiple aggregators,
• 7:23
  at least you can lose room for discussion and debate about methods and how
• 7:28
  bestto
• 7:28
  SPEAKER_02: go about doing this sort of thing.
• 7:30
  SPEAKER_02: So I love that you guys have done this.
• 7:32
  SPEAKER_02: I was very excited.
• 7:33
  SPEAKER_02: I'm actually going to bring up the...
• 7:37
  site so this is FiftyPlusOne I've zoomed in a little bit so it looks better
• 7:42
  forpeople
• 7:43
  at home watching yeah we like to do lots of charts and stuff on this but I
• 7:48
  thought
• 7:48
  at the very least I could show people this obviously you guys have
• 7:51
  presidential
• 7:52
  SPEAKER_02: approval which is such a mainstay of any sort of
• 7:57
  SPEAKER_02: pulling aggregation for understanding,
• 8:00
  SPEAKER_02: obviously,
• 8:01
  like what people think about the person who is the most conspicuous
• 8:04
  individualin
• 8:06
  SPEAKER_02: government.
• 8:07
  SPEAKER_02: So,
• 8:08
  SPEAKER_02: you know,
• 8:09
  SPEAKER_02: sort of looking forward,
• 8:10
  SPEAKER_02: I know that you guys,
• 8:12
  SPEAKER_02: you wanted to have,
• 8:13
  SPEAKER_02: I think you wanted to be more accessible.
• 8:15
  SPEAKER_02: You don't want it to be dependent on sort of a media organization.
• 8:19
  So I was kind of curious what that means for sort of access for people in terms
• 8:24
  of
• 8:24
  SPEAKER_02: like your data and,
• 8:25
  SPEAKER_02: you know,
• 8:26
  SPEAKER_02: how people can get access to that.
• 8:29
  Yeah,
• 8:29
  I'm going to pull the curtain back a little behind the scenes with what
• 8:32
  happened with 538.
• 8:34
  SPEAKER_01: Basically,
• 8:35
  SPEAKER_01: this service we're providing people here,
• 8:38
  SPEAKER_01: polling average and collection of information,
• 8:40
  SPEAKER_01: it's monetizable,
• 8:43
  SPEAKER_01: but only if you actually make the effort to monetize it.
• 8:47
  And you need to monetize it to pay server costs and people like Mary employing
• 8:51
  the
• 8:51
  SPEAKER_01: data eventually,
• 8:52
  SPEAKER_01: hopefully,
• 8:52
  SPEAKER_01: maybe.
• 8:54
  And that means that something basically has to be locked behind a paywall.
• 9:00
  And we wanted the main takeaway of a public opinion in America to be publicly
• 9:06
  available.
• 9:07
  SPEAKER_01: That's why you can go to this website and look at our averages,
• 9:11
  SPEAKER_01: which have taken a lot of research to do.
• 9:15
  SPEAKER_01: calculate the average itself.
• 9:18
  And then there are some things that we're going to lock behind the paywall so
• 9:20
  that
• 9:20
  SPEAKER_01: we can support the sort of public interest admission of the company.
• 9:24
  SPEAKER_01: So that's stuff like cross tabs,
• 9:28
  SPEAKER_01: any downloadable CSV of the actual polling data itself,
• 9:32
  and all that stuff's available to people who want to pay an API subscription
• 9:36
  forus
• 9:36
  SPEAKER_01: so they can download it to their computer.
• 9:39
  SPEAKER_01: Or,
• 9:40
  SPEAKER_01: and this is something we're adding soon,
• 9:42
  a visual tabular data as well on all of these pages for various approval
• 9:48
  averages.
• 9:49
  SPEAKER_01: But the big thing is we do think that there's value in this indicator.
• 9:54
  SPEAKER_01: People are looking right here on their screen, like disapprove plus 16.
• 9:58
  SPEAKER_01: So that's something that we wanted to keep public if at all possible.
• 10:04
  SPEAKER_01: I don't know, Mary might have something to add.
• 10:06
  SPEAKER_00: Yeah,
• 10:06
  SPEAKER_00: I mean,
• 10:07
  SPEAKER_00: I think what's really exciting about this project,
• 10:11
  SPEAKER_00: starting out as a small startup,
• 10:13
  isthatwehavealotofflexibilitytodoalotofthingsthathaven'tbeendone
• 10:19
  SPEAKER_00: before.
• 10:19
  SPEAKER_00: So for example,
• 10:20
  SPEAKER_00: for API subscribers,
• 10:22
  you can download complete crosstabs for presidential approval for generic
• 10:26
  ballot.
• 10:27
  SPEAKER_00: Hopefully, will be some more crosstab collections available soon.
• 10:31
  SPEAKER_00: There's a lot more we can do that
• 10:35
  SPEAKER_00: You know, there may be certain journalistic reasons.
• 10:37
  You don't want to be like super public with like a big crosstabs polling
• 10:40
  average
• 10:41
  SPEAKER_00: because it's really,
• 10:41
  SPEAKER_00: really noisy.
• 10:43
  SPEAKER_00: It's hard to interpret.
• 10:44
  SPEAKER_00: It's difficult to explain to the public what that really means.
• 10:48
  Butforresearcherswhowanttobeabletodownloaditanduseitinmodelsor
• 10:52
  SPEAKER_00: whatever it is that they are doing or other...
• 10:57
  SPEAKER_00: media entities that wanna use that responsibly,
• 11:01
  wecanmakethatstuffavailableandreallyhaveamoreexpansivesetofdata
• 11:05
  SPEAKER_00: available for people for whom it is particularly useful.
• 11:09
  SPEAKER_00: With respect to like the tabular stuff,
• 11:11
  SPEAKER_00: as Elliot mentioned,
• 11:13
  SPEAKER_00: we're hoping by end of year,
• 11:14
  SPEAKER_00: there will be like a really low level subscription.
• 11:16
  SPEAKER_00: If you love this data,
• 11:17
  SPEAKER_00: you wanna play with this data,
• 11:18
  SPEAKER_00: you wanna sort and filter the tables and look at all kinds of stuff,
• 11:21
  SPEAKER_00: there will be like a way for you to support our work.
• 11:25
  for like a relatively small amount of money so that you can continue to
• 11:30
  understand
• 11:32
  SPEAKER_00: the public better,
• 11:33
  SPEAKER_00: which is the goal of polling,
• 11:35
  SPEAKER_00: and make sure that we can do this in perpetuity.
• 11:39
  SPEAKER_02: Yeah,
• 11:40
  SPEAKER_02: just to switch up the visual here for folks,
• 11:43
  SPEAKER_02: I'm going to bring up the generic ballot,
• 11:45
  which is the question that basically asks folks who they plan to vote for in
• 11:50
  the
• 11:50
  SPEAKER_02: next House election for the US House of Representatives.
• 11:54
  SPEAKER_02: So looking ahead to 2026 midterms.
• 11:57
  SPEAKER_02: And so, and actually sort of, you know, just to turn to a...
• 12:01
  SPEAKER_02: the kind of the larger subject here,
• 12:03
  SPEAKER_02: obviously,
• 12:03
  SPEAKER_02: if we're sitting here talking about polling,
• 12:04
  SPEAKER_02: we're talking about aggregating,
• 12:06
  SPEAKER_02: we're talking about,
• 12:08
  SPEAKER_02: you know,
• 12:09
  SPEAKER_02: trying to derive meaning from numbers,
• 12:12
  SPEAKER_02: asking people why they think,
• 12:13
  SPEAKER_02: you know,
• 12:13
  SPEAKER_02: what do they think about certain topics or certain elections?
• 12:18
  I guess I wanted to sort of talk with you guys about why you think polling is so
• 12:23
  important.
• 12:24
  SPEAKER_02: Elliot, I know you've written a book where you go into this.
• 12:28
  SPEAKER_01: Oh boy, how much time do we have?
• 12:30
  SPEAKER_01: No, I know, I know.
• 12:31
  SPEAKER_02: Well, we're trying to keep it to around a half hour, I know.
• 12:33
  SPEAKER_02: So don't do the whole book.
• 12:36
  SPEAKER_02: But obviously,
• 12:38
  SPEAKER_02: I agree with you guys,
• 12:39
  SPEAKER_02: but I wanted to hear it from,
• 12:41
  SPEAKER_02: I figure the listeners,
• 12:43
  SPEAKER_02: viewers,
• 12:43
  SPEAKER_02: they would all benefit from hearing it from you.
• 12:48
  soiyouknowthesimplestexplanationthoughonesuitableforatimelimited
• 12:53
  podcastjustboilsdowntothefollowingumweliveinademocracysowhatthe
• 12:58
  people think matters that's that's pretty much it our system of government
• 13:02
  is
• 13:02
  predicatedontheideathatifpeoplewantsomethingonaveragethatthingis
• 13:07
  supposed to happen um and also that there's you know safeguards in place uh
• 13:11
  like
• 13:11
  SPEAKER_01: for civil rights and etc but
• 13:13
  SPEAKER_01: The numerical majority,
• 13:14
  SPEAKER_01: James Madison wrote,
• 13:16
  SPEAKER_01: is the principle on which all democracies flourish.
• 13:19
  SPEAKER_01: So if that's the case,
• 13:20
  there are lots of questions about politics in our lives that aren't sort of
• 13:25
  aren't
• 13:25
  SPEAKER_01: consulted in the electoral process.
• 13:28
  SPEAKER_01: And there are issues that arise between elections to questions of war,
• 13:32
  SPEAKER_01: of famine,
• 13:34
  of just day to day politics in America that politicians don't talk about in
• 13:40
  their
• 13:40
  SPEAKER_01: campaigns.
• 13:42
  So that public opinion isn't represented in government unless you have
• 13:45
  anotherway
• 13:46
  SPEAKER_01: to measure it,
• 13:47
  SPEAKER_01: number one,
• 13:48
  SPEAKER_01: and unless people are paying attention to it,
• 13:50
  SPEAKER_01: number two.
• 13:51
  So I guess my academic work is mostly focused on the first thing is the
• 13:56
  measurement of it.
• 13:57
  But what I like about projects like FiftyPlusOne is we're giving all of that
• 14:01
  data information.
• 14:03
  tothepublicandtryingtotryingtoelevateitsothatwecanfulfillthat
• 14:07
  mission that's my sort of stump speech that's my really idealistic uh
• 14:11
  idealistic
• 14:12
  framing obviously this falls short sometimes and we'll talk about some
• 14:15
  caseswhen
• 14:15
  polls fall short but that's my take mary i think it also has a pretty good take
• 14:19
  on
• 14:19
  SPEAKER_00: this I mean here's the thing
• 14:23
  SPEAKER_00: polls are flawed, we are aware.
• 14:25
  SPEAKER_00: There's a margin of error.
• 14:26
  SPEAKER_00: You can see it on the screen right now, right?
• 14:28
  SPEAKER_00: There's all kinds of things that can go wrong.
• 14:32
  SPEAKER_00: But if you don't have polls, what do you have?
• 14:36
  SPEAKER_00: You're just guessing, right?
• 14:38
  SPEAKER_00: So it is an imperfect instrument,
• 14:40
  but it is the best instrument we have available to even think about
• 14:45
  approachinghow
• 14:47
  SPEAKER_00: how Americans view their lives and their government.
• 14:51
  And so if we want to...
• 14:52
  I mean,
• 14:54
  obviously,
• 14:54
  I would like polls to be less flawed.
• 14:56
  SPEAKER_00: I think the work that pollsters do to improve year after year,
• 15:01
  SPEAKER_00: cycle after cycle,
• 15:02
  SPEAKER_00: is critical and highly valuable.
• 15:04
  SPEAKER_00: And it...
• 15:06
  SPEAKER_00: only can make what we do in aggregating that data better.
• 15:12
  SPEAKER_00: And if we say,
• 15:14
  SPEAKER_00: well,
• 15:14
  SPEAKER_00: polls are flawed,
• 15:14
  SPEAKER_00: we give up on them,
• 15:16
  then I guess you have to rely on punditry to understand what Americans think.
• 15:20
  SPEAKER_00: And I think that's probably a pretty bad way to approach politics.
• 15:27
  SPEAKER_02: Well, you know, you guys sort of hinted at it, but I was also curious,
• 15:32
  giventhatyou'veobviouslyembarkedandyouhaveplentyofexperiencewith
• 15:37
  SPEAKER_02: aggregating data,
• 15:38
  SPEAKER_02: but sort of now doing it with FiftyPlusOne,
• 15:40
  sort of the current and future challenges for polling and for taking that
• 15:45
  dataand
• 15:45
  SPEAKER_02: making sense of it in a,
• 15:47
  SPEAKER_02: and as Elliot was pointing out,
• 15:48
  SPEAKER_02: a responsible way,
• 15:50
  SPEAKER_02: or Mary,
• 15:51
  SPEAKER_02: two of you have both pointed out,
• 15:52
  SPEAKER_02: in a responsible way,
• 15:53
  SPEAKER_02: taking data and using it transparently and,
• 15:59
  SPEAKER_02: understanding the limitations of that data.
• 16:01
  SPEAKER_02: And so I guess I was,
• 16:03
  SPEAKER_02: I'd love to hear your thoughts because obviously I have plenty of my own,
• 16:05
  but I was curious about your thoughts on certain sort of the current and
• 16:09
  future
• 16:09
  SPEAKER_02: challenges for polling and maybe for aggregating.
• 16:13
  SPEAKER_00: I mean,
• 16:13
  SPEAKER_00: I think the challenge,
• 16:15
  SPEAKER_00: and we saw this in the 2025 races,
• 16:18
  there is a big challenge in understanding the difference between a
• 16:24
  registeredvoter
• 16:24
  SPEAKER_00: and a likely voter,
• 16:26
  SPEAKER_00: right?
• 16:26
  SPEAKER_00: And so when you look at like,
• 16:27
  SPEAKER_00: for example,
• 16:28
  SPEAKER_00: the 2025 polling in Virginia,
• 16:30
  we had lots of surveys that were treating the likely voter universe like it
• 16:34
  would
• 16:34
  SPEAKER_00: be something like 2021 or 2024,
• 16:36
  SPEAKER_00: which were both more Republican-leaning environments.
• 16:39
  SPEAKER_00: Right.
• 16:39
  SPEAKER_00: So like having a good handle on who's actually going to show up to vote,
• 16:47
  SPEAKER_00: I think is a particular challenge,
• 16:48
  SPEAKER_00: especially in this moment where there is some question about whether some
• 16:53
  significant portion of the electorate is motivated primarily by voting for
• 16:58
  Donald
• 16:58
  SPEAKER_00: Trump,
• 16:59
  who will never be on a ballot again unless he wants to be like governor of
• 17:03
  Florida
• 17:03
  SPEAKER_00: or something.
• 17:04
  SPEAKER_00: I don't know.
• 17:07
  So I think that's one of the fundamental challenges when it comes to horse
• 17:11
  race
• 17:11
  SPEAKER_00: polling,
• 17:11
  SPEAKER_00: election polling,
• 17:13
  is understanding who's actually gonna turn up to the polls and how you can
• 17:18
  lookat
• 17:19
  historical voting records or asking voters whether or not they plan to vote
• 17:23
  and
• 17:24
  synthesize that into a likely voter model that actually resembles the
• 17:27
  likelyvoter
• 17:28
  universe that will turn up that is in the future that you cannot yet observe.
• 17:33
  SPEAKER_01: The fundamental problem for polling,
• 17:35
  SPEAKER_01: the thing that we're trying to solve with all of our methods,
• 17:40
  SPEAKER_01: with all the design and research choices,
• 17:42
  SPEAKER_01: with all of our modeling,
• 17:44
  is that the people who don't pick up the phone can be different than the people
• 17:46
  who
• 17:47
  SPEAKER_01: do.
• 17:48
  SPEAKER_01: Back when response rates were like 100%, this wasn't a big concern.
• 17:53
  SPEAKER_01: But today, they're less than 1%, sometimes less than half a percentage.
• 17:58
  SPEAKER_01: So if you want 1000 people, you've got to call 100,000 or 200,000.
• 18:01
  SPEAKER_01: And you're missing,
• 18:03
  you're potentially missing a big chunk of the electorate that thinks
• 18:06
  differently
• 18:07
  SPEAKER_01: than the types of people you can on the phone or,
• 18:09
  SPEAKER_01: you know,
• 18:09
  SPEAKER_01: most public opinion research today isn't actually on the phone,
• 18:12
  SPEAKER_01: it's on the web.
• 18:13
  SPEAKER_01: So or on the web.
• 18:15
  SPEAKER_01: Well,
• 18:16
  the most common methodology we see from pollsters that I would consider the
• 18:20
  highest
• 18:20
  quality is a hybrid model where it's partially over live phone calls and
• 18:24
  partially
• 18:25
  SPEAKER_00: SMS to web surveying.
• 18:28
  SPEAKER_01: Yeah.
• 18:29
  And that's great because it can cancel out some of the biases of either
• 18:33
  metric.
• 18:33
  Butifyouhavesomegroupofthepopulationthat'smuchmorewillingtotakea
• 18:38
  SPEAKER_01: survey like in 2016,
• 18:41
  SPEAKER_01: 2020 and 2024 Democrats,
• 18:42
  SPEAKER_01: regardless of the way you're calling those people,
• 18:45
  SPEAKER_01: even combining the sample still runs into problems.
• 18:48
  SPEAKER_01: So that has been the problem with polling, period, historically.
• 18:52
  SPEAKER_01: It will continue to be the problem in the future.
• 18:56
  And the techniques that pollsters have available to them to account for this
• 19:01
  are
• 19:02
  SPEAKER_01: actually getting a lot better.
• 19:03
  SPEAKER_01: So I'm pretty optimistic.
• 19:05
  Historically,
• 19:06
  pollsters have only really been able to say,
• 19:08
  oh,
• 19:08
  ifwe'reunder-representing
• 19:09
  SPEAKER_01: people based on their demographics,
• 19:11
  SPEAKER_01: so their race,
• 19:11
  SPEAKER_01: their age,
• 19:12
  SPEAKER_01: potentially their education,
• 19:14
  SPEAKER_01: depending on the survey.
• 19:15
  SPEAKER_01: They've been able to balance their samples.
• 19:17
  SPEAKER_01: Now we know that non-response,
• 19:19
  SPEAKER_01: not picking up the phone when a pollster calls you,
• 19:21
  can be correlated with your party and who you vote for in previous elections.
• 19:26
  SPEAKER_01: So pollsters can make adjustments based on those two as well.
• 19:29
  SPEAKER_01: Mary points out that this can go wrong in 2025.
• 19:33
  pollsters who are adjusting their samples in Virginia to be representative
• 19:37
  oflike
• 19:37
  SPEAKER_01: the presidential electorate in 2024,
• 19:39
  SPEAKER_01: where the previous governor electorate in 2021 still made problems.
• 19:42
  SPEAKER_01: They sort of introduced errors by making the wrong adjustments.
• 19:46
  SPEAKER_01: But if you do this the smart way and the best pollsters in 2025 did,
• 19:50
  SPEAKER_01: then you can still have really great accurate results,
• 19:53
  SPEAKER_01: even if,
• 19:54
  SPEAKER_01: you know,
• 19:54
  SPEAKER_01: only a fraction of the population is answering.
• 19:57
  SPEAKER_01: answering your survey, but it means you have to be really careful.
• 19:58
  SPEAKER_01: You have to really think through your decisions.
• 20:00
  SPEAKER_01: You have to be a professional.
• 20:02
  SPEAKER_01: You can't just,
• 20:02
  SPEAKER_01: you know,
• 20:03
  SPEAKER_01: can't just get him,
• 20:04
  SPEAKER_01: you know,
• 20:04
  get responses from Lucid or SurveyMonkey from your friends and call out a
• 20:08
  poll.
• 20:08
  SPEAKER_01: So you do have to be smart about it.
• 20:10
  SPEAKER_00: Yeah,
• 20:10
  SPEAKER_00: I mean,
• 20:11
  I think what both Elliot and I are saying here is that the fundamental
• 20:13
  challenge
• 20:14
  that will always be in front of people trying to measure public opinion is
• 20:18
  ensuring
• 20:20
  SPEAKER_00: representativeness,
• 20:22
  SPEAKER_00: is that the right word,
• 20:23
  SPEAKER_00: in your survey.
• 20:24
  SPEAKER_00: Whether that's in the underlying sample or in the target universe,
• 20:27
  SPEAKER_00: these are both big questions,
• 20:29
  SPEAKER_00: right?
• 20:29
  So like Elliot highlights the difficulty of getting the underlying sample
• 20:33
  toberepresentative
• 20:35
  SPEAKER_00: of the population.
• 20:36
  I'm highlighting the difficulty of like estimating what the population
• 20:39
  actuallyis
• 20:41
  SPEAKER_00: in the electorate,
• 20:41
  butbothofthosearefundamentallyabouthowdoweensurethatthesurveyis
• 20:45
  SPEAKER_00: representative of the population that we care to represent.
• 20:48
  SPEAKER_02: Yeah, exactly.
• 20:49
  SPEAKER_02: And we wrote a lot about this at 538,
• 20:52
  SPEAKER_02: obviously,
• 20:53
  butthedifficultyisthatifyouhaveakindofabadsamplefromthestartora
• 20:58
  SPEAKER_02: pretty unrepresentative sample,
• 20:59
  youcan'tnecessarilyweightyourwayoutofthatbyadjustingitforvarious
• 21:04
  SPEAKER_02: things to try to make it look more like the population.
• 21:06
  SPEAKER_02: If you're still missing people and if there's a fundamental,
• 21:09
  SPEAKER_02: as Elliot was getting,
• 21:10
  SPEAKER_02: a fundamental difference in some political perspectives,
• 21:14
  SPEAKER_02: on average for people who are less likely to respond.
• 21:17
  SPEAKER_02: And so if you're missing them and they're even they even look,
• 21:19
  SPEAKER_02: you know,
• 21:19
  SPEAKER_02: for instance,
• 21:20
  like a Republican who doesn't respond to a poll might look different in
• 21:23
  certain
• 21:23
  SPEAKER_02: important ways from a Republican who does answer.
• 21:25
  SPEAKER_03: Yeah.
• 21:26
  SPEAKER_02: You know,
• 21:26
  and we saw that a lot with with polling in the presidential elections in 2020
• 21:30
  and
• 21:30
  SPEAKER_02: 2024 and talk about,
• 21:30
  SPEAKER_02: you know.
• 21:34
  SPEAKER_02: more likely to respond, Republicans may have been a little less Trumpy.
• 21:38
  SPEAKER_02: And so you were missing,
• 21:41
  SPEAKER_02: perhaps to some extent,
• 21:42
  SPEAKER_02: or underestimating,
• 21:42
  SPEAKER_02: I should say,
• 21:43
  SPEAKER_02: not missing necessarily,
• 21:44
  SPEAKER_02: but underestimating Trump's support,
• 21:47
  SPEAKER_02: at least to some extent,
• 21:49
  SPEAKER_02: in both those elections.
• 21:50
  SPEAKER_02: Now,
• 21:50
  SPEAKER_02: this brings me actually to a question that I know,
• 21:53
  SPEAKER_02: Elliot,
• 21:53
  SPEAKER_02: you wrote something the other day about,
• 21:55
  but thinking about the 2025 election and some of the polling we saw there,
• 21:59
  SPEAKER_02: and I thought,
• 22:00
  SPEAKER_02: you know,
• 22:02
  Maybe it's too easy to lead with the case that was sort of the most notable miss
• 22:07
  in
• 22:07
  SPEAKER_02: terms of what the polls had.
• 22:09
  SPEAKER_02: But New Jersey,
• 22:11
  SPEAKER_02: where generally speaking,
• 22:14
  SPEAKER_02: polling aggregators,
• 22:16
  SPEAKER_02: about five or six points generally,
• 22:18
  SPEAKER_02: in terms of margin for Mikie Sherrill,
• 22:21
  SPEAKER_02: the Democratic nominee,
• 22:22
  SPEAKER_02: and she ended up winning by about 14.
• 22:25
  uh and obviously that's a pretty significant under shot under shoot uh
• 22:30
  however
• 22:30
  whateverwhateveristhepropertermthereandiiguessIwantedsomeofyour
• 22:37
  thoughtslikeit'syou'retryingtofigureoutwhattheelectorateisgoing
• 22:41
  to look like so I know that was definitely one of the challenges here but also
• 22:45
  some
• 22:45
  SPEAKER_02: bigger questions Elliot I know you talked a bit about
• 22:49
  SPEAKER_02: people trying to weight their samples to the presidential electorate.
• 22:53
  SPEAKER_02: And obviously 2025's electorate is not going to look the same as 2024.
• 22:57
  ButIguessifyouwanttojuststartoutwithsomeofyourinitialthoughtson
• 23:01
  SPEAKER_02: those challenges.
• 23:04
  SPEAKER_01: Yeah,
• 23:04
  this is a great example because there's a lot of different pollsters doing a
• 23:09
  lotof
• 23:10
  SPEAKER_01: different things in New Jersey and getting wildly different results.
• 23:14
  SPEAKER_01: I will point out in your screenshot, I think FiftyPlusOne does the best.
• 23:17
  SPEAKER_01: So a little feather in our cap.
• 23:21
  butcongratstoyoucongratstoyoubutthesoI'lljustpicktwoI'llpickthe
• 23:25
  worstpollsterandthebestpollsterinnewjerseyandandthenmaybewecan
• 23:31
  learnsomethingheretheworstpollsterintheNewJerseyraceisatlasintel
• 23:35
  whichsortofparadoxicallywasratedthebestpollsterinthe2024raceand
• 23:41
  SPEAKER_01: they have the sort of lowest statistical bias and the best
• 23:45
  SPEAKER_01: accuracy in 2024.
• 23:47
  SPEAKER_01: So hats off to them that year.
• 23:50
  SPEAKER_01: But they,
• 23:52
  SPEAKER_01: a week before the election or a couple of days before the election,
• 23:54
  SPEAKER_01: had Sherrill winning by just 1, 50 to 49 over Ciattarelli.
• 23:58
  SPEAKER_01: And then on the other end of the spectrum,
• 24:00
  SPEAKER_01: we have YouGov,
• 24:01
  who I believe has Mikie Sherrill plus 10 in their survey closest to the
• 24:07
  election.
• 24:08
  SPEAKER_01: And they have the lowest error.
• 24:09
  SPEAKER_01: So they have what ends up,
• 24:11
  SPEAKER_01: I guess,
• 24:11
  SPEAKER_01: being a four point miss,
• 24:13
  SPEAKER_01: which is inside the margin of error or maybe just outside.
• 24:16
  SPEAKER_02: I should point out that I had some stats,
• 24:19
  SPEAKER_02: actually,
• 24:19
  that the average error in 538 data from 98 through 2022 for governor's races,
• 24:25
  SPEAKER_02: the average error was five points,
• 24:26
  SPEAKER_02: basically.
• 24:27
  SPEAKER_02: So four point miss is not, you know, that's typical.
• 24:31
  SPEAKER_02: That's typical.
• 24:32
  SPEAKER_01: Yeah, that's expected.
• 24:33
  You should expect,
• 24:34
  on average,
• 24:35
  a survey to be off given the historical patterns here.
• 24:38
  SPEAKER_01: You shouldn't expect them to be perfect.
• 24:40
  But there are some key differences in how YouGov and Atlas Intel were
• 24:44
  collecting
• 24:44
  SPEAKER_01: and processing their data.
• 24:46
  SPEAKER_01: So AtlasIntel,
• 24:47
  SPEAKER_01: they run this,
• 24:48
  SPEAKER_01: I guess,
• 24:49
  very AI computationally complex survey recruitment method where they put
• 24:55
  adson
• 24:56
  SPEAKER_01: your phone.
• 24:57
  SPEAKER_01: That was supposed to be sarcastic, by the way.
• 24:59
  SPEAKER_01: if you're an Instagram user,
• 25:00
  SPEAKER_01: and then feed you,
• 25:01
  SPEAKER_01: and I think maybe they use other technologies as well,
• 25:03
  SPEAKER_01: but then they feed you into a web form to fill out their survey.
• 25:08
  SPEAKER_01: That's how they collect their data.
• 25:10
  SPEAKER_01: They do set targets, apparently.
• 25:13
  We don't have a whole lot of information,
• 25:14
  butitseemsliketheysettargets
• 25:15
  targets based off of demographics and then also weight their survey by past
• 25:20
  vote
• 25:21
  SPEAKER_01: and party affiliation somewhat ad hoc,
• 25:25
  SPEAKER_01: which,
• 25:26
  SPEAKER_01: you know,
• 25:28
  it seems like the variables that they're waiting on change from survey to
• 25:31
  survey
• 25:32
  SPEAKER_01: context to context.
• 25:32
  That's not necessarily like other pollsters also change how they do their
• 25:36
  polls
• 25:37
  SPEAKER_01: from state to state.
• 25:37
  SPEAKER_01: So.
• 25:38
  SPEAKER_01: Not a super red flag.
• 25:39
  SPEAKER_01: But the way that YouGov is conducting their data,
• 25:41
  SPEAKER_01: I think it's closer to what you would want your ideal poll to look like.
• 25:45
  YouGov has this massive 2 million people panel of respondents on their web
• 25:51
  platform online.
• 25:54
  And they can pick people from that platform to respond based on their
• 25:59
  demographics
• 26:00
  SPEAKER_01: and their partisanship.
• 26:02
  SPEAKER_01: So they can say,
• 26:02
  SPEAKER_01: you know,
• 26:03
  I want a sample of people based on their partisanship and their demographics
• 26:06
  of,
• 26:07
  SPEAKER_01: in this case,
• 26:07
  SPEAKER_01: New Jersey,
• 26:08
  that match the demographics and partisanship of New Jersey registered
• 26:12
  voters.
• 26:12
  SPEAKER_01: And then they can apply a likely voter screen on top of that.
• 26:15
  SPEAKER_01: If people say they're really likely to vote,
• 26:18
  theyassignthosepeoplesomeprobabilityofvotingandthatgetstakeninto
• 26:23
  SPEAKER_01: account in their waiting step.
• 26:25
  And they let the sample guide what the likely voter universe is going to look
• 26:29
  like.
• 26:30
  SPEAKER_01: instead of the Atlas Intel approach,
• 26:32
  whichistomandatethatthelikelyvotersampleisgoingtohavesomecertain
• 26:36
  SPEAKER_01: characteristics.
• 26:38
  SPEAKER_01: If it indeed for AtlasIntel was waiting on past vote,
• 26:42
  SPEAKER_01: making the electorate look like it did in 2021 or 2024,
• 26:44
  SPEAKER_01: that caused them to have
• 26:47
  such a massive misfire and I would say that that's pretty likely based on how
• 26:51
  they
• 26:51
  SPEAKER_01: do their other surveys,
• 26:53
  then that's just sort of suggestive that like they're putting their thumb
• 26:57
  toomuch
• 26:57
  on the scale to try to make these adjustments and kind of need to dial back and
• 27:01
  let
• 27:01
  SPEAKER_01: the sample lead the way on the likely voter filter.
• 27:06
  SPEAKER_00: Yeah,
• 27:06
  SPEAKER_00: and that methodology,
• 27:07
  SPEAKER_00: that online targeted ad methodology is very,
• 27:09
  SPEAKER_00: very tricky because you might get different click-through rates,
• 27:15
  SPEAKER_00: again,
• 27:15
  SPEAKER_00: based on different demographics.
• 27:16
  And it's a lot more difficult to control with quotas than some other kinds of
• 27:23
  methodologies.
• 27:23
  So just to flag,
• 27:24
  like not to say that I think it's always a bad methodology.
• 27:27
  Obviously,
• 27:28
  as Elliot said,
• 27:28
  Atlas Intel was one of the standout pollsters of 2024.
• 27:33
  SPEAKER_00: But it can be a tricky methodology to wield.
• 27:37
  SPEAKER_02: Yeah, I mean, perhaps... Oh, sorry.
• 27:39
  SPEAKER_01: And I'll just say, also, it can break down over time, right?
• 27:42
  So if you have a variable that's correlated with non-response that you wait
• 27:48
  byin
• 27:48
  SPEAKER_01: one year,
• 27:48
  SPEAKER_01: but the rest of your data generating process isn't super representative,
• 27:52
  SPEAKER_01: so you don't have representative...
• 27:54
  SPEAKER_01: cross tabs or something, for example.
• 27:56
  SPEAKER_01: And I guess by way of sidebar,
• 27:57
  like AtlasIntel's poll in New Jersey had the Republican winning 60% of the
• 28:01
  vote
• 28:02
  SPEAKER_01: among black voters.
• 28:03
  SPEAKER_01: That's probably like they just had a biased composition of black voters.
• 28:06
  SPEAKER_01: And that's worth like nine.
• 28:08
  SPEAKER_01: I did the math that was worth nine points.
• 28:09
  If they had the right sample there,
• 28:10
  they would have had Mikie Sherrill up nine.
• 28:13
  SPEAKER_01: by way of sidebar, like those sorts of things.
• 28:16
  SPEAKER_01: If you're really reliant on your weighting model,
• 28:18
  SPEAKER_01: having the right correlations to non-response,
• 28:21
  SPEAKER_01: then you are very,
• 28:23
  very subject to the non-response environment and the composition of the
• 28:27
  electorate
• 28:27
  SPEAKER_01: in a way you wouldn't be if you had a more robust survey.
• 28:31
  SPEAKER_01: So there is a bit of like fragility here,
• 28:34
  maybe in the way they're doing their surveys that makes them not as accurate
• 28:38
  as
• 28:38
  SPEAKER_01: they would have claimed,
• 28:39
  SPEAKER_01: I think,
• 28:39
  SPEAKER_01: over the long term.
• 28:41
  SPEAKER_02: And perhaps it speaks also,
• 28:44
  SPEAKER_02: again,
• 28:45
  SPEAKER_02: polling for a presidential election in 2024,
• 28:48
  SPEAKER_02: not to say it was easier or something,
• 28:50
  SPEAKER_02: but you did have 2020 pass vote perhaps as a more reliable tool.
• 28:56
  And obviously a lot of pollsters employed that as a waiting mechanism when
• 29:01
  looking
• 29:01
  SPEAKER_02: at the 2024 electorate.
• 29:04
  SPEAKER_02: And I know that was a very popular thing.
• 29:06
  SPEAKER_02: And it just so happens that the 2024 presidential error,
• 29:10
  SPEAKER_02: Elliot,
• 29:10
  SPEAKER_02: I think you were a part of it,
• 29:11
  but the Association for American Public Opinion Research sort of
• 29:16
  post-mortemon
• 29:17
  SPEAKER_02: 2024 polling found still underestimated Trump,
• 29:22
  SPEAKER_02: but not by nearly as much as it did in 2020 or 2016.
• 29:27
  SPEAKER_02: And I know that one part of that was that many pollsters turned to
• 29:33
  waiting on past presidential vote as a way of trying to ensure that their
• 29:37
  samples
• 29:38
  were not overly Democratic leaning and that they were perhaps adjusting for
• 29:43
  the
• 29:43
  fact that there was some important element of the population that was being
• 29:48
  missed.
• 29:49
  SPEAKER_02: or was less likely to respond.
• 29:52
  SPEAKER_02: And so when I think about that though for a,
• 29:54
  SPEAKER_02: now to be clear,
• 29:55
  SPEAKER_02: New Jersey actually had pretty high turnout for a gubernatorial election.
• 29:57
  SPEAKER_02: It was like the highest in recent memory for New Jersey.
• 30:01
  It was about 50,
• 30:01
  51% of the voting eligible population actually showed up,
• 30:04
  which was very high.
• 30:05
  SPEAKER_02: It was only about 41% in 2021.
• 30:09
  SPEAKER_02: But you're still talking about,
• 30:11
  SPEAKER_02: in New Jersey,
• 30:12
  two-thirds of the voting eligible population showing up for the 2024
• 30:15
  presidential
• 30:16
  SPEAKER_02: election and then half of it showing up
• 30:18
  SPEAKER_02: for a gubernatorial election.
• 30:19
  So you are talking about a different universe of voters and one where what
• 30:25
  that 16,
• 30:27
  SPEAKER_02: 17% that didn't show up,
• 30:28
  SPEAKER_02: what that group looks like is also important.
• 30:31
  SPEAKER_02: And so in terms of thinking about weighting and what you're weighting to,
• 30:35
  SPEAKER_02: I think maybe that speaks,
• 30:36
  SPEAKER_02: Elliot,
• 30:36
  SPEAKER_02: to you saying,
• 30:37
  talkingaboutsortoflettingthesampleleadthewayalittlebitmorejust
• 30:41
  SPEAKER_02: because it will look different.
• 30:43
  SPEAKER_02: Yeah.
• 30:44
  SPEAKER_00: Yeah.
• 30:44
  I think about this a lot for 2028 because waiting to pass presidential vote
• 30:49
  worked
• 30:49
  SPEAKER_00: well in 2020 and 2024,
• 30:51
  in part because we have the same Republican nominee and we know that Donald
• 30:56
  Trump
• 30:56
  SPEAKER_00: drove turnout among a certain subset of the electorate.
• 30:58
  SPEAKER_00: So I'm really curious and very nervous about 2028 polling,
• 31:04
  SPEAKER_00: whether without Trump on the ballot,
• 31:06
  SPEAKER_00: that technique is going to be as effective as
• 31:08
  Andtheunfortunatethingiswecannotknowtheanswertothatuntilafterthe
• 31:13
  SPEAKER_00: election is over.
• 31:14
  Wait,
• 31:15
  Mary,
• 31:15
  are you saying you can't predict the direction of polling error?
• 31:20
  Well,
• 31:20
  I mean,
• 31:21
  I'm actually saying something a little different than that,
• 31:23
  actually.
• 31:23
  SPEAKER_00: I know, sorry.
• 31:24
  SPEAKER_02: This has been a pet peeve of mine for years.
• 31:26
  Thisideaofhowdoweadjustthesampletoensurerepresentativenessofthe
• 31:31
  SPEAKER_00: population that we're attempting to represent.
• 31:34
  SPEAKER_00: Now, in the past couple of elections, because we've had sort of like
• 31:38
  SPEAKER_00: Similar dynamics,
• 31:39
  especially on the Republican side with the same nominee for three elections
• 31:42
  ina
• 31:42
  SPEAKER_00: row.
• 31:43
  It turns out that guessing that the electorate is going to look pretty
• 31:48
  similarto
• 31:48
  SPEAKER_00: the last election has been a pretty effective strategy.
• 31:51
  But when we have a new Republican nominee on the ticket,
• 31:54
  is it going to be as effective?
• 31:56
  AretheselikelowerpropensityDonaldTrumpvotersgoingtoshowupina2028
• 32:00
  SPEAKER_00: election to like pull the lever for JD
• 32:02
  SPEAKER_00: Vance?
• 32:02
  SPEAKER_00: I don't know.
• 32:03
  SPEAKER_00: Nobody knows.
• 32:05
  Sowe'llseehowthat'slikeifyou'dwaitedon2012votein2016consistently
• 32:11
  you would have missed some swings or at least perhaps underestimated some
• 32:15
  swings
• 32:15
  especially in certain states uh so yeah i mean it speaks to the challenge this
• 32:20
  is
• 32:20
  this at least has been a little bit of a short-term solution but it's unclear
• 32:24
  about
• 32:24
  SPEAKER_02: its longevity
• 32:26
  SPEAKER_00: Which,
• 32:27
  SPEAKER_00: like I said,
• 32:27
  just makes me nervous for the next set of polls for a presidential election.
• 32:32
  SPEAKER_00: If pollsters overlearn the lessons of the last couple of years,
• 32:35
  we could find ourselves in a position where the polls are very wrong in a
• 32:38
  different
• 32:39
  SPEAKER_00: kind of way.
• 32:40
  SPEAKER_00: Not to say directionally, but a different kind of problem.
• 32:45
  SPEAKER_01: We need to get to a point,
• 32:46
  SPEAKER_01: probably,
• 32:47
  SPEAKER_01: maybe to get back to your point,
• 32:49
  SPEAKER_01: Geoffrey,
• 32:49
  SPEAKER_01: where people expect polling error.
• 32:51
  SPEAKER_01: This is my entire argument in my book.
• 32:53
  Expect them to be wrong by about two or three points on average in a
• 32:56
  presidential
• 32:57
  SPEAKER_01: race and more in other contexts,
• 32:58
  SPEAKER_01: because that's how wrong they've been historically.
• 33:00
  SPEAKER_01: If you lower your expectations for them,
• 33:02
  you unlock so much more utility for surveys in other contexts than
• 33:08
  pre-election
• 33:08
  SPEAKER_01: polling because
• 33:12
  you're not expecting perfection and then throwing the baby out with the
• 33:15
  bathwater,
• 33:15
  I say.
• 33:16
  SPEAKER_00: Yeah,
• 33:17
  andIthinkthisisactuallysomethingthatwasreallyimportantforusat
• 33:20
  SPEAKER_00: 538 and will continue to be important for us at FiftyPlusOne moving
• 33:23
  SPEAKER_00: forward is how do we communicate in words,
• 33:26
  SPEAKER_00: in visuals,
• 33:26
  SPEAKER_00: in all of the things we do,
• 33:28
  SPEAKER_00: uncertainty about what we are saying,
• 33:31
  SPEAKER_00: right?
• 33:31
  SPEAKER_00: So like you guys both know how many meetings we had at 538 trying to
• 33:35
  SPEAKER_00: figure out forecast design where we can clearly communicate like,
• 33:39
  SPEAKER_00: hey,
• 33:40
  SPEAKER_00: this is not like the prediction.
• 33:44
  This is a range of predictions with varying probabilities and trying to
• 33:49
  really
• 33:50
  SPEAKER_00: communicate with people that like,
• 33:52
  we're trying to tell you the set of possible outcomes and how likely those
• 33:55
  possible
• 33:56
  SPEAKER_00: outcomes are,
• 33:57
  SPEAKER_00: not the thing that will happen.
• 33:59
  SPEAKER_00: If we knew the thing that would happen, why would we run the election?
• 34:03
  SPEAKER_02: I mean,
• 34:03
  SPEAKER_02: yeah,
• 34:04
  SPEAKER_02: and I actually,
• 34:05
  SPEAKER_02: I brought up an old table from a piece from 2023,
• 34:08
  SPEAKER_02: which was just most,
• 34:10
  SPEAKER_02: the quick one I could quickest pull up here,
• 34:13
  basically showing like weighted average error in polling for different
• 34:19
  kindsof
• 34:19
  SPEAKER_02: races from 538 on the old site.
• 34:24
  SPEAKER_02: And, you know, the average error
• 34:27
  for some of these races was like five,
• 34:29
  was nine points into presidential primary.
• 34:32
  SPEAKER_02: And that makes sense.
• 34:32
  That's like the most,
• 34:34
  people are more likely to shift,
• 34:36
  make up their minds later.
• 34:38
  SPEAKER_02: They're choosing among people that they tend to,
• 34:40
  SPEAKER_02: they would tend to like multiple candidates because they're a Democrat or
• 34:43
  SPEAKER_02: Republican for the most part.
• 34:44
  SPEAKER_00: And candidates drop out all the time.
• 34:46
  SPEAKER_02: Candidates drop out.
• 34:47
  SPEAKER_02: So things can shift very suddenly and very quickly.
• 34:50
  And then in the general,
• 34:51
  you know,
• 34:51
  for presidential,
• 34:52
  it was about four points.
• 34:54
  SPEAKER_02: which not actually, you know, this is for state and national polls.
• 35:01
  SPEAKER_02: So, and then for Senate, Gov, more like five points, for House, six.
• 35:07
  And,
• 35:07
  you know,
• 35:07
  just based on the fact that,
• 35:09
  as Elliot was saying,
• 35:09
  we should expect some error.
• 35:11
  SPEAKER_02: And so when we come in below those kinds of rates,
• 35:14
  SPEAKER_02: it actually means that the polls did pretty well,
• 35:16
  SPEAKER_02: which maybe is my chance,
• 35:18
  SPEAKER_02: and we should,
• 35:18
  SPEAKER_02: maybe we can finish on this,
• 35:19
  SPEAKER_02: is that I know you guys were both involved
• 35:22
  SPEAKER_02: in some polling in Virginia.
• 35:25
  SPEAKER_02: And I'll just let the header sit there with State Navigate.
• 35:31
  SPEAKER_02: In fact, there's Mary's name right there.
• 35:33
  SPEAKER_02: At the top, Chaz Nuttycombe, who runs State Navigate.
• 35:37
  SPEAKER_02: And basically talking about,
• 35:38
  SPEAKER_02: I just thought it'd be cool to hear you guys talk for a minute about
• 35:42
  SPEAKER_02: about this poll and how it did differ,
• 35:44
  SPEAKER_02: it did come closer to the sort of the right,
• 35:48
  SPEAKER_02: look,
• 35:49
  SPEAKER_02: obviously all polls are snapshots in time,
• 35:51
  SPEAKER_02: but this one being just before the election,
• 35:54
  SPEAKER_02: we would hope that pollsters would be getting somewhat close
• 35:58
  to the final outcome if they're getting a reasonably good read of the
• 36:02
  electorate.
• 36:03
  SPEAKER_02: And in your case,
• 36:05
  SPEAKER_02: you know,
• 36:05
  SPEAKER_02: Spanberger's going to win by 15-ish,
• 36:08
  SPEAKER_02: 14-15,
• 36:08
  SPEAKER_02: and you had over the 30-point lead.
• 36:10
  SPEAKER_02: So obviously- Well,
• 36:11
  SPEAKER_00: and if I can toot our own horn a little bit,
• 36:13
  SPEAKER_00: I think we had Hashmi at plus 10 in this survey.
• 36:16
  SPEAKER_02: And she's going to win by like 11.
• 36:18
  SPEAKER_00: And Jones by five.
• 36:20
  SPEAKER_02: And he's gonna win by about six.
• 36:21
  SPEAKER_02: So yeah, yeah.
• 36:22
  SPEAKER_00: Pretty good, pretty good.
• 36:26
  SPEAKER_00: Okay, a couple of things I wanna say about this.
• 36:28
  SPEAKER_00: So first of all, I mean, Chaz knows like everything about Virginia.
• 36:34
  And so when we were thinking about the composition of the likely electorate
• 36:37
  in
• 36:37
  SPEAKER_00: Virginia,
• 36:38
  we made the decision to model the electorate as something closer to a
• 36:41
  2017-like
• 36:42
  SPEAKER_00: environment than a 2021 or 2024-like environment.
• 36:46
  That's generally what you expect in Virginia in these off-year
• 36:50
  gubernatorial
• 36:50
  SPEAKER_00: elections is that when a Republican is in power,
• 36:55
  SPEAKER_00: you have a more Democratic-leaning environment.
• 36:57
  When a Democrat is in power,
• 36:58
  you have a more Republican-leaning environment.
• 37:00
  And so we went back to the most recent Democratic-leaning gubernatorial
• 37:04
  election
• 37:05
  SPEAKER_00: with respect to party identification.
• 37:09
  SPEAKER_00: in particular when we were thinking about weighting the survey.
• 37:14
  SPEAKER_00: Then the second thing we did is we tried lots of different,
• 37:18
  SPEAKER_00: this is entirely text to SMS to web,
• 37:21
  SPEAKER_00: text to web survey.
• 37:22
  Sowetriedabunchofdifferentthingsinourtextmessagestomakesurewehad
• 37:26
  SPEAKER_00: representation from the groups that we were trying to target.
• 37:29
  SPEAKER_00: For example,
• 37:31
  SPEAKER_00: after the first few days of fielding,
• 37:32
  SPEAKER_00: we were like,
• 37:33
  SPEAKER_00: man,
• 37:33
  SPEAKER_00: we don't have enough young people.
• 37:35
  SPEAKER_00: So we had a Gen Z-er write the text.
• 37:38
  SPEAKER_00: No, we just asked someone we know who's 20.
• 37:40
  SPEAKER_00: We were like,
• 37:41
  SPEAKER_00: hey,
• 37:41
  SPEAKER_00: if you were texting your friend and you wanted them to click this link,
• 37:44
  SPEAKER_00: how would you write this text?
• 37:45
  SPEAKER_00: And they wrote it for us.
• 37:46
  SPEAKER_00: And then we sent that out.
• 37:47
  SPEAKER_00: And it tripled or quadrupled our response rate among young voters.
• 37:52
  SPEAKER_00: Great.
• 37:53
  SPEAKER_01: Amazing.
• 37:54
  SPEAKER_01: I'm trying to find the actual text that was sent out,
• 37:57
  SPEAKER_01: but I think it was just like,
• 37:58
  SPEAKER_01: not enough people are answering our poll,
• 38:00
  SPEAKER_01: LOL,
• 38:00
  SPEAKER_01: please PLS help or something.
• 38:02
  SPEAKER_00: Yes, yes.
• 38:03
  SPEAKER_00: It was all lowercase.
• 38:05
  SPEAKER_00: It was all lowercase.
• 38:06
  SPEAKER_00: And it was like, yo, we need more young people to answer our poll.
• 38:10
  SPEAKER_00: LOL, please help.
• 38:11
  SPEAKER_00: And please is spelled PLS.
• 38:13
  SPEAKER_00: And then it has the link.
• 38:14
  SPEAKER_00: Honestly, listen, I am 40 years old.
• 38:16
  SPEAKER_00: This sounded ridiculous to me, but the young people clicked the link.
• 38:20
  SPEAKER_00: Okay.
• 38:22
  SPEAKER_02: And you heard from them.
• 38:25
  SPEAKER_00: I don't want to say we had,
• 38:27
  SPEAKER_00: at that point,
• 38:27
  SPEAKER_00: once we did that,
• 38:28
  SPEAKER_00: a stellar response rate among young people,
• 38:30
  SPEAKER_00: but it was better.
• 38:31
  So we tried a bunch of different things to try to make sure we increased that
• 38:35
  representation.
• 38:37
  And we used history to inform the composition of the likely electorate in a
• 38:42
  waythatIthink
• 38:47
  understood the thermostatic nature of these kinds of things that not all
• 38:51
  posters did.
• 38:53
  SPEAKER_01: I will say what we did was kind of risky.
• 38:54
  Like we were asserting our own prior about what the composition of the
• 38:59
  electorate should be.
• 39:01
  SPEAKER_01: And that creates the potential that our assumption was wrong.
• 39:06
  This is basically what all the other pollsters,
• 39:07
  they just use the wrong baseline,
• 39:09
  right?
• 39:09
  They used maybe the partisan composition according to past vote in the 2024
• 39:12
  election.
• 39:14
  I think we had a good reason to know that that was gonna be wrong ahead of time.
• 39:16
  SPEAKER_01: You just sort of think through the consequences of that.
• 39:19
  SPEAKER_01: And we were like, no, we don't wanna do that.
• 39:20
  SPEAKER_01: The electorate's gonna be different.
• 39:21
  SPEAKER_01: Other pollsters should have also,
• 39:23
  SPEAKER_01: I think,
• 39:23
  SPEAKER_01: seen that coming if indeed they were waiting by pass vote.
• 39:28
  SPEAKER_01: But,
• 39:28
  SPEAKER_01: you know,
• 39:28
  this this shows what type of overperformance you can get for your surveys
• 39:33
  whenyou
• 39:35
  SPEAKER_01: are really,
• 39:35
  SPEAKER_01: really intimately aware of the how the electorate behaves.
• 39:40
  And like you have domain knowledge in the case of Chaz for the population
• 39:44
  you're trying to.
• 39:45
  SPEAKER_01: to survey.
• 39:46
  SPEAKER_01: So in the future,
• 39:47
  I'd like to let the sample lead us more to use a probabilistic likely voter
• 39:51
  filter,
• 39:52
  SPEAKER_01: for example,
• 39:53
  SPEAKER_01: to wait based on the registered population of the poll.
• 39:55
  SPEAKER_01: But at least in this case,
• 39:57
  we either got lucky or you can't really argue with the results either way,
• 40:01
  SPEAKER_01: I guess.
• 40:02
  SPEAKER_01: Well,
• 40:02
  SPEAKER_02: I'm sure as you guys hopefully get to do this more,
• 40:06
  SPEAKER_02: you'll get to try out different things.
• 40:09
  I hope that you're able to keep doing this kind of work with State Navigate,
• 40:12
  SPEAKER_02: which is also a great site.
• 40:14
  SPEAKER_02: I hope to have Chaz on at some point.
• 40:16
  SPEAKER_02: Maybe just talk about Virginia in the future.
• 40:18
  Chaz is one of the only people I would put up against me for knowledge of
• 40:22
  Virginia politics.
• 40:25
  SPEAKER_02: So he'd be a perfect person to have on for that.
• 40:28
  SPEAKER_02: Mary Radcliffe,
• 40:30
  SPEAKER_02: Elliot Morris,
• 40:31
  Ireallywanttothankyouguysforcomingonwithmeandchattingwithmeonthe
• 40:34
  SPEAKER_02: Decision Desk HQ podcast.
• 40:36
  SPEAKER_02: They are running FiftyPlusOne,
• 40:40
  SPEAKER_02: spelled out,
• 40:41
  SPEAKER_02: polling site,
• 40:42
  SPEAKER_02: doing great work on aggregation and trying to understand
• 40:48
  what people are thinking and their attitudes about the important questions
• 40:51
  of our moment.
• 40:51
  SPEAKER_02: So again, I'm Geoffrey Skelley.
• 40:54
  SPEAKER_02: This has been the Decision Desk HQ podcast, and thanks for listening.