How To Read 2020 Polls Like A FiveThirtyEighter

Beware: Reading polls can be hazardous to your health. Symptoms include cherry-picking, overconfidence, falling for junky numbers and rushing to judgment. Thankfully, we have a cure. Building on an old checklist from former FiveThirtyEight political writer Harry Enten, here are some guidelines you should bear in mind when you’re interpreting political polling — in primary season and beyond.

What to watch for during the primaries

People who try to discredit early primary polls by pointing out that, say, Jeb Bush led early polls of the GOP field in 2016 are being disingenuous. Should these polls be treated with caution? Sure, but national primary polls conducted in the calendar year before the election are actually somewhat predictive of who the eventual nominee will be. Earlier this year, fellow FiveThirtyEight analyst Geoffrey Skelley looked at early primary polling since 1972 and found that candidates who polled better in the months before the primaries wound up doing better in the eventual primaries. In fact, those who averaged 35 percent or higher in the polls rarely lost the nomination.

And if we go one step further and account for a candidate’s level of name recognition, early national primary polls were even more telling of who might win the nomination. As you can see in the chart below, a low-name-recognition candidate whose polling average climbed past 10 percent in the first half of the year before the primaries had at least a 1 in 4 shot at winning, which actually put them ahead of a high-name-recognition candidate polling at the same level.

This is why we believe that national primary polls are useful (even this far out) despite the fact that they are technically measuring an election that will never happen — we don’t hold a national primary. For this reason, early-state polls are important, too, especially if they look different from national polls. History is littered with examples of national underdogs who pulled off surprising wins in Iowa or New Hampshire, then rode the momentum all the way to the nomination. And according to analysis from RealClearPolitics, shortly after Thanksgiving is historically when polls of Iowa and New Hampshire start to come into closer alignment with the eventual results.

But don’t put too much faith in early primary polls (or even late ones — they have a much higher error, on average, than general-election polls). Voters’ preferences are much more fluid in primaries than they are in general elections, in large part because partisanship, a reliable cue in general elections, is removed from the equation. And voters may vacillate between the multiple candidates they like and even change their mind at the last minute, perhaps in an effort to vote tactically (i.e., vote for their second choice because that candidate has a better chance of beating a third candidate whom the voter likes less than their first or second choice).

On the flip side, early general-election polls are pretty much worthless. They are hypothetical match-ups between candidates who haven’t had a chance to make their case to the public, who haven’t had to withstand tough attacks and who still aren’t on many Americans’ radar. And these polls aren’t terribly predictive of the eventual result either. From 1944 to 2012, polls that tested the eventual Democratic and Republican nominees about a year before the election (specifically, in November and December of the previous year) missed the final margin by almost 11 percentage points, on average — though it’s worth noting that they were more accurate in 2016, missing by around 3 points.¹

In other words, at this stage in the cycle, primary polls can be useful but are by no means infallible, while general-election polls can safely be ignored. That may seem frustrating, but just remember that pollsters aren’t trying to make predictions; they’re simply trying to capture an accurate snapshot of public opinion at a given moment in time.

What to keep in mind generally

There are some guidelines you should remember at any time of the year, however. First, some pollsters are more accurate than others. We consider the gold standard of polling methodology to be pollsters that use live people (as opposed to robocalls) to conduct interviews over the phone, that call cell phones as well as landlines and that participate in the American Association for Public Opinion Research’s Transparency Initiative or the Roper Center for Public Opinion Research archive. That said, the polling industry is changing; there are some good online pollsters too. You can use FiveThirtyEight’s Pollster Ratings to check what methodology each pollster uses and how good its track record has been. (And if a pollster doesn’t show up in our Pollster Ratings, that might be a red flag.)

Another reason to pay attention to the pollster is for comparison purposes. Because pollsters sometimes have consistent house effects (their polls overestimate the same party over and over), it can be tricky to compare results from different pollsters. (For this reason, FiveThirtyEight’s models adjust polls to account for house effects.) When looking for trends in the data over time, it’s better to compare a poll to previous surveys done by that same pollster. Otherwise, what looks like a rise or fall in the numbers could just be the result of a different methodological decision or, especially for non-horse-race questions, the way the question is worded. The order in which questions are asked can matter too; for example, asking a bunch of questions about health care and then asking for whom respondents would vote might bias them to pick the candidate they think is best on health care.

In addition, note who is being polled and what the margin of error is. Polls conducted among likely voters are the best approximation of who will actually cast a ballot, although when you’re still several months away from an election, polls of registered voters are much more common, and that’s fine. For non-electoral public opinion questions, like the president’s approval rating, many polls use a sample that will try to match the demographic profile of all adults in the U.S., and that’s fine, too. As for margin of error … just remember that it exists! For example, if a poll of the 2018 Florida governor race showed former Tallahassee Mayor Andrew Gillum ahead of former Rep. Ron DeSantis 47 percent to 46 percent with a margin of error of plus or minus 4 points, you’d want to keep in mind that DeSantis may actually have been leading at the time. Remember, too, that the margin of error applies to each candidate’s polling number, not to the difference between the candidates. So if both numbers are off by the margin of error, the difference between them could be off by twice as much. In this case, that could mean Gillum dropping to 43 percent and DeSantis jumping up to 50 percent, going from a 1-point deficit to a 7-point lead.

Sample size is important too — a smaller sample means a larger margin of error — but good polling is expensive, so the best pollsters may wind up with smaller samples. And that’s OK. As long as you heed the margin of error, a poll with a sample size of, say, 300 isn’t inherently untrustworthy. That said, don’t dive too much into one poll’s crosstabs — that’s where sample sizes do get unacceptably small and margins of error get unacceptably big. This is one reason not to trust commentators who try to “unskew” a poll by tinkering with its demographic breakdown, or who say that a poll’s results among, say, black voters are unbelievable and therefore the whole poll is too. These people are usually trying to manufacture better results for their side, anyway.

Speaking of which, consider the motive of whoever is sharing the survey. Polls sponsored by a candidate or interest group will probably be overly favorable to their cause. You should be especially suspicious of internal polls that lack details on how they were conducted (e.g., when they were conducted, who was polled, their sample size and their pollster). If you get your news from a partisan outlet, it may also selectively cover only polls that are good for its side. And even the mainstream media might be inclined to overhype a poll as “shocking” or a margin as “razor-thin” because it makes for a better headline.

Next, beware of polls that have drastically different results from all the others. They often turn out to be outliers — although not always (every new trend starts with one poll), which is why you shouldn’t throw them out either. Instead, just use a polling average, which aggregates multiple polls and helps you put the outlier into proper context. We at FiveThirtyEight use averages for that very reason.

And even if a new trend does emerge, wait a bit before declaring it the new normal. Big events — candidate announcements, debates, conventions — can have dramatic effects on the polls, but they are often fleeting.

Finally, come to terms with the fact that polls won’t perfectly predict the final results. Polls are a lot more accurate than people sometimes give them credit for, but polling error is real. Since 1998, polls conducted within a few weeks of the election have missed by an average of 3-10 points, depending on the type of campaign. So trust the polls, but hold onto some uncertainty right up until the moment election results start rolling in.