Does Donald Trump lead the race for the Republican presidential nomination? Or does he lead it by a “yuuuuuge” margin? Bloomberg recently released a poll that gave Trump 24 percent to Ben Carson’s 20 percent. On the same day, Ipsos released a poll that put Trump at 37 percent to Carson’s 14 percent. Normally, I’d suggest you average the results and move on. But these two polls are emblematic of a deep divide this year first noted by Jonathan Robinson: The Bloomberg poll was conducted over the phone with live interviewers; the Ipsos poll was conducted online.
The problem
Trump has averaged 23.4 percent in live-interview polls since entering the race in mid-June. That’s 5.9 percentage points lower than his standing according to automated phone polls (29.3 percent) and 5.7 percentage points lower than his support in Internet polls (29.1 percent). Here’s the gap since Trump entered the race (to simplify things, I’ve combined automated phone and Internet polls):
And the divide isn’t confined to national polls. Trump is also doing better in Iowa and New Hampshire in non-live-interview polls.
Trump has averaged 24.5 percent and 33.2 percent in non-live polls in Iowa and New Hampshire, respectively. In live-interview polls in those same states, he has averaged 20.4 percent and 22.3 percent. In surveys released the same November weekend, Suffolk University found Trump’s support at 22 percent in New Hampshire, while YouGov released a poll pegging him at 32 percent. Imagine if this gap holds up through early February: It’s possible that we’ll head into Iowa and New Hampshire with little idea of what’s going to happen.
And it’s not just a matter of good pollsters versus bad pollsters: Every online pollster has averaged more Trump-friendly results than the live-interviewer average. NBC News, SurveyMonkey and The Wall Street Journal all typically conduct top-notch polls. SurveyMonkey, for example, was one of the few pollsters to do well in the 2014 U.S. midterms and 2015 United Kingdom elections, and the NBC News/Wall Street Journal poll earns an A- in FiveThirtyEight’s pollster ratings. Yet, Trump has averaged 26.8 percent in the NBC News/SurveyMonkey surveys1 conducted online and 22 percent in the NBC News/Wall Street Journal polls done live by phone. In many instances, they have produced different results within days of each other.
In previous years, most analysts and reporters probably would have put considerably more weight on the live-interview polls, or just ignored the automated and Internet polls altogether. That’s because most (though not all) analysts thought live-interview polls, in which each member of the electorate with a landline phone has an equal chance of being selected, were more accurate than Internet polls, most of which rely — at least somewhat — on volunteer respondents who might not be fully representative of the population, and automated polls, which cannot reach cellphones.
But this year, it’s considerably more difficult to ignore non-live-interview polls. Many more polls are being conducted over the Internet, as Nate Cohn of The New York Times has pointed out. In 2011 and 2012, 65 percent of national Republican primary polls were done by live interview, compared with 21 percent online and 14 percent automated. Since Trump entered the race, just 37 percent of national primary polls have been conducted by live interviewers, while 50 percent have been conducted over the Internet and 13 percent by automated phone call.
Moreover, media outlets are increasingly willing to sponsor Internet polls. For example, NBC News has partnered with SurveyMonkey,2 and The New York Times has partnered with YouGov. The New York Times and NBC News previously refused to report on Internet-based polls in which at least some respondents volunteered to be in the sample.3 They are part of a new age in which media companies sponsor both live-interview and Internet surveys, in part because of the cost of live-interview polls and in part because online polls have been fairly accurate in general elections.
Now that the media treats Internet polls basically like any other poll, online pollsters that produce weekly results such as Ipsos and Morning Consult (which show very good results for Trump) frequently dominate the news cycle. This year, no live-interview pollsters are producing weekly results, so the type of poll that favors Trump is in the news more often.
Why we can’t be sure who is right
Who might have a better handle on Trump? It’s difficult to say because there are pollsters I trust on both sides of the divide. There are a lot of smart people at SurveyMonkey and YouGov. SurveyMonkey, especially, has done ridiculously well recently. We can’t test anything until the voting starts, so at this point, it’s probably best to take both live-interview and non-live-interview polls into consideration.
But if forced, I’d go with live-interview polls
Here’s why:
1. Internet polling is mostly untested in primaries. When we get close to the actual voting, you’ll hear people like me advise you to trust the polling (imperfect as it is) over metrics like lawn signs or crowd size. That belief stems from decades of electoral history during which live-interview polling has given us a good, though imprecise, idea of who is going to win. Internet polling has done fairly well in general elections, but it has barely been tested in presidential primaries. The only consistent Internet-based primary pollsters in 2012 were YouGov and Zogby Interactive, and they didn’t survey state primaries with any regularity. This isn’t just stodginess about the old ways being better; the Bayesian “trust the polls” prior for Internet surveys isn’t as strong as it is for live-interview polls.
2. We saw gaps between live-interview and non-live-interview polls in 2012, and the live-interview polls tended to be more predictive. Over the course of 2011, we saw many anti-establishment challengers to Mitt Romney rise in the polls before voting started. Remember Michele Bachmann, Herman Cain, Newt Gingrich and Rick Perry? They each had their moment as the main opponent to Romney. As Steve Koczela first pointed out, any time one of them got close to Romney, he or she tended to do better in the non-live-interview polls than the live-interview polls.
According to the HuffPost Pollster aggregate of national surveys, they averaged 3.7 percentage points higher at their peak in non-live-interview polls than in live-interview polls. Romney, in contrast, did better in 2011 in live-interview surveys (21.6 percent) than in non-live-interview polls (18.3 percent). In Internet polls specifically, 16.7 percent of respondents supported Romney in 2011.
Joshua Clinton and Steven Rogers investigated state polling and found that automated polls tended to be far less accurate than live-interview polls further out from a given primary in 2012. They also found signs that automated polls herded toward live-interview polls as the primaries approached. Clinton and Rogers didn’t test Internet polls, but it’s worrisome that the Internet polls this year more closely match the automated polls, which have been less accurate in the past.
3. Internet polls may be picking up nonvoters. One of the biggest fears of using Internet polls is that they’re not representative of the electorate. You can weight to get rid of some disparities in a volunteer sample — if you have too few elderly respondents, for example, then the elderly respondents you do have get more weight in the average. But weighting may not be enough; the type of older Republican who signs up for an Internet poll may differ politically from the older Republican who answers only a telephone poll. Although this isn’t a big deal in general elections in which party identification tends to determine vote choice, it’s unclear how well Internet polls that weight by party identification will do in the primary.4
A recent YouGov poll sponsored by the University of Massachusetts confirmed these fears. UMass decided to match the YouGov likely voters to the list of active registered voters to see whether they could be confirmed as registered to vote. Among confirmed voters, Trump did 3 percentage points worse. Meanwhile, the other candidates did, on net, 10 percentage points better. Thus, even though there were fewer undecided respondents among those confirmed as registered to vote — so, all else being equal, every candidate’s vote share should have increased — Trump lost ground.
Automated phone polls, of course, can call actual voters through a list sample. They can’t, however, call cellphones (it’s the law). In order to make up for this deficiency, pollsters who perform them often use Internet panels, which themselves may not be representative.
4. Arguments against the accuracy of live-interview polls don’t make much sense. Trump fans like to point out that he leads in polls with larger sample sizes. Since Trump entered the race, the average national live-interview poll has had only 434 respondents, compared with 732 for non-live-interview polls. That makes those leads sound convincing, right? Not really. The theoretical margin of error for a sample size of 732 is only 1.1 percentage points smaller than for a sample size of 434. In other words, differing sample sizes don’t come anywhere close to explaining the difference in Trump’s support between live-interview and other polls. And although it’s possible that declining response rates are affecting the accuracy of telephone polls this year, that hasn’t been a problem in the past. The Pew Research Center studied the topic in 2012, when it calculated the response rate to its surveys to be just 9 percent and found “no significant differences between the surveys on party identification, leaned party identification or political ideology” in surveys with lower response rates.
What about the idea that people don’t want to tell pollsters they support Trump for fear of being thought politically incorrect? Jonathan Robinson floated this idea when he first noticed the gap between live-interview and non-live-interview polls. I have a number of problems with this theory.
First, since when are Trump supporters afraid to admit they like Trump? Much of the reason they like Trump in the first place is because he “tells it like it is,” and his supporters often let people know it. Now, all of a sudden, they’re getting shy with pollsters?
Second, if Trump voters were afraid to admit that they were behind him, we’d expect them to say they were undecided (see the Bradley effect). But, on average, 11.2 percent of voters are undecided in live-interview national Republican primary polls and 7.4 percent in non-live-interview national primary polls. Even if we assigned all these additional undecided Republicans to Trump (which probably isn’t right), this 3.8 percentage point gap doesn’t account for the 5.8 percentage point difference in Trump support between live-interview and non-live-interview surveys.
Third, as I pointed out above, the gap between live- and non-live polls also occurred in 2012, when Trump didn’t run.
All that said
Trump leads national primary polls among Republicans any way you cut it. He has already lasted much longer than a lot of people (including me) thought he would. It wouldn’t be shocking to see him break the old rules when it comes to polling as well. All we can say for certain right now is that Trump’s standing in live-interview and non-live-interview polls differs significantly. We’ll have to wait to see which is more accurate, or if the gap shrinks the closer we get to the primary.
Dhrumil Mehta contributing research