If you take a data-driven approach to election forecasting, it’s not immediately clear how much you might expect to learn by traveling around the country talking to voters. Polls can reach far more voters (and a far more representative sample of them) than a reporter might by knocking on doors or accosting voters at the shopping center. And if you’re not careful, it can be easy to cherry-pick the anecdotal evidence and read the “vibrations” in a way that confirms your biases.
But last month, I spent a week and a half on the ground in the United Kingdom. I visited a number of locations across the nation as part of an edition of BBC’s “Panorama” program, which will air Monday night in the U.K. I visited shipyards, bingo halls, town squares, quite a few pubs and even a sheep auction. Along with the host of the program, Richard Bacon, I spoke with dozens of voters.
Nothing that I learned changed my forecast of the U.K. election, which will be held May 7. That’s partly because I, personally, don’t have a forecast of the U.K. election. Instead, after a less-than-brilliant performance going at it on our own in 2010, FiveThirtyEight has partnered with the three U.K.-based academics behind electionforecast.co.uk, whose forecast you can find here.
But my visit did help me understand more about why the polls behave as they do in the U.K. and why the three academics behind our forecast — Chris Hanretty, Ben Lauderdale and Nick Vivyan — designed their model as they did. It also helped me understand more about the differences between the U.S. and U.K. election systems — and why this U.K. election is so hard to call.
1. Tactical voting can have a major impact in the U.K. When there are only two major candidates, the choice isn’t very complicated. Which candidate do you like better? Great — go ahead and pick her.
But the U.K. has become less and less of a two-party system. While the Conservatives and Labour collectively accounted for about 90 percent of the vote through the election of 1970, they’ll be down to somewhere in the neighborhood of 65 percent to 70 percent this year.
So voters in the U.K. have two types of tactical concerns that don’t really apply in two-way elections in the U.S.
- First: Does the candidate you prefer have a chance to win locally? If not, your vote might be wasted.
- Second: Will the candidate’s party have any influence in who forms the next government?
There’s a lot of academic evidence for tactical voting. But it also wasn’t hard to find voters who were thinking about these considerations. We met with some left-of-center voters in North Devon, for instance, who might ordinarily prefer to vote for Labour or the Green Party. But because the top two candidates in their constituency are a Liberal Democrat and a Conservative, they said they might vote for the Liberal Democrat instead.
For what it’s worth — maybe not much — the voters we spoke with seemed less likely to be swayed by the second type of tactical consideration: What type of influence is a candidate’s party likely to have in the next government? That’s not because they didn’t care about this — they did — but because they saw the post-May 7 landscape as messy and unpredictable. Better to vote for the candidate you like — provided she has a chance to win locally — and hope for the best after that.
Incidentally, this type of tactical voting can mean the polls themselves have influence on the race. If voters in that Devon constituency saw a poll showing that the Labour candidate was within a few points of the lead, they might vote for her after all. There’s even the potential for a polling surge to become a self-fulfilling prophecy, as can sometimes happen in the early stages of the U.S. presidential primaries.
2. There really are “shy Tories.” The Shy Tory Factor refers to a phenomenon, observed in several U.K. elections in the 1990s, in which Tories (Conservatives) outperformed their polls on election day. One explanation for this is that these voters were less likely to disclose their vote to pollsters, claiming they were undecided instead.
Indeed, we came across several voters who initially told us that they were undecided, only to prove quite ready to declare a candidate preference after gentle questioning. A young entrepreneur we met from an immigrant family in Bury, for example, eventually said he planned to vote Conservative — in contrast to most of his friends and relatives, who were voting for Labour.
But none of the voters we encountered were really all that shy. They were willing to talk about their vote in front of a BBC camera crew, after all. But the polls may be underrepresenting voters who are less enthusiastic and outspoken, like “no” voters in last year’s Scottish referendum, for instance. While there are some “shy” Tories, there are undoubtedly some shy Labour voters and shy Liberal Democrats, too.
Is there any way to account for this in a forecast? Our U.K. academic partners have found that swings from the previous election tend to be overstated in the polls. For instance, if a party that won 10 percent of the vote last time around is polling at 20 percent, they might project it to win more like 17 percent or 18 percent of the vote instead. Some of this — there are other reasons that we’ll get to in a moment — may be because voters who would prefer to preserve the status quo are less enthusiastic and less likely to show up in polls.
3. Incumbency can matter, and its effects won’t always show up in the polls. Another way to put this is that incumbents may overperform their polls on election day. There’s a good reason for this: Most polls in the U.K. don’t mention the names of the candidates. Instead, they’ll ask a “generic ballot” question like this one — “How do you plan to vote, if at all, in the general election that will take place on the 7th May?” — and then provide a list of the parties rather than the candidates.
Even polls of individual constituencies — and there are far more of them this year — sometimes just ask about the parties rather than the candidates. So well-known and well-liked incumbents — like Nick Harvey, the Liberal Democrat MP from the North Devon constituency we visited — may overperform their polls when voters see their names on the ballot.
4. SNP and UKIP voters have more in common than you might assume. We visited one constituency, Boston and Skegness, where the right-wing party, the U.K. Independence Party (UKIP), has a shot to pick up a seat. UKIP has campaigned heavily on anti-immigrant and anti-EU policies, and UKIP voters in Skegness almost invariably brought these issues up when we spoke with them. A few — not all or necessarily most, but some — also made statements that were xenophobic and racist.
These voters have a lot of economic frustrations. The constituency’s per-capita income is among England’s lowest — median earnings were under £17,000 per year (about $26,000) in Boston and Skegness in 2011.
Glasgow, meanwhile, which we also visited, has turned into a hotbed of support for the Scottish National Party (SNP). It voted for independence last year — and Labour is all but certain to lose most of the seats it holds in and around the city, once a Labour stronghold. Glasgow also has low incomes, sluggish economic growth, low life expectancies and — in some places like the Glasgow Central constituency we visited — abject poverty.
SNP voters and UKIP voters wouldn’t see eye to eye on most issues. SNP is explicitly left-wing and supports EU integration. But economic welfare is a powerful determinant of voting in the U.K., as it is in the United States, and they’re both drawing support from economically marginalized areas.
I don’t mean to presume that UKIP voters don’t care about immigration or that SNP voters don’t care about Scottish independence. But voters are complex, and if you talk to enough of them, you’ll find that they have all sorts of concerns that the national parties don’t have much to do with. One elderly voter in Devon, for instance, said he wished the government would do more about loneliness among pensioners, while one UKIP voter in Skegness was frustrated by her inability to find child care for her developmentally disabled son. Voters may choose their parties — influenced by personal circumstances, economic conditions and other things — and then adopt their policy positions later.1
5. The centrism of the major parties may have enabled the SNP surge. In North Devon, the Lib Dem stronghold, I asked every voter I met to explain the principal philosophical and policy differences between Labour and the Liberal Democrats. From my point of view as a “dumb American,” it’s hard to find all that many policies on which they really disagree.
Most voters instead responded with descriptions of the geographic and demographic differences between the parties — like that Liberal Democrats have traditionally been rural, for example, and that the party has strong historical roots in Methodism. They also seemed to agree that the differences had become harder to articulate since Labour became “New Labour” and moved to the center under Tony Blair — while Conservatives, simultaneously, have de-emphasized divisive cultural issues.2
This centrism stands in contrast to the U.S., where the parties have become more and more polarized by pretty much every available measure. By contrast, in the United Kingdom, voters frustrated with their personal situations and with the state of their communities may need to move to one of the wings to reject the status quo. That can manifest itself in a left-wing resurgence in some communities and a right-wing one in others.
What makes SNP unique, however, is that its vote is concentrated entirely in — of course — Scotland. So even though SNP may win no more than perhaps 4 percent or 5 percent of the vote throughout Great Britain3 — far less than UKIP and perhaps even less than the Green Party — it will translate into the party winning most, perhaps even almost all, of the 59 seats in Scotland.4
6. Some people still think I’m a “magic nerd.” I gained a lot of notoriety in the 2008 and 2012 U.S. presidential elections by forecasting that Barack Obama was quite likely to win even though many media accounts portrayed the races as being too close to call.
I’m glad my predictions turned out right. But partly as a result, I’m now sometimes known as the guy who can make highly confident and precise predictions in the face of what everyone says is impossible uncertainty.
The irony is that U.S. elections are really the exception and not the rule. Because of a fairly unique set of circumstances — the pathologies of the American political media on the one hand, and the simplicity of the two-party system on the other hand — it’s often prudent to bet fairly heavily on the favorite.
But far more often, as I describe in the book, experts are too confident in their predictions. (This tends to hold whether they use statistical systems to make forecasts or just wing it instead.)
And consider the challenges this year’s U.K. election presents:
- There’s still the possibility of a late polling swing, especially given the multi-party nature of the contest.
- The polls could be off on election day itself — because of “shy Tories” or for other reasons. Since 1979, the final U.K. polls have missed the spread between the Labour and Conservative vote by an average of 4 percentage points.
- Although we’re getting some help from local constituency polls, it’s still quite challenging to know how the vote share will translate into the number of seats.
- And most importantly, it appears highly unlikely that any party will receive an outright majority on May 7. And the aftermath could be extremely messy. It’s possible that the Conservatives will win the most seats but that Labour will seek to form a government with the consent of SNP, for instance.
We do know a few things, of course. SNP is going to gain a ton of seats. The Liberal Democrats will lose some. UKIP isn’t likely to win as many as the party had hoped. There probably won’t be a majority for any party.
And it wouldn’t take that much of a late polling shift to make sorting out the election aftermath easier. If the Conservatives beat their current polls by a couple of percentage points, they’ll be close enough to a majority that they could form another coalition with the Liberal Democrats. If Labour beats their polls by the same amount, they’ll have the plurality of seats, and Ed Miliband should have lots of options for how to form a government.
But what if the polls are exactly right? Who’s going to be the next prime minister? Sometimes you can gather all the data you like and the best forecast is ¯\_(ã)_/¯.
Check out our 2015 general election predictions and full U.K. election coverage.