Many are looking at Scotland’s independence referendum as a referendum on poll accuracy as well. And the general consensus is that polls failed the test.
One analyst called it “a loss not only for the pro-independence movement … but also for the pollsters.” A Guardian column blamed “systematic error across the models the pollsters used.”
The evidence motivating all the hand-wringing: Polls showed the race close, with some even having “yes” ahead. But in the end, “no” won by 10 points.
Having done no polls of our own, I had no dog in the fight, but I think the critique is overdrawn.
More than 70 polls were conducted on this issue during 2014; just two had “yes” ahead — hardly abject failure. California’s Proposition 19, legalizing marijuana, lost by 7 points. There, 24 polls were conducted, and half of them had the “yes” side ahead.
We don’t yet know the outcome of California’s Proposition 46 on medical malpractice — it’s on the November ballot. But two public polls have been done by the Field Institute, within two months of each other. One had “yes” ahead by 28 points, the other had “yes” behind by 3. More than a dozen polls were made public on Proposition 8, the anti-gay-marriage measure, which won by 5 points; only four of them had the correct side winning (including ours).
So the polls in Scotland were more consistently likely to predict the winner than surveys in many other races.
Indeed, the coverage was more misleading than the polls, taken together. Almost all the breathless discussion of impending victory for Scottish independence was occasioned by the only two polls during September that showed “yes” ahead. Sixteen others had “no” leading.
It’s no surprise the commentariat jumped on the outliers. “Scotland votes for independence, breaking up the United Kingdom, 307 years after it joined” is a more compelling story than “nothing will change in Britain.” However, the attention lavished on the former story line was wildly out of proportion to its likelihood, given all of the available data.
Adding to the intensity of the critique was an annoying habit of the British press that conveys a sense of unjustified finality. By reporting results after excluding undecideds, the polls in Scotland make the results seem more resistant to change than they are.
It’s frankly misleading to report a poll that finds 42 percent voting “no” and 40 percent “yes” with 18 percent undecided as 52 percent “no” to 48 percent “yes.” The actual numbers point to a fluid situation liable to change, whereas the reportage suggests a slightly greater margin but much less room for movement. Undecideds are in principle different from those who say their minds are made up. Excluding them assumes they will cast ballots in the same way as those who have already determined how they will vote — an assumption not justified by any analysis of the data itself.
Finally, any reasonable examination must allow for the possibility of change. I’m frustrated by those who criticize polls three or four weeks out from election day for being “wrong.” In that period, at least one side, and often both, are laboring intensively and spending effusively to make the polls wrong, that is, to change the outcome from what it would have been to something that is better for them.
Expecting a poll weeks or months out to reflect the final outcome is to expect the campaign to make no difference. That might be true much of the time, but it is hardly a foregone conclusion.
YouGov re-interviewed its earlier respondents on election day in Scotland and found that some had in fact changed their minds, moving toward a “no” vote in response to a vast, multilayered “no” campaign.
Independence lost, but the polls, properly interpreted, did pretty well.
Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the Majority Leader of the Senate and the Democratic Whip in the House.