National polls certainly picked the right winner. Every poll released in the month before the election found Biden leading — and he easily won the popular vote.
Averaged together — always the best way to look at the data — state polls also predicted the presidential winner in every state but two. In Georgia, the averaged polls gave President Trump a slim one-point lead but voters preferred Biden by an even smaller margin. In Florida, polling suggested a Biden advantage of less than one point, but Trump won by more that 3 percent.
Senate polling produced one giant miss and two smaller ones. Every Maine poll showed Democrat Sara Gideon ahead, though incumbent Sen. Susan Collins (R) prevailed by a handsome margin. Surveys in North Carolina and in one of the Georgia seats put Democrats very narrowly ahead. Voters in both states cast a few more ballots for the Republicans.
Of course, projecting the winner correctly is an important — but pretty low — bar for measuring poll accuracy.
Even horse-race polls should give us more subtle and nuanced information than simply who will win.
In battleground states, there were some big misses on the projected magnitude of the two presidential candidates’ leads, almost all under-accounting for Trump’s support. Polls predicted a tiny one-point Trump margin in Ohio. He won the Buckeye State by more than eight. Polls suggested Biden would prevail in Wisconsin by nearly seven points; in reality, his margin was less than one.
Those misses weren’t statistically random, though. On average, the polls misjudged Biden’s support by a meager one point, while crucially undercounting (or under-predicting) Trump’s support by more than three points.
Those facts are consistent with at least two theories: that “
shy Trump voters” were reluctant to tell pollsters they backed the president; or that late undecideds broke to Trump.
A look at the same data for battleground Senate polls undermines the first explanation.
Leaving aside Maine, which obviously suffered from unique problems, Senate surveys accurately pegged the vote for Democrats, but consistently underestimated support for Republicans.
On average, public Senate polls came within a point of the vote for Democratic candidates but missed support for Republicans by almost four times as much and, again, in every case understated GOP support.
With support for every GOP candidate in a competitive Senate race underestimated, it’s hard to sustain the shy-Trump-voter theory. Something more systematic is happening. It does appear that it’s more difficult to get Republicans to participate in polls in numbers equal to their strength.
However, the notion that late undecideds broke for the GOP receives some support from the exit polls. Late deciders were far more plentiful in 2016, but in that year, they split about evenly. In this election, they delivered
lopsided margins to Trump.
Decades ago, pollsters’ inside secret of was that undecideds broke overwhelmingly to the challenger.
One of my first races pitted an unknown against a House incumbent in Connecticut. Six weeks out, we were behind by 48 percent to 25 percent. I cautioned the candidate to ignore the margin, noting that most of those undecideds would come our way. And in the end, we won.
But what was once a law of polling no longer rules.
For the last 20 years, we’ve gone through election cycles where undecideds broke evenly and where they broke to incumbents. This year, as in some past elections, undecideds went overwhelmingly to Republicans.
Human brains and eyes naturally gravitate to margins — because margins quickly summarize the data. But like all summaries, they can obscure, or ignore, important information.
There’s a difference between a contest where a candidate leads 53 percent to 45 percent, with 2 percent undecided, and one where the lead is 46 percent to 38 percent with 16 percent undecided. The eight-point margin is the same, but the contours of the race, and the outcomes, could well be quite different.
Focusing on margins while failing to understand the role of undecideds is not an error in the poll, it’s a mistake in analysis and interpretation. So is relying on any one poll instead of an average.
The lesson of this cycle is not the demise of polling, but the need to include all the data and perhaps most importantly, to devise a way to predict, in advance, how the undecideds will fall.
For decades, I’ve searched in vain for this great white whale. I hope someone finally reels it in.
Polling isn’t broken. But we too often miss its hidden signals.
Mark Mellman is president of The Mellman Group, a polling and consulting firm.
National polls certainly picked the right winner. Every poll released in the month before the election found Biden leading — and he easily won the popular vote.
Leaving aside Maine, which obviously suffered from unique problems, Senate surveys accurately pegged the vote for Democrats, but consistently underestimated support for Republicans.
On average, public Senate polls came within a point of the vote for Democratic candidates but missed support for Republicans by almost four times as much and, again, in every case understated GOP support.
Decades ago, pollsters’ inside secret of was that undecideds broke overwhelmingly to the challenger.
One of my first races pitted an unknown against a House incumbent in Connecticut. Six weeks out, we were behind by 48 percent to 25 percent. I cautioned the candidate to ignore the margin, noting that most of those undecideds would come our way. And in the end, we won.
Human brains and eyes naturally gravitate to margins — because margins quickly summarize the data. But like all summaries, they can obscure, or ignore, important information.
There’s a difference between a contest where a candidate leads 53 percent to 45 percent, with 2 percent undecided, and one where the lead is 46 percent to 38 percent with 16 percent undecided. The eight-point margin is the same, but the contours of the race, and the outcomes, could well be quite different.