Just how likely is it that the polls will be right?
Notice first, I said, polls, plural, with an “s.”
There is no particular reason any one poll should be “right” — except ours, of course. That’s what the statistical margin of error is all about.
The results should fit under a classic bell curve, with most congregating near the correct answer but a number a little off, and a smaller number further off.
In other words, in a race that really sports a 4-point margin you would expect to see a lot of the polls concentrated in the 3- to 5-point range, fewer with margins of 2 or 6, but some showing a 1- or 9-point race, and maybe even a couple minus-1 or plus-10, or even minus-2 and plus-11.
These are just statistical facts of life. Nothing can really be done to obviate them.
Of course, statistical sampling is just one source of survey error. We’ve explored many others over the years: failure to contact hard-to-reach voters, question order effects, problematic interviewers or bad interviewing devices, failure to determine who is really going to vote and who isn’t, undecideds deciding, “decided” voters switching at the last moment, and many others.
Thus, in many years the range of poll results can be fairly wide. In 1984, when Ronald Reagan won reelection by an 18-point margin, polls in the final week pegged it anywhere from 10 to 25 points.
In 2004, when George W. Bush won by 5, polls conducted the last week said he was anywhere from 2 points down to 6 points ahead.
Averaging is an old statistical approach to reducing the margin of sampling error and allowing random nonsampling errors to cancel each other out or be minimized.
We’ve been studying poll averages for over a dozen years, and public aggregators like RealClearPolitics and FiveThirtyEight advanced the effort.
So how good are these averages at foretelling the ultimate Election Day results?
Pretty damn good, actually.
FiveThirtyEight did some of the leg work, and I did some computations.
Since 1976, on average, the average of the polls in the final week of the campaign is just 2.1 points off the actual margin — damn close.
Indeed, the averages were quite close in the years I cited above, when the range of results was wide. In 1984, the poll average missed the actual results by just 1 point, and in 2004 the average was less than 1 point away from the final mark.
Poll averages were furthest off in 1980, when they predicted Reagan would beat Jimmy Carter by 2.5 percentage points. Reagan actually won by 9.74 — an error of 7.24 points.
It was a complicated year, as independent John Anderson took nearly 7 percent.
More important, that final week’s average was based on just 4 polls.
The more polls, the more reliable the average should be, and, indeed, it is.
In years with nine or more polls during that final week, the average error shrinks to just 1.5 percentage points.
There is little fodder in the facts for those who claim polls are becoming less accurate.
Make no mistake, polling is hard, and good polling, these days, is even harder. But there’s no evidence it’s getting worse.
In 2004 and 2008, the final-week poll averages were less than 1 point away from the actual margin.
In 2012, they were 2.7 points off the mark — a little wider but still quite good.
Moreover, the averages were farther off in 1980, 1996 and 2000 than in 2012. No evidence of a trend toward greater error.
So, what do the averages suggest about election 2016?
We don’t have the final week’s polls yet. But in the polls completed in the penultimate week, Hillary Clinton leads by an average of a little over 3 points.
By next week we’ll know.
Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the minority leader of the Senate and the Democratic whip in the House.