Margin of error is simultaneously the most under,- and most over-used concept in polling, in part because so many people don’t really understand it.
A Democratic gubernatorial primary survey in Maryland provides an opportunity to re-engage with these issues.
Between May 29 and June 3, the great pollsters at The Washington Post, together with the University of Maryland, interviewed 532 likely Democratic primary voters and found former NAACP President Ben Jealous with 21 percent of the vote, followed by Prince George’s County Executive Rushern Baker at 16 percent.
Five other candidates split 24 percent of voters, while 39 percent were undecided.
Trying to give the results some meaning, the paper noted on a chart that the “difference between the candidates is 5 points; a difference of 7.5 points is needed to be statistically significant.”
The article reporting on the poll stated flatly, Mr. Jealous’s “margin is not statistically significant.”
This language might cause a reasonable reader to believe that Jealous’s lead was meaningless, or at least not very meaningful.
That interpretation would be reinforced by the following communique from the pollsters, “Our standard for a lead being statistically significant is 95% confidence, the traditional gauge. We call advantages that are significant at 90-95% ‘slight.”
If Mr. Jealous prevails by 5 points on election night, few will report it was a “slight” victory.
So how is his 5-point poll lead transformed into “insignificance?”
To be clear, nothing’s inaccurate in the Post’s claims.
Like most other discussions of these issues by pollsters, they are just misleading.
You might assume categorizing findings as “significant” or “insignificant” results from careful, scientific assessment. It doesn’t. It was an arbitrary decision made by a lone statistician in the 1920s, who pronounced that if, in this case, there’s a 95 percent chance that Jealous leads, it’s “statistically significant,” but if the chances are only 93 percent, it’s not significant.
“Significance,” as used in statistics, is completely unrelated to the dictionary definition of the word — importance — a great cause of confusion.
Let me give you the other side of the statistical coin. It is also true that, according to this poll, there is nearly an 88 percent chance that Jealous leads.
That’s not unimportant, or inconsequential, or even “slight.”
If you had the opportunity to place a bet you would win 88 percent of time, would you do it? I think so. In fact, I think you’d make that bet every single time — and be richer for it.
This isn’t just some idiosyncratic Mellmanism.
In 2016, The American Statistical Association released a statement approved by dozens of leading statisticians, urging that “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold [e.g. 95%] …. A conclusion does not become ‘true’ on one side of the divide and ‘false’ on the other.”
Someone may well respond with the latest survey, by The Baltimore Sun and the University of Baltimore taken May 29 to June 6 which found Jealous and Baker tied.
“See,” they could retort, “another poll shows them tied. Jealous’s lead wasn’t real after all.”
And it may not be. Or it may be that the horserace changed in those few days, or that one pollster or the other did a better job of not violating the randomness assumptions on which all these statistics are based.
However, if you put the two different polls together, the numbers indicate a 79% chance of a Jealous lead.
Will Jealous win on Election Day?
Polls today, especially with 4 of 10 voters undecided, can’t predict that. Opponents will be fighting every day to chip away at Jealous’s likely lead.
But the former NAACP president probably led last week.
Mellman is president of The Mellman Group and has helped elect 30 U.S. senators, 12 governors and dozens of House members. Mellman served as pollster to Senate Democratic leaders for over 20 years and as president of the American Association of Political Consultants.