Nov

05

2012

## The Polls Tell Us Less than We Think

I have a bold prediction for Tuesday: almost all the presidential polls out there (and there are LOTS of them) will prove to be accurate. And yet at this moment, no one really knows who will win.

The factor in these polls that most people forget about, or fail to understand, is the statistical concept “margin of error.” Most Americans give too much credence to surveys (64% of likely voters know that). It’s not that the polls are rigged (at least not most of them). The polls are often found to be “inaccurate” because we expect them to be more “accurate” than they claim to be. Every presidential poll has a margin of error (MoE), which is the pollster’s way of saying “here’s the statistical range we feel confident about.” But if we ignore the MoE, or misapply it, we will not know what the poll is really saying.

For example, let’s look at the latest presidential polls taken in Michigan:

Baydoun/Foster – 1913 LV, 2.2 MoE, Obama 46, Romney 47

PPP – 700 LV, 3.7 MoE, Obama 52, Romney 46

Rasmussen – 750 LV, 4.0 MoE, Obama 52, Romney 47

Detroit News – 600 LV, 4.0 MoE, Obama 48, Romney 45

EPIC-MRA – 600 LV, 4.0 MoE, Obama 48, Romney 42

The margin of error indicates the range (plus or minus the given number) that will be true 95% of the time. The 95% figure comes into play, though it rarely gets mentioned, because it is the standard confidence interval. It means, in the Detroit News poll for example, that if they ran the same survey 100 times, in 95 of those times Obama would get 48% of the voter preference, plus or minus 4 points. This indicates the range of percentages about which the pollster feels confident. The Detroit News is confident (or at least was when the poll was taken) that somewhere between 44% and 52% of the likely voters prefer Obama. Or to put it another way, they think 19 out of 20 polls (i.e., the 95% confidence interval) will have Obama’s percentage between 44-52 and Romney’s between 41-49.

Make sense? I hope so because I want to make a few other important observations about MoE.

1. The margin of error does not refer to the gap between two candidates in the polls but to the polling numbers for each candidate. The +/- range is for the percentages themselves, not for the difference between the two percentages. So if Romney leads Obama by 6 percentage points in Florida (51-45 in a recent poll) and the margin of error is plus or minus 3.5, the poll does not give Romney a decisive edge. True, it’s better to have the polls say you are ahead than behind, but what the poll indicates is that 95% of the time this survey should show Romney with 47.5-54.5 percent of the vote and Obama with 41.5-48.5 percent of the vote. The MoE refers to the percentages, not to the gap. So according to this Florida poll Obama might squeak out a 1 point win in Florida or Romney could cruise to a double digit victory.

2. Remember that the MoE has a confidence interval of 95 percent. This means 1 out 20 times you are going to get a strange outlier. If you’ve seen a wild poll or two in this election season that’s way out there compared to almost every other poll, that’s what pollsters expect. Five percent of the time the poll will be really different, not necessarily wrong, just really different.

3. Because of the MoE, polls tell us less than we think, even though candidates want to make more of them than they should. Look again at the Michigan polls above. The top one is the most recent and the bottom one less recent (in this case, only a week earlier). The Romney camp could say, “Look, we have the momentum in Michigan!” On the other hand, the Obama camp could say, “Look, we’ve been ahead in this state for months and every other poll is in our favor!” In reality, the polls say that the state is too close to call. In none of these polls is Obama’s range without overlap with Romney’s range. Obama fans should not think they have Michigan in the bag. Everyone of these polls says that Romney could actually be ahead of Obama. And yet, Romney fans should simmer down too. The latest poll with Romney up by 1 point is exactly what every other poll said was possible 95 times out of 100. It may not indicate momentum at all.

All this is to suggest that in this season’s plethora of polls pundits have been too quick to make big conclusions and rewrite campaigns narratives based on a recent poll which, given the margin of error, may say pretty much the same thing as the last poll.

The national polling indicates a dead heat, and in 10 or 12 battleground states the polls are too close to call. You can make a case that Obama will win handily, that Romney will make a good showing in several blue states but hasn’t broken through in the polls and will fall short in almost all of them. You can also make the case for a big Romney win, that last minute voters will break for the the challenger and Obama will lose almost every state where his numbers have been below 50 percent.

And it could be as close as Bush-Gore in 2000. Tuesday will tell.

Unless it really is Bush-Gore all over again.

## 10 Comments

I love Monday morning humor!

McAlvey have any input into this? Not bad having him on your team.

[...] The Polls Tell us Less Than We Think - Kevin DeYoung writes a helpful article about what the polls tell us. If you haven’t figured it out already, the polls are just too close to call. [...]

Pollsters also get to decide (guess) the ratio of democratic to republican voters on election day, by looking at 08 participation, etc. this often is where pollsters can steer the results by predicting one party participation over the other, and how some are predicting a Romney landslide while others are calling it a squeaker. It’s finally almost over!

And here I thought you were going to say that the big factor the polls don’t take into account is cheating.

In all seriousness, that could do it for Obama.

Is this the Monday Morning Humor entry?

I don’t believe the pollsters weight the results according to a set ration of D’s and R’s. The do weight the results according to other factors like race and age. In other words, most of these polls try to take into account what sort of people tend to vote and in what percentages. But the partisan breakdown is simply a reflection of what people tell them in a given survey. It doesn’t indicate bias, but it can indicate that the poll is not very representative.

And here I thought you were going to say that the big factor the polls don’t take into account is cheating.In all seriousness, that could do it for Obama.Do you have any support for this, or are you just making it up?

Kevin,

Unfortunately, you don’t know quite as much as you think, statistics-wise. The margin of error and confidence level is in the method itself, not in the particular result we get from the method. For instance,

It means, in the Detroit News poll for example, that if they ran the same survey 100 times,

in 95 of those times Obama would get 48% of the voter preference, plus or minus 4 points.

is incorrect. The correct conclusion would be that if the Detroit News repeated the same survey 100 times, 95 of the results would be within the margin of error of the true (unknown) population proportion that support Obama. The difficulty is that we don’t know if the one poll we have (48%) is one of the 19/20 that is within the margin of +/- 4% of the true value or of the 1/20 that isn’t.

So indeed, we do know even less than we thought we knew!

The five thirty eight blog is an excellent source of information, and reading along is a nice introduction to the methods and models and used in polling. At the end of the day, the best we get are statistical probabilities and likely outcomes, not definitive results.

http://fivethirtyeight.blogs.nytimes.com

@Kevin, well put!

“

So indeed, we do know even less than we thought we knew!/“