Explanation for Wildly Divergent Polls (e. g. after first presidential debate)

[mshan]

http://video.pbs.org/video/2290469964

Specific comments start around 7:20 mark, but whole extended clip is also good (5 minute mark and 6 minute mark are also good starting points to get interesting tidbits on how uncertain polling can be).


Rassmussen: http://electoral-vote.com/evp2012/Info/rasmussen.html

Gallup seems to be other big outlier this cycle.

"Perhaps 90 or 95 percent of the time, taking a simple average of the polls will work just about as well as the more complicated FiveThirtyEight method. But this is rare instance where taking all the polls at face value may be a mistake, and the additional checks-and-balances the FiveThirtyEight method applies are worth the trouble."

http://forums.anandtech.com/showthread.php?t=2266842

 

[DominionSeraph]

Thread needs poll.

 

[Balt]

I think Nate and the 538 methodology will either be validated or embarrassed in this election. The 2008 election was relatively easy to call for many reasons. The 2012 election will be far more difficult.

Personally, I think he will be validated. By November 7th, we will know for sure.

 

[Farang]

I think Nate and the 538 methodology will either be validated or embarrassed in this election. The 2008 election was relatively easy to call for many reasons. The 2012 election will be far more difficult.

Personally, I think he will be validated. By November 7th, we will know for sure.

I think chances are slim that he will be embarrassed. It seems to me his model is most useful as a prediction model through the course of a cycle (i.e. tamping down convention bounces). Most of the stuff he writes about pertains to explaining trends (or lack thereof) in data.

The day before the election, however, his model seems to mostly be a poll aggregator that weighs various polls based on past performance. Those usually do pretty well, especially the more data you give them. i.e. if Romney won it wouldn't be embarrassing that he said in September Obama had a 85% chance, as that may well have been true at the time.

 

[Rainsford]

I think Nate and the 538 methodology will either be validated or embarrassed in this election. The 2008 election was relatively easy to call for many reasons. The 2012 election will be far more difficult.

Personally, I think he will be validated. By November 7th, we will know for sure.

Certainly you wouldn't have been taking a big risk to just guess that Obama would beat McCain in 2008...but calling states right is more difficult. 538 called 49 of the 50 states right (the site gave Indiana to McCain) and missed an electoral vote in Nebraska around Omaha that Obama also won. 538 also got all the Senate races right. The 2008 election may have been relatively easy to call just in terms of who would be President, but calling it that accurately seems pretty good to me.

 

[Rainsford]

I think chances are slim that he will be embarrassed. It seems to me his model is most useful as a prediction model through the course of a cycle (i.e. tamping down convention bounces). Most of the stuff he writes about pertains to explaining trends (or lack thereof) in data.

The day before the election, however, his model seems to mostly be a poll aggregator that weighs various polls based on past performance. Those usually do pretty well, especially the more data you give them. i.e. if Romney won it wouldn't be embarrassing that he said in September Obama had a 85% chance, as that may well have been true at the time.

I would agree, although I suspect at least some folks won't see it that way. The idea of having a theory based on currently available information and changing it as new information becomes available seems to be viewed as an admission of intellectual defeat by some people.

 

[buckshot24]

I think Nate and the 538 methodology will either be validated or embarrassed in this election. The 2008 election was relatively easy to call for many reasons. The 2012 election will be far more difficult.

Personally, I think he will be validated. By November 7th, we will know for sure.

His post today states that the prediction could be wildly volatile during the last stages of an election. He could be right on the money come election day. But really, assuming that there is a 33% chance of a Romney victory, if Romney wins it isn't anything to be ashamed about.

 

[buckshot24]

I would agree, although I suspect at least some folks won't see it that way. The idea of having a theory based on currently available information and changing it as new information becomes available seems to be viewed as an admission of intellectual defeat by some people.

I was sort of thinking the same thing. What good was the forecast in May now? It isn't going to matter what the prediction was back then.

 

[Rainsford]

I was sort of thinking the same thing. What good was the forecast in May now? It isn't going to matter what the prediction was back then.

Forecasts definitely don't have much use in terms of predictions when you look back like that, that's for sure. It's like the weather forecast for today from a week ago.

On the other hand, it is kind of interesting to look at what caused the predictions to change over time or what makes them different from the actual results. The best example I can come up with this time around (so far) is what the historical predictions show about the impact of the first debate. There was a pretty clear upward trend for Obama that stopped pretty much immediately after the first debate and started rapidly shifting the race towards Romney. Whatever the final election result, looking back at the forecasts pre and post-debate definitely helps define the impact of Romney and Obama's respective first debate performances.

 

[mshan]

"The post-debate-1 correction was a nearly five-point swing. It was complete within one day. This means that the post-debate-day media meltdown could not have caused the swing – though it certainly helped cement perceptions. This crash is likely to be caused in part by changed morale on both sides: hope among Romney’s supporters and despair among Obama supporters, and a consequent change in whether they meet the criteria for being a “likely voter.” The other component would be the possibility that minds were changed by suddenly-moderate-Mitt and stumbling-Barack."

(Nate Silver and Sam Wang were both on NPR Science Friday two days ago. IIRC, one of them said that Democratic registrations were up 3 or 4% above Republican registrations in swing states. One reader comment to a post-first debate article at PEC political science professor type noted that pollsters had suddenly changed methodogy* (to a more inaccurate previous methodology, or were just plain not revealing methodology now, in the immediate aftermath of first debate, when 3 heavily Republican leaning pollsters suddenly all came out with polls way before anyone else)

"But here is something interesting. National polls do not match the state polls – and it is state races that determine the outcome, via the Electoral College. In the Meta-Analysis that Andrew Ferguson and I report on this website, Obama has been ahead all along."

"The likeliest cause of this discrepancy is that in states where it matters most such as Ohio, Nevada, and Colorado, the two candidates are genuinely overperforming/underperforming. Also, as I noted in 2008, winners in non-swing states often outperform polls. So there is some question about the accuracy of likely-voter screens when a local result is very unequal. Which leads us back to state polls as a better measure of the race.

In this graph the Meta-margin appears to be a lagging indicator. However, it’s not an apples-to-apples comparison, since I am showing the calculation as it unfolded, day by day. It can move quickly if more state polls are reported – as has occurred since Debate #2.

I will be interested to see if on Election Day, the national vote and the electoral vote count still show this discrepancy. If both are accurate, President Obama’s re-elect probability is about 90% – but his probability of winning the popular vote is lower, about 70%."

http://election.princeton.edu/2012/10/21/anatomy-of-a-bounce/#more-7753

 

[mshan]

*

Louis in NY // Oct 5, 2012 at 6:04 pm

"Sam,

I’ve been following your site for some time and finally decided to comment. As someone who has worked in politics, international relations, and have taught politics as an adjunct, I find your analysis and predictions quite refreshing and straight forward.

My question is as follows: does your Meta-margin model allow for changes/variation in polling methodology, especially when it’s the same pollster? My observation, as of late, is that many of the pollsters are seemingly self serving by changing their methodology.

You’ve done a very good job, as others, in outlining the variables from the standard polling as compared to robocalls, yet I notice that in many recent polls I’ve seen multiple instances when a pollster has not disclosed methodology or has changed from their previous approach; in some cases, seemingly going ‘backward’ and utilizing a less accurate methodology.

Looking forward to your reply. In the meantime, keep up the great work!




Sam Wang // Oct 5, 2012 at 6:14 pm

Thank you for writing. I have set up the code to be purposely blind as to pollster identity. No house effects and so on. If likely-voter and registered-voter are available from the same pollster, we take likely-voter data.

We considered having it accept at most one poll per state for a given pollster, which would be a justifiable move. However, at this point it would require redrawing the graph, something I don’t want to do midseason. In practice, pollster duplication is a major factor only in the early part of the campaign."

http://election.princeton.edu/2012/10/05/predictions-october-5th-presidentsenate/ (scroll down into reader comments)

 

[Farang]

Gallup not backing down.. Romney +7 today. Odd

 

[buckshot24]

Gallup not backing down.. Romney +7 today. Odd

They have Romney up 3 now in their registered voter poll. Romney was down just a few days ago.