"First, to quote
Matthews’s claim:
John Sides and the team at the Monkey Cage have a model that uses GDP, the president’s party and approval rating, incumbency, and district-level presidential vote, rather than House polling. Their model gets the seat margin wrong by 2.61 seats, on average, much lower than Wang’s error. It gives Republicans a three out of four chance of keeping the House.
This is seemingly a convincing criticism. However, at its heart is some misdirection. The “2.61 seats” statement is revealing because it is too small to be realistic. It is the same weakness I detected in the FiveThirtyEight “four-factor” model yesterday: Overfitting of small residuals is basically
chasing noise, and leads to massive uncertainties.
Now, the
Monkey Cage crew is aware of this issue. To quote them:
The standard error for the vote share estimate is 5.6%; for seat share, 8.7%. That’s a lot of uncertainty. It means there is at least a little probability of some pretty crazy outcomes. It explains why there is still a 1 in 4 chance that the Democrats will get the 25 seats they need to retake the House, when our own median prediction is only one seat.
In other words, the “median gain of one seat” sounds precise…but is meaningless.
Let me make the point graphically. Here are our two national-popular-vote predictions plotted side by side…but with uncertainties included:
For those of you unfamiliar with this kind of plot, the data points are the values that get reported in the popular press. The horizontal lines are error bars. They indicate the confidence with which we know the median. A large error bar indicates high uncertainty.
As you can see, our two ranges are perfectly consistent – but the PEC estimate gives much more certainty – and information. In contrast, their range, from R+13% to D+9%, contains many possibilities that we can be confident will not happen in November. If the Republicans win by 10 points, I will personally wash Dylan Matthews’s car with a toothbrush.
What about
seat count, the ultimate measure of House control? Same story:
In short, their model indicates a three in four chance of GOP control because their uncertainty is massive. Do you think the Republicans will attain a 278-157 majority?
In some sense, our two calculations are consistent. However, what I presented is not a complex model in the same sense, but a precise short-term projection of likely outcomes.
http://election.princeton.edu/2012/09/21/monkeying-around-with-fundamentals-based-models/#more-5986