Originally posted by: cquark
Crooked Timber debunks Slate's and others supposed debunking:As Chris said, with respect to the Lancet study on excess Iraqi deaths, ?I can predict with certainty that there will be numerous posts on weblogs supporting the war attacking the study?. Score several Cassandra points for Chris, they weren?t slow in coming. You can have the know-nothing rightwing flack variety or the handwringing liberal variety. And to be honest, the standard of critique is enough to make you weep.
Taking the complaints that seem to have been raised about this study:
?That is, a one in twenty chance that the effect simply does not exist? (from Tech Central Station). The author of the TCS piece appears to believe that because the Lancet study published a 95% confidence interval, there is a 5% chance that there was no effect. The problem with this critique is that it is not true.
?a relative risk ratio of anything less than three is regarded as statistically insignificant?. This is also from TCS, and also, simply, not true. Interesting to note that TCS appear to have upped the ante on this piece of bogus epidemiology; historically when they have been talking about passive smoking, the threshold for relative risk ratios has been two. Which is also bollocks. The TCS author appears to have a very shaky grasp of the statistical concepts he is using.
?This isn?t an estimate. It?s a dart board?. The critique here, from Slate, is that the 95% confidence interval for the estimate of excess deaths (8,000 to 200,000) is so wide that it?s meaningless. It?s wrong. Although there are a lot of numbers between 8,000 and 200,000, one of the ones that isn?t is a little number called zero. That?s quite startling. One might have hoped that there was at least some chance that the Iraq war might have had a positive effect on death rates in Iraq. But the confidence interval from this piece of work suggests that there would be only a 2.5% chance of getting this sort of result from the sample if the true effect of the invasion had been favourable. A curious basis for a humanitarian intervention; ?we must invade, because Saddam is killing thousands of his citizens every year, and we will kill only 8,000 more?.
The estimate of prewar mortality is too low. The idea here is that the sample chosen for the survey had a mortality rate of about 5 per 1000 in the two years before the invasion. And, because the death rate for the period 1985-90 was 6.8 per 1000 according to UN figures, this in some way suggests that the estimates are at fault.
This critique is more interesting, but hardly devastating. For one thing, the contention that the Iraqi death rate did not fall from 6.8 to around 5 during the 1990s is based on ?it must have done? rather than on hard numbers. Since the 6.8 number includes (as far as I can tell) atrocities committed by Saddam during the period which were not repeated in 2000-03, I am less convinced than the Slate author that the discrepancy strikes such a huge blow to the study?s credibility. In any case, since the study compares own-averages of the clusters before and after the invasion, anyone wanting to make this critique needs to come up with a convincing explanation of why it is that the study had a lower death-rate than the national average before the invasion and not after the invasion.
?various bog standard methodological quibbles are really really devastating?. This line of attack is usually associated with Steven Milloy, so I will nickname it the ?devastating critique?. The example I found was here. The modus operandi is to take a decent piece of statistical research carried out by someone who got his hands dirty with the data, point out a few areas in which it differs from the Platonic Form of the Epidemiological Study (if you?re dealing with a really good study, it does your work for you here by alerting you to the specific difficulties), and then say something like ?sheeeesh, how did this ever get published?!?!?. I?ve done it myself a few times, but that?s hardly a recommendation.
The Chicago Boyz blog post is an excellent example of the ?Devastating Critique?. Surprise surprise, estimating civilian casualties is a difficult business. That?s why the confidence interval is so wide. They don?t actually raise any principled reasons why the confidence interval ought to be wider than the one published, and therefore they aren?t raising any questions which would make us think that this confidence interval should include zero.
It gives a different number from Iraq Body Count. so it must be wrong. This critique is also fairly stupid. The IBC numbers are compiled from well-sourced English language press reports. They therefore represent a lower bound on any credible estimate of casualties, not a definitive number. Thousands of people die in the UK every day; how many of them make it into the papers? How may into the Arabic language press?
One can score extra points for intellectual dishonesty on this count by citing Oxblog to try to imply that IBC is in some way an overestimate (and therefore, of course, to push that confidence interval in the direction of zero). As the link I?ve provided shows, the Oxblog critique (which I don?t agree with) refers in the main to whether documented casualties can be blamed on the Americans; there is no well-founded challenge to suggest that the people IBC lists as dead are in fact consuming oxygen.
There is something intrinsically suspect about accelerated peer review. As John pointed out not so long ago, the time taken for peer review is determined by academic procrastination above all other factors. Every academic paper could complete its peer review very quickly if the reviewers got their finger out because they thought it was important. The suggestion that people are trying to make here is that reviewers for the Lancet usually spend six months humming and hawing over the data, to the exclusion of all other activity, and that this process was short-circuited by politically motivated editors wanting to rush something into print without anyone having a proper look at it. No such six month scrutiny ever takes place, and this objection is also Simply Not True.
The 100,000 figure should not have been headlined. Another staple critique of epidemiological studies one doesn?t like. It is true of more or less any study you hear of, since you never hear of studies that don?t have interesting headlines. In all honesty, I don?t like these extrapolated numbers, never have and never will. I don?t like linear models and I don?t like extrapolation. However, it?s a venial sin rather than a mortal one, and I have never, ever, at all, heard of anyone criticising it in a study that they otherwise liked. (Simple thought experiment; if the results of the study had been talking about 100,000 fewer deaths, would this critique have been made by the same people? Like hell).
The important thing as far as I?m concerned is the position of zero in the confidence interval; it seems very unlikely indeed that the process described could have given this sample if it was not the case that the invasion had made the death rate in Iraq worse rather than better. And this conclusion of the study is basically unchallenged. In fact, it?s in a better position than ?unchallenged?; it?s been challenged so weakly and on such spurious grounds that my Bayesian assessment has been updated in its favour, on the basis that if those who disliked the study?s conclusion had any real ammunition against it, the published critiques would not have been so weak.
Good find, thanks!
Facebook
Twitter