Introduction
I will be using this blog to post links and comments related to two articles measuring mortality in Iraq which appeared in the Lancet in 2004 and 2006. The articles are:
1) Les Roberts, Riyadh Lafta, Richard Garfield, Jamal Khudhairi, and Gilbert Burnham, "Mortality before and after the 2003 invasion of Iraq: cluster sample survey", The Lancet, 29 October 2004.
2) Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts, "Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey", The Lancet, 11 October 2006.
The best overview of the articles, including associated links, is at Wikipedia.
I first became involved in this debate on Deltoid, a blog maintained by Tim Lambert. My initial set of comments on Roberts et al. (2004) can be found here. My first comment was:
And then it was down the rabbit hole for me.
I am indirectly responsible for making a summary version of the data associated with Roberts el al. (2004) public. I have lobbied various people associated with both studies to make more data available. The fact that the authors have repeatedly refused to do so is my single biggest complaint against them.
The best summary of critiques and rebuttals to Roberts et al. (2004) is by Daniel Davies at Crooked Timber.
My current opinion is that people like Davies and Lambert are correct that a large, large number of the critics of both studies are wrong, even idiotic. These critics have no idea what they are talking about. In Lambert's marvelous phrase, the studies have been "flypaper for innumerates." But, just because many of the critics have been incoherent does not mean that there are not significant concerns that a reasonable person might have with the studies and the behavior of the authors. One purpose of this blog is to bring these concerns to the forefront of the discussion.
This is not a blog about the Iraq War. One can believe that the Lancet studies are 100% correct and still be either pro-war or anti-war. As an occasional teacher, I am more interested in the use of the debate surrounding these articles as a jumping off point for a discussion of statistical methodology and the proper way to conduct academic research. I am a big fan of the replication standard.
I come to this debate as a natural skeptic. I may appear highly critical of these articles, but this is mainly because I am highly critical of many peer-reviewed journal articles, especially those articles that refuse to be transparent about their methodology and/or data. My own guess is that the Lancet authors have overestimated the increase in mortality from pre-war to post-war. But, unlike many critics, I think that the estimates in both articles are not unreasonable.
If I had to bet, I would provide much wider confidence intervals than either the Lancet authors or most of their critics. Burnham et al. (2006) estimate 650,000 "excess deaths" since the start of the war with a 95% confidence interval of 400,000 to 950,000. My own estimate would center around 300,000 and range from 0 to 1.2 million. Obviously, no one is really interested in my estimate --- derived as it is from reading the literature and associate debates --- but I thought it reasonable to be upfront about my prior beliefs.
1) Les Roberts, Riyadh Lafta, Richard Garfield, Jamal Khudhairi, and Gilbert Burnham, "Mortality before and after the 2003 invasion of Iraq: cluster sample survey", The Lancet, 29 October 2004.
2) Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts, "Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey", The Lancet, 11 October 2006.
The best overview of the articles, including associated links, is at Wikipedia.
I first became involved in this debate on Deltoid, a blog maintained by Tim Lambert. My initial set of comments on Roberts et al. (2004) can be found here. My first comment was:
jre claims that, with regard to the Lancet student, “not a single valid objection [has] been raised against its methodology or its results.”
Hmmm. Allow me to raise one. The central problem with the Lancet study was that it was conducted by people who, before the war started, were against the war, people who felt that the war was likely to increase civilian casualities and who, therefore, had a expectation/desire (unconscious or otherwise) to find the result that they found.
Consider the Iraqis who did the actual door-to-door surveying. Do you think that they appreciated having such a well paying job? Do you think that they hoped for more such work? If you were them, would you be tempted to shade the results just a little so that the person paying you was happy?
Consider a counter-factual world in which the Pentagon (or the AEI or the Heritage Foundation) did a study with exactly the same methodology, sampling plan and so on. If that study had shown 100,000 fewer deaths, would you have been as quick to believe it?
Again, this objection in no way invalidates the study. But don’t pretend that valid objections have not and could not be made.
And then it was down the rabbit hole for me.
I am indirectly responsible for making a summary version of the data associated with Roberts el al. (2004) public. I have lobbied various people associated with both studies to make more data available. The fact that the authors have repeatedly refused to do so is my single biggest complaint against them.
The best summary of critiques and rebuttals to Roberts et al. (2004) is by Daniel Davies at Crooked Timber.
My current opinion is that people like Davies and Lambert are correct that a large, large number of the critics of both studies are wrong, even idiotic. These critics have no idea what they are talking about. In Lambert's marvelous phrase, the studies have been "flypaper for innumerates." But, just because many of the critics have been incoherent does not mean that there are not significant concerns that a reasonable person might have with the studies and the behavior of the authors. One purpose of this blog is to bring these concerns to the forefront of the discussion.
This is not a blog about the Iraq War. One can believe that the Lancet studies are 100% correct and still be either pro-war or anti-war. As an occasional teacher, I am more interested in the use of the debate surrounding these articles as a jumping off point for a discussion of statistical methodology and the proper way to conduct academic research. I am a big fan of the replication standard.
I come to this debate as a natural skeptic. I may appear highly critical of these articles, but this is mainly because I am highly critical of many peer-reviewed journal articles, especially those articles that refuse to be transparent about their methodology and/or data. My own guess is that the Lancet authors have overestimated the increase in mortality from pre-war to post-war. But, unlike many critics, I think that the estimates in both articles are not unreasonable.
If I had to bet, I would provide much wider confidence intervals than either the Lancet authors or most of their critics. Burnham et al. (2006) estimate 650,000 "excess deaths" since the start of the war with a 95% confidence interval of 400,000 to 950,000. My own estimate would center around 300,000 and range from 0 to 1.2 million. Obviously, no one is really interested in my estimate --- derived as it is from reading the literature and associate debates --- but I thought it reasonable to be upfront about my prior beliefs.
posted by David Kane at 3:41 AM
0 comments
