TCS Daily Article
I have a brief article in TCS Daily entitled "Thoughts from a Lancet Skeptic." The resulting comment thread is not, uh, very useful. Some editing was done on the piece after I submitted it. For those who care, here is the original version (note the nice things I had to say about Daniel Davies).
-----
Thoughts From A Lancet Skeptic
The British medical journal The Lancet published two articles about increases in mortality in Iraq caused by the US-led war and occupation. The authors estimated "excess deaths," the number of Iraqis who had died in excess of what would have happened if the pre-invasion mortality rate had not changed. The first paper (termed L1), with lead author Les Roberts, argued that 98,000 Iraqis had died because of the war and occupation through September 2004. The second (L2) updated the estimate to 655,000 through July 2006. Both articles were published shortly before the US November elections in an explicit attempt to affect the US political process.
Taken together, these two articles are the most important and controversial scientific publications of the decade, generating endless discussion among policymakers and the public. President Bush and former British Prime Minister Tony Blair were questioned about the authors's claims in press conferences. However, the estimates for excess deaths provided by the Lancet authors are almost certainly too high and possibly fraudulent. I was among the first scientists to make this claim. I will not be the last.
An Accidental Critic
How did I come to participate in this dispute? I have no background in survey research and no expertise in Iraq. But I do make a habit of reading blogs written by smart people with different views from my own. Indeed, there is no better way to test your beliefs than to confront the best proponents of alternate theories. To that end, I read Tim Lambert's blog Deltoid. Lambert, whatever his other faults, is careful and intelligent. He gets the details right. He is also a fierce critic of many of the articles and authors (i.e., Michael Fumento and Tim Ball) at TCS Daily. Lambert has defended the Lancet surveys extensively and, for the most part, his defense is correct. Indeed, the Lancet authors have benefited from the stupidity of many of their critics, as documented by bloggers like Daniel Davies. In Lambert's marvelous phrase, the articles have been like "flypaper for innumerates."
Yet just because many of the Lancet critics are clueless does not mean that the articles themselves are correct. My first reaction to L1 was that its confidence interval for excess deaths, 8,000 to 194,000, was suspiciously close to zero. Every good statistician is skeptical of a result which just barely rejects what researchers call the "null hypothesis," in this case that mortality in Iraq was unchanged after the invasion. A small change in the model assumptions could easily make the effect go away. Since it was obvious that the Lancet authors had political motivations/ambitions (Roberts ran for Congress in 2006), I thought that they were probably guilty of cherry-picking their model, at least to some extent. They would not be the first researchers to do so.
Yet these suspicions were, for me, overwhelmed by my disgust with the behavior of the authors and their supporters. Although they provided some summary data for L1, they refused to divulge the household-level data and computer code that would help outside researchers (like me) to replicate their results. This is not the way that scientist ought to act. But, since life is short and I am only a part-time academic, I left the issue behind after exchanging some e-mails with the authors in 2005. And that's where things would have stayed for me had the authors not published L2 in 2006. At that point, I felt a moral obligation to get to the bottom of this story. And so, here I am.
Criticisms
In August, I made a presentation at the annual meeting of the American Statistical Association which argued that the results of the first Lancet survey were internally inconsistent. The technical details are opaque at best but the implication is that the authors purposely presented their data (by including outlier data from Falluja in some parts of the analysis and excluding it elsewhere) to mislead. Specifically, the 8,000 to 194,000 confidence interval is claimed to be "conservative" because it excludes the carnage in Falluja. I show that including Falluja would have widened the confidence interval enough to include zero, thereby not allowing the authors to reject the null hypothesis of no increase in mortality.
In English, my claim is that the authors specifically refused to provide the confidence interval for excess deaths using all their data because they knew that doing so would provide too much ammunition to their critics. Even today, they stubbornly decline to tell me or anyone else what the confidence intervals would be with Falluja included.
But their behavior is even worse than that. They refuse to share the underlying data and computer code with the wider scientific community. Although some of the Lancet authors have conducted themselves professionally throughout the dispute (I would especially commend Gilbert Burnham and Shannon Doocy), Les Roberts has forced a sort of worst common denominator behavior on the team as a whole, including Elizabeth Johnson, the statistical consultant who performed the actual analysis.
That paper was discussed at Deltoid and then picked up from there by Michelle Malkin. Suddenly, I was part of the Right Wing Noise Machine, even invited as a guest on my local talk radio station. Alas, I misinterpreted the orders from my Rovian overlords and spent most of the time defending the Lancet authors from the innumerate complaints that the host was making. He choose not to keep me on the air long enough to get to the point of my actual critique.
Fortunately, other scientists are working on the topic. Colin Kahl writes that the Lancet estimates are "dubious." Fritz Scheuren, past president of the American Statistical Association, claims that the response rates from L2 are "not credible." Stephen Fienberg, one of the most respected statisticians in the country, insisted that Les Roberts' refusal to share data with Michael Spagat and his co-authors was:
If scientists like Kahl, Scheuren, Fienberg, Spagat and others think that your results are flawed and your behavior suspect, then you have a problem.
The Future
Where is the debate going? I sometimes worry that, like so many other left/right disputes, this will never be resolved, that we will never be sure whether or not the Lancet articles were fraudulent. Will these estimates be the Chambers/Hiss debate of the 21st century? I hope not. Fortunately, other scientists are hard at work on the topic, reanalyzing the data produced in L2 and conducting new surveys. Both critics and supporters of the Lancet results should be prepared to update their estimates in the face of this new evidence. If independent scientists publish results that are similar to those of the Lancet authors, then I will recant my criticism. Will Lancet supporters like Lambert and Davies do the same when the results go against their beliefs? I have my doubts.
-----
Thoughts From A Lancet Skeptic
The British medical journal The Lancet published two articles about increases in mortality in Iraq caused by the US-led war and occupation. The authors estimated "excess deaths," the number of Iraqis who had died in excess of what would have happened if the pre-invasion mortality rate had not changed. The first paper (termed L1), with lead author Les Roberts, argued that 98,000 Iraqis had died because of the war and occupation through September 2004. The second (L2) updated the estimate to 655,000 through July 2006. Both articles were published shortly before the US November elections in an explicit attempt to affect the US political process.
Taken together, these two articles are the most important and controversial scientific publications of the decade, generating endless discussion among policymakers and the public. President Bush and former British Prime Minister Tony Blair were questioned about the authors's claims in press conferences. However, the estimates for excess deaths provided by the Lancet authors are almost certainly too high and possibly fraudulent. I was among the first scientists to make this claim. I will not be the last.
An Accidental Critic
How did I come to participate in this dispute? I have no background in survey research and no expertise in Iraq. But I do make a habit of reading blogs written by smart people with different views from my own. Indeed, there is no better way to test your beliefs than to confront the best proponents of alternate theories. To that end, I read Tim Lambert's blog Deltoid. Lambert, whatever his other faults, is careful and intelligent. He gets the details right. He is also a fierce critic of many of the articles and authors (i.e., Michael Fumento and Tim Ball) at TCS Daily. Lambert has defended the Lancet surveys extensively and, for the most part, his defense is correct. Indeed, the Lancet authors have benefited from the stupidity of many of their critics, as documented by bloggers like Daniel Davies. In Lambert's marvelous phrase, the articles have been like "flypaper for innumerates."
Yet just because many of the Lancet critics are clueless does not mean that the articles themselves are correct. My first reaction to L1 was that its confidence interval for excess deaths, 8,000 to 194,000, was suspiciously close to zero. Every good statistician is skeptical of a result which just barely rejects what researchers call the "null hypothesis," in this case that mortality in Iraq was unchanged after the invasion. A small change in the model assumptions could easily make the effect go away. Since it was obvious that the Lancet authors had political motivations/ambitions (Roberts ran for Congress in 2006), I thought that they were probably guilty of cherry-picking their model, at least to some extent. They would not be the first researchers to do so.
Yet these suspicions were, for me, overwhelmed by my disgust with the behavior of the authors and their supporters. Although they provided some summary data for L1, they refused to divulge the household-level data and computer code that would help outside researchers (like me) to replicate their results. This is not the way that scientist ought to act. But, since life is short and I am only a part-time academic, I left the issue behind after exchanging some e-mails with the authors in 2005. And that's where things would have stayed for me had the authors not published L2 in 2006. At that point, I felt a moral obligation to get to the bottom of this story. And so, here I am.
Criticisms
In August, I made a presentation at the annual meeting of the American Statistical Association which argued that the results of the first Lancet survey were internally inconsistent. The technical details are opaque at best but the implication is that the authors purposely presented their data (by including outlier data from Falluja in some parts of the analysis and excluding it elsewhere) to mislead. Specifically, the 8,000 to 194,000 confidence interval is claimed to be "conservative" because it excludes the carnage in Falluja. I show that including Falluja would have widened the confidence interval enough to include zero, thereby not allowing the authors to reject the null hypothesis of no increase in mortality.
In English, my claim is that the authors specifically refused to provide the confidence interval for excess deaths using all their data because they knew that doing so would provide too much ammunition to their critics. Even today, they stubbornly decline to tell me or anyone else what the confidence intervals would be with Falluja included.
But their behavior is even worse than that. They refuse to share the underlying data and computer code with the wider scientific community. Although some of the Lancet authors have conducted themselves professionally throughout the dispute (I would especially commend Gilbert Burnham and Shannon Doocy), Les Roberts has forced a sort of worst common denominator behavior on the team as a whole, including Elizabeth Johnson, the statistical consultant who performed the actual analysis.
That paper was discussed at Deltoid and then picked up from there by Michelle Malkin. Suddenly, I was part of the Right Wing Noise Machine, even invited as a guest on my local talk radio station. Alas, I misinterpreted the orders from my Rovian overlords and spent most of the time defending the Lancet authors from the innumerate complaints that the host was making. He choose not to keep me on the air long enough to get to the point of my actual critique.
Fortunately, other scientists are working on the topic. Colin Kahl writes that the Lancet estimates are "dubious." Fritz Scheuren, past president of the American Statistical Association, claims that the response rates from L2 are "not credible." Stephen Fienberg, one of the most respected statisticians in the country, insisted that Les Roberts' refusal to share data with Michael Spagat and his co-authors was:
just the wrong response. I, as an editor, would not publish a study for which the data was not shared.
If scientists like Kahl, Scheuren, Fienberg, Spagat and others think that your results are flawed and your behavior suspect, then you have a problem.
The Future
Where is the debate going? I sometimes worry that, like so many other left/right disputes, this will never be resolved, that we will never be sure whether or not the Lancet articles were fraudulent. Will these estimates be the Chambers/Hiss debate of the 21st century? I hope not. Fortunately, other scientists are hard at work on the topic, reanalyzing the data produced in L2 and conducting new surveys. Both critics and supporters of the Lancet results should be prepared to update their estimates in the face of this new evidence. If independent scientists publish results that are similar to those of the Lancet authors, then I will recant my criticism. Will Lancet supporters like Lambert and Davies do the same when the results go against their beliefs? I have my doubts.
posted by David Kane at 6:24 AM
0 comments
