Old Posts
I am deleting some Lancet posts that I placed on EphBlog and putting them here.
------------------
October 21, 2006: Dust Up
Many readers will be perplexed about Billy's comment on the previous post.
Billy is referring to a post I wrote about the Burnham et al (2006) paper about the number of excess deaths in Iraq resulting from invasion and occupation. (That post was taken down and replace with this.) For background reading, you can start with Dan Drezner '90. I have been involved in the debate since the first Lancet study, Roberts et al (2004). The post I wrote generated criticism here, here and here.
Now, were it not for Billy's comment, I probably wouldn't have mentioned any of this. This blog is about All Things Eph, after all, not All Things David Kane. But if it were a different Eph than I involved in the debate, I would certainly be following it here. The reason that I published the original post (and related ones) at the SSS Blog was that they seemed to belong there (or someplace like there) and not here.
Now, if this were all happening to some other Eph, I would invite him to start an Eph Dairy devoted to the topic here, to post as often as he liked about his endless debates about the study. I would encourage this Eph to make most of his posts brief, with continuations below the break, so that the main page were not so cluttered with ramblings that were not of broad interest to our audience.
Should I follow the inverse of Kant's categorical imperative and treat myself the way I would treat any other Eph. Perhaps. For now, I'll leave it to the readers of EphBlog, especially my fellow authors. Should I start an Eph Diary about my criticisms of these Lancet articles?
------------------
August 5, 2007: Eph Diarist on Lancet/Iraq?
Regular readers will know that I am heavily involved (here, here and here) in work critical of estimates of mortality in Iraq published in The Lancet. I think more of this belongs in EphBlog. Do you agree?
Our Eph Diarist feature tries to bring the non-Williams related work of an Eph to a broader audience. If an Eph asked me if she could write regular entries about her efforts on this topic, I would gladly publish her, perhaps with a request that she put most of each entry "below the fold" (as I have done with this one) so that readers uninterested in the topic are not distracted. If people complained that this, like Derek's Red Sox Diary, did not belong on EphBlog, I would tell them to not read her stuff.
But, since I am the one interested in doing this, I can hardly judge my own case. So, regular readers, what do you think? I will probably post an entry or two just to give people a flavor. I like to think that this is great stuff. The substantive issue, mortality in Iraq, is important. The statistical details are subtle. Seeing how science works behind the scenes --- working papers, conference presentations, peer review --- is interesting.
And, if you find all this boring, don't read the entries!
------------------
August 6, 2007: "Not Credible"
The most damning comment about the Lancet estimates of mortality in Iraq came in the Wednesday afternoon session.
The presentation by Les Roberts focussed on the second Lancet survey (termed L2 by we aficionados). One discussant was Fritz Scheuren of NORC at the University of Chicago. Scheuren, besides being a distinguished statistician, is a past president of the American Statistical Association (the organization in charge of the entire conference).
Scheuren termed the response rate in L2 "not credible." (That is an exact quote. He used the phrase several times and clearly meant it.) Comments:
1) It was amazing to see such an accusation at a professional gathering. Scheuren was not suggesting that Roberts himself was guilty of fraud. Instead, he seems to believe that there is no way that the Iraqi interviewers achieved a 98% response rate.
2) Scheuren is an expert on surveys in conflict zones.
If a statistician of Scheuren's caliber and experience says that you have a problem, you have a problem.
3) Note that this point (a 98%+ response rate is ipso facto evidence of fraud) was what first made me (in)famous in Lancet circles last November. Here is a copy of the original post and associated links. Scheuren seems to agree with me. Does ace Lancet defender Daniel Davies consider Scheuren guilty of "devious hack-work?"
4) During the Q&A, Scheuren and Roberts got into a back and forth on data sharing. Scheuren made clear that, while he did not view the response rate as "credible," he did not think that it was absolutely impossible either. He was ready to be convinced if Roberts were to share more data with him. Specifically, Scheuren wants to see the results broke down by interviewer/team. For example, if one of the two main teams reported 100% response rates while the other reported 96%, Scheuren would (I think) be a lot more skeptical of the first team's results. Roberts mumbled something about that data not being on the computer (which is plausible). But Roberts also expressed no interest in providing Scheuren (or anybody else) with more data. At one point, Roberts said that, if it were up to him, the Lancet authors would not have shared any of their data with anyone else.
I will have further updates on what happened in Salt Lake City over the next couple of days. When the history of this sad saga is written, the moment when a past president of the American Statistical Society termed the Lancet numbers "not credible" will mark the end of the beginning of the debunking of these flawed studies.
Let me end with the same challenge that I have been issuing for the last year: Is there another survey like the one conducted by L2 (nationwide sample with a single contact attempt) for which the response rate is as high or higher than 98%? I have never been able to find one. Can anyone? Any survey in any country on any topic at any time? And, if we can't, what are the odds that one of the most controversial surveys of the decade should just happen, by chance, to have the highest response rate ever?
"Not credible," indeed.
[And from the comment thread to that post.]
Excellent catch on this article.
I think that I asked the most "courtroom-style" question. More on that later.
I think that "blasted" is a fair description of the reaction to Roberts' refusal to share data with his critics. In terms of a "major flaw," too bad the reporter did not ask me! I give good quote.
Harsh! More on what else Fienberg had to say soon.
Says who? I have never read of such an accusation. Has anyone? This sounds to me like something that Roberts spoon fed to the reporter. "See? I get this criticism all the time . . . "
Really? Roberts admitted this (sort of) with regard to L1 but has since backed away. I have never seen him admit this for L2. Has he? Perhaps this came up at lunch because I did not see Roberts do this in the main presentation.
This is not inconsistent with what I wrote above. Scheuren is a careful guy. He can not point to a flaw in the data that he has seen. He wants to get more detailed data. He still does not find the results "credible."
More heat (generated by Roberts?) than light. Scheuren does not want to know the "researchers' identities." None of us do. We just want to know that Researcher A reported this data; Researcher B reported that data; and so on. Roberts has an annoying habit of pretending that his critics are asking for names when he knows we just want to see if the results vary greatly by interviewer.
------------------
August 7, 2007: Holocaust Denial
Besides his formal presentation, Les Roberts also participated in a luncheon roundtable entitled "Media Coverage Regarding Data on Mortality in Iraq."
I was able to pull up and chair and listen in. Roberts is a charming and engaging speaker. Most of the ten or so people around the table were clearly fans. For me, the best part was when Roberts, perhaps playing a bit to his audience, compared people who disagree with the Lancet estimates to people who deny the Holocaust.
Nice!
I have heard of Roberts using this comparison in the past, but this was the first time that I got a chance to witness it myself. He comes close to making that accusation in his speech at Brown, but shies away from making the comparison explicit, perhaps because he knew that the tape was running. He frames the issue by pointing out (correctly!) that Iranian President Mahmoud Ahmadinejad makes people angry when he describes the Holocaust as a "myth." Roberts thinks that President Bush dismissing the Lancet estimates amounts to the same thing.
It is not clear if Roberts thinks that Fritz Scheuren, past president of the American Statistical Association, is no better than famous Holocaust denier David Irving. Clarification welcome!
------------------
October 21, 2006: Dust Up
Many readers will be perplexed about Billy's comment on the previous post.
Dude, don't you have anything to say about the dust-up at the SSS Blog?
Billy is referring to a post I wrote about the Burnham et al (2006) paper about the number of excess deaths in Iraq resulting from invasion and occupation. (That post was taken down and replace with this.) For background reading, you can start with Dan Drezner '90. I have been involved in the debate since the first Lancet study, Roberts et al (2004). The post I wrote generated criticism here, here and here.
Now, were it not for Billy's comment, I probably wouldn't have mentioned any of this. This blog is about All Things Eph, after all, not All Things David Kane. But if it were a different Eph than I involved in the debate, I would certainly be following it here. The reason that I published the original post (and related ones) at the SSS Blog was that they seemed to belong there (or someplace like there) and not here.
Now, if this were all happening to some other Eph, I would invite him to start an Eph Dairy devoted to the topic here, to post as often as he liked about his endless debates about the study. I would encourage this Eph to make most of his posts brief, with continuations below the break, so that the main page were not so cluttered with ramblings that were not of broad interest to our audience.
Should I follow the inverse of Kant's categorical imperative and treat myself the way I would treat any other Eph. Perhaps. For now, I'll leave it to the readers of EphBlog, especially my fellow authors. Should I start an Eph Diary about my criticisms of these Lancet articles?
------------------
August 5, 2007: Eph Diarist on Lancet/Iraq?
Regular readers will know that I am heavily involved (here, here and here) in work critical of estimates of mortality in Iraq published in The Lancet. I think more of this belongs in EphBlog. Do you agree?
Our Eph Diarist feature tries to bring the non-Williams related work of an Eph to a broader audience. If an Eph asked me if she could write regular entries about her efforts on this topic, I would gladly publish her, perhaps with a request that she put most of each entry "below the fold" (as I have done with this one) so that readers uninterested in the topic are not distracted. If people complained that this, like Derek's Red Sox Diary, did not belong on EphBlog, I would tell them to not read her stuff.
But, since I am the one interested in doing this, I can hardly judge my own case. So, regular readers, what do you think? I will probably post an entry or two just to give people a flavor. I like to think that this is great stuff. The substantive issue, mortality in Iraq, is important. The statistical details are subtle. Seeing how science works behind the scenes --- working papers, conference presentations, peer review --- is interesting.
And, if you find all this boring, don't read the entries!
------------------
August 6, 2007: "Not Credible"
The most damning comment about the Lancet estimates of mortality in Iraq came in the Wednesday afternoon session.
The presentation by Les Roberts focussed on the second Lancet survey (termed L2 by we aficionados). One discussant was Fritz Scheuren of NORC at the University of Chicago. Scheuren, besides being a distinguished statistician, is a past president of the American Statistical Association (the organization in charge of the entire conference).
Scheuren termed the response rate in L2 "not credible." (That is an exact quote. He used the phrase several times and clearly meant it.) Comments:
1) It was amazing to see such an accusation at a professional gathering. Scheuren was not suggesting that Roberts himself was guilty of fraud. Instead, he seems to believe that there is no way that the Iraqi interviewers achieved a 98% response rate.
2) Scheuren is an expert on surveys in conflict zones.
Fritz Scheuren, Ph.D., is a statistical consultant for HRDAG [Human Rights Data Analysis Group]. Recently, Dr. Scheuren consulted on the methods of statistical analysis for Peru's Truth and Reconciliation Commission and reviewed the report of the analysis. In late August of 2003, he visited Peru, along with Dr. Ball and Jana Asher, to meet with representatives of political, military, and civil society groups to explain the technical basis of the findings detailed in the report. In recent years, Dr. Scheuren has advised on the overall direction and approach to the statistical analysis for several HRDAG projects, including Guatemala and Kosovo. As a top statistician in the field, he has also provided critical peer-review of HRDAG work.
If a statistician of Scheuren's caliber and experience says that you have a problem, you have a problem.
3) Note that this point (a 98%+ response rate is ipso facto evidence of fraud) was what first made me (in)famous in Lancet circles last November. Here is a copy of the original post and associated links. Scheuren seems to agree with me. Does ace Lancet defender Daniel Davies consider Scheuren guilty of "devious hack-work?"
4) During the Q&A, Scheuren and Roberts got into a back and forth on data sharing. Scheuren made clear that, while he did not view the response rate as "credible," he did not think that it was absolutely impossible either. He was ready to be convinced if Roberts were to share more data with him. Specifically, Scheuren wants to see the results broke down by interviewer/team. For example, if one of the two main teams reported 100% response rates while the other reported 96%, Scheuren would (I think) be a lot more skeptical of the first team's results. Roberts mumbled something about that data not being on the computer (which is plausible). But Roberts also expressed no interest in providing Scheuren (or anybody else) with more data. At one point, Roberts said that, if it were up to him, the Lancet authors would not have shared any of their data with anyone else.
I will have further updates on what happened in Salt Lake City over the next couple of days. When the history of this sad saga is written, the moment when a past president of the American Statistical Society termed the Lancet numbers "not credible" will mark the end of the beginning of the debunking of these flawed studies.
Let me end with the same challenge that I have been issuing for the last year: Is there another survey like the one conducted by L2 (nationwide sample with a single contact attempt) for which the response rate is as high or higher than 98%? I have never been able to find one. Can anyone? Any survey in any country on any topic at any time? And, if we can't, what are the odds that one of the most controversial surveys of the decade should just happen, by chance, to have the highest response rate ever?
"Not credible," indeed.
[And from the comment thread to that post.]
Excellent catch on this article.
Statisticians grill surveyor of Iraqi casualties of war
By Paul Foy
The Associated Press
The courtroom-style questioning came in a packed ballroom in Salt Lake City at the world's largest gathering of statisticians.
I think that I asked the most "courtroom-style" question. More on that later.
On the hot seat: A globe-trotting researcher who says his team's surveys of Iraqi households projected nearly 655,000 had died in the war as of July 2006, a number still 10 times higher than conventional estimates.
Leslie F. Roberts and others from Johns Hopkins University took accounts of births and deaths in some Iraqi households to estimate that the country's death rate had more than doubled after the 2003 invasion.
Number crunchers this week quibbled with Roberts' survey methods and blasted his refusal to release all his raw data for scrutiny - or any data to his worst critics. Some discounted him as an advocate for world peace, although none could find a major flaw in his surveys or analysis.
I think that "blasted" is a fair description of the reaction to Roberts' refusal to share data with his critics. In terms of a "major flaw," too bad the reporter did not ask me! I give good quote.
However, Stephen Fienberg, a professor of statistics at Carnegie Mellon University in Pittsburgh, said: ''I thought the surveys were pretty good for non-statisticians.''
Harsh! More on what else Fienberg had to say soon.
Roberts, an epidemiologist, said he is opening a new front in the study of public health hazards: War. He has conducted about 30 mortality studies since 1990 in conflicts around the globe, including the Congo, where he was similarly accused of exaggerating war-related deaths.
Says who? I have never read of such an accusation. Has anyone? This sounds to me like something that Roberts spoon fed to the reporter. "See? I get this criticism all the time . . . "
Roberts organized two surveys of mortality in Iraqi households that were published last October in Britain's premier the medical journal, The Lancet. He acknowledged that the timing was meant to influence midterm U.S. elections.
Really? Roberts admitted this (sort of) with regard to L1 but has since backed away. I have never seen him admit this for L2. Has he? Perhaps this came up at lunch because I did not see Roberts do this in the main presentation.
''It puts you in a position where you are going to get attacked,'' said Fritz Scheuren, a senior fellow at the University of Chicago's National Opinion Research Center, who is trying to organize another Iraqi survey to see if he can match Roberts' results.
Scheuren, the American Statistical Association's former president, said he couldn't find anything wrong with The Lancet surveys.
This is not inconsistent with what I wrote above. Scheuren is a careful guy. He can not point to a flaw in the data that he has seen. He wants to get more detailed data. He still does not find the results "credible."
He complained, however, that he wasn't able to get Roberts to reveal which of his Iraqi surveyors conducted which surveys, information that could reveal any bias in workers who compile consistently implausible results.
Roberts said he won't release the researchers' identities for fear of exposing them to retaliation. The Iraqi government has strongly disputed the findings.
More heat (generated by Roberts?) than light. Scheuren does not want to know the "researchers' identities." None of us do. We just want to know that Researcher A reported this data; Researcher B reported that data; and so on. Roberts has an annoying habit of pretending that his critics are asking for names when he knows we just want to see if the results vary greatly by interviewer.
------------------
August 7, 2007: Holocaust Denial
Besides his formal presentation, Les Roberts also participated in a luncheon roundtable entitled "Media Coverage Regarding Data on Mortality in Iraq."
I was able to pull up and chair and listen in. Roberts is a charming and engaging speaker. Most of the ten or so people around the table were clearly fans. For me, the best part was when Roberts, perhaps playing a bit to his audience, compared people who disagree with the Lancet estimates to people who deny the Holocaust.
Nice!
I have heard of Roberts using this comparison in the past, but this was the first time that I got a chance to witness it myself. He comes close to making that accusation in his speech at Brown, but shies away from making the comparison explicit, perhaps because he knew that the tape was running. He frames the issue by pointing out (correctly!) that Iranian President Mahmoud Ahmadinejad makes people angry when he describes the Holocaust as a "myth." Roberts thinks that President Bush dismissing the Lancet estimates amounts to the same thing.
It is not clear if Roberts thinks that Fritz Scheuren, past president of the American Statistical Association, is no better than famous Holocaust denier David Irving. Clarification welcome!
posted by David Kane at 12:08 PM
0 Comments:
Post a Comment
<< Home