Lancet On Iraqi Mortality

Dubious Polling

2010-03-02T08:04:00.000-08:00

From the "2010 Top Ten "Dubious Polling" Awards" by Pollsters George Bishop and David Moore.

With this article, veteran pollsters, authors and political scientists George F. Bishop and David W. Moore issue their Second Annual Top Ten “Dubious Polling” Awards. These awards are intended to mark for posterity some of the most risible and outrageous pronouncements by polling organizations during the previous year.

...

“STONEWALLING/COVERUP” Award

WINNERS: The John Hopkins Bloomberg School of Public Health and one of its professors, Dr. Gilbert Burnham, for stonewalling in the face of serious questions about a flawed survey project, which reported more than 600,000 Iraqi deaths from 2003 to 2006. The head researcher was formally censured by the American Association for Public Opinion Research (AAPOR) for covering up his data collection efforts, but the Bloomberg School refuses to investigate the methodology. (Ah, the wisdom of the three monkeys: “See no evil, hear no evil, speak no evil!”).

BACKGROUND: In 2006, the British medical journal, The Lancet, published the results of a survey, designed and supervised by Dr. Gilbert Burnham of the John Hopkins Bloomberg School of Public Health, and his colleagues.* The survey purported to show that about 600,000 Iraqi deaths occurred in Iraq by July 2006, as a consequence of the invasion of Iraq.

A lot of people were against the war, but jacking up the body count with bad studies is not a good tactic for anyone. According to economics professor Michael Spagat of Royal Holloway College, these results were anywhere from seven to 14 times as high as other credible estimates, including those made by the non-partisan Iraq Body Count, a consortium of U.S. and U.K researchers, also concerned about the human toll of the war.

Such large differences in estimates led other researchers to question the methodology of the study. But contrary to scientific norms, Burnham refused to provide details about how the survey was conducted. When a complaint was lodged with AAPOR, its standards committee also tried to obtain such details, but was rebuffed. That led to the censure.

What exactly were John Hopkins Bloomberg School, and Burnham, et. al., hiding? AAPOR asked for the kind of information that any scientist doing this type of work should release: a copy of the questionnaire, the consent statement that interviewees have to see, a full description of the selection process, a summary of the disposition of all sample cases, and how the mortality rate was calculated.

John Hopkins Bloomberg School initially stood behind the study, but then eventually concluded that Burnham had made some unauthorized changes in his methodology, and thus “the School has suspended Dr. Burnham’s privileges to serve as a principal investigator on projects involving human subjects research.”

But the Bloomberg School has not come clean with the problems of the research project. Their press release admitted that their internal review “did not evaluate aspects of the sampling methodology or statistical approach of the study.” Instead, Bloomberg asserts, “It is expected that the scientific community will continue to debate the best methods for estimating excess mortality in conflict situations in appropriate academic forums.”

Let’s see: The Bloomberg School will not attempt to evaluate what experts believe is almost certainly a faulty methodology, saying the scientific community should make the evaluation. But then the school advises Burnham not to release details about his methods, so the scientific community can’t have the information it needs for a definitive assessment.

Sounds like a cop-out and a Catch 22, all rolled into one!

And we thought Richard Nixon was tricky.

* Burnham G, Lafta R, Doocy S, Roberts L. 2006a. ‘Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey’. The Lancet 368:1421-1428. It can be accessed online at http://brusselstribunal.org/pdf/lancet111006.pdf.

Response Rates

2010-02-04T12:25:00.000-08:00

My hard working research assistant Daniel Suo spent a bunch of time looking for surveys with a response rate as high as Lancet II. We never found a single one. He also gathered some quotes from survey experts. Here they are:

"-Any U.S. survey with contact rates higher than the figures you quoted from the Census Bureau would be suspicious. It is not easy to find people at home AND who will answer the door.
-Figures like 90% would indicate that we are NOT dealing with a random selection of households. Some market research surveys that use quota samples instruct interviewers to go to a certain block and then knock on every nth door until they find someone at home. That's how you get contact rates of 100%. "
- Professor Tom Piazza, University of California at Berkeley

"in today's survey environment, there is no such thing as an extremely high response
rate on a door-to-door survey with just one contact attempt. The demographics work against
it, along with increasing resistance to surveys."
- Professor John Goyder, University of Waterloo

" I think the best estimate is from Groves and Couper, page 82 - - 49% contacted on first attempt.
Mind you, only some of those consented to the interview, so this is simply contactability. JG"
- Professor John Goyder, University of Waterloo

"If I had to pick a number, I'd say that response rates
above 90% on the first contact for door to door surveys of people are
unusual these days."
- Professor Jerry Reiter, Duke University

"I think it is impossible to get even close to 100% on first attempt, people out, or not answering doors.
I would ask them to define what they mean buy response rates, they may be playing with the definition to suit their needs."
- Randa Bell , CMRP, PRC Vice President- Marketing and Client Services ASDE Survey Sampler, Inc

"if you can
get 25% or above I think you can be confident presenting your results to an
academic crowd."
- Professor Phillip Howard, University of Washington

"I don't know of any study that gets even close to 100 percent nor a
systematic list of studies focusing on their response rate. "
- Professor Sidney Verba, Harvard University

"In Mexico, single-contact-attempt door-to-door survey have an average response rate of 70%. There is not such a thing as a 90% or higher response rate (at least not that I know). "
- Pablo Paras

2009-11-19T09:12:00.000-08:00

From this article in the New York Times:

One big question hovering over McChrystal is whether his experience in Iraq truly prepares him for the multiheaded challenge that faces him now. For nearly five years, McChrystal served as chief of the Joint Special Operations Command, which oversees the military’s commando units, including the Army Delta Force and the Navy Seals. (Until recently, the Pentagon refused to acknowledge that the command even existed.)

As JSOC’s commander, McChrystal spent no time trying to win over the Iraqis or training Iraqi forces or building the governing capacity of the Iraqi state. In Iraq (and, for about a third of his time, in Afghanistan), McChrystal’s job, and that of the men under his command, was, almost exclusively, to kill and capture insurgents and terrorists.

The rescue of Iraq from the cataclysm that it had become by 2006 is an epic tale of grit and blood and luck. By February of that year, Iraq had descended into a full-blown civil war, with a thousand civilians dying every month. Its central actors were the gunmen of Al Qaeda, who, with their suicide bombers, carried out large-scale massacres of Shiite civilians; and the Shiite militias, some of them in Iraqi uniforms, who retaliated by massacring thousands of young Sunni men.

Emphasis added. Is this turning into the conventional wisdom? Instead of the 25,000 or so deaths per month that Lancet 2 would suggest for this period, the New York Times is now going with 1,000? That would be in the same neighborhood (even less than?) IBC.

Old Posts

2009-06-04T12:08:00.001-07:00

I am deleting some Lancet posts that I placed on EphBlog and putting them here.

------------------
October 21, 2006: Dust Up

Many readers will be perplexed about Billy's comment on the previous post.

Dude, don't you have anything to say about the dust-up at the SSS Blog?

Billy is referring to a post I wrote about the Burnham et al (2006) paper about the number of excess deaths in Iraq resulting from invasion and occupation. (That post was taken down and replace with this.) For background reading, you can start with Dan Drezner '90. I have been involved in the debate since the first Lancet study, Roberts et al (2004). The post I wrote generated criticism here, here and here.

Now, were it not for Billy's comment, I probably wouldn't have mentioned any of this. This blog is about All Things Eph, after all, not All Things David Kane. But if it were a different Eph than I involved in the debate, I would certainly be following it here. The reason that I published the original post (and related ones) at the SSS Blog was that they seemed to belong there (or someplace like there) and not here.

Now, if this were all happening to some other Eph, I would invite him to start an Eph Dairy devoted to the topic here, to post as often as he liked about his endless debates about the study. I would encourage this Eph to make most of his posts brief, with continuations below the break, so that the main page were not so cluttered with ramblings that were not of broad interest to our audience.

Should I follow the inverse of Kant's categorical imperative and treat myself the way I would treat any other Eph. Perhaps. For now, I'll leave it to the readers of EphBlog, especially my fellow authors. Should I start an Eph Diary about my criticisms of these Lancet articles?

------------------
August 5, 2007: Eph Diarist on Lancet/Iraq?

Regular readers will know that I am heavily involved (here, here and here) in work critical of estimates of mortality in Iraq published in The Lancet. I think more of this belongs in EphBlog. Do you agree?

Our Eph Diarist feature tries to bring the non-Williams related work of an Eph to a broader audience. If an Eph asked me if she could write regular entries about her efforts on this topic, I would gladly publish her, perhaps with a request that she put most of each entry "below the fold" (as I have done with this one) so that readers uninterested in the topic are not distracted. If people complained that this, like Derek's Red Sox Diary, did not belong on EphBlog, I would tell them to not read her stuff.

But, since I am the one interested in doing this, I can hardly judge my own case. So, regular readers, what do you think? I will probably post an entry or two just to give people a flavor. I like to think that this is great stuff. The substantive issue, mortality in Iraq, is important. The statistical details are subtle. Seeing how science works behind the scenes --- working papers, conference presentations, peer review --- is interesting.

And, if you find all this boring, don't read the entries!

------------------
August 6, 2007: "Not Credible"

The most damning comment about the Lancet estimates of mortality in Iraq came in the Wednesday afternoon session.

The presentation by Les Roberts focussed on the second Lancet survey (termed L2 by we aficionados). One discussant was Fritz Scheuren of NORC at the University of Chicago. Scheuren, besides being a distinguished statistician, is a past president of the American Statistical Association (the organization in charge of the entire conference).

Scheuren termed the response rate in L2 "not credible." (That is an exact quote. He used the phrase several times and clearly meant it.) Comments:

1) It was amazing to see such an accusation at a professional gathering. Scheuren was not suggesting that Roberts himself was guilty of fraud. Instead, he seems to believe that there is no way that the Iraqi interviewers achieved a 98% response rate.

2) Scheuren is an expert on surveys in conflict zones.

Fritz Scheuren, Ph.D., is a statistical consultant for HRDAG [Human Rights Data Analysis Group]. Recently, Dr. Scheuren consulted on the methods of statistical analysis for Peru's Truth and Reconciliation Commission and reviewed the report of the analysis. In late August of 2003, he visited Peru, along with Dr. Ball and Jana Asher, to meet with representatives of political, military, and civil society groups to explain the technical basis of the findings detailed in the report. In recent years, Dr. Scheuren has advised on the overall direction and approach to the statistical analysis for several HRDAG projects, including Guatemala and Kosovo. As a top statistician in the field, he has also provided critical peer-review of HRDAG work.

If a statistician of Scheuren's caliber and experience says that you have a problem, you have a problem.

3) Note that this point (a 98%+ response rate is ipso facto evidence of fraud) was what first made me (in)famous in Lancet circles last November. Here is a copy of the original post and associated links. Scheuren seems to agree with me. Does ace Lancet defender Daniel Davies consider Scheuren guilty of "devious hack-work?"

4) During the Q&A, Scheuren and Roberts got into a back and forth on data sharing. Scheuren made clear that, while he did not view the response rate as "credible," he did not think that it was absolutely impossible either. He was ready to be convinced if Roberts were to share more data with him. Specifically, Scheuren wants to see the results broke down by interviewer/team. For example, if one of the two main teams reported 100% response rates while the other reported 96%, Scheuren would (I think) be a lot more skeptical of the first team's results. Roberts mumbled something about that data not being on the computer (which is plausible). But Roberts also expressed no interest in providing Scheuren (or anybody else) with more data. At one point, Roberts said that, if it were up to him, the Lancet authors would not have shared any of their data with anyone else.

I will have further updates on what happened in Salt Lake City over the next couple of days. When the history of this sad saga is written, the moment when a past president of the American Statistical Society termed the Lancet numbers "not credible" will mark the end of the beginning of the debunking of these flawed studies.

Let me end with the same challenge that I have been issuing for the last year: Is there another survey like the one conducted by L2 (nationwide sample with a single contact attempt) for which the response rate is as high or higher than 98%? I have never been able to find one. Can anyone? Any survey in any country on any topic at any time? And, if we can't, what are the odds that one of the most controversial surveys of the decade should just happen, by chance, to have the highest response rate ever?

"Not credible," indeed.

[And from the comment thread to that post.]

Excellent catch on this article.

Statisticians grill surveyor of Iraqi casualties of war
By Paul Foy
The Associated Press

The courtroom-style questioning came in a packed ballroom in Salt Lake City at the world's largest gathering of statisticians.

I think that I asked the most "courtroom-style" question. More on that later.

On the hot seat: A globe-trotting researcher who says his team's surveys of Iraqi households projected nearly 655,000 had died in the war as of July 2006, a number still 10 times higher than conventional estimates.

Leslie F. Roberts and others from Johns Hopkins University took accounts of births and deaths in some Iraqi households to estimate that the country's death rate had more than doubled after the 2003 invasion.

Number crunchers this week quibbled with Roberts' survey methods and blasted his refusal to release all his raw data for scrutiny - or any data to his worst critics. Some discounted him as an advocate for world peace, although none could find a major flaw in his surveys or analysis.

I think that "blasted" is a fair description of the reaction to Roberts' refusal to share data with his critics. In terms of a "major flaw," too bad the reporter did not ask me! I give good quote.

However, Stephen Fienberg, a professor of statistics at Carnegie Mellon University in Pittsburgh, said: ''I thought the surveys were pretty good for non-statisticians.''

Harsh! More on what else Fienberg had to say soon.

Roberts, an epidemiologist, said he is opening a new front in the study of public health hazards: War. He has conducted about 30 mortality studies since 1990 in conflicts around the globe, including the Congo, where he was similarly accused of exaggerating war-related deaths.

Says who? I have never read of such an accusation. Has anyone? This sounds to me like something that Roberts spoon fed to the reporter. "See? I get this criticism all the time . . . "

Roberts organized two surveys of mortality in Iraqi households that were published last October in Britain's premier the medical journal, The Lancet. He acknowledged that the timing was meant to influence midterm U.S. elections.

Really? Roberts admitted this (sort of) with regard to L1 but has since backed away. I have never seen him admit this for L2. Has he? Perhaps this came up at lunch because I did not see Roberts do this in the main presentation.

''It puts you in a position where you are going to get attacked,'' said Fritz Scheuren, a senior fellow at the University of Chicago's National Opinion Research Center, who is trying to organize another Iraqi survey to see if he can match Roberts' results.

Scheuren, the American Statistical Association's former president, said he couldn't find anything wrong with The Lancet surveys.

This is not inconsistent with what I wrote above. Scheuren is a careful guy. He can not point to a flaw in the data that he has seen. He wants to get more detailed data. He still does not find the results "credible."

He complained, however, that he wasn't able to get Roberts to reveal which of his Iraqi surveyors conducted which surveys, information that could reveal any bias in workers who compile consistently implausible results.

Roberts said he won't release the researchers' identities for fear of exposing them to retaliation. The Iraqi government has strongly disputed the findings.

More heat (generated by Roberts?) than light. Scheuren does not want to know the "researchers' identities." None of us do. We just want to know that Researcher A reported this data; Researcher B reported that data; and so on. Roberts has an annoying habit of pretending that his critics are asking for names when he knows we just want to see if the results vary greatly by interviewer.

------------------
August 7, 2007: Holocaust Denial

Besides his formal presentation, Les Roberts also participated in a luncheon roundtable entitled "Media Coverage Regarding Data on Mortality in Iraq."

I was able to pull up and chair and listen in. Roberts is a charming and engaging speaker. Most of the ten or so people around the table were clearly fans. For me, the best part was when Roberts, perhaps playing a bit to his audience, compared people who disagree with the Lancet estimates to people who deny the Holocaust.

Nice!

I have heard of Roberts using this comparison in the past, but this was the first time that I got a chance to witness it myself. He comes close to making that accusation in his speech at Brown, but shies away from making the comparison explicit, perhaps because he knew that the tape was running. He frames the issue by pointing out (correctly!) that Iranian President Mahmoud Ahmadinejad makes people angry when he describes the Holocaust as a "myth." Roberts thinks that President Bush dismissing the Lancet estimates amounts to the same thing.

It is not clear if Roberts thinks that Fritz Scheuren, past president of the American Statistical Association, is no better than famous Holocaust denier David Irving. Clarification welcome!

Burnham Sanctioned

2009-02-26T06:06:00.000-08:00

Two weeks ago, I claimed that a partial victory for we Lancet skeptics would come with the "Censure of Roberts/Burnham by Johns Hopkins." I declare victory!

Iraq Researcher Sanctioned

Associated Press
Tuesday, February 24, 2009; Page A04

The Johns Hopkins Bloomberg School of Public Health is sanctioning the lead author of a 2006 study that suggested massive civilian deaths in Iraq.

The school announced yesterday that it is barring Gilbert M. Burnham from serving as a principal investigator on projects involving human subjects, saying he violated school policies by collecting the names of those interviewed.

The school completed an internal review of the study, which estimated that nearly 655,000 Iraqis had died because of the U.S.-led invasion and war in Iraq. The review found that inclusion of identifiers did not affect the results of the study.

The school says the paper published in the Lancet medical journal incorrectly stated that identifying data were not collected. A correction will be submitted.

Now, "sanction" is not the same thing as "censure," but it is close enough for blog-work. The original Hopkins news release is here. (Hat tip to Tim Lambert.) Key sections:

The Bloomberg School of Public Health’s IRB acted properly in determining that the original study protocol was exempt from review by the full IRB under federal regulations. The original protocol explicitly stated that no names of study participants or living household members would be collected. The protocol also included an appropriate script to secure verbal consent from study participants, rather than a written consent process that would have included participants’ signatures.

Note that the interests of Hopkins and Gilbert Burnham are not necessarily aligned, although both wish the whole debate would go away. Hopkins wants to ensure that the Federal Government isn't angry with it since Hopkins is so dependent on federal money. Hopkins' key concern is to demonstrate that the university did nothing wrong. If that means throwing Burnham under the bus, so be it.

An examination was conducted of all the original data collection forms, numbering over 1,800 forms, which included review by a translator.

...

A review of the original data collection forms revealed that researchers in the field used data collection forms that were different from the form included in the original protocol. The forms included space for the names of respondents or householders, which were recorded on many of the records.

If the interviewers weren't even using the correct form (and did not visit the correct governorates), then why should we believe that they did anything else correct?

Left unexplained is whose fault this is. Did Burnham hand Lafta the wrong form? On purpose? By mistake? Or did Burnham give Lafta the correct form and then Lafta just used the wrong one? On purpose or by mistake? Recall (pdf) that the process included two face-to-face meetings.

American and Iraqi team members met twice across the border in Jordan, first to plan the survey and later to analyze the findings.

My guess is that Burnham did everything correctly (he seems like a careful and professional guy to me) but then Lafta just screwed it all up, either out of sloppiness or because field work is hard, even for the most diligent of researchers. And then Burnham got in real trouble because, after the fact, he tried to cover for Lafta. It is always the cover-up, never the crime.

And that is the real issue for Burnham's credibility. If he had just fessed up immediately, then that would be one thing. But, instead, he made a series of misleading claims about what happened. Consider just six examples:

1) Burnham's February 2007 talk at MIT (video here).

There were limitations to record keeping; we had some criticism that we could not produce a record showing which households were visited and who names of people were, and so forth.

We intentionally did not record that, because we felt that if the team were stopped at a checkpoint, of which there are lots of checkpoints, and the records were gone through, some of you may have had this experience, where you stop at a checkpoint, people go through all your papers, read everything, and they find certain neighborhoods. That might have increased risk, which we didn't want to do.

That's not true. The survey forms included names and Burnham knew that at the time he gave his talk to MIT. Note that these comments were part of the main presentation, something that Burnham had probably given on multiple occasions, not an off-the-cuff answer to a question from the audience. If Burnham said this at MIT, he probably said it elsewhere.

2) Burnham wrote to blogger Tim Lambert in February 2008:

As far as the survey forms, we have all the original field survey forms. Immediately following the study we met up with Riyadh (in this very hotel I am now in) and Shannon, Riyadh and I went through the data his team had computer entered, and verified each entry line-by-line against the original paper forms from the field. We rechecked each data item, and went through the whole survey process cluster-by-cluster. We considered each death, and what the circumstances were and how to classify it. Back in Baltimore as we were in the analysis we checked with Riyadh over any questions that came up subsequently. We have the details on the surveys carried at each of the clusters. We do not have the unique identifiers as we made it clear this information was not to be part of the database for ethical reasons to protect the participants and the interviewers.

Again, this statement is incorrect. Burnham did have the full names of at least some (or many? or most?) of the survey participants.

3) Burnham in an interview with the National Journal.

Hopkins's guidelines say that substantial modification of survey procedures without approval by the university's IRB can be deemed a serious violation. "All intentional noncompliance is considered to be serious," the guidelines state.

The survey "was carried out as we designed it," Burnham told National Journal.

Not true. The survey design called, explicitly, for the names of participants to not be collected. But the names were collected. Even if one tries to defend Burnham (see below) by insisting that he did not realize, until recently, that full names were collected, we still have the problem that Burnham has always known that Lafta used the wrong forms, forms with a space for the names of the participants. Using the wrong form is, obviously, not carry out the survey "as we designed it."

4) A description in Johns Hopkins Magazine.

Concern for the safety of interviewers and respondents alike produced two more decisions. First, they would not record identifiers like the names and addresses of people interviewed. Burnham feared retribution if a hostile militia at a checkpoint found a record of households visited by the Iraqi survey teams.

Now, a Burnham defender might argue that a) Burnham is not quoted, so this misstatement is not his fault and b) there is no misstatement here since "would not" is not the same as "did not." Perhaps. But the article clearly featured extensive cooperation from Burnham (and Doocy). They have some responsibility to make sure that the author describes their work accurately. And, to the extent that a mistake is made, they have an obligation to correct it. If Johns Hopkins Magazine can not trust professors from Johns Hopkins to clearly describe their research, then what is the world coming to?

The point of this example is not so much that it is damning in and of itself. It isn't. The point is that it is part of a pattern of Burnham giving his listeners a false impressions of whether or not names were collected. He makes everyone think that names were not collected even though they were.

5) The official Q&A from the Hopkins page, since removed (along with other material) for obvious reasons.

Approval specified that no unique identifiers would be collected from households visited by researchers, including complete names, addresses, telephone numbers or other information which could potentially put the households at risk. While household demographics were collected in both Iraq studies, personal information such as the date, location and cause of death was collected only for deceased household members. Research regulations do not consider a dead person to be a human subject and informed consent is not required for uniquely identifying information on the characteristics and circumstances of death. Informed consent was obtained from the principal respondent in each household before interviews were conducted.

"Personal information" was, we now know, collected for some living household members. Although Burnham's name does not appear on this Q&A, he was and is the Director of Center for Refugee and Disaster Response. It is hard to believe that he did not sign off on the document before it was posted.

6) "The Human Cost of the War in Iraq" (pdf), the official companion piece to Burnham et al (2006).

The survey was explained to the head of household or spouse, and their consent to participate was obtained. For ethical reasons, no names were written down, and no incentives were provided to participate.

Since names were written down, this statement is also not true. One might defend Burnham's misstatement at the MIT lecture as being an honest mistake, made while speaking off-the-cuff. One might argue that an informal e-mail to a blogger like Tim Lambert does not require perfect accuracy. One might believe that interviewers from the National Journal and Johns Hopkins Magazine were mistaken. One might assume that that Burnham never saw/approved that Q&A on the homepage of the Center that he heads.

But here we have a formal report with a statement that Burnham knew was both untrue and important. (If it were a trivial issue, then Hopkins would not have sanctioned him and no correction to the Lancet would be necessary.) How many other untrue statements are there?

Even worse, all the other L2 authors are also authors of this paper. Now, it could be that some of those authors (like Les Roberts) never saw the actual forms and so might be guilty of nothing more than an honest mistake. But Shannon Doocy saw the forms. She knew that this statement was not true.

Is Burnham telling the truth even now? Good question! Consider this article from the Baltimore Sun:

The Johns Hopkins University has disciplined the lead author of a widely publicized study that reported widespread civilian deaths in Iraq as a result of the U.S. invasion.

Discipline, sanction, censure. Whatever. They all work for me!

Because of the difficulty of carrying out research in Iraq during the war, Burnham and his team partnered with Iraqi doctors at a university in Iraq. Burnham, working out of Jordan, said he made it clear to the doctors that they could collect the first names of children and adults, to help keep the information straight, but that last names could not be collected.

Huh? This seems really fishy to me. I assume that collecting first names was not a part of the protocol that Burnham submitted to Hopkins. I have certainly never heard of anything like this. (Counter examples welcome!) You either write down someone's name or you don't. So, Burnham knew, before the interviewers went out that they were using a form with a spot for names? And he didn't say, "Wait! That's not the form you are supposed to use." The whole thing makes no sense. Unless Burnham is trying to cover for Lafta, trying his best to make his actions in recording names less damning.

And that is key to defending the results of the study. If Lafta screwed this up, then there is every reason to think that he screwed up other stuff, either on purpose or by mistake. If he wrote down names when Burnham told him not to, then how does anyone know if he followed the assigned sampling plan? We don't.

But if Burnham can take the blame himself, can claim that he told/allowed Lafta to write down partial names, then, perhaps, the rest of Lafta's work can still be trusted. I do not believe that Burnham really told Lafta that he could write down partial names.

You disagree? Fine. Let's ask Shannon Doocy. She was with Burnham in Jordan. She was a co-author of the study. She knows Lafta. Presumably, she was in the room when Burnham/Lafta discussed the planing for the study. I bet that she can't/won't back up Burnham on this point. She's not 66 and tenured . . .

When the surveys came back to him in Jordan, it appeared that some had last names. Many were in Arabic. Burnham said he asked his Iraqi partners and was told that the names were not complete, which he accepted. But Hopkins, in its investigation, found that the data form used in the surveys was different from what was originally proposed, and included space for names of respondents. Hopkins found that full names were collected.

Oh, what a tangled web we weave . . .

If I were Baltimore Sun reporter Stephen Kiehl, I would ask some follow up questions:

1) How many of the 1,800+ forms had names? How many of those names were full names (as opposed to just first names)? Were those names all in Arabic, all in English or a mixture? How many of the names were in English?

"Many" might be 20 or 100 or 500. But, given that the forms featured lots of English words (cause of death and so on), I would bet that many (hundreds?) names were in English. So, Burnham looked at a name like "Nouri al-Maliki" or "Iyad Allawi" and said, "Sure. I accept that these names are not complete." Unlikely! Again, it would be nice to ask Shannon Doocy some questions as well.

2) What is Burnham's take on the fact that his "Iraqi partners" misled him? (Was this just Lafta or Lafta and some other interviewers? My understanding had been was that it was just Lafta who brought the forms to Burnham/Doocy in Jordan. And note that Lafta is not just a "partner," he is an official co-author on the study.) Here we have documented proof that one author of the study misled another author of the study. How much faith does Burnham think we should have in a study in which the authors are lying to each other?

3) Why did Burnham mislead so many news organizations? Consider his (and Roberts') letter (also here) to the National Journal:

In the ethical review process conducted with the Bloomberg School of Public Health's Institutional Review Board, we indicated that we would not record unique identifiers, such as full names, street addresses, or any data (including details from death certificates) that might identify the subjects and put them at risk.

Oh, snap! Did you catch that? I thought that this was just going to be one of many examples of where Burnham lied, after the fact, about not collecting names. But note that he is not lying! Sneaky! He and Roberts do not claim (falsely) to not have collected names. They claim (truthfully) to having told Hopkins that they "would not" record names. Hah! I did not catch that the first time around because I assumed a basic level of honesty from Burnham. My mistake! (Further discussion here.) Comments:

1) This is circumstantial evidence that, at least by this point, Roberts was in on the lie.

2) I suspect that "full names" rather than just "names" is used to preserve wiggle room on various dimensions.

Likely Correction?

2009-02-24T18:42:00.001-08:00

The Johns Hopkins statement includes:

The paper in The Lancet incorrectly stated that identifying data were not collected. An erratum will be submitted to The Lancet to correct the text of the 2006 paper on this point.

I am confused about how this is going to work. The only relevant statement that I could find in the paper was:

The survey purpose was explained to the head of household or spouse, and oral consent was obtained. Participants were assured that no unique identifiers would be gathered.

For all we know, this is a true statement. Perhaps this is what participants were told, albeit incorrectly. If that is so, then a hard core Lancet defender, like Les Roberts, might maintain that no correction is necessary. In fact, don't all the authors of a paper need to agree on any correction? I suppose that a single author, like Burnham, can say whatever he wants, but I wouldn't think that a single author has the right to make an official change unless the other authors agree. At most, a single author can just demand that his name be removed from the publication. And, say what you will about Les Roberts, but he is a head-strong fellow. Not that there is anything wrong with that! What if Burnham wants to make a correction but Roberts doesn't?

Perhaps there is some other reference in the paper that I have missed.

Moreover, wouldn't this also be a good time to correct the mistaken description in the paper of the sampling scheme. After all, even Roberts/Burnham admit that it is not accurate.

Anyway, I don't have a strong opinion on what will happen. I am just curious what others predict . . .

Misleading Statements on Unique Identifiers

2009-02-24T09:37:00.000-08:00

I had assumed that it would be easy to find places where Roberts/Burnham claimed that names were not collected. Turns out that it isn't. Looks like they knew that this fact might eventually come out so they tried to mislead without lying. Consider their letter to Science.

Those who work in conflict situations know that checkpoints often scrutinize written materials carried by those stopped, and their purpose may be questioned. Unique identifiers, such as neighborhoods, streets, and houses, would pose a risk not only to those in survey locations, but also to the survey teams. Protection of human subjects is always paramount in field research. Not including unique identifiers was specified in the approval the study received from the Johns Hopkins Bloomberg School of Public Health Committee on Human Research. At no time did the teams “destroy” details, as Bohannon contends. Not recording unique identifiers does not compromise the validity of our results.

Were Burnham/Roberts lying? Tough to say! I (and everyone else who read those words two years ago) assumed that the names of the participants were not collected. (We now know that at least some full names were recorded.) But the above passage is written in such a way as to give the impression that no names were collected --- "Not including unique identifiers was specified in the approval the study received" --- without explicitly stating that this had been done. Clever!

Recall how the paper itself handled the issue.

The survey purpose was explained to the head of household or spouse, and oral consent was obtained. Participants were assured that no unique identifiers would be gathered.

Again, this could be a truthful statement. Who knows what the interviewers said to the participants? But it completely misleads the reader into thinking that names had not been collected when, in fact, they had been collected.

So, hard core Lancet defenders will insist that no correction is necessary. Then why is Johns Hopkins insisting that one be made?

The paper in The Lancet incorrectly stated that identifying data were not collected. An erratum will be submitted to The Lancet to correct the text of the 2006 paper on this point.

If you agree with Hopkins that a correction is necessary, then you should also believe that Roberts/Burnham owe an apology/correction to, among others, the readers of Science. They were just as likely to have been misled.

But also consider this June 2007 letter to Science from Burnham, Roberts and Doocy.

At the time the study was published, Iraqi colleagues requested that we delay release, as they were very fearful that somehow streets, houses, and neighborhoods might be identified through the data with severe consequences. We agreed to wait for 6 months and have now made the data available.

From the beginning, we have taken the position that protecting participants is paramount, and thus we will not be releasing any identifiers below the level of governorate. The demand by some for street names seems to arise from the erroneous belief that only main streets were sampled, when in fact, where there were residential streets that did not intersect with the selected commercial street, these too were included in the sampling frame for identification of the start house. In any event, our interviewers reported that most, although not all, violent deaths occurred away from the location of residence.

Does this make much sense? To the extent that their "Iraqi colleagues" were "very fearful," it would not be because of specific neighborhoods being identified via statistical magic but because the full names of participants could be released. That's the real problem. By pretending otherwise, Burnham/Roberts/Doocy continued to mislead all of us into thinking that the worst conceivable data release (from the point of view of participant safety) might allow identification of a specific street when, in fact, the worst possible release would involve the actual names of participants. Again, there is nothing in the above that is literally a lie --- Who knows what their "Iraqi colleagues" told them? --- but there is no doubt that readers were misled, again, into thinking that names had not been recorded.

L2 Sampling Details

2009-02-19T18:28:00.000-08:00

This post serves to collect information from various sources about the details of the L2 sampling procedure.

Summary: L2 authors have given different (and conflicting) accounts of exactly what the interviewers did and/or were supposed to do. They have made no final statement about what sampling plan was followed. Anyone who claims to "know" what the sampling plan in L2 was is lying.

Let's begin with the description from the paper itself.

As a first stage of sampling, 50 clusters were selected systematically by Governorate with a population proportional to size approach, on the basis of the 2004 UNDP/Iraqi Ministry of Planning population estimates (table 1). At the second stage of sampling, the Governorate’s constituent administrative units were listed by population or estimated population, and location(s) were selected randomly proportionate to population size. The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed. For this study, a household was defined as a unit that ate together, and had a separate entrance from the street or a separate apartment entrance.

The problem with this description, as highlighted by the "main street bias" work in Johnson et al (2008), is that there are houses/streets that are not included in the sampling frame. Consider this figure from Johnson et al.

It is obvious that some houses are not within the sample frame.

Now, just because the stated procedure has this problem does not invalidate it. Given the time/resource constraints faced by L2, this approach is perfectly reasonable. The critical thing is not so much the exact approach used as it is transparency on the part of the L2 authors as to their procedure.

The best single source for an overview of L2 sampling issues is Mike Spagat's "Ethical and Data-Integrity Problems in the Second Lancet Survey of Mortality in Iraq" (pdf), forthcoming in Defence and Peace Economics. Below are some of the key sections:

The authors of L2 have still not fully disclosed their sample design (Bohannon, 2008, Spagat, 2007). Gilbert Burnham and Les Roberts have stated frequently that the L2 field teams did not follow the sampling methodology that was published in the Lancet but they have not supplied a viable alternative. Burnham and Roberts have also issued a series of contradictory statements about their sampling procedures and have either destroyed or not collected evidence necessary to evaluate these procedures.

See Spagat for citations.

These lists of main streets are at the core of the claimed sampling methodology. Yet, the L2 authors have refused to provide these lists or even clarify where they came from.

The footnote associated with this sentence is particularly damning.

For example, Seppo Laaksonen, a professor of survey methodology in Helsinki, requested and was denied any information on main streets, even the average number of main streets per cluster (Laaksonen, 2008).

This is the sort of behavior that makes me (and most statisticians) incredibly suspicious of L2. It is fine, perhaps, for the L2 authors to ignore me. I am just some random guy on the internet. It is fine, perhaps, for them to ignore Spagat et al since Spagat has been so critical of their work. But to refuse to answer a question from a professor of survey methodology with no particular axe to grind is just pathetic. If you are an academic, you answer questions from your fellow academics.

Back to Spagat:

Gilbert Burnham did make aspects of the sampling methodology fairly concrete in Biever (2007), an interview with the New Scientist.

“The interviewers wrote the principal streets in a cluster on pieces of paper and randomly selected one. They walked down that street, wrote down the surrounding residential streets and randomly picked one. Finally, they walked down the selected street, numbered the houses and used a random number table to pick one. That was our starting house, and the interviewers knocked on doors until they’d surveyed 40 households…. The team took care to destroy the pieces of paper which could have identified households if interviewers were searched at checkpoints.” (Biever, 2007, emphasis added.)

Whatever its strengths or weaknesses, this does seem to be a procedure that can be followed in the field. The L2 authors may no longer be able to specify their sample design since these pieces of paper have been destroyed. But they should be able to supply lists of principal streets or at least specify how many such streets there were per governorate.

Burnham explains that the sampling information was destroyed to protect the identities of respondents, but this explanation is inadequate. Pieces of paper with lists of principal streets and surrounding streets would be of no use for identifying households included in the survey. Even lists of all of the households on a street that was actually sampled would not be usable for identifying particular L2 respondents. On the other hand, the L2 data-entry form that Riyadh Lafta submitted to the WHO contains spaces for listing the name of each head of household in addition to names of people who died or were born during the L2 sampling period. If the field teams could travel around with pieces of paper containing the names of their respondents plus many of their family members then they did not have to destroy lists of streets. Finally, as noted above in section 2, the lists of L2’s respondents would have been widely known at the local level in any case.

The L2 authors have often dismissed the possibility of sampling bias by stating that they did not actually follow the sampling procedures that they claimed to have followed in their Lancet publication. For example, Burnham and Roberts (2006a) write that they had removed the following sentence from their description of their sampling methodology at the suggestion of peer reviewers and the editorial staff at the Lancet:

"As far as selection of the start houses, in areas where there were residential streets that did not cross the main avenues in the area selected, these were included in the random street selection process, in an effort to reduce the selection bias that more busy streets would have." (Burnham and Roberts, 2006a)

Combining this with Gilbert Burnham’s New Scientist interview already quoted (Biever, 2007) would imply that at each location:

A. Field teams wrote names of main streets on pieces of paper and selected one street at random.

B. The field teams then walked down this street writing down names of cross streets on pieces of paper and selected one of these at random.

C. The field teams then became aware of all other streets in the area that did not cross the main avenues and may have selected one of these instead of one of the cross streets written on pieces of paper. This wide selection was done according to an undisclosed procedure.

The Biever (2007) description of Burnham does outline a sampling procedure that could have been followed and is broadly consistent with the published methodology. If other types of streets, beyond those that would be covered by the published methodology, were included in the sampling procedures then the authors need to specify how these streets were included. More fundamentally, how did the field teams discover the existence of such streets that could not be seen by walking down principal streets as described by Burnham in Biever (2007)? The L2 field teams would not have brought detailed street maps with them into each selected area or else it would not have been necessary to walk down selected principal streets writing down names of surrounding streets on pieces of paper. We can also rule out the possibility that the teams completely canvassed entire neighborhoods and built up detailed street maps from scratch in each location. Developing such detailed street maps would have been very time consuming and the L2 field teams had to follow an extremely compressed schedule that required them to perform forty interviews in a day (Hicks, 2006).

In Giles (2007), an article in Nature, Burnham and Roberts suggested one possible explanation on how the field teams had managed to augment their street lists beyond streets that could be seen by walking down a main street but this suggestion was rejected by an L2 field-team member interviewed by Nature:

“But again, details are unclear. Roberts and Gilbert Burnham, also at Johns Hopkins, say local people were asked to identify pockets of homes away from the centre; the Iraqi interviewer says the team never worked with locals on this issue.” (Giles, 2007)

Even if locals had identified such “pockets of homes away from the centre” the authors still would have to specify how these were included in the randomization procedures. Indeed, involving local residents in selecting the streets to be sampled would seem to be at odds with random selection of households. Locals could, for example, lead the survey teams to particularly violent areas.

Burnham and Roberts have induced further confusion about their sample design by issuing a series of contradictory statements.

"The sites were selected entirely at random, so all households had an equal chance of being included." (Burnham et al, 2006b, emphasis added)

"Our study team worked very hard to ensure that our sample households were selected at random. We set up rigorous guidelines and methods so that any street block within our chosen village had an equal chance of being selected." (Burnham and Roberts, 2006b, emphasis added)

“… we had an equal chance of picking a main street as a back street.” (The National Interest, 2006).

These statements contradict each other and the methodology published in the Lancet. Some streets are much longer than others. Some streets are much more densely populated than others. Such varied units cannot all have equal probability of selection. If, for example, every street block had an equal chance of selection then households on densely populated street blocks would have lower selection probabilities than households on sparsely populated street block. If main streets are more densely populated on average than back streets are and main streets and back streets have equal selection probabilities then households on main streets would have lower selection probabilities than households on back streets.

All this is bad enough. Yet Lancet defenders assert that these contradictions are unimportant, that such discrepancies are inevitable when researchers try to explain their work to the public, that much of the "conflict" is just caused by a reasonable effort on behalf of the Lancey authors to simplify.

And that is a reasonable defense. But, to work, there must come a point at which we can point to a specific statement that describes the sampling plan used. What is the truth? Normally, the truth would be whatever is written in the published paper, but the Lancet authors now claim that the published work is not accurate. (And they have failed to publish a correction in the Lancet.) The truth could also be the official sampling plan that the authors created prior to starting the project. They must have written up a plan ahead of time. If they were to release that now, they could point to that.

So, if the truth is not in the paper, or in a correction to the paper or in an official document released by the authors, where is it? Good question! No one knows.

Burnham gave a presentation at MIT in February 2007. (See here for video, transcript and extensive commentary.) Let me quote my prior commentary.

Burnham claims that they did not restrict the sample to streets that crossed their main streets. Instead, they made a list of "all the residential streets that either crossed it or were in that immediate area." This is just gibberish.

First, if this was what they actually did, why didn't they describe it that way in the article? Second, given the time constraints, there was no way that the teams had enough time to list all such side streets. Third, even if the interviewers did do it this way, the problem of Main Street Bias would still exist, except it would be more Center Of Town Bias. Some side streets are in the "immediate area" of just one main street (or often in the area of none) and other side streets (especially those toward the center of a town or neighborhood) are near more than one. The later are much more likely to be included in the sample.

Again, it is not unreasonable for there to be some confusion in the immediate aftermath of publication about exactly what the interviewers did. My point here is not to attack Burnham for an in artful phrase like "equal chance" as used in an interview. But, given that this was such an important matter of public concern, I would expect Burnham to have his story straight by the time that he spoke at MIT, 4 months after publication.

But things are even worse that that!

Consider the Q&A that the Lancet authors posted in early 2008 on the Johns Hopkins web page. Note the description of the sampling:

Sampling for the 2006 study was designed to give all households in Iraq an equal chance of being included.

This statement can not possibly be true. How can the authors, more than a year after the study was published, assert such an obvious falsehood about the single most important criticism of the study?

But, even though this is a false statement, it is at least a clear one. Want to know what the official sample plan was? Don't check the paper. (It's wrong.) Don't look for corrections. (None were made.) Don't look for the original planning documents. (They were never released.) But, at least there is an official statement on the Johns Hopkins website. When Tim Lambert and other Lancet defenders want to claim that the sampling plan was such and such, they could link here as evidence.

But then the Lancet authors deleted that page! (This was in a conjunction with various other false statements made on the web page as well as other unexplained deletions.) So, the official word is that there is no official word. The last clear statement made by the Lancet team --- a statement that, one assumes, supersedes the published paper and other interviews given by the authors --- was removed from the web without any explanation.

Anyone who claims to know the sampling plan used in L2 is lying.

Pope of Debunkers

2009-02-15T18:35:00.000-08:00

Robert Shone's blog, Dissident 93, covers a variety of Lancet topics. Here is a good summary of why so much of the criticism of the Main Street Bias work is incoherent. Via his blog is also this useful Science article.

www.sciencemag.org SCIENCE VOL 319 18 JANUARY 2008 273
CREDIT: ADAPTED FROM C. A. BROWNSTEIN ET AL., NEJM 358, 5 (2008), MASSACHUSETTS MEDICAL SOCIETY
NEWS OF THE WEEK
A team led by the World Health Organization
(WHO) has produced a new estimate of
the number of Iraqis who died violently in
the first 40 months following the U.S.–led
invasion: between 104,000 and 223,000.
This figure, published online last week by
the New England Journal of Medicine, hews
close to some other attempts to quantify the
toll but comes in far below a controversial
2006 study led by researchers at Johns Hopkins
University in Baltimore, Maryland.
That group estimated approximately
600,000 violent deaths during the same
period. The discrepancy has prompted critics
to renew their charge that the Johns Hopkins
results are not credible.
Data from a war zone are never fully reliable;
the best researchers can hope for is
“getting the numbers roughly right,” says
Fritz Scheuren, a statistician at the University
of Chicago in Illinois and past president
of the American Statistical Association.
Escalating violence in Iraq after 2003 put a
limit on quality control, but researchers do
have a quantitative starting point: the casualty
tally made by Iraq Body Count, a nonprofit
advocacy group based in London. By
controlling for multiple accounts of the same
car bombs and shootings, the group estimates
from media reports that between
81,000 and 88,000 violent deaths have
occurred in Iraq since the invasion. The figure
is useful as “a lower bound on the true
number,” says Jon Pedersen, a statistician at
the Fafo Institute for Applied International
Studies in Oslo, Norway.
To get the upper bound, says Pedersen,
you have to knock on doors in what is known
as a two-stage cluster survey. That’s the
method used by the WHO and Johns Hopkins
teams, among others. Researchers divide the
country into regions and then sample clusters
of households within each. Finally, they
extrapolate mortality rates from those clusters
to the total population.
Epidemiologists Les Roberts and Gilbert
Burnham of Johns Hopkins published the
first Iraq cluster study in November 2004 in
The Lancet. They used data collected by
Roberts and an Iraqi team, which, in September
2004, surveyed 988 households in
33 clusters across the country. They arrived
at a figure of 98,000 “extra” deaths since the
invasion, about half due to violence. Soon
after this, a team led by Pedersen and the
United Nations Development Programme,
which had used a much larger sample of
21,668 households in 2200 clusters, produced
an estimate for roughly the same
period of about 25,000 violent deaths.
As the invasion gave way to occupation
and insurgency, Rober ts and Burnham
mounted another study. This time they left
the surveying entirely to the Iraqi team,
communicating from abroad. Published in
October 2006 in The Lancet, the second
survey—based on 1849 households in
47 clusters—estimated that 601,000 Iraqis
died violent deaths between the 2003 invasion
and July 2006. To many, the number
seemed unrealistically high. Some also
faulted the authors for not fully answering
questions about the survey’s methods
(Science, 20 October 2006, p. 396).
Now comes the WHO survey. Conducted
with the help of the Iraqi government, it is by
far the most comprehensive mor tality
assessment to date. Interviewers visited
9345 homes in more than 1000 clusters. But
its estimate of 151,000 violent deaths has
come in for some criticism, too. Unlike other
Iraq casualty surveys, this one includes an
upward adjustment of 35% to account for
“underreporting” of deaths due to migration,
memory lapse, and dishonesty. “That is
really an arbitrary fudge factor,” says
Debarati Guha-Sapir, an epidemiologist at
the WHO Collaborating Centre for Research
on the Epidemiology of Disasters in Brussels,
Belgium. But the number falls squarely
within the range produced by a meta-analysis
of al l avai lable mor tal i ty studies by
Guha-Sapir and fellow centre epidemiologist
Olivier Degomme. The Johns Hopkins
figure is an outlier, she says.
Why the Hopkins study came up with
such a high figure is not clear. Criticism of
the study has in fact intensified since Burnham
and Roberts released a data set to
selected peers last year. “It did
not include the standard kinds of
data,” says Seppo Laaksonen, a
statistician at the University of
Helsinki in Finland and a specialist
in survey methodology.
For example, he says, it was
impossible “to check the objectivity
and randomness of cluster
selection.” Scheuren, who also
received the data, wanted to
compare results obtained by different
interviewers to “get a handle
on noise” and check for fabrication
by surveyors. Rober ts
declined to provide all the
details, according to Scheuren,
saying that he was concerned
that this would risk the safety of
the interviewers.
Burnham told Science, however,
that the Johns Hopkins
team does not have such detailed
information. “Our goal was to
reduce any type of risk to the community
and the par ticipants,” says Burnham.
“While we have much of the raw data, we
requested that anything designating the
interviewers or the location of the neighborhoods
visited not be sent to us.” Laaksonen
responds that he would not have published
“any f igures for the country” if he didn’t
have direct access to such raw information
from surveyors.
Burnham is not retreating. Because the
WHO survey was conducted by Iraqi government
personnel, “people may have been
hesitant to answer honestly,” he says. He
claims that unlike those in the WHO study,
nearly all of the deaths tallied by the 2006
Lancet study were verified with death certificates.
Even if the debate may be drawing
to a close about whether the number of violent
deaths in postinvasion Iraq could be as
high as 600,000, the argument about methods
is clearly far from settled.
–JOHN BOHANNON
Downloaded from www.sciencemag.org on January 17, 2008

End Game

2009-02-11T06:42:00.000-08:00

How will the controversy over the Lancet articles end? Good question. I can't predict the future, but here are some benchmarks for defining "victory," at least for we Lancet critics.

1) Both Roberts et al (2004) and Burnham et al (2006) are withdrawn by the Lancet. This is the primary goal that many of us are aiming for. If scientists don't police the accuracy of the scientific literature, then who will?

2) The Lancet authors are forced to be more transparent in their research. There is still some (small) chance that the Lancet papers are accurately, or at least not fraudulent. If the authors were to provide access to their data/methods to all researchers, then that would be a victory for replication and the scientific process. My primary motivation from the start has been to insistent on the importance of scientific work that is open to replication. That is the primary norm that I want to defend. If the Lancet authors don't have to share their data or explain their statistics, then why should any other scientist have to?

3) Censure of Roberts/Burnham by Johns Hopkins or (in the case of Roberts) Columbia. If leading research universities don't uphold the norms of open and transparent scientific inquiry, who will? The most interesting aspect of the recent censure of Burnham by AAPOR was news of an open investigation by Johns Hopkins.

In a highly unusual rebuke, the American Association for Public Opinion Research today said the author of a widely debated survey on "excess deaths" in Iraq had violated its code of professional ethics by refusing to disclose details of his work. The author's institution later disclosed to ABC News that it, too, is investigating the study.

AAPOR, in a statement, said that in an eight-month investigation, Gilbert Burnham, a faculty member at the Johns Hopkins Bloomberg School of Public Health, "repeatedly refused to make public essential facts about his research on civilian deaths in Iraq."

Hours later, the school itself disclosed its own investigation of the Iraq casualties report "to determine if any violation of the school's rules or guidelines for the conduct of research occurred." It said the review "is nearing completion."

...

The inquiry by the Johns Hopkins Bloomberg School was disclosed in an e-mail from Tim Parsons, the school's public affairs director, as follows:

"The level of civilian mortality in Iraq is a controversial subject. Questions have been raised regarding the findings and methodology of the 2006 Iraq mortality study conducted by Dr. Gilbert Burnham and published in The Lancet. The Johns Hopkins Bloomberg School of Public Health takes any allegation of scientific or professional misconduct very seriously. It believes that the correct forum for discussing the reported findings of the Lancet study and the general methodology that led to those findings is in the regular exchange of views in the scientific literature. The Bloomberg School of Public Health has undertaken a review of the study to determine if any violation of the school's rules or guidelines for the conduct of research occurred in the conduct of the study. That review is nearing completion and the school is unable to discuss the results at this time."

I hesitate to speculate on what Hopkins will conclude.

4) A change in policy by the Lancet to require authors to allow for replication of their work. More and more journals are moving in this direction and I expect the Lancet to follow suit at some point.

Will any of these come to pass? I don't know. But, heading into year 5 of this controversy, I am pleased with the progress that we have made so far.

Tiny Chance

2009-02-06T17:39:00.000-08:00

The question of how the 2004 Lancet authors viewed the Fallujah "outlier" is often of interest. Consider this letter to the editor of The Independent.

You reported ("Polish hostage held in iraq is released unharmed", 21 November) the Foreign Secretary's response to our study published in The Lancet of civilian deaths in Iraq. It is heartening that Jack Straw has addressed the topic in such detail. However, his response includes an apparent misreading of our results.

Our study found that violence was widespread and up 58-fold after the invasion; that from 32 of the neighbourhoods we visited we estimated 98,000 excess deaths; and that from the sample of the most war-torn communities represented by 30 households in Fallujah more people had probably died than in all of the rest of the country combined.

Fallujah is the only insight into those cities experiencing extreme violence (ie Ramadi, Tallafar, Fallujah, Najaf); all the others were passed over in our sample by random chance. If the Fallujah duster is representative, there were about 200,000 excess deaths above the 98,000.

Perhaps Fallujah is so unique that it represents only Fallujah, implying that it represents only 50-70,000 additional deaths. There is a tiny chance that the neighborhood we visited in Fallujah was worse than the average experience, and only corresponds with a couple of tens of thousands of deaths. We also explain why, given study limitations, our estimate is likely to be low. Therefore, when taken in total, we concluded that the civilian death toll was at least around 100,000 and probably higher, not between 8,000 and l94,000 as Mr. Straw states. While far higher than the Iraq Ministry of health surveillance estimates, on 17 August the minister himself described surveillance in Iraq as geographically incomplete, insensitive and missing most health events.

We, the occupying nations, should aspire to acknowledge the dignity of every life lost, and to monitor trends and causes of deaths to better serve the Iraqis, and in doing so, sooner end this deadly occupation.

Les Roberts, Gilbert Burnham Centre for International Emergency, Disaster and Refugee Studies, Johns Hopkins Bloomberg School of Public Health, Baltimore; Richard Garfield School of Nursing, Columbia University, New York, USA

I would not be surprised if the initial version of the paper that they submitted to the Lancet included the Fallujah data without much/any discussion of what an outlier it was. The authors clearly believed that the data from Falluja is representative of what was happening in large parts of Iraq. Perhaps the reviewers made a stink, thus leading to the sometimes-Fallujah-in-sometime-Fallujah-out nature of the final paper.

UPDATE: Whoops! I blogged about this letter before.

Counterfactual

2008-11-24T12:55:00.001-08:00

I can't vouch for most of the analysis here (and is it tongue-in-cheek?), but it is an interesting take on the question of what mortality would have been in Iraq in a counterfactual world without the US-led invasion and occupation.

The sanctions regime, which began in 1990, destroyed Iraq’s economy (reducing GDP by as much as three quarters) and impoverished millions of Iraqis. Particular attention was given at the time to its effect on children. The contemporary critics of the sanctions pointed out that before the sanctions began, the child mortality rate was about 50 per 1000; during the sanctions, on one accounting the rate soared to about 128 per 1000 (click on "basic indicators" here). More conservative estimates were in the range of a doubling of child mortality. Using the more conservative estimate, at one million births per year, this works out to an annual difference of 50,000 children surviving to the age of 5 (for various qualifications, see here). Today, the child mortality rate is below the pre-sanctions figure, and so every year in excess of 50,000 more Iraqi children survive than during the sanctions. The data are hotly contested but the trends are unmistakable and will continue to strengthen if security improves. Meanwhile, violent deaths of civilians, while still far too high, are declining; a very cautious estimate of 500-800 per month, based on the most recent reports on the Iraq Body Count website, is much lower than the avoided deaths of children compared to the sanctions regime. A conservative estimate is that more than 40,000 Iraqis survive per year today than during the sanctions regime, and probably most of them children. The tight correlation between GDP and child mortality across countries bolsters this conclusion.

Let’s suppose that the sanctions regime had continued for 10 years, from 2003 to 2013, and further that security flattens out—it doesn’t get worse, but it doesn’t get better. Under these assumptions, 400,000 Iraqi children would have died if the war had not occurred and the sanctions regime continued. Now, almost 100,000 Iraqis died during the war, and so one of the war’s benefits is that it saves the lives of 300,000 Iraqis (over 10 years).

I have always been suspicious of child mortality estimates during the sanctions period, and of old data in general. I just link to this piece to highlight the difficulties in honestly estimating "excess deaths." All of this also ignores the question of what Saddam's policies would have been had he stayed in power. If he would have been as "peaceful" in the future as he was in 2002, then excess deaths is one number. But, if he had become less peaceful (more in line with the previous decades of neighbor-invasion (Iran, Kuwait) and internal-slaughtering (Shia, Kurds), then excess deaths are much lower, even negative, as a result of the invasion.

But, as always, my main focus is on the Lancet surveys. They make an explicit comparison to mortality in 2002 to March 2003, so these discussions are largely irrelevant. (Hat tip Andrew Sullivan.)

IBC Estimates

2008-11-16T10:24:00.000-08:00

I am a big fan of IBC but, at the same time, it is important to remember that they are reliant on news reports. This can lead, obviously, to under-counting since not every death makes the news. But it can also lead to over-counting since the news is not always reliable.

Consider this AP article from November 15. (Thanks to a friend for the pointer.)

Counting the dead gets more complicated in Iraq

By KIM GAMEL – 1 day ago

BAGHDAD (AP) — This much is agreed — a double bombing in Baghdad struck a school bus and those responding to the first blast. But the difference in casualty figures was stark. Iraqi officials said 31 people died; the U.S. military put the death toll at five.

The conflicting reports from Monday's attack are emblematic of a spate of recent bombings that have raised fears of a resurgence in violence.

There have always been disagreements because accurately counting the dead in the chaos of Iraq's war has never been easy. Yet discrepancies appear to be widening as the political stakes grow.

U.S. officials privately say that some officials in the Shiite-dominated Interior and Health ministries could possibly have political or personal motives for inflating casualty numbers for bombings in mainly Shiite areas.

Iraqi officials insist their tallies are more accurate, saying the figures are based on death certificates issued by hospitals and the number of wounded who receive treatment.

The increase in attacks comes at a sensitive time for Iraq, with political tensions heating up ahead of provincial elections that are due to be held by Jan. 31 and are expected to shift some power to the disaffected Sunni minority.

Sunni and Shiite extremists also oppose negotiations that are under way over a proposed U.S.-Iraqi security pact that would extend the presence of American forces in the country beyond the end of the year.

U.S. officers acknowledge the difficulties in establishing accurate numbers but express confidence their figures come close thanks to an increased presence of American troops who have spread throughout the community to work with their Iraqi counterparts.

American soldiers race to the site of bombings, often with the Iraqi security forces that are responding to the attacks. Those troops interview witnesses as well as rescue crews to reach a consensus on casualties and the type of attack, the military said.

"We do have enough soldiers throughout the battle space that when there is an explosion, we hear it and basically we run to the sound of the gun," said Lt. Col. Steve Stover, a spokesman for U.S. forces in Baghdad.

If the U.S. military is unable to promptly reach the scene, it depends on casualty reports from American transition teams working with the Iraqi security forces that do.

It's a change from earlier in the war, when American troops focused on fighting insurgents and often declined to provide civilian casualty information.

Adel Muhsin, a top Health Ministry official, said the Americans generally aren't getting the full picture, pointing out that victims often die of their wounds at the hospital or on the way there.

"The U.S. figures are based on the preliminary reports. Usually, the first view is not completely dependable because the site is still chaotic," Muhsin said. "Preliminary and initial figures taken shortly after the explosion tend to be small."

John Pike, a military and security analyst who runs the respected Web site GlobalSecurity.org, said methodology and politics on both sides could be at play in the differing numbers.

He noted that followers of anti-U.S. Shiite cleric Muqtada al-Sadr, a rival of Prime Minister Nouri al-Maliki's government, wield influence in the Health and Interior ministries.

"I could easily imagine that the U.S. government would want to get a low number, which would suggest that peace is at hand, and that al-Sadr might want a high number to suggest that maybe al-Maliki is not doing such a good job after all," Pike said. "You could imagine both methodological and political explanations for the discrepancy."

The issue came to the fore this week when a series of bombings targeting Shiite areas over three consecutive days rattled the growing sense of confidence among Iraqis about security gains.

_ On Monday, the Interior Ministry said 27 people were killed in the bus attack. Police and hospital officials contacted by Associated Press staff put the death toll at 31, which would make it the deadliest blast in the capital in six weeks. The U.S. military, citing Iraqi army figures, said five were killed.

_ On Tuesday, Iraqi police and hospital officials said three people died when twin blasts hit a newspaper delivery truck and nearby vendors. The U.S. military said 18 people were wounded, including three Iraqi policemen and 15 civilians.

_ On Wednesday, Iraqi police and hospital officials said 23 people were killed in a series of bombings in mostly Shiite areas of the capital. The U.S. military said one civilian died and 46 were wounded in four separate attacks.

Iraq Body Count, an independent organization that tracks media reports as well as official figures, put the death tolls at 27 in Monday's attack, four on Tuesday and 28 on Wednesday.

So, IBC estimates 59 civilian deaths for these three attacks. The US military estimates 6 dead (assuming no deaths in the Tuesday attack). Who is right? I don't know. But the insistence by many that IBC represents a minimum estimate is suspect.

The are other similar examples. Consider these two strories (here and here) from Bob Owens. Both of these incidents are still included in the IBC database (here and here) even though Owens provides fairly compelling evidence that the incidents were fake.

Does that reflect poorly on IBC? No! They are doing what they have always promised to do: catalog reported civilian casualties. They do an amazing job, especially by showing everyone their raw data and including supporting details. It is not their fault that media reports are sometimes false. And, in these cases, the specific reports could be true. The evidence presented by Owens is powerful but not conclusive. And, to be fair to Owens, it is impossible for him to prove a negative, that these deaths did not occur.

Yet this means that those who insist that IBC must be a (dramatic) under-estimate should take more seriously the claim that not every reported death actually occurred. In at least these cases, there is reason to believe that the IBC count is too high.

Spagat's Latest

2008-08-18T14:10:00.000-07:00

Mike Spagat writes to me and his fellow panelists from JSM.

I've just posted a new paper on my website which can be considered a companion piece to the one I presented at JSM. Comments are very welcome.

http://personal.rhul.ac.uk/uhte/014/Mainstreaming.pdf

I've also posted a summary of my JSM paper:

http://personal.rhul.ac.uk/uhte/014/Summary.pdf

Then, having received some worthwhile comments, I've slightly modified my presentation:

http://personal.rhul.ac.uk/uhte/014/Denver.pdf

This is great stuff. I still disagree with Mike about whether or not there is "evidence for extrapolation" in L2 (see page 18 of his Denver presentation), but the rest of his analysis is spot on. If you are new to this debate, start with Mike's "Mainstreaming" paper at the first link above.

Peer Reviewed Articles Critical of L1/L2

2008-08-18T13:47:00.000-07:00

Here are 4 peer-reviewed, published articles that are highly critical of different aspects of the Lancet articles. I agree with much of this and disagree with some. But there is no doubt that lots of experts in public health and/or statistics find the work of Roberts et al to be deeply suspect.

Daponte (2007), Wartime estimates of Iraqi civilian casualties. International Review of the Red Cross, 89(868), 943-957.

Johnson, N., Spagat, M., Gourley, S., Onnela, J., Reinert, G, (2008), Bias in epidemiological studies of conflict mortality, Journal of Peace Research, 45(5), 653-664.

Laaksonen, Seppo, (2008). Retrospective Two-Stage Cluster Sampling for Mortality in Iraq. International Journal of Market Research 50, 3, 403-417.

Rosenblum, M. and van der Laan, M.J. (2008) Confidence Intervals for the Population Mean Tailored to Small Sample Sizes, with Applications to Survey Sampling. International Journal of Biostatistics (to appear in June issue). Technical Report Available at: U.C. Berkeley Division of Biostatistics Working Paper Series. Working Paper 237.

Lancet Comments on My Article

2008-08-18T13:36:00.000-07:00

I sent a draft of my article on confidence intervals in L1 to the Lancet last year. My goal was to get them to force Roberts et al to answer some questions. The Lancet editor though it best to consider my article for publication. That was fine with me, although I agree with the eventual denial that my paper is too late and narrow to merit inclusion in the Lancet. For those who care, who are the reviewers thoughts, most of which I agree with (especially with regard to my silly mistake in the formula for covariance). I publish them here (with permission) since other Lancet aficionados may find them interesting, especially the APPENDIX at the end which seems to reveal some secrets to the original peer-review for L1.

Manuscript reference number: THELANCET-D-07-05168
Title: Comments on the Confidence Intervals of Roberts et al. (2004)

Dear Dr Kane,

Many thanks for submitting your manuscript to The Lancet. Clearly we felt we should seek advice on this and following external peer review, several editors here have discussed the manuscript. Essentially the reviewers all advise us that they are comfortable with the Roberts et al analysis but less comfortable with the arguments you advance. Our decision is, therefore, that we should not publish your submission.

The reviewers' comments and some editorial points that may be of interest to you are presented in the paragraphs below. I hope you find these comments helpful.

Reviewer #1: The main message of this paper is that there is a disagreement among
the three 95% confidence intervals for the variables CMRpost, CMRpre
and
RR = CMRpost
CMRpre
,
published by Roberts et al. in The Lancet 364: 1857-1864, (2004). For showing this the author assume that the published 95% confidence intervals for the variables CMRpost and CMRpre are correct, and then argues that the 95% confidence interval for the variable RR is not. An argument based on the calculation of the probability Pr[CMRpost - CMRpre < 0] is used.
My first comments is: I do not think the author needs to use this argument.

All we need is a distribution for RR and to derive from it the 95% confidence interval. But what statistical model should be considered for the variable RR? Roberts et al. �page 1859 second column� seem to assume a log-linear regression model but nothing is said about the regressors they used.

In this situation the simplest thing to do is to consider a normal distribution for the CMRs, as done in the paper. Under normality I have computed the confidence interval for RR, and find that it does not coincide with that given by Roberts et al.

Indeed, assuming, as in the paper, that CMRpre follows the normal distribution N(5, .662), and CMRpost is N(12.3, 5.562) distributed for which the 95% confidence intervals for CMRpre and CMRpost are the published (3.7, 6.3) and (1.4, 23.2), respectively, and assuming that the variables CMRpre and CMRpost are uncorrelated, the density of RR, whose values are denoted by w, turns out to be

f (w) = 1 Z 8 -8 |t| exp �-(t w - 12.3)2 (t - 5)2
2 (0.66) (5.56) 2(5.562) 2(0.662) �dt.

This density looks like that given in Figure 4 in page 10 of the paper.
>From this density the 95% confidence interval for RR is (0.28, 4.99)
which does not coincide with that of Robert et al.

Furthermore, if the confidence intervals for CMRpre and CMRpost were constructed with a t-distribution with a degree of freedom = 3, the confidence interval for RR is wider than the above one.

Therefore, I essentially agree with the inconsistency of the three confidence intervals as it is claimed in the paper. Having said that, here is my second question.

Second question: What is the message after the finding? If the message is that the paper by Roberts et al. is a quite obscure paper concerning to the statistical methodology used, I completely agree. But if themessage is that the data support the null hypothesis that mortality in Iraq is unchanged simply because the 95% confidence interval on f (w) contains the point w = 1, I disagree.

The reason for my disagreement is that to distinguish between estimation and testing was strongly recommended by Jeffreys (1961, pp. 245-249), mainly because the methods commonly used for estimation are typically not suitable for hypothesis testing. In particular, your hidden jump between the confidence interval and the non existence of evidence against the null, has no justification. We need to carry out a test (preferable a Bayesian test) based on the whole data set before to accept or reject a null.
A minor question. Is there something Bayesian in this paper? In my opinion it is frequentist paper with no Bayesian thinking at all. Well, after all what the paper proves is that a frequentist confidence interval is suspect.

A final comment: To cook the data with the intuition is an extremely bad and dangerous practice. It is sometimes used when your statistical model is clearly a wrong model unable to give you a sensible answer.

Reviewer #2:
1. I do not think this piece is suitable as a Lancet article, as it is essentially a statistical discussion of a previous paper. It might be suitable as a letter. However, I understood that you have a policy of only accepting correspondence on previous Lancet papers within a limited period after publication. The Roberts paper was published over two years ago, and has since been superseded by the later Burnham study in 2006. I am not convinced that it makes sense to publish this material now, especially as it is not based on any new facts that have emerged since the original publication.

2. In any case, I believe that the main statistical argument put forward by Kane is incorrect. He uses the confidence interval for the post-invasion mortality rate, as reported by Roberts, to deduce that the confidence interval for the relative risk is too narrow. The wide confidence interval for post-invasion mortality (especially when Falluja is included in the analysis) is based on cluster-sample methodology and correctly takes account of between-cluster variability. This is reflected in the very large design effect (29.3 including Falluja, 2.0 excluding Falluja). However the relative risk estimate is based on comparing pre-invasion and post-invasion mortality within clusters. This is a before and after comparison, and can be thought of as a kind of "matched analysis". If pre- and post-invasion rates are correlated, this will provide a more precise estimate of the relative risk than is implied by looking at the precision of the pre- and post-invasion rates individually. I
assume that this is the reason why Roberts' confidence interval for the RR is narrower than implied by Kane's calculations. In fact Kane explicitly states that he is assuming zero covariance, which does not seem appropriate given this study design. (In any case, his formula for CMR does not seem correct - the last term in the variance should be -2 x Cov(pre,post), I think.

3. In my original review of the Roberts paper, I pointed out that their confidence interval on the RR did not make allowance for the possible variation in relative risk between clusters. In response to this, they reanalysed their data (with assistance from Prof Zeger at Johns Hopkins who is a world expert on the analysis of correlated data!) and the confidence interval in the published paper does make allowance for this source of variation.

4. Kane also states that this overall relative risk estimate (including the Falluja data) is the main finding of the Roberts paper. I would question this. I think the main take-home message of the paper was their estimate of nearly 100,000 excess deaths post-invasion. This is actually based on the excess risk estimated after excluding the data from Falluja. And the authors give a wide confidence interval for this, with a lower confidence limit of 8,000 excess deaths which is much more conservative than the estimate based on the full dataset including Falluja.

5. As mentioned above, this study has in any case been largely superseded by the larger survey carried out in Iraq in 2006 by the same team and published in The Lancet in October 2006. This confirms the earlier findings, but gives much more accurate mortality estimates and provides stronger evidence that there has been a true increase in mortality. The relevance of a statistical critique of the earlier paper at this point is therefore a bit questionable.

6. Finally, one of the most compelling aspects of the evidence presented by Roberts and his team is the dramatic change in cause of death and the age and sex distribution of deaths. Kane makes no mention of these findings.

Reviewer #4:
Report for Lancet on paper/research letter by David Kane on; Confidence intervals of Roberts et al. (2004)

Main considerations:
1. Original fast track referees commented extensively, see APPENDIX, on the issue raised by Kane; but allowed that it be addressed either analytically or in Discussion - including by discussion of how cluster sampling re mortality could be improved in a future study.
2. In his Abstract and Introduction, Kane selectively cites 2.5 fold increase (95% CI 1.6-4.2) whereas, as acknowledged in footnote 2, the Findings section in Roberts et al (2004) also cited sensitivity analysis: 'If we exclude the Falluja data, the risk of death is 1.5-fold (1.1-2.3) after the invasion. We estimate that 98,000 more deaths than expected (8,000 - 194,000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included'. In so doing, Roberts et al. (2004) had addressed the methodological concerns of referees by at least one of the options offered to them. Importantly, their Findings exposed readers to the considerable method-variation.
3. Thus, Kane is covering ground that Lancet, its referees and the authors had traversed - at issue now are 3 points:
a) should Lancet readers have been confronted with Table 1 comparison of CMR versus CMR* to re-present to them why (1.1-2.3) needed to be in the frame besides (1.6-4.2)?
b) should Lancet readers have been confronted with confidence limits, such as in Table 2, which were nonsense albeit the presented point estimate was valid?
c) Do Kane's analyses via assumptions of i) normality and independence, or ii) unimodality and independence add quantitative understanding of why the lower bound of 1.6 is too high?

My answers are:
A. that Lancet and Roberts et al. sufficiently addressed a) by including method sensitivity in the Finding, not just in text of their paper.
B. On b), Roberts et al. might have under-scored even more so why they refrained from providing non-applicable confidence limits because a sceptic, such as Kane, may consider that their exclusion was to blind the reader to methodological problems. However, Lancet published a commentary which specifically addressed the issue of extrapolation from sampled clusters that may have differed in their potential for exposure to hostilities; and how other intelligence sources might be brought to bear to improve the estimation.

C. The final question relates to Kane's own contribution (with or without Michael Spagat as co-author who is currently acknowledged). Unfortunately, Dr Kane's estimate is merely an illustration of the problem that referees' alluded to: namely, if confidence interval is derived from 'rest+Falluja' it is so wide (because mice and elephant) that the uncertainty - as displayed in Dr Kane's Figure 3 - is so wide that no definite inference can be drawn. Yet, if Dr Kane had applied the same analyses to 'the rest' (as Roberts et al. did) , then his p8 distribution (last line) would read N (2.9, 1.352), and - as reported by Roberts et al. - would show statistically significant excess mortality in the non-Falluja clusters (cf 95% CI for RR of 1.1-2.3) . . . to which Falluja can only add.

Neither referees nor Roberts et al (2004) nor Lancet were in any doubt about the criticality of the second reported CI in Findings.

Since 2004, the same research team has conducted a follow-up analysis with more clusters. Despite the longer recall interval, there is good central agreement for the 14 month period post March 2003 with the earlier findings by Roberts et al.

SEE ALSO: article by Scott Zieger in June issue of Significance, 2007 who gave statistical advice to Roberts et al.

APPENDIX
A.1 'no statistical test comparing pre and post conflict CMRs; CIs overlap for CMRs (did compare RR, however, which was significantly different);
A.2 ' unclear why one should remove Falluja cluster if authors felt methodology done correctly. As mentioned above, should have sampled more clusters and fewer persons per cluster to improve methodology and avoid this issue', and related later comment;
A.3 'need to compare violent deaths in various clusters with external events that occurred in Iraq to help validate and interpret data'.

B. no mentions.

C.1 'The main weakness of the paper lies in the extrapolation from the CMR to a total number of deaths fro the whole country. Cluster surveys still need to be validated as a tool to estimate mortality, and design effects are known to be high due to the clustering of violent deaths. Extrapolating the results to the whole country cannot be done without great caution';
C.2 'Excluding the Falluja data narrows down the mortality rate confidence intervals, but what happened in Falluja is not an exception in Iraq and should be taken into account. On the other hand, keeping the data in gives such an important design effect that CI are too wide to make any conclusion. Similarly for the increased relative risk and the extrapolated number of deaths: excluding Falluja gives CIs on the limit of significance';
C.3 'The authors (correctly) discuss the particularity of Falluja extensively, but do not take this into account when extrapolating the figures for the whole country. The quarter of million deaths are presented as a fact;
C.4 'The results of this survey clearly illustrate the limitations of cluster sampling as a method fro estimating mortality, as is explained in $9 of the Limitations section.

D.1 'There is one important statistical issue related to the cluster sampling that I do not think they have addressed adequately, and this concerns the precision of their estimate of excess mortality. To obtain this estimate, they first obtain the mortality rate ratio for the post-invasion versus pre-invasion periods for the sampled households, together with a 95% CI for this rate ratio . . . valid point estimate of excess mortality . . . mortality rate ratio takes no account of the variation in the period effect BETWEEN clusters. This CI will validly estimate the precision of the rate ratio WITHIN THE SELECTED CLUSTERS . . . the remainder of this report is an excellent tutorial/explanatory note on this precise issue . . . and ends as follows:
D.2 'I would suggest that either the analysis is repeated so that the excess mortality estimates have confidence limits in which correct allowance is made fro the cluster sample design. Or this limitation (the use of a 'fixed effects' model) should be explicitly mentioned in the discussion. The emphasis would then be on the fact that these are VALID POINT ESTIMATES, and are very substantial, and on the striking change in causation of mortality. But there would be a clearer recognition that the excess mortality estimates at national level are very imprecise.

E.1 Information which conveys details about the sampling scheme and its resultant selection of clusters is suggested in Table 1. Table 1 also includes the basic paired data per selected cluster: person-months of observation before & after conflict; total births before & after conflict before & after conflict. Table 2 gives additional cause-specific information about deaths before & after conflict.
E.2 by comparison, authors' Figures 1, 2 and 3 do not sufficiently respect study design and obscure actual numbers, Figure 3 seems to convey that there were no deaths [non-violent, or violent] in Falluja in the 14.5 months before conflict.
E.3 From the methods as described, I am unable to check what allowance(s) have been made (and where) for design effects in computing standard errors fro paired versus unpaired estimates.
E.4 What RR fro total mortality (after versus before) was survey designed to be powerful to detect [if this genuinely was a prior consideration]?
E.5 Improvements to sample scheme could/should be related to how authors have tried to deal with 'Falluja' effect . . . For the future, stratified sampling might be proposed for stratum a= a priori heavily-conflicted Governorates & stratum b= other Governorates . . . Even in the present case, this stratified approach could be explored posthoc in deciding how to multiply up from the sample results for Governorates which were/were not a priori known to be heavily-conflicted (rather than ignore Falluja & apply non-Falluja rates to all Iraq) - see also specific comments, P7, last paragraph;
E.6 UNCLEAR how/if design effect was allowed fro in relative risk CIs.

The above comments were made on the penultimate draft. There was a further round of refereeing by one or more of A to E.

Thanks for sending us this critique.
best regardfs
Stuart

The Lancet

Influencing the Election

2008-07-31T06:24:00.000-07:00

One of the on-going disputes is whether or not Les Roberts and his co-authors attempted to influence that US election with the publication of L1 (or L2). There is no doubt that they did. Consider the original AP article on L1.

Researchers have estimated that as many as 100,000 more Iraqis -- many of them women and children -- died since the start of the U.S.-led invasion of Iraq than would have been expected otherwise, based on the death rate before the war.

Writing in the British-based medical journal The Lancet, the American and Iraqi researchers concluded that violence accounted for most of the extra deaths and that airstrikes by the U.S.-led coalition were a major factor.

There is no official figure for the number of Iraqis killed since the conflict began, but some non-governmental estimates range from 10,000 to 30,000. As of Thursday, 1,106 U.S. servicemen had been killed, according to the U.S. Defense Department.

The scientists who wrote the report concede that the data they based their projections on were of ``limited precision,'' because the quality of the information depends on the accuracy of the household interviews used for the study. The interviewers were Iraqi, most of them doctors.

Designed and conducted by researchers at Johns Hopkins University, Columbia University and the Al-Mustansiriya University in Baghdad, the study was published Thursday on The Lancet's Web site.

The survey attributed most of the extra deaths to violence and said airstrikes by coalition forces caused most of the violent deaths.

``Most individuals reportedly killed by coalition forces were women and children,'' the researchers wrote.

The report was released just days before the U.S. presidential election, and the lead researcher said he wanted it that way. The Lancet routinely publishes papers on the Web before they appear in print, particularly if it considers the findings of urgent public health interest.

Those reports then appear later in the print issue of the journal. The journal's spokesmen said they were uncertain which print issue the Iraqi report would appear in and said it was too late to make Friday's issue, and possibly too late for the Nov. 5 edition.

Les Roberts, the lead researcher from Johns Hopkins, said the article's timing was up to him.

``I emailed it in on Sept. 30 under the condition that it came out before the election,'' Roberts told The Associated Press. ``My motive in doing that was not to skew the election. My motive was that if this came out during the campaign, both candidates would be forced to pledge to protect civilian lives in Iraq.

``I was opposed to the war and I still think that the war was a bad idea, but I think that our science has transcended our perspectives,'' Roberts said.

Comments:

1) As here, Roberts often tries to deny that he sought to affect the election even as he admits to doing so. What does it even mean to "skew the election?" The easiest way to think about the issue is to compare two worlds: In world A, L1 comes out after the election. In that world, Bush and Kerry spend the week before the election campaigning, arguing about issues X, Y, and Z. Both candidates seek to focus the debate on topics most likely to benefit them. Iraqi (civilian) mortality plays a role in that world but it is a small one. The less that Iraqi civilian mortality is discussed, the better off that Bush is (I think). In world B, L1 comes out before the election. (This is the world we actually live in.) Iraqi mortality is much more a part of the campaign then it was in world A. Issues X, Y and Z are still discussed, but less than they were in world A. (There are only so many hours in the day, questions that reporters can ask, speeches that Bush and Kerry can give.)

The causal effect of issuing L1 before the election rather than after is the difference between world A and world B. Roberts sought to force the candidates to "pledge to protect civilian lives in Iraq." He sought to influence what issues the candidates addressed. He wanted to change the debate from what it would have been without L1 so that more time/energy/attention was focussed on Iraqi mortality.

And that is fine! It's a free country and Roberts has the right to try to influence the electoral process. He is smart enough to know that his actions, alone, are unlikely to be the deciding factor in which candidate wins. But that does not change the fact that he sought to force the candidates to address an issue that they would not have otherwise addressed were it not for his decision to publish L1 before the election.

2) Can Roberts (or any author) insist on a specific publication date when working with journal like the Lancet? I guess so, if the article is desirable enough, from the editors point of view. Given what we know about Lancet editor Richard Horton's politics (and don't forget his YouTube videos here and here), it seems likely that he was also in favor of the article coming out before the US elections.

3) A similar story on the importance of timing applies to L2. Paul Foy reported in 2006 that:

Roberts organized two surveys of mortality in Iraqi households that were published last October in Britain's premier medical journal, The Lancet. He acknowledged that the timing was meant to influence midterm U.S. elections.

This is confused in that L1 was published in October 2004 and L2 in October 2006, but midterm elections were (obviously) only in 2006.

4) Fox News is probably quoting Burnham out of context here.

We wanted to get the survey out before the election if at all possible, but our agenda on this is concern for the humanitarian issues.

As always, I think that Burnham is a good guy. He is really focussed on humanitarian issues. But there is no compelling scientific or humanitarian reason why the studies needed to come out in the two weeks prior to the US elections. You can make the case (and I have seen Horton, for example, make it) that the key issue is not that the studies come out before the election but that they come out as soon as possible. That is a reasonable belief. Alas, the Lancet studies did not work that way. It took only 6 weeks for L1 to go from survey finish (mid September 2004) to publication. Why, if getting the information out fast is what matters, did it take 14 weeks for L2? (The survey work was completed in early July 2006.) The obvious explanation is that the authors and/or editors wanted the articles to come out just before the US elections. They worked more than twice as fast on L1 than on L2 in order to make that happen.

5) Roberts has spent the last 5 years backpedaling. He realizes that his credibility is damaged if people know that he sought to "influence" the US elections in 2004 and 2006. See our many transcripts for examples. The Hopkins Q&A (since deleted) included this tidbit.

At no time did study authors Les Roberts or Gilbert Burnham say that the release of their mortality studies was timed to affect the outcome of elections.

The only way for this to be true is for "outcome of elections" to be defined as "whether Bush or Kerry won." It is reasonable to think that Roberts/Burnham were smart enough to realize that their articles would not swing the election one way or the other. But there is no doubt that they (or at least Roberts) have admitted that they sought to influence the course of the campaign by forcing the candidates to address the issue of Iraqi mortality when they would have otherwise spent their time on other topics.

Eight Transcripts

2008-07-16T13:33:00.000-07:00

Looking for more reading material? I am here to help. Consider these 8 transcripts of talks (7 from Les Roberts and 1 Gilbert Burnham) over the last few years. There are some interesting/annoying passages, but nothing much beyond what we would expect. We have:

roberts_2006_talk_with_slides
roberts_on_npr
charles-goyette-interview-of-les-roberts
roberts_at_university_of_minnesota
roberts_at_watson_institute
roberts_in_trumansburg
roberts_on_democracy_now
burnham_interview_with_lancet

I hope to post comments and highlights at some point, but no promises. Only feel like reading one? Start with the first.

Propaganda

2008-07-16T13:29:00.000-07:00

This article (pdf) by Mark van der Laan and Leon de Winter takes a hard anti-Lancet line.

Given all of this, one needs to wonder if this large estimated number of violent deaths is not only due to statistical uncertainty (100.000-1000.000), but possibly also due to one or more of the potential biases mentioned above (and biases not mentioned at all because of a lack of space). Could it be that The Lancet’s survey is juggling with statistics and defies common sense?

We conclude that it is virtually impossible to judge the value of the original data collected in the 47 clusters. We also conclude that the estimates based upon these data are extremely unreliable and cannot stand a decent scientific evaluation. It may be that the number of violent deaths is much higher than previously reported, but this specific report, just like the October 2004 report, cannot support the estimates that have been flying around the world on October 29, 2006. It is not science. It is propaganda.

It surprising to see an academic of van der Laan's stature use such strong language about a peer-reviewed article.

Daponte article

2008-07-10T20:29:00.000-07:00

Must read article from Yale researcher Beth Osborne Daponte.

Challenges exist when making reliable and valid estimates of civilian mortality due to war. This article first discusses a framework used to examine war’s impact on civilians and then considers challenges common to each statistical approach taken to estimate civilian casualties. It examines the different approaches that have been used to estimate civilian casualties associated with the recent fighting in Iraq to date and compares the results of different approaches. The author concludes by proposing that after fighting has ceased, other approaches to estimating Iraqi civilian mortality, such as post-war retrospective surveys and demographic analysis, should be employed.

Daponte's article is fair and professional. If you only have time to read 15 pages about the debate over Iraqi mortality, this is the paper for you. Bottom line:

Perhaps the best that the public can be given is exactly what IBC provides – a running tally of deaths derived from knowledge about incidents. While imperfect, that knowledge, supplemented by the wealth of data of the Iraq Living Conditions Survey and Iraq Family Health Survey (which have their own limitations), provides enough information in the light of the circumstances. At a later date, additional surveys can be conducted to determine the impact and/or do demographic analysis. But for now, the Iraq Body Count’s imperfect figures combined with the date of the ILCS and IFHS may suffice.

Exactly right. No survey is perfect, but combining the information from IBC, ILCS and IFHS is the best way to get a handle on Iraqi mortality. But what about those Lancet surveys? Why does Daponte not even mention them in her conclusion? Because she thinks that they are highly suspect. Read the whole thing, but my favorite quotes are:

The estimates from these students [DK --- I think that this should have been "studies"] have been lauded but also questioned, partially because the researchers have misinterpreted their own figures but also because of fundamental questions about the representativeness of the achieved survey sample.

The [Lancet I] authors misinterpreted the analysis of the data . . .

Problems with the analysis of the data also plagued the second effort.

The pre-war CDR that the two Lancet studies yield seems too low. That is not to say that it is wrong, but the authors should provide a credible explanation as to why their pre-war CDR is nearly half that of what the UN Population Division estimates for pre-war Iraq. Since Burnham et al. arrive at their estimate of Iraqi ‘‘excess deaths’’ by taking the difference in the pre-war and wartime crude death rates and applying it to a population, if the pre-war mortality rate was too low and/or if the population estimates are too high (e.g., do not take into account the refugee movement out of Iraq), then the resulting number of ‘‘excess deaths’’ would be too high, yielding inflated estimates. Unfortunately, the authors have not adequately addressed these issues.

Burnham et al. sent interviewers to the field to ask respondents for information, knowing that this could put interviewers’ lives at risk. In doing so, the research team was professionally irresponsible. Further, in an effort to ‘‘protect interviewers’’ (even though they had already put them in danger), they sacrificed the scientific randomization that the research relies upon.

Further, one should question how a proposal to conduct this research made it through the Institutional Review Board at a US university.

However, unlike the Lancet studies, the ILCS was careful in its attribution of the root causes of civilian casualties in Iraq.

Les Roberts likes to claim that no one with expertise in estimating conflict mortality criticizes the Lancet results. He should stop making such false claims.

Daniel Davies likes to get his boots on when he thinks that someone is unfairly criticizing the Lancet studies. Time to get walking! Davies will have a hard time portraying Daponte as either incompetent or a Neocon stooge of the Bush administration. If Daponte doesn't think that the Lancet estimates are worth paying attention to, why does Davies defend them so relentlessly?

Just asking!

Obermeyer, Murray and Gakidou

2008-07-01T03:50:00.000-07:00

Thanks to Donald Johnson writing at Deltoid for the pointer to this article from the British Medical Journal, "Fifty years of violent war deaths from Vietnam to Bosnia: analysis of data from the world health survey programme" by Ziad Obermeyer, Christopher J L Murray, and Emmanuela Gakidou. Below are my comments, expanded from my initial thoughts at Deltoid. I refer to the authors by their initials: OMG.

OMG's key claim, for purposes of Lancet aficionados, is that "media estimates capture on average a third of the number of deaths estimates from a population based surveys." This matters because Les Roberts has been running around for years claiming that passive surveillance (as Iraq Body Count uses) is a horrible method of estimating mortality and never (except possibly in places like Bosnia) captures more than a small percentage of all deaths. (In fairness to Roberts, this is a new article and so, perhaps, his previous claims were justified by the research he had access to at the time.)

Will Roberts now acknowledge this? Time will tell.

The paragraph most directly relevant to disputes over Iraq mortality is:

As a final point of comparison, we applied our correction method, derived from the comparison of survey estimates with Uppsala/PRIO data, to data from the Iraq Body Count project’s most recent report of 86,539 (the midpoint of the 82,772 to 90,305 range reported in April 2008) dead in Iraq since 2003. Our adjusted estimate of 184,000 violent deaths related to war falls between the Iraq Family Health Survey estimate of 151,000 (104,000 to 223,000) and the 601,000 estimate from the second Iraq mortality survey by Burnham and colleagues. [footnotes omitted]

Comments:

1) Tim Lambert enjoys making a listing of various (reputable) estimates of mortality in Iraq. Example here Now, I might argue that, given all the problems that ORB has had, its estimate does not belong in Tim's collection. But there can be no doubt that OMG's estimate does belong. Will Tim add it?

2) If your main interest is judging the quality of L2, then a better comparison would have used the IBC numbers to July 2006 (mid-point 47,668), thus covering the same time period as L2 and IFHS. I am not sure what the exact formula is that allows OMG to go from 86,539 to 184,000. Assume that we can just apply this ratio (184,000/86,539 = 2.13) to the IBC estimate of 47,668. That would yield a violent death estimate of 102,000. Recall that, 2 years ago, Jon Pedersen estimated violent deaths at 100,000. Moreover, the IFHS estimate would be lowered from 151,000 to around 100,000 if you removed the "arbitrary fudge factor" (in Debarati Guha-Sapir's marvelous phrasing) that IFHS employs.

Call me crazy, but I would say that the emerging scientific consensus is that approximately 100,000 excess war-related violent deaths had occurred in Iraq through June 2006. This is 1/3 lower than 150,000 (0 -- 500,000) estimate that I was comfortable with earlier in this year. Given all the new research since then, I update my estimate to 100,000 with a 95% confidence intervals of (0 -- 300,000).

[Those who think that a lower bound of zero is too low should remember that the definition of "excess" implies a comparison to what would have happened in a counterfactual world without a US invasion/occupation. Although I do not think that it is likely that Saddam would have engaged in substantial internal (against Kurds/Shia) or external (against Iran/Kuwait) aggression, it is not impossible that he would have. Those (possible) violent deaths were prevented by the war. If the comparison is against mortality in Iraq in 2002, then the lower bound should be raised substantially.]

So, if IFHS and IBC+OMG are consistent with each other and with the opinions of informed observers like Pedersen and Guha-Sapir, why does L2 estimate violent mortality approximately 6 times higher? I think that the raw data underlying L2 is not reliable.

Two Transcripts

2008-06-04T05:58:00.000-07:00

Here are two transcripts from talks by Lancet authors in 2007: Gilbert Burnham speaking at MIT (link and pdf) and Les Roberts in a radio interview (link and pdf). Comments:

1) These transcripts were done by my interns. They did their best, but if you decide to quote from these, you should check the quote against the original source as well.

2) I have already highlighted (here and here) some of the more important parts of Burnham's talk. The main benefit of this version is that includes copies of his slides, making it much easier to follow the talk. On page 16, we have a slide showing that the confidence intervals for excess deaths including Falluja would be much wider (and include 0), just as I have been arguing for two years now. Pages 21-24 highlight a different version of the sampling plan than is described in the paper. Burnham claims that they did not restrict the sample to streets that crossed their main streets. Instead, they made a list of "all the residential streets that either crossed it or were in that immediate area." This is just gibberish.

First, if this was what they actually did, why didn't they describe it that way in the article? Second, given the time constraints, there was no way that the teams had enough time to list all such side streets. Third, even if the interviewers did do it this way, the problem of Main Street Bias would still exist, except it would be more Center Of Town Bias. Some side streets are in the "immediate area" of just one main street (or often in the area of none) and other side streets (especially those toward the center of a town or neighborhood) are near more than one. The later are much more likely to be included in the sample.

3) I do not think that the Roberts interview is nearly as interesting, although it is amusing to note how he makes fun of "economists" (read: Spagat) and "political scientists" (read: me) early in the talk. I think that his claims (page 7ff) that the Lancet surveys are consistent with UNDP results from 2004 are not correct either, but I have not been following that dispute closely enough to comment more.

4) I also liked this statement from Roberts about the fact that 2/3 of the violent deaths from L1 were in Falluja.

So we, in 2004, came out with an estimate that we thought a 100,000 excess deaths
had occurred. When we did that, we set the Fallujah numbers aside, we didn't think they were wrong, but the vast, as you say 2/3 of all the violent deaths we found were there in this one 30 house cluster in Fallujah, and if in 2004, the fall of 2004, you had been watching the news this would have been very plausible. First, there was that big, incursion into Fallujah into I think April, in which because the press coverage was so negative for the Americans they ended up backing off and pulling back out, and in the month or two months before we went in the city had been surrounded by American forces and was being shelled on a daily basis.

It is absurd to think that US forces were lobby artillery shells into Falluja on a daily basis in July, August or September of 2004.

UPDATE: A reader adds these comments on the Roberts interview.

For me the real shocker is that he claims multiple times that the BBC poll found 17% of households "lost a household member to violence".

Also, he says that that there is "overwhelming" statistical evidence that 30 clusters is enough. Where is the evidence?

On sampling he actually describes the procedure as originally described in the Lancet. You might want to point out the contradiction with Burnham's version that you already treat.

He misrepresents the L1 sampling procedures as standard when they were anything but. It's worth another look at the L1 paper on this. They paired governorates that they claimed with no evidence were similarly violent then they randomly selected one of each pair. Definitely not standard. They then improperly ignored this when calculating their CI's.

He admits that Lafta "didn't really understand all the fancy statistics and sampling methods…"

Just complete gobledegook on how visiting more clusters could only raise his estimates.

Totally misrepresents ILCS. This boils down to the same thing as was in your previous post on the IFHS as an overestimate. Pedersen wrote a paper saying that all surveys underestimate (I think this is probably wrong but OK) so he raises his estimate. But he points out explicitly that the argument about surveys underestimating applies just as much to L1 as it does to the ILCS. So Roberts tries to compare and adjusted ILCS to an unadjusted L1. His 37,000 figure isn't right either but it's not egregiously low.

The he says "most of the bodies coming into morgues are not from natural causes but from violence". That's like saying most deaths investigated by the police are violent. Does that prove that most deaths are violent? The morgues get only suspicious deaths and unclaimed bodies.

UPDATE 2009-02-24: Recent events have made a particular section of Burnham's talk highly relevant. The transcript around page 35 of the pdf is not exactly correct. Here is a better transcription.

There were limitations to record keeping; we had some criticism that we could not produce a record showing which households were visited and who names of people were, and so forth.

We intentionally did not record that, because we felt that if the team were stopped at a checkpoint, of which there are lots of checkpoints, and the records were gone through, some of you may have had this experience, where you stop at a checkpoint, people go through all your papers, read everything, and they find certain neighborhoods. That might have increased risk, which we didn't want to do.

There can be no doubt that Burnham told the MIT audience that the survey team did not record names. But we now know that they did.

IFHS As An Overestimate?

2008-05-12T08:43:00.000-07:00

I have spent so much time and energy fighting the claim, made by Lancet defenders, that IFHS is an underestimate of violent mortality in Iraq, that I had never really considered the other side. Might IFHS be an overestimate? (Thanks to a reader for pointing this out.)

Consider one of the adjustments made in IFHS:

To estimate the most probable rate of violent deaths after the invasion and the range of uncertainty, we performed Monte Carlo simulations that took into account the survey sampling errors that were estimated with the use of the jackknife procedure and uncertainty regarding the missing cluster-adjustment factors, the level of underreporting, and the projected population numbers. We assumed that the level of underreporting was 35% (95% uncertainty range, 20 to 50), and its uncertainty was normally distributed.

I had initially misread this as referring to the issues of clusters that the interviewers had not been able to visit because they were too dangerous. Looking more closely, I now see that this "underreporting" has nothing to do with missing clusters. Instead, I think, the concern is that some households might have, for whatever reason, failed to inform interviewers about violent deaths. I also think that "underreporting" covers the concern that entire households may have been killed or that families with higher than average mortality would have been more likely to leave the country. In either case, no household would be left to interview, thereby leading to underreporting.

But 35% is a big number! Where does it come from? I can't see any discussion in the paper or literature references. Why not use 5% or 200%? Note that the Lancet papers make no adjustment for underreporting, although they do discuss the issue. The Lancet authors use concerns with underreporting to justify, reasonably enough, that their estimates are "conservative."

But that means that we should not be comparing the 151,000 violent deaths estimate from IFHS with the 601,000 violent deaths from L2! The first number adjusts for underreporting while the second does not. This is an apples-versus-oranges comparison.

We can get a sense of the magnitude of this issue by comparing Table 3 and 4 in IFHS. In Table 3, the overall violent mortality rate is 1.09 (0.81 -- 1.50) per 1,000 per year without any adjustment for underreporting, but with adjustments for the missing clusters. In Table 4, on the other hand, we have these estimates for the 3 years after the invasion: 1.77, 1.56 and 1.67. The key point is, as the legend indicates, that these numbers are adjusted for "underreporting," unlike those in Table 3.

In fact, the main results section of the paper makes clear that this is a big issue.

Interviewers visited 89.4% of 1086 household clusters during the study period; the household response rate was 96.2%. From January 2002 through June 2006, there were 1325 reported deaths. After adjustment for missing clusters, the overall rate of death per 1000 person-years was 5.31 (95% confidence interval [CI], 4.89 to 5.77); the estimated rate of violence-related death was 1.09 (95% CI, 0.81 to 1.50). When underreporting was taken into account, the rate of violence-related death was estimated to be 1.67 (95% uncertainty range, 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range, 104,000 to 223,000) from March 2003 through June 2006.

The adjustment from underreporting raises the violent death rate about 50%, from 1.09 to 1.67. Both those numbers include the adjustment for missing clusters. That 50% increase does not match perfectly against the 35% figure quoted above, but there are a lot of messy details to consider so we are safe in assuming that the 1.09 to 1.67 increase is accurate.

So, back of the envelope, the appropriate number to compare to the 601,000 excess violent deaths from L2 is not the 151,000 cited by IFHS. Instead, we should use 100,000 or so, the excess death estimate which is implied by an violent death rate of 1.09. [1.67 divided by 1.09 times 151,000 equals 98,600, but I am rounding up to be "conservative."] In other words, once we adjust for the use of underreporting, we see that the L2 estimate is six times larger than the one provided by IFHS, not only four times larger.

(Of course, one could also adjust the L2 number up to account for underreporting. Assuming an adjustment factor consistent with IFHS yields an excess violent death estimate of around 900,000.)

The main point is that we want to compare L2 with IFHS on an equal footing, either with adjustment or without. The current convention --- of which I am as guilty as anyone --- of comparing 601,000 (L2) with 151,000 (IFHS) and then concluding that the L2 estimate is 4 times higher is fatally flawed. To compare L2 and IFHS sensibly, we must either adjust L2 upward (so that it also adjusts for underreporting) or adjust IFHS downward (so that it does not adjust). Whichever you prefer, the conclusion is the same: the L2 estimate of violent deaths is six times larger than IFHS.

The next step in the analysis is to do the same for the issue of missing clusters. Both IFHS and L2 have missing clusters, clusters that were too dangerous for the interviewers to enter. In such cases, it is reasonable to believe that mortality might be higher in those clusters than in ones that the interviewers were able to visit. (But see here for a contrary view.) IFHS adjusts for this bias, but L2 does not. So, to compare the two surveys on an equal footing, we need to either remove this adjustment from IFHS or add it to L2. Doing so will make L2 an even greater multiple of IFHS than it already is.

UPDATE: Note that one can think about this change as either an increase in 53% (1.09 to 1.67) or a decrease of 35% (1.67 to 1.09). I have also been told that Deberati Guha Sapir refers to this as an "arbitrary fudge," but I am still tracking down that reference.

UPDATE II: Here is the quote in context:

Now comes the WHO survey. Conducted with the help of the Iraqi government, it is by far the most comprehensive mortality assessment to date. Interviewers visited 9345 homes in more than 1000 clusters. But its estimate of 151,000 violent deaths has come in for some criticism, too. Unlike other Iraq casualty surveys, this one includes an upward adjustment of 35% to account for “underreporting” of deaths due to migration, memory lapse, and dishonesty. “That is really an arbitrary fudge factor,” says Debarati Guha-Sapir, an epidemiologist at the WHO Collaborating Centre for Research on the Epidemiology of Disasters in Brussels, Belgium. But the number falls squarely within the range produced by a meta-analysis of all available mortality studies by Guha-Sapir and fellow centre epidemiologist Olivier Degomme. The Johns Hopkins figure is an outlier, she says.

UPDATE III: L2 also had a problem with clusters that were to violent to visit. From the paper:

The survey was done between May 20 and July 10, 2006. Only 47 of the sought 50 clusters were included in this analysis. On two occasions, miscommunication resulted in clusters not being visited in Muthanna and Dahuk, and instead being included in other Governorates. In Wassit, insecurity caused the team to choose the next nearest population area, in accordance with the study protocol. Later it was discovered that this second site was actually across the boundary in Baghdad Governorate.

So, we know that there was at least one cluster that was too violent to visit and we know that they had a "protocol" to deal with this problem. (It is unclear how many such clusters there were.) This means that a fair comparison between L2 and IFHS would need to either adjust both for this problem or adjust neither. The problem --- and, again, I am as guilty as anyone --- is that the standard 151,000 (IFHS) to 601,000 (L2) comparison adjusts only the former but not the later.

Fortunately, Table 4 in the Supplementary material (pdf) shows that the post-invasion violent mortality rate unadjusted for missing clusters is 0.80 (0.63 1.03). So, we need to cut the 151,000 IFHS estimate for excess deaths (which is based on a mortality rate of 1.67) in more than half in order to provide a fair comparison to the 601,000 figure from L2. Scaling appropriately, the IFHS estimate would be 72,000.

Summary: Using the same assumptions for L2 and IFHS (no adjustments for underreporting or for clusters that could not be visited) generates estimates that differ by more than a factor of 8: 601,000 to 72,000.

Disappearing Questions

2008-05-12T07:48:00.000-07:00

The Johns Hopkins webpage continues to be airbrushed in ways both small and large. Previous discussion here and here. Consider some changes:

1) The main page used to have a link to "Answers to Questions About Iraq Mortality Surveys." That link has now disappeared. And, along with the link, there is no longer a set of questions and answers.

2) I have not had the time or inclination to follow this closely but it seems like they had one version of this up for some time. Then, for whatever reason, they decided to delete that and create a second version (saved here as pdf). But now that version is gone too.

3) My guess is that all these changes are caused by the involvement of the lawyers from Johns Hopkins. If it were up to, say, Les Roberts (and perhaps Gilbert Burnham) all this material would still be available. But the lawyers have told them to either be quiet or to be very sure that everything they claim is correct. Since there are numerous problems with the claims made in these documents, Johns Hopkins has decided that the simplest course of action is to post nothing.

"Civilian" Casualties

2008-05-06T11:07:00.000-07:00

Consider the original news release about L1, before our friends from Hopkins dump it down the memory hole. The title is "Iraqi Civilian Deaths Increase Dramatically After Invasion." Lancet aficionados will recall that this news release (and, I think, the original Lancet editorial) caused some controversy because they both discussed "civilian" deaths. Needless to say, L1 did not differentiate between civilians and non-civilians in its methodology, so the authors have no idea whether the excess mortality was driven by military deaths (whether Saddam-era soldiers during the initial invasion or insurgents thereafter) or civilian ones. (Leaving aside deductions one might make by focusing on female/child/elderly deaths.)

Anyway, there was some dispute about this in the first few months (before I started paying attention to the debate) but I always thought that none of this mattered, that it was an honest mistake caused by someone who did not read the study closely. And, indeed, if I just point to the title of the press release, this would be reasonable. After all, Roberts/Burnham do not make the titles for press releases (one assumes). So, we should not blame them if some secretary in the news office, presumably acting in good faith, gives this release a misleading title.

But note how the release begins.

Civilian deaths have risen dramatically in Iraq since the country was invaded in March 2003, according to a survey conducted by researchers from the Johns Hopkins Bloomberg School of Public Health, Columbia University School of Nursing and Al-Mustansiriya University in Baghdad. The researchers found that the majority of deaths were attributed to violence, which were primarily the result of military actions by Coalition forces. Most of those killed by Coalition forces were women and children. However, the researchers stressed that they found no evidence of improper conduct by the Coalition soldiers.

The survey is the first countrywide attempt to calculate the number of civilian deaths in Iraq since the war began.

Leave aside the anti-coalition slurs here, we see that the author of the news release maintains that this is a count of civilians, rather than just Iraqis. Again, this could be just a mistake by someone in the Hopkins press office, but one expects more care to be taken with the actual body of a news release rather than just its title. Indeed, one would expect the news office to show a news release to the researchers whose work it is describing before the news release is made public. Did Roberts/Burnham know ahead of time that the news release would make claims about civilians? Did they approve this ahead of time? Again, both are busy academics, so we might want to give them the benefit of the doubt. Perhaps they never saw the release, either before publication or even now. But then we read:

“Our findings need to be independently verified with a larger sample group. However, I think our survey demonstrates the importance of collecting civilian casualty information during a war and that it can be done,” said lead author Les Roberts, PhD, an associate with the Bloomberg School of Public Health’s Center for International Emergency, Disaster and Refugee Studies.

...

“There is a real necessity for accurate monitoring of civilian deaths during combat situations. Otherwise it is impossible to know the extent of the problems civilians may be facing or how to protect them,” explained study co-author Gilbert Burnham, MD, associate professor of International Health at the Bloomberg School of Public Health and director of the Center for International, Disaster and Refugee Studies.

Burnham and Roberts themselves used the term "civilian(s)" in direct quotes! There is no need to blame Tim Parsons, the media contact at Hopkins. He, or whoever wrote the news release and its title, were just repeating what the professor had told them, that the Lancet survey showed a dramatic rise in civilian mortality.

This is, in many ways, a minor sin. There is no doubt that thousands of civilians have died in Iraq. But the fact that Roberts/Burnham were happy to mislead readers about exactly what their study measured indicates that we need to maintain a skeptical attitude about the claims that they make.

Missing Answers

2008-04-18T12:20:00.000-07:00

Is it just me, or are weird things happening at Johns Hopkins? For several months (at least), they have had a set of questions and answers about the Lancet surveys here. But that has now been deleted. What is going on? Here is the Google cache and here is a permanent copy.

Again, this could be nothing. Perhaps they are just re-organizing where things are located. Perhaps they are rewriting some of the material to make it more clear.

Or perhaps the lawyers are starting to get involved . . .

Sleazy Switcheroo

2008-04-16T09:53:00.000-07:00

UPDATE: The letter now seems to have been deleted altogether. Or perhaps it has been moved somewhere else. Comments welcome!

UPDATE II: Here is a Google cache of the second version of the letter. And here is a permanent copy. If anyone really cared, I could probably find a screen shot of the first version. I have confirmed that the National Journal published the first version of the letter but can't find a link to that.

----------------
As I first mention in this Deltoid thread, Gilbert Burnham and Les Roberts have done a very sleazy thing.

They initially sent a letter of complaint to the National Journal. That letter was posted here originally but has since moved to here. I think that this move is a technical issue unrelated to my complaint.) Anyway, that letter made an incorrect claim about what the National Journal reported about missing death certificate. Burnham and Roberts claimed that:

The statement on missing certificates is wrong. Three clusters did not have the presence of certificates noted, and in all there were 120 deaths in which the interviewers neglected to note their presence.

The problem is that this statement is clearly false.

I e-mailed Burnham, Roberts and co-author/data-maven Shannon Doocy about this issue on February 7th. We engaged in a spirited back and forth via e-mail. Doocy, as always, knew the data backwards and forwards and, while she was too polite to say so directly, confirmed my position. Roberts defended his position for a bit, misunderstood what the data showed and then, eventually, failed to reply.

On March 4th, I e-mailed the General Counsel of Johns Hopkins, Stephen Dunham, with my complaint, cc'ing Roberts, Burnham and others. I received no response.

And now, mirable dictu, the original letter has been changed in place! The offending section now reads:

The statement on missing certificates is wrong. There were 83 deaths (13%) in which the interviewers neglected to note their presence and these deaths were distributed across 20 clusters.

How is this sleazy? Let me count the ways.

1) Neither Roberts, Burnham nor Dunham ever admitted that the original claims were wrong. They never apologized to Munro/Cannon for incorrectly impugning their article. They never acknowledged that Spagat/my empirical work was correct.

2) They did not inform anyone that they have made this change. They swapped the wording in the proverbial dead of night, perhaps while hoping that no one would notice. Since I am the one who brought this error to their attention, Burnham and Roberts should have let me know that they were (sort of) correcting it.

3) The letter as it is now posted is a fraud. Burnham and Roberts still claim that "The following letter was submitted to the editors of the National Journal on January 7, 2008." In fact, this is not the letter that they submitted to the National Journal.

4) I do not think that they have told the editors of the National Journal that their previous letter to them was in error and that they are falsely claiming to the world to have sent a letter that they did not send.

5) They have not apologized to either the authors (Munro and Cannon) nor to the researchers (Michael Spagat and me) for their false accusations.

6) And, perhaps worst of all, they continue to try to mislead. They still claim that "The statement on missing certificates is wrong." This is a falsehood. Every single statement abut missing certificates in the National Journal article is correct.

Data for L1

2008-03-09T15:50:00.000-07:00

Those interested can download (some of) the data from L1 here. Comments:

1) Thanks to Tim Lambert for posting that data. I wonder if Les Roberts has ever asked him to remove it? When we chatted briefly at JSM last summer, Roberts seemed to accuse me of posting his data on my blog. I did not do so since he didn't give me his permission. I only included the data in my R package after Lambert made the data public. I am not sure what I would do if Roberts asked me to remove the data from my package. On the one hand, I take proper scholarly behavior seriously. It is their data and I would not share it without permission. Indeed, I have been asked by several folks to provide a copy of the L2 data. I have never done so (even though I could have without getting caught) because I don't think that scientists ought to behave that way. But the L1 data is a trickier case because Lambert has already placed the data in the public domain. (In fact, I made sure to download the data from his site rather than use the copy that Roberts gave to me.) Fortunately, Roberts has not asked me to remove the data from the R package, so I am covered. Also, I have discussed the issue with Burnham and Doocy, who have offered no objections.

2) The back story was that I was trying to get the data out of Roberts and was cc'ing Lambert both out of politeness (since it was his blog Deltoid that got me involved in the debate) but also out of a sense that Roberts was, I thought, more likely to play nice if Lambert, a vigorous supporter of L1, was listening in on the conversation.

3) I have not looked closely at the cluster level data that is available above for L1. My main focus has been on what happens when you include and exclude Falluja from the analysis. But I am still somewhat suspicious of the raw data here, and not just the ludicrous outlier that is Falluja. For example, the range of average household size seems ridiculously large. Assume that each cluster included 30 households. Then, the average household size in Karbala 1 is just 4.6, rising to 11.5 in Thaura (Baghdad). (Hat tip to Mike Spagat.) Does that seem reasonable? I guess that household size might be larger in the city than the country (?), but more than 2 times larger? Is that consistent with data from other surveys?

4) Recall that Roberts (can't find the link) only participated in the surveys for the first 8 clusters. At that point, there was some problem with the police and his interviewers argued that he (as an American) was putting them in danger. So, he spent the rest of the time in Baghdad. The obvious question is: Were the results very different between the clusters that ~~Burnham~~ Roberts supervised and the ones that he did not? A quick glance suggests that this is tough to know because, judging from the dates, he participated in all (?) the clusters in Baghdad, so looking at the results for just "his" clusters would need to adjust for that in some way. I am too busy to tackle this now.

5) Note that two of the strangest clusters were Falluja and, again, Thaura. Recall the stories about how these two (or at least Falluja) were saved to the end of the process because they were so dangerous. I have always been quite suspicious of results that come at the tail end of these surveys, as the famous cluster 33 does in L2. You can see that these were the last two because they feature the latest "date finished" in the data.

6) Why don't the authors provide data from L1 at the same level of detail that they do for L2? Excellent question. I have pushed both Burnham and Roberts on this point. The official story is that this data is "no longer available." That is a direct quote from an e-mail to me. I have tried to find out what this means. Surely the data is available at least to them. They did not lose it? Or destroy it? Elizabeth Johnson, the statistical consultant, is a careful and serious scholar. I am virtually certain that she kept a back-up copy, along with documentation and computer code. Why won't Roberts share this with us? He may not have anything to hide but he sure acts like he does.

Recent Articles on L2

2008-02-13T12:55:00.000-08:00

Several recent (to me) articles about the Lancet surveys are worth reading.

First, a tour de force (pdf) from Michael Spagat: "Ethical and Data-Integrity Problems in the Second Lancet Survey of Mortality in Iraq." This is the paper that I wanted to write last summer, but was not smart or organized enough to do. Michael will be presenting this paper at JSM in August.

Second, "Estimating mortality in civil conflicts: lessons from Iraq" by Debarati Guha-Sapir and Olivier Degomme (pdf) is critical of L2. The authors identify several "errors and methodological weaknesses" of L2 and write:

Our re-estimation of total war-related death toll for Iraq from the invasion until June 2006 is therefore around 125,000.

This is less than 1/4 of the comparable L2 estimate of 601,000.

Third, a 2006 working paper (pdf) by Mark J. van der Laan. He argues that the reported confidence interval for L2 is significantly too narrow.

Fourth, the IFHS study. They estimate about 151,000 violent deaths over the same period as L2. Again, this is around 1/4 the L2 estimate. M. Ali, one of the authors, will be presenting on my panel at JSM.

Good stuff all.

Where does that leave my estimate of excess war deaths? Time to update with the information in these studies. Recall that I wrote two years ago:

If I had to bet, I would provide much wider confidence intervals than either the Lancet authors or most of their critics. Burnham et al. (2006) estimate 650,000 "excess deaths" since the start of the war with a 95% confidence interval of 400,000 to 950,000. My own estimate would center around 300,000 and range from 0 to 1.2 million.

Then, last month, I updated to:

So, my new estimate is 150,000 (at first glance, this new paper seems much better than L1 or L2) with a confidence interval of 0 to 500,000.

This still seems OK, but I think that the upper bound can start to come down. When experience scholars like the folks at IFHS and CRED come up with independent (?) estimates with upper confidence intervals well below 500,000, then I can be fairly sure that this is too conservative. So, now I go with 125,000 (shifting a little lower than IFHS because I am impressed with the view that the IFHS estimate does too much adjustment for dangerous clusters that were not sampled) and a range of 0 to 300,000.

All of this, of course, depends on the assumption that the mortality rate in Iraq, in the absence of war, would have been similar to that of Iraq in 2002 to 2003. In other words, I assume that Sadam would not have attacked Iran, gassed the Kurds, taken revenge on the Shiites and so on. Whatever probability you assign to those events, you should decrease your excess death estimates accordingly.

Another False Statement

2008-02-10T15:14:00.000-08:00

My general take (as here) is that Les Roberts is much more likely to say something false than Gilbert Burnham is. Perhaps I need to rethink this assumption. Consider Burnham's comment about/to the Wall Street Journal two years ago.

Mr. Moore did not question our methodology, but rather the number of clusters we used to develop a representative sample. Our study used 47 randomly selected clusters of 40 households each. In his critique, Mr. Moore did not note that our survey sample included 12,801 people living in 47 clusters, which is the equivalent to a survey of 3,700 randomly selected individuals. As a comparison, a 3,700-person survey is nearly 3 times larger than the average U.S. political survey that reports a margin of error of +/-3%.

By what law of political arithmetic is a survey with 12,801 people living in 47 clusters "equivalent" to a survey of 3,700 randomly selected individuals? It all depends on the design effect, on how clustered the response of interest is. And you can't know how large the design effect is until you do the survey. Again, I think that some of Moore's critique was weak, but this is confusing at best.

In addition, Mr. Moore claimed that the Hopkins study did not include any demographic data. The survey did collect demographic data, such as age and sex, related to violence, although they are not the same details Mr. Moore’s company would have collected for public opinion polls. The characteristics of households in our study are similar to other accounts of households in Iraq and the region, though the household size for the 2006 study is smaller (6.9) than found in the 2004 survey (7.9).

The debate over demographic data is key. Moore did make that claim. Roberts then responded that the survey did collect that data, as everyone would have expected it to. This made Moore look like an idiot. See here for relevant links. It was only 6 months later, once the data had been made available to me and others, that it became clear that little meaningful demographic data had been collected. But, by that time, most people had forgotten the debate. But not we Lancet fanatics!

This debate came up on Deltoid here. Note my painstaking efforts to force Lambert to print a correction after he slandered Neil Munro. One of the reasons, probably, that Lambert would be in such a rush to attack Munro on this is because Burnham and others continue to mislead on this point. Note the weasel wording:

The survey did collect demographic data, such as age and sex, related to violence, although they are not the same details Mr. Moore’s company would have collected for public opinion polls.

The survey collected age and sex for deaths, whether or not they were "related to violence." So, if someone died from a heart attack before the war started, his age and sex were recorded. But no age information was collected for the individual residents of each household. This means that it is impossible to know whether or not the sample matches up with other information about the population structure of Iraq. Although lots of information might be included under the heading "demographic," the minimum details are age and sex. If you are not collecting age/sex information, you do not get to claim that you are collecting "demographic" data.

Now, Burnham's sentence above is so convoluted that it is tough to be certain that it is false. But the next one is clearly untrue.

The characteristics of households in our study are similar to other accounts of households in Iraq and the region

That's just false. They did not collect ages, so they have no idea if the "characteristics of households" match "other accounts." Burnham is just making stuff up, allowing the reader to believe that they collected age/sex information (and that it matches other surveys) when, in fact, they did no such thing.

Again, it might be possible for a lawyer to argue (unlike in our last case) that nothing in Burnham's letter is literally false, but, put together, his statements are completely misleading. He should be embarrassed.

UPDATE: One more item. Tim Lambert (because he is a serious guy who gets the details correct) took the time to post a correction to this item from 2006. (It would be nice if Tim gave me a little credit and if he dated these corrections. The subsequent set of comments make a lot more sense if you know that his correction was added in 2008.) But the best part is that, even in 2008, Roberts does not know what went on!

I was wrong! Shannon cleaned and analyzed the data. I never saw the raw forms. We collected age and gender on everyone in 2004. That was the plan in 2006. My understanding is that they did this for some houses in the start but as that was the most lengthy part of the interview they just started recording how many people were in the house and the age and gender of the dead.

Nice story. But that's not what they did! Or at least is not what they claimed to have done. The data that has been distributed does list the number of males and females in each house but not their ages. But Roberts claims that they just recorded number of household residents. Well, which is it? Did they just make up the genders of household residents or did they collect that data? If I were smart, I would know a quick way to test to see if they were making stuff up. If they were, you would probably see too many houses with similar numbers of men and women and not enough extreme households of all men or all women. (Of course, this assumes a certain default model of household formation.)

Another item to add to the long list of things to investigate . . .

False Statement on Missing Certificates

2008-01-14T15:25:00.000-08:00

See UPDATE below.

Gilbert Burnham is an honest scientist doing his best to get to the truth, but he did not collect the data for L2; he did not travel to Iraq; he has only second-hand knowledge of what the interviewers did (as opposed to what they were supposed to do). Les Roberts is an ideologue and former candidate for Congress who will say most anything to advance the cause. Neither of them has, as far as I know, done any meaningful work with the actual data underlying L2, i.e., they rely on (the very smart) Shannon Doocy and Elizabeth Johnon to do the analysis. Put all this together and have them write a letter to the National Journal in response to Munro and Cannon's article on the L2. The results are not pretty. The NJ article included this information on missing certificates.

Under pressure from critics, the authors did release a disk of the surveyors' collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single province of Nineveh. That means the surveyors reported zero missing or lost certificates for 180 violent deaths in 15 provinces outside Nineveh. The odds against such perfection are at least 10,000 to 1, Spagat told NJ. Also, surveyors recorded another 70 violent deaths and 13 nonviolent deaths without explaining the presence or absence of certificates in the database. In a subsequent MIT lecture, Burnham said that the surveyors sometimes forgot to ask for the certificates.

Having looked at the raw data, I believe the above analysis is 100% correct. Burnham and Roberts (BR) respond that:

The statement on missing certificates is wrong. Three clusters did not have the presence of certificates noted, and in all there were 120 deaths in which the interviewers neglected to note their presence.

First, it is a bad sign that BR do not specify which "statement" they are disagreeing with. Besides the section that I quoted (which I think is what they are referring to), there are few other references to missing certificates in the article. and none seems connected to BR's comment. Second, the article does not even present what I consider the most damning aspect of the missing certificate issue: interviewers were much more likely to "forget" to ask for certificates for violent deaths and for more recent deaths. Their forgetfulness was anything but random. Third, and most importantly, what BR claim in the above is false. Let's go to the data!

> library(lancet.iraqmortality)
> x <- prep.deaths()
> summary(x$certificate)
no yes forgot
45 501 83

There were 45 deaths in which the interviewers asked for certificates but for which no certificates were found. There were 83 deaths in which the interviewers forgot to ask. (In the raw data, these cases are marked as NA. I code them as forgot because this is what Gilbert Burnham claimed happened in these cases.)

> y <- subset(x, x$certificate == "forgot")
> dim(y)
[1] 83 14
> table(y$cluster)[table(y$cluster) != 0]

1 2 14 16 18 20 22 23 24 25 30 31 32 33 34 39 40 42 46 51
1 1 3 1 1 2 1 4 10 1 1 1 4 24 7 6 8 2 1 4
> length(table(y$cluster)[table(y$cluster) != 0])
[1] 20

So, there were 20 (not 3) clusters in which the interviewers forgot to ask for at least some death certificates. But maybe Burnham and Roberts are referring to clusters in which interviewers did ask but no certificates were available?

> y <- subset(x, x$certificate == "no")
> dim(y)
[1] 45 14
> table(y$cluster)[table(y$cluster) != 0]

2 4 5 11 12 13 25 26 34 35 36 37 41 45
2 2 1 4 1 6 1 2 20 1 1 2 1 1
> length(table(y$cluster)[table(y$cluster) != 0])
[1] 14
>

There were 14 such clusters. (And, minor note, there are 128 (not 120) cases in which interviewers either forgot to ask for the death certificate or did ask but did not get to see it.)

So, what are BR talking about? Who knows? My guess is that Roberts wrote this (without really looking at the data) and then convinced Burnham to sign off. Burnham would never, I think, purposely misrepresent the data. Roberts will say whatever it takes to convince people that the L2 results are basically accurate.

As a rule of thumb, you should double-check every claim that Les Roberts makes about the raw data. Much of what he says is true. Yet a lot of really important claims are false.

I think that BR are trying to use one of their favorite tricks: looking at all the data together when the critics just want to make a point about the data from violent deaths, where we believe the real problems are. But since BR can't even provide a competent summary of the data, the whole effort makes no sense. They really should check all their empirical claims with Shannon Doocy before making them.

UPDATE 2008-04-10. I just noticed that Burnham/Roberts have changed the text of the letter without telling anyone. Classy! Now the letter is a lie because it is no longer what was "submitted to the editors of the National Journal on January 7, 2008." The offending passage reads.

The statement on missing certificates is wrong. There were 83 deaths (13%) in which the interviewers neglected to note their presence and these deaths were distributed across 20 clusters.

1) The numerical claims are now correct, as I show above.

2) But now the claim that the "statement on missing certificates is wrong" makes no sense. Nothing in Munro's article is contradicted by these numbers.

3) It is sleazy for Burnham/Roberts to make this correction without giving me credit. (I e-mailed them about it and, after not getting satisfaction, brought the issue to the attention of the General Counsel of Johns Hopkins).

4) It is sleazy for Burnham/Roberts to pretend that this was the original version of the letter.

5) It is false to (still) claim that "The following letter was submitted to the editors of the National Journal on January 7, 2008." This was not the letter that they submitted to the National Journal.

Tweaked

2008-01-13T20:10:00.000-08:00

The always informed Tim Lambert provides an approving reference to Rebecca Goldin's post at stats.org on the National Journal article. Goldin is a serious statistician. She provides an excellent overview of the dispute over the accuracy of L2.

The National Journal team did its homework, interviewing many experts (rather than conservative pundits) and categorizing the potential flaws of the study into different headings. However suspicious some facts surrounding the Lancet study might be (such as the anti-war position held by the scientists conducting the study), only two criticisms cited by the Journal raise any alarms.

One is “main street bias,” the idea that the Lancet study authors over-sampled regions near main streets, which were in turn more likely to be home to victims of car-bombs or other violence. The other is fraud – not by those who wrote the Lancet article, but by those in the field, doing the interviews under minimal supervision.

Goldin is obviously not a knee-jerk Lancet defender or attacker. I agree with her that these are, far and away, the most important criticisms of L2. I also agree with some, but not all, of her criticisms of Munro.

The Journal made a convincing argument that the data may well have been tweaked, in part based on the theory that faked data has patterns that true data rarely fit into; for example, invented people reported as killed may be more likely to be 30 or 40 than 32 or 43. It doesn’t seem unusual if any individual is 30, but it’s awfully strange if all of the deaths consist of 30-year-olds. Apparently, those conducting the Lancet study did not put enough checks in place to ensure that interviewers didn’t pad the books. The data look like inventiveness may have played a role, based on which death certificates the survey conductors reported to have seen, and which they didn’t.

So, the data "may well have been tweaked" and "look like inventiveness may have played a role." In other words, there are good reasons for suspecting "fraud," as many of us have for more than a year. Goldin is correct to note that "we should be careful in reading too much into any particular statistical anomaly" and she is right to worry about that "If those looking for fault in the Lancet study only considered a few possible ways in which the data didn’t look random, then unusually distributed data is far more damning than if they considered many, many ways and found one."

Luckily, I was among the first people to look closely at the data (and certainly the first to describe (pdf) the problems with it). I can confirm that just about the very first thing that I looked at was whether the rate of "forgetting" to ask for death certificates was correlated to date or type of death. And, sure enough, it was! We can be sure that the interviewers did not just "forget" to ask for death certificates, that they purposely asked sometimes and did not ask others.

Does that invalidate the whole study? No. Yet it provides further evidence that the US authors (like Gilbert Burnham) had only the foggiest idea of what the Iraqi interviewers were up to. And it makes any reasonable person suspicious of what else the interviewers were up to. If they felt comfortable picking and choosing which families to ask about death certificates then how can we be sure that they didn't similarly pick and choose which neighborhoods to place clusters in and which houses to visit?

At the end of the day, "tweaked" and "inventiveness" are just nice terms for "fraud." Neither Goldin nor I know, for a fact, that there was fraud in the Lancet data collection process, but much of the circumstantial evidence points in that direction.

Roberts Quotes on IFHS

2008-01-12T20:51:00.000-08:00

If I were Les Roberts, I would either keep quite about IFHS or say nothing too critical. I would know that harsh comments, while perhaps helpful in the short term in convincing people that the true number of violent deaths is much higher than 151,000, are likely to cause the IFHS authors to write another, even more critical paper. In fact, if I were Roberts, I would be very worried about the IFHS authors getting so pissed off that they might climb of the Scheuren bandwagon and demand more detailed data from L2 and/or demand data from L1.

Fortunately (for me!), Roberts does not think that way. Instead, he seems to be going out of his way to attack the IFHS results, just as he has done in the past with IBC. Here are a couple of quotes:

A paragraph in the published abstract of the report, blandly titled "Adjustment for Reporting Bias" contains an implicit confession of the subjectivity with which the authors reached their conclusions. As Sprey points out, "they say 'the level of completeness in reporting of death was 62%,' but they give no real explanation of how they arrive at that figure." Les Roberts, one of the principal authors of the Johns Hopkins studies, has commented: "We confirmed our deaths with death certificates, they did not. As the NEJM study's interviewers worked for one side in this conflict, [the U.S.- sponsored government] it is likely that people would be unwilling to admit violent deaths to the study workers."

...

If any further confirmation of the essential worthlessness of the NEJM effort, it comes in the bizarre conclusion that violent deaths in the Iraqi population have not increased over the course of the occupation. As Iraq has descended into a bloody civil war during that time, it should seem obvious to the meanest intelligence that violent deaths have to have increased. Indeed, even Iraq Body Count tracks the same rate of increase as the Hopkins survey, while NEJM settles for a mere 7% in recent years. As Roberts points out: "They roughly found a steady rate of violence from 2003 - 2006. Baghdad morgue data, Najaf burial data, Pentagon attack data, and our data all show a dramatic increase over 2005 and 2006."

Next Steps

2008-01-12T16:25:00.000-08:00

Assume for a second that you think some of the data in L2 is either fraudulent or a product of a corrupted survey process. What data would you want to look at from IFHS to test that hypothesis?

The problem is that L2 supporters can reasonably quibble with how IFHS "adjusts" for clusters that were too dangerous to visit. IFHS reports that:

Of the 1086 originally selected clusters, 115 (10.6%) were not visited because of problems with security. These clusters were located in Anbar (61.7% of the unvisited clusters), Baghdad (26.9%), Nineveh (10.4%), and Wasit (0.8%). Since past mortality is likely to be higher in these clusters than in those that were visited during the IFHS, we imputed mortality figures for the missing clusters in Anbar and Baghdad with the use of information from the Iraq Body Count on the distribution of deaths among provinces to estimate the ratio of rates of death in these areas to those in other provinces with high death rates.

(It is not clear to me why the authors did not perform a similar adjustment because of the missing clusters in Nineveh and Wasit. Perhaps the number of missing clusters was too small to matter? Perhaps the IBC data was not detailed enough to work with?)

Anyway, reasonable people will disagree over whether or not the IFHS adjustments for this problem were too small (L2 supporters) or too large (IBC supporters). On my list of to-dos is trying to calculate what the violent death estimate is without the adjustment. The trick to avoiding the whole morass is just to ignore these governorates altogether. Instead, throw out these 4, plus the 3 in Kurdistan (where everyone agrees things have been peaceful) and focus on the 11 others. Or just the subset of these 11 in which L2 (implausibly) reports extremely high violent mortality.

The beauty of this approach is that the IFHS estimates for these "complete" governorates require no adjustments. No clusters were skipped. Every household was checked. There is no good reason for the IFHS and L2 estimates to be that different. The confidence intervals for the IFHS will be much narrower because there is no extra uncertainty associated with the adjustment for missing clusters. Also, because we are ignoring Kurdistan, we will get a much more focussed look at the differences between IFHS and L2.

And, back of the envelope, those differences will be huge. As the IFHS authors note:

All three sources agreed on the low mortality in Kurdistan. Of all the violent deaths occurring in Iraq, the proportion in Baghdad was 54% in the IFHS, 60% in the Iraq Body Count, and only 26% in the study by Burnham et al.

In other words, the big differences between L2 and IFHS are not in Baghdad. (In fact, and also on the to-do list, it is not clear to me that L2 and IFHS disagree that much about Baghdad, especially if we throw-out the deeply suspect results from cluster 33 in L2.) So, the disagreement in the remaining clusters will be large. Combine a large raw difference with narrow confidence intervals for the IFHS estimates, and you have a recipe for, as the Marxists say, heightening the contradictions.

Reconciliation

2008-01-12T09:56:00.000-08:00

One issue that has come up in the Lancetosphere is how much the IBC estimates are undercounts and what light, if any, IFHS and other sources shed on the topic. This is a hard problem with no easy solution. Here is my preliminary take:

Assume for a second that the IFHS estimate of 150,000 total violent deaths for March 2003 through June 2006 is correct. How does that gibe with the 50,000 civilian violent deaths as reported by IBC? (Actually numbers are 151,000 for IFHS and 48,000 for IBC, but I am rounding with abandon.) Both are, obviously, much lower than the 600,000 estimate from L2. (Since there were almost no pre-war violent deaths in L2, "violent deaths" and "excess violent deaths" are the virtually identical in L2.) IFHS is 1/4 L2 and IBC is 1/3 IFHS.

First, the US military was killing, during the invasion, thousands of Iraqi troops and, after the invasion, thousands of insurgents. Those numbers are counted in the IFHS total but not by IBC. Donald Johnson points out this article which reports that the US military claims that it killed 19,000 insurgents from June 2003 through September 2007. Guestimating from their table, it seems like the toll up to June 2006 might be 13,000. But this does not include Iraqi military deaths for the invasion itself. Skimming this Wikipedia article, 10,000 seems a reasonable estimate. So, rounding up, we might have about 25,000 Iraqi soldiers and insurgents killed by US forces during this period. I do not have a sense of whether this is more likely an overestimate or an underestimate.

Second, insurgents were killing many non-civilians in the Iraqi population. The deaths of Iraqi soldiers are not counted by IBC but are included in IFHS. I think that the same is true for police officers. Anyway, it is certainly the case that thousands of Iraqi combatants have died, presumably somewhere between US combat deaths around 2,000 and insurgent deaths of 13,000. Call it 5,000.

Third, there has been a great deal of insurgent/insurgent violence. That is what happens when a civil war starts. Some of these are clearly "civilian deaths." When the local Sadr militia picks up a Sunni man minding his own business and kills him, that is clearly a non-combatant death and should be counted by IBC. But when two armed groups are fighting, as in much of the intra-Shia violence, it is not clear if those deaths are counted (by IBC) as civilian or if they should be. Could there be another 20,000 such deaths? Sure.

So, we have 30,000 Iraqi military/insurgent deaths caused by the US military plus 20,000 Iraqi deaths involving combatants in the civil war. (Again, I don't put any particular faith in these numbers and haven't look closely for good data. The point is that there are certainly tens of thousands of deaths, at a minimum, that are included in IFHS but are, by definition, excluded from IBC.

In summary, there are 50,000 IBC civilian deaths plus 50,000 soldier/insurgent/combatant deaths yielding 100,000 total. But this is 50,000 less than IFHS. Fine. I don't see that discrepancy as a big one. Could IBC be off by a factor of 2? Sure! Could IBC be correct but my summary of Iraqi combatant deaths be off by a factor of 2? Sure!

In general, the IBC and IFHS numbers are broadly consistent (as both the IFHS and OBC principal investigators would no doubt agree) because IBC is counting a subset of the deaths captured by IFHS.

Said and Unsaid

2008-01-12T09:00:00.000-08:00

There is much fun discussion in the Lancetosphere about the IFHS study. Lancet defenders are doing everything they can to insinuate that the IFHS results support or validate or, at least, are not inconsistent with L1 and L2. And, to some degree, they are reasonable to do so. For example, both L1 and L2 report that Iraq has 18 governorates. IFHS agrees!

Perhaps that is a bit snarky, but it is important to distinguish between what the authors of a study actually say and what you conclude about the studies data, models and results. For example, the IFHS authors say nothing about L1. They do not even cite it among their references. Perhaps they think that L1 is a great paper and their results support it 100%. Perhaps they think that L1 is completely wrong and should be retracted by the Lancet. They don't say and so we don't know.

But the IFHS authors do make very specific claims about L2. From the abstract:

When underreporting was taken into account, the rate of violence-related death was estimated to be 1.67 (95% uncertainty range, 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range, 104,000 to 223,000) from March 2003 through June 2006.

Conclusions: Violence is a leading cause of death for Iraqi adults and was the main cause of death in men between the ages of 15 and 59 years during the first 3 years after the 2003 invasion. Although the estimated range is substantially lower than a recent survey-based estimate, it nonetheless points to a massive death toll, only one of the many health and human consequences of an ongoing humanitarian crisis.

In other words, the number of violent deaths estimated by L2 is too high. They are making a claim that a specific number from L2, the 600,000 violent deaths, is an overestimate. (Note that "violent deaths" (IFHS) and "excess violent deaths" (L2) refer to almost exactly the same underlying number because there were almost no violent deaths before the war.)

To mention in the abstract that a different peer-reviewed article is wrong is a strong statement of the beliefs of the IFHS authors.

The most striking difference in rates of death was between those in the study by Burnham et al. and those in the two other data sources for the six high-mortality provinces, which accounted for 64% of all deaths in the study by Burnham et al.

What is most surprising, to me, is how critical the IFHS authors are of L2 (Burnham et al), how they go out of their way to imply that L2 is wrong. Note especially Figure 1. If your main point was the L2 was wrong, I do not think that you could have constructed this figure in a more accusatory fashion.

Of course, just because the IFHS authors think that L2 is wrong does not mean that it is. But Lancet supporters should not pretend that the IFHS authors think that L2 (or L1) is correct about anything when they provide almost nothing but criticisms of L2. Now, that criticism is couched in the polite language of an academic paper, but, given the constraints of acceptable dialog in the New England Journal of Medicine, could the IFHS authors written anything more critical of L2? I don't see how.

There was greater agreement regarding mortality from nonviolent causes between the IFHS study (372 deaths per day) and the study by Burnham et al. (416 deaths per day)

This is one of the few (only?) places were IFHS offers support to L2. But this support is more damning than helpful because none of the serious critics of L2 have complained about their estimates of post-invasion non-violent mortality. After all, the increase that L2 found (50,000) was not even statistically significant. As Michael Spagat wrote:

The main problem with the comparison highlighted by the L2 authors [in asserting that L2 validates L1] is that it is of all excess deaths, not just violent deaths. All suggestions of possible bias in L2 that we know of, sampling or non-sampling, pertain to violent deaths. The available facts simply do not support a claim that L1 and L2 suggest very similar numbers of violent deaths. By persistently conflating non-violent deaths with violent deaths the L2 authors have obscured this essential point.

Correct. To say that the L2 estimates of non-violent death (not "excess non-violent death") are reasonable is to congratulate them on the fact that they got the number of governorates in Iraq correct. It is true, but faint praise.

The most striking difference in rates of death was between those in the study by Burnham et al. and those in the two other data sources for the six high-mortality provinces, which accounted for 64% of all deaths in the study by Burnham et al.

People like me think that the reason for this is that the L2 data from those provinces is fraudulent, either made up out of whole cloth or derived from a corrupted sampling scheme. One of my guesses is:

The L2 interviewers, for whatever reason, decide that they want to report more deaths. They go to a cluster and, following the procedure outlined in Burnham's discussion, gather all the neighborhood kids around and tell them about the survey. So far, so good. But then they ask the kids, "Has anyone in the neighborhood died in the last couple years, especially violently?" The kids know this and tell them. Then the interviewers preferentially select those houses, either a picking and choosing around the neighborhood or just placing the 40 house cluster in the part of the neighborhood that, by chance, had the most deaths. In that scenario, all the data is "accurate" in the sense that no one is making anything up, but the mortality estimate will be much too high.

Access to the demographic data might allow us to catch that because the houses would have way more young men than a random sample should have, given what we know about the age/sex distribution in Iraq from sources like ILCS.

Now, of course, I wasn't in Iraq for the surveys. But a corrupted interview process would help explain why the L2 results in these provinces are so wildly divergent from those of IFHS and IBC. Note also that this could have occurred without the knowledge of any of the L2 authors. Only Lafta was in Iraq for the survey and, since there were two teams operating independently, the fraud (if that is the cause) might have happened without his knowledge. This is why people like Fritz Scheuren want to see the underlying data classified by interviewer. This is why everyone would have liked to see demographic information. Remind me again about why Les Roberts lied to Tim Lambert about the whether or not age data was collected for individual households.

Back to IFHS. Table 4 is another example of the IFHS authors arguing, politely, that the L2 results are completely implausible. Could any critic of L2 constructed this table in a more damning fashion? Not that I can see.

And the final statement comes in the Discussion.

The IFHS results for trends and distribution of deaths according to province are consistent with what has been reported from the scanning of press reports for civilian casualties through the Iraq Body Count project. The estimated number of deaths in the IFHS is about three times as high as that reported by the Iraq Body Count. Both sources indicate that the 2006 study by Burnham et al. considerably overestimated the number of violent deaths. For instance, to reach the 925 violent deaths per day reported by Burnham et al. for June 2005 through June 2006, as many as 87% of violent deaths would have been missed in the IFHS and more than 90% in the Iraq Body Count. This level of underreporting is highly improbable, given the internal and external consistency of the data and the much larger sample size and quality-control measures taken in the implementation of the IFHS.

"Highly improbable" is New England Journal of Medicinese for "total crap."

And this is where things get fun. Lancet redoubts like Crooked Timber and Deltoid are filled with hopeful attempts to rescue L2 from IFHS, to argue that, by some excess death calculation that the IFHS authors do not use, IFHS supports L2. Perhaps. All of us spend time trying to use the raw data and models of a given paper to answer questions that the papers authors fail to address. Who knows? Perhaps the next paper from the IFHS team will be an excess death calculation that matches perfectly with L1 and L2. Hah! Anyone with experience reading scientific papers knows that the IFHS paper could not have been more critical of L2. Future work from these authors will almost certainly continue in the same vein. Anyone want to bet otherwise?

There is no way to spin the IFHS paper as anything less than a total rejection of the L2 violent death estimates. That doesn't mean that the IFHS authors are right, but we need to be clear on what they say.

IFHS Versus L1

2008-01-12T08:49:00.000-08:00

Some Lancet supporters claim that the results from IFHS support L1. I think that this is wrong. Consider a comment by me from Crooked Timber.

The issue before us is: How does the 151,000 estimate of violent deaths in all of Iraq from IFHS compare with L1? Now, since the surveys use different terminology over a different time scale, we will not be able to make an exact comparison. But, L1 reports 73 violent deaths in all of Iraq post-invasion compared to 1 pre-invasion (from the phantom US bombing runs, no doubt.) (See Table 2.) Speaking very roughly each excess death in the sample corresponds to 3,000 or so deaths in the population. So, for all of Iraq, there were around 200,000 violent deaths in L1 through September 2004. (I am obviously skirting over the, in this context, unimportant distinction between violent deaths and excess violent deaths.)

IFHS estimates 151,000 violent deaths through June 2006. Relative to IFHS, the L1 estimate is ludicrously high.

Again, this is just back of the envelope, but I wanted to help Tim clean out his garage, following Kieran’s kind suggestion.

Now, Tim might argue that we need to exclude Falluja for this that or the other reason. Fine. If the L1 authors had just dropped Falluja from all of their analysis (or included it everywhere or done both), I wouldn’t have objected so much. But they picked and chose. Yet, in this context, you don’t get to play that game. The IFHS authors estimate 151,000. That is their number. You can either try to get numbers out of L1 that are comparable to that (as I do above). Or you can claim that such a comparison is impossible. But you can’t just claim the comparison works for the subset of the IFHS data that you want to look at.

Crooked Timber Comment

2008-01-12T07:26:00.000-08:00

I comment that I made at Crooked Timber is worth repeating here. I have also been active in Deltoid recently: here, here, here and here. There is no rest for the wicked. Of course, the master plan is to convince smart folks like Tim Lambert and Daniel Davies that the underlying data from L2 is suspect. Once they are on my side, things will go much faster.

----------
Did someone call for me? Hello Crooked Timber! I have been banned from Henry and dsquared's threads, but, since Kieran has not banned me, I assume he does not object to my contribution. Kieran writes:

A study like this gives us good reason to substantially revise our estimate of the total number of excess deaths downward. The Burhnam et al estimate of excess deaths looks like it was too high, assuming that the new survey is basically reliable.

Exactly correct! Last year, my estimates were:

If I had to bet, I would provide much wider confidence intervals than either the Lancet authors or most of their critics. Burnham et al. (2006) estimate 650,000 "excess deaths" since the start of the war with a 95% confidence interval of 400,000 to 950,000. My own estimate would center around 300,000 and range from 0 to 1.2 million. Obviously, no one is really interested in my estimate --- derived as it is from reading the literature and associate debates --- but I thought it reasonable to be upfront about my prior beliefs.

In retrospect, I should have placed more weight on the informed estimates of people like Jon Pedersen. He estimated violent deaths at 100,000 (1/6th of the L2 estimate) and that sure matched up nicely with the 150,000 from IFHS. So, my new estimate is 150,000 (at first glance, this new paper seems much better than L1 or L2) with a confidence interval of 0 to 500,000.

It would be interesting to read the current estimates of folks like Kieran and dsquared. Note that I wrote a month ago:

Where is the debate going? I sometimes worry that, like so many other left/right disputes, this will never be resolved, that we will never be sure whether or not the Lancet articles were fraudulent. Will these estimates be the Chambers/Hiss debate of the 21st century? I hope not. Fortunately, other scientists are hard at work on the topic, reanalyzing the data produced in L2 and conducting new surveys. Both critics and supporters of the Lancet results should be prepared to update their estimates in the face of this new evidence. If independent scientists publish results that are similar to those of the Lancet authors, then I will recant my criticism. Will Lancet supporters like Lambert and Davies do the same when the results go against their beliefs? I have my doubts.

I should not have doubted Kieran's willingness to update his estimates. My apologies! dsquared, on the other hand, is acting about how I suspected. Is there any new information that would cause him to doubt the L2 results?

Because more stuff is coming! What's most interesting about IFHS is how they went out of their way to attack L2. They didn't need to do that. They could have been much nicer. They could have spun the story as Roberts and dsquared would like to. Instead, they go for the jugular, as much as you can in the NEJM. They highlight how their confidence interval rejects the L2 range by more much more than 100,000 deaths. They don't just argue that they are right. They argue that L2 is very, very wrong.

Who is right? Time will tell. Did everyone catch how Horton was shoving the L2 authors off the sled in the National Journal article?

Today, the journal's editor tacitly concedes discomfort with the Iraqi death estimates. "Anything [the authors] can do to strengthen the credibility of the Lancet paper," Horton told NJ, "would be very welcome." If clear evidence of misconduct is presented to The Lancet, "we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry."

Hardly a ringing endorsement! Perhaps Richard Horton knows/suspects that something is not right with the L2 data . . .

Where is this going? The wheels of science grind slowly, but they grind very fine indeed. If the data underlying L1/L2 is fake, then the Lancet papers will be the most important scientific fraud of the decade. Think that is impossible? Think again.

What can the Crooked Timber community do? Act like scholars and scientists. (As Kieran does in this post.) Keep an open mind. Consider all the evidence. Look at the underlying data. Study the statistical models. Replicate the results. Make our findings public.

One small step would be for dsquared to allow me to publish comments in his threads on this topic. But perhaps open discussion and debate is not what he is looking for.

National Journal Article

2008-01-04T06:29:00.000-08:00

Neil Munro's National Journal article is out. I haven't had a chance to read it closely, but my quotes are not as contextualized as I would like them to be.

Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent -- and repeatable -- to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?

"The authors refuse to provide anyone with the underlying data," said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at Harvard University.

That is correct, but it is important to note that the authors' behaviour was much better in L2 than in L1. In L2, most researchers were provided with some of the data. I attribute this to goodwill and professionalism on the part of lead author Gilbert Burnham. But, it is still pathetic that they refuse to share the data with Spagat et al and that they have yet to (will never?) allow Scheuren and others to see if there are problems with different interviewers providing anomalous results. In L1, their behaviour has been horrible, due mostly, I believe, to Les Roberts' attitude. No one has seen the underlying data for L1, other than cluster-level summaries. This is not the way that scientists ought to behave.

On this topic, I wish that Munro had quoted me about the fact that, as far as I know, no scientific team has ever granted data access, however incomplete, to some critics but not others. It is inexcusable for the Lancet authors to show data to me but not to Spagat.

To Kane, the study's reported response rate of more than 98 percent "makes no sense," if only because many male heads of households would be at work or elsewhere during the day and Iraqi women would likely refuse to participate. On the other hand, Kieran J. Healy, a sociologist at the University of Arizona, found that in four previous unrelated surveys, the polling response in Iraq was typically in the 90percent range.

Again, this is an accurate quote, but the context is off. My key point is not about how much time Iraqi men are away or how likely Iraqi women are to participate in a survey. Who knows? My key point is that there has never been a single-contact survey with 98%+ participation, in any country at any time on any topic. Never. What are the odds that the most controversial survey of the decade with achieve an unprecendented repsonse rate? More background here.

The authors should not have included the July data in their report because the survey was scheduled to end on June 30, according to Debarati Guha-Sapir, director of the World Health Organization's Collaborating Center for Research on the Epidemiology of Disasters at the University of Louvain in Belgium. Because of the study's methodology, those 24 deaths ultimately added 48,000 to the national death toll and tripled the authors' estimate for total car bomb deaths to 76,000. That figure is 15 times the 5,046 car bomb killings that Iraq Body Count recorded up to August 2006.

According to a data table reviewed by Spagat and Kane, the team recorded the violent deaths as taking place in early July and did not explain why they failed to see death certificates for any of the 24 victims. The surveyors did remember, however, to ask for the death certificate of the one person who had died peacefully in that cluster.

First, where is documentation for the claim that the survey was supposed to end on June 30th? I have never heard of that. In fact, I doubt it. When you go and start fieldwork for a survey in some war-torn country, you certainly have a plan and a schedule that you hope to keep. I believe that they wanted to finish by June 30th. But why would the study protocal require that? It wouldn't. There is no reason to put on such a straight-jacket. Instead, the protocal called for getting 50 clusters. However long that took is how long it would take.

Of course, I am still deeply suspicious of the results for that cluster. Finding a whole bunch of deaths at the end of the survey --- and in a category, car bombs, that you wanted/expected to much larger than the data that you had gathered so far --- is awfully convenient, just as finding scores of deaths in Falluja at the very end of L1 was convient. But the June 30 date is, as far as a I can tell irrelevant.

Second, I agree that the authors did not explain why they did not ask for death certificates in that specific case. But a plausible explanation would be that the deaths happened a day or two before the survey and that, therefore, the interviewers knew that the families would not yet have death certificates available. So, why ask for them? As always, the authors should be a lot more transparent and willing to answer questions, but I think that they have plausible responses to these issues.

None of which means that I believe those answers. My guess continues to be that the/some interview teams went to a neighborhood and asked the kids who had died and then interviewed those houses preferentially. I suspect that they went out looking in early July for a neighborhood with car-bomb deaths, even went to that specific neighborhood after they heard about the car-bomb on the news. But suspicions are not proof. I would be happy to bet, however, that Lafta was a part of the team that did those interviews, just as he was the one to go to Falluja for L1.

I think that the issue about car-bomb deaths that is most damning is how the authors pretended in the paper that there was a gradual rise in such deaths, consistent with news reports and IBC, over the course of the time period when, in fact, car-bomb deaths were constant for the two years prior to July 2006. Alas, Munroe does not make that point.

I think that the table associated with the article is fine as far as it goes. But I wish that Munro had used my tables which show how "forgetting" to ask for death certificates was much more common for later deaths and for violent ones. That is the damning evidence that something more than forgetfulness was going on when the interviewers failed to even check for death certificates.

But all these are quibbles. Munro has done a fine job in gathering all sorts of evidence and arguments. I spoke with him several times and there is no doubt that he understands the ins and outs of the debate.

I hope to have more substantive comments on the article in due course.

TCS Daily Article

2007-12-12T06:24:00.001-08:00

I have a brief article in TCS Daily entitled "Thoughts from a Lancet Skeptic." The resulting comment thread is not, uh, very useful. Some editing was done on the piece after I submitted it. For those who care, here is the original version (note the nice things I had to say about Daniel Davies).

-----
Thoughts From A Lancet Skeptic

The British medical journal The Lancet published two articles about increases in mortality in Iraq caused by the US-led war and occupation. The authors estimated "excess deaths," the number of Iraqis who had died in excess of what would have happened if the pre-invasion mortality rate had not changed. The first paper (termed L1), with lead author Les Roberts, argued that 98,000 Iraqis had died because of the war and occupation through September 2004. The second (L2) updated the estimate to 655,000 through July 2006. Both articles were published shortly before the US November elections in an explicit attempt to affect the US political process.

Taken together, these two articles are the most important and controversial scientific publications of the decade, generating endless discussion among policymakers and the public. President Bush and former British Prime Minister Tony Blair were questioned about the authors's claims in press conferences. However, the estimates for excess deaths provided by the Lancet authors are almost certainly too high and possibly fraudulent. I was among the first scientists to make this claim. I will not be the last.

An Accidental Critic

How did I come to participate in this dispute? I have no background in survey research and no expertise in Iraq. But I do make a habit of reading blogs written by smart people with different views from my own. Indeed, there is no better way to test your beliefs than to confront the best proponents of alternate theories. To that end, I read Tim Lambert's blog Deltoid. Lambert, whatever his other faults, is careful and intelligent. He gets the details right. He is also a fierce critic of many of the articles and authors (i.e., Michael Fumento and Tim Ball) at TCS Daily. Lambert has defended the Lancet surveys extensively and, for the most part, his defense is correct. Indeed, the Lancet authors have benefited from the stupidity of many of their critics, as documented by bloggers like Daniel Davies. In Lambert's marvelous phrase, the articles have been like "flypaper for innumerates."

Yet just because many of the Lancet critics are clueless does not mean that the articles themselves are correct. My first reaction to L1 was that its confidence interval for excess deaths, 8,000 to 194,000, was suspiciously close to zero. Every good statistician is skeptical of a result which just barely rejects what researchers call the "null hypothesis," in this case that mortality in Iraq was unchanged after the invasion. A small change in the model assumptions could easily make the effect go away. Since it was obvious that the Lancet authors had political motivations/ambitions (Roberts ran for Congress in 2006), I thought that they were probably guilty of cherry-picking their model, at least to some extent. They would not be the first researchers to do so.

Yet these suspicions were, for me, overwhelmed by my disgust with the behavior of the authors and their supporters. Although they provided some summary data for L1, they refused to divulge the household-level data and computer code that would help outside researchers (like me) to replicate their results. This is not the way that scientist ought to act. But, since life is short and I am only a part-time academic, I left the issue behind after exchanging some e-mails with the authors in 2005. And that's where things would have stayed for me had the authors not published L2 in 2006. At that point, I felt a moral obligation to get to the bottom of this story. And so, here I am.

Criticisms

In August, I made a presentation at the annual meeting of the American Statistical Association which argued that the results of the first Lancet survey were internally inconsistent. The technical details are opaque at best but the implication is that the authors purposely presented their data (by including outlier data from Falluja in some parts of the analysis and excluding it elsewhere) to mislead. Specifically, the 8,000 to 194,000 confidence interval is claimed to be "conservative" because it excludes the carnage in Falluja. I show that including Falluja would have widened the confidence interval enough to include zero, thereby not allowing the authors to reject the null hypothesis of no increase in mortality.

In English, my claim is that the authors specifically refused to provide the confidence interval for excess deaths using all their data because they knew that doing so would provide too much ammunition to their critics. Even today, they stubbornly decline to tell me or anyone else what the confidence intervals would be with Falluja included.

But their behavior is even worse than that. They refuse to share the underlying data and computer code with the wider scientific community. Although some of the Lancet authors have conducted themselves professionally throughout the dispute (I would especially commend Gilbert Burnham and Shannon Doocy), Les Roberts has forced a sort of worst common denominator behavior on the team as a whole, including Elizabeth Johnson, the statistical consultant who performed the actual analysis.

That paper was discussed at Deltoid and then picked up from there by Michelle Malkin. Suddenly, I was part of the Right Wing Noise Machine, even invited as a guest on my local talk radio station. Alas, I misinterpreted the orders from my Rovian overlords and spent most of the time defending the Lancet authors from the innumerate complaints that the host was making. He choose not to keep me on the air long enough to get to the point of my actual critique.

Fortunately, other scientists are working on the topic. Colin Kahl writes that the Lancet estimates are "dubious." Fritz Scheuren, past president of the American Statistical Association, claims that the response rates from L2 are "not credible." Stephen Fienberg, one of the most respected statisticians in the country, insisted that Les Roberts' refusal to share data with Michael Spagat and his co-authors was:

just the wrong response. I, as an editor, would not publish a study for which the data was not shared.

If scientists like Kahl, Scheuren, Fienberg, Spagat and others think that your results are flawed and your behavior suspect, then you have a problem.

The Future

Where is the debate going? I sometimes worry that, like so many other left/right disputes, this will never be resolved, that we will never be sure whether or not the Lancet articles were fraudulent. Will these estimates be the Chambers/Hiss debate of the 21st century? I hope not. Fortunately, other scientists are hard at work on the topic, reanalyzing the data produced in L2 and conducting new surveys. Both critics and supporters of the Lancet results should be prepared to update their estimates in the face of this new evidence. If independent scientists publish results that are similar to those of the Lancet authors, then I will recant my criticism. Will Lancet supporters like Lambert and Davies do the same when the results go against their beliefs? I have my doubts.

100% Response Rate

2007-10-25T11:40:00.000-07:00

My main source of suspicion with regard to Lancet II concerns its amazingly high response rate. See previous discussion here, here and here. Kieran Healy counters that concern by writing:

The immediate problem with this charge is that, as it turns out, phenomenally high response rates are apparently very common in Iraq, and not just in this survey. UK Polling Report says the following:

The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that – incredibly – response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.

If this is correct, then the only bit of circumstantial evidence that Kane proffers in support of his insinuation is in fact a misconception based on his own ignorance.

Related discussion here. Healy is a serious scholar so his objections are worth careful study. The paper and associated slides are available. Keep in mind that we have three levels of references: Healy talking about what UK Polling Report (a blog written by Anthony Wells, not a formal organization) says about a paper by John Heald of Opinion Research Business (ORB).

Looking at the paper and slides, it is clear that, although Heald (presumably) presented the paper, the other author, Munqith Daghir of IIACSS in Iraq was responsible for the polling results. (It is not clear to me if Heald has ever been to Iraq, much less conducted a poll there.) ORB seems (?) like a competent organization, but its website makes little mention of doing work in Iraq. Instead, it reports that:

Research professionals at ORB have worked with over 15 world leaders on their Public Affairs initiatives including studies in the UK, US, Russia, South Africa, Malaysia, Taiwan, Turkey, Bulgaria, Lithuania, Malta and Gibraltar.

This report indicates that ORB first started to collaborate with Daghir sometime in 2005. The paper referenced by Healy/Wells reports that:

One man who listened to what his people allegedly thought about the invasion was Dr Munqeth Daghir, a lecturer in the University. He was caught between occupying forces saying that they had come to liberate Iraq and protect them from the Baathist regime and Iraqi exiles slowly returning saying that they wanted to help and could represent the real Iraqi. However, he knew that neither was a real reflection of what Iraqis really thought and wanted.

So, having read a book about market research and polling in between avoiding bombs that were dropping on his city, the first poll was conducted in Baghdad amongst a representative sample of 1,000 adults. That was in April 2003 and more than three years later and having the benefit of carrying out more than 150 polls, we want to demonstrate the advantages and difficulties of polling in Iraq and then quantify what people really thought both then (post conflict) and now.

Daghir is to be congratulated for making a life for himself as a pollster amidst the chaos of Iraq. But "having read a book about market research and polling" is not, shall we say, the most impressive pollster resume the world has ever seen. The entire article continues in a similar style. This is not so much a "paper" on the polling situation in Iraq as an advertisement for Daghir and his company. Now, there is nothing wrong with advertisements, and I have no reason to believe that Daghir and IIACSS aren't high quality pollsters, but Healy cites all this to demonstrate that the response rate for Lancet II is not an outlier result, that it is typical of Iraqi polls. Note again the quote from Wells which Healy selects and then comments on.

Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.

If this is correct, then the only bit of circumstantial evidence that Kane proffers in support of his insinuation is in fact a misconception based on his own ignorance.

Speaking of "ignorance," it is not clear if Healy (or even Anthony Wells) ever read the report in question. It does not mention ORB! It seems to me that Wells/Healy are mistaken, that ORB had no "experience of polling in Iraq" prior to 2005, that "they" did none of this work. Instead, Daghir/IIACSS have done some subcontracting for ORB in 2005/2006 and both IIACSS and ORB are eager to do more business together. But, Healy/Wells have no business claiming that ORB had anything to do with the poll results prior to 2005.

Consider this January 2006 here with Daghir.

In under three years he [Daghir] has run over 100 surveys for the UN, international NGOs, foreign governments, ad agencies, the media and FMCG companies. He has also carried out research on behalf of UK-based research agency ORB for almost a year now, looking at public attitudes towards topical issues and is about to explore attitudes towards tobacco.

In other words, Daghir has been working with ORB for "almost a year" as of January 2006. They had no relationship that we know of back in November 2003 when the implausible 100% response rate poll was conducted.

At that time [April 2003] I hadn’t much money. All of our savings in the bank had been looted. We hadn’t received the payment for most of the government projects we had done. So we were mostly bankrupt. I sold my car, and my partners sold things to fund this project. And 14 of my students on masters and PhD degree courses agreed to work with me for free. My daughter and my son worked as data punchers. I used my own computer with a generator. We started like a family business, in a big room with my son, my daughter, my partner’s son, my partner’s daughter, working together with our students. I told them how to code the questionnaire, how to enter the data. After two weeks I started the fieldwork.

It is a nice story and Daghir deserves credit for his ambition and bravery. But, he was novice, self-taught surveyer. What are the odds that he got everything correct the first time?

I knew that Baghdad is distributed into nine different areas, and how many citizens lived in each one. But to tell the truth, I didn’t know anything about the real random systematic sample. We did it randomly by going to any house we wanted to go to. So it wasn’t a perfect sample.

Indeed.

The key table is on page 2 of the report. There are six surveys listed. The first one, from November 2003, reports a 100% "Response Rate" in sample of 1,167. This is, of course, absurd. A proper poll picks out the 1,000 or so people it wants to sample from a larger population. It then searches for those people. You can never, ever find them all, no matter how "friendly" the local population. A charitable interpretation is that these are more "participation rates" than "response rates." (Background on terminology here.) In other words, the interviewers kept on looking for people to contact, perhaps by going around a market, perhaps by traveling from house to house. Some houses were empty. Some people refused to answer the door and/or talk with them. But, of those that they did contact, most were willing to participate. (The fact that the first column is labeled "Total Contacts" makes this plausible.)

The main problem with Lancet II is not that participation rates were high. That's true for polling all over Iraq. The problem is with the contact rates. These polls seem to be ignoring that aspect in their reporting. No poll of 1,000 people finds everyone.

And, even if we wanted to believe this outlandish result, it would still have little bearing on the Lancet II data because the polls are almost 3 years apart. Perhaps response rates were extremely high in November 2003, but the Lancet II interviews were done in 2006. The remaining 5 polls in the table have an average response rate of 87%, with none higher than 91%. How did the Lancet surveys do 10% better?

Kahl on Casualties

2007-10-24T09:38:00.000-07:00

As time passes, more and more critiques of the Lancet results are appearing in the peer-reviewed academic literature. One of the first is from Colin Kahl. (Hat tip to Henry Farrell.) The citation is: Colin Kahl (2007), “In the Crossfire or the Crosshairs? Norms, Civilian Casualties, and U.S. Conduct in Iraq,” International Security 32:7-46. These quotes come from pages 11--13.

There are numerous estimates of civilian casualties in Iraq, all of which are problematic. Iraq Body Count (IBC), a nonprofit organization that derives estimates from English-language and translated media sources, suggests that as many as 7,393 civilians may have been killed during major combat. Another estimate by the Project on Defense Alternatives, which corroborated media reports with hospital and burial records and filtered out likely combatant fatalities, suggests that 3,230–4,327 Iraqi civilians were killed.12 In comparative perspective, it is significant to note that the number of Iraqi civilians killed during the 2003 invasion was similar to that of the 1991 Persian Gulf War, even though the mission objectives in 2003 required coalition forces to operate much more extensively in Iraqi cities, where they confronted both regular Iraqi units and irregular Fedayeen fighters largely indistinguishable from the civilian population.13

For the SO/COIN period, the two leading estimates come from IBC and the Brookings Institution. IBC estimates that as many as 54,303 Iraqi civilians were killed from May 1, 2003, through the end of 2006. The maximum number attributed to U.S. forces or crossfire is 4,399. That figure, however, is probably an underestimate because it includes only 87 deaths from U.S. actions at checkpoints and during convoy operations, known as “escalation of force” (EOF) incidents. If estimated EOF fatalities based on U.S. military figures are added to the IBC numbers and adjusted for double counting, the modified IBC total is 5,429 deaths attributable to U.S. forces and crossfire from the declared end of major combat through the end of 2006 (representing 10 percent of the total violent deaths).14 Over the same time period, the Brookings Iraq Index—which supplements IBC data with hospital and morgue records reviewed by the UN Assistance Mission for Iraq (UNAMI), and makes a further upward adjustment to account for likely undercounts from media reporting—estimates that 76,552 Iraqi civilians were killed.15 The Brookings Iraq Index does not provide a comprehensive estimate of deaths attributable to U.S. forces or crossfire, but assuming the same percentage documented by IBC and adding EOF incidents produces an adjusted Brookings estimate of 8,685 deaths in the SO/COIN period through the end of 2006.

Another well-known study, based on an Iraqi household survey and reported in the British medical journal The Lancet in October 2006, estimated 601,000 excess violent deaths since the invasion.16 Although widely cited as an estimate of civilian deaths, the majority of the 300 postinvasion violent deaths recorded in the sample of 1,849 households were military-aged men (59 percent aged 15–44; 78 percent aged 15–59), and no attempt was made to differentiate between combatants and noncombatants. Moreover, the Lancet estimate is so much higher than other available tallies, and its findings suggest so many implausible implications given other available data on the nature of the conflict, that its conclusions seem dubious, perhaps reflecting some significant sampling bias or reporting error.17 Indeed, a much larger household survey funded by the UN Development Programme in 2004 suggests that the IBC and Brookings figures are probably much closer to the actual number of civilian deaths.18 Jon Pedersen, the research director for the UNDP study, recently stated that he believes the Lancet numbers are “high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much.”19 Consequently, the Lancet numbers are not used for the analysis here.

The associated footnotes are:

12. Iraq Body Count, http://www.iraqbodycount.org; and Carl Conetta, “The Wages of War: Iraqi Combatant and Noncombatant Fatalities in the 2003 Confiict,” Research Monograph, No. 8 (Cambridge, Mass.: Project on Defense Alternatives, October 20, 2003), http://www.comw.org/pda/0310rm8.html.
13. Estimates of direct civilian deaths during the Gulf War range from 3,500 to 15,000. Conetta, “The Wages of War”; and Medact, Collateral Damage: The Health and Environmental Costs of War on Iraq (London: Grayston Center, November 2002), p. 2.
14. ”Murder Is Certain,” Economist, March 23, 2006; and Josh White, Charles Lane, and Julie Tate, “Homicide Charges Rare in Iraq War,” Washington Post, August 28, 2006.
15. Michael O’Hanlon and Jason H. Campbell, Iraq Index (Washington, D.C.: Brookings, April 26, 2007), p. 15.
16. Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts, “Mortality after the 2003 Invasion of Iraq: A Cross-Sectional Cluster Sample Survey,” Lancet, October 11, 2006, pp. 1–8.
17. Although 92 percent of those asked by the research team provided death certificates, the study’s estimate is 12 times higher than those provided by UNAMI and the Los Angeles Times, based on a review of death certificates from Iraqi morgues and hospitals for the comparable time period. United Nations Assistance Mission for Iraq, Human Rights Office, “Human Rights Report,” May 1–June 30, 2006; and Louise Roug and Doug Smith, “War’s Iraqi Death Toll Tops 50,000,” Los Angeles Times, June 25, 2006. Moreover, the implied number of wounded and car bomb victims, heterogeneous distribution of victims, very high (and historically unusual) ratio of excess violent deaths to nonviolent deaths, mysterious postinvasion drop in child mortality, low estimate of prewar (sanctions era) mortality, and scale and trend in deaths attributed to the coalition all seem highly implausible given everything else that is known about the conºict, suggesting that the Lancet estimate is the by-product of some significant sampling bias or other problem with how the survey was conducted. See Beth Osborne Daponte, “The Civilian Death Toll in IraqWar Proves a Nebulous Statistic,” Taipei Times, January 28, 2007; Hamit Dardagan, John Sloboda, and Josh Dougherty, “Reality Checks: Some Responses to the Latest Lancet Estimates,” Iraq Body Count, October 16, 2006, http://iraqbodycount.org/press/pr14.php; Jim Giles, “Death Toll in Iraq: Survey Team Takes on Its Critics,” Nature, March 1, 2007, pp. 6–7; Madelyn Hsiao-Rei Hicks, “Mortality after the 2003 Invasion of Iraq: Were Valid and Ethical Field Methods Used in This Survey?” Households in Conflict Network, Research Design Note, No. 3 (Brighton, U.K.: Institute for Development Studies, University of Sussex, December 2006); Neil F. Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, and Gesine Reinert, “Bias in Epidemiological Studies of Conflict Mortality,” Households in Conflict Network, Research Design Note, No. 2 (Brighton, U.K.: Institute for Development Studies, University of Sussex, 2006); Fred Kaplan, “Number Crunching: Taking Another Look at the Lancet’s Iraq Study,” Slate, October 20, 2006, http://www.slate.com/id/2151926/; Jeffrey White and Loring White, “Death in Iraq: A Critical Examination of the Lancet Paper,” PolicyWatch, No. 1155 (Washington, D.C.: Washington Institute, October 18, 2006), http://www.washingtoninstitute.org/templateC05.php?CID?2523; and “Correspondence: Mortality in Iraq,” Lancet, January 13, 2007, pp. 101–104.
18. The UNDP-funded study surveyed 21,688 households and estimated that there were 24,000 Iraqi war-related fatalities (both military and civilian) through mid-2004. Central Organization for Statistics and Information and Fafo Institute of Applied International Studies, Iraq Living Conditions Survey, 2004, Vol. 2: Analytical Report (Baghdad: Ministry of Planning and Development Cooperation, 2005), p. 54. Although the UNDP estimate is 1.6 times higher than the IBC estimate during the comparable time period (see Hami Dardagan, John Sloboda, and Josh Dougherty, “Speculation Is No Substitute: A Defence of Iraq Body Count,” April 2006, http://reports.iraqbodycount.org/d/a_defence_of_ibc.pdf, Table 3), once the estimated 9,200 Iraqi military casualties from the major combat phase identified by the Project on Defense Alternatives are subtracted (Conetta, “The Wages of War”), the difference nearly disappears.
19. Quoted in Jefferson Morley, “Iraq’s Civilian Death Toll ‘Horrible’—Or Worse?” Washington Post, World Opinion Roundup, October 19, 2006, http://blog.washingtonpost.com/worldopinionroundup/2006/10/is_iraqs_civilian_death_toll_h.html.

Apologies for the poor copy-and-paste job. See the actual article for a nicer version.

If Kahl doesn't trust the Lancet numbers, then why do you?

Not in My Journal

2007-10-22T18:55:00.000-07:00

One of the more dramatic moments in the JSM session devoted to the Lancet surveys occurred during the question and answer session. (When AP reporter Paul Foy described the "courtroom-style questioning," I am pretty sure that he was, at least in part, referring to this incident.) This was actually the first question and it went, as luck would have it, to me. I asked something like:

As many audience members know, the Ethical Guidelines of the American Statistical Association require us to share data with other scientists for purposes of replication. Although you and your co-authors have shared some of your data with some other researchers, you have refused to share data with Michael Spagat and his co-authors. Why should we give credence to what you have said today if you refuse to adhere to the ethical guidelines of the ASA?

Comments:

1) No doubt my actual questions included more "uh's" and "you know's." It was certainly not phrased as nicely. But my meaning got across. Roberts and the other panelists addressed my point directly.

2) The Ethical Guidelines for Statistical Practice for the American Statistical Association require that statisticians:

Share data used in published studies to aid peer review and replication, but exercise due caution to protect proprietary and confidential data, including all data which might inappropriately reveal respondent identities.

Now, it is not clear that any of the Lancet authors are even members of the ASA, so these guidelines may not even apply to them. Roberts on occasion tries to claim that he is protecting interview/interviewee safety by not releasing the data. But he can't get away with that dodge for L2 since that data has been shared. There is just no reason to not share the data with critics like Spagat et al when you have already shared the data with critics like me.

3) The audience, of course, does not read the details of the ethical guidelines or follow the Lancet debate closely. But they certainly found it absurd that a scientist would share his data with some critics but not others. The reason that the Q&A went in the direction of "courtroom-style questioning" was the skepticism caused by Roberts et al behavior. I have yet to find, after a year of looking, another example where the data behind a published study was shared with some academic critics but not others.

What happened next? Roberts gave his usual shpiel, which mostly focuses in what a waste of time it is to have this discussion, to even talk about an already published study. Instead, critics should be getting new data, going to Iraq and so on. Best part was when he said that "If it were up to me, we would not have released the data to anyone." I was not the only member of the audience to be shocked by that admission. But it is consistent with what I can glean of the internal dynamics of the Lancet team. Roberts would really prefer to not share anything with anyone, but cooler heads (like Burnham, I think) occasionally prevail.

The other participants than chimed in. They all agreed that scientists should share their data. They all urged Roberts to do so. (All the subsequent questions from the floor that addresses this point seemed to agree with this position.) But the best comment came from Stephen Fienberg, the chair of the session and a professor at Carnegie Mellon.

That's just the wrong response. I, as an editor, would not publish a study for which the data was not shared.

Unfortunately, the editor of the Lancet feels differently. I wonder why.

AP Story on JSM Meeting

2007-10-12T18:50:00.000-07:00

Here is the AP story by Paul Foy on the JSM session about the Lancet surveys.

The courtroom-style questioning came in a packed ballroom at the world's largest gathering of statisticians.

On the hot seat: a globe-trotting researcher who says his team's surveys of Iraqi households projected nearly 655,000 had died in the war as of July 2006, a number still ten times higher than conventional estimates.

Leslie F. Roberts and others from John Hopkins University took accounts of births and deaths in some Iraqi households to estimate that the country's death rate had more than doubled after the 2003 invasion.

Number crunchers this week quibbled with Roberts's survey methods and blasted his refusal to release all his raw data for scrutiny -- or any data to his worst critics. Some discounted him as an advocate for world peace, although none could find a major flaw in his surveys or analysis.

"Most of the criticism I heard was carping," said Stephen Fienberg, a professor of statistics at Carnegie Mellon University in Pittsburgh. "I thought the surveys were pretty good for non-statisticians."

Roberts, an epidemiologist, said he's opening a new front in the study of public health hazards: war. He has conducted about 30 mortality studies since 1990 in conflicts around the globe, including the Congo, where he was similarly accused of exaggerating war-related deaths.

Roberts organized two surveys of mortality in Iraqi households that were published last October in Britain's premier medical journal, The Lancet. He acknowledged that the timing was meant to influence midterm U.S. elections.

"It puts you in a position where you are going to get attacked," said Fritz Scheuren, a senior fellow at the University of Chicago's National Opinion Research Center, who is trying to organize another Iraqi survey to see if he can match Roberts's results.

Scheuren, the American Statistical Association's former president, said he couldn't find anything wrong with The Lancet surveys.

He complained, however, that he wasn't able to get Roberts to reveal which of his Iraqi surveyors conducted which surveys, information that could reveal any bias in workers who compile consistently implausible results.

Roberts, now a lecturer at Columbia University, said he won't release the researchers' identities for fear of exposing them to death or retaliation. The new Iraqi government has strongly disputed the findings.

The estimate was not meant to be precise, but practically every health statistic in America comes from similar sampling techniques, Roberts and his team said.

Scheuren said it's OK for a scientist to be an advocate for peace but Roberts's work deserves more scrutiny because of it.

"It happens that war is political," Roberts responded Wednesday after his grilling at a convention of more than 5,000 statisticians. "So I think those criticisms are valid. I am swimming in a much more political world that these statisticians."

In The Lancet article, Roberts and other researchers produced eight pages of text, maps, charts and many figures, but the number was the show-stopper: 654,965 war-related deaths, from the 2003 invasion of Iraq to last summer.

The estimate covered everything from battlefield casualties to civilians dying for lack of routine medical care. President Bush last fall said the estimate wasn't credible.

"Do I believe 650,000? No," Fienberg said. "Do I believe a lot of people are being killed? Yes."

Roberts and the statisticians agree that estimates of as few as 66,000 war-related Iraqi civilian deaths are based on sporadic media reports or incomplete accounts from hospitals and morgues.

Other Iraqis end up in mass graves or are promptly and randomly buried by relatives under Muslim law, Roberts said.

He said he personally supervised a 2004 survey of mortality in Iraqi households, slipping into the country from Jordan on the floor of a sport utility vehicle with $20,000 hidden in his shoes. It was too dangerous for him to return in 2006, but he said he had a trained team of Iraqi doctors in white coats ready for that door-to-door survey.

In all, the researchers questioned 2,850 randomly selected households, and Roberts said both surveys confirmed a soaring death rate after the invasion. Their short questionnaire sought to find out how many family members were born or had died since Jan. 1, 2002, establishing a baseline for a post-invasion death rate.

"Verifying these numbers wouldn't be too difficult, so why isn't it being done?" asked Jana Asher, a graduate student at Carnegie Mellon University, who dissected Roberts's surveys and found little fault with them.

Has Roberts denied the claim that the timing of the report was "meant to influence midterm U.S. elections."? Not that I have seen. See some discussion on Deltoid.

Bias Presentation

2007-10-03T06:50:00.000-07:00

This presentation (pdf) by Michael Spagat (Royal Holloway College, University of London) is good stuff, giving a more global view of the Main Street Bias issue.

Truther

2007-09-05T17:28:00.000-07:00

Les Roberts thinks that critics who question his studies are the same as those who deny that 3,000 people died on 9/11. Really.

And another thought is that -- quite unrelated -- if someone said in the 9/11 attacks, “I think only 200 or 300 people really died,” we would be really, really upset. And I think in the long view, the danger of discarding this study, if it’s correct, is that, at a moment when we as a society should be showing contrition, our leaders have essentially expressed indifference to an extraordinary level of suffering. And that’s just the wrong message in terms of either our long-term security or peace in the Middle East.

Is this better or worse than his other analogy of Lancet critics as Holocaust deniers?

Details on Falluja

2007-09-05T17:15:00.000-07:00

The L1 authors consistently maintain that Falluja was not an outlier in the sense that it should be discarded. Instead, it was representative of many parts of Iraq. This is from a letter they wrote to the Independent.

Our study found that violence was widespread and up 58-fold after the invasion; that from 32 of the neighbourhoods we visited we estimated 98,000 excess deaths; and that from the sample of the most war-torn communities represented by 30 households in Fallujah more people had probably died than in all of the rest of the country combined.

Fallujah is the only insight into those cities experiencing extreme violence (ie Ramadi, Tallafar, Fallujah, Najaf); all the others were passed over in our sample by random chance. If the Fallujah duster is representative, there were about 200,000 excess deaths above the 98,000.

Perhaps Fallujah is so unique that it represents only Fallujah, implying that it represents only 50-70,000 additional deaths. There is a tiny chance that the neighborhood we visited in Fallujah was worse than the average experience, and only corresponds with a couple of tens of thousands of deaths. We also explain why, given study limitations, our estimate is likely to be low. Therefore, when taken in total, we concluded that the civilian death toll was at least around 100,000 and probably higher, not between 8,000 and l94,000 as Mr. Straw states.

I am still a little hazy on where the 50-70,000 number comes from and what it implies for the population of Falluja.

War Crime

2007-09-05T17:04:00.000-07:00

The antics of Lancet editor Richard Horton are endless fun. Here he describes the invasion of Iraq as a "monstrous war crime."

This Labour government, which includes Gordon Brown as much as it does Tony Blair, is party to a war crime of monstrous proportions. Yet our political consensus prevents any judicial or civil society response. Britain is paralysed by its own indifference.

At a time when we are celebrating our enlightened abolition of slavery 200 years ago, we are continuing to commit one of the worst international abuses of human rights of the past half-century. It is inexplicable how we allowed this to happen. It is inexplicable why we are not demanding this government's mass resignation.

Two hundred years from now, the Iraq war will be mourned as the moment when Britain violated its delicate democratic constitution and joined the ranks of nations that use extreme pre-emptive killing as a tactic of foreign policy. Some anniversary that will be.

This is consistent with what Roberts occasionally says (can't find a link just now) about whether or not he would ever release detailed data from L1. He would, if the International Criminal Court needed the evidence for war crime prosecutions, presumably against civilians leaders like Bush and Blair.

Amer and Scheuren

2007-09-04T10:41:00.000-07:00

One of the more annoying tendencies of Lancet defenders is that they refuse to recognize the breadth of academic criticism against the Lancet papers. Over time, I hope to gather some of that criticism together. Here is one example.

Several ASA statisticians have gone to Iraq since 2003—Jana Asher and Mary Gray, to mention two. An article by Asher about her experience appeared in the December 2006 issue of Amstat News; Gray’s work in Iraqi Kurdistan can be found in the April 2007 issue. So, the involvement of statisticians in Iraq is not unusual.

The recent controversy over the estimated Iraq war deaths that appeared in the October 11, 2006, issue of The Lancet has drawn considerable statistical attention to the situation in Iraq, as the results in the article did not appear credible. As mentioned by Gray, a follow-up to The Lancet piece occurred at a February session of the Washington Statistical Society (WSS), organized by Wendy Rotz, with talks by Asher and David Marker. The jury is still out on The Lancet results. Statisticians must assess the process used, not the outcome. But there is a lot wrong with the process, leading to the conjecture that the results are most probably wrong.

Indeed. Do these authors have standing for believing that the Lancet results are "probably wrong?" Judge for yourself.

Safaa Amer holds her PhD in statistics from Oregon State University. ... Fritz Scheuren is a vice president for statistics, who works with Amer at NORC at the University of Chicago. He put together the What Is a Survey? booklet some years ago. In his 2005 ASA presidential speech (March 2007, JASA), he advocated a pro bono role for statisticians.

I look forward to reading Daniel Davies thoughts on why Amer and Scheuren are just another two members of the ever-expanding list of anti-Lance cargo culters. Time to get those boots on!

Missing Data

2007-09-03T04:46:00.001-07:00

I have been having fun on Deltoid recently (here and here). One annoying issue is whether or not the method by which SG replicated the L1 CMR estimates is obviously correct and/or the only reasonable approach. I don't think it is. (Given that it matches the formula in the spreadsheet distributed by Les Roberts, I am happy to assume that it matches the approach used in the paper.) Although I am a layman when it comes to demography, it seems obvious that any statistican would question whether just adding up all the deaths and dividing by person-months is the best way to estimate the crude mortality rate for Iraq. Such a proceedure ignores the fact that non-response varies across clusters.

Consider a simple example in which you have two clusters with 50 attempted interviews in each using a one year look-back period. In cluster A, you interview all 50 households. There are 10 people in each house and a total of 20 deaths. The CMR is cluster A is then 4% (20 deaths divided by 500 person-years). But, in cluster B, only 10 households agree to be interviewed. The other 40 refuse. There are also 10 people in each of the 10 households. There is one death, giving a CMR of 1% for cluster B.

Question: What is the best estimate of the CMR for the whole country given this data from two clusters? Answer: It depends! Certainly, the formula cited as obvious by SG, Robert Chung and others at Deltoid is not clearly the best answer (although I agree that it is a reasonable one). To see why, note that this formula just adds up the total deaths (20 + 1 = 21) and divides by the total person years in the population that agreed to be interviewed (500 + 100 = 600) to give a CMR of 3.5% (21/600). In other words, the overall estimate is very close to the estimate for cluster A because the actual sample size from cluster A is so much larger than that for cluster B even though the sampling plan called for equal sample sizes.

Ignoring non-response causes you to weigh clusters with higher-response rates more heavily even though, a priori, there is no particularly good reason to do so.

A different approach would be to treat the two cluster as independent and just average the resulting CMRs from each cluster. Such an approach would give 2.5%. Now, the difference between 2.5% and 3.5% isn't that big. Indeed, the differences in the two approaches for the Lancet data are even smaller. But the idea that one is clearly right and the other wrong is just stupid. It all depends on what you think the cause of the non-response is.

(A further complication is that household sizes may differ across clusters. In that case, it makes sense to either sum populations or to weight the clusters by the total population in each cluster. But the main point I am making here has to do with non-response.)

Love Postings

2007-07-25T18:12:00.000-07:00

Another collection of Lancet critiques.

More Quotes From Burnham

2007-06-06T14:35:00.000-07:00

Previously, I commented on aspects of Gilbert Burnham's talk at MIT. Now, I want to record some key quotes.

Around 18:00 discussing the violent deaths which occurred before the war started. Note that this discussion concerns Lancet I.

Now, one of the big differences was the number of deaths due to violence. Now, our definition of violence wasn't car crashes or roof-falling-in-on-the-house, but these were intentional violence. So, they would be things like gun shots, missiles and so forth. And there were a few deaths that were before the invasion that were violent, and many of these were probably --- not "many," there were only a couple --- but these were associated probably with incidents around the No Fly zones, the process of radar units locking on to air planes and then firing back on the artillery and missile sites and so forth.

I previously covered this topic in the context of Lancet II. The topic of Falluja and confidence intervals starts around 18:30.

Around 20:30 we have this.

I might add that, in this second survey, I intentionally asked the team to oversample in Falluja, to take three samples, if they chose Falluja in the random selection, which indeed happened. And so we did three samples, three clusters instead of the one which the population would have merited. And then we randomly selected one of the three, which turned out to be the one that had the lowest mortality in the three clusters. But the experience in the first one was really true to what we found in the subsequent one, that there was very, very widespread destruction in Falluja, nearly every household had lost somebody in that survey.

Around 21:30 (starting discussion of Lancet II):

In 2006 (in working with John [Tirman of MIT]) we went back to look at things again from May to July of 2006. And, as John knows, we had great intention to get this out as early as we could before the elections, so we would not be again accused of trying to do something with the elections. But, one thing and another, movement of money, ethical approvals and so forth, we didn't get it done quite as soon as we would have liked.

It is still somewhat a mystery why there were no clusters in Muthanna and Dahuk. The paper refers to "miscommunication." Burnham says (around 22:30) in referring to a slide which shows the distribution of clusters by governorate, "the last two, which probably should have gotten included, because of some mix ups in communication, they did not get sampled."

My guess is that the US authors wanted at least one cluster in each of the 18 governorates. Their technical document (one hopes!) specified the procedure by which this would occur. (Presumably, this involved assigning one cluster to each governorate and then allocating the remaining 32 clusters in proportion to governorate population.) Unfortunately, the US authors did not (?) actually do any of the sampling. Instead, the Iraqi interviewers performed the random allocation but did not follow the stated procedure.

Yet this leaves the mystery of which two clusters were discarded as a result of the mistake. That is, the US authors wanted a cluster in each of Muthanna and Dahuk. Yet, they did not get them. But how does anyone know which of the 50 clusters which were actually sampled were the "mistakes", the ones that were used in place of those two? How could anyone know which clusters to discard? I would really like to see the data from the two discarded clusters (as well as the one boundary mistake from Wassit).

I love this description (around 24:00) of the process.

They [the interviewers] went out house to house in their white coats so that they couldn't be mistaken for being somebody else. They, first off, rounded up the children to explain what this survey was about, sent out the children to the households to explain to the neighbors what was going on and so forth, to try and reduce the risks that were involved.

What possible bias could this cause?!

Around 25:00 we get:

So we had pairs [of interviewers], male and female. Generally speaking, women will speak to women. So, if the head of the household who is home, or the person who is home, was a woman, the male interviewer would often go to the next house and get started there. So, with these two pair of interviewers for each of the clusters of 40 households, we were able to finish a cluster in a day. One of the things that we did not want to do was to come back the next day.

On forgetting to ask (at 26:00).

So this time our intent was to ask every household where a death was reported for a death certificate. Now, if you have done survey work you know you don't always get all the answers you want. And, in this case, in 13% of cases the interviewers forgot to ask for the death certificate.

There is more good stuff. I hope to post more in a few weeks.

UPDATE: Here are more highlights.

Around 26:30 there is a description of the sampling methodology that is not totally consistent with other descriptions. There are also handy slides providing a visual guide to the procedure.

Around 28:30, Burnham notes that 11 of the clusters were in the same geographic area (town or village) as in the first survey but never in the exact same location. It would be interesting to see how that subset matches up between 2004 and 2006.

Around 30:00 shows an interesting histogram of the rate of violent death (post-invasion?) in each cluster. Speculates that Falluja and Dyalla (sp?) are the two biggest outliers but notes that the distribution looks pretty normal. In other words, violent deaths are not that clumped together. Also shows an interesting plot of death by age. Looks reasonable for women (high as babies then lower then high for elderly). The patterns for males, however, are very different. Highest mortality is for fighting age men. This is good evidence that there is a war going on, as we know.

[Side comment: But doesn't this dispute the central claim that lots of civilians are getting killed? One storyline is that US forces are dropping lots of bombs and lots of car bombs are going off in market places. In such a world, one would expect lots of females deaths as well as male deaths. Why would men be that much more likely to die in their house from an air strike or in the market from a car bomb than women? But, if most of the male deaths are "combatants" (take a look at the deaths from gunfire to cross check), then this is precisely the pattern we would expect to see. (Of course, many/most of the male victims of death squads and whatnot are not combatants as we would use the term.)

Shows the mortality by governorate and mentions that some of these estimates are imprecise because there is only one or a small number of clusters in the governorate. I ought to look into this more closely. Is there any statistical significance to the three groupings that they use to classify governorates? I have my doubts. This isn't really in point but it might make for a pretty graphic.

Around 39:00 talks about "deaths among civilians." False! They never let up with that canard. They measure deaths. They have no idea who the dead individuals were (civilians, insurgents, Iraqi army, et cetera).

Some discussion of survivor bias. Notes that there was one neighborhood in Falluja (from 2006 study?) in which all the members of seven families had been killed. Reports that it was common to have one or two such families in other clusters. I do not think that any of this is mentioned in the paper or distributed as part of the data.

Responds to MSB critique at 43:00.

A group of physicists in Oxford complained that our sampling method favored the principal streets where they interpreted that most of the killings may have occurred. In fact, we went out of our way to try to include all the streets in the sampling frame. And, of course, the other thing we found is most killings occurred away from home anyway. And so that probably didn't add much to bias, if it had even existed.

I love the discussion about sampling Falluja at around 54:00. But the important quote is above. Guy paying for survey says, "If you happen to randomly choose Falluja (and I hope you do!), let's do three samples." Guy getting paid to do the survey says, "Guess what! We randomly selected Falluja." Guy with the money. "That makes me happy!" Burnham also mentions the problems of estimating population in Falluja. Commonly accepted number for before the war is 500,000. After US military action, the number was probably at 200,000.

Around 57:00 gets called out by a questioner about inappropriate use of the term "civilians" in describing the victims. Fesses up that he should use the term "Iraqis."

Around 1 hour, tells the story of a former student at US AID who reports that, when the survey came out, some unnamed office at the White House sent over a copy for them to try and discredit. They tried to but couldn't come up with anything (the project ruined their Friday afternoon) and so reported back that the study could not be discredited.

Detailed discussion of the randomization procedure around 1:07:00.

They [the Iraqi interviewers] did it on pieces of paper, they'd write down street numbers [or street names?] of pieces of paper. And then they would randomly pick out one of those from the hat, as it were. The things that we were doing in the 1960s, they did that. We got criticism because at the end of the process they destroyed all the little pieces of paper which, you know, I think that most of us would have done anyway.

...

Once they selected the street, then they numbered the houses, from that street, from 1 to wherever the end of the street was. And then they randomly, using serial numbers on money, they randomly selected a start number, to start from that house, and then from there they went to the nearest front door, nearest front door, until they had a total of 40 houses.

...

If the neighborhood was too dangerous, then you could either come back or you could select another one. Now, where this was a real problem was in Basrah. And they had to go to Basrah three or four times before it was finally safe enough to go. But they decided that they were going to go to the neighborhood that was selected even if it was a bit dangerous at the moment, they'd come back and do that one; they wanted to stick with the one that was selected.

A discussion of the Wassit neighborhood problem follows. Burnham also mentions "rules" about the process for selecting a different neighborhood if the chosen one was too dangerous. They were supposed to look for a nearby neighborhood which was "similar" to the one selected.

It sure would be nice to have a sense of the how often this happened and to see a copy of the technical document which guided the survey teams.

Old Roberts Interview

2007-05-16T04:25:00.000-07:00

This interview with Les Robert is two years old (and only covers Lancet I) but still interesting. Highlights:

Americans are so hated that I couldn’t go around talking to people. We would pick a random point in each “cluster” — each village or town we surveyed.

I would show our Iraqi team how to pick a random point in a town, how to use a Global Positioning System to draw a map of the town and drive to the right point, how to find the few houses closer to the point.

Always in the first few houses there’s some that are a bit quirky. There might be a cousin visiting and you have to decide whether you include him in the sample. We worked through the first few clusters together to go through those issues.

I’d walk around on the street with our interview team. Then I’d go get in a car and hide, and the Iraqis would visit the houses by themselves. I was almost never out in public.

My driver had three brothers so he had access to four different cars and he would pick me up in a different one each morning. We’d leave at different times and use different routes.

I only went out with the interviewers for the first eight days. On the eighth day the police picked up our interviewers while I was in the car watching and that was a pretty bad experience.

After that we were convinced that interviewers knew what they were doing, and they didn’t want me there. For about 15 days I just stayed in a hotel room and didn’t go out.

It would be interesting to compare the data for Lancet I for the days that Roberts accompanied the team and the days he did not. Other reports (can't find the link) note that Roberts did not go to Fallujah. Note that the paper reports that the survey actually occurred over a 13 day period: September 8 through September 20.

Two thirds of all violent deaths in your survey were in one city — Fallujah. Why were the deaths so concentrated in this city?

The city was shelled extensively in the weeks before we were interviewing. We went and attempted to interview 30 households. Almost half of the houses we went to were empty.

We skipped over them and went to other houses. We think that our findings, if anything, underestimated the number of deaths because of the number of empty and destroyed houses. Some of the families probably fled, but many are probably dead.

Of those families sticking around in Fallujah, a quarter lost a family member in the few months leading up to the interview. Who knows how many have died since the assault on the city in November.

I get very angry about the coverage of Fallujah. I heard a show last week on public radio in the US. They said that it is believed that half the 200,000 people who used to live in the city had returned. Well, the ministry of health told us the population used to be 310,000.

The US press has been manipulated. Things don’t sound as bad if you say that 50 percent rather than 30 percent of the population are back.

During the invasion of Fallujah, Pentagon spokespeople said again and again that they believed 3,000 to 5,000 mainly foreign combatants were left in the city and that most of the civilian population had left.

Well, they went in, they killed a lot of people — estimates range from 600 to 2,100 — and they captured 1,600 prisoners.

Only 30 of the prisoners were identified as foreign combatants — only 2 percent of those captured. In my country no one was held to account for what was either a lie or an absurd intelligence failure.

I know terrible things happened in Fallujah, but no one has been given a chance to get good information about what is going on.

What was the reaction to your survey when it was published?

The coverage in the press varied enormously. It was very different in the US and in Europe. I had more interviews with European newspapers and radio shows than I did with American ones. The interviews I had in America were with the left wing, marginal media, which doesn’t have a very wide audience.

I have been trying to track down the population of pre-war Fallujah. Roberts reports 310,000, which seems a reasonable number. I think that an understanding of the data collection process for Fallujah would be enlightening. If you, like John Pedersen, believe that the Lancet estimates are too high, then the place to start understanding how that might have occurred is with Fallujah.

Roberts Talk at Brown

2007-05-14T09:12:00.000-07:00

Here is a April lecture by Les Roberts given at Brown.

Highlights:

1) He mentions that, in Lancet I, the measured mortality rate in Falluja was 20%.

2) He highlights the accuracy of the high estimate of the proportion of deaths from air strikes given in Lancet I. He notes that this is different from other conflicts in which it is hard for someone to know which side is responsible for a given death. Who knows which side fired the artillery shell? But, in Iraq, only one side has air power so Roberts can be sure that these deaths are due to the coalition.

3) Repeats the claim that "most" of the authors were in favor of the war (although he notes that they did not discuss it at the time).

4) Shows a slide (which I really can't make out) that seems similar to one from Burnham's MIT talk. It seems to be a histogram of cluster death rates. Most of the observations form a nice normal distribution but there are at least two big outliers and a meaningful skew to the right. I am having trouble seeing this in the raw data. The simple deaths per cluster have a skew but no big outliers. Is there a negative correlation between deaths and family size? I hope to be able to reproduce this graphic by the end of the week. It could also be that the data here includes three clusters from Falluja rather than just the one that I think was distributed to me.

5) Roberts asserts that IBC's funding dried up after the first report and that their criticisms of his work are motivated by funding issues. He also asserts that there is a link (some are colleagues at Oxford) between the IBC folks and Spagat et al.

6) Roberts calls the Main Street Bias critique "absurd."

7) Asserts that most of the deaths that they recorded (might just be referring to Lancet II at this point) are not listed by IBC. How can he determine that?

8) Claims that Bagdad has lower mortality than the country as a whole.

9) Amazingly partisan at toward the end of the speech. See the video for details, but it is safe to say that Roberts is no fan of Republicans in general or President Bush in particular.

10) Compares estimates which are much lower than the Lancet data to Holocaust denial. Nice!

Probably

2007-05-14T08:33:00.000-07:00

As penance for my sins, I try to watch every video presentation by the Lancet authors. Here is a recent one by Les Roberts.

Best part? In passing, he mentions that the documents used by Dan Rather in the famous Bush/AWOL story were "probably forged." Probably?!? The world is probably round, too.

Roberts Comments on ILCS and Politics

2007-04-20T13:41:00.000-07:00

Les Roberts dishes some dirt against ILCS in this interview, starting at around 2:50. Calls Jon Perdersen a "great researcher out of Norway." One question on death with a couple of follow-on questions if a death had occurred. Survey was conducted by "Iraq governmental employees of Saddam's era." (Roberts employees that such people do not do high quality work. But were all his interviewers doctors (i.e., working for the government before the war) similar situated. Notes Pedersen's decision to re-sample on that question after the death rate came in too low. Roberts implies that, even with this correction, the estimated death rate is too low from UNDP/ILCS.

I haven't followed this dispute closely enough to comment. Roberts certainly believes that the ILCS results are consistent with his. Best part was the dialog on political views at around 5:50. Interview mentions an assumption about Roberts' politics, a concern that they did the study because they are opposed to the US/allied presence in Iraq.

Roberts: Well, that's just not true.

Interviewer: (incredulous) You're not opposed to the US presence in Iraq.

Roberts: Actually, the majority of the first authors I know were in favor of the invasion of Iraq.

Really? That would be deeply surprising to me. Burnham, at his MIT talk certainly gives the impression that he was opposed to the invasion (although I don't recall him addressing that point). Everything that I have read about Roberts suggests that he was anti-war from the start (although I can't find a good link right now). If those two were anti-war, then all three of the remaining authors (Riyadh Lafta, Richard Garfield, Jamal Khudhairi) would have to be pro-invasion.

Anyway, Roberts then goes on a riff about how public researchers are always "against" the thing that they study. No one who studies measles is pro-measles. Roberts is against death. "Full stop." Roberts says "A lot of people have died here and that is completely unacceptable."

Roberts does a great mocking of a US southern accent at the very end. Not nice!

Very, Very Wide

2007-04-20T13:27:00.000-07:00

Very thorough interview of Les Roberts on a (British?) news show. Highly recommended. Key part for my concern about confidence intervals comes in Roberts discussion of the 2004 study at around 3:50.

There was one place, the city of Falluja which had just been devasted by shelling and bombing. It was so far out of whack with all the other [clusters] that it just made our confidence intervals very, very wide.

Unfortunately, the interviewer quickly moves on to other issues. Compare that statement with Burnham's.

Now this is what the confidence intervals would look like. There is a 10% probability that it was less than 44,000 and only a 2.5% chance that it was less than 8,000. If we put Falluja into it, the top end of the confidence interval would be infinity. It really skewed things so badly that we decided that we should just leave it out and be conservative.

Seems fairly obvious to me that, if Falluja is included in Lancet I, the null hypothesis of no increase in mortality can not be rejected. But that's an empirical question. If the authors would release the data for Lancet I, we can figure this out.

Soldz Article

2007-04-17T13:08:00.000-07:00

One of the most fun parts of participating in this debate is getting to meet some of the people involved. I had an enjoyable lunch with Stephen Soldz last week. His blog has all sorts of useful links, and his article on Lancet I is a fine example of a fair and balanced analysis.

If your a fan of the controversy surrounding the Lancet articles (either pro or con) and you find yourself near Cambridge, MA, please let me know. Lunch is on me!

Data!

2007-04-15T12:29:00.000-07:00

The authors have made a version of the data for Lancet II available to me. Thanks! To be honest, I had worried that they wouldn't do so. I was wrong to worry. I appreciate the authors sharing this data with me (and others). I also thank them (in advance!) for answering the questions that I and others will have.

A central value of any good scientific community is transparency of methodology. If you want to claim that X is true, you need to explain in detail why X is true and demonstrate how other scientists, especially those who disagree with you, can replicate your results. The Lancet authors have taken the first step along this road. Good for them.

Lafta Visit

2007-04-14T23:50:00.000-07:00

Via Tim Lambert, we have more details of Riyadh Lafta's upcoming visit to Canada. (Lafta was the key Iraqi doctor for both Lancet I and II.) I think that he organized and supervised all the survey work.

A highly regarded Iraqi epidemiologist who wants to tell Americans about an alarming rise in cancer levels among Iraqi children will come to Canada instead because he couldn't get a visa to the United States.

Does Lafta have a degree in epidemiology or statistics or public health? Just curious. Lack of a degree does not imply incompetence, but if the only two surveys he has conducted have been for these articles, I am not sure that I would describe him as an "epidemiologist."

Once in Canada, Dr. Lafta will present estimates that paint a damning portrait of the war's ravages on children: that birth defects are on the rise since the war began, and that the number of children dying from cancers such as leukemia has risen tenfold.

Huh? There is no doubt that car bombs are bad for children, but what possible war-related mechanism could there be for a 10-fold increase in leukemia? Also, the Iraqi government can't keep track of death rates but it has accurate statistics on childhood leukemia? That seems, uh, implausible.

Dr. Lafta was born in Baghdad in 1960, was trained as a physician at Baghdad University College and then worked for 14 years for the Ministry of Health under Saddam Hussein. He became the head of the communicable disease department and then the primary-care department of Diyala province in northern Iraq.

It would be nice to have more details on this time-line. Did Lafta's 14 years at MoH include his time at Diyala? Was Lafta at MoH until the invasion in 2003? One way to make the dates work would be for Rafta to finish his training in 1989 (at age 29) and then spend then next 14 years until 2003 at MoH. 29 would certainly be a standard age for finishing training in the US (but no doubt Iraqi training schedules are different).

In order to have a sense of how qualified Rafta was to organize these surveys, we need to have a better sense of his background. If he did a lot of such surveys during his 14 years at MoH (either nationwide or just in Diyala), then he would be an excellent choice. If did no prior surveys, then he still might have done a great job for the Lancet team, but we need more details. How did the Lancet authors and Rafta meet each other?

Also, I guess that I am safe in assuming that Lafta is Sunni. Isn't it also the case that most/all higher ranking officials in the pre-war government (whether at MoH or elsewhere) were members of the Baath party? Now, obviously, just because one is a Sunni and/or a former member of the Baath party does not that one's professional work is suspect. But, in any survey situation, it is a concern if the interviewers are too "different" from the interviewees. (I have already discussed the issue of (the lack of?) Kurdish speakers among the interviewers.)

Consider the problem that Shia/Kurdish households might, not so much lie in order to deceive when confronted with Sunni interviewers as give those interviewers that answers that they seemed to "want." If the interviewers seemed to want to find some post-war deaths, then give them so post war deaths.

The analogous situation in the US in 1965 might be white interviewers asking black households in, say, Mississippi about police brutality. A black interviewee might well decide that the better part of valor was giving the white interviewer the answer that the interviewee thinks the interviewer wants. Tell the nice white lady that there are no problems with police brutality (even though there are). In this scenario, there is no subterfuge or fraud on the part of the interviewer. She is honestly trying to find the correct answer, honestly writes down what the interviewee reports.

I do not think that this is a major concern but it would be nice to know if all the interviewers were Sunni. My guess is that they are. (How many Shia became English-fluent doctors in Saddam's Iraq?) Again, this does not mean that they are lying, but it does suggest that we should look for patterns in the responses that the y recorded.

Infinity

2007-04-14T23:27:00.000-07:00

As noted previously, Gilbert Burnham's presentation at MIT featured information about the results for Lancet I. Burnham is describing their estimate of excess deaths due to the war. Their estimate was 98,000 if the data from Falluja is ignored. The estimated death rate in Falluja was much higher than elsewhere so removing Falluja was described as "conservative," by both the authors and other commentators. Recall the findings from the paper (pdf) itself.

The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion. We estimate that 98 000 more deaths than expected (8000–194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.

Any empirical researcher is vaguely suspicious of results which just barely reject the null hypothesis. That 8,000 figure is awfully close to zero. Would other reasonable parameterizations generate the same result? Perhaps. But, given that this excluded Falluja, there is nothing to worry about, right? The war must have increased the mortality rate.

But note how Burnham described things (quote starts around 19:30) during the talk.

We got a huge amount of criticisms for these confidence intervals, and I'll come to this confidence interval in just a bit. But we had a confidence interval at the low end of 8,000 and at the high end of 194,000.

...

Now this is what the confidence intervals would look like. There is a 10% probability that it was less than 44,000 and only a 2.5% chance that it was less than 8,000. If we put Falluja into it, the top end of the confidence interval would be infinity. It really skewed things so badly that we decided that we should just leave it out and be conservative.

Huh? He can't possibly mean that the top end would be infinity. I would think (albeit, not as an expert on confidence intervals in cluster sampling) that a top end of infinity would imply a bottom end of the confidence interval at infinity as well. Wouldn't it have to? Note that excess deaths is a real valued variable. If it were bounded by zero, then, of course, you might have a confidence interval which was bounded below but not above.

For real valued variables, I do not think that I have ever seen an applied situation with actual data in which the confidence interval was infinity above but not below. Has anyone else? Almost every confidence interval of a real valued variable is symmetric. If it is infinite at the top, then it is infinite at the bottom.

Also, the fact that it is infinite at the top suggests that this was not a bootstrap confidence interval. I think that it is very hard (impossible?) to get infinity for bootstrap confidence intervals. A bootstrap always gives you back something. That would suggest that they were using an analytic calculation of some kind, one that failed to converge.

And that wouldn't be surprising. If the Falluja data is enough of an outlier, then the software has trouble calculating a confidence interval. It spits back infinity of NA or NaN or whatever.

Assume for now that the confidence interval was infinity on both sides. This would mean that only by ignoring the Falluja data were the authors able to reject the null hypothesis of zero excess deaths due to the war. It would be highly misleading to report a rejection of the null hypothesis which required throwing out some of the data without at least telling your readers how fragile this result is.

Perhaps Burnham just meant "really big" instead of "infinity." But, if so, then the same large range was probably present at the lower limits as well. That is, even if the upper confidence interval was just some huge number and not literally infinity, that would suggest that the lower bound was similarly large and, probably, covered zero. (This might depend on how much higher the mean estimate was, of course.) In either case, if getting a statistically significant result requires throwing out some of the data --- even though you have no reason for thinking that data is less accurate than other collected data --- you must report this fact to your readers.

Yet all of this discussion is in direct contradiction to the paper. The authors claim to have calculated a confidence interval (for the increase in risk) include the Falluja data.

The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period.

A 4.2 upper confidence level for the relative risk is neither infinite nor, I think, particularly high. Might not a raging war make things 4 times more dangerous in a country? Why would Burnham claim that the upper confidence limit is infinite while the paper reports a reasonable number? Is there some reason why one might have a reasonable upper limit for the relative risk but not for the total numbers of deaths? Not that I can think of.

No doubt I am just missing something obvious. Clarifications would be welcome.

ESOMAR Podcast

2007-04-10T06:02:00.000-07:00

Great podcast from October 2006 with many of the key players involved in Iraqi polls. Matthew Warshaw of D3 Systems is one of the key people in the recent ABC poll as well as the World Public Opinion.com polls. Most of his discussion focuses on Afghanistan. Fascinating stuff on the knitty-gritty details.

The section on Iraq features Johnny Heald of Opinion Research Business and Munqeth Daghir of MEC Communications. Daghir's story is interesting and moving. He had no experience in polling but read a book (by "Dr. Gallup") just after the invasion and decided to get involved in polling, creating a company from scratch. Heald has been polling with Daghir for two years (since 2004, I guess). Daghir's first poll was in Baghdad, three weeks after the invasion. Daghir's had interesting comments on the "culture" of fear surrounding interviews, how people don't want to reveal their opinions, are afraid of the militias and so on.

Daghir discusses how different polling for him is now (2006) compared to when he started (2003). At the start, he just had one book and no experience. Now, he has gotten lots of help from ESOMAR (which I think is an international organization of market researchers) and research experts from around the world, people like Heald. Now, his polls are "according to international standards", before, not so much. The 19th minute is the key section. There has been a "great change in the methodology." Now, Daghir's polls are "totally different." Daghir discussed having a lot of trouble selling the initial polls.

Pedersen Quote

2007-04-09T14:32:00.000-07:00

A reader writes in with this sourcing for the Pedersen quote.

No one disputes that Iraq has grown much more deadly. The question is how much.

A 2004 study by the same authors of the Lancet article estimated 98,000 violent deaths in the first 18 months after the invasion, a figure four times higher than the findings of a much larger survey done at approximately the same time by Norweigan researchers working for the United Nations. That study, the Iraqi Living Conditions Survey, estimated 23,743 civilian deaths in the first 13 months of the conflict.

In a telephone interview, Jon Pedersen, research director for the 2004 study, said several factors probably account for researchers' different findings.

One key issue is how researchers extrapolate from the deaths identified in their field research to a death toll for the whole country. Pedersen noted that the Lancet study is based on a pre-invasion mortality rate of 5.5 deaths per thousand people. The U.N., he said, used the figure of 9 deaths per thousand. Extrapolating from the lower pre-invasion mortality rate would yield a greater increase in post-invasion deaths, he noted. If the higher pre-invasion mortality rate is more accurate, then the deaths attributable to the war would be lower.

Another difficulty, he said, is doing accurate research under dangerous conditions. Pedersen wondered how tightly the study's authors could oversee the interviews as they were conducted throughout Iraq.

The JHU study, he noted, asked Iraqis only about mortality. The U.N. study asked Iraqis about many aspects of their living conditions. Pedersen said his study probably underestimated deaths caused by the war because the interviews did not focus on the issue, while the Lancet article probably overstated them because no other subject was discussed.

Pedersen said he thinks the Lancet numbers are "high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much."

"Regardless of the numbers that are possible," he added, "we are seeing a situation that is pretty horrible."

Agreed. But, if Pedersen is correct, how did Lancet II come up with a number 6 times greater than the truth? Was there a problem in their statistical software? Did they make a mistake in some arcane cluster sampling calculation? Unlikely. The most likely source of error, as I have said for many months, is the raw data. Are all the supporters of Lancet II 100% convinced that Pedersen is wrong?

It seems to me that the truth is most likely to lie somewhere between Lancet II's estimate and Pedersen's.

Data Now Available

2007-04-09T10:33:00.000-07:00

The data for Lancet II (but not, it appears, Lancet I) is now available.

Six months have passed since the publication of the study and we feel the time is now right to make the data set available to academic and other scientific groups whom we judge have the technical capacity to objectively analyze the data.

...

These data will be released on request to recognized academic institutions or scientific groups with biostatistical and epidemiological analytic capacity.

1. The data will be provided to organizations or groups without publicly stated views that would cause doubt about their objectivity in analyzing the data.

For someone like me who wants access to the data, the concern, of course, is that there is wiggle room in the definitions for "technical capacity" and "biostatistical and epidemiological analytic capacity." By any reasonable standard, IQSS and KCM (my firm) would qualify in terms of statistical sophistication, but who knows? I have submitted requests using each organization on the theory that if IQSS does not meet their criteria, then KCM will (or vice versa). My hope is to get this resolved in the next day or two so that I can revise my abstract accordingly. The deadline for doing so is April 11.

Data Availability

2007-04-05T07:27:00.000-07:00

I sent the following letter to most of the Lancet authors.

Hi,

My name is David Kane and I am a fellow at IQSS at Harvard. I will be presenting a paper at JSM in Salt Lake City.

Questions:

1) One of my main goals in this paper is to avoid the mistakes of Hicks and Spagat/Johnson/Gourley by ensuring that every description of your work that I publish is *accurate*. Is there someone that I might direct questions to in order to achieve this? I am happy to e-mail all of you or just a subset or just one contact person. Whichever you prefer.

2) I could not find e-mail addresses for all the authors of both papers. Would you mind passing this message on to Riyadh Lafta and Jamal Khudhairi?

3) Can you give me an update on data availability? I know that there has been talk about making the data available to researchers like me but I have been unable to confirm this. Do you still expect to make that data available? If so, when? As you can see, my current abstract claims that the data is not available and I won't be able to edit that abstract after April 11. If you could make the data available to me (with all identifying information removed, of course) before April 11, then I could update my abstract appropriately.

4) It is fair to say that I am a critic of the estimates in your papers (I think that Jon Pedersen's are probably closer to the mark), but I consider myself a scientist first. My main goal here is the truth.

Thanks for your time,

Dave Kane

PS. I have cc'd several people that I have communicated with on this topic. Hope you don't mind.

The dig against Hicks/Spagat et. al. might have been a little harsh. Is it there fault that the written description of the Lancet methodology is not easy to parse? But I thought that politeness was the better part of valor in this case. My main goal is to get access to the data and to get at least one of the authors to answer my questions.

UPDATE: Edited to embed link to fix formatting.

ABC Poll Methodology

2007-04-04T18:08:00.000-07:00

I haven't had time to review the new ABC poll closely, but their methodology description is commendably thorough.

This survey was conducted for ABC News, USA Today, the BBC and ARD German TV by D3 Systems of Vienna, Va., and KA Research Ltd. of Istanbul. Interviews were conducted in person, in Arabic or Kurdish, among a random national sample of 2,212 Iraqis aged 18 and up from Feb. 25-March 5, 2007.

Four hundred and fifty-eight sampling points were distributed proportionate to population size in each of Iraq's 18 provinces, then in each of the 102 districts within the provinces, then by simple random sampling among Iraq's nearly 11,000 villages or neighborhoods, with urban/rural stratification at each stage.

Maps or grids were used to select random starting points within each sampling point, with household selection by random interval and within-household selection by the "next-birthday" method. An average of five interviews were conducted per sampling point. Three of the 458 sampling points were inaccessible for security reasons and were substituted with randomly selected replacements.

Interviews were conducted by 103 trained Iraqi interviewers with 27 supervisors. Just over half of interviews were back checked by supervisors — 28 percent by direct observation, 14 percent by revisits and 10 percent by phone.

In addition to the national sample, oversamples were drawn in Anbar province, Sadr City, Basra city and Kirkuk city to allow for more reliable analysis in those areas. Population data came from 2005 estimates by the Iraq Ministry of Planning. The sample was weighted by sex, age, education, urban/rural status and population of province.

The survey had a contact rate of 90 percent and a cooperation rate of 62 percent for a net response rate of 56 percent. Including an estimated design effect of 1.51, the results have a margin of sampling error of 2.5 percentage points at the 95 percent confidence level.

Sounds thorough and professional. But then why is the contact rate only 90% (compared to Lancet II's 99%) and the cooperation rate only 62% (compare to Lancet II's 99%)? Isn't it a sign of incompetence on the part of D3/KA that there response rate is more than 40 percentage points lower than that of Lancet II (56% as compared to 98%)?

No, I think that this is evidence that the Lancet interviewers made up data, at least some of it. What other explanation could there be? The D3/KA interviewers were so rude that people didn't want to talk to them? The Lancet interviewers were so charming that virtually everyone was willing to answer their questions?

Other explanations are welcome.

Pedersen Quote

2007-03-29T01:41:00.000-07:00

Did Jon Pedersen really say this?

I know many, many people who have served in Iraq, worked in Iraq providing aid, contracts, or working on Iraq issues inside and outside the government over the entire course of the war. Not one of them thinks the Lancet numbers can be correct. As Jon Pedersen, the lead researcher for the UNDP household cluster survey conducted in 2004 (the only other, and much larger, study to use the Lancet method) recently stated, the Lancet numbers are "high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much"

I have seen a quote along these lines before, but can't find an authoritative citation. The link given does not work for me.

Random Numbers

2007-03-17T08:15:00.000-07:00

A central concern of critics is the relationship between the US authors of the paper and the Iraqi survey teams. What did the US authors tell the Iraqis to do in terms of the exact survey procedures? What did the Iraqis actually do? What did the Iraqis tell the US authors that they did? How can the US authors (and the rest of us) know for sure what the Iraqis did?

A small but revealing example of this involves the sampling procedures. The main paper reports:

Sampling followed the same approach used in 2004, except that selection of survey sites was by random numbers applied to streets or blocks rather than with global positioning units (GPS), since surveyors felt that being seen with a GPS unit could put their lives at risk.

...

As a first stage of sampling, 50 clusters were selected systematically by Governorate with a population proportional to size approach, on the basis of the 2004 UNDP/Iraqi Ministry of Planning population estimates (table 1). At the second stage of sampling, the Governorate’s constituent administrative units were listed by population or estimated population, and location(s) were selected randomly proportionate to population size. The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected.

The "Human Cost" paper reports the same.

Selection of households to be interviewed must be completely random to be sure the results are free of bias. For this survey, all households had an equal chance of being selected. A series of completely random choices were made. First the location of each of the 50 clusters was chosen according the geographic distribution of the population in Iraq. This is known as the first stage of sampling in which the governates (provinces) where the survey would be conducted were selected. This sampling process went on randomly to select the town (or section of the town), the neighborhood, and then the actual house where the survey would start. This was all done using random numbers. Once the start house was selected, an interview was conducted there and then in the next 39 nearest houses.

A perfectly sensible procedure. Random numbers are, indeed, widely used in surveys. But is this actually what happened? Consider Gilbert Burnham's recent speech at MIT. Watch from 1:07 to 1:10 in the video. Some crank (i.e., me) is trying to understand precisely what the procedure was.

According Burnham, the team did not use random numbers! Instead, he mentioned two approaches. Once is to write down all the names of the candidate streets on pieces of papers and then "randomly" select among them by hand. Also, for selecting the specific house, Burnham reports:

Once they selected the streets, then they numbered the houses on that street from one to whatever the end of that street was. And then they randomly, using serial numbers on money, they randomly selected a start number and started with that house, and from that they went to the nearest front door, the nearest front door, nearest front door, nearest front door, until they had a total of 40 houses.

"[S]erial numbers on money" is not the same thing as "random numbers," as any statistician will tell you. First, there is no guarantee that the serial numbers on currency are random. Who knows if there are more 1's than 7's on Iraqi (or US) currency? Second, using serial numbers makes it much easier for an unscrupulous interviewer to cheat.

Now, I don't actually worry that this caused a major problem. Putting street names on a piece of paper and picking one out of a hat is a fairly random process. Once you have picked a street, I wouldn't think that it matters much which house you start with. Even a malicious interviewer would have trouble, I would think, knowing which house to pick, whatever answer he might "want" to get.

But note that this might be a concern. The interviewers went to a neighborhood and, before starting the survey --- before picking the start house? --- told the local kids about the survey. From those kids, one could learn which house suffered several deaths. One could check this by seeing if there is a tendency for houses near the start of the survey in a given cluster to have higher death rates than houses at the end of the survey.

Anyway, my complaint is that this is another example of the methodology described in the article not being accurate. Don't claim to use random numbers while actually using some other process. And, if the article is incorrect in its claim about the process used to select which houses to interview, what else is it incorrect about?

The Lancet ought to publish a correction.

Timing Comments

2007-03-16T21:06:00.000-07:00

Kevin Wagner sent in these comments on the timing issue.

There are at least three claims about the Lancet study's procedures and the feasibility of its methods to which Les Roberts has given answers that would entail he either was misinformed about some of the fundamental aspects of his survey or was willing to play fast and loose with his rhetoric to dissuade inquiry.

The first two instances come from an article by the journal Nature, a copy of which is cited here:

http://psychoanalystsopposewar.org/blog/2007/03/01/nature-on-iraq-mortality-study/

1) "Roberts and Gilbert Burnham, also at Johns Hopkins, say local people were asked to identify pockets of homes away from the centre; the Iraqi interviewer says the team never worked with locals on this issue."

To my knowledge, this has never been addressed further by Roberts. It's an outright contradiction of the authors' claim with the interviewer's. It raises the questions of how much the study's authors accurately knew about their interviewers' behavior and also potential bias introduced from a failure to follow the study's methods on the parts of the interviewers.

2) "The US authors subsequently said that each team split into two pairs, a workload that is "doable", says Paul Spiegel, an epidemiologist at the United Nations High Commission for Refugees in Geneva, who carried out similar surveys in Kosovo and Ethiopia. After being asked by Nature whether even this system allowed enough time, author Les Roberts of Johns Hopkins said that the four individuals in a team often worked independently. But an Iraqi researcher involved in the data collection, who asked not to be named because he fears that press attention could make him the target of attacks, told Nature this never happened. Roberts later said that he had been referring to the procedure used in a 2004 mortality survey carried out in Iraq with the same team (L. Roberts et al. Lancet 364, 1857–1864; 2004)."

This clarification is problematic. Apparently, in response to a follow up question on the 2006 study, Roberts replies with something relevant only to 2004, i.e. that the interviewers worked independently. They did work independently in 2004 and did not [so he, Burnham and the interviewer claim] in 2006. Moreover, it is damaging to his case for the 2006 study's feasibility. In 2004, he had six effective interviewers. In 2006, he had four two-person teams, i.e., four effective interviewers. How would responding that he effectively had two more interviewers in 2004 than in 2006 prove that his 2006 timeframe was feasible?

3) Last is Roberts' response on to queries on the interview timeframe and his reply to Tim Lambert on same.

http://www.radioopensource.org/les-roberts-weighs-in-on-lancet-controversy/

He states:

"The two main criticisms which were in both the *Nature* article and *The Times* article are completely without merit. They said there wasn't enough time to have done the interviews. We had eight interviewers working ten hour days for 49 days, they had two hours in the field to ask each household five questions. They had time."

They had eight interviewers which he says worked 49 ten hour days. 8 interviewers x 10 hours x 49 days / 2000 houses in the sample = 1.98 hours.

However, as noted before, those 8 worked in two-person teams. 4x10x49/2000=.98.

In any case, how is the total time that was devoted to the entire project supposed to relate to the actual time they spent in the field? One to one correspondence? Every second they had for the project was spent in the field? Short of that, what does Roberts mean when he says they had 2 hours in the field per household?

http://scienceblogs.com/deltoid/2007/03/london_times_hatchet_job_on_la.php

Then Tim Lambert says he got a reply from Roberts in a post in the above blog entry:

"I asked Roberts and he says he agrees with Burnham about the interview length: 15-20 minutes."

So, again, what exactly is Roberts claiming when he says the interviewers had two hours in the field per household to ask five questions? How is this claim accurate or relevant? Interviews took 20 minutes, on the high side, according to Roberts. To what is "two hours in the field" supposed to relate? Did the interviewers really have an additional hour and 40 minutes per household in the field as his claim implies?

I am not sure how much I worry about this issue, especially since Burnham has told us that, on occasion, single interviewers would work alone (although we have no idea how often that occurred). It is further evidence that the US authors have trouble knowing exactly what the interviewers did. Related comments here.

Burnham Presentation: Notes and Comments

2007-03-16T17:41:00.000-07:00

I attended Gilbert Burnham's recent presentation at MIT. Tim Lambert provides a link to the video. I have not reviewed the video, but here are some notes that I made of the talk. (These might be incorrect; doublecheck the video.)

The work of the authors was split up about as you might expect. Burnham organized; Lafta organized the survey; Doocy did the statistics; Roberts wrote the article. None of the authors besides Lafta were ever in Iraq.

He mention something about the confidence intervals of the first survey going to infinity if the Fallujah data were included. I am not sure if I understood this point correctly. Recall every assumes that the results of Lancet I would have been even stronger if the Fallujah data were included. Thus, excluding Fallujah was "conservative." But perhaps this only applies to the mean of the distribution. It could be (?) that including an outlier widens the confidence intervals so much that one can no longer reject the null hypothesis of no increase in excess mortality. I think that I or someone else made this point during the discussion 2 years ago, but I am too lazy to look for a citation. If this is correct, then it strikes me as something that the readers of Lancet I deserved to know. I should come back to this point later. My best guess now is that I just misheard Burnham and he was just talking about the mean estimate going very high, i.e., to infinity.

Burnham mentioned that he wanted to oversample Fallujah if it were chosen again. And, mirable dictu, it was chosen. But isn't that a little suspicious? Again, my basic hypothesis is not so much that the surveyors were malicious as that they gave the Americans what they thought the Americans wanted. So, they knew that Burnham wanted to check out Fallujah again so they ensured (?) that Fallujah was "randomly" selected for Lancet II. (Does this mean that all the sampling was done by the Iraqis?) Also, if Fallujah were purposely oversampled, then one would need to adjust the estimates for this fact. Not easy to do! Did Doocy do this? How? Given that the reason that Burnham chose Fallujah to be oversampled was because he knew it was much more violent than other parts of Iraq, one would need to do some adjustment. I guess that Doocy could use the extra samples just to estimate Anbar more accurately and then combine the Anbar estimate with the rest of the country. But I don't recall any discussion of this in the paper.

The authors wanted to get the results of the survey published well before the election in order to avoid the controversy which plagued Lancet I, but they weren't able to do so because of fund raising and dealing with ethical reviews and the like. This makes little sense to me. Raising money and dealing with IRBs is tough and time consuming, but the survey work was complete by July 10, 2006. So, by that date, all (?) the issues of money and ethics were done. Lancet I went from survey completion to publication is one month. At the same rate, Lancet II could have been published in August. Now, one month is a very quick time from survey to publication, so no one would expect such a result. Indeed, getting to press within 3 months is still unusual. But I still expect that Lancet editor Richard Thorton likes October even year publication dates for these articles.

All the interviewers were from (the same, I think) community medicine center in Bagdhad

Tim Lambert comments: "The IBC made vociferous attacks on the studies because they want to defend their methods, and Les Roberts suggests that IBC are trying to stop the donations from drying up." I thought that this was a low blow, and an unusual comment in an academic seminar. But, it could also be true. Burnham made clear that this was Roberts's opinion, not his.

There was useful background on the conduct of the survey. Once the team had picked the street (house?) to start at, it would tell all the children in the neighborhood (who had gathered to see the strangers in their white coats) what the survey was about. The children would spread the word around the neighborhood, alleviating suspicion and making everyone comfortable.

Tim Lambert comments: "They will soon release the data (with identifying material removed) to other researchers." I hope so! There still seemed (to me) to be some hedging on this, that only "qualified" researchers would be allowed to see the data, that they (or their institutions) would be expected to testify somehow to data security. With luck, this won't be a problem, but I have my doubts.

Burnham claimed that the reason that the interviewers only asked for death certificates in 87% of the reported deaths was that they "forgot." This strikes me as an implausible (but testable) claim. Some blogospherians had speculated that the 13% where this wasn't done were purposely for choices made to avoid danger or trauma. Could you ask for a death certificate with a grief-stricken mother wailing on the front step? This claim is testable because we should see a pattern in the cases where no certificate was asked for. It should be focused on one or two teams and should occur early in the surveys rather than later. Checking this is an example of the analysis we will be able to do once the data is available. Related to this is the issue of the survey form itself. Has this been made publicly available? It should be! Also, any competently designed form would feature a check box for this (and every other question). If it is on the form, then how could one forget 13% of the time?

One of the biggest surprises (to me) was Burnham's admission that the teams operated independently at times. I think that the drill was that a team of 4 would go to a cluster, pick a main street, then a side street, then a house. Two teams of two would start knocking on doors. So each sub-team would have to 20 houses to finish this cluster. But, Burnham said, these two person teams would sometimes (how often?) operate independently. In other words, as one person "finished up" an interview in one house, her partner would leave and start on the next house himself. This procedure helps to alleviate concerns that there wasn't "enough time" to do the surveys but it does mean that if only one of the eight interviewers were malicious, she would have opportunity to make her results whatever she wanted. This is another reason for looking closely at the raw data. Mortality rates should be similar for all four members of the one team. (The other team, operating in different clusters, might have different results.

He mentioned the difficulty of getting good population estimates for Fallujah. He mentioned 500,000 as being a number for before the war but 200,000 after all the US military activity.

The slides make clear that the main-street bias (MSB) issue is less serious than one might suppose (and that the methodology write up in the paper is misleading --- unintentionally, I think). But the whole issue is quite complex and I hope to return to it later.

Again, these are from my messy notes. Check the video for what Burnham actually said.

SSS Removal

2007-03-15T10:53:00.000-07:00

Thanks to a comment which mentioned this link to the original removal of my SSS post.

18 October 2006
REMOVED: A case for fraud?

Amy Perfors

David Kane's most recent guest post about the Lancet study has been removed. Since this is not a normal practice for us, explanations for why (and why we posted it in the first place) are below the fold.

Why remove it? The tone is unacceptable, the facts are shoddy, and the ideas are not endorsed by myself, the other authors on the sidebar, or the Harvard IQSS.

Why post it in the first place, given this? Here I admit to an error in judgment on my part. I see my job as head of the Author's Committee as doing the somewhat mundane and boring tasks of coordinating and inspiring our posters, not exercising editorial control. I was uncomfortable with the post even before putting it up, but I also hate censorship, and -- since I don't know this field or this study very well -- I couldn't say with complete confidence that my discomfort was totally justified. I decided to err on the side of expressing something I was uncomfortable with, rather than stifling it. Again, that was probably an error with regards to this post, and I apologize. It was not up to the standards we aspire to here, and does not reflect our views.

Posted by Amy Perfors at October 18, 2006 03:45 PM

I think that there were comments to this post (before it was, in turn, removed) but don't know if they are preserved anywhere. Note that my original title included a question mark. I do not know whether the Lancet results are correct, but if there is fraud, it is most likely to be found in the (lack of) quality of the raw data, not in any arcana of cluster calculations. The statistics here are fine.

Child Mortality

2007-03-13T08:02:00.000-07:00

Comment threads at Crooked Timber are often interesting. Daniel Davies claims:

The thing is, that there is a “marker” in the Times article – as in, a statement that is not true and that is obviously not true to anyone who has read the article. It is in the following paragraph:

Dr Richard Garfield, an American academic who had collaborated with the authors on an earlier study, declined to join this one because he did not think that the risk to the interviewers was justifiable. Together with Professor Hans Rosling and Dr Johan Von Schreeb at the Karolinska Institute in Stockholm, Dr Garfield wrote to The Lancet to insist there must be a “substantial reporting error” because Burnham et al suggest that child deaths had dropped by two thirds since the invasion. The idea that war prevents children dying, Dr Garfield implies, points to something amiss.

This is not true. As table 2 of the study shows, infant mortality remained constant in the survey (when you adjust for the greater number of months in the post-war recall period) while child deaths increased substantially. They did not drop by two thirds, or indeed drop at all. Von Schreeb, Rosling and Garfield did not say they dropped either (presumably because they have read the survey). They said that the crude estimate of under-15 mortality was substantially lower than other estimates of under-5 mortality in Iraq, and that this implied that there may have been substantial under-reporting of child deaths. They then suggested that this reporting error might lead to additional uncertainty in the estimates of roughly the same size as the sampling error – +/- 30%. Note that, for bonus hack points, the “plus” sign in “+/- 30%” is not ornamental, and to treat Von Schreeb et al as providing evidence that the study was an overestimate is Kaplan’s Fallacy. This is my reason for believing that Anjana Ahuja didn’t read the research; it’s an error that could easily have been made in transcribing notes of a half-understood conversation but couldn’t have been made at all if you read the articles.

"ragout" writes:

On the question of whether the Times article is a “bad piece of science journalism,” I much prefer the Times’ version to Daniel’s. Specifically, Daniel summarizes Garfield and other critics as saying “that the crude estimate of under-15 mortality was substantially lower than other estimates of under-5 mortality in Iraq.”

But Daniel’s version is misleading. The critics were not quoting mortality rates as such, which would be deaths of the under-15 per kid under-15. If the critics had really compared under-15 mortality to under-5 mortality, as Daniel says, the critics would indeed be foolish.

But since the critics are prominent scientists, they certainly did not do anything so foolish. Instead, they compared under-15 deaths per birth in the Lancet study to under-5 deaths per birth in another study. The critics’ rely on the fact that, as a matter of logic, if there are X deaths of kids under 15, there must be less than X deaths of kids under 5.

The Lancet study has 36 under-15 deaths per 1000 births, and another pre-war study has 100 or so under-5 deaths per 1000 births. If follows that the Lancet study found an under-5 death rate less than 1/3 of the pre-war study. This is exactly what the Times article says, and Daniel obscures.

Second, Daniel claims that “infant mortality remained constant in the [Lancet] survey.” But as far as I can see there is no data in the paper to calculate pre and post war infant mortality. The paper just reports total births, not pre and post war births. Daniel, without telling the reader, is implicitly assuming that the birth rate remained constant (which hardly seems consistent with the drastic increase in violence).

I don't have the energy to dive into this one right now. Ragout seems to get the better of it but Davies is almost always correct (in my experience) in this sort of analysis. First pass, it seems like Davies is correct to criticize the wording of the original news article but that ragout (and Gilbert?) are correct on the substance. Is this point worth exploring further? Perhaps.

If the Lancet II survey team "made up" large portions of the data, then we would expect to find all sorts of anomalies like this. It is very hard (I would guess!) to make up data that "hangs together" and is consistent with other known information. Or could the difference be the fault of the previous survey? Or could it be due to chance?

ILCS Response Rates

2007-03-12T18:17:00.000-07:00

A common response to concerns about the high response rate achieved in Lancet II is to note that the Iraq Living Conditions Survey 2004 (ILCS) reported a response rate of 98.5%. The technical sophistication and competence of ILCS has been universally praised.

In each governorate, 1,100 households were selected for interview, with the exception of Baghdad, where 3,300 households were selected. The sample thus consisted of 22,000 households. Of these, 21,668 were actually interviewed.

In fact, the response rate for ILCS is higher than that for Lancet II (98.3%). So, what is the problem?

The issue is that ILCS was conducted in a much more thorough fashion than Lancet II.

COSIT staff were extensively trained in implementing the survey tool by researchers from the Fafo AIS. The first round of training took place in Amman, Jordan during the first three weeks of February 2003. Core staff from COSIT’s offices in each governorate were present, in addition to administrative staff from the headquarters in Baghdad. Training of local staff was subsequently conducted at six locations within Iraq during the first two weeks of March 2003 by COSIT’s core staff under supervision from Fafo.

Fieldwork started on March 22, 2004, and was completed by May 25, 2004. Data collection in the Governorates of Erbil and Dahouk were implemented and completed in August 2004.

After each selected PSU had been mapped and listed, interviewers were sent to the 10 selected households. Interviewers were organized in teams of five, with individual supervisors who continuously provided guidance and checked the quality of all incoming interviews. When necessary, interviewers were sent back to the households to reconfirm information. Furthermore, supervisors from COSIT’s headquarters in Baghdad and Fafo staff also visited the interviewer teams.

Upon completion of the interviews, the information was sent to the governorate office for registration and inspection, then to the Baghdad main office for coding and data entry. During the data entry process, extensive quality control was implemented, and questionnaires were sent back to the field for re-interviewing or update both by COSIT’s Baghdad office and by Fafo headquarters in Oslo.

Completed data files were continuously sent to Fafo’s headquarters in Oslo, Norway, where further quality checks were implemented. In instances where problems arose, direct communication was made with COSIT. Several times during the fieldwork, COSIT arranged meetings with its offices’ heads in order to inform them of problems that had surfaced and resolve them.

In other words, ILCS interviewers went back to the sample households again and again and again. This is quite different from the procedure in Lancet II. In that case, a cluster was visited on just one day. In fact, it appears that houses were just checked one time. What good fortune that there was almost always someone (head of house or spouse) at home!

What we need, obviously, is more information about the initial ICLS samples. How many households were present for this first interview? How many immediately agreed to participate? Only this level of detail will tell us if the final ILCS response rate is relevant in evaluating the reliability of the Lancet II sample.

Kurdish Speakers?

2007-03-12T17:45:00.000-07:00

One aspect of the debate that has always confused me is the issue of Kurdish speakers. My (correct?) understanding is that Iraq has a large Kurdish population which does not speak Arabic and/or is unlikely to participate in a survey conducted by a non-Kurdish speaker. References welcome! Lancet II reports that all the interviewers spoke English and Arabic. It does not mention Kurdish. Does this mean that none of the interviewers spoke Kurdish? That is what I would expect. Does that matter?

Consider the trouble that the ICLS went through to translate its survey into both Arabic and Kurdish.

The questionnaires reflect the nature of the survey. Two questionnaires were used: one general questionnaire for each household, answered by the household head or a member of the household with knowledge of all members was the respondent; and one targeted questionnaire used to interview women of the household aged between 15 and 54 years. The first questionnaire dealt with housing and infrastructure, household economy, basic demography, and the education, health, and labour force characteristics of the household members; the second focused on the women’s reproductive history and children’s health.

Three versions of the questionnaires—one Arabic and two Kurdish—were used in the field.. Although the questionnaires were developed in English, they were translated twice—once into Arabic or Kurdish, then back again into English—in order to verify the translations and ensure that all members of the survey team had a common understanding.

Compared to many surveys, the questionnaires were quite long, with a median interviewing time of 83 minutes. Fifty percent of the interviews lasted between 60 and 105.

Either ICLS wasted a lot of time and money unnecessarily translating the survey into Kurdish or there is no way that Lancet II got a 98% response rate without using Kurdish speakers. Or is there a third possibility?

Data Validity

2007-03-12T13:24:00.000-07:00

This article on Lancet II and the news reactions thereto is interesting.

As a biostatistician, the Bloomberg School's Zeger has thought a lot about the study. "I am so impressed by Gil because he was able to conduct a scientific survey on a shoestring budget under very difficult circumstances," he says. He does not dismiss all concerns about the methodology. "It was the best science that could be done under the circumstances. We're always making decisions absent scientific-quality data — that's public health practice." But he draws an important distinction between practice and science. "We tend to have a different standard for scientific research. This study was on the research end. It was published in a scientific journal. There are a lot of aspects that are below the reporting standards you would have if you were doing a U.S. clinical trial, for example: the documentation for each case, the ability to reproduce the results, detailed information about how everything was done. I think it would be useful for the school and the public health community to think through these kinds of issues.

"[But] it's absolutely appropriate, on very limited resources, to go into a place like Iraq and make an estimate of excess mortality to use in planning and making decisions. My own sense is I would rather err on the side of generating potentially useful data, with all of the caveats. I think noisy data is better than no data." Zeger notes that the tests of the data's validity, built into the second survey at his recommendation, all checked out. He admits the numbers are hard to grasp, especially the study's estimate that from June 2005 to June 2006, Iraqis were dying at a rate of 1,000 per day. "That's a lot of bodies," he says. "I have a hard time getting my mind around that. But as a scientist, what do you do? That's the number."

I certainly agree that noisy data is better than no data, but only if we have access to all the details of where that noisy data comes from. What "vaidity" checks is Zeger talking about?

Links Related to Moore

2007-03-12T12:23:00.000-07:00

Here are some links related to the unimpressive Steven E. Moore Wall Street Journal op-ed.

Stats.org
Tim Lambert

The cluster comments made by Moore seem wrong (as demonstrated by Lambert and others. But this point is, at least, not unreasonable.

With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census.

Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.

And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it --- try using Google to find a survey that does not ask demographic questions.

I don't know of a survey that doesn't ask demographic questions, but I haven't really looked. Why would demographics be useful? First, it is (perhaps) a check against randomization mistakes and/or fraudulent data. If the demographics of the survey don't match the demographics of the population, then something is wrong. (But if the match isn't too far off, then the concern isn't that great.) Second, you could use demographics to adjust the survey results. If, say, Sunnis made up 50% of the survey but are, we think, at most 25% of the population, then you would want to adjust the results accordingly.

I wonder why the survey form did not include demographics. Unlike Moore, I doubt that the reason is nefarious. But it is endlessly annoying that the Lancet authors refuse to even release the questions or actual survey forms.

UPDATE: Lambert points out that plenty of demographic data (age and gender of household residents) was collected. Moore's response:

Despite Les' eloquent response, he has yet to reveal any comparison of demographic information for the 2006 survey to the 1997 Iraqi census, the 2003 update to that census, the 2004 UNDP/ILCS survey or any other demographic instrument.

Seems like Roberts/Lambert is correct on this one. It still drives me nuts that we don't have access to the raw data.

IRI Response Rate

2007-03-12T11:38:00.000-07:00

It appears that Brookings' excellent Iraq Index project may allow me to track every nationwide poll conducted in Iraq in 2006. If so, I could collect the response rates for each and then see whether or not Lancet II is an outlier.

One example is this poll by the International Republican Institute from March 2006. The accompanying Powerpoint slides report a response rate of 93% (2,804 out of 3,000). This is higher than the rates we have seen from WPO but lower than that of Lancet II. I will contact IRI to get more details.

More Plausible Respone Rates

2007-03-12T08:50:00.000-07:00

As a follow up to this post, consider this January 2006 poll conducted for World Public Opinion.org. Methodological details (pdf):

The survey was designed and analyzed by the Program on International Policy Attitudes for WorldPublicOpinion.org. Field work was conducted through D3 Systems and its partner KA Research in Iraq. Face-to-face interviews were conducted among a national random sample of 1,000 Iraqi adults 18 years and older. An over sample of 150 Iraqi Sunni Arabs from predominantly Sunni Arab provinces (Anbar, Diyalah and Salah Al-Din) was carried out to provide additional precision with this group. The total sample thus was 1,150 Iraqi adults. The data were weighted to the following targets (Shia Arab, 55%, Sunni Arab 22%, Kurd 18%, other 5%) in order to properly represent the Iraqi ethnic/religious communities.

The sample design was a multi-stage area probability sample conducted in all 18 Iraqi provinces including Baghdad. Urban and rural areas were proportionally represented. A total of 5 sampling points (4 urban and 1 rural) of the 116 employed were replaced for security reasons with substitutes in the same province and urban/rural classification. Among all the cases drawn into the sample, a 94% contact rate and 74% completion rate were achieved.

The contact and completion rates are almost identical to those of the November 2006 survey. Again, how can these rates be so much lower than those for Lancet II? Now, it could still be that the surveyors employed by WPO (D3 and KA) are not actually surveying people, are just filling out the forms themselves. Anyone doing this is well advised to report plausible contact/completion rates rather than 100%, even though high participation is what the client "wants." Excessive contact/completion rates are the first thing that a wise client checks to ensure against fraud.

But if we assume that D3 and KA have done a proper job --- in nationwide surveys which bracket Lancet II --- it because hard to understand what magic the Lancet survey teams (composed of physicians who, I think, do no other survey work outside of the two Lancet articles) perform in order to produce such high rates. How are the Lancet survey teams able to reach 99% of the intended households while D3/KA can only contact 94%? How are the Lancet teams able to convince virtually everyone they meet (completion rate over 99%) to finish the survey, while D3/KA can't persuade more than 3/4 of the people they meet to finish? Are D3/KA survey teams rude or scary or socially awkward? Are the Lancet teams more friendly or engaging or persuasive?

Perhaps. Yet I think that one survey team is not telling the whole truth, has tried mightily to give its bosses what the bosses want to hear.

Plausible Response Rates

2007-03-12T08:25:00.000-07:00

To the extent that I (and others) find the 98.3% response rate for Lancet II completely implausible, it behooves us to highlight similar surveys with more believable response rates. This report from World Public Opinion.org (WPO) provides such an example, especially appealing because it featured a nationwide sample and was conducted in September 2006, just a few months after the fieldwork for Lancet II. The headline, "Baghdad Shias Believe Killings May Increase Once U.S.-led Forces Depart but Large Majorities Still Support Withdrawal Within a Year," indicates that this is hardly the work of crazed Neocons.

What was the response rate for this poll? Details here (pdf):

The survey was designed and analyzed by the Program on International Policy Attitudes for WorldPublicOpinion.org. Field work was conducted through D3 Systems and its partner KA Research in Iraq. Face-to-face interviews were conducted among a national random sample of 1,000 Iraqi adults 18 years and older. An over sample of 150 Iraqi Sunni Arabs from predominantly Sunni Arab provinces (Anbar, Diyalah and Salah Al-Din) was carried out to provide additional precision with this group. The total sample thus was 1,150 Iraqi adults. The data were weighted to the following targets (Shia Arab, 55%, Sunni Arab 22%, Kurd 18%, other 5%) in order to properly represent the Iraqi ethnic/religious communities.

The sample design was a multi-stage area probability sample conducted in all 18 Iraqi provinces including Baghdad. Urban and rural areas were proportionally represented. Only one rural sampling point of the 115 employed were replaced for security reasons with substitutes in the same province and urban/rural classification. Among all the cases drawn into the sample, a 93% contact rate and 72% completion rate were achieved.

Since response rate is contact rate times completion (or participation) rate, the total response rate for this survey is 67%. (It could be that the completion rate here is actually from the whole sample, in other words, the 72% figure is the one we should use.) How can it be that Lancet II could achieve such a dramatically higher response rate? Note that both aspects of the overall response rate were lower for this survey. WPO had more trouble finding the individuals which it wanted to survey (99% versus 93%) and, having found those individuals, it had much more trouble convincing them to participate in the survey (99% versus either 72% or, if completion includes the contact problems, 77%).

If Lancet II had reported contact and participation rates more like those of the WPO, I would be much less suspicious of their results.

More Response Rate Details

2007-03-11T15:49:00.000-07:00

As a follow up to this post on response rates, I want to dive into the details in one of the polls that Kieran Healy cites because it helps to highlight just how implausible a 98.3% response rate for Lancet II is. Healy, among other examples, cites a Gallup poll (pdf) with a 97% response rate. He implies that, if Gallup (regularly?) gets a 97% response rate, than a 98% rate for Lancet II is perfectly plausible.

Here (from page 11) are all the detail we have on this 97% rate.

Face-to-face interviews were conducted among 1,178 adults who resided in urban areas within the governorate of Baghdad. Interviews were carried out between August 28 and September 4. The response rate was 97 percent; 3 percent of those selected refused to participate in the study.

A probability-based sample was drawn utilizing 1997 census data. Census districts were utilized as primary sampling units (PSUs). A total of 122 PSUs were chosen using probability-proportional-to-size methods. About 10 interviews, one per household, were conducted at each location. Interviewers were given all relevant address details for each PSU. Within each selected household, respondents were selected using the Kish method.

For the results based on this sample, one can say with 95% confidence that the margin of error is approximately ± 2.7%.

Now, it would be nice to have more details on this survey and, in time, I hope to find someone with lots of experience conducting polls in Iraq specifically and third world countries in general, but there are several issues to keep in mind.

Recall that non-response generally falls into two categories: failure to contact and, given that someone has been contacted, a refusal to participate.

Yet Gallup only provides us information about the response rate for people who were not absent. In other words, we have no information on the contact rate. We just know the participation rate. (It could be that Gallup is being sloppy and that this 3% who "refused to participate in the study" includes those who were not home.)

So the correct comparison of this 97% rate is not with the 98.3% response rate for Lancet II but the 99.2% participation rate.

Again, the difference between 97% and 99.2% may not seem large. And it isn't.

First, I just want to point out that the Lancet II response rates are higher than any other survey (with one possible exception to be addressed later). Second, one could just as easily say that the refusal rate for this Gallup poll is more than three times higher than the refusal rate for Lancet II. Why would that be? Why would households be so much less willing to participate in this Gallup than they are in the Lancet II? Third, the closer one gets to 100% participation, the more difficult it is to make progress. There is a much larger difference between 97% and 99% participation rates than between 60% and 62% rates because the marginal 2% increase is much harder to achieve the closer you get to 100%. Fourth, certain aspects of Lancet II should make it harder to have a higher participation rates. For example, this Gallup poll did not require the presence of the head of the household or spouse. Any resident adult would do.

Until we know more details about opinion polling in Iraq, it is tough to know what to do with very high response rates, rates much higher than anything we see in the US. But it is still a mystery why the rates for Lancet II should be so much higher than any other survey.

Iraqanalysis

2007-03-11T14:36:00.000-07:00

Iraqanalysis.org seems to be a useful site. I especially liked their thoughts on survey response rates. See the link for details.

Terminology

2007-03-11T13:23:00.000-07:00

In looking hard at the response rate for Lancet II, it is helpful to use some terminology. Recall (from page 4):

[A] final sample of 1849 households in 47 randomly selected clusters. In 16 (0·9%) dwellings, residents were absent; 15 (0·8%) households refused to participate.

I am unable to find web-based standard definitions for survey terms (suggestions welcome). So, I define the contact rate as the percentage of households (of those that the interviewers attempted to contact) in which residents were not absent. In this survey, the contact rate was 1833/1849 or 99.1%. The participation rate is the percentage of households (of those contacted) which agree to participate in the survey. In this case, it is 1818/1833 or 99.2%. The response rate is then the number of participating households divided by the number of households at which contact was attempted, or 1818/1849 = 98.3%

One confusing aspect is that the Lancet authors do not explain how they handled the head of household issue. On page 2 we have "The survey purpose was explained to the head of household or spouse, and oral consent was obtained." This implies that either the head of household or the spouse was the source of the information. This makes sense since such people are likely to have knowledge of the inhabitants over the last couple of years. But how often were the household head and spouse both absent? It would seem unlikely that this never happened. Did the interviewers code this as "residents" being "absent" and include these in those 16 cases? Or did they just get information from whatever adult was present?

I do not think that this is a critical issue, but there is value in getting all the details correct.

Response Rate

2007-03-10T10:16:00.000-08:00

Imagine that we conduct 100 independent polls, each of 10,000 people, in the US. The polls are all supposed to use the exact same procedure. Will all the response rates be exactly the same?

Of course not. One poll might have a 60% response rate and another a 64% response rate, not because of fraud or malfeasance but just because of random chance. In fact, we would expect there to be a distribution of response rates, perhaps centered around 60% with a 2% standard deviation. Although the poll with a 64% response is higher than the vast majority of the other polls, at least one poll out of the 100 needs to be the highest, just as one will be the lowest. An extreme result is no proof of fraud. This is all the more so since there is no human way to ensure that all 100 polls use the exact same procedure. At the very least, different individuals will be conducting the polls, or the same individuals will be conducting the polls on different days.

But what if the results of 99 of the polls produce a nice normal distribution centered on 60% with a 2% standard deviation but the 100th poll features a 99% response rate? What would be a reasonable conclusion?

First, this could just be random. Perhaps poll results are fat-tailed, and so extreme results are to be expected. Second, this could just be an honest mistake. Perhaps the interviewers in the 100th poll mismarked the forms. Perhaps the forms were marked directly but there was an error in the automatic reader. Third, this excessively high response rate might be evidence of fraud, might indicate that the reviewers for this poll did not bother to interview anyone and just filled out the forms themselves. Without more information, it is hard to know which of these three explanations is correct or if there is something else going on.

Readers can judge for themselves, but if anyone reports a 99% response rate for a US poll, I think that the second and third explanation (honest mistake or fraud) are the most likely. I can find no evidence that poll results are fat-tailed. For determining the usefulness of the poll results, it doesn't really matter whether the problem is a mistake or a fraud. In either case, the results of the poll are not reliable.

It should be obvious how this theoretical concerns relates to the Lancet II. I argue that the 99% response rate is ludicrously high, way higher than the rate for almost all polls on almost all subjects in almost all countries in the world. Kieran Healy takes me to task and writes:

Kane says, “I can not find a single example of a survey with a 99%+ response rates in a large sample for any survey topic in any country ever.” I googled around a bit looking for information on previous Iraqi polls and their response rates. It took about two minutes. Here is the methodological statement for a poll conducted by Oxford Research International for ABC News (and others, including Time and the BBC) in November of 2005. The report says, “The survey had a contact rate of 98 percent and a cooperation rate of 84 percent for a total response rate of 82 percent.” Here is one from the International Republican Institute, done in July. The PowerPoint slides for that one say that “A total sample of 2,849 valid interviews were obtained from a total sample of 3,120 rendering a response rate of 91 percent.” And here is a report put out in 2003 by the former Coalition Provisional Authority, summarizing surveys conducted by the Office of Research and Gallup. In the former, “The overall response rate was 89 percent, ranging from 93% in Baghdad to 100% in Suleymania and Erbil.” In the latter, “Face-to-face interviews were conducted among 1,178 adults who resided in urban areas within the governorate of Baghdad … The response rate was 97 percent.” So much for Iraqi surveys with extraordinary response rates being hard to find.

See the original post for links. Now, it is hard to know what to make of this. Healy finds the results for 4 polls. Their response rates are 82%, 91%, 89% and 97%. The average here is 89.75%. Let's round up to 90%.

So, I claim to not be able to find any poll with a response rate higher than 99% (the response rate in the Lancet II). Healy claims that I am wrong and, for evidence, cites 4 polls with response rates lower than 99%. Am I missing something? Isn't he just providing further evidence for my concerns. If, of the hundreds (?) of polls conducted in Iraq, Lancet II features the highest response rate, isn't that cause for concern? (Note that Healy, in an earlier portion of the same post cites a poll with 100% response rate. I hope to return to that specific example at a later date.)

Again, one poll will, by definition, have the highest response rate. A priori, there is no reason why Lancet II might not be that poll. But it is a bit worrying that the poll with the most controversial result of any poll conducted in Iraq in the last 2 years would also have the highest response rate of any poll. What are the odds of that? If response rate and controversy are independent, then this would be a surprising result. If they are correlated (perhaps people are more likely to want to participate in a poll about death rates than in a poll on less controversial topics), then this is to be expected.

In any event, the annoyance comes when someone like Henry Farrell on Crooked Timber writes:

I don’t have very much respect for David Kane (it isn’t me who was accused of fraud). What bugs me as much as the initial offensive accusation is that he never to my knowledge apologized afterwards or sought to retract his accusation (if I’d done something similar, god forbid, I hope that I’d have apologized abjectly to the offended parties; I’d likely have disappeared entirely from public debate immediately thereafter).

What accusation does Henry want me to retract? The main point of my initial post was a) that any problems with Lancet II are likely to lie with the interviewers, not with the specific clustering formulas and other arcana used by the authors in their statistical analysis and b) there is some evidence that the response rate for Lancet II is excessively high. Why such concerns make me untouchable is unclear.

Lancet On Iraqi Mortality

Dubious Polling

Response Rates

Old Posts

Burnham Sanctioned

Likely Correction?

Misleading Statements on Unique Identifiers

L2 Sampling Details

Pope of Debunkers

End Game

Tiny Chance

Counterfactual

IBC Estimates

Spagat's Latest

Peer Reviewed Articles Critical of L1/L2

Lancet Comments on My Article

Influencing the Election

Eight Transcripts

Propaganda

Daponte article

Obermeyer, Murray and Gakidou

Two Transcripts

IFHS As An Overestimate?

Disappearing Questions

"Civilian" Casualties

Missing Answers

Sleazy Switcheroo

Data for L1

Recent Articles on L2

Another False Statement

False Statement on Missing Certificates

Tweaked

Roberts Quotes on IFHS

Next Steps

Reconciliation

Said and Unsaid

IFHS Versus L1

Crooked Timber Comment

National Journal Article

TCS Daily Article

100% Response Rate

Kahl on Casualties

Not in My Journal

AP Story on JSM Meeting

Bias Presentation

Truther

Details on Falluja

War Crime

Amer and Scheuren

Missing Data

Love Postings

More Quotes From Burnham

Old Roberts Interview

Roberts Talk at Brown

Probably

Roberts Comments on ILCS and Politics

Very, Very Wide

More Questions

Soldz Article

Data!

Lafta Visit

Infinity

ESOMAR Podcast

Pedersen Quote

Data Now Available

Data Availability

ABC Poll Methodology

Pedersen Quote

Random Numbers

Timing Comments

Burnham Presentation: Notes and Comments

SSS Removal

Child Mortality

ILCS Response Rates

Kurdish Speakers?

Data Validity

Links Related to Moore

IRI Response Rate

More Plausible Respone Rates

Plausible Response Rates