100% Response Rate

My main source of suspicion with regard to Lancet II concerns its amazingly high response rate. See previous discussion here, here and here. Kieran Healy counters that concern by writing:

The immediate problem with this charge is that, as it turns out, phenomenally high response rates are apparently very common in Iraq, and not just in this survey. UK Polling Report says the following:

The report suggests that over 98% of people contacted agreed to be interviewed. For anyone involved in market research in this country the figure just sounds stupid. Phone polls here tend to get a response rate of something like 1 in 6. However, the truth is that – incredibly – response rates this high are the norm in Iraq. Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.

If this is correct, then the only bit of circumstantial evidence that Kane proffers in support of his insinuation is in fact a misconception based on his own ignorance.

Related discussion here. Healy is a serious scholar so his objections are worth careful study. The paper and associated slides are available. Keep in mind that we have three levels of references: Healy talking about what UK Polling Report (a blog written by Anthony Wells, not a formal organization) says about a paper by John Heald of Opinion Research Business (ORB).

Looking at the paper and slides, it is clear that, although Heald (presumably) presented the paper, the other author, Munqith Daghir of IIACSS in Iraq was responsible for the polling results. (It is not clear to me if Heald has ever been to Iraq, much less conducted a poll there.) ORB seems (?) like a competent organization, but its website makes little mention of doing work in Iraq. Instead, it reports that:

Research professionals at ORB have worked with over 15 world leaders on their Public Affairs initiatives including studies in the UK, US, Russia, South Africa, Malaysia, Taiwan, Turkey, Bulgaria, Lithuania, Malta and Gibraltar.

This report indicates that ORB first started to collaborate with Daghir sometime in 2005. The paper referenced by Healy/Wells reports that:

One man who listened to what his people allegedly thought about the invasion was Dr Munqeth Daghir, a lecturer in the University. He was caught between occupying forces saying that they had come to liberate Iraq and protect them from the Baathist regime and Iraqi exiles slowly returning saying that they wanted to help and could represent the real Iraqi. However, he knew that neither was a real reflection of what Iraqis really thought and wanted.

So, having read a book about market research and polling in between avoiding bombs that were dropping on his city, the first poll was conducted in Baghdad amongst a representative sample of 1,000 adults. That was in April 2003 and more than three years later and having the benefit of carrying out more than 150 polls, we want to demonstrate the advantages and difficulties of polling in Iraq and then quantify what people really thought both then (post conflict) and now.

Daghir is to be congratulated for making a life for himself as a pollster amidst the chaos of Iraq. But "having read a book about market research and polling" is not, shall we say, the most impressive pollster resume the world has ever seen. The entire article continues in a similar style. This is not so much a "paper" on the polling situation in Iraq as an advertisement for Daghir and his company. Now, there is nothing wrong with advertisements, and I have no reason to believe that Daghir and IIACSS aren't high quality pollsters, but Healy cites all this to demonstrate that the response rate for Lancet II is not an outlier result, that it is typical of Iraqi polls. Note again the quote from Wells which Healy selects and then comments on.

Earlier this year Johnny Heald of ORB gave a paper at the ESOMAR conference about his company’s experience of polling in Iraq – they’ve done over 150 polls since the invasion, and get response rates in the region of 95%. In November 2003 they did a poll that got a response rate of 100%. That isn’t rounding up. They contacted 1067 people, and 1067 agreed to be interviewed.

If this is correct, then the only bit of circumstantial evidence that Kane proffers in support of his insinuation is in fact a misconception based on his own ignorance.

Speaking of "ignorance," it is not clear if Healy (or even Anthony Wells) ever read the report in question. It does not mention ORB! It seems to me that Wells/Healy are mistaken, that ORB had no "experience of polling in Iraq" prior to 2005, that "they" did none of this work. Instead, Daghir/IIACSS have done some subcontracting for ORB in 2005/2006 and both IIACSS and ORB are eager to do more business together. But, Healy/Wells have no business claiming that ORB had anything to do with the poll results prior to 2005.

Consider this January 2006 here with Daghir.

In under three years he [Daghir] has run over 100 surveys for the UN, international NGOs, foreign governments, ad agencies, the media and FMCG companies. He has also carried out research on behalf of UK-based research agency ORB for almost a year now, looking at public attitudes towards topical issues and is about to explore attitudes towards tobacco.

In other words, Daghir has been working with ORB for "almost a year" as of January 2006. They had no relationship that we know of back in November 2003 when the implausible 100% response rate poll was conducted.

At that time [April 2003] I hadn’t much money. All of our savings in the bank had been looted. We hadn’t received the payment for most of the government projects we had done. So we were mostly bankrupt. I sold my car, and my partners sold things to fund this project. And 14 of my students on masters and PhD degree courses agreed to work with me for free. My daughter and my son worked as data punchers. I used my own computer with a generator. We started like a family business, in a big room with my son, my daughter, my partner’s son, my partner’s daughter, working together with our students. I told them how to code the questionnaire, how to enter the data. After two weeks I started the fieldwork.

It is a nice story and Daghir deserves credit for his ambition and bravery. But, he was novice, self-taught surveyer. What are the odds that he got everything correct the first time?

I knew that Baghdad is distributed into nine different areas, and how many citizens lived in each one. But to tell the truth, I didn’t know anything about the real random systematic sample. We did it randomly by going to any house we wanted to go to. So it wasn’t a perfect sample.

Indeed.

The key table is on page 2 of the report. There are six surveys listed. The first one, from November 2003, reports a 100% "Response Rate" in sample of 1,167. This is, of course, absurd. A proper poll picks out the 1,000 or so people it wants to sample from a larger population. It then searches for those people. You can never, ever find them all, no matter how "friendly" the local population. A charitable interpretation is that these are more "participation rates" than "response rates." (Background on terminology here.) In other words, the interviewers kept on looking for people to contact, perhaps by going around a market, perhaps by traveling from house to house. Some houses were empty. Some people refused to answer the door and/or talk with them. But, of those that they did contact, most were willing to participate. (The fact that the first column is labeled "Total Contacts" makes this plausible.)

The main problem with Lancet II is not that participation rates were high. That's true for polling all over Iraq. The problem is with the contact rates. These polls seem to be ignoring that aspect in their reporting. No poll of 1,000 people finds everyone.

And, even if we wanted to believe this outlandish result, it would still have little bearing on the Lancet II data because the polls are almost 3 years apart. Perhaps response rates were extremely high in November 2003, but the Lancet II interviews were done in 2006. The remaining 5 polls in the table have an average response rate of 87%, with none higher than 91%. How did the Lancet surveys do 10% better?

Kahl on Casualties

As time passes, more and more critiques of the Lancet results are appearing in the peer-reviewed academic literature. One of the first is from Colin Kahl. (Hat tip to Henry Farrell.) The citation is: Colin Kahl (2007), “In the Crossfire or the Crosshairs? Norms, Civilian Casualties, and U.S. Conduct in Iraq,” International Security 32:7-46. These quotes come from pages 11--13.

There are numerous estimates of civilian casualties in Iraq, all of which are problematic. Iraq Body Count (IBC), a nonprofit organization that derives estimates from English-language and translated media sources, suggests that as many as 7,393 civilians may have been killed during major combat. Another estimate by the Project on Defense Alternatives, which corroborated media reports with hospital and burial records and filtered out likely combatant fatalities, suggests that 3,230–4,327 Iraqi civilians were killed.12 In comparative perspective, it is significant to note that the number of Iraqi civilians killed during the 2003 invasion was similar to that of the 1991 Persian Gulf War, even though the mission objectives in 2003 required coalition forces to operate much more extensively in Iraqi cities, where they confronted both regular Iraqi units and irregular Fedayeen fighters largely indistinguishable from the civilian population.13

For the SO/COIN period, the two leading estimates come from IBC and the Brookings Institution. IBC estimates that as many as 54,303 Iraqi civilians were killed from May 1, 2003, through the end of 2006. The maximum number attributed to U.S. forces or crossfire is 4,399. That figure, however, is probably an underestimate because it includes only 87 deaths from U.S. actions at checkpoints and during convoy operations, known as “escalation of force” (EOF) incidents. If estimated EOF fatalities based on U.S. military figures are added to the IBC numbers and adjusted for double counting, the modified IBC total is 5,429 deaths attributable to U.S. forces and crossfire from the declared end of major combat through the end of 2006 (representing 10 percent of the total violent deaths).14 Over the same time period, the Brookings Iraq Index—which supplements IBC data with hospital and morgue records reviewed by the UN Assistance Mission for Iraq (UNAMI), and makes a further upward adjustment to account for likely undercounts from media reporting—estimates that 76,552 Iraqi civilians were killed.15 The Brookings Iraq Index does not provide a comprehensive estimate of deaths attributable to U.S. forces or crossfire, but assuming the same percentage documented by IBC and adding EOF incidents produces an adjusted Brookings estimate of 8,685 deaths in the SO/COIN period through the end of 2006.

Another well-known study, based on an Iraqi household survey and reported in the British medical journal The Lancet in October 2006, estimated 601,000 excess violent deaths since the invasion.16 Although widely cited as an estimate of civilian deaths, the majority of the 300 postinvasion violent deaths recorded in the sample of 1,849 households were military-aged men (59 percent aged 15–44; 78 percent aged 15–59), and no attempt was made to differentiate between combatants and noncombatants. Moreover, the Lancet estimate is so much higher than other available tallies, and its findings suggest so many implausible implications given other available data on the nature of the conflict, that its conclusions seem dubious, perhaps reflecting some significant sampling bias or reporting error.17 Indeed, a much larger household survey funded by the UN Development Programme in 2004 suggests that the IBC and Brookings figures are probably much closer to the actual number of civilian deaths.18 Jon Pedersen, the research director for the UNDP study, recently stated that he believes the Lancet numbers are “high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much.”19 Consequently, the Lancet numbers are not used for the analysis here.

The associated footnotes are:

12. Iraq Body Count, http://www.iraqbodycount.org; and Carl Conetta, “The Wages of War: Iraqi Combatant and Noncombatant Fatalities in the 2003 Confiict,” Research Monograph, No. 8 (Cambridge, Mass.: Project on Defense Alternatives, October 20, 2003), http://www.comw.org/pda/0310rm8.html.
13. Estimates of direct civilian deaths during the Gulf War range from 3,500 to 15,000. Conetta, “The Wages of War”; and Medact, Collateral Damage: The Health and Environmental Costs of War on Iraq (London: Grayston Center, November 2002), p. 2.
14. ”Murder Is Certain,” Economist, March 23, 2006; and Josh White, Charles Lane, and Julie Tate, “Homicide Charges Rare in Iraq War,” Washington Post, August 28, 2006.
15. Michael O’Hanlon and Jason H. Campbell, Iraq Index (Washington, D.C.: Brookings, April 26, 2007), p. 15.
16. Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts, “Mortality after the 2003 Invasion of Iraq: A Cross-Sectional Cluster Sample Survey,” Lancet, October 11, 2006, pp. 1–8.
17. Although 92 percent of those asked by the research team provided death certificates, the study’s estimate is 12 times higher than those provided by UNAMI and the Los Angeles Times, based on a review of death certificates from Iraqi morgues and hospitals for the comparable time period. United Nations Assistance Mission for Iraq, Human Rights Office, “Human Rights Report,” May 1–June 30, 2006; and Louise Roug and Doug Smith, “War’s Iraqi Death Toll Tops 50,000,” Los Angeles Times, June 25, 2006. Moreover, the implied number of wounded and car bomb victims, heterogeneous distribution of victims, very high (and historically unusual) ratio of excess violent deaths to nonviolent deaths, mysterious postinvasion drop in child mortality, low estimate of prewar (sanctions era) mortality, and scale and trend in deaths attributed to the coalition all seem highly implausible given everything else that is known about the conºict, suggesting that the Lancet estimate is the by-product of some significant sampling bias or other problem with how the survey was conducted. See Beth Osborne Daponte, “The Civilian Death Toll in IraqWar Proves a Nebulous Statistic,” Taipei Times, January 28, 2007; Hamit Dardagan, John Sloboda, and Josh Dougherty, “Reality Checks: Some Responses to the Latest Lancet Estimates,” Iraq Body Count, October 16, 2006, http://iraqbodycount.org/press/pr14.php; Jim Giles, “Death Toll in Iraq: Survey Team Takes on Its Critics,” Nature, March 1, 2007, pp. 6–7; Madelyn Hsiao-Rei Hicks, “Mortality after the 2003 Invasion of Iraq: Were Valid and Ethical Field Methods Used in This Survey?” Households in Conflict Network, Research Design Note, No. 3 (Brighton, U.K.: Institute for Development Studies, University of Sussex, December 2006); Neil F. Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, and Gesine Reinert, “Bias in Epidemiological Studies of Conflict Mortality,” Households in Conflict Network, Research Design Note, No. 2 (Brighton, U.K.: Institute for Development Studies, University of Sussex, 2006); Fred Kaplan, “Number Crunching: Taking Another Look at the Lancet’s Iraq Study,” Slate, October 20, 2006, http://www.slate.com/id/2151926/; Jeffrey White and Loring White, “Death in Iraq: A Critical Examination of the Lancet Paper,” PolicyWatch, No. 1155 (Washington, D.C.: Washington Institute, October 18, 2006), http://www.washingtoninstitute.org/templateC05.php?CID?2523; and “Correspondence: Mortality in Iraq,” Lancet, January 13, 2007, pp. 101–104.
18. The UNDP-funded study surveyed 21,688 households and estimated that there were 24,000 Iraqi war-related fatalities (both military and civilian) through mid-2004. Central Organization for Statistics and Information and Fafo Institute of Applied International Studies, Iraq Living Conditions Survey, 2004, Vol. 2: Analytical Report (Baghdad: Ministry of Planning and Development Cooperation, 2005), p. 54. Although the UNDP estimate is 1.6 times higher than the IBC estimate during the comparable time period (see Hami Dardagan, John Sloboda, and Josh Dougherty, “Speculation Is No Substitute: A Defence of Iraq Body Count,” April 2006, http://reports.iraqbodycount.org/d/a_defence_of_ibc.pdf, Table 3), once the estimated 9,200 Iraqi military casualties from the major combat phase identified by the Project on Defense Alternatives are subtracted (Conetta, “The Wages of War”), the difference nearly disappears.
19. Quoted in Jefferson Morley, “Iraq’s Civilian Death Toll ‘Horrible’—Or Worse?” Washington Post, World Opinion Roundup, October 19, 2006, http://blog.washingtonpost.com/worldopinionroundup/2006/10/is_iraqs_civilian_death_toll_h.html.

Apologies for the poor copy-and-paste job. See the actual article for a nicer version.

If Kahl doesn't trust the Lancet numbers, then why do you?

Not in My Journal

One of the more dramatic moments in the JSM session devoted to the Lancet surveys occurred during the question and answer session. (When AP reporter Paul Foy described the "courtroom-style questioning," I am pretty sure that he was, at least in part, referring to this incident.) This was actually the first question and it went, as luck would have it, to me. I asked something like:

As many audience members know, the Ethical Guidelines of the American Statistical Association require us to share data with other scientists for purposes of replication. Although you and your co-authors have shared some of your data with some other researchers, you have refused to share data with Michael Spagat and his co-authors. Why should we give credence to what you have said today if you refuse to adhere to the ethical guidelines of the ASA?

Comments:

1) No doubt my actual questions included more "uh's" and "you know's." It was certainly not phrased as nicely. But my meaning got across. Roberts and the other panelists addressed my point directly.

2) The Ethical Guidelines for Statistical Practice for the American Statistical Association require that statisticians:

Share data used in published studies to aid peer review and replication, but exercise due caution to protect proprietary and confidential data, including all data which might inappropriately reveal respondent identities.

Now, it is not clear that any of the Lancet authors are even members of the ASA, so these guidelines may not even apply to them. Roberts on occasion tries to claim that he is protecting interview/interviewee safety by not releasing the data. But he can't get away with that dodge for L2 since that data has been shared. There is just no reason to not share the data with critics like Spagat et al when you have already shared the data with critics like me.

3) The audience, of course, does not read the details of the ethical guidelines or follow the Lancet debate closely. But they certainly found it absurd that a scientist would share his data with some critics but not others. The reason that the Q&A went in the direction of "courtroom-style questioning" was the skepticism caused by Roberts et al behavior. I have yet to find, after a year of looking, another example where the data behind a published study was shared with some academic critics but not others.

What happened next? Roberts gave his usual shpiel, which mostly focuses in what a waste of time it is to have this discussion, to even talk about an already published study. Instead, critics should be getting new data, going to Iraq and so on. Best part was when he said that "If it were up to me, we would not have released the data to anyone." I was not the only member of the audience to be shocked by that admission. But it is consistent with what I can glean of the internal dynamics of the Lancet team. Roberts would really prefer to not share anything with anyone, but cooler heads (like Burnham, I think) occasionally prevail.

The other participants than chimed in. They all agreed that scientists should share their data. They all urged Roberts to do so. (All the subsequent questions from the floor that addresses this point seemed to agree with this position.) But the best comment came from Stephen Fienberg, the chair of the session and a professor at Carnegie Mellon.

That's just the wrong response. I, as an editor, would not publish a study for which the data was not shared.

Unfortunately, the editor of the Lancet feels differently. I wonder why.

AP Story on JSM Meeting

Here is the AP story by Paul Foy on the JSM session about the Lancet surveys.

The courtroom-style questioning came in a packed ballroom at the world's largest gathering of statisticians.

On the hot seat: a globe-trotting researcher who says his team's surveys of Iraqi households projected nearly 655,000 had died in the war as of July 2006, a number still ten times higher than conventional estimates.

Leslie F. Roberts and others from John Hopkins University took accounts of births and deaths in some Iraqi households to estimate that the country's death rate had more than doubled after the 2003 invasion.

Number crunchers this week quibbled with Roberts's survey methods and blasted his refusal to release all his raw data for scrutiny -- or any data to his worst critics. Some discounted him as an advocate for world peace, although none could find a major flaw in his surveys or analysis.

"Most of the criticism I heard was carping," said Stephen Fienberg, a professor of statistics at Carnegie Mellon University in Pittsburgh. "I thought the surveys were pretty good for non-statisticians."

Roberts, an epidemiologist, said he's opening a new front in the study of public health hazards: war. He has conducted about 30 mortality studies since 1990 in conflicts around the globe, including the Congo, where he was similarly accused of exaggerating war-related deaths.

Roberts organized two surveys of mortality in Iraqi households that were published last October in Britain's premier medical journal, The Lancet. He acknowledged that the timing was meant to influence midterm U.S. elections.

"It puts you in a position where you are going to get attacked," said Fritz Scheuren, a senior fellow at the University of Chicago's National Opinion Research Center, who is trying to organize another Iraqi survey to see if he can match Roberts's results.

Scheuren, the American Statistical Association's former president, said he couldn't find anything wrong with The Lancet surveys.

He complained, however, that he wasn't able to get Roberts to reveal which of his Iraqi surveyors conducted which surveys, information that could reveal any bias in workers who compile consistently implausible results.

Roberts, now a lecturer at Columbia University, said he won't release the researchers' identities for fear of exposing them to death or retaliation. The new Iraqi government has strongly disputed the findings.

The estimate was not meant to be precise, but practically every health statistic in America comes from similar sampling techniques, Roberts and his team said.

Scheuren said it's OK for a scientist to be an advocate for peace but Roberts's work deserves more scrutiny because of it.

"It happens that war is political," Roberts responded Wednesday after his grilling at a convention of more than 5,000 statisticians. "So I think those criticisms are valid. I am swimming in a much more political world that these statisticians."

In The Lancet article, Roberts and other researchers produced eight pages of text, maps, charts and many figures, but the number was the show-stopper: 654,965 war-related deaths, from the 2003 invasion of Iraq to last summer.

The estimate covered everything from battlefield casualties to civilians dying for lack of routine medical care. President Bush last fall said the estimate wasn't credible.

"Do I believe 650,000? No," Fienberg said. "Do I believe a lot of people are being killed? Yes."

Roberts and the statisticians agree that estimates of as few as 66,000 war-related Iraqi civilian deaths are based on sporadic media reports or incomplete accounts from hospitals and morgues.

Other Iraqis end up in mass graves or are promptly and randomly buried by relatives under Muslim law, Roberts said.

He said he personally supervised a 2004 survey of mortality in Iraqi households, slipping into the country from Jordan on the floor of a sport utility vehicle with $20,000 hidden in his shoes. It was too dangerous for him to return in 2006, but he said he had a trained team of Iraqi doctors in white coats ready for that door-to-door survey.

In all, the researchers questioned 2,850 randomly selected households, and Roberts said both surveys confirmed a soaring death rate after the invasion. Their short questionnaire sought to find out how many family members were born or had died since Jan. 1, 2002, establishing a baseline for a post-invasion death rate.

"Verifying these numbers wouldn't be too difficult, so why isn't it being done?" asked Jana Asher, a graduate student at Carnegie Mellon University, who dissected Roberts's surveys and found little fault with them.

Has Roberts denied the claim that the timing of the report was "meant to influence midterm U.S. elections."? Not that I have seen. See some discussion on Deltoid.

Bias Presentation

This presentation (pdf) by Michael Spagat (Royal Holloway College, University of London) is good stuff, giving a more global view of the Main Street Bias issue.