Roberts Comments on ILCS and Politics

Les Roberts dishes some dirt against ILCS in this interview, starting at around 2:50. Calls Jon Perdersen a "great researcher out of Norway." One question on death with a couple of follow-on questions if a death had occurred. Survey was conducted by "Iraq governmental employees of Saddam's era." (Roberts employees that such people do not do high quality work. But were all his interviewers doctors (i.e., working for the government before the war) similar situated. Notes Pedersen's decision to re-sample on that question after the death rate came in too low. Roberts implies that, even with this correction, the estimated death rate is too low from UNDP/ILCS.

I haven't followed this dispute closely enough to comment. Roberts certainly believes that the ILCS results are consistent with his. Best part was the dialog on political views at around 5:50. Interview mentions an assumption about Roberts' politics, a concern that they did the study because they are opposed to the US/allied presence in Iraq.

Roberts: Well, that's just not true.

Interviewer: (incredulous) You're not opposed to the US presence in Iraq.

Roberts: Actually, the majority of the first authors I know were in favor of the invasion of Iraq.

Really? That would be deeply surprising to me. Burnham, at his MIT talk certainly gives the impression that he was opposed to the invasion (although I don't recall him addressing that point). Everything that I have read about Roberts suggests that he was anti-war from the start (although I can't find a good link right now). If those two were anti-war, then all three of the remaining authors (Riyadh Lafta, Richard Garfield, Jamal Khudhairi) would have to be pro-invasion.

Anyway, Roberts then goes on a riff about how public researchers are always "against" the thing that they study. No one who studies measles is pro-measles. Roberts is against death. "Full stop." Roberts says "A lot of people have died here and that is completely unacceptable."

Roberts does a great mocking of a US southern accent at the very end. Not nice!

Very, Very Wide

Very thorough interview of Les Roberts on a (British?) news show. Highly recommended. Key part for my concern about confidence intervals comes in Roberts discussion of the 2004 study at around 3:50.

There was one place, the city of Falluja which had just been devasted by shelling and bombing. It was so far out of whack with all the other [clusters] that it just made our confidence intervals very, very wide.

Unfortunately, the interviewer quickly moves on to other issues. Compare that statement with Burnham's.

Now this is what the confidence intervals would look like. There is a 10% probability that it was less than 44,000 and only a 2.5% chance that it was less than 8,000. If we put Falluja into it, the top end of the confidence interval would be infinity. It really skewed things so badly that we decided that we should just leave it out and be conservative.

Seems fairly obvious to me that, if Falluja is included in Lancet I, the null hypothesis of no increase in mortality can not be rejected. But that's an empirical question. If the authors would release the data for Lancet I, we can figure this out.

More Questions

I continue to have a cordial correspondence with the Lancet authors. Special thanks to Gilbert Burnham for answering my questions. For those who care, my latest set of questions was as follows.

Hope all is well and thank you for sharing some of the data from Burnham et al (2006) with me.

1) Is data from Roberts et al (2004) available as well? It is a great idea (as you do the the 2006 article) to try and match up data from the two separate studies. I would like to do the same.

2) Is a copy of the interview form available, either in English or Arabic? This may have been posted at some point but I am unable to find a copy. I would like to get to the bottom of the whole not-enough-time-to-do-the-survey dispute (which I don't attach a lot of credence to) by examining just how long the survey is.

3) It is my understanding that none of the interviewers spoke Kurdish. Is that correct?

4) Is a copy of the studies precise methodology available? I am looking for whatever written material were given to the interviewers which specified the exact procedures to be used in selecting clusters, dealing with security issues and the like. It is impossible to evaluate the merit (if any) of the main street bias critique without understanding the steps used by the interviewer teams.

5) In his talk at MIT in February, Dr. Burnham stated (with regard to Roberts et al (2004):

"Now this is what the confidence intervals would look like. There is a 10% probability that it was less than 44,000 and only a 2.5% chance that it was less than 8,000. If we put Falluja into it, the top end of the confidence interval would be infinity. It really skewed things so badly that we decided that we should just leave it out and be conservative."

Is that an accurate statement about the effect of including the Falluja data?

As always, thanks for your time.

Dave Kane

I won't be printing any replies without the express permission of the senders, of course. If any readers know the answers (surely the survey form has been made public somewhere?), please contact me.

Soldz Article

One of the most fun parts of participating in this debate is getting to meet some of the people involved. I had an enjoyable lunch with Stephen Soldz last week. His blog has all sorts of useful links, and his article on Lancet I is a fine example of a fair and balanced analysis.

If your a fan of the controversy surrounding the Lancet articles (either pro or con) and you find yourself near Cambridge, MA, please let me know. Lunch is on me!

Data!

The authors have made a version of the data for Lancet II available to me. Thanks! To be honest, I had worried that they wouldn't do so. I was wrong to worry. I appreciate the authors sharing this data with me (and others). I also thank them (in advance!) for answering the questions that I and others will have.

A central value of any good scientific community is transparency of methodology. If you want to claim that X is true, you need to explain in detail why X is true and demonstrate how other scientists, especially those who disagree with you, can replicate your results. The Lancet authors have taken the first step along this road. Good for them.

Lafta Visit

Via Tim Lambert, we have more details of Riyadh Lafta's upcoming visit to Canada. (Lafta was the key Iraqi doctor for both Lancet I and II.) I think that he organized and supervised all the survey work.

A highly regarded Iraqi epidemiologist who wants to tell Americans about an alarming rise in cancer levels among Iraqi children will come to Canada instead because he couldn't get a visa to the United States.

Does Lafta have a degree in epidemiology or statistics or public health? Just curious. Lack of a degree does not imply incompetence, but if the only two surveys he has conducted have been for these articles, I am not sure that I would describe him as an "epidemiologist."

Once in Canada, Dr. Lafta will present estimates that paint a damning portrait of the war's ravages on children: that birth defects are on the rise since the war began, and that the number of children dying from cancers such as leukemia has risen tenfold.

Huh? There is no doubt that car bombs are bad for children, but what possible war-related mechanism could there be for a 10-fold increase in leukemia? Also, the Iraqi government can't keep track of death rates but it has accurate statistics on childhood leukemia? That seems, uh, implausible.

Dr. Lafta was born in Baghdad in 1960, was trained as a physician at Baghdad University College and then worked for 14 years for the Ministry of Health under Saddam Hussein. He became the head of the communicable disease department and then the primary-care department of Diyala province in northern Iraq.

It would be nice to have more details on this time-line. Did Lafta's 14 years at MoH include his time at Diyala? Was Lafta at MoH until the invasion in 2003? One way to make the dates work would be for Rafta to finish his training in 1989 (at age 29) and then spend then next 14 years until 2003 at MoH. 29 would certainly be a standard age for finishing training in the US (but no doubt Iraqi training schedules are different).

In order to have a sense of how qualified Rafta was to organize these surveys, we need to have a better sense of his background. If he did a lot of such surveys during his 14 years at MoH (either nationwide or just in Diyala), then he would be an excellent choice. If did no prior surveys, then he still might have done a great job for the Lancet team, but we need more details. How did the Lancet authors and Rafta meet each other?

Also, I guess that I am safe in assuming that Lafta is Sunni. Isn't it also the case that most/all higher ranking officials in the pre-war government (whether at MoH or elsewhere) were members of the Baath party? Now, obviously, just because one is a Sunni and/or a former member of the Baath party does not that one's professional work is suspect. But, in any survey situation, it is a concern if the interviewers are too "different" from the interviewees. (I have already discussed the issue of (the lack of?) Kurdish speakers among the interviewers.)

Consider the problem that Shia/Kurdish households might, not so much lie in order to deceive when confronted with Sunni interviewers as give those interviewers that answers that they seemed to "want." If the interviewers seemed to want to find some post-war deaths, then give them so post war deaths.

The analogous situation in the US in 1965 might be white interviewers asking black households in, say, Mississippi about police brutality. A black interviewee might well decide that the better part of valor was giving the white interviewer the answer that the interviewee thinks the interviewer wants. Tell the nice white lady that there are no problems with police brutality (even though there are). In this scenario, there is no subterfuge or fraud on the part of the interviewer. She is honestly trying to find the correct answer, honestly writes down what the interviewee reports.

I do not think that this is a major concern but it would be nice to know if all the interviewers were Sunni. My guess is that they are. (How many Shia became English-fluent doctors in Saddam's Iraq?) Again, this does not mean that they are lying, but it does suggest that we should look for patterns in the responses that the y recorded.

Infinity

As noted previously, Gilbert Burnham's presentation at MIT featured information about the results for Lancet I. Burnham is describing their estimate of excess deaths due to the war. Their estimate was 98,000 if the data from Falluja is ignored. The estimated death rate in Falluja was much higher than elsewhere so removing Falluja was described as "conservative," by both the authors and other commentators. Recall the findings from the paper (pdf) itself.

The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion. We estimate that 98 000 more deaths than expected (8000–194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.

Any empirical researcher is vaguely suspicious of results which just barely reject the null hypothesis. That 8,000 figure is awfully close to zero. Would other reasonable parameterizations generate the same result? Perhaps. But, given that this excluded Falluja, there is nothing to worry about, right? The war must have increased the mortality rate.

But note how Burnham described things (quote starts around 19:30) during the talk.

We got a huge amount of criticisms for these confidence intervals, and I'll come to this confidence interval in just a bit. But we had a confidence interval at the low end of 8,000 and at the high end of 194,000.

...

Now this is what the confidence intervals would look like. There is a 10% probability that it was less than 44,000 and only a 2.5% chance that it was less than 8,000. If we put Falluja into it, the top end of the confidence interval would be infinity. It really skewed things so badly that we decided that we should just leave it out and be conservative.

Huh? He can't possibly mean that the top end would be infinity. I would think (albeit, not as an expert on confidence intervals in cluster sampling) that a top end of infinity would imply a bottom end of the confidence interval at infinity as well. Wouldn't it have to? Note that excess deaths is a real valued variable. If it were bounded by zero, then, of course, you might have a confidence interval which was bounded below but not above.

For real valued variables, I do not think that I have ever seen an applied situation with actual data in which the confidence interval was infinity above but not below. Has anyone else? Almost every confidence interval of a real valued variable is symmetric. If it is infinite at the top, then it is infinite at the bottom.

Also, the fact that it is infinite at the top suggests that this was not a bootstrap confidence interval. I think that it is very hard (impossible?) to get infinity for bootstrap confidence intervals. A bootstrap always gives you back something. That would suggest that they were using an analytic calculation of some kind, one that failed to converge.

And that wouldn't be surprising. If the Falluja data is enough of an outlier, then the software has trouble calculating a confidence interval. It spits back infinity of NA or NaN or whatever.

Assume for now that the confidence interval was infinity on both sides. This would mean that only by ignoring the Falluja data were the authors able to reject the null hypothesis of zero excess deaths due to the war. It would be highly misleading to report a rejection of the null hypothesis which required throwing out some of the data without at least telling your readers how fragile this result is.

Perhaps Burnham just meant "really big" instead of "infinity." But, if so, then the same large range was probably present at the lower limits as well. That is, even if the upper confidence interval was just some huge number and not literally infinity, that would suggest that the lower bound was similarly large and, probably, covered zero. (This might depend on how much higher the mean estimate was, of course.) In either case, if getting a statistically significant result requires throwing out some of the data --- even though you have no reason for thinking that data is less accurate than other collected data --- you must report this fact to your readers.

Yet all of this discussion is in direct contradiction to the paper. The authors claim to have calculated a confidence interval (for the increase in risk) include the Falluja data.

The risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared with the preinvasion period.

A 4.2 upper confidence level for the relative risk is neither infinite nor, I think, particularly high. Might not a raging war make things 4 times more dangerous in a country? Why would Burnham claim that the upper confidence limit is infinite while the paper reports a reasonable number? Is there some reason why one might have a reasonable upper limit for the relative risk but not for the total numbers of deaths? Not that I can think of.

No doubt I am just missing something obvious. Clarifications would be welcome.

ESOMAR Podcast

Great podcast from October 2006 with many of the key players involved in Iraqi polls. Matthew Warshaw of D3 Systems is one of the key people in the recent ABC poll as well as the World Public Opinion.com polls. Most of his discussion focuses on Afghanistan. Fascinating stuff on the knitty-gritty details.

The section on Iraq features Johnny Heald of Opinion Research Business and Munqeth Daghir of MEC Communications. Daghir's story is interesting and moving. He had no experience in polling but read a book (by "Dr. Gallup") just after the invasion and decided to get involved in polling, creating a company from scratch. Heald has been polling with Daghir for two years (since 2004, I guess). Daghir's first poll was in Baghdad, three weeks after the invasion. Daghir's had interesting comments on the "culture" of fear surrounding interviews, how people don't want to reveal their opinions, are afraid of the militias and so on.

Daghir discusses how different polling for him is now (2006) compared to when he started (2003). At the start, he just had one book and no experience. Now, he has gotten lots of help from ESOMAR (which I think is an international organization of market researchers) and research experts from around the world, people like Heald. Now, his polls are "according to international standards", before, not so much. The 19th minute is the key section. There has been a "great change in the methodology." Now, Daghir's polls are "totally different." Daghir discussed having a lot of trouble selling the initial polls.

Pedersen Quote

A reader writes in with this sourcing for the Pedersen quote.

No one disputes that Iraq has grown much more deadly. The question is how much.

A 2004 study by the same authors of the Lancet article estimated 98,000 violent deaths in the first 18 months after the invasion, a figure four times higher than the findings of a much larger survey done at approximately the same time by Norweigan researchers working for the United Nations. That study, the Iraqi Living Conditions Survey, estimated 23,743 civilian deaths in the first 13 months of the conflict.

In a telephone interview, Jon Pedersen, research director for the 2004 study, said several factors probably account for researchers' different findings.

One key issue is how researchers extrapolate from the deaths identified in their field research to a death toll for the whole country. Pedersen noted that the Lancet study is based on a pre-invasion mortality rate of 5.5 deaths per thousand people. The U.N., he said, used the figure of 9 deaths per thousand. Extrapolating from the lower pre-invasion mortality rate would yield a greater increase in post-invasion deaths, he noted. If the higher pre-invasion mortality rate is more accurate, then the deaths attributable to the war would be lower.

Another difficulty, he said, is doing accurate research under dangerous conditions. Pedersen wondered how tightly the study's authors could oversee the interviews as they were conducted throughout Iraq.

The JHU study, he noted, asked Iraqis only about mortality. The U.N. study asked Iraqis about many aspects of their living conditions. Pedersen said his study probably underestimated deaths caused by the war because the interviews did not focus on the issue, while the Lancet article probably overstated them because no other subject was discussed.

Pedersen said he thinks the Lancet numbers are "high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much."

"Regardless of the numbers that are possible," he added, "we are seeing a situation that is pretty horrible."

Agreed. But, if Pedersen is correct, how did Lancet II come up with a number 6 times greater than the truth? Was there a problem in their statistical software? Did they make a mistake in some arcane cluster sampling calculation? Unlikely. The most likely source of error, as I have said for many months, is the raw data. Are all the supporters of Lancet II 100% convinced that Pedersen is wrong?

It seems to me that the truth is most likely to lie somewhere between Lancet II's estimate and Pedersen's.

Data Now Available

The data for Lancet II (but not, it appears, Lancet I) is now available.

Six months have passed since the publication of the study and we feel the time is now right to make the data set available to academic and other scientific groups whom we judge have the technical capacity to objectively analyze the data.

...

These data will be released on request to recognized academic institutions or scientific groups with biostatistical and epidemiological analytic capacity.

1. The data will be provided to organizations or groups without publicly stated views that would cause doubt about their objectivity in analyzing the data.

For someone like me who wants access to the data, the concern, of course, is that there is wiggle room in the definitions for "technical capacity" and "biostatistical and epidemiological analytic capacity." By any reasonable standard, IQSS and KCM (my firm) would qualify in terms of statistical sophistication, but who knows? I have submitted requests using each organization on the theory that if IQSS does not meet their criteria, then KCM will (or vice versa). My hope is to get this resolved in the next day or two so that I can revise my abstract accordingly. The deadline for doing so is April 11.

Data Availability

I sent the following letter to most of the Lancet authors.

Hi,

My name is David Kane and I am a fellow at IQSS at Harvard. I will be presenting a paper at JSM in Salt Lake City.

Questions:

1) One of my main goals in this paper is to avoid the mistakes of Hicks and Spagat/Johnson/Gourley by ensuring that every description of your work that I publish is *accurate*. Is there someone that I might direct questions to in order to achieve this? I am happy to e-mail all of you or just a subset or just one contact person. Whichever you prefer.

2) I could not find e-mail addresses for all the authors of both papers. Would you mind passing this message on to Riyadh Lafta and Jamal Khudhairi?

3) Can you give me an update on data availability? I know that there has been talk about making the data available to researchers like me but I have been unable to confirm this. Do you still expect to make that data available? If so, when? As you can see, my current abstract claims that the data is not available and I won't be able to edit that abstract after April 11. If you could make the data available to me (with all identifying information removed, of course) before April 11, then I could update my abstract appropriately.

4) It is fair to say that I am a critic of the estimates in your papers (I think that Jon Pedersen's are probably closer to the mark), but I consider myself a scientist first. My main goal here is the truth.


Thanks for your time,

Dave Kane

PS. I have cc'd several people that I have communicated with on this topic. Hope you don't mind.

The dig against Hicks/Spagat et. al. might have been a little harsh. Is it there fault that the written description of the Lancet methodology is not easy to parse? But I thought that politeness was the better part of valor in this case. My main goal is to get access to the data and to get at least one of the authors to answer my questions.

UPDATE: Edited to embed link to fix formatting.

ABC Poll Methodology

I haven't had time to review the new ABC poll closely, but their methodology description is commendably thorough.

This survey was conducted for ABC News, USA Today, the BBC and ARD German TV by D3 Systems of Vienna, Va., and KA Research Ltd. of Istanbul. Interviews were conducted in person, in Arabic or Kurdish, among a random national sample of 2,212 Iraqis aged 18 and up from Feb. 25-March 5, 2007.

Four hundred and fifty-eight sampling points were distributed proportionate to population size in each of Iraq's 18 provinces, then in each of the 102 districts within the provinces, then by simple random sampling among Iraq's nearly 11,000 villages or neighborhoods, with urban/rural stratification at each stage.

Maps or grids were used to select random starting points within each sampling point, with household selection by random interval and within-household selection by the "next-birthday" method. An average of five interviews were conducted per sampling point. Three of the 458 sampling points were inaccessible for security reasons and were substituted with randomly selected replacements.

Interviews were conducted by 103 trained Iraqi interviewers with 27 supervisors. Just over half of interviews were back checked by supervisors — 28 percent by direct observation, 14 percent by revisits and 10 percent by phone.

In addition to the national sample, oversamples were drawn in Anbar province, Sadr City, Basra city and Kirkuk city to allow for more reliable analysis in those areas. Population data came from 2005 estimates by the Iraq Ministry of Planning. The sample was weighted by sex, age, education, urban/rural status and population of province.

The survey had a contact rate of 90 percent and a cooperation rate of 62 percent for a net response rate of 56 percent. Including an estimated design effect of 1.51, the results have a margin of sampling error of 2.5 percentage points at the 95 percent confidence level.

Sounds thorough and professional. But then why is the contact rate only 90% (compared to Lancet II's 99%) and the cooperation rate only 62% (compare to Lancet II's 99%)? Isn't it a sign of incompetence on the part of D3/KA that there response rate is more than 40 percentage points lower than that of Lancet II (56% as compared to 98%)?

No, I think that this is evidence that the Lancet interviewers made up data, at least some of it. What other explanation could there be? The D3/KA interviewers were so rude that people didn't want to talk to them? The Lancet interviewers were so charming that virtually everyone was willing to answer their questions?

Other explanations are welcome.