Said and Unsaid

There is much fun discussion in the Lancetosphere about the IFHS study. Lancet defenders are doing everything they can to insinuate that the IFHS results support or validate or, at least, are not inconsistent with L1 and L2. And, to some degree, they are reasonable to do so. For example, both L1 and L2 report that Iraq has 18 governorates. IFHS agrees!

Perhaps that is a bit snarky, but it is important to distinguish between what the authors of a study actually say and what you conclude about the studies data, models and results. For example, the IFHS authors say nothing about L1. They do not even cite it among their references. Perhaps they think that L1 is a great paper and their results support it 100%. Perhaps they think that L1 is completely wrong and should be retracted by the Lancet. They don't say and so we don't know.

But the IFHS authors do make very specific claims about L2. From the abstract:

When underreporting was taken into account, the rate of violence-related death was estimated to be 1.67 (95% uncertainty range, 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range, 104,000 to 223,000) from March 2003 through June 2006.

Conclusions: Violence is a leading cause of death for Iraqi adults and was the main cause of death in men between the ages of 15 and 59 years during the first 3 years after the 2003 invasion. Although the estimated range is substantially lower than a recent survey-based estimate, it nonetheless points to a massive death toll, only one of the many health and human consequences of an ongoing humanitarian crisis.

In other words, the number of violent deaths estimated by L2 is too high. They are making a claim that a specific number from L2, the 600,000 violent deaths, is an overestimate. (Note that "violent deaths" (IFHS) and "excess violent deaths" (L2) refer to almost exactly the same underlying number because there were almost no violent deaths before the war.)

To mention in the abstract that a different peer-reviewed article is wrong is a strong statement of the beliefs of the IFHS authors.

The most striking difference in rates of death was between those in the study by Burnham et al. and those in the two other data sources for the six high-mortality provinces, which accounted for 64% of all deaths in the study by Burnham et al.

What is most surprising, to me, is how critical the IFHS authors are of L2 (Burnham et al), how they go out of their way to imply that L2 is wrong. Note especially Figure 1. If your main point was the L2 was wrong, I do not think that you could have constructed this figure in a more accusatory fashion.

Of course, just because the IFHS authors think that L2 is wrong does not mean that it is. But Lancet supporters should not pretend that the IFHS authors think that L2 (or L1) is correct about anything when they provide almost nothing but criticisms of L2. Now, that criticism is couched in the polite language of an academic paper, but, given the constraints of acceptable dialog in the New England Journal of Medicine, could the IFHS authors written anything more critical of L2? I don't see how.

There was greater agreement regarding mortality from nonviolent causes between the IFHS study (372 deaths per day) and the study by Burnham et al. (416 deaths per day)

This is one of the few (only?) places were IFHS offers support to L2. But this support is more damning than helpful because none of the serious critics of L2 have complained about their estimates of post-invasion non-violent mortality. After all, the increase that L2 found (50,000) was not even statistically significant. As Michael Spagat wrote:

The main problem with the comparison highlighted by the L2 authors [in asserting that L2 validates L1] is that it is of all excess deaths, not just violent deaths. All suggestions of possible bias in L2 that we know of, sampling or non-sampling, pertain to violent deaths. The available facts simply do not support a claim that L1 and L2 suggest very similar numbers of violent deaths. By persistently conflating non-violent deaths with violent deaths the L2 authors have obscured this essential point.

Correct. To say that the L2 estimates of non-violent death (not "excess non-violent death") are reasonable is to congratulate them on the fact that they got the number of governorates in Iraq correct. It is true, but faint praise.

The most striking difference in rates of death was between those in the study by Burnham et al. and those in the two other data sources for the six high-mortality provinces, which accounted for 64% of all deaths in the study by Burnham et al.

People like me think that the reason for this is that the L2 data from those provinces is fraudulent, either made up out of whole cloth or derived from a corrupted sampling scheme. One of my guesses is:

The L2 interviewers, for whatever reason, decide that they want to report more deaths. They go to a cluster and, following the procedure outlined in Burnham's discussion, gather all the neighborhood kids around and tell them about the survey. So far, so good. But then they ask the kids, "Has anyone in the neighborhood died in the last couple years, especially violently?" The kids know this and tell them. Then the interviewers preferentially select those houses, either a picking and choosing around the neighborhood or just placing the 40 house cluster in the part of the neighborhood that, by chance, had the most deaths. In that scenario, all the data is "accurate" in the sense that no one is making anything up, but the mortality estimate will be much too high.

Access to the demographic data might allow us to catch that because the houses would have way more young men than a random sample should have, given what we know about the age/sex distribution in Iraq from sources like ILCS.

Now, of course, I wasn't in Iraq for the surveys. But a corrupted interview process would help explain why the L2 results in these provinces are so wildly divergent from those of IFHS and IBC. Note also that this could have occurred without the knowledge of any of the L2 authors. Only Lafta was in Iraq for the survey and, since there were two teams operating independently, the fraud (if that is the cause) might have happened without his knowledge. This is why people like Fritz Scheuren want to see the underlying data classified by interviewer. This is why everyone would have liked to see demographic information. Remind me again about why Les Roberts lied to Tim Lambert about the whether or not age data was collected for individual households.

Back to IFHS. Table 4 is another example of the IFHS authors arguing, politely, that the L2 results are completely implausible. Could any critic of L2 constructed this table in a more damning fashion? Not that I can see.

And the final statement comes in the Discussion.

The IFHS results for trends and distribution of deaths according to province are consistent with what has been reported from the scanning of press reports for civilian casualties through the Iraq Body Count project. The estimated number of deaths in the IFHS is about three times as high as that reported by the Iraq Body Count. Both sources indicate that the 2006 study by Burnham et al. considerably overestimated the number of violent deaths. For instance, to reach the 925 violent deaths per day reported by Burnham et al. for June 2005 through June 2006, as many as 87% of violent deaths would have been missed in the IFHS and more than 90% in the Iraq Body Count. This level of underreporting is highly improbable, given the internal and external consistency of the data and the much larger sample size and quality-control measures taken in the implementation of the IFHS.

"Highly improbable" is New England Journal of Medicinese for "total crap."

And this is where things get fun. Lancet redoubts like Crooked Timber and Deltoid are filled with hopeful attempts to rescue L2 from IFHS, to argue that, by some excess death calculation that the IFHS authors do not use, IFHS supports L2. Perhaps. All of us spend time trying to use the raw data and models of a given paper to answer questions that the papers authors fail to address. Who knows? Perhaps the next paper from the IFHS team will be an excess death calculation that matches perfectly with L1 and L2. Hah! Anyone with experience reading scientific papers knows that the IFHS paper could not have been more critical of L2. Future work from these authors will almost certainly continue in the same vein. Anyone want to bet otherwise?

There is no way to spin the IFHS paper as anything less than a total rejection of the L2 violent death estimates. That doesn't mean that the IFHS authors are right, but we need to be clear on what they say.