Another False Statement

My general take (as here) is that Les Roberts is much more likely to say something false than Gilbert Burnham is. Perhaps I need to rethink this assumption. Consider Burnham's comment about/to the Wall Street Journal two years ago.

Mr. Moore did not question our methodology, but rather the number of clusters we used to develop a representative sample. Our study used 47 randomly selected clusters of 40 households each. In his critique, Mr. Moore did not note that our survey sample included 12,801 people living in 47 clusters, which is the equivalent to a survey of 3,700 randomly selected individuals. As a comparison, a 3,700-person survey is nearly 3 times larger than the average U.S. political survey that reports a margin of error of +/-3%.

By what law of political arithmetic is a survey with 12,801 people living in 47 clusters "equivalent" to a survey of 3,700 randomly selected individuals? It all depends on the design effect, on how clustered the response of interest is. And you can't know how large the design effect is until you do the survey. Again, I think that some of Moore's critique was weak, but this is confusing at best.

In addition, Mr. Moore claimed that the Hopkins study did not include any demographic data. The survey did collect demographic data, such as age and sex, related to violence, although they are not the same details Mr. Moore’s company would have collected for public opinion polls. The characteristics of households in our study are similar to other accounts of households in Iraq and the region, though the household size for the 2006 study is smaller (6.9) than found in the 2004 survey (7.9).

The debate over demographic data is key. Moore did make that claim. Roberts then responded that the survey did collect that data, as everyone would have expected it to. This made Moore look like an idiot. See here for relevant links. It was only 6 months later, once the data had been made available to me and others, that it became clear that little meaningful demographic data had been collected. But, by that time, most people had forgotten the debate. But not we Lancet fanatics!

This debate came up on Deltoid here. Note my painstaking efforts to force Lambert to print a correction after he slandered Neil Munro. One of the reasons, probably, that Lambert would be in such a rush to attack Munro on this is because Burnham and others continue to mislead on this point. Note the weasel wording:

The survey did collect demographic data, such as age and sex, related to violence, although they are not the same details Mr. Moore’s company would have collected for public opinion polls.

The survey collected age and sex for deaths, whether or not they were "related to violence." So, if someone died from a heart attack before the war started, his age and sex were recorded. But no age information was collected for the individual residents of each household. This means that it is impossible to know whether or not the sample matches up with other information about the population structure of Iraq. Although lots of information might be included under the heading "demographic," the minimum details are age and sex. If you are not collecting age/sex information, you do not get to claim that you are collecting "demographic" data.

Now, Burnham's sentence above is so convoluted that it is tough to be certain that it is false. But the next one is clearly untrue.

The characteristics of households in our study are similar to other accounts of households in Iraq and the region

That's just false. They did not collect ages, so they have no idea if the "characteristics of households" match "other accounts." Burnham is just making stuff up, allowing the reader to believe that they collected age/sex information (and that it matches other surveys) when, in fact, they did no such thing.

Again, it might be possible for a lawyer to argue (unlike in our last case) that nothing in Burnham's letter is literally false, but, put together, his statements are completely misleading. He should be embarrassed.

UPDATE: One more item. Tim Lambert (because he is a serious guy who gets the details correct) took the time to post a correction to this item from 2006. (It would be nice if Tim gave me a little credit and if he dated these corrections. The subsequent set of comments make a lot more sense if you know that his correction was added in 2008.) But the best part is that, even in 2008, Roberts does not know what went on!

I was wrong! Shannon cleaned and analyzed the data. I never saw the raw forms. We collected age and gender on everyone in 2004. That was the plan in 2006. My understanding is that they did this for some houses in the start but as that was the most lengthy part of the interview they just started recording how many people were in the house and the age and gender of the dead.

Nice story. But that's not what they did! Or at least is not what they claimed to have done. The data that has been distributed does list the number of males and females in each house but not their ages. But Roberts claims that they just recorded number of household residents. Well, which is it? Did they just make up the genders of household residents or did they collect that data? If I were smart, I would know a quick way to test to see if they were making stuff up. If they were, you would probably see too many houses with similar numbers of men and women and not enough extreme households of all men or all women. (Of course, this assumes a certain default model of household formation.)

Another item to add to the long list of things to investigate . . .