Links Related to Moore

Here are some links related to the unimpressive Steven E. Moore Wall Street Journal op-ed.

Stats.org
Tim Lambert

The cluster comments made by Moore seem wrong (as demonstrated by Lambert and others. But this point is, at least, not unreasonable.

With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census.

Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.

And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it --- try using Google to find a survey that does not ask demographic questions.

I don't know of a survey that doesn't ask demographic questions, but I haven't really looked. Why would demographics be useful? First, it is (perhaps) a check against randomization mistakes and/or fraudulent data. If the demographics of the survey don't match the demographics of the population, then something is wrong. (But if the match isn't too far off, then the concern isn't that great.) Second, you could use demographics to adjust the survey results. If, say, Sunnis made up 50% of the survey but are, we think, at most 25% of the population, then you would want to adjust the results accordingly.

I wonder why the survey form did not include demographics. Unlike Moore, I doubt that the reason is nefarious. But it is endlessly annoying that the Lancet authors refuse to even release the questions or actual survey forms.

UPDATE: Lambert points out that plenty of demographic data (age and gender of household residents) was collected. Moore's response:

Despite Les' eloquent response, he has yet to reveal any comparison of demographic information for the 2006 survey to the 1997 Iraqi census, the 2003 update to that census, the 2004 UNDP/ILCS survey or any other demographic instrument.

Seems like Roberts/Lambert is correct on this one. It still drives me nuts that we don't have access to the raw data.