Pope of Debunkers

Robert Shone's blog, Dissident 93, covers a variety of Lancet topics. Here is a good summary of why so much of the criticism of the Main Street Bias work is incoherent. Via his blog is also this useful Science article.

www.sciencemag.org SCIENCE VOL 319 18 JANUARY 2008 273
CREDIT: ADAPTED FROM C. A. BROWNSTEIN ET AL., NEJM 358, 5 (2008), MASSACHUSETTS MEDICAL SOCIETY
NEWS OF THE WEEK
A team led by the World Health Organization
(WHO) has produced a new estimate of
the number of Iraqis who died violently in
the first 40 months following the U.S.–led
invasion: between 104,000 and 223,000.
This figure, published online last week by
the New England Journal of Medicine, hews
close to some other attempts to quantify the
toll but comes in far below a controversial
2006 study led by researchers at Johns Hopkins
University in Baltimore, Maryland.
That group estimated approximately
600,000 violent deaths during the same
period. The discrepancy has prompted critics
to renew their charge that the Johns Hopkins
results are not credible.
Data from a war zone are never fully reliable;
the best researchers can hope for is
“getting the numbers roughly right,” says
Fritz Scheuren, a statistician at the University
of Chicago in Illinois and past president
of the American Statistical Association.
Escalating violence in Iraq after 2003 put a
limit on quality control, but researchers do
have a quantitative starting point: the casualty
tally made by Iraq Body Count, a nonprofit
advocacy group based in London. By
controlling for multiple accounts of the same
car bombs and shootings, the group estimates
from media reports that between
81,000 and 88,000 violent deaths have
occurred in Iraq since the invasion. The figure
is useful as “a lower bound on the true
number,” says Jon Pedersen, a statistician at
the Fafo Institute for Applied International
Studies in Oslo, Norway.
To get the upper bound, says Pedersen,
you have to knock on doors in what is known
as a two-stage cluster survey. That’s the
method used by the WHO and Johns Hopkins
teams, among others. Researchers divide the
country into regions and then sample clusters
of households within each. Finally, they
extrapolate mortality rates from those clusters
to the total population.
Epidemiologists Les Roberts and Gilbert
Burnham of Johns Hopkins published the
first Iraq cluster study in November 2004 in
The Lancet. They used data collected by
Roberts and an Iraqi team, which, in September
2004, surveyed 988 households in
33 clusters across the country. They arrived
at a figure of 98,000 “extra” deaths since the
invasion, about half due to violence. Soon
after this, a team led by Pedersen and the
United Nations Development Programme,
which had used a much larger sample of
21,668 households in 2200 clusters, produced
an estimate for roughly the same
period of about 25,000 violent deaths.
As the invasion gave way to occupation
and insurgency, Rober ts and Burnham
mounted another study. This time they left
the surveying entirely to the Iraqi team,
communicating from abroad. Published in
October 2006 in The Lancet, the second
survey—based on 1849 households in
47 clusters—estimated that 601,000 Iraqis
died violent deaths between the 2003 invasion
and July 2006. To many, the number
seemed unrealistically high. Some also
faulted the authors for not fully answering
questions about the survey’s methods
(Science, 20 October 2006, p. 396).
Now comes the WHO survey. Conducted
with the help of the Iraqi government, it is by
far the most comprehensive mor tality
assessment to date. Interviewers visited
9345 homes in more than 1000 clusters. But
its estimate of 151,000 violent deaths has
come in for some criticism, too. Unlike other
Iraq casualty surveys, this one includes an
upward adjustment of 35% to account for
“underreporting” of deaths due to migration,
memory lapse, and dishonesty. “That is
really an arbitrary fudge factor,” says
Debarati Guha-Sapir, an epidemiologist at
the WHO Collaborating Centre for Research
on the Epidemiology of Disasters in Brussels,
Belgium. But the number falls squarely
within the range produced by a meta-analysis
of al l avai lable mor tal i ty studies by
Guha-Sapir and fellow centre epidemiologist
Olivier Degomme. The Johns Hopkins
figure is an outlier, she says.
Why the Hopkins study came up with
such a high figure is not clear. Criticism of
the study has in fact intensified since Burnham
and Roberts released a data set to
selected peers last year. “It did
not include the standard kinds of
data,” says Seppo Laaksonen, a
statistician at the University of
Helsinki in Finland and a specialist
in survey methodology.
For example, he says, it was
impossible “to check the objectivity
and randomness of cluster
selection.” Scheuren, who also
received the data, wanted to
compare results obtained by different
interviewers to “get a handle
on noise” and check for fabrication
by surveyors. Rober ts
declined to provide all the
details, according to Scheuren,
saying that he was concerned
that this would risk the safety of
the interviewers.
Burnham told Science, however,
that the Johns Hopkins
team does not have such detailed
information. “Our goal was to
reduce any type of risk to the community
and the par ticipants,” says Burnham.
“While we have much of the raw data, we
requested that anything designating the
interviewers or the location of the neighborhoods
visited not be sent to us.” Laaksonen
responds that he would not have published
“any f igures for the country” if he didn’t
have direct access to such raw information
from surveyors.
Burnham is not retreating. Because the
WHO survey was conducted by Iraqi government
personnel, “people may have been
hesitant to answer honestly,” he says. He
claims that unlike those in the WHO study,
nearly all of the deaths tallied by the 2006
Lancet study were verified with death certificates.
Even if the debate may be drawing
to a close about whether the number of violent
deaths in postinvasion Iraq could be as
high as 600,000, the argument about methods
is clearly far from settled.
–JOHN BOHANNON
Downloaded from www.sciencemag.org on January 17, 2008