Tuesday, March 02, 2010

Dubious Polling

From the "2010 Top Ten "Dubious Polling" Awards" by Pollsters George Bishop and David Moore.


With this article, veteran pollsters, authors and political scientists George F. Bishop and David W. Moore issue their Second Annual Top Ten “Dubious Polling” Awards. These awards are intended to mark for posterity some of the most risible and outrageous pronouncements by polling organizations during the previous year.

...

“STONEWALLING/COVERUP” Award


WINNERS: The John Hopkins Bloomberg School of Public Health and one of its professors, Dr. Gilbert Burnham, for stonewalling in the face of serious questions about a flawed survey project, which reported more than 600,000 Iraqi deaths from 2003 to 2006. The head researcher was formally censured by the American Association for Public Opinion Research (AAPOR) for covering up his data collection efforts, but the Bloomberg School refuses to investigate the methodology. (Ah, the wisdom of the three monkeys: “See no evil, hear no evil, speak no evil!”).

BACKGROUND: In 2006, the British medical journal, The Lancet, published the results of a survey, designed and supervised by Dr. Gilbert Burnham of the John Hopkins Bloomberg School of Public Health, and his colleagues.* The survey purported to show that about 600,000 Iraqi deaths occurred in Iraq by July 2006, as a consequence of the invasion of Iraq.

A lot of people were against the war, but jacking up the body count with bad studies is not a good tactic for anyone. According to economics professor Michael Spagat of Royal Holloway College, these results were anywhere from seven to 14 times as high as other credible estimates, including those made by the non-partisan Iraq Body Count, a consortium of U.S. and U.K researchers, also concerned about the human toll of the war.

Such large differences in estimates led other researchers to question the methodology of the study. But contrary to scientific norms, Burnham refused to provide details about how the survey was conducted. When a complaint was lodged with AAPOR, its standards committee also tried to obtain such details, but was rebuffed. That led to the censure.

What exactly were John Hopkins Bloomberg School, and Burnham, et. al., hiding? AAPOR asked for the kind of information that any scientist doing this type of work should release: a copy of the questionnaire, the consent statement that interviewees have to see, a full description of the selection process, a summary of the disposition of all sample cases, and how the mortality rate was calculated.

John Hopkins Bloomberg School initially stood behind the study, but then eventually concluded that Burnham had made some unauthorized changes in his methodology, and thus “the School has suspended Dr. Burnham’s privileges to serve as a principal investigator on projects involving human subjects research.”

But the Bloomberg School has not come clean with the problems of the research project. Their press release admitted that their internal review “did not evaluate aspects of the sampling methodology or statistical approach of the study.” Instead, Bloomberg asserts, “It is expected that the scientific community will continue to debate the best methods for estimating excess mortality in conflict situations in appropriate academic forums.”

Let’s see: The Bloomberg School will not attempt to evaluate what experts believe is almost certainly a faulty methodology, saying the scientific community should make the evaluation. But then the school advises Burnham not to release details about his methods, so the scientific community can’t have the information it needs for a definitive assessment.

Sounds like a cop-out and a Catch 22, all rolled into one!

And we thought Richard Nixon was tricky.

* Burnham G, Lafta R, Doocy S, Roberts L. 2006a. ‘Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey’. The Lancet 368:1421-1428. It can be accessed online at http://brusselstribunal.org/pdf/lancet111006.pdf.

3 Comments:

Blogger 林翊娟 said...

IS VERY GOOD..............................

11:10 PM  
Anonymous Anonymous said...

'According to economics professor Michael Spagat of Royal Holloway College'

Michael 'Plan Colombia' Spagat? Does he have an undeclared conflict of interest?

http://dailysketcher.blogspot.com/2006/11/professors-michael-spagat-neil.html

'The important thing to notice however is that Professor Michael Spagat uses data from IBC for all his conflict analysis research that includes Iraq. This also applies to Professor Neil Johnson.

Obviously if all their reserach is based on IBC figures, then their findings would be virtually useless if the Lancet figures were correct.

Hence their desire to discredit it.

They are most definitely not 'honest brokers' in this affair:'

1:42 AM  
Anonymous Anonymous said...

Of course there is this too

http://scienceblogs.com/deltoid/2008/02/spagat_goes_off_the_deep_end.php


With all those points and lines, it's not hard to find two points that line up with one of the four possible points of L2. (In fact, as well as the ones Spagat shows, L1 and DRC also work). Once you've done this, all you have to is erase all the other points and lines to wind up with Spagat's graph. Needless to say, doing this is dishonest cherry picking, especially when you are doing it to accuse researchers of fraud.

Spagat says

"This graphic was passed to me by researchers who asked to remain anonymous."

And no wonder!

Since Spagat didn't produce the deceitful graphic, he isn't guilty of fraud, just of incompetence. As is David Kane, who describes Spagat's paper as a "tour de force".

1:47 AM  

Post a Comment

<< Home