False Statement on Missing Certificates
See UPDATE below.
Gilbert Burnham is an honest scientist doing his best to get to the truth, but he did not collect the data for L2; he did not travel to Iraq; he has only second-hand knowledge of what the interviewers did (as opposed to what they were supposed to do). Les Roberts is an ideologue and former candidate for Congress who will say most anything to advance the cause. Neither of them has, as far as I know, done any meaningful work with the actual data underlying L2, i.e., they rely on (the very smart) Shannon Doocy and Elizabeth Johnon to do the analysis. Put all this together and have them write a letter to the National Journal in response to Munro and Cannon's article on the L2. The results are not pretty. The NJ article included this information on missing certificates.
Having looked at the raw data, I believe the above analysis is 100% correct. Burnham and Roberts (BR) respond that:
First, it is a bad sign that BR do not specify which "statement" they are disagreeing with. Besides the section that I quoted (which I think is what they are referring to), there are few other references to missing certificates in the article. and none seems connected to BR's comment. Second, the article does not even present what I consider the most damning aspect of the missing certificate issue: interviewers were much more likely to "forget" to ask for certificates for violent deaths and for more recent deaths. Their forgetfulness was anything but random. Third, and most importantly, what BR claim in the above is false. Let's go to the data!
> library(lancet.iraqmortality)
> x <- prep.deaths()
> summary(x$certificate)
no yes forgot
45 501 83
There were 45 deaths in which the interviewers asked for certificates but for which no certificates were found. There were 83 deaths in which the interviewers forgot to ask. (In the raw data, these cases are marked as NA. I code them as forgot because this is what Gilbert Burnham claimed happened in these cases.)
> y <- subset(x, x$certificate == "forgot")
> dim(y)
[1] 83 14
> table(y$cluster)[table(y$cluster) != 0]
1 2 14 16 18 20 22 23 24 25 30 31 32 33 34 39 40 42 46 51
1 1 3 1 1 2 1 4 10 1 1 1 4 24 7 6 8 2 1 4
> length(table(y$cluster)[table(y$cluster) != 0])
[1] 20
So, there were 20 (not 3) clusters in which the interviewers forgot to ask for at least some death certificates. But maybe Burnham and Roberts are referring to clusters in which interviewers did ask but no certificates were available?
> y <- subset(x, x$certificate == "no")
> dim(y)
[1] 45 14
> table(y$cluster)[table(y$cluster) != 0]
2 4 5 11 12 13 25 26 34 35 36 37 41 45
2 2 1 4 1 6 1 2 20 1 1 2 1 1
> length(table(y$cluster)[table(y$cluster) != 0])
[1] 14
>
There were 14 such clusters. (And, minor note, there are 128 (not 120) cases in which interviewers either forgot to ask for the death certificate or did ask but did not get to see it.)
So, what are BR talking about? Who knows? My guess is that Roberts wrote this (without really looking at the data) and then convinced Burnham to sign off. Burnham would never, I think, purposely misrepresent the data. Roberts will say whatever it takes to convince people that the L2 results are basically accurate.
As a rule of thumb, you should double-check every claim that Les Roberts makes about the raw data. Much of what he says is true. Yet a lot of really important claims are false.
I think that BR are trying to use one of their favorite tricks: looking at all the data together when the critics just want to make a point about the data from violent deaths, where we believe the real problems are. But since BR can't even provide a competent summary of the data, the whole effort makes no sense. They really should check all their empirical claims with Shannon Doocy before making them.
UPDATE 2008-04-10. I just noticed that Burnham/Roberts have changed the text of the letter without telling anyone. Classy! Now the letter is a lie because it is no longer what was "submitted to the editors of the National Journal on January 7, 2008." The offending passage reads.
1) The numerical claims are now correct, as I show above.
2) But now the claim that the "statement on missing certificates is wrong" makes no sense. Nothing in Munro's article is contradicted by these numbers.
3) It is sleazy for Burnham/Roberts to make this correction without giving me credit. (I e-mailed them about it and, after not getting satisfaction, brought the issue to the attention of the General Counsel of Johns Hopkins).
4) It is sleazy for Burnham/Roberts to pretend that this was the original version of the letter.
5) It is false to (still) claim that "The following letter was submitted to the editors of the National Journal on January 7, 2008." This was not the letter that they submitted to the National Journal.
Gilbert Burnham is an honest scientist doing his best to get to the truth, but he did not collect the data for L2; he did not travel to Iraq; he has only second-hand knowledge of what the interviewers did (as opposed to what they were supposed to do). Les Roberts is an ideologue and former candidate for Congress who will say most anything to advance the cause. Neither of them has, as far as I know, done any meaningful work with the actual data underlying L2, i.e., they rely on (the very smart) Shannon Doocy and Elizabeth Johnon to do the analysis. Put all this together and have them write a letter to the National Journal in response to Munro and Cannon's article on the L2. The results are not pretty. The NJ article included this information on missing certificates.
Under pressure from critics, the authors did release a disk of the surveyors' collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single province of Nineveh. That means the surveyors reported zero missing or lost certificates for 180 violent deaths in 15 provinces outside Nineveh. The odds against such perfection are at least 10,000 to 1, Spagat told NJ. Also, surveyors recorded another 70 violent deaths and 13 nonviolent deaths without explaining the presence or absence of certificates in the database. In a subsequent MIT lecture, Burnham said that the surveyors sometimes forgot to ask for the certificates.
Having looked at the raw data, I believe the above analysis is 100% correct. Burnham and Roberts (BR) respond that:
The statement on missing certificates is wrong. Three clusters did not have the presence of certificates noted, and in all there were 120 deaths in which the interviewers neglected to note their presence.
First, it is a bad sign that BR do not specify which "statement" they are disagreeing with. Besides the section that I quoted (which I think is what they are referring to), there are few other references to missing certificates in the article. and none seems connected to BR's comment. Second, the article does not even present what I consider the most damning aspect of the missing certificate issue: interviewers were much more likely to "forget" to ask for certificates for violent deaths and for more recent deaths. Their forgetfulness was anything but random. Third, and most importantly, what BR claim in the above is false. Let's go to the data!
> library(lancet.iraqmortality)
> x <- prep.deaths()
> summary(x$certificate)
no yes forgot
45 501 83
There were 45 deaths in which the interviewers asked for certificates but for which no certificates were found. There were 83 deaths in which the interviewers forgot to ask. (In the raw data, these cases are marked as NA. I code them as forgot because this is what Gilbert Burnham claimed happened in these cases.)
> y <- subset(x, x$certificate == "forgot")
> dim(y)
[1] 83 14
> table(y$cluster)[table(y$cluster) != 0]
1 2 14 16 18 20 22 23 24 25 30 31 32 33 34 39 40 42 46 51
1 1 3 1 1 2 1 4 10 1 1 1 4 24 7 6 8 2 1 4
> length(table(y$cluster)[table(y$cluster) != 0])
[1] 20
So, there were 20 (not 3) clusters in which the interviewers forgot to ask for at least some death certificates. But maybe Burnham and Roberts are referring to clusters in which interviewers did ask but no certificates were available?
> y <- subset(x, x$certificate == "no")
> dim(y)
[1] 45 14
> table(y$cluster)[table(y$cluster) != 0]
2 4 5 11 12 13 25 26 34 35 36 37 41 45
2 2 1 4 1 6 1 2 20 1 1 2 1 1
> length(table(y$cluster)[table(y$cluster) != 0])
[1] 14
>
There were 14 such clusters. (And, minor note, there are 128 (not 120) cases in which interviewers either forgot to ask for the death certificate or did ask but did not get to see it.)
So, what are BR talking about? Who knows? My guess is that Roberts wrote this (without really looking at the data) and then convinced Burnham to sign off. Burnham would never, I think, purposely misrepresent the data. Roberts will say whatever it takes to convince people that the L2 results are basically accurate.
As a rule of thumb, you should double-check every claim that Les Roberts makes about the raw data. Much of what he says is true. Yet a lot of really important claims are false.
I think that BR are trying to use one of their favorite tricks: looking at all the data together when the critics just want to make a point about the data from violent deaths, where we believe the real problems are. But since BR can't even provide a competent summary of the data, the whole effort makes no sense. They really should check all their empirical claims with Shannon Doocy before making them.
UPDATE 2008-04-10. I just noticed that Burnham/Roberts have changed the text of the letter without telling anyone. Classy! Now the letter is a lie because it is no longer what was "submitted to the editors of the National Journal on January 7, 2008." The offending passage reads.
The statement on missing certificates is wrong. There were 83 deaths (13%) in which the interviewers neglected to note their presence and these deaths were distributed across 20 clusters.
1) The numerical claims are now correct, as I show above.
2) But now the claim that the "statement on missing certificates is wrong" makes no sense. Nothing in Munro's article is contradicted by these numbers.
3) It is sleazy for Burnham/Roberts to make this correction without giving me credit. (I e-mailed them about it and, after not getting satisfaction, brought the issue to the attention of the General Counsel of Johns Hopkins).
4) It is sleazy for Burnham/Roberts to pretend that this was the original version of the letter.
5) It is false to (still) claim that "The following letter was submitted to the editors of the National Journal on January 7, 2008." This was not the letter that they submitted to the National Journal.
posted by David Kane at 3:25 PM
2 Comments:
sorry David, but i m pretty sure that the single false statement here, again is yours.
here is what the Lancet authors said:
The statement on missing certificates is wrong. Three clusters did not have the presence of certificates noted, and in all there were 120 deaths in which the interviewers neglected to note their presence.
and here your attempt to contradict it:
(And, minor note, there are 128 (not 120) cases in which interviewers either forgot to ask for the death certificate or did ask but did not get to see it.)
as even you noticed, that is not a contradiction.
So, there were 20 (not 3) clusters in which the interviewers forgot to ask for at least some death certificates.
the Lancet authors rather obviously spoke about clusters, in which ALL or the vast majority of death certificates were not noted.
your analysis of few missing certificates in other clusters does NOT contradict that point.
instead it is weakening the point that Munro is making, claiming that ALL (violent deaths) certificates were missing from a single cluster..
But maybe Burnham and Roberts are referring to clusters in which interviewers did ask but no certificates were available?
...
There were 14 such clusters.
same argument as above. no contradiction of the Lancet repla, but in sharp contrast to Munro, who claims (citing Spagat), that the missing certificates for non-violent death are spread over 8 (eight!) provinces.
http://news.nationaljournal.com/articles/databomb/index.htm
why do you attack Burnham and Roberts, when you are unable to contradict them, but not Munro, whom you clearly contradicted?
Thanks for the comment sod, but you are mistaken. You claim that:
------
the Lancet authors rather obviously spoke about clusters, in which ALL or the vast majority of death certificates were not noted. your analysis of few missing certificates in other clusters does NOT contradict that point.
-------
First, I had a lengthy e-mail exchange with the authors and they never made that claim. Les Roberts defended the statement as made.
Second, even your defense does not work. Let's go to the data!
> x <- prep.deaths()
> y <- as.matrix(table(x$cluster, x$certificate))
> head(y)
no yes forgot
1 0 22 1
2 2 18 1
3 0 5 0
4 2 9 0
5 1 8 0
6 0 15 0
> y[y[,2] < y[,3],]
no yes forgot
24 0 8 10
32 0 1 4
33 0 1 24
39 0 0 6
40 0 0 8
>
In other words, there are 5 clusters (not three) in which "ALL or the vast majority of death certificates were not noted".
Again, my e-mail exchange with the authors confirmed that Les Roberts has only a hazy knowledge of the details of the data. Even after Shannon Doocy confirmed my analysis to him, he still refuses to remove the letter or apologize.
Post a Comment
<< Home