Not in My Journal

One of the more dramatic moments in the JSM session devoted to the Lancet surveys occurred during the question and answer session. (When AP reporter Paul Foy described the "courtroom-style questioning," I am pretty sure that he was, at least in part, referring to this incident.) This was actually the first question and it went, as luck would have it, to me. I asked something like:

As many audience members know, the Ethical Guidelines of the American Statistical Association require us to share data with other scientists for purposes of replication. Although you and your co-authors have shared some of your data with some other researchers, you have refused to share data with Michael Spagat and his co-authors. Why should we give credence to what you have said today if you refuse to adhere to the ethical guidelines of the ASA?

Comments:

1) No doubt my actual questions included more "uh's" and "you know's." It was certainly not phrased as nicely. But my meaning got across. Roberts and the other panelists addressed my point directly.

2) The Ethical Guidelines for Statistical Practice for the American Statistical Association require that statisticians:

Share data used in published studies to aid peer review and replication, but exercise due caution to protect proprietary and confidential data, including all data which might inappropriately reveal respondent identities.

Now, it is not clear that any of the Lancet authors are even members of the ASA, so these guidelines may not even apply to them. Roberts on occasion tries to claim that he is protecting interview/interviewee safety by not releasing the data. But he can't get away with that dodge for L2 since that data has been shared. There is just no reason to not share the data with critics like Spagat et al when you have already shared the data with critics like me.

3) The audience, of course, does not read the details of the ethical guidelines or follow the Lancet debate closely. But they certainly found it absurd that a scientist would share his data with some critics but not others. The reason that the Q&A went in the direction of "courtroom-style questioning" was the skepticism caused by Roberts et al behavior. I have yet to find, after a year of looking, another example where the data behind a published study was shared with some academic critics but not others.

What happened next? Roberts gave his usual shpiel, which mostly focuses in what a waste of time it is to have this discussion, to even talk about an already published study. Instead, critics should be getting new data, going to Iraq and so on. Best part was when he said that "If it were up to me, we would not have released the data to anyone." I was not the only member of the audience to be shocked by that admission. But it is consistent with what I can glean of the internal dynamics of the Lancet team. Roberts would really prefer to not share anything with anyone, but cooler heads (like Burnham, I think) occasionally prevail.

The other participants than chimed in. They all agreed that scientists should share their data. They all urged Roberts to do so. (All the subsequent questions from the floor that addresses this point seemed to agree with this position.) But the best comment came from Stephen Fienberg, the chair of the session and a professor at Carnegie Mellon.

That's just the wrong response. I, as an editor, would not publish a study for which the data was not shared.

Unfortunately, the editor of the Lancet feels differently. I wonder why.