Plausible Response Rates

To the extent that I (and others) find the 98.3% response rate for Lancet II completely implausible, it behooves us to highlight similar surveys with more believable response rates. This report from World Public Opinion.org (WPO) provides such an example, especially appealing because it featured a nationwide sample and was conducted in September 2006, just a few months after the fieldwork for Lancet II. The headline, "Baghdad Shias Believe Killings May Increase Once U.S.-led Forces Depart but Large Majorities Still Support Withdrawal Within a Year," indicates that this is hardly the work of crazed Neocons.

What was the response rate for this poll? Details here (pdf):

The survey was designed and analyzed by the Program on International Policy Attitudes for WorldPublicOpinion.org. Field work was conducted through D3 Systems and its partner KA Research in Iraq. Face-to-face interviews were conducted among a national random sample of 1,000 Iraqi adults 18 years and older. An over sample of 150 Iraqi Sunni Arabs from predominantly Sunni Arab provinces (Anbar, Diyalah and Salah Al-Din) was carried out to provide additional precision with this group. The total sample thus was 1,150 Iraqi adults. The data were weighted to the following targets (Shia Arab, 55%, Sunni Arab 22%, Kurd 18%, other 5%) in order to properly represent the Iraqi ethnic/religious communities.

The sample design was a multi-stage area probability sample conducted in all 18 Iraqi provinces including Baghdad. Urban and rural areas were proportionally represented. Only one rural sampling point of the 115 employed were replaced for security reasons with substitutes in the same province and urban/rural classification. Among all the cases drawn into the sample, a 93% contact rate and 72% completion rate were achieved.

Since response rate is contact rate times completion (or participation) rate, the total response rate for this survey is 67%. (It could be that the completion rate here is actually from the whole sample, in other words, the 72% figure is the one we should use.) How can it be that Lancet II could achieve such a dramatically higher response rate? Note that both aspects of the overall response rate were lower for this survey. WPO had more trouble finding the individuals which it wanted to survey (99% versus 93%) and, having found those individuals, it had much more trouble convincing them to participate in the survey (99% versus either 72% or, if completion includes the contact problems, 77%).

If Lancet II had reported contact and participation rates more like those of the WPO, I would be much less suspicious of their results.