More Response Rate Details

As a follow up to this post on response rates, I want to dive into the details in one of the polls that Kieran Healy cites because it helps to highlight just how implausible a 98.3% response rate for Lancet II is. Healy, among other examples, cites a Gallup poll (pdf) with a 97% response rate. He implies that, if Gallup (regularly?) gets a 97% response rate, than a 98% rate for Lancet II is perfectly plausible.

Here (from page 11) are all the detail we have on this 97% rate.

Face-to-face interviews were conducted among 1,178 adults who resided in urban areas within the governorate of Baghdad. Interviews were carried out between August 28 and September 4. The response rate was 97 percent; 3 percent of those selected refused to participate in the study.

A probability-based sample was drawn utilizing 1997 census data. Census districts were utilized as primary sampling units (PSUs). A total of 122 PSUs were chosen using probability-proportional-to-size methods. About 10 interviews, one per household, were conducted at each location. Interviewers were given all relevant address details for each PSU. Within each selected household, respondents were selected using the Kish method.

For the results based on this sample, one can say with 95% confidence that the margin of error is approximately ± 2.7%.

Now, it would be nice to have more details on this survey and, in time, I hope to find someone with lots of experience conducting polls in Iraq specifically and third world countries in general, but there are several issues to keep in mind.

Recall that non-response generally falls into two categories: failure to contact and, given that someone has been contacted, a refusal to participate.

Yet Gallup only provides us information about the response rate for people who were not absent. In other words, we have no information on the contact rate. We just know the participation rate. (It could be that Gallup is being sloppy and that this 3% who "refused to participate in the study" includes those who were not home.)

So the correct comparison of this 97% rate is not with the 98.3% response rate for Lancet II but the 99.2% participation rate.

Again, the difference between 97% and 99.2% may not seem large. And it isn't.

First, I just want to point out that the Lancet II response rates are higher than any other survey (with one possible exception to be addressed later). Second, one could just as easily say that the refusal rate for this Gallup poll is more than three times higher than the refusal rate for Lancet II. Why would that be? Why would households be so much less willing to participate in this Gallup than they are in the Lancet II? Third, the closer one gets to 100% participation, the more difficult it is to make progress. There is a much larger difference between 97% and 99% participation rates than between 60% and 62% rates because the marginal 2% increase is much harder to achieve the closer you get to 100%. Fourth, certain aspects of Lancet II should make it harder to have a higher participation rates. For example, this Gallup poll did not require the presence of the head of the household or spouse. Any resident adult would do.

Until we know more details about opinion polling in Iraq, it is tough to know what to do with very high response rates, rates much higher than anything we see in the US. But it is still a mystery why the rates for Lancet II should be so much higher than any other survey.