POSTED BY: | August 12, 2010

Pew Research Center

Estimates of telephone coverage released by the National Center for Health Statistics found that 25% of households (and 23% of adults) in the second half of 2009 had no landline service and only cell phone service (just 2% of households had no telephone service of any type).

For certain subgroups in the population, the numbers are considerably higher: 30% of Hispanics are cell-only, as are 49% of adults ages 25-29.

Yet pollsters and other survey researchers who use the telephone as the principal means of reaching potential respondents face a difficult decision as to whether to include cell phones in their samples. Doing so adds significantly to the cost and complexity of conducting surveys at a time when respondent cooperation is becoming increasingly difficult to obtain.

The following analysis updates and extends a previous Pew Research Center study of possible non-coverage bias in social and political surveys conducted by telephone. The study compares weighted estimates from landline samples to those obtained from combined samples of landline and cell respondents. Items selected include nearly all of the key indicators regularly tracked by the Pew Research Center for the People & the Press and the Pew Internet & American Life Project (e.g., presidential approval, party affiliation, internet use, broadband adoption, sending and receiving text messages on a cell phone), as well as a sampling of other timely indicators (e.g., agreement with the Tea Party, approval of health care reform, use of cell phones to play music).

Although still modest in size for most of the survey estimates of the general public examined in this review, non-coverage bias is now appearing regularly in landline telephone samples. Of 72 questions examined, 43 show differences of 0, 1 or 2 percentage points between the landline and dual frame weighted samples. Twenty-nine of the differences are 3 percentage points or more, all of which are statistically significant. Only one difference is as large as 7 points, while four others are 5 points and seven are 4 points.

For some estimates, even a small amount of bias may have important substantive consequences for the political or social implications of the research. Since the decline of landline coverage has not been uniform across demographic groups, non-coverage bias among certain subgroups may be even larger than for the full sample. As a result, some key subgroups in surveys based only on landlines may be severely underrepresented.

For example, among all adults, landline surveys underestimate wireless internet use by 2 percentage points, but the bias is 8 points among African Americans. Conversely, adults ages 50 and older are significantly overrepresented in landline samples, comprising 66% of the average landline sample when they should be only 40% of the sample.

The complete report may be found here.