The Impact Of "Cell-Onlys" On Public Opinion Polling

The proportion of Americans who rely solely on a cell phone for their telephone service continues to grow, as does the share who still have a landline phone but do most of their calling on their cell phone. With these changes, there is an increased concern that polls conducted only on landline telephones may not accurately measure public opinion. A new Pew Research Center study finds that, while different demographically, Americans who mostly or exclusively rely on cell phones are not substantially different from the landline population in their basic political attitudes and preferences.

On key political measures such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation, respondents reached on cell phones hold attitudes that are very similar to those reached on landline telephones. Analysis of two separate nationwide studies shows that including interviews conducted by cell phone does not substantially change any key survey findings.

These findings are based on two surveys of adults, conducted Oct. 17-23 and Dec. 19-30, 2007 by the Pew Research Center for the People & the Press. The surveys included interviews with a total of 2,596 adults reached in a conventional landline sample, as well as 841 adults interviewed on their cell phones, using a sample drawn from a nationally representative cell telephone number database. Of those reached on a cell phone, 312 people (or 37%) reported that their cell phone is their only phone.

When data from both samples are combined and weighted to match the U.S. population on key demographic measures, the results are virtually identical to those from the landline survey alone. Across more than 100 political and attitudinal questions on the surveys, including cell phone interviews does not change the results by more than two points in the vast majority of comparisons, and in only one comparison is the difference as large as 4 points.

In particular, there is no evidence that the polling in the Democratic and Republican nomination contests is biased by the fact that most polls rely only on landline interviews. In the December national poll, support for no candidate in the landline sample changed by more than two points when the preferences of cell phone respondents were blended in. The same was true in the October national poll.

There is no doubt that Americans who rely solely on cell phones differ from the rest of the public in some key respects. However, in most cases these differences are the result of their demographic characteristics, particularly the fact they tend to be very young. Since adjustments for age are made in standard landline surveys, adding the cell-only component to the survey substantially increases the raw number of younger people surveyed, but does not alter the overall weight of younger respondents in the final estimates.

In most respects, the political attitudes and behaviors of younger people who are cell-only do not differ substantially from younger people surveys do reach on landlines, meaning that the overall results are virtually identical to those from the landline survey alone.

However, on some non-political topics, and in surveys of certain groups in addition to young people, studies have shown that the inclusion of cell phones in the sample design makes a difference in the combined results. An earlier study by the Pew Research Center for the People & the Press found that blending landline and cell phone samples resulted in higher estimates of young people ages 18 to 25 using new technologies. In addition, small but significant differences were found for lifestyle measures such as attending church and alcohol consumption. In another study from the National Health Interview Survey, Blumberg and Luke (2007) found that for surveys of low-income adults and young adults, the estimates for health risk behaviors, HIV testing, exercise and obesity were all changed when cell phones were included in the sample.

In addition to testing the impact of cell phone sampling, the October and December Pew studies demonstrate the feasibility of including cell phones in telephone surveys. The response rates for the cell and landline samples were virtually identical in both studies, and there is no evidence that the quality of data gathered from cell phone surveys is lower than in landline surveys. Including cell phones, however, is very costly. On average, a cell phone interview costs approximately three times as much as a comparable landline interview.

Although the inclusion of cell phone samples is very costly, and may make little difference in the substantive conclusions one would draw from political surveys, other aspects of the dual frame design provide particular benefits that may argue for the adoption of this type of sampling frame design. Chief among these benefits is the improved demographic representation for certain groups and the attendant increase in the sizes of the samples of these groups for further analysis. This is because it is easier to reach by cell phone than by landline certain groups of respondents who have both types of service.

The inclusion of a cell phone sample may be essential in surveys of population groups that have high rates of cell-only households. More generally, with an estimated 14% of Americans relying solely on cell phones, their exclusion from opinion surveys may call into question the credibility of polls in the mind of the public.

Overview of Differences

Results from both the December and October polls show that the cell-only respondents have somewhat different attitudes and behaviors from those reached on landline telephones. In the December survey, which focused on the public's campaign news sources, cell-only respondents were significantly less likely to say they have watched a presidential debate on television, but more likely to have seen debate video online. This reflects a more general pattern: cell-only Americans are somewhat less likely to rely on newspapers and network evening news for campaign information, but more apt to get campaign news from the internet, late night comedy shows, and to use social networking sites. Not surprisingly, these behaviors are characteristic of younger respondents in general - whether cell-only or not - and the blended results for none of these measures change by no more than two percentage points.

The October survey included questions that asked registered voters about the importance of 16 issues to their vote. There were a few significant differences between the landline respondents and those who were cell-only: the latter group was 14 points less likely to say Social Security would be important to their vote, and somewhat more likely to say immigration would be important. Again, these differences are understandable, given the fact that cell-only respondents are younger (and thus less concerned about Social Security) and more likely to be Hispanic (who are more concerned about immigration). When the cell-only respondents were combined with the landline respondents, none of the overall survey estimates changed by more than one percentage point.

While including interviews conducted by cell phone in a national sample does not substantially affect survey findings, it does improve the overall representativeness of the sample by reaching more respondents in otherwise hard to reach subpopulations. This reduces sampling error for these groups, and may also mean that the survey requires less statistical adjustment to match the demographic profile of the population. Less clear is whether adding cell phone interviews is the most efficient use of resources. Cell phone interviews cost approximately three times as much as landline interviews, and the sample sizes of underrepresented groups can be boosted more cheaply by simply expanding the overall sample size of the landline survey.

Profile of Cell-Only Respondents

One of the most striking differences between cell-only respondents and people reached on a landline telephone is their age. Nearly half of the cell-only respondents (46%) are under age 30 compared to only 12% in the landline sample. Related to their younger age, only 26% of cell-only respondents are married, compared with 57% percent of those in the landline sample. Similarly, about half of cell-only respondents have never been married (51%), compared with only 16% in the landline sample.

In addition, the landline sample includes a higher proportion of college graduates than the cell-only group (38% vs. 26%), which may also reflect the greater use of cell phones among young people who are still in college. The income distribution also is quite different for the landline and cell-only groups; 29% of people in the landline sample have household incomes of at least $75,000 annually, compared with just 16% in the cell-only group. Similarly, nearly twice as many cell-only than landline respondents earn less than $30,000 a year (41% vs. 21%).

Overall, the landline sample includes more whites (82% vs. 68%) than the cell-only group while the cell-only group includes a greater proportion of minorities. In the cell-only group, there are more African-Americans (19% vs. 11%), Hispanics (13% vs. 6%), and Asians (5% vs. 1%) compared with the landline sample. The cell-only group also includes a larger percentage of males than the landline group (61% vs. 48%). Finally, more cell-only respondents than landline respondents are religiously unaffiliated (27% vs. 14%).

The "Dual" Households

In this study, cell phone interviews were conducted with cell-only individuals (those who have no landline phone), as well as with those who were reached by cell phone but also have a landline telephone. Since these so-called dual-phone respondents could, in fact, be contacted on a landline telephone, some prior studies did not interview them, focusing only on those reachable only on a cell phone.

However, the current study includes dual-phone respondents regardless of whether they were reached on their landline or cell phones. This choice reflects the fact that about half (47%) of the dual-phone respondents who were reached on their cell phone say that they receive more of their calls on their cell phone, in most cases a lot more. While it may be possible to reach these respondents on their landline telephone, it may be more difficult to do so.

The crux of the issue is whether the dual users reached by cell phone are different from those reached by landline. For the most part, the answer is no. Among the dual users, more males than females were reached by cell phone (56% male, compared with 48% male among dual users reached by landline). And more than twice as many Hispanics were reached by cell phone (11% vs. 5%). Those reached by cell phone were somewhat younger (57% under age 50, compared with 47% among those reached by landline). Across a broad range of attitudinal questions in the two surveys, there was very little difference between the dual users reached by cell phone and those reached by landline.

Young Landline vs. Cell Users

In some respects, young people who rely solely on cell phones are quite different demographically from young people who have landline telephones. Much of the difference is driven by the fact that, even within the 18-29 year-old age group, the average age of cell-only respondents is much younger than of landline respondents. Among respondents under age 30, a greater proportion of cell-only respondents than landline respondents are under age 25 (70% vs. 55%). In part because of their younger age, fewer young cell-only people are married (15% vs. 32%) and fewer have children (19% vs. 31%). Nearly half of young people under the age of 30 who rely exclusively on their cell phones (48%) have household incomes of less than $30,000 a year, compared with about a third (32%) of those in the same age category with landline telephones. There also is a substantial gender difference, with men outnumbering women in the cell-only sample (62% vs. 38%), compared with a more even balance in the landline sample (48% male, 52% female).

However, there are no significant differences in education between young people with landlines and those that are cell-only. While both groups have comparable numbers of whites and African Americans, a greater proportion of cell-only people are Asian (8% vs. 2% of the landline sample) Finally, fewer cell-only young people than those with landlines attend religious services once a week (24% vs. 36%) probably because more are religiously unaffiliated (36% vs. 26%).

Although cell-only and landline users under the age of 30 differ demographically, there are very few differences in their political attitudes, ideology, and partisan affiliation. Comparable majorities of young people in the landline and cell-only samples express dissatisfaction with the way things are going in the country, and about the same proportions in both groups disapprove of President Bush's job performance. Slightly more cell-only than landline people affiliate with the Democratic Party; however, ideologically, more cell-only people report they are conservative than their landline counterparts and neither of these differences are significant.

When it comes to the campaign, young people who are cell-only report that they regularly learn about the campaign from a larger number of news sources (an average of 1.98 "regular" sources per person) than do those with landline telephones (an average of 1.66). In contrast, slightly fewer cell-only young people say they have given a lot or some thought to the 2008 presidential campaign and slightly fewer are registered voters than are those with landlines. Among those registered to vote, there are no differences in whether young people are likely to vote in the presidential primary.

Young people have similar views on the situation in Iraq regardless of whether they rely exclusively on their cell phones or have landline telephones. Identical percentages of cell-only and landline young people (55% each) say that the United States made the wrong decision in using military force against Iraq and that the U.S. military effort there is not going well (53%). Compared with those who have landlines, a slightly larger number of cell-only young people think that the U.S. should keep military troops in Iraq until the situation is stabilized (43% vs. 35%).

Overall, these results suggest that the political attitudes of young people do not vary much by telephone status. As a result, while their inclusion in the study substantially increases the number of younger people interviewed, it does not substantially change overall survey estimates.

Practical Considerations in Conducting Interviews on Cell Phones

This study and several others conducted by the Pew Research Center, as well as those by other survey organizations, demonstrate that it is feasible to conduct random sample surveys by cell phone. But the process is costly, requiring significant additional effort by the survey field house and some additional work in data processing and weighting. Exclusive of the fixed study costs such as CATI programming, pre-testing surveys, creating demographic banners, the marginal cost of a cell phone interview in these two studies was approximately three times larger than the marginal cost of a landline interview. And in terms of reaching the most critical "cell-only" respondents, previous studies suggest that such interviews cost four to five times more than comparable landline interviews, largely because of the additional screening necessary to locate cell-only respondents.

The cost differential for calling cell phones is a result of several operational differences between calling in the landline and cell sample frames. One of the largest differences results from the fact that, due to federal regulations, telephone numbers in the cell frame must be manually dialed by the interviewer. For landline numbers, an "auto-dialer" is used to take a number from the sample and actually dial it before transferring the call to the interviewer.

Another difference is that a significant number of people reached in the cell frame turned out to be under the age of 18 and thus ineligible for the survey. In fact, more than four-in-ten (42%) of the cell phone respondents who were willing to cooperate with the survey could not be interviewed because the phone belonged to an underage person. None of the cooperating households in the landline frame was excluded because they contained no adults. This aspect of the cell sample, along with the fact that the cell phone frame reaches a higher percentage of individuals who do not speak English, meant that the percentage of contacted individuals eligible for the survey was far lower in the cell frame - just 45% and 40% in October and December, respectively, compared with 86% and 85% in the landline frame.

A third difference is that respondents in the cell frame were offered a modest cash reimbursement to offset the cost of airtime they might incur while taking the survey. Beyond the expenses incurred, the collection of contact information in order to reimburse respondents, and the attendant administrative and processing costs, adds to the overall cost of interviewing in the cell frame. The vast majority of respondents (85% in October and 80% in December) who agreed to participate in the interview provided the necessary name and mailing address to receive the reimbursement.

To test the potential impact of different amounts of reimbursement, cell phone respondents in October were randomly assigned to be offered either $10 or $20. Somewhat surprisingly, there was virtually no difference in the response rate between those offered $10 and those offered $20. (There also was no difference in the percentage of cooperating respondents who provided a name and address for reimbursement at the end of the interview.)

Apart from the eligibility rates and the cost differential, however, there were remarkable similarities between the cell and landline samples in several aspects of the fieldwork. The contact and cooperation rates between the cell and landline samples were nearly identical. Similarly, the breakoff rate - the percentage of people who begin the interview but do not complete it - was the same in each sample. As a result, overall response rates were very similar in the cell and landline samples - 23% in each sample in October, and 18% in the landline sample in December - compared with 22% in the cell sample.

Quality of Responses: Landline vs. Cell Phone Interviews

Differences in the ways that people use landline telephones and cell phones could potentially affect the quality of data collected in surveys sampling both kinds of phone numbers. For example, if people are more distracted or more accustomed to short conversations on a cell phone compared to when they use a landline, then they may not respond as carefully when interviewed on a cell phone. However, studies on this topic have found no substantive differences between the quality of answers recorded in landline interviews and those recorded in cell phone interviews. Results from recent Pew surveys are generally consistent with this finding.

People interviewed in cell samples were less likely to refuse to answer or say "don't know" on at least one question than those interviewed in the landline sample. This result, however, simply reflects the different characteristics of people reachable by landline versus those reachable on a cell phone. For example, adults ages 60 and older are more likely than younger people to decline to answer questions; they also are much more likely to be interviewed in the landline sample. After accounting for such demographic differences, there is no perceptible difference in the rates of Refused/Don't know responses between cell phone and landline samples.

Another way to gain insight into how carefully people respond is through interviewer evaluations. Immediately after completing each interview, interviewers recorded their impressions of the respondent's level of cooperation and level of distraction (each on a four-point scale).

There is a slight suggestion that the cell sample respondents were more cooperative and less distracted than those reached on landlines, but again the difference may be attributable to factors other than the type of phone used by the respondent. The difference in the age distributions of the two samples is one factor. The monetary reimbursement, which was offered only to persons in the cell sample, also may have an effect. Presumably, cell sample respondents' knowledge that they would be remunerated had a positive effect on their attitude during the interview. By this logic, if an incentive had been offered to the landline sample as well, the rates of cooperation would be even more similar.

Benefits of Conducting Cell Phone Samples

Surveys that rely only on landline interviews are more likely to produce biased estimates if the segment of the public unreachable on a landline differs substantially from the landline public. If the cell-only respondents are not very different from the landline respondents, the survey estimates will not be biased by the absence of the cell-only group. For example, the landline survey finds that 54% of Americans favor bringing troops home from Iraq; among the cell-only respondents, 55% favor a U.S. troop withdrawal. Thus the overall survey estimate is unaffected when the cell-only respondents are blended in. One way to consider the impact of adding cell-only interviews to a survey is to ask the question: How different would the cell-only have to be for the total survey estimates to be affected by their inclusion?

For example, in the unlikely instance that 100% of the cell-only adults favored a troop withdrawal from Iraq, and landline respondents remained divided (with 54% favoring withdrawal), then the combined survey estimate would shift to 59% - a five-point increase. The standard survey alone would underestimate national support for withdrawal. Alternatively, if just 30% of the cell-only respondents favored withdrawal, the combined estimate would be 50%, four points below the current estimate from the landline survey.

These effects are potentially greater when analyzing subgroups in the population, such as young people, who are less likely to be reached on a landline. For example, if 100% of cell-only young people (ages 18-29) favored a troop withdrawal, the combined sample estimate for this age group would be 72% in favor of withdrawal, rather than the 60% that the landline sample of young people produces.

Guarding Against Bias

Findings reported here and in other studies demonstrate that standard landline samples still perform well relative to more expensive designs that combine landline and cell phone samples. Currently, this holds true for most overall population estimates. The potential for bias, however, is greater for estimates for subgroups that tend to rely more on cell phones, such as young adults, blacks, Hispanics.

Indeed, for such groups, several standard sample estimates differ from the corresponding combined sample estimates. For example, 46% of Hispanics align with the Democratic Party, based on the standard landline sample. Based on the combined sample, however, 43% of Hispanics consider themselves Democrats. On other items, the standard and combined samples yield similar results, even on estimates for young adults.

When there is a difference between the standard and combined estimates, the natural question is which figure is more accurate. Benchmark data from the American Community Survey (a large multi-mode survey conducted by the Census Bureau) shows that the answer varies.

The combined survey sample yields more accurate estimates for Hispanics on two of the characteristics evaluated here. With regard to African Americans, the combined sample estimate of the proportion of the black population who are parents of children under age 18 is more accurate than the corresponding landline sample estimate. However, the combined sample estimate for the marriage rate among blacks is less accurate. For all 18-29 year olds, the combined sample appears to be slightly less biased in estimating the marriage rate and the proportion who are parents of children under 18.

These results demonstrate that a combined sample is not always superior to a standard sample (and vice versa). This may seem counterintuitive given that the combined sample, by definition, does a better job covering the population (both landline and cell phone users). The primary explanation for the shortcomings of both the standard and combined designs appears to be non-response: Everyone with a telephone has a chance of being interviewed in the combined design, but most either do not answer the call or decline to be interviewed. Those who do respond in landline or cell samples sometimes differ systematically on items in the survey from those who do not participate.

Sample Sizes of Groups Relying Mostly on Cell Phones

One potential advantage of a dual-frame survey is that it may be possible to complete more interviews with groups who rely more on cell phones. For example, 28% of cell phone respondents are under age 30. This is more than double the rate of young adults in landline samples (12%). Thus, a sample of 1,000 cell interviews would yield roughly 280 adults age 18 to 30, while an equally-sized sample of landline numbers would yield roughly 120 adults in this age group.

Having a larger sample size is important because it means more precise estimates. Roughly speaking, the margin of error on an estimate for young adults is 6% with a sample size of 280. With the smaller sample size of 120, the estimate is less reliable and the margin of error is about 9%.

Currently, these advantages are not being realized, largely because of the cost. Cell phone interviews are approximately three times more expensive than landline interviews. Young adults, however, are not three times more likely to be reached in the cell sample (only about twice as likely). When the survey budget is held fixed, the most effective way to maximize the number of interviews - even for groups like 18-29 year olds who rely more heavily on cell phones - is to allocate the entire budget to increasing the overall number of landline interviews. This is because roughly three landline interviews can be completed for the same cost as every one cell phone interview.

Over time the cost differential between landline and cell interviews may narrow. It also is possible that the prevalence of various subgroups may become lower in landline samples and higher in cell samples. Such developments would imply greater sample sizes under a dual frame design (for fixed cost) relative to sample sizes expected under current conditions.