Methodology

Collecting Survey Data

Survey researchers employ a variety of techniques in the collection of survey data. People can be contacted and surveyed using several different modes: by an interviewer in-person or on the telephone (either a landline or cell phone), via the internet or by paper questionnaires (delivered in person or in the mail).

The choice of mode affects who can theoretically be interviewed in the survey, the availability of an effective way to sample people in the population, how people can be contacted and selected to be respondents and who responds to the survey. In addition, factors related to the mode-such as the presence of an interviewer and whether information is communicated aurally or visually-can influence how people respond.

Survey response rates often vary for each mode and can be affected by aspects of the survey design (e.g., number of calls/contacts, length of field period, etc.). In recent years surveyors have been faced with declining response rates for most types of surveys, which we discuss in more detail in the section the problem of declining response rates.

In addition to landline telephone surveys, the Pew Research Center for the People & the Press also conducts cell phone surveys, web surveys and mixed-mode surveys, where people can be surveyed by more than one mode. We discuss these types of surveys in the following sections and provide examples from surveys that used each method. In addition, some of our surveys involve reinterviewing people who we have previously surveyed to see if their attitudes or behaviors have changed. For example, in presidential election years we interview voters who were first contacted earlier in the fall again following the election in order to understand how their opinions may have changed from when they were interviewed previously.

Cell phone surveys

Telephone surveys have traditionally been conducted only by landline telephone. However, as the proportion of Americans who rely solely or mostly on a cell phone for their telephone services now approaches 20%, more surveys are including interviews with people on their cell phones. Cell phone surveys are conducted in conjunction with a landline survey to improve coverage. The data are then combined for analysis. In addition to the issues associated with sampling cell phones, there are also unique challenges that arise when surveying people on their cell phones.

table

One of the most important considerations when conducting cell phone surveys is that the costs are substantially higher than for a traditional landline survey. The cost of a completed cell phone interview is two to two-and-a-half times more than a completed landline interview. Although some of the fixed costs associated with landline surveys are not duplicated when a cell phone sample is added (in particular, programming the questionnaire and pre-testing), other fixed costs are higher (data processing and weighting are more complex in dual-frame surveys).

One of the most important additional costs of cell phone surveys is screening for eligible respondents. A significant number of people reached by cell are under the age of 18 and thus are not eligible for most of our surveys of adults. Cell phone surveys also cost more because federal regulations require cell phone numbers to be dialed manually (whereas auto-dialers can be used to dial landline numbers before calls are transferred to interviewers). In addition, respondents (including those to Pew Research surveys) are often offered cash reimbursements to help offset any costs they might incur for completing the survey on their cell phone. The cost of these payments, as well as the additional time necessary for interviewers to collect contact information in order to reimburse respondents, adds to the cost of conducting cell phone surveys (see Calling Cell Phones in '08 Pre-Election Polls for more information).

In addition to costs, most cell phones have caller identification or other screening devices that allow people to see the number that is calling before deciding to answer. There also is wide variation in how people use their cell phones (e.g., whether they are turned on all the time, used only during work hours or only for emergencies). Cell phone surveys are also influenced by the respondents' environment (as are landline surveys, but people responding to landline surveys are always at home). In particular, legal restrictions on the use of cell phones while driving, as well as concerns about safety, have raised the issue of whether people should be responding to surveys on their cell phones while driving in a car. In addition, people often talk on their cell phones in more open places where they may have less privacy. These concerns have led some surveyors (including the Pew Research Center) to ask cell phone respondents whether they are in a safe place and whether they can speak freely before continuing with the interview. Lastly, the quality of connection may influence whether an interview can be completed at that time, and interruptions may be more common on cell phones.

Although some research suggests that response rates may be lower for cell phone surveys, recent analysis of Pew Research data suggests that response rates and cooperation rates are similar for landline and cell samples. Similarly, although some have suggested that respondents may be more distracted during an interview, our research has not found substantive differences in the quality of responses between landline and cell phone interviews; interviewer ratings of respondent cooperation and levels of distraction were similar in the cell and landline samples, with cell phone respondents showing slightly greater cooperation and less distraction.

Related Reports and Commentaries

Back to top

Internet surveys

The number of internet surveys being conducted has increased dramatically in the last 10 years. Web surveys have proliferated because they can be used to conduct random sample surveys of members of selected populations who have access to the internet and the skills necessary to complete a survey on the Web (e.g., students, members of voluntary associations, etc.). The Pew Research Center has conducted several mixed-mode surveys of journalists where respondents completed the survey via the Web or by telephone.

Although more surveys are being conducted via the Web, internet surveys have not replaced other modes for surveys of the general population. Internet surveys of the general population can be subject to significant biases resulting from undercoverage and nonresponse. Not everyone in the U.S. has access to the internet and there are significant demographic differences between those who do have access and those who do not. People with lower incomes, less education, living in rural areas or age 65 and older are underrepresented among those who use the internet and those with high-speed internet access (see the Pew Research Center's Internet and American Life Project for the latest trends). People also vary a great deal in the skills necessary to complete a Web survey. There also is no systematic way to sample the general population using the internet. There is no national list of email addresses from which people could be sampled, and there is no standard convention for email addresses, as there is for phone numbers, that would allow random sampling. Further, many people have multiple email addresses.

Because of these limitations, surveyors use two main strategies for surveying the general population using the internet. One strategy relies on convenience samples of internet users. Some researchers rely on panels of respondents who opt-in or volunteer to participate in surveys, while others use one-time surveys that invite participation from whoever sees the survey invitation. These surveys are subject to the same limitations facing other surveys using nonprobability-based samples: the relationship between the sample and the population is unknown so there is no theoretical basis for computing or reporting a margin of sampling error and thus for knowing how representative the sample is of the population as a whole. (also see American Association For Public Opinion Research - Opt-In Surveys and Margin of Error). We discuss results comparing responses from an online survey to those obtained using our traditional RDD telephone survey in Online Polling Offer Mixed Results.

Another strategy for creating an online panel is to use another mode (mail, telephone or face-to-face) to randomly recruit people to participate and then have them respond to surveys via the Web. Recruitment using probability-based sampling via another mode allows surveyors to estimate a margin of error for the survey (see Why probability sampling for more information).

Related Reports and Commentaries

Back to top

The problem of declining response rates

As Americans are faced with more demands on their time and a growing number of unsolicited telephone calls, many have armed themselves with increasingly sophisticated technology for screening their calls (e.g., voice mail, caller identification, call blocking and privacy managers) and are exercising more choice over when and how they can be contacted. As a result, fewer Americans are participating in telephone polls than was the case when telephone surveys first became prevalent. As a consequence, response rates have been declining over the past decade or more.

The Pew Research Center for the People & the Press has conducted two survey experiments to gauge the effects of respondent cooperation on the validity of the results. These experiments compare responses from a standard survey, conducted with commonly utilized polling techniques over a five-day period, and a survey conducted over a much longer period that employed more rigorous techniques aimed at obtaining a high rate of response.

Findings from both the 2003 study Polls Face Growing Resistance, But Still Representative and the 1997 study Conservative Opinions Not Underestimated, But Racial Hostility Missed indicate that carefully conducted polls continue to obtain representative samples of the public and provide accurate data about the views and experiences of Americans. These results are also reported in Public Opinion Quarterly.

Related Reports and Commentaries

Back to top

Mixed-mode surveys

Survey researchers employ a variety of techniques in the collection of survey data. People can be contacted and surveyed using several different modes: by an interviewer in-person or on the telephone (either a landline or cell phone), via the internet or by paper questionnaires delivered in person or in the mail.

The choice of mode affects who can theoretically be interviewed in the survey, the availability of an effective way to sample people in the population, how people can be contacted and selected to be respondents, and who responds to the survey. In addition, factors related to the mode, such as the presence of an interviewer and whether information is communicated aurally or visually, can affect how people respond.

Although the Pew Research Center primarily conducts telephone surveys, we also occasionally conduct mixed-mode surveys, where people are surveyed by more than one mode. For example, we have conducted several mixed-mode surveys of journalists where respondents complete the survey via the web or by telephone.

Related Reports and Commentaries

Back to top

Reinterviews

Reinterviews are typically used to examine whether individuals have changed their opinions, behaviors, or circumstances (such as employment, health status or income) over time. Survey designs that include reinterviews are called panel surveys. The key feature of this survey design is that the same individuals who were interviewed at the time of the first survey are interviewed again at a later date. The Pew Research Center for the People & the Press sometimes conducts reinterviews, especially to learn more about whether and how voters' opinions change during the course of a presidential election campaign. For an example from the 2008 presidential campaign see High Marks for the Campaign, a High Bar for Obama. For an example comparing foreign policy opinions before and after the events of September 11, 2001 see America's New Internationalist Point of View.

Some of the reports listed below used reinterviews primarily to ask follow-up questions about respondents' opinions rather than to analyze opinion change on the same issues. Survey reports of this sort include Beyond Red vs. Blue and Voters Like Campaign 2004, But Too Much 'Mud-Slinging'.

Related Reports and Commentaries

Back to top