Methodology

Sampling

For most of our national surveys of the general public, we conduct telephone surveys using a random digit sample of landline and cell phone numbers in the continental United States. Some of these general population surveys include additional, larger samples of subgroups, such as African Americans or young people. (These are called "oversamples.") We also conduct surveys of people in particular states or regions, where our sample is limited to residents of these areas, and international surveys that involve sampling people in multiple countries. Lastly, we occasionally conduct surveys of special populations, such as foreign policy officials or journalists. The importance of probably sampling is discussed more broadly in the final section "Why probability sampling."

Random digit dialing - our standard method

The typical Pew Research Center for the People & the Press national survey selects a random digit sample of both landline and cell phone numbers in the continental United States. As the proportion of Americans who rely solely or mostly on cell phones for their telephone service continues to grow, sampling both landline and cell phone numbers helps to ensure that our surveys represent all adults who have access to either. We sample landline and cell phone numbers to yield a ratio of approximately three landline interviews to each cell phone interview. This ratio is based on an analysis that attempts to balance cost and fieldwork considerations as well as to improve the overall demographic composition of the sample (in terms of age, race/ethnicity and education). This ratio also ensures a minimum number of cell only respondents in each survey.

table

The design of the landline sample ensures representation of both listed and unlisted numbers (including those not yet listed) by using random digit dialing. This method uses random generation of the last two digits of telephone numbers selected on the basis of the area code, telephone exchange, and bank number. A bank is defined as 100 contiguous telephone numbers, for example 800-555-1200 to 800-555-1299. The telephone exchanges are selected to be proportionally stratified by county and by telephone exchange within the county. That is, the number of telephone numbers randomly sampled from within a given county is proportional to that county's share of telephone numbers in the U.S. Only banks of telephone numbers containing three or more listed residential numbers are selected.

The cell phone sample is drawn through systematic sampling from dedicated wireless banks of 100 contiguous numbers and shared service banks with no directory-listed landline numbers (to ensure that the cell phone sample does not include banks that are also included in the landline sample). The sample is designed to be representative both geographically and by large and small wireless carriers.

Both the landline and cell samples are released for interviewing in replicates, which are small random samples of the larger sample. Using replicates to control the release of telephone numbers ensures that the complete call procedures are followed for the entire sample. The use of replicates also ensures that the regional distribution of numbers called is appropriate. This also works to increase the representativeness of the sample.

When interviewers reach someone on a landline phone, they ask to speak with "the youngest male, 18 years of age or older, who is now at home." If there is no eligible male at home, interviewers ask to speak with "the youngest female, 18 years of age or older, who is now at home." This method of selecting respondents within each household improves participation among young people who are often more difficult to interview than older people because of their lifestyles. Unlike a landline phone, a cell phone is assumed in Pew Research polls to be a personal device. This means that, for those in the cell sample, no effort is made to give other household members a chance to be interviewed. Instead, interviewers ask if the person who answers the cell phone is 18 years of age or older to determine if they are eligible to complete the survey (also see Cell phones for more information.)

Back to top

Cell phones

As the proportion of Americans who rely solely or mostly on a cell phone for their telephone services now approaches 20%, the representativeness of telephone surveys based on a random sample of households with landline telephones has come under increased scrutiny. More pollsters and survey methodologists are researching the influence of cell phones on telephone surveying. Public Opinion Quarterly dedicated a special issue to the topic of cell phones in 2007 Cell Phone Numbers and Telephone Surveying in the U.S. The Pew Research Center for the People & the Press decided to call cell phones for our Fall 2008 election surveys and is now including a cell phone sample in nearly all polls.

table

One of the main challenges of surveying cell phone users is drawing a representative sample of this group. Drawing samples for all telephone surveys is now more complicated because of the introduction of cell phone numbers and number portability (i.e., where people can keep their numbers when they move or change service providers or port a landline number to a cell phone). Telephone numbers are assigned different prefixes, which can be used to identify whether the number is for a landline or cell phone, but there are also mixed or shared prefixes that include both landline and cell numbers. In addition, people who forward their calls (e.g., from their landline number at home or work to their cell) may appear as a landline number even when they are actually talking on their cell phones.

Most telephone surveys use the household as the sampling unit because landline telephone numbers have typically been shared among all members living in a household. Once a sample of landline telephone numbers is drawn, a separate selection procedure is used to give all adults living in a given household a chance of selection (such as asking for the youngest adult male or female). However, the situation is more complicated for cell phone users because cell phones are often considered individual rather than shared devices, so the person who answers the phone usually becomes the respondent, whether he or she is the primary user of the phone or shares the cell phone with others. Although some surveyors have experimented with selecting among the users of a shared cell phone, it is still uncertain whether the benefits of this approach outweigh the disadvantages, such as potentially lower response rates. In addition, many people under the age of 18 (and thus not eligible for most national surveys) have cell phones. Substantial time and costs are incurred screening out these ineligible respondents.

Several additional issues arise when identifying the geographic location of a cell phone number. The geographic information that can be derived from cell phone numbers is much less precise than for landline telephone numbers. The boundaries of wireless service areas are often larger than landline service areas. The geographic information is based on the rate center where the phone was purchased, rather than where the person lives. And many people move without changing their cell phone numbers. Based on a comparison of geographic information provided with the sample to that derived from respondents' self-reported zip codes, as many as 10% of respondents live in a different state and nearly 40% in a different county than the sample would predict. This issue is of particular concern for sampling cell phones within a geographic area. Although respondents who do not live in the area may be identified by a screener question, people who do live in the area -- but have cell phones from elsewhere - are likely to be excluded from the survey. In addition, cell phone penetration rates are not available for most counties or states. Because of this, there is no way to select cell phone numbers proportional to their size within these geographic areas.

In addition to the different procedures necessary for sampling cell phone numbers, there are also substantial challenges with interviewing people on their cell phones. Those challenges are discussed in more detail in Cell phone surveys.

Related Reports and Commentaries

Back to top

Oversamples

For some surveys, it is important to ensure that there are enough members of a certain subgroup in the population so that more reliable estimates can be reported for that group. To do this, we oversample members of the subgroup by selecting more people from this group than would typically be done if everyone in the sample had an equal chance of being selected. Because the margin of sampling error is related to the size of the sample, increasing the sample size for a particular subgroup through the use of oversampling allows for estimates to be made with a smaller margin of error. A survey that includes an oversample also typically weights the results so that members in the oversampled group are weighted to their actual proportion in the population; this allows for the overall survey results to represent both the national population and the oversampled subgroup.

For example, African Americans make up about 11% of the total U.S. population, according to the U.S. Census. A survey with a sample size of 1,000 would only include approximately 110 African Americans. The margin of sampling error for African Americans then would be around 10 percentage points. Estimates about African Americans from this survey thus could fall within a 20 point range, which is often too imprecise for many detailed analyses that surveyors want to perform. In contrast, oversampling African Americans so that there are roughly 450 interviews completed with people in this group reduces the margin of sampling error to about 5 percentage points and improves the reliability of estimates that can be made. Unless a listed sample is available or people can be selected from prior surveys, oversampling a particular group usually involves incurring the additional costs associated with screening for eligible respondents.

An alternative to oversampling certain groups is to increase the overall sample size for the survey. This option is especially desirable if there are multiple groups of interest that would need to be oversampled. However, this approach often increases costs because the overall number of completed interviews needs to be increased substantially to improve the representation of the subgroup(s) of interest.

The studies reported below included an oversample in the survey design. Among other groups, these studies included oversamples of young people, African Americans, internet users, and parents of young children. See the "About the Survey" section in each report for details.

Related Reports and Commentaries

Back to top

Regional

Many surveys conducted in the U.S. are not national in scope but instead are designed to represent residents of a single community, city, county, state, or region. These surveys tend to have similar sampling procedures to many national surveys, but the people sampled are limited to the geographic area of interest. For example, residents of states can be sampled using the random digit dialing procedures described for our national surveys. However, there may be considerable error with sampling cell phones in a particular geographic area as discussed in the section on sampling cell phones.

The Pew Research Center has conducted a number of state surveys, especially in the context of upcoming presidential primaries. And, we have conducted some special surveys of metropolitan area residents of Philadelphia, New York and Washington.

Related Reports and Commentaries

Primary poll reports

Other regional reports

Back to top

International

International polling poses special challenges to surveyors. The sampling procedures are specific to each country and may affect what mode is selected for data collection. In addition, for multi-country studies data collection needs to be coordinated across multiple countries and data collection organizations. One of the most important aspects of international surveying is the attention given to how concepts and questions are interpreted across multiple cultures and languages.

Most of the international surveys conducted by the Pew Research Center are through the Pew Global Attitudes Project. The Pew Global Attitudes surveys were launched on a regular basis in 2002. The Project has conducted a series of worldwide public opinion surveys that encompasses a broad array of subjects, ranging from people's assessments of their own lives to their views about the current state of the world and important issues of the day. Depending on the mode best suited to a country, the surveys are conducted by telephone or face-to-face. Most of the national surveys are representative of the entire population of a given country though at times data collection is limited in some way - typically to more urban areas - and results in a survey that is not fully representative nationally.

Back to top

Elites and other special populations

Representative surveys can be conducted with almost any population imaginable. It is common for surveyors to want to collect information from experts or elites in particular fields (such as policy-makers, elected officials, scientists or news editors) and other special populations (such as special interest groups, people working in particular sectors, etc.). The principles of drawing a representative sample are the same whether the sample is of the general population or some other group. Decisions must be made about the size of the sample and the level of precision desired so that the survey can provide accurate estimates for the population of interest, and any subgroups within the population that will be analyzed.

Some special challenges arise when sampling these populations. In particular, it may be difficult to find a sampling frame or list for the population of interest and this may influence how the population is defined. In addition, information may be available for only some ways of contacting potential respondents (e.g., email addresses but not phone numbers) and may vary for people within the sample. If most members in the population of interest have internet access and email addresses are available, the internet often provides a convenient and inexpensive way to survey experts or other special populations.

The Pew Research Center occasionally conducts surveys of opinion leaders, especially those in public policy roles. The opinion of elites is often compared with that of the general public to better determine whether these groups have similar or different opinions. The Pew Research Center has conducted several surveys designed to be representative of a special population including surveying journalists, Muslim Americans, Howard Dean's campaign supporters during the 2004 presidential primary campaign, political campaign consultants and constituent groups from a sample of federal agencies.

Related Reports and Commentaries

Back to top

Why probability sampling

Probability sampling, where a small randomly selected sample of the population can be used to estimate the distribution of an attitude or opinion in the entire population with statistical confidence, provides the foundation for survey research and political polling. The basis of probability-based random sampling is that every member of the population must have a known, non-zero chance of being selected. Probability sampling provides the means by which the margin of sampling error can be calculated and the level of confidence in survey estimates reported. Sampling error results from collecting data from some rather than all members of the population and is highly dependent on the size of the sample.

Sampling Error

We report a margin of sampling error for the total sample for each survey and sometimes for key subgroups (e.g., registered voters, Democrats, Republicans, etc.). For example, the sampling error for a typical Pew Research Center for the People & the Press national survey of 1500 completed interviews is plus or minus 3 percentage points with a 95% confidence interval. This means that in 95 out of every 100 samples of the same size and type, the results we would obtain will vary by no more than plus or minus 3 percentage points from the result we would get if we could interview every member of the population. Thus, the chances are very high (95 out of 100) that any sample we draw will be within 3 points of the true population value.

Also see American Association for Public Opinion Research - What is the Margin of Sampling Error?

 

Nonprobability-based sampling

For some surveys, it may not be feasible or practical to draw a random probability sample of the population. In these cases, various types of nonprobability-based samples may be used, such as convenience, volunteer, purposive, and quota samples. Nonprobability samples are often used for qualitative or exploratory research, such as focus groups or in-depth interviews. Many internet panel surveys, which have increased in recent years, rely on volunteer samples. Some internet panels, however, have been developed to randomly recruit people using another mode so that random probability sampling methods can be used. For nonprobability-based samples, the relationship between the sample and the population is unknown. That means there is no theoretical basis for computing or reporting a margin of sampling error and thus for knowing how representative the sample is of the population as a whole.

Also see American Association for Public Opinion Research - Opt-In Surveys and Margin of Error for a discussion of sampling error in nonprobability-based surveys.

Back to top

1 A small proportion of Americans (about 2%) do not have access to any phone.
2 In practice this assumption is not always correct, as some people share cell phones. But it is still uncertain whether the benefits of sampling among the users of a shared cell phone outweigh the disadvantages, such as lower response rates.