Survey Sampling

Sampling refers to the process of choosing a sample of elements from a total population of elements.

Probability vs. Non-Probability Sampling

Statisticians distinguish between two broad categories of sampling.

  • Probability sampling. With probability sampling, every element of the population has a known probability of being included in the sample.
  • Non-probability sampling. With non-probability sampling, we cannot specify the probability that each element will be included in the sample.

Each approach has advantages and disadvantages. The main advantages of non-probability sampling are convenience and cost. However, with non-probability samples, we cannot make probability statements about our sample statistics. For example, we cannot compute a confidence interval for an estimation problem or a region of acceptance for a hypothesis test.

Probability samples, in contrast, allow us to make probability statements about sample statistics. We can estimate the extent to which a sample statistic is likely to differ from a population parameter. The remainder of this tutorial focuses on probability sampling.

Quality of Survey Results

When researchers describe the quality of survey results, they may use one or more of the following terms.

  • Accuracy. Accuracy refers to how close a sample statistic is to a population parameter. Thus, if you know that a sample mean is 99 and the true population mean is 100, you can make a statement about the sample accuracy. For example, you might say the sample mean is accurate to within 1 unit.
  • Precision. Precision refers to how close estimates from different samples are to each other. For example, the standard error is a measure of precision. When the standard error is small, estimates from different samples will be close in value; and vice versa. Precision is inversely related to standard error. When the standard error is small, sample estimates are more precise; when the standard error is large, sample estimates are less precise.

  • Margin of error. The margin of error expresses the maximum expected difference between the true population parameter and a sample estimate of that parameter. To be meaningful, the margin of error should be qualified by a probability statement. For example, a pollster might report that 50% of voters will choose the Democratic candidate. To indicate the quality of the survey result, the pollster might add that the margin of error is +5%, with a confidence level of 90%. This means that if the same sampling method were used to select different samples and if a margin of error were computed for each sample, the true percentage of Democratic voters would fall within the margin of error 90% of the time.

In upcoming lessons, we'll show how to measure precision and margin of error with different sampling methods and with different population parameters (i.e., mean scores, proportions, and total scores).

Sample Design

A sample design can be described by two factors.

  • Sampling method. Sampling method refers to the rules and procedures by which some elements of the population are included in the sample. Later in this tutorial, we will describe three common sampling methods: simple random sampling, stratified sampling, and cluster sampling.
  • Estimator. The estimation process for calculating sample statistics is called the estimator. Different sampling methods may use different estimators. For example, the formula for computing a mean score with a simple random sample is different from the formula for computing a mean score with a stratified sample. Similarly, the formula for the standard error may vary from one sampling method to the next.

The "best" sample design depends on survey objectives and on survey resources. For example, a researcher might select the most economical design that provides a desired level of precision. Or, if the budget is limited, a researcher might choose the design that provides the greatest precision without going over budget. Or other factors might guide the choice of sample design.