Increasing survey accuracy

Editor's note: Norman Frendberg is president of Consumer Insights, Rochester, New York.

Maximizing survey accuracy is the ultimate goal of every survey researcher. Several approaches to addressing this ongoing challenge follow. In order to minimize survey error, we must first identify the three types of error:

1. Sampling error- This error occurs when we survey only a sample of the population rather than every person, i.e., we may survey 1,000 households rather than the approximate 93 million households comprising the total U.S. population.

2. Two types of non-sampling error-

2a. Observational error- This type of error includes incorrect measurements caused by a variety of factors such as the respondent's failure to recall information accurately (e.g., the last brand bought).

An observational error may also occur in the information processing phase during activities such as keypunching.

2b. Non-observational error- The occurrence of this type of error results from the inability to obtain information from qualified respondents. For example, respondents may be unavailable or refuse to be interviewed. Additionally, there may be potential respondents who are excluded from the survey as in the case of those without phones in a telephone study.

Sampling error

The sampling error can be reduced simply by increasing the sample size. The increases in sample size necessary to reduce sampling error are illustrated in Table 1.

Let's assume that among a random sample of 200 respondents, 20% indicate they "definitely would buy" our new product. We would be 95% confident that this score among the total population is ±6 percentage points, (i.e., between 14% to 26%). Increasing the sample size by a factor of five to 1,000 respondents reduces the sampling error by half to ±3 percentage points. However, for most mail-intercept or phone survey research, such an increase in sample size would drastically increase the study cost. Increasing the sample size to reduce sampling error may fail to represent a cost-effective approach to increasing overall survey accuracy.

Table 1
Sampling Error at 95% Confidence on Sample Measure of 20%

However, decreasing sampling error has the unique distinction of being a measurable source of error in survey research as well as the most commonly understood error.

Non-sampling error

Survey accuracy can also increase as a result of decreasing non-sampling error, although the exact margin of improvement is not measurable.

Reducing non-sampling error can be accomplished by employing a wide variety of techniques. However, creating a comprehensive list of techniques is impossible since a particular method may be suitable for one study, but totally inappropriate for another.

Observational errors

One way to decrease observational error is by enhancing the communication between interviewer and respondent, thereby improving the collection of accurate data.

Photo exhibits can be effectively used to clarify choices for a respondent and aid in memory recall when appropriate. For example, when conducting mall-intercept panty hose studies, respondents are often shown color exhibits illustrating package fronts of different styles (e.g., regular, control top) for the major brands. Respondents can refer to the illustration when asked about brand and style usage, and many times will mention a product by the package color.

In another study, a photo exhibit helped respondents determine the weight of their dogs. The illustration provided information on dog breeds and approximate weights, which served as a visual reference for the respondent. This process furnished helpful information that yielded more accurate data, even in the case of a mixed breed.

Reduction in observational errors can also be achieved in the data collection process. Questions can be designed so that recording information is easier for the interviewers, which results in a higher degree of error-free data.

In the following example we asked respondents, "Have you ever heard of [brand name] ice cream?" The response to this question could be recorded in several ways, two of which follow:

OPTION #1 Interview Instruction:
If "yes," circle answer for each brand respondent has heard of:

OPTION #2 Interview Instruction:
Circle "yes" for all brands respondent has heard of and "no" for those brands respondent has never heard of:

Both methods are functional, but option #2 requires the interviewer to record an answer for each question. The process in this second option is easier, more complete and therefore, probably derives greater accuracy.

Non-observational errors

A key component of non-observational error is the respondent's refusal to be interviewed. One approach to increasing survey accuracy by reducing the refusal rate involves offering a cash incentive. Not only does this technique enhance survey accuracy, but it may actually save money as indicated below. The following data were collected several years ago illustrating the actual cost of screening respondents using a $2 cash incentive for various incidence rates. At a cost of $15/hour, a $2 cash incentive actually saves money below the 30% incidence level.

Cost Analysis by Incidence Rate for Client Cost of $15 Per Hour*

Summary

Although enlarging the sample size does reduce error, this approach may be quite costly for a small return of increased accuracy. Increasing survey accuracy can be best achieved in a cost effective manner by decreasing non-sampling error. Three ways to reduce these non-sampling errors are: use of visual communication aids during the interview process, improving the recording of answers by interviewers, and offering cash incentives to respondents.

* Frederick Wiseman, Marianne Schafer, Richard Schafer, "An Experimental Test of the Effects of a Monetary Incentive of Cooperation Rates and Data Collection Costs in Central Location Interviewing," Journal of Marketing Research, pp. 439-442, November, 1983.