Editor's note: Howard Waddell is president of Decision Resource, a Miami research firm.

With the exception of consumer mail panels, little marketing research in this country is done by mail. Read the sales literature of most full-service research companies and you will learn about the number of phone stations, CATI capability, focus group facilities and so on, but few firms say anything about mail surveys. The fact is, most researchers do not take mail surveys seriously. Until a few years ago, I was one of those people.

Almost any textbook on marketing research will tell you that telephone surveys are generally superior to mail surveys. Lower non-response error is usually the reason given why. It is a common perception that because such a high percentage of people do not return mail surveys, one cannot extrapolate survey results to the entire population being surveyed. It is also perceived that the non-response error of telephone surveys is much lower than that of mail surveys. Perhaps it's time to revisit these issues.

If the non-response error associated with telephone surveys is truly less than that with mail surveys, then the number of different people called by telephone to complete a single interview should be consistently lower than the number of questionnaires mailed to receive a completed response.

There are two factors that determine telephone survey response rates or, inversely, the number of calls necessary to complete a given number of interviews. The first is the proportion of people called who can be reached. The second is the proportion of people reached who agree to be surveyed (and do, in fact, complete the interview).

With more working couples and pervasive use of telephone answering machines, it is becoming increasingly difficult to reach people on the telephone. That only 40 percent to 45 percent of consumers can be reached by phone on a first try should surprise no one. (Reaching the right person within the household is yet another problem, but we'll ignore that here.)

With the increasingly high volume of telemarketing calls, including those that start off with a few "survey" questions, the percentage of people who are even willing to participate in phone surveys is no higher than about 50 percent.

The arithmetic is simple. If calls are made to l,000 households with good phone numbers, 450 might be reached on the first call. With a survey participation rate of 50 percent, one would expect 225 completed interviews. As additional calls are made to the people who could not be reached, more interviews will be completed. But there is a point of diminishing returns because the refusal rate isn't going to change.

If each household were called three times, and it is assumed that the likelihood of getting through remains at 45 percent, one would establish contact with 834 households and complete 417 interviews. The response rate, therefore, would be 41.7 percent. (The upper limit of the response rate, achieved when all households are contacted, is equal to the 100 percent minus the refusal rate, or 50 percent.)

A critical question must be asked. Can response rates better than 41.7 percent can be routinely achieved with mail surveys? The answer is that they can. With mail panels, response rates of 70-80 percent are not uncommon. Even without mail panels, response rates for consumer mail surveys of 50 percent can be routinely achieved if the surveys are done properly.

The single most important aspect of doing mail surveys properly is focusing on the cost per response received, not the cost per questionnaire mailed. Taking that approach can mean the difference between a 10 percent response rate and a 50 percent, 60 percent or even 70 percent response rate. It should also be pointed out that doing the things that are necessary to minimize the cost per response will, more than likely, maximize the total response rate as well.

Buyers of telephone surveys know that it is not uncommon for their research suppliers to ask for four, five or even six times as many telephone numbers as completed interviews required. If research buyers believe that the non-response error for telephone surveys is lower than for mail surveys, they should limit their supplier to only two times as many telephone numbers as required interviews.

Additional benefits

In addition to a lower non-response error, mail surveys have additional benefits relative to telephone surveys.

  • In telephone surveys, interviewers can unintentionally influence how people answer questions. It is not difficult to imagine a variety of scenarios in which a male respondent might answer questions one way if they were asked by a woman, and another way if they were asked by a man. (The same could be true when the sexes are reversed.) Telephone research buyers should ask to see their data cross-tabbed by sex of the interviewer.

Interviewers can also influence responses in a more direct, but still unintentional way. Consider an actual incident that happened a few years ago. A manufacturer of windows was conducting a telephone survey of homeowners to measure, among other things, aided awareness of their brand. I received one of the calls. The interviewer read me a list of company names and asked me to tell her which ones I had heard of. The list was rather lengthy and, after about 10 names, she and I were both growing a bit impatient because I knew none of the companies. Then we came to Storm-Tite (a fictitious name). "Surely, you have heard of them!" she said with considerable irritation in her voice. Although I had heard of Storm-Tite and said so, would I have admitted my ignorance to this interviewer if I had not? While this was, presumably, an isolated incident, it could not happen in a mail survey.

  • Knowing they have no anonymity, respondents to telephone surveys are more likely to color their answers or not answer some questions at all. Consider the following example. If, in a survey, people are asked how much money they make, some might exaggerate the figure if they feel they should be making more. Others might not answer the question if they suspect their answer will not be treated confidentially.

Logic, alone, would suggest that these response problems would be worse if the respondent believes the interviewer knows who he or she is. When a question is asked over the telephone, the link between the answers given and the person giving them is obviously there. In a mail survey, the link may or may not be there, depending on the construction of the questionnaire. However, when it is not there, one can expect more candid answers.

  • With mail surveys, respondents have time to think about their answers. During a telephone interview, the respondent has no opportunity to review any subjective answers to determine if they accurately reflect his or her opinions. Asked to indicate the importance of a variety of factors influencing purchase behavior, the mail survey respondent can review all the answers and make changes, if necessary. Anyone who has reviewed mail survey questionnaires knows that changes to answers are not uncommon. Reflection by the respondent during a telephone survey is generally not possible.
  • With mail surveys, it is possible to include illustrations or detailed descriptions of products or services. With phone surveys, it's not.
  • Mail surveys are usually less expensive than telephone surveys. With a 50 percent mail survey response rate, the direct cost to acquire a completed four-page questionnaire as part of a nationwide consumer survey would usually be between $4 and $6. The cost to complete an interview in a comparable telephone survey would certainly be no less, and probably more.

There are, of course, situations where mail surveys are inappropriate. First, mail surveys are not suitable for unaided or top-of-mind awareness measurement. Also, because mail surveys generally take more time than telephone surveys, they do not make sense when results are needed quickly. A telephone sway can be done overnight. Mail surveys cannot.

Undeserved reputation

To conclude, it is fair to say that mail surveys have an undeserved bad reputation that is perpetuated in marketing classrooms, even today. Also, few sellers of research have any experience or expertise in the subject. That's not surprising since there is little demand for mail surveys and the margin dollars for the suppliers are less for a mail survey. If mail surveys cost more than telephone surveys, there might be more marketing emphasis on the virtues of the mail survey.