Designing a questionnaire that dives beneath the surface

Editor’s note: Gillian Humphreys is a vice president and a board director of NFO CFgroup, a Toronto research firm. Joanne McNeish is acting director, marketing research at Canada Post Corporation, Ottawa.

Before diving right into the topic of questionnaire design, we would first like to set the stage by describing our overall philosophy to research. We believe that a good researcher, and indeed a good research user, should be a diver who plunges beneath the surface of the information gleaned from customers.

Of course, the realities of time and budget constraints often dictate how deeply we can dive and what equipment we require. Sometimes we need a mouthpiece and mask to skim the surface; other times we’ll become deep sea divers with full scuba equipment. However, it’s imperative before asking your customers a single question that you ask yourself many questions. This leads to the most important part of the research process.

Our focus in this article is on customers, those purchasers of products with whom we already have a relationship. They could be business customers or consumers.

Setting objectives

This first step in any research project is the written statement of clear research objectives. This is critical in developing a good working relationship between the ultimate user of the research and the research consultant. Beginning with unclear objectives can end in wasted time, effort and money, ultimately leading to dissatisfaction with the outcome of the research. In addition, the choice of data collection method, the questionnaire content, and the analysis plan are all driven by these objectives. Our guidelines for researchers are:

• Don’t expect a survey to make a decision for you. It will simply give you information enabling you to make a better decision than you would without this knowledge.
• Research cannot correct market conditions, share levels, customer awareness or perception; it can only measure them.
• Be sure you know what decision you’re trying to make before embarking on the research process.
• Focus on your information needs. Don’t be tempted to expand your list to include things that might be “nice to know,” but are irrelevant to the issue at hand.
• The fewer and more specific the objectives, the better.
• Recognize that it’s unlikely that any survey can answer all your questions.
• Agreeing on what action will be taken and on what basis before the research starts will help clarify the objectives. In fact, defining the action standards will increase the odds that the research will in fact be used to make decisions.
• Always consider the time and budget available, as well as the ability to take action on the information gathered.

In some circumstances, the best choice is not to do any research at all. For instance, when the business decision has already been made, or must be made before any results can possibly be available, or when the available budget is simply inadequate for the type of research required. And surveys can’t replace your need to think through your problem and decide what to do.

Finally, to ensure understanding and satisfaction with the outcome of the research, we recommend that you provide the objectives in writing and gain agreement from all those who will be involved in the process.

Designing the questionnaire

General considerations

• What is a questionnaire? A questionnaire is a set of questions designed to generate the data points necessary for accomplishing the objectives of a research study. It provides standardization and uniformity in the data-gathering process so each respondent sees or hears the same words or questions and, where an interviewer is involved, that person asks identical questions. A lack of standardization and uniformity can lead to non-sampling errors and impede your ability to project the answers to the population you’re evaluating. Moreover, you increase the risk of making a wrong decision or taking a wrong action based on the research.
• How many questions? Once objectives are set, you can start formulating your questions. Is there a “right” number of questions? As with the objectives, the questionnaire should be short, to-the-point, and effective. Typically, we believe we should be asking for no more than 15 minutes of a customer’s time. To do this, we always keep the objectives on hand when designing or reviewing a questionnaire to ensure each question actually relates to a specific objective. This way, you don’t waste your customer’s valuable time with nice-to-know questions.
• How should it flow? There should also be a logical flow to your questions. A questionnaire should be viewed as the launching pad for a conversation between the interviewer and the respondent or, in the case of self-completed questionnaires by mail or the Internet, between your company and the respondent. Not only is it very frustrating for a respondent to make this conversation disjointed, but it can also influence the quality of the information gathered.

The chart below gives you a general overview of typical questionnaire sequencing. The key to effective question sequencing is ensuring success for the respondent as he or she moves through the questionnaire. First, you must get the right person answering the questions and early on give him or her the sense that “This is easy, I know the answers to these questions.” Then you can begin to progress to questions requiring more thought on the part of the respondent. Finally, consumer or business demographic information that’s not a necessary part of screening the respondent should come last. These questions are of little interest to the respondent (but of great interest to the researcher), but they signal respondents that the interview is almost over.

Remember, most respondents want to cooperate and will generally try to answer any question, however silly or irrelevant. But the fact that you can ask almost any question doesn’t mean you should.

One other important consideration with the questionnaire is the process that must take place after the interview is complete. The respondents’ answers must be simple to tabulate, fast to check for completeness, easy to translate into clear data elements for analysis, and simple to transfer back into findings responding to the objectives of the research.

How to use qualitative research to refine the questions

In a world of increasingly faster decisions, it’s tempting to bypass the research process and move directly into the development and execution of a quantitative study without the benefit of qualitative research and/or a pretest of the questionnaire with your target respondent.

This would be a mistake. Using qualitative research before quantitative helps ensure you’re targeting the appropriate people, using meaningful language on the questionnaire, and can substantiate numerically the motivations developed at this stage.

You can use qualitative research simply to pretest the quantitative questionnaire so you can observe and record the unspoken reaction to it. This allows you to make improvements and changes to the questionnaires before you begin the data collection process. However, at an earlier stage in questionnaire development, qualitative research can best be used to help you:

• hear and understand the language of the customer;
• educate yourself to an unfamiliar environment, in particular, needs-satisfaction, usage situations and problems;
• gain insights into topics that might otherwise be impossible to pursue with structured research methods;
• identify the sequence of information most appropriate for the respondent;
• provide contextual information about the market that can be substantiated numerically in the subsequent quantitative study.

You can also conduct qualitative research after the quantitative to help explain your findings and provide additional insights into the “hard numbers.”

How to word the questions

Communicating clearly is even more important in the wording of a questionnaire. The ability to gain accurate insight from the data depends largely on using simple words, asking questions with specific, rather than general, references, and speaking in the language the respondent typically uses in the situation being explored.

An added concern in Canada is the need to conduct surveys in English and French at a minimum and in more languages if the company’s target group includes an ethnic population or operates in foreign markets.

There are three general question types where mistakes are most often made:

1) Non-specific questions. There are some words that are used so often that we believe we understand their meaning. In fact, there meaning depends more on the reader’s or listener’s interpretation.

A question asked like this, “What media do you use regularly?” leaves the definition of “regularly” to each respondent for interpretation.

Note also that the use of the word “media” may not be appropriate with a consumer respondent, but is clear and meaningful to a respondent who makes media-buying decisions.

Asked for a specified time period, you control and understand the answer to the question, “Which media have you used in the past 12 months?”

2) Leading questions. A leading question leads the respondent toward a specific response because of the wording or the structure of the question itself. If you truly are interested in obtaining information, rather than simply supporting your own opinion, you must not lead your respondents in any particular direction. Even if you’re using your survey results for advocacy, as support in a legal conflict, or for promotional purposes, your credibility will be undermined if the questions are leading.

A leading question would be, “Over the past year, Company A has taken steps to ensure that all courier shipments are delivered by 10 a.m. Would you say that Company A courier deliveries now generally arrive earlier than they did a year ago, or not?”

A non-leading version of this would be, “Compared with a year ago, would you say that Company A courier deliveries now generally arrive earlier or later (ROTATE EARLIER/LATER) than they did, or would you say there is no difference?”

Presenting the words “earlier” first and “later” second to some respondents and presenting the others with the reverse further decreases the likelihood of leading your respondents toward any particular view.

3) Sensitive questions. Respondents are often willing to reveal quite intimate details about themselves when questioned, but only if you create a trusting environment. This means the questions must be appropriately written, the interviewers well-trained, and the interviewer given sufficient time to establish rapport with the respondent. An experienced, well-trained interviewer will approach the interview situation with confidence. Not only does this create a rapport and put respondents at ease, but it also encourages an environment in which, if a question is clearly not deterring the interviewer, the respondent is likelier to answer.

Interviewers who are uncomfortable asking a respondent for sensitive information, such as financial data, are much likelier to meet with resistance from the respondent than interviewers who treat this type of question as if it’s of no special importance to them.

Creating a sense of trust is particularly important in the case of Web surveys. Clearly, respondents must be assured their information is secure. Sensitivity to some extent is in the eye of the beholder, with respondent perceptions of who is asking the questions, and for what purpose, influencing willingness to cooperate. This is particularly true in the case of Web-based research where the challenges in signaling these important pieces of information are much more apparent.

In an e-mail or on a Web-based survey, the respondent often lacks the normal cues to the identity of the company, the country from which they’re operating, and the security that their personal information is not being shared without their knowledge. Unlike telephone numbers or addresses, respondents know there is no directory of e-mail addresses.

Some businesses have specific rules prohibiting employees from completing surveys. Fortunately, there are very few of these companies.

Detailed competitive information is probably the hardest type of data to obtain from a survey. There is a separate branch of marketing research called competitive intelligence. Competitive intelligence focuses on the collection of information about companies rather than the behavior and attitudes of individuals or organizations.

For instance, it’s unrealistic to expect a customer to share with an interviewer the details of pricing arrangements they might have in place with your competitors, even if the information is being collected by a neutral third party. You can sometimes gather this type of information from a mystery shopping project, combined with secondary research from Web sites, for instance, rather than an individual.

One question you should never ask

A marketing research questionnaire is a vehicle through which you can ask almost any question. However, there is one question you should never ask. Research can reflect the image of your organization. Never, as part of the research, ask a respondent if a sales representative could call on him or her or try to sell that person a product. People then feel tricked, and their opinion of the organization or brand drops accordingly, along with their opinion of legitimate marketing research.

The bias question: survey sponsorship

Although, undoubtedly, the identification of your company in customer surveys improves your response levels, it can also create a positive or negative bias in responses. Bias in research terms is exactly that - an unknown impact on the outcome. In some cases, where, for instance, you’re asking customers to compare you or your products with your competitors’, it’s clearly inappropriate to identify yourself if you want a clean, unbiased assessment.

For Web-based surveys, it’s beneficial to have a recognizable survey sponsor since there are often no visual clues as to who is asking the questions and what will be done with the information. However, in most cases, it’s more appropriate to have a strong visual identity for, and support from, the research company that you’re using rather than for your own company.

Exploring the impact of incentives

In the U.S., most surveys offer incentives, in part because once this practice has been introduced, the more respondents come to expect a reward and are less willing to cooperate without it. We have largely avoided this situation in Canada where respondents still provide information for free.

Why do respondents participate in surveys?

• the interview is an opportunity for social interaction with a sympathetic listener;
• respondents are curious about the interview experience;
• respondents have a sincere desire to help companies develop better products or services.

Typically, in Canada, we offer incentives only for long or complex surveys, Internet surveys, or business surveys. The main advantages to offering incentives are the improvement of response rates, hence representativeness, and the creation of good public relations, particularly when survey sponsorship is transparent and revealed.

There are disadvantages, however, namely the introduction of bias (e.g., are those who participate different from those who don’t?) and cost, particularly if you are surveying a large customer base.

The most common forms of incentives are money and prize draws. For the latter, you must conform to Canadian law, which is quite stringent, particularly in Quebec. The literature suggests that small dollar amounts are the most compelling, and, contrary to intuition, a larger monetary amount does not necessarily improve response rates. In fact, it often reduces response rates to a level equal to or lower than no incentive at all.

How frequent should customer contact be?

Your customers are your most valuable resource, so it’s important not to over-utilize them. No matter how good the questionnaire, too many requests will reduce the response rate, and will also reflect poorly on your organization.

Customer contact from a large organization can often be so frequent it’s annoying. For example, we discovered that the contract signer of a large retail organization had been contacted seven times in one year to complete a survey of one sort or another. It seems different groups were conducting research on different products at different times of the year. On the seventh request to participate in research, the client insisted on talking with the marketing research manager to explain his problem. He commented that, while he had been happy to provide information through a questionnaire, he was not planning to do so in the future because there was never any noticeable change in the areas he considered important. Needless to say, we have improved our structural approach to research and worked directly with this customer on the issues of importance to him. Luckily, in this scenario, the respondent let us know that he was dissatisfied with the research process. What impression are you leaving with your customers?

The problem is, you are often selling a number of different products to the same company. There are different points of contact within a customer company that are necessary to consider to get to the user rather than a contract signer or other gatekeeper.

Large companies are often inundated with requests for information and often from the wrong, or inappropriate, person. On the other hand, interviewers often ignore smaller companies because their dollar value is not high. It’s much easier, however, to get to the decision-makers in smaller companies, and they are typically more willing to participate in surveys.

Do people lie on surveys?

Marketing research has as its basic tenet the anonymity of the respondent. It provides objective information on the group rather than the individual. Database marketing aims to sell products to specific individuals. In the ideal world of the marketer, the merging of databases with marketing research would benefit the customer and the selling organization by allowing for:

• a true understanding of the value and potential of a segment;
• the best way to communicate with the customer;
• meaningful enhancements to products specific to individual customers.

Since marketers are interested mainly in the most profitable customers, and customers know this, they are increasingly concerned with the way their personal information is being used. They don’t understand the distinctions between the two disciplines, and they’re either not seeing the benefits promised to them or they’ve come to expect more benefits over time (e.g., more points, greater discounts).

Canadians are beginning to react by reclaiming control of the situation. Their ability to influence governments has resulted in the passage of privacy legislation (Bill C-6). And consumers are using technology, such as answering machines and anonymity software (call screening) to provide an additional layer of privacy protection.

Most importantly, they are:

• refusing to participate at all (declining response rates);
• refusing to participate without an incentive;
• refusing to provide certain information;
• refusing to provide accurate information.

Despite these limitations, we still find remarkable consistency when repeating studies. So, if people lie on surveys, they do so systematically - an unlikely hypothesis.

Why spend time and money designing an effective questionnaire?

Representativeness depends mainly on two things: proper sampling procedures and good response rates. It’s important to understand, though, that even if a finding is statistically significant, it may have no real meaning or significance, nor help you understand a good approximation of the truth of the product experience.

Customers are less likely to refuse an interview and more willing to give honest opinions if:

• they are the right person to answer the questions;
• they are assured of confidentiality;
• they are approached in a professional way;
• the survey is brief;
• the questions are well written;
• the questionnaire flows in a logical fashion.

Successful project

Once you’ve determined how best and how far to dive beneath the surface and what “equipment” you’ll need for the dive, then you can carry out a successful research project, including a well-designed questionnaire.