Editor’s note: David Ashley is a market research specialist at the U.S. Small Business Administration, Washington, D.C.

“It is a capital mistake to theorize in advance of the facts.” Easy for Sherlock Holmes to say, but then, he never administered a questionnaire. Although writing a good questionnaire takes time, following some guidelines can make the process easier and more productive.

Given that survey rules are seemingly numerous, how can you make sense out of writing an effective questionnaire? Remember, a questionnaire is best developed by first dovetailing your secondary research with primary research. Primary research might include a focus group or interviews. After you have laid the foundation, begin drafting the questions - all the while keeping true to your research design and research objectives set forth at the beginning.

  • First, it’s best to be clear and concise about what you are asking. Be vigilant in ensuring that the questions are sufficiently related, thus enabling comparative analysis between and among concepts. When writing a question, ask yourself what would be a natural follow-up question. Is that follow-up question in the questionnaire? Stand-alone questions will likely limit your analysis to descriptive observations when generally an analytical approach is more revealing.
  • Testing the questions and topics in a focus group is desirable. A hodgepodge of unstructured questions might yield unhelpful results and discourage survey participation since the questionnaire might appear to be lacking a clear structure or natural flow. Conducting a focus group beforehand will help identify the most effective ways, or words, to use when asking questions - especially sensitive questions. Consider asking general questions at the beginning and invasive/sensitive questions towards the end. The respondent is best led into the questionnaire gradually and naturally. Pelting the respondent with thought-provoking questions too early might discourage the respondent from completing the questionnaire. Also, you should benefit from having done sufficient participant screening to avoid using valuable space asking unnecessary questions such as “Do you own a cell phone?” if you have obtained a mailing list of cell phone owners for the survey.
  • Ensure questions are grammatically simple, brief, and use the respondents’ core vocabulary - as identified perhaps in a focus group, interviews, or secondary research. Avoid jargon and acronyms as they will likely only be of interest to the questionnaire author. Testing the questionnaire before it’s distributed will better your odds that the respondents will interpret the questions the same way once the final draft is ready for distribution. Consider the question, “Where do you get your allergy shot?” One might respond, “At the doctor’s office” and another respondent might reply, “In my right arm.” Keep in mind that one’s frame of reference is driven by their individual, and unique, experiences, background, culture, etc.
  • Be wary of asking respondents to guess what they might do. Asking, “Will you read Consumer Reportsbefore you buy your next car?” might lead you to respond “yes” because you might feel that’s what you should do. However, when in the throes of being wooed by the proverbial unctuous sales rep, you might not actually read Consumer Reportsbefore you buy. Instead ask, “The last time you bought a car did you review Consumer Reports?” It is generally more revealing to ask what people actually did rather than what they might do. This approach clarifies what actually happened as opposed to what people say will happen. Consider that during election polling, spinmeisters are generally more interested in “likely voters,” as measured by those with a history of voting, rather than members of the general public who might say they will vote but wind up not voting.
  • Avoid asking questions the respondent is not qualified to answer or questions asking the respondent to recall specific examples when a general recollection will suffice. For example, “How much did you pay for milk last month?” could be rephrased, “The last time you bought milk, did it cost more or less than the previous time you bought milk?” In this case, the respondent’s specific recollection might be less accurate than the memory of overall milk price - as compared to the previous milk purchases. Correct general data is preferable to incorrect, guessed-at, specifics. Also undesirable is asking two questions at once. “Did you receive fast and friendly service?” sadly appears to be a mainstay of restaurant surveys. Well, the waiter might have been as friendly as the day is long - which coincidently was the length of time it took to be served. It might be difficult to answer this question accurately.
  • Other question pitfalls are both overstating assumptions made in the question - “Are you against cigarettes because they can kill people?” - and asking respondents to answer on behalf of the general population - “Will the public support the president’s proposal?” You might believe cigarettes are harmful to your health but you might not feel they “kill people.” On the latter question, how can you be expected to speak on behalf of the public at large? Please note however, if you are gauging the respondent’s perception about the public, it might be appropriate to ask them to speak about the public - but make sure your intention to measure perception is clear. Avoid using the respondent’s perception about the public as factual data about what the public might do.
  • Asking for information obtainable from other sources, such as internal databases, might have a negative effect on your response rate. How does it feel after you have dutifully punched in your credit number over the phone only to be asked by the agent upon getting through, “May I have your credit card number?” Respondents appear to have dwindling tolerance for being asked questions the questioner should know or for being asked the same question twice. You might want to couch the same questions in a different light if you are trying to measure questionnaire reliability - a measure of the consistence of answers a respondent provides.

Lies, lies and damned statistics

Reviewing the four types of data will position you to maximize the effectiveness of your questions. The proper use of these data types will greatly enhance your analysis and final product. Use nominal (the least sophisticated data), ordinal, interval, and ratio (the most sophisticated data) questions throughout the survey. A cross-section of the four types of data will maximize your data analysis options. Remember that more sophisticated data can be revised down to lower-level level data during the analysis phase - but the reverse is not advisable. The hint here is to use as much higher-end data as appropriate.

Scaling is also an important component of creating an effective questionnaire. When possible you should use similar scaling throughout the questionnaire lest you risk confusing the respondent. Similar scaling facilitates data analysis and makes it easier for the respondent - which in turn generally increases the respondents’ willingness to complete the questionnaire. Although they are useful, be aware that odd-numbered scales might result in a disproportionate number of middle, or neutral, responses. For highly controversial issues, a middle response such as 3 on a 1-to-5 scale might be the respondent’s finessing of what might likely really be on his mind. Questions that are deemed truly more prone to a neutral attitude (which can be somewhat clarified during the secondary research and focus group phase) might not be negatively affected by giving the respondent the neutral option as there is ostensibly no need for the safe haven of a neutral response. There is no ironclad right or wrong on the odd vs. even issue; just be aware of the implications of your choice. Generally, the best choice between the odd vs. even argument lies in the question itself. Some questions are better asked with an odd scale and others with an even scale.

Use interval scales to measure attitudinal responses and ratio scales whenever appropriate as they can always be converted to interval or ordinal data for crosstabulation. Although open-ended questions have advantages, summarizing response can be tedious and difficult. Generally, open-ended questions should be placed towards the end of the questionnaire and should be used sparingly since they require more time to complete and are more difficult to quantify. Probed open-ended questions such as, “List three specific attributes you like about this product” might yield more useful information than unprobed questions such as, “Please comment on the product.” The latter question is so open that you will likely get a wide array of answers - some more useful than others. Giving the respondent some framework within which to respond will generally result in the respondent’s completing the open-ended part of the questionnaire more precisely.

Life is tough enough, so avoid using a scale beyond 10 points. Ten-point scales are often used since respondents can generally relate to such scales relatively easily. Nine- and eight-point scales are not generally desirable since one man’s “7” might be another’s “6” even though they share the same opinion, they are simply interpreting the same scale differently since there are too many gradations within the scale. Seven- and five-point scaling introduce the neutral response issue. If you are using an odd scale, the middle response is less obvious to the respondent on a seven-point scale than on a five-point scale – although most often you will find five-point scales. A three-point scale might be used for ordinal response questions such as “small,” “medium,” and “large” and a two-point scale might be useful for semantic-extremity responses such as simply “yes” or “no.” Scaling is also dependent on the level of attitudinal diversity in your sample. Generally the more diverse the respondents’ opinions, the broader the scale should be. However, the scale should not be so large as to cause too much differing interpretation as to the intervals within the scale. Often, through a focus group or secondary research, the researcher can gain a general understanding of what scales are appropriate. No one scale is necessarily right or wrong, but some scales are better than others given certain circumstances. The appropriate scale depends on the nature of the question itself and the information you hope to glean from the responses. Ten-, six-, five-, and four-point scales are generally the most common.

Make sure the questionnaire is attractive and has a consistent appearance throughout. Start with a statement informing the respondents of the survey’s purpose. Remember to ask for participation rather than assuming they will participate. Be careful not to include too many fancy fonts, underlining, and bolding. Using plenty of white space will make the questionnaire uncluttered and numbering the questions will facilitate responding.

Incentives and follow-up calls/e-mails are effective ways to increase questionnaire response rates. An introductory letter/e-mail will alert the respondent of an impending survey. Obtaining an appropriate sponsor or affiliation might improve your standing with respondents and increase their likelihood of completing the survey. Personalizing the questionnaire and offering a monetary incentive or other gift also help increase survey response rates.

Don’t despair

A well-written questionnaire will increase your response rate. Be aware that you will most likely discover that a questionnaire you thought was perfect will have holes in it once your respondents complete it. However, don’t despair, it happens to the best survey researchers around! It’s difficult to predict precisely how respondents will react to and interpret your questionnaire. Consequently, you might want to revise your next questionnaire to account for issues raised in previous questionnaires. Finally, remember these two key snippets - keep the questionnaire as brief as possible and test it with an appropriate group before it’s administered.

Now send it out and hope for the best!