Proven methods for reliable results
Editor’s note: Lee Smith is president of InsightExpress, a Stamford, Conn., research company.
While online research is faster and easier to implement than its traditional counterparts, that doesn’t mean that it’s as easy as it looks. Without understanding the best practices of surveying using this powerful medium, you can fall into making some all-too-common mistakes and pitfalls.
In many ways, the methodologies for conducting online research are very similar to those employed for traditional research. Objectives must be defined (for example, consider the characteristics of your audience); surveys must be designed to encourage participation; and careful thought must be given to achieving meaningful results. Additionally, because of its nature, the online environment requires other factors be considered before the survey is pushed to respondents. This article outlines five of the most important factors for conducting online research. Keep them in mind and you’re almost certain to generate stellar online results.
1. Define objectives - what do you want to know?
Defining what questions you are addressing in your research is a critical component in developing the foundation for a well-designed online survey.
The objective of any survey is to gather the information needed to make a sound decision. Typical questions include:
- What problem needs to be solved?
- What decision(s) are needed to solve that problem?
- What kind of information is needed to make the decision(s)?
- Who can provide that information?
Once these questions are answered, it is easier to come up with a clear, concise statement mapping out the objective of the survey. If the survey has more than one objective, multiple surveys may be a better course of action than asking too much in one survey.
For example, an individual in charge of developing a marketing plan for the coming year for a national restaurant chain could use online research to shape the plan. The marketing director can find out what the restaurant’s customers think of its food, service, atmosphere, menu choices and prices. What’s more, the marketing director would know if those opinions vary significantly according to a number of key demographic variables, including age, gender, education level, income, marital status, presence of children, and/or geographic location. Other considerations might include determining whether the opinions of frequent customers differ from those of infrequent customers.
Before proceeding with a survey, it’s important for that marketer to review objectives. Does it really make sense to talk only to customers of his or her restaurant (at least right now)? After all, feedback from customers is important, but for comparison reasons, it’s important to hear from people who frequent competing restaurants Also, how would the marketer know where his restaurant stands if, for instance, customers say his restaurant lacks enough menu choices?
By casting a broader net and conducting the first survey with a less-targeted audience (adults, ages 18-55, for example), the marketer is able to set some benchmarks for customer satisfaction with comparable restaurants — simply by asking a couple of questions about which restaurants the respondents frequent. Following up with a second survey designed to get more specific input from their own customers, exclusively, is the logical next step. One of the benefits of conducting this type of research online is that the lower overall cost of surveying means marketers can afford to conduct more research with less overall expense.
2. Identify targets - look who’s online now
Nearly two-thirds of all households are online today - meaning that the profile of the average adult American Internet user is rapidly converging with that of the average American (see chart).
 |
These fundamental demographic shifts, in combination with higher respondent cooperation and survey completion rates, make it feasible to migrate a variety of research applications from traditional methodologies to the online environment - with impressive results.
It would be safe to assume that in the restaurant example above, where restaurant-goers were sought, the profile of the typical Web user probably comes closer to meeting the survey’s targeting needs than that of the U.S. adult population in general.
In fact, more and more people, in virtually all types of demographic categories, are using the Web, making it more likely that members of a survey’s target audience will be available online. Still, it’s important to give thought to defining the audience and how to approach them, especially when members of low-incidence populations are sought.
It is also possible to take advantage of an online survey to address issues that respondents might not agree to discuss in a phone or face-to-face interview. For example, recent studies have indicated that people are more willing to answer questions on sensitive issues on the Web than they are in phone interviews. Online research respondents also appear to be more willing to answer open-ended questions than those responding to mail or telephone interviews. Once a baseline is established about the potential survey audience, questions can be developed.
3. Designing the perfect survey - it all depends on how you ask
On the face of it, conducting survey research can seem tantalizingly simple. Ask some questions, get some answers, add it all up and find out what everyone thinks. But it’s not quite that simple. In fact, how a survey is designed may be the single most important factor in determining both response rates and the reliability of what is learned from the responses. Some of the rules of good survey design apply to all forms of research, whether the study is conducted in person, via mail, over the phone or on the Internet. And then there are other considerations that are specific to the Web.
Ideally, how long should a survey be? As short as possible. That reliable piece of survey design advice is especially valuable when it comes to Web surveys. The Web, after all, is the ultimate medium for instant gratification. Users go online, get what they are looking for, and get going. It’s been perfectly tailored for people who are always on the run. Therefore, surveys should be designed for that same mentality.
Of course, as in any situation, there are a few exceptions to consider. If, for example, the survey is targeting a group of people who have a vested interest in the outcome of the research (i.e., employees, members of an organization or customers who’ve made a major long-term investment in a particular product or service), more questions can be asked without the same kinds of concern for response rates or bias. Or if the potential respondents are members of a panel who have agreed in advance to answer questionnaires of a given length, higher levels of cooperation can be expected.
Once the survey length is established, the question order must be set. Tempting as it may be to start with demographic questions and save the “good stuff” for last, it is not advisable to do so. Research has repeatedly shown that people are more likely to complete a survey - and provide honest answers - if they can get right down to business and start answering questions related to what the survey is about at the very beginning. The first question should be the one that captures the respondent’s interest, is easy to understand and easy to answer. Save the drop-down boxes and multiple-answer grids for later.
Experienced researchers know that there is almost always more than one way to ask a question. The best online survey questions are the ones that are the shortest, simplest and most direct. For example, if you wanted to assess customer satisfaction for the restaurant chain from the example above, don’t ask, “How was your last meal at the chain restaurant?” A question like that would solicit mostly single-word answers — “fine,” “good,” “great,” but would not provide any valuable information.
Rather, construct a series of questions that allow for specific feedback about everything from the food and the prices to the service and the atmosphere. Here are a few tips that can ensure the right questions are asked:
- Ask one thing at a time. Surveys can be confusing if they ask questions like, “Did the food and service at Joe’s Place meet your expectations?” Instead, break that into “Did the service meet your expectations?” and “Did the food meet your expectations?”
- Watch out for bias. It can be very easy to telegraph the “right” answers to some questions. For example, by asking, “Given the near-drought conditions facing many cities today, do you think that Joe’s Place should continue its policy of serving each customer a glass of water automatically?” it is likely the response would be the same for nearly all respondents. After all, who wants to advocate wasting water?
- Avoid loaded questions. Be careful to present all sides of an issue when asking for an opinion. “Should Joe’s Place offer a separate children’s menu, or are there enough choices for children on the current menu?” doesn’t give respondents an opportunity to consider other options when it comes to ordering for their children. A better idea might be to ask “Which of the following do you think would help meet the needs of families with children eating at Joe’s Place?” and follow it with a list of several options (including a children’s menu, offering half-portions, etc.) along with directions to select all that apply.
The quality of the questions asked will have an impact on the quality of the answers received.
4. Invitations, incentives and driving the survey home
Most market researchers don’t consider themselves to be salespeople. In fact, the market research community has gone to great lengths to distance research from sales - for obvious reasons.
But the truth is, market researchers do have to sell. Because no matter how well objectives have been defined, how precisely audiences have been targeted, and how adeptly the questionnaire has been constructed, the survey will be meaningless if people do not respond. That requires selling.
Recent years have seen a decline in response rates for most traditional survey methodologies - thanks in part to the proliferation of answering machines, Caller-ID systems, heavy telemarketing and direct mail campaigning. The American public has been telling researchers that they will not be bothered.
Online researchers, however, face a different set of challenges. The same factors that can make soliciting respondents for online survey less intrusive - and therefore, more readily welcomed - can also make the invitations easy to ignore. So it’s important to give careful consideration to both the medium and the message.
Among the most popular methods for inviting people to participate in a survey:
- E-mail - This is clearly the way to go when the audience is well defined, and when there is an available list of e-mail addresses and permission to send them e-mail. Lists can be purchased with qualified names or companies can use lists from their own databases of customers, prospects, Web site visitors, members, employees and other contacts.
- Web site banners, buttons and other links - The obvious choice if the desire is to receive feedback from visitors to a specific site. It is also the best way to recruit for low-incidence populations by issuing an invitation on Web sites that are known to draw an audience similar to the one the survey is trying to reach.
- Pop-up surveys - These are just like the surprise ads that pop up when users visit Web sites, and can be annoying to potential respondents. Pop-ups should be used judiciously, and only on sites that are very strongly related to the content of the survey.
Whichever method is chosen, it is important to make the most of that fleeting opportunity to capture a respondent’s attention and get him or her to say yes to participate in it. Be sure to communicate, right up front, why taking five or 10 minutes out of an already busy day to answer survey questions is a worthwhile exercise. Some generic starting points:
- “We’d like to know what you look for in a family restaurant, so we can provide you with the best possible food, service and value.”
- “With your input, we’ll be able to offer you more of what you’re looking for - in terms of products, service and value.”
- “Your opinion is important to us because...”
Then be as specific as possible. For those creating an e-mail invitation, be as personal as possible, such as mentioning the recipient’s name, any information about recent transactions, and acknowledging any past correspondence. The more that is done to emphasize a known relationship with the recipient, the better the chance that he or she will agree to participate in the survey.
To incent or not to incent - now that is a question! Market researchers have been offering potential respondents incentives for as long as anyone can remember. Years ago, it was a shiny new quarter tucked inside a mail survey with the benevolent suggestion that the recipient give it to a deserving child. Today it can be anything from cash to merchandise, services, discounts and even donations to charitable organizations. Whatever such compensation is called, it is important to note that it is simply another means of answering the respondent’s “What’s in it for me?” The key, of course, is offering an incentive that’s attractive enough to boost response, without it being so spectacular that results are biased because the sample was skewed.
Many researchers refuse to offer incentives at all - citing potential bias (which is nearly impossible to estimate accurately) and budget constraints. But even they admit it may sometimes be necessary to employ incentives to help boost response from low-incidence populations or those who typically don’t respond under other circumstances.
To determine what kind of impact incentives might have on research, surveyors should consider splitting the sample and testing an offer-based invitation against one where no offer was made. While it is almost certain that a higher response rate will be achieved with the incentive, the surveyor can determine just how high that number will be, providing the opportunity to evaluate potential bias issues for future surveys.
5. The home stretch - get what you need, when you need it
Once the survey has been finalized, it’s important to closely monitor results as they arrive. With online surveys, users can perform real-time analyses, from charting to data mining to multivariate linear regressions. What’s more, it all can be conducted directly from the desktop - either at work, on the road or even from home.
Monitoring the results can help marketers better understand the nuances of their surveys and their questions, and helps make for better follow-up surveys. In some instances, if the questions prove to be too open-ended or are not achieving the desired results, questions may be rephrased and the survey re-launched immediately.