Avoid the Field of Dreams fallacy

Editor’s note: Lawrence Kilbourne is director of professional services at WebSurveyor Corporation, a Herndon, Va. research firm.

“If you build it, they will come,” is a famous line from the 1989 movie Field of Dreams that describes how many people feel about online surveys. Unfortunately this “Field of Dreams fallacy” dooms many online survey efforts to poor response rates, bad data and recriminations over what went wrong.

There’s no question that online surveys are popular because of their ease of use and the speed with which they can be used to gather data. Unfortunately many find that putting a survey online doesn’t automatically translate into higher response rates or better data. The cautionary message here is that the ease by which online surveys can be constructed should not mislead you into thinking that creating and conducting a good survey is therefore easy. The good news is that you can avoid these problems by taking a few important steps at the beginning of your online survey effort.

“Begin with the end in mind,” is one of the central messages of Stephen Covey’s The 7 Habits of Highly Effective People. It applies equally to the creation and execution of successful surveys. Good surveys begin with a goal: to obtain an answer to an important question through the accumulation of sound data. That question (notice I did not say that constellation of questions) focuses the survey, ensures that the survey questions are relevant, and provides a story line around which the data can be gathered. Remember: Good surveys seek to clarify through the accumulation of data, not to simply accumulate data and hope that something interesting jumps out from it.

Rule #1: begin with the end in mind.

Focused surveys have an additional benefit: they are generally short - meaning that they can be completed easily and quickly - and don’t leave the survey taker wondering, “What in the world are they trying to figure out?” By making it easy for respondents to take the survey, you increase response rates and obtain more accurate data. When creating surveys it is always important to bear in mind not only “What am I trying to determine?” but also, “How easy or difficult am I making it for the person I want to take this survey?” A common source of poor survey response rates is a failure to consider the latter question when engaged in the process of survey creation.

Rule #2: make it easy to take.

In many aspects, the Field of Dreams fallacy really boils down to a failure to keep in mind those who will be taking your survey. It’s a bit like narcissism - call it survey narcissism. If I focus solely on what I hope to obtain by way of data, I completely lose sight of the fact that unless the survey appeals to potential respondents, I may end up with no responses and no data.

Rule #3: make your survey appealing.

There are a number of components that go into accomplishing this. First, unlike the movie Field of Dreams, people will not come simply because you have created a survey. They must be invited to take it, and the form and content of the invitation are critical to the success or failure of your effort. For our purposes we are going to assume that the invitation will come via e-mail. The simplicity, speed and ubiquity of e-mail set it off decisively from other media that could be used to issue the invitation (e.g., direct mail). The initial question to be taken up is what form will the e-mail take: text or HTML? There are pros and cons to each, so let’s briefly survey them.

HTML e-mails have a natural advantage over text e-mails because they are graphical. Well-done graphics appeal to the eye. In our information-saturated world, few of us will take the time to read a 1,000-word - or even a 100-word - invitation. The ubiquity of e-mail cuts both ways, and a common reaction to the growing stack of e-mails we find in our inboxes is to quickly delete those that don’t convey a compelling message easily grasped.

Graphic HTML e-mails overcome this by actually showing the message and how to respond. Putting big radio buttons within the e-mail invitation saying “Click here to take the survey” makes it easy for the recipient to respond (remember Rule #2?).

To be sure, HTML e-mails have their drawbacks as well. The most obvious is that many e-mail programs block graphics, so there is a certain percentage of the universe that will not receive your survey invitation. Beyond that hurdle is a second: the growing adoption of spam filters that react to warning signs, like e-mails with attachments or with graphics embedded in them. The impact of spam filters can be mitigated to a large degree by the judicious use of language within the HTML e-mail (e.g., avoiding red-flag phrases like “Win a free…,” as well as the use of symbols like !!! or $$$). There are even spam testers available that provide a scoring of how likely your e-mail is to be rejected by the most common spam filters.

The primary advantage of text e-mails is that they are unlikely to trigger spam filters (assuming you don’t use the red-flag phrases and symbols noted above). That said, however, there’s little else that makes them appealing. Because they are composed of text, they must be comprehended in a linear fashion: reading line to line. This is inherently a longer process than comprehending something visually. The risk is that before the reader has fully read your message, he or she may simply decide to move on. As a general rule, text survey invitations are best reserved for internal audiences where a higher response rate is assumed.

Rule #3a: Make your e-mail invitation appealing.

No matter how appealing your e-mail invitation, unless a recipient actually opens it your cause is lost. Under the onslaught of solicited and unsolicited e-mails, many today have adopted a survival strategy of quickly scanning the subject lines in their in-box and simply deleting those e-mails that are suspicious looking or unappealing.

Rule #3b: Make the subject line to your e-mail appealing.

This is more art than science, but some basic rules exist that can reliably guide us. First, begin with the end in mind (remember Rule #1?). The whole purpose of the subject line is to entice someone to go one step further and open the e-mail. So the subject line must be compelling - literally as well as figuratively. “Please take my survey” is not likely to garner many responses. “Survey invitation: We’d like your opinion on…” is going in a better direction. The best way to gauge a subject line’s appeal is to try it out yourself, on friends or business associates in an actual e-mail. Ask them how it appears in their e-mail window (does it show up in its entirety, or only in part?), and whether they would open it to read more.

Second, keep it accurate. Do not equate deceptive subject lines with compelling ones. You first risk the wrath of the recipient once they open the e-mail and discover that its message does not match what was promised in the subject line. But more importantly, you run afoul of the federal CAN-SPAM Act, which provides for criminal as well as civil penalties for e-mailers who use deceptive or false subject lines in their e-mails. It’s not only a best ethical practice to be honest and straightforward in your subject line, it’s the law!

To this point we’ve succeeded in creating a survey that is user-friendly, as well as an invitation that is appealing and a subject line that is compelling. Our ballpark looks built, to continue the analogy we began with, will they come now?

Even with all that has been done it is likely that response rates will still be poor. The lesson here is that for a survey invitation to be successful, in addition to everything above, there must be something in it for the survey respondent. Call it a quid pro quo, if you want to be fancy, or just a fair deal. Either way, there must be some incentive for an individual to complete your survey. You are asking people to take time out of their busy day to provide you with information you deem important or perhaps critical. What are you prepared to offer in response for their time and effort?

Survey incentives can take many forms, and there isn’t time or space here to go into them in detail. But some basic guidelines can be laid out. First, incentives don’t necessarily have to cost a great deal. Use your creativity: can you offer a service or a discount on a purchase in lieu of cash? Second, even if you opt for a cash incentive it doesn’t have to cost you an arm and a leg. Third, incent the first x number of people who respond. Telling people in your e-mail invitation that the first 100 or 200 respondents will receive a small incentive drives up response rates and gets people to take the survey immediately rather than think about taking it later. Finally, use your graphics in the e-mail invitation to show the incentive, as well as talking about it in the text.

Once people understand that you are willing to fairly compensate them in exchange for their time and assistance, then you can finally answer the question “Will they come?” in the affirmative. Hence our final rule: Give people a reason to take your survey.

Make it easy

Online survey tools have made it easy to construct surveys and collect data. By following the four rules outlined here, you can make it easy to get the data you’re looking for - and be assured that if you build it, they will come.