Editor's note: Bill Eaton is vice president of Creative Research Systems, Inc., Petaluma, Calif.
It's difficult to open a marketing magazine these days without seeing details of a new survey conducted via the Internet. It's the latest and greatest toy for all those researchers who love the black box approach to research, where the medium is more important than the message.
Like most other methods there are advantages and disadvantages to surveys conducted via the Internet. Let's start with the advantages.
They are extremely fast. You can post a questionnaire on a Web site and start to receive replies almost instantly from Net surfers. Within a few hours you will probably have an adequate sized sample. This involves less effort, but is no quicker than a phone survey with an adequate number of stations.
The second advantage is that there is almost no cost involved. Once the questionnaire is posted on a Web site there are no further printing, postage or long-distance costs. Compared to traditional methods where every extra interview costs money, this is extremely attractive.
So Internet surveys are fast, economical - and deadly!
Internet surveys and the Golden Rules of Research
The inherent problems of Internet surveying are rooted in the three basic rules that I learned in Research 101 many years ago. Rule #1 is that you cannot ask men for the opinions of women, Republicans for the beliefs of Democrats or users for the thoughts of non-users. In statistical terms, a sample represents the universe from which it was drawn - and no one else. And who makes up the Internet universe?
The universe you are surveying on the Internet is a well-educated, high-tech and high-income (and even nerdy) one. Respondents may represent the Steve Jobs and Bill Gates of this world (albeit with less cash) or even the Kevin Mitnicks but not much else. So rule #1 is strike #1 - you cannot use the Internet for anything other then surveys of very specific, Internet-based groups
Let's take a look at rule #2. This says you go look for your sample - you don't let your sample come to you. When one of the major causes of concern in survey research is the low and declining percentage of people willing to undertake a survey when requested, what are we to make of volunteers who surf Web sites? That they are not representative of much, other than Net surfers! In addition, volunteer respondents usually have stronger than average opinions for or against a subject - so strong that they are prepared to seek out someone to unload these opinions on. While some Internet surveys do seek out specific respondents, most simply pick up opinions from Web surfers who cruise that site. So your volunteer respondents almost certainly have stronger opinions than your average consumer. Strike #2!
Rule #3 - you can only vote once. Tammany Hall is not allowed in survey research. This is normally controlled by careful fieldwork and back-checking of paper questionnaires or with phone numbers selected to avoid duplication. With Internet surveys, quality control is almost non-existent - most of the Internet survey programs cannot detect repeat voting, and even those that do can be disabled by smart hackers, who take a perverted intellectual pride in trashing safeguards of this type.
If you don't believe me, ask InfoWorld what happened to their March 1997 Readers Choice survey - conducted for the first time by Internet. The results were so skewed by repeat voting for one product that the entire survey was publicly abandoned and the editor asked for readers' help to avoid the problem again. And it's not the first time. Last year another survey of two well known competing software programs had a similar result - one side loaded the vote to the point that it became blatantly obvious.
What is even more dangerous is the knowledge that there almost certainly are many more fraudulent surveys that remain undetected. No one knows how many, but I hate to think how many business decisions were based on the results of such surveys. So Strike #3!
Can Internet surveys work? Maybe - if you can control for the three problem areas. One, if there are strong controls to avoid multiple polling. Two, if the surveys are properly distributed to a genuine sample and the results properly back-checked. Three (and most important), the results must not be extrapolated beyond Internet users.
Surfin' USA?
A 1995 survey by the Electronic Industries Association gave modem ownership as 16 percent. Obviously, not everyone used a modem on their PC so the regular Internet usership level was probably about 10 percent. One year later a survey by the Wirthlin Research Group claimed that 20 percent or about 42 million adult Americans had used the Internet but this figure was basically an "ever used" figure. The claimed regular user total was about 14 percent. Probably the most recent survey was conducted in Spring 1997 by CommerceNet/Nielsen. It revealed that 22 percent of Americans had "recently used the Internet." However about a quarter of this group had not used the World Wide Web, indicating that they were probably using the Internet for E-mail only. Allowing for minor over-claiming about the topic of the month and the growth between the surveys probably no more than 12 percent of Americans are regular users of the Web.
Except for television at 98 percent, the telephone (with ownership at 96 percent) is the closest thing to a universal appliance in American homes. Even in the most rural and poorest homes it's about 90 percent, so you get as good coverage of America as you are likely to get.
Now ask yourself: If telephone ownership in the USA were limited to 12 percent of the population, would you use it for market surveys? I doubt it and even then a phone conversation is a one-on-one operation - you don't have to surf AT&T at random until you find a survey waiting to be answered.
Although computer prices have declined (and continue to decline) dramatically, a minimalist set up is still around $1,000 - compared to a $10 monthly rental for a phone. PC industry leaders talk of $500 Net PCs, but none have reached the market yet and even $500 is expensive in comparison to a phone.
Computers also require some technical training, and there is a core sector of American society that is unable or unwilling to undertake this. Studies (not conducted by Internet) have shown that, even when available at no cost, some people simply do not want to adapt to the new technology. Some of this follows the Diffusion of Innovation theory where the introduction of a new product follows the classic bell curve. Currently computer ownership has just reached the early majority phase and Internet users are still in the early adopter phase. Obviously these levels will continue to grow. If, however, growth follows the pattern of other new technologies (phones, TV, etc.) it will be at least a generation before Internet surveys are reasonably representative and they may never become fully representative, as the final group of non-adopters may remain significantly large due to cost and/or inability to adopt to the new technology.
Haves and have-nots
Will Internet surveys work better in the future? As usership spreads to representative levels, they may be usable for the information generation but given the current progress of American society towards two classes of information haves and have-nots, there may remain a significant class who cannot be reached by this method. So for now (and for the foreseeable future) the law is three strikes and you're out!