Editor’s note: Jeff Rosenblum is director of Internet research, and Chris Grecco is director of quantitative research, at King Brown & Partners, a San Francisco research firm.
The Internet is well on its way to becoming the world’s most compelling entertainment, information and communication tool. Access to e-mail, databases, corporate information, newsgroups, and multimedia content like audio and video files enable users from around the world to access and exchange information more quickly and easily than most could have ever imagined.
Given these capabilities, it is not surprising that the Internet is revolutionizing many industries, including market research. Internet-based research is becoming increasingly popular as companies regularly conduct on-line studies more quickly and more cost effectively than with traditional methods.
However, Internet-based research is not universally approved for all research projects. There are clear and consistent arguments against Internet-based research, and before developing an Internet-based research system, our company, King, Brown & Partners, Inc., found it necessary to address some key questions about conducting research on-line. These included: Are the demographic characteristics of the on-line survey population consistent with those who are not on-line? Are there attitudinal or experiential differences inherent in having on-line access that bias the way respondents perceive products or services? Are there measurable benefits of using the Internet as a data collection tool?
Power and ability
We addressed these arguments against Internet research and weighed them against the Internet’s power and ability to enhance the research process. Confident in the potential of this new research tool, KB&P launched an Internet-based research system in early 1996.
After completing many on-line projects, we have gained an appreciation for the research capabilities of this medium. We have found that the Internet has fulfilled its promise of being an effective and efficient tool and on-line projects have become the fastest growing area of research at KB&P. Some of the benefits we have experienced include:
- Cost and time savings: Compared to traditional research methodologies, on-line studies are conducted with an average savings of more than 40 percent in cost, with commensurate reductions in cycle time.
- Increased accuracy: While some audiences are more difficult to contact on-line, other targets are significantly easier to reach and more receptive to completing surveys via the Internet (e.g., regular Web users, MIS managers, employees for personnel surveys).
- Increased concept testing capabilities: By enhancing the questionnaire instrument with graphics or multimedia elements in surveys, Internet-based research is a more compelling stimulus environment than traditional methods. Respondents who see or hear a new product or advertising concept provide more valid and richer responses than those who simply hear the concept read to them over the telephone.
- Greater survey control: On-line surveys have greater control with regards to interview bias, sampling, skip patterns, awareness testing and stimulus materials.
Case study I: Internet-based research to develop Web sites
Our firm began working with the Discovery Channel in 1996. The first projects that we conducted were related to their Web site, Discovery Channel On-line (DCOL). At the time, DCOL management was interested measuring site usage, perceptions, demographic characteristics and Internet usage of its visitors. Because the audience was exclusively Internet based, on-line research was the most appropriate methodology.
Using a Java-based intercept link on various pages within DCOL, we surveyed a relatively non-biased audience in a fraction of the time and cost that it would have taken using a telephone or mail survey. Perhaps even more important, however, was that the study simply would not have been feasible to conduct via traditional means. Despite the fact that the site garners tens of thousands of visitors each day, trying to find DCOL users randomly by telephone (i.e., "dialing for dollars"), would have been cost-prohibitive.
Using the intercept method, we were able to conduct roughly 600 random interviews in one week. Because all survey responses were immediately downloaded to a back-end database (no keypunching needed), data cleaning and tabulation was completed in 24 hours. Preliminary findings based on the 600 completed surveys were available within 10 days of launching the study.
More impressive than the cost and time savings was the accuracy of the findings. By interviewing respondents while they were in the process of using the site, all of the issues covered in the survey were fresh in respondents’ minds. And, with the large dataset, we were able to undertake detailed subgroup analyses with statistical validity and then report differences with confidence.
Another benefit of conducting Web-based research is the ability to conduct follow-up projects quickly and easily. All information from respondents was downloaded into a proprietary DCOL database. When DCOL management is interested in exploring specific issues related to the research findings or among a smaller group of its users (e.g., nature or science lovers) we can quickly recontact those users via e-mail and invite them to participate in a more targeted quantitative survey or qualitative exploration.
Case study II: Conducting on-line research to explore issues related to off-line services
While Internet-based methodologies are a relatively obvious solution for most projects related to commercial Web sites or on-line services, many questions remain about using the Internet for conducting research related to off-line products and services.
We have found that there are arguments both for and against using on-line research, and that they usually fall into either of two schools of thought:
1. As long as the demographic characteristics and attitudinal differences between on-line users and non-users are isolated, the only difference between the two methods is that one audience has an Internet connection and the other does not.
2. Those with an Internet connection can have substantially different demographic and psychographic profiles, so the research may yield significantly different findings.
We believe that there is no definitive answer and that each project must be analyzed on a case-by-case basis. We find that using a combination of both on-line and off-line methodologies is often the ideal approach. On-line methodologies provide substantial cost and time savings. Traditional methodologies ensure that the on-line data is consistent with proven techniques.
We recently used a dual-methodological approach when conducting research for an off-line Discovery Channel product. Discovery Channel recently purchased The Nature Company’s retail operations. When the purchase was finalized, Discovery management needed to understand customers’ usage, perceptions, and satisfaction of the store and its products for baseline measurements.
Because we had many research objectives to cover and our survey was extensive, we knew it would take respondents a considerable amount of time to complete. We realized that we could achieve a better response rate and save money on interviewing costs by allowing respondents to take the survey on their own instead of being interviewed at the store locations. Thus, we decided to collect the data by distributing postcards to in-store customers, inviting them to complete the survey either by calling a toll-free 800 number or visiting an on-line survey located on a KB&P Web server.
One in five completed the on-line version, which is roughly the same percentage as the number of U.S. households with on-line access. This resulted in savings of 20 percent off of the data entry costs.
The only significant differences we found between the on-line and telephone surveys were on demographic characteristics. Not surprisingly, those who responded on-line were more likely to be male, educated, and affluent, characteristics consistent with the general Internet community. When these demographic characteristics were isolated and normalized, responses to key survey questions on store perceptions and levels of satisfaction were consistent between the two methods.
Thus, we conclude that giving potential respondents the option to complete the survey on-line does not create survey biases. And, in fact, it may decrease non-response bias by including respondents who would not ordinarily respond to surveys through traditional means.
Looking forward
Despite our successes at conducting on-line research, we still don’t feel that Internet-based research is the ideal solution for all projects at all times. We believe that qualitative studies will continue to be better-deployed in-person than on-line for the foreseeable future. We also understand that although quantitative research has proven to be extremely effective when conducted on-line, there are times when it simply will not prove to be the best technique (for example, conducting research with the elderly or the poor).
Still, for most projects when clients are Internet-based or involved with high technology products, we will almost always recommend conducting quantitative projects on-line. For other studies, we will continue to use a case-by-case perspective and weigh the advantages and disadvantages of each approach before finalizing a methodology. As with The Nature Company visitor survey, we may conduct a portion of the research off-line to ensure that our on-line findings are consistent with non-Internet enabled respondents.
Much of our decision-making revolves around one simple axiom: No research is perfect. Think for a moment about telephone research (which has not been recently scrutinized to the same extent as on-line research). We currently rely on interviewers to call strangers in the evening and ask them to answer questions about a product that they may or may not be interested in, at a time that may be inconvenient, without compensation. Oftentimes prospective respondents are either not home, do not want to participate at that time or do not want to spend time on the phone with a stranger. Now, compare that to the typical Internet-based survey, which is often more accurately targeted, is self-administered, contains multimedia stimulus, and can be taken at a time that is convenient for respondents. In this example, which method do you think provides a lower level of bias and more reliable results?
We believe that on-line research often is, and will increasingly become, the most appropriate approach. The future of market research is around the corner. And it will be on-line.