Fulfilling the promise of the Web

Editor’s note: Douglas Rivers is co-founder and CEO of InterSurvey, a Palo Alto, Calif., research firm.

The hype is true: the Web is revolutionizing market research. But will it be a revolutionary step forward or backward? Web interviewing has obvious advantages over traditional telephone and central location interviewing. But much of what is proclaimed to be revolutionary and new about Web surveys is not an advance, but a half-century regression in terms of data quality and research standards.

For instance, quota sampling, a.k.a. “demographic balancing,” has recently experienced a revival on the Internet. No one has made a serious argument in the last half-century that quota sampling is anything other than a cheap and expedient alternative to scientifically valid sampling techniques. Yet quota sampling is the foundation for most current Web surveys.

The return of discredited methodologies, like quota sampling, represents the lure of expediency and wishful thinking over hard-earned experience.

It is not difficult to understand the appeal of conducting interviews over the Internet. Web-based surveys can include multimedia content that is impossible to deliver over the telephone. Turnaround time is reduced from weeks to days or even hours. And by dispensing with interviewers and telephone charges, the cost of conducting a survey can be reduced substantially.

But getting data faster and less expensively is of no benefit if the data aren’t any good. Is it possible to take advantage of the Internet without compromising data quality? I think the answer is yes, but it requires a different approach to the typical Web survey. The purpose of this article is to explain what’s wrong with conventional Web surveys for consumer market research and how a new approach addresses these problems.

Five myths about Web surveys

It is understandable that enthusiasm for a new technology like Web interviewing would cause a temporary suspension of disbelief. But it’s time to assess the problems with Web surveys and solve them, rather than ignore them.

Myth 1: Internet penetration is so high now that there is no longer any issue about the projectability of Web surveys.
Estimates of Internet penetration are all over the map, but even the most optimistic numbers put only about a half of the U.S. population on-line. The most authoritative data, collected by the Bureau of the Census in December 1998, showed that only 42 percent of American homes had a computer and only about a third had Internet access from some point, either at home or elsewhere. More recent commercial estimates (October 1999) put the number of Americans on-line between 65 and 101 million - far below the approximately 229 million who live in households with telephones.

The problem of population coverage for Web surveys is particularly serious for consumer market research. Internet usage is very low in some important population segments. For example, only 12 percent of minority women report accessing the Internet. Web users are still more affluent, educated, and urban than the average American. If you want accurate data about most consumer populations, you shouldn’t restrict yourself to people on the Web.

And of course, a survey can only represent those who have some chance of being sampled. A survey of Internet users is, by definition, uninformative about non-Internet users. If the target population contains few or no people without Internet access, then this isn’t a problem. But for most consumer studies, population coverage is and will remain a serious issue for some time.

Myth 2: It’s just a matter of time before enough people are on the Internet so that Web surveys can replace telephone and in-person surveys.
Okay, the Web population doesn’t look like America, but it will soon, so people argue it’s just a matter of time before the population coverage problem goes away. But there is a more fundamental problem with Web surveys that won’t disappear even if everyone has Internet access: there is no direct way to sample e-mail addresses.

In telephone surveys, households can be selected by random digit dialing (RDD). Because every phone number has exactly 10 digits, it is easy to generate a random set of phone numbers. There are some complications because households can have more than one phone line and different numbers of people, but it is well understood how to handle these issues.

Unfortunately, there is no analogue of RDD for e-mail addresses. Nor is there any listing of e-mail of addresses that could serve as a sampling frame. Even if such a listing existed, it would be considered spamming to send out survey requests to a sample of such e-mail addresses.

Today most Web surveys recruit “samples” of Web users employing banner ads, pop-up windows, opt-in e-mail lists, and similar devices. These are “convenience samples,” chosen haphazardly and with no theoretical underpinnings. It is tempting to believe that Web surveys, because they are conducted on the Internet, are representative of Web users.

The people who accept these invitations, however, are not randomly selected and are demonstrably unrepresentative of Web users. For example, teenagers are much more likely to frequent on-line game sites and chat rooms than to fill out surveys. Web survey takers tend to be older and better educated than average Web users. For some reason, men are more likely to take Web surveys than women, even though the gender gap on the Web has almost disappeared. Thus these samples don’t represent anything in particular.

There are valid ways to sample people on the Web. It is possible to sample visitors to a Web site (through pop-up windows) or customers who have purchased on-line (using the e-mail address they provided when purchasing). And some services, such as Nielsen//NetRatings, have drawn RDD samples of Web users. But none of these approaches gives a valid sample of general consumer populations.

Myth 3: Revolutionary new techniques, such as “demographic balancing” and “propensity score adjustment,” have solved the Web sampling problem.
For individual research studies, Web panel vendors draw samples of their panel members intended to be representative of the target population for that study. One advantage of Web interviewing is that screening can be done quickly and cheaply. Samples can be “demographic balanced” by selecting members to fill various demographic quotas.

This is not a new technique. Before the failure of the pre-election polls in the 1948 U.S. presidential race, the majority of surveys were conducted using quota sampling. Most reputable survey organizations, such as Gallup and Roper, abandoned quota sampling after 1948, but it never disappeared entirely from market research. Quota samples sometimes give reasonably accurate estimates, since the sample is at least representative of the population on selected demographics. But, because it isn’t a probability sample, one never knows how reliable the results are. You can’t compute a margin of error for a quota sample and there is no guarantee that the results aren’t seriously biased. It’s inexpensive, but you get what you pay for.

Some Web survey vendors claim to have developed revolutionary new techniques that can adjust for biases in their quota samples. Unfortunately, just because it is new does not mean it is better. Researchers have long understood that sample elements should be weighted by the reciprocal of the probability that an element was selected. If the sample inclusion probability is solely a function of demographic characteristics, then it is appropriate to weight the sample by the ratio of population elements to sample elements with those characteristics. This is a standard form of non-response adjustment that works so long as the underlying assumption - that selection into the sample depends only upon the selected demographic characteristics - is correct. Most samples require some weighting, but this is not a new technique, and it does not provide a solution for badly skewed samples.

Propensity score adjustment is a newer technique that was developed for handling non-random assignment of treatments in experiments. Some have suggested that it be applied to Web surveys, but the technique is not applicable to this problem. If the “treatments” are participation and non-participation in a Web survey, the Web sample contains no non-participants that could be used to estimate the propensity score. Nor does one learn anything by conducting a parallel telephone survey, since the probability of being in the telephone sample is unrelated to the propensity score needed to weight the Web sample.

Myth 4: Web surveys can be conducted almost for free.
The promise of free or almost-free research is surely one of the most seductive aspects of Web surveys. Without interviewers or telephone charges, the cost of conducting a Web survey can be much lower than a traditional telephone or central location study. The costs of operating a Web survey, aside from recruiting panel members, are mostly fixed and independent of sample size. One server can process thousands or even millions of interviews.

There are two reasons why the prospect of very inexpensive surveying is an illusion. First, for better or worse, people are coming to expect to be paid for providing information on the Web. Some companies will provide free computers or Internet service or even cash for users to provide demographic information and view ads. It is very likely that respondents will increasingly demand compensation for participating in market research surveys. Second, demographic weighting requires very large samples to be effective. Small probability samples can be quite reliable. (A probability sample of size 400, for example, has a margin of error of ± 5 percent, which is adequate for many purposes.) Weighting on many demographics, which the typical Internet survey needs, requires a much larger sample. The costs of providing participation incentives and increasing sample size mean that Web surveying won’t be free and may end up being nearly as expensive as traditional techniques.

Myth 5: Web surveys can contain streaming audio and video.
One of the main appeals of Web surveys is that they have the capability of including multimedia content through a Web browser. Certainly Web surveys have the potential to include visual material that is impossible to convey in telephone interviews and that simulates in-person interviewing, even if the respondent is located thousands of miles away.

Today, most users still have low-speed dial-up connections that are too slow to support video streaming. Even high-quality audio or still images are too large to download quickly for many users. And, heterogeneous software configurations make it impossible to use multimedia even when users have the necessary hardware and adequate connection speeds.

Web surveys are written for the least capable hardware and software in the sample. As a result, most existing Web surveys resemble CATI questionnaires - mostly text with perhaps a few images, but nothing that fully takes advantage of the potential of Web interviewing.

It will be between five and 10 years before a substantial number of households have broadband access to the Internet. Consequently, an alternative to streaming multimedia is required for the foreseeable future.

The promise of the Web

The goal of InterSurvey was to create a Web-based survey capability that fulfills the promise of Web interviewing without sacrificing the data quality essential for reliable consumer market research. Our approach was to separate the problem of sampling from interviewing. Off the Web, we have reliable methods of sampling. On the Web, once we overcome the problem of a heterogeneous hardware base, we have a promising platform for interviewing. The solution is to combine the two.

The first step in creating a valid panel for consumer research is to recruit households using random digit dialing. Each household is contacted and provided with free hardware - a WebTV receiver - and Internet access, as well as other incentives. Each member of the selected households is enrolled in the panel and provided with their own password-protected e-mail account. For particular research studies, panel members are sub-sampled. When a survey is ready, respondents receive notification by e-mail that a questionnaire is waiting for them. Panel members get short surveys (five to 10 minutes in length) on a variety of different subjects several times per month.

The WebTV supplies reliable and consistent delivery of multimedia images to every household by caching audio and video files that are too large to download. When surveys are to contain video, the WebTV unit is notified and downloads the necessary files during off hours. As a result, when a respondent takes a survey, the multimedia content is immediately available and appears to be embedded within the questionnaire. InterSurvey is also the first research panel that makes it possible to interview a representative sample of the entire U.S. population (or any sub-population) over the Web. The sample is representative of the entire population because it uses valid probability sampling techniques and does not exclude households because they lack computers or Internet access. All of the panel’s households can be interviewed over the Web since they have been provided with Web access.

While the expense of providing hardware to every household is significant, the use of a panel design allows the cost to be amortized over the life of the panel. Another advantage is that extensive profile information can be collected. This allows surveys to be targeted at specific subpopulations with minimal screening costs. Surveys can be shortened, reducing the burden upon respondents, while increasing the amount of demographic information collected compared to one-shot surveys.

There are, of course, some drawbacks to panels. The most frequent objections involve the usual questions about panel experience and attrition effects. To combat these possible effects, it is essential to constantly refresh the panel by retiring old members and adding new ones. Potential experience effects are further minimized by varying survey content (so that respondents are not asked repeatedly about the same topics) and minimizing respondent burden (by avoiding long questionnaires). Fortunately, panel experience effects can be assessed by comparing new entrants into the panel with experienced members. One does not need to assume anything about panel experience effects; these can be assessed empirically.

Panel demographics

After conducting a series of pilot experiments in the summer of 1999, InterSurvey began recruiting panel members in September. At the time of this writing (December 1999), 18,661 individual members in 7,986 households have been recruited. The panel is expected to grow to about 250,000 persons by 2001.

Recruitment procedures involve a combination of advance mailings (to households with listed phone numbers), cash incentives, and up to 24 callbacks over a one-month field period. Initial results have been encouraging - almost 60 percent of contacted households agree to participate and overall response rates compare very favorably to those typical for commercial telephone surveys and panels.

Not surprisingly, given the recruitment procedures and high response rates, the panel matches population demographics closely. There are a few groups, such as persons over 75, those with less than a high school education, household incomes under $30,000, and African-Americans, that are under-represented. But overall these differences are fairly modest.

Panel members exhibit very high levels of cooperation and, to date, low levels of attrition. Once a person has been recruited into the panel, they are very likely to complete surveys assigned to them and to do so very quickly. Completion rates for active panel members are about 90 percent, with almost 60 percent of all surveys completed within 24 hours of assignment. Only 5 percent of recruited panel households have dropped out to date.

The future of survey research

Competition with telemarketing is killing telephone interviewing. Although it is still possible with enough time, effort, and expense to recruit a high-quality sample via RDD, we are paying more and getting less with telephone interviewing. Can the Internet supplant telephones as the primary mode of interviewing for consumer market research? Time will tell.

But the question should not be whether the Internet is used for interviewing, but how it should be used. It is possible to use the Web for research without sacrificing data quality. Web-based research needn’t neglect the large numbers of consumers who aren’t on-line. It doesn’t have to abandon reliable sampling techniques. And it can do all of this with multimedia beyond the capabilities of most users’ computers today.