Editor's note: James Watt, Ph.D., is professor of communication sciences, University of Connecticut and vice president for product development, Swift Interactive Technologies, Inc.
To paraphrase a long-distance carrier's commercials: if you haven't done Internet survey research, you will. Because there are some very powerful reasons why you should consider using the Internet for quantitative survey research.
First, there is the speed with which a questionnaire can be created, distributed to respondents, and the data returned. Since printing, mailing and data keying delays are eliminated, you can have data in hand within hours of writing a questionnaire. Data are obtained in electronic form, so statistical analysis software can be programmed to process standard questionnaires and return statistical summaries and charts automatically.
A second reason to consider Internet surveys is cost. Printing, mailing, keying and interviewer costs are eliminated, and the incremental costs of each respondent are typically low, so studies with large numbers of respondents can be done at substantial savings compared to mail or telephone surveys.
Of course, there are some offsetting costs of preparing and distributing an Internet questionnaire. These costs range widely, according to the type of Internet interviewing used. The Internet survey is always cheaper by a substantial margin than a telephone survey, is only slightly more expensive than a mail survey for surveys with fewer than about 500 respondents, and becomes increasingly less expensive than mail for more than 500 respondents.
An often overlooked benefit of Internet survey research is the ease with which an Internet survey can be quickly modified. For example, early data returns may suggest additional questions that should be asked. Changing or adding questions on the fly would be nearly impossible with a mail questionnaire and difficult with a telephone questionnaire, but can be achieved in a matter of minutes with some Internet survey systems.
Internet questionnaires delivered with the Web have some unique advantages. They can be made visually pleasing with attractive fonts and graphics. The graphical and hypertext features of the Web can be used to present products for reaction or to explain service offerings. For respondents with current versions of Netscape or Internet Explorer, the two most popular Web browsers, audio and video can be added to the questionnaire. This multimedia ability of Web-delivered questionnaires is unique.
Appropriate populations for Internet survey research
Not all populations are candidates for Internet survey research. The general consumer population is often a poor fit, because fewer than 10 percent of the U.S. households regularly use Internet services (although more are connected, many are infrequent users). There is also a potential problem in the general population with reluctance to use computers, as well as some fear of the intentions of those who use the Internet to ask questions. This fear has been fueled by sensational media accounts of "cyberstalkers" and con artists who prey on Internet users.
However, there are some exceptions to this broad statement. For example, computer products purchasers and users of Internet services are both ideal populations. Both populations are likely to have very high connectivity and neither are likely to have high levels of cyberphobia. Consumers who have purchased products or services using the Internet are not likely to be fearful of Internet surveys. Web-delivered questionnaires can be made part of the purchase transaction (for customer satisfaction studies, for example), with attendant high levels of motivation and participation from the respondents.
Business and professional users of Internet services are also an excellent population to reach with Internet surveys. Over 80 percent of businesses are currently estimated to have Internet connections, with the number expected to reach 90 percent by next year. Business users are likely to have experience with the Internet and to recognize its convenience in replying to questionnaires. In business-to-business research, product and service demonstrations are often crucial. Web-delivered questionnaires, with their ability to weave text and audio-visual demonstrations into the questionnaire, are an excellent way to reach a business population.
Internet questionnaires can frequently be used to supplement traditional methods of collecting questionnaire data. The portion of the target population that uses the Internet can be reached cheaply and quickly with Internet questionnaires, while those not connected can be reached by mail or telephone. Supplementing traditional survey methods provides some immediate cost savings, as well as a migration path toward fuller Internet interviewing in the future as the connectivity of the general population increases.
Internet samples
Internet samples fall into three categories: unrestricted, screened and recruited.
In an unrestricted sample, anyone on the Internet who desires may complete the questionnaire. These samples may have poor representativeness due to self-selection of the respondents. The rate of participation (completion rate in traditional survey terms) is generally low. Unrestricted samples do have utility in applications like point-of-sale surveys for Web commerce, Web site user profiles, bingo card-like customer interest surveys, or recruitment of potential focus group members.
Screened samples adjust for the unrepresentativeness of the self-selected respondents by imposing quotas based on some desired sample characteristics. These are often demographic characteristics such as gender, income and geographic region, or product-related criteria such as past purchase behavior, job responsibilities or current product use. The applications for screened samples are generally similar to those for unrestricted samples.
Screened sample questionnaires typically use a branching or skip pattern for asking screening questions to determine whether or not the full questionnaire should be presented to a respondent. Some Web survey systems can make immediate market segment calculations that assign a respondent to a particular segment based on screening questions, then select the appropriate questionnaire to match the respondent's segment.
Alternatively, some Internet research providers maintain a "panel house" that recruits respondents who fill out a preliminary classification questionnaire. This information is used to classify respondents into demographic segments. Clients specify the desired segments, and the respondents who match the desired demographics are permitted to fill out the questionnaires of all clients who specify that segment. This approach is somewhat less flexible than using tailored screening questions that are unique to the survey being conducted, and also raises questions about the representativeness of respondents who are willing to spend the time to fill out many different questionnaires for different clients.
Recruited samples are used for targeted populations in surveys that require more control over the make-up of the sample. Respondents are recruited by telephone, mail, E-mail, or in person. After qualification, they are sent the questionnaire by E-mail, or are directed to a Web site that contains a link to the questionnaire. At Web sites, passwords are normally used to restrict access to the questionnaire to the recruited sample members. Since the makeup of the sample is known, completions can be monitored, and follow-up messages can be sent to those who do not complete the questionnaire, in order to improve the participation rate.
Recruited samples are ideal in applications that already have a database from which to recruit the sample. For example, a good application would be a survey that used a customer database to recruit respondents for a purchaser satisfaction study. Another application might be the construction of a consumer panel for tracking research. The convenience of filling out a short Internet survey as compared to a paper diary that must be mailed back should increase the participation rate and the accuracy of the answers.
Different methods of conducting Internet surveys
E-mail questionnaires. The questionnaire is prepared like a simple E-mail message, and is sent to a list of known E-mail addresses. The respondent fills in the answers, and E-mails the form plus replies back to the research organization. A computer program is typically used to prepare the questionnaire, the E-mail address list, and to extract the data from the replies.
E-mail questionnaires are simple to construct and fast to distribute. By showing up in the respondent's E-mailbox, they demand immediate attention.
However, they are generally limited to plain text, although graphics can be sent as E-mail attachments that are decoded separately from the questionnaire text. Many standard questionnaire layout techniques, such as creating grids of questions and scale responses, cannot be done in a visually attractive way in E-mail. There is no check for validity of data until the whole questionnaire is returned, so there is virtually no opportunity to request that the respondent reenter bad data. The respondent may damage the questionnaire text in the process of responding, making automatic data extraction impossible and requiring hand coding of damaged responses. In addition, all question skips are carried out by the respondent, who is given a set of instructions embedded in the text ("If you replied 'yes' to this question, skip to Question 23"). This can result in illegal skip patterns, which may require more hand recoding, or result in missing data or rejected questionnaires.
Converted CATI systems. A software translator program takes questionnaires programmed in the CATI vendor's questionnaire construction language and translates them for distribution over the Web. The Web server may be located in the research supplier's facility, or time may be rented from a service bureau that has the CATI system installed. The Web server is linked to a database that receives the respondents' replies and stores them.
Converted CATI systems have the good sample and quota management typical of CATI programs. They also inherit the ability to set up complex skip patterns for screening and to adapt to respondents' replies. They can do data verification at the time of entry, and request reentry of illegal data immediately. Converted CATI systems provide quick migration to Internet interviewing for current users of a particular CATI system and permit reuse of existing programmed questionnaires. In some systems, progress of the Internet survey can be monitored while data is being collected, with some intermediate data extracts available for a fee (daily summaries, for example).
On the negative side, the CATI systems on which these Internet survey products are based were designed for a telephone interviewer working from a computer screen. Respondent screen formatting is somewhat limited as a result. In addition, the CATI languages frequently do not take advantage of the Web's ability to present graphics and audio-visual material. The researcher is locked into a single CATI system provider's technology, which is only a small disadvantage if the researcher is already using that CATI system, but a larger one if the researcher is not. Finally, the converted CATI systems are expensive to purchase and use.
Converted disk-by-mail systems. These are similar to converted CATI systems. Disk-by-mail systems provide a questionnaire construction tool that creates a program file on a floppy disk that the respondent subsequently runs on a personal computer. The program presents the questions on the computer screen and records the answers on the program floppy disk, which is then mailed back to the research organization. The converted disk-by-mail system adapts the questionnaire for presentation via the Web, and provides a data management program to record the answers provided by the respondents.
Converted disk-by-mail systems have the same skip pattern management and data verification advantages of converted CATI systems, with the addition of more flexible questionnaire construction tools that include graphical and audio/visual material. However, they inherit the limitations on quota management of the disk-by-mail approach, which is designed to present a single questionnaire to a single respondent. They typically require that the user manage his/her own Web site and install and maintain the software on that site.
Web CGI programs. In this approach to Internet survey research, each questionnaire is programmed directly in HTML (the presentation language used by the WWW) using a computer script language such as PERL or a programming language such as Visual Basic. The programmed questionnaire is placed on a Web server at the client's location or on a server located in a service bureau. The program uses the Common Gateway Interface (CGI) of the WWW to place respondents' replies into a database. Database queries can be programmed to give periodic reports of the data to date, including statistical analyses.
The CGI programming approach is the most flexible of all. Complex question skips and data verification and reentry can be achieved, and programming languages can use the full capability of the Web. Since all questionnaires are custom programmed, Web CGI programs are not tied to a proprietary CATI language, or a single technology vendor. Database operations and queries can be programmed to adapt to virtually any special reporting need of the researcher.
This flexibility comes with a cost, however. Since questionnaires and database operations are essentially custom computer programs that must be created and debugged by highly-trained programmers, they are expensive. The computer languages contain no special tools for tasks like screening, quota management and question skip pattern management, so programming these features in each questionnaire further increases the cost.
The CGI program must be placed on a Web server system to distribute the questionnaires and collect the data. This can be the research client's Web server, or a server provided by the research supplier. If the survey is placed on the client's Web server, time for programming and debugging can be difficult to schedule. Large corporate sites often require several administrative approvals before any modifications of the site can be made, and technical staff are frequently leery of allowing an outside programmer to place a program on their site.
Web survey systems. These are software systems specifically designed for Web questionnaire construction and delivery. In essence, they combine the survey administration tools of a CATI system with the flexibility of CGI programming. They consist of an integrated questionnaire designer, Web server, database, and data delivery program, designed for use by non-programmers.
In a typical use, the questionnaire is constructed with an easy-to-use questionnaire editor using a visual interface, then automatically transmitted to the Web server system. The Web server distributes questionnaire and files responses in a database. The user can query the server at any time via the Web for completion statistics, descriptive statistics on responses, and graphical displays of data. Data can be downloaded from the server at any time for analysis at the researcher's location. The questionnaire construction and data display programs reside on the user's computer system, while the Web server is located in a survey technology provider's office.
Web survey systems include tools that allow non-programmers to create complex questionnaires that are visually appealing. The complexity of skip patterns and data verification that can be achieved approaches that of the CGI programming approach. Users do not have to maintain a Web site or database, so there is less disruption of clients' Web sites and computing facilities. Sample quota control is as good as that provided by converted CATI systems. In addition, tools to personalize questionnaires with database information (like inserting the respondent's name in a questionnaire delivered to a restricted sample respondent) and to add graphics and sound without programming are often included.
Web survey systems typically have a lower cost per completed interview than converted CATI, converted disk-by-mail, or CGI programs, although they are more expensive than E-mail surveys for small surveys (under 500 respondents). The lower cost results from the efficiencies of using software tools designed specifically for Web use, and from the cost-sharing of Internet access costs and hardware costs that a central server system provides.
Like converted CATI, converted disk-by-mail, and CGI programming, Web survey systems use the more passive Web retrieval for questionnaires. E-mail, although it has many limitations, is more immediately attention-demanding. Also, for current users of CATI systems, migration of existing questionnaires to Web survey systems is more difficult than migration to a converted CATI system. Questionnaires must be manually cut and pasted into the Web survey questionnaire constructor.
The kind of Internet survey technology to use for a project depends on the circumstances of survey. This grid summarizes the strengths and weaknesses of each.
E-Mail Converted Converted Web CGI Web Survey
CATI Disk-By-Mail Programs Systems
Ease of Excellent Fair Good Poor Excellent
creation/
modification
Ease of Access Poor Fair Good Excellent Excellent
to Preliminary (with extra programming)
Data
Sample Quota Poor Excellent Fair Excellent Excellent
Control (with extra programming)
Data Validity Poor Good Good Excellent Excellent
Checks (with extra programming)
Demand of Excellent Good Good Good Good
Respondent's
Attention
Personalization of Fair Fair Poor Excellent Excellent
Questionnaires (with extra programming)
Conversion of
Existing
Questionnaires Fair Excellent Good Good Good
Expertise Low High Moderate Very High Moderate
Required by
Questionnaire
Creator
Cost per Inexpensive Expensive Expensive Very Expen. Moderate
Completion to Inexpensive
Definite advantages
Internet survey research is not appropriate for all populations and all projects, but for many applications it provides definite advantages. For populations already using the Internet, or for "early adopter" populations, quantitative survey research on the Internet can give faster results at a lower cost than traditional methods. Internet questionnaires can be used to supplement traditional quantitative data collection methods as a way of reducing the overall cost of a project or as the beginning of a migration to all-Internet surveys in the future.