Don't eliminate - migrate!

Editor’s note: Heather Woodward is Internet research product manager at FGI Research, Chapel Hill, N.C.

You have a dilemma. One of your largest and most important research projects is facing potential budget cuts. In the middle of a tough economy, you will not have enough money to do the sizable telephone awareness and ad tracking study that you have successfully conducted for the past two years. What to do? You don’t want to cancel the study and lose the benefits of the research, but a shorter, less involved telephone study or a smaller sample size will not give you enough information to justify the expense of the research. What you really need is the same study you’ve been doing all along. An unsolvable problem? As recently as a few years ago, yes, but in today’s research marketplace, there are solutions for problems like these. As we’ll demonstrate in the following case study, online research tools and methodologies can be used to successfully replace costly telephone studies if carefully designed and properly managed.

Searching for solutions

The following case study is based on a large annual research program for a Fortune 500 telecommunications company (which will be referred to as the telecom company). The study in question was a 12-minute telephone interview with a complex quota structure based on geography, daily completion targets and technology use. In its original format, 30,000 surveys were conducted each day of the year with an RDD telephone sample. A comparable mail study would be as costly as telephone, with postage costs and lower response rates canceling out any benefit gained by eliminating human fieldwork hours. The only feasible option from a cost perspective is the Web.

Of course, there are trade-offs to converting to an online data collection methodology, and these will need to be addressed and understood up front before any decision is made. The primary advantages for the purposes of this kind of study are largely related to cost implications, but certainly not exclusively. Some of these advantages are:

  • With an online study, there are no costs for telephone charges, interviewer time, supervisory or quality control time within a typical phone center environment. Online data collection also dramatically reduces the level of professional project management support that is typically spent interacting with a phone center. In terms of the investment of project management time, the online version would require about half the current level of professional effort compared to the telephone equivalent.
  • In the online world, there are no time-zone constraints or limitations. Interviews roll in at all hours of the day or night, which accelerates fieldwork and allows respondents to complete the survey at a time that’s convenient for them, instead of during their dinner hour or favorite television show.
  • There are options available to you in an online format that are simply not feasible with a telephone study, such as the use of images for awareness measurement (logos, screen shots of advertisements, etc.). Even streaming video can be incorporated, provided the respondent has the appropriate software — like Real Player or Macromedia’s Flash player — to view video images.
  • The Web affords an ideal structure for reporting on project progress, with the capability to make quota reporting and topline data available online in real time without the need for faxes or e-mails back and forth.

Some of the disadvantages are:

  • End users of the research may have concerns about moving to a still relatively new platform for data collection. They may fear that the results will be vastly different from a comparable telephone study, or that the technology is simply too new to be reliable.
  • Certain demographic distributions can be different in an online study, resulting in concerns about skew in certain subgroups. Typically, online data collection results in a younger, more affluent and less ethnically-diverse sample than the U.S. population on the whole.
  • For a study of this magnitude, a great deal of sample is required, and it will need to mirror the demographic makeup of the U.S. population as closely as possible. There is no true equivalent to a random digit-dialing sample online.
  • The most significant trade-off to consider is the projectability of the survey results to the broader U.S. population. The online method most often utilizes as its sample frame an online panel of respondents. Almost invariably, online panel samples comprise individuals who have explicitly opted-in to participate in research studies; RDD frames for the most part reflect true EPSEM (equal probability of selection method) sampling rules, enabling survey results to be reliably projected to the population universe being measured. In most cases, however, careful sampling of the online frame — attempting to replicate the broader universe as closely as possible — will yield comparable results to the online method. This is true so long as the subject of the research is not totally contingent upon Internet access within households.

Making it work

In the fourth quarter of 2002, this is the dilemma we were facing. How could we design this study for the online environment, and equally important, make sure everyone was comfortable with the decision to migrate this project? It was decided that we would conduct a 30-day parallel test, running the project concurrently on the Web and on the telephone, with a period of analysis at the back end to determine the feasibility of continuing with this research in 2003 with the appropriate data collection method, if at all.

The questionnaire was programmed in the Web environment, precisely replicating the complex quota structure and hierarchical logic of the telephone study. Modifications were made to accommodate the differences in question layouts in the new format and references to interaction between the respondent and an interviewer.

The most significant revision that had to be made to the questionnaire was the structure of the many unaided awareness questions (“What company do you think of first when considering…”) included in the survey. This is a relatively simple matter on the telephone, with a prelist viewable by the interviewer, but not to the respondent. On the Web, responses to this crucial question are collected via input to a text box in a Web browser. With the many possible permutations of the spelling of something as simple as a company name, the capturing and subsequent use of the specific responses within the same survey can quickly become complicated, particularly if the skip logic later in the survey requires restoration of these “open-ended” responses. This was managed by using a combination of text fields and text restores to affix total unaided and aided awareness into the database for future reference by the script.

As described before, the sample frame was an online panel of United States consumers, selected proportionally by geography and U.S. Census demographic data, and then weighted after data collection. (Identical back-end weighting was applied to the telephone control group as well, and in both frames, and is used to mitigate the effects of non-response bias. This is a different application to front-end weighting used in sampling.) E-mail invitations and reminders were staggered throughout the 30-day fieldwork window to correspond with the daily dialing quotas for the telephone study, ensuring that the distribution of the completes was spread as evenly as possible throughout fieldwork.

Data collection began for the parallel test the first week of December 2002, and ended the first week of January 2003, allowing for a sample size of about 3,000 for each method.  We began looking at the data after the first week of fieldwork to identify differences between the methods for key data points.

The results

Our first investigations focused on the demographic makeup of the households that participated in the study. Our principal concern was that the online data would return vastly different breakdowns from the telephone data, but in fact, this was not the case. Due to careful sampling at the front end, we were able to avoid significant differences in most of these breakdowns.

Respondent age was broken out into the following categories: 18-19, 20-24, 25-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59, 60-64, 65-69, and 70+. Among these groups, there were only two where the difference between the unweighted Web and phone data exceeded 3 percentage points. Those groups were 45-49, and 70+.

The mean number of people residing in the household on the telephone was 2.65, and the mean for Web was 2.46, a very slight difference.

For the household income breakdowns, the only significant difference was among households reporting an annual income of under $15,000, with the Web data reflecting a percentage 3.74 percent lower than the telephone data. Not a surprising difference, with Internet penetration continuing to skew towards higher income households.

Reaching minorities continues to be a difficulty for online research, which was evidenced by the race and ethnic origin distributions in the data from the Web component of the test. All of our ethnic groups had differences of greater than 3 percent between the Web and the telephone data, with the frequency of minorities completing the interview significantly lower online. This will continue to crop up in online research until the distribution of minorities with Internet access becomes comparable with their presence in the U.S. population. Until then, these issues must be handled with weighting. For the purposes of this study, we weighted the Internet data to the telephone data to provide the most analogous means of comparison.

Gender breakdowns were not hugely different between the two methods, although the difference for both genders was significant. In both methods, a much higher distribution of women completed the interview than men.

The findings from the demographic analysis provided us with what we needed to weight the data appropriately, but the most interesting results of our comparisons came from the ad awareness section of the questionnaire, which is one of the focal points of the research.

This section was designed to gauge recall and sponsor identification of ads for the telecom company and its largest competitors. Each respondent was asked about two advertisements, one for the telecom company and one for a competitor. These ads were randomly selected from a list of ads that were running during the fieldwork window. In the telephone version, text descriptions of the ads would be read to the respondent, who was then asked to indicate if they recalled seeing or hearing these ads. Then those who did recall the ads were asked to identify (unaided) the companies associated with those ads. In the online version, the text descriptions of the ads were presented on the screen for the respondent for the same exercise.

What we found when we compared the data for this section between the two methods was a systematic increase in the online version, both in awareness of the ads and in correct identification of the sponsor of the ads. This was the case for the telecom company as well as for each of the three competitors who were tracked.  For the telecom company, for example, only 28 percent of respondents remembered the ad they were asked about on the telephone, compared with almost 40 percent online.  Of those who remembered the ad, 34 percent correctly identified the telecom company on the telephone, compared with 75 percent on the Web! With the exception of one competitor whose historical performance has been poor in this section, these increases were similarly significant.

We believe the primary factor driving this increase for the online version is the ability of the respondent to see the text on the screen and respond to it in their own time, rather than having to make a split-second decision while on the phone with an interviewer. We may be pulling in some of the people who would report no awareness on the phone because they didn’t have the opportunity to really think about it. In addition, there is the real possibility that our subject matter impacted awareness here. Because the companies we asked about were purveyors of technology-related services, an Internet audience may inherently be more aware of such products and services when compared to the general population. Both factors will require further investigation to quantify the impact on our data.

A happy ending

After the data comparisons were completed, we found that what differences exist between the two methods could be accounted for and worked around. In many cases, there were no significant differences at all. Where necessary and possible, revisions were made to the project structure to minimize these differences (adjustments to quotas) and maximize the new options available (incorporation of screen shots from television ads for ad recall and brand awareness). The project was renewed for 2003, utilizing the new method. The telecom company will be able to conduct research that will meet its business needs, even in the midst of steep budget cuts. Bottom line: don’t let the combination of budget cuts and the increasing cost of telephone data collection spell the end for your research. Your solution may just await you online.

Recommendations for transitioning a telephone study to the Web

  • Identify your universe first. If you need to interview a specific minority group that is largely unrepresented on the Internet, a totally online study may not be for you. Investigate the possibility of a multimode study, with online AND telephone components. However, most studies with the general population or with a client’s customer base can be easily transitioned to an online environment.
  • Pay close attention to your sampling. Duplicating your frame online is one of the most important aspects of this transition. If you have a general population study, make sure your sampling is targeted to U.S. Census data up front.
  • Run a parallel test for at least 30 days with your original method to determine which data points, if any, yield significant differences in the new method. Use this information to make appropriate adjustments to your questionnaire.
  • Keep your questionnaire as short as possible. Online surveys do move faster than telephone, due to the absence of the interviewer/respondent interaction, but even online respondents are subject to survey fatigue, and you may experience high dropout rates after a certain point, and introduce the risk of bias when the respondent grows tired of the survey. If you can, keep your script at 20 minutes or less.
  • Take your time. Because of the ability to field surveys quickly, the inclination to rush the fieldwork process is significant. However, that which is true in the telephone world about respondent availability to complete surveys still holds true for the Web. If you field your study in a three-day window without a weekend, you will systematically eliminate those people who only check their e-mail at that time.  Try to schedule your fieldwork to cover at least part of a week and one full weekend to maximize your representation.