Demographics and online survey response rates

Editor’s note: Kurt Knapton and Steve Myers are both officers of e-Rewards, Inc., a Dallas-based online sample provider.

Response rates are vitally important to survey-based research studies because the level of error and the studies’ findings are ultimately linked to the response rates. The Council of American Survey Research Organization’s (CASRO) definition of a research survey response rate is “the ratio of the number of interviews to the number of eligible units in the sample” (CASRO 1982). It is important to calculate response rates accurately because they are one measure of the potential bias in the research data, with a high response rate indicating a lower potential bias. Alternatively, when low response rates occur, a non-response bias may exist, whereby there is a systematic difference between those who do and do not respond to a survey measurement instrument. If and when non-response bias is present (e.g., non-respondents differ significantly from respondents) then results can be false or misleading, and results cannot be generalized to the entire population being studied.

Response rate improvement methods

Since low response rates can produce unreliable research results, a fair amount of research literature exists examining response rates over time. Generally, response rates have been found to be declining, not only in traditional research modes such as direct mail and telephone research (Bickart and Schmittlein, 1999), but more recently in online research as well (Sheehan 2001). Because lower response rates can lead to non-response bias, achieving higher response rates is an imperative for the researcher.

A number of research studies have examined response rates across multiple research modes (e.g., direct mail, telephone, Web), and there is a significant amount of research in the marketing and social science literature regarding methods to increase response rates (Dillman, 1978). The two methods shown to be the most effective are: 1) providing currency incentives, and 2) attempting follow-up/reminder contact. It has been repeatedly shown that providing financial incentives as a persuasive motivator for reluctant respondents is a viable and effective technique for increasing response rates. For example, Goetz, Tyler and Cook (1984) concluded that financial incentives increased response rates and showed no differences in the demographics of the incentive and non-incentive groups. Shettle and Mooney (1999) reached a similar conclusion, while Mason, Lesser and Dillman (2003) found that non-response error may actually be reduced when financial incentives are used in student population studies. Additionally, in a meta-analysis of 38 studies reported in the 1970s and 1980s, Church (1993) concluded that a small financial incentive increased response rates by 19 percent. McDaniel and Rao (1980) compared the accuracy of incentivized respondents vs. non-incentivized respondents and found that the incentivized group provided more accurate information and was more diligent when completing their questionnaires. Independent of financial incentives, it has been consistently shown that sending reminder notifications following the initial survey is also an effective way of increasing survey response rates. For example, Dillman (1978) reports that reminder notifications increase response rates among mailed surveys. King (2002) investigated response rates and data collection strategies for Web-based employee surveys and determined that reminder notifications resulted in an increase in company response rates, regardless of industry type or company size. In the same study, King estimates that online response rates will be 8-15 percent lower when organizations do not send a follow-up reminder. Finally, Kanuk and Berenson (1975) examined over 75 articles that addressed increasing mail survey response rates and found that follow-up contact and the use of currency incentives were the only two methodological procedures that had any empirical impact on response rates. This data suggests that proven methods to increase response rates exist and may be used by researchers as an effective way to diminish potential non-response bias.

Online research and non-responders

The inherent advantages of online research coupled with the lower response rates being garnered via traditional research modes such as mail and telephone has led to an overall industry endorsement of Web-based surveys. As online market research becomes a larger portion of the research mix, the question arises whether patterns of non-response observed historically in traditional research modes are also present in online market research. Furthermore, would such patterns of non-response tend to occur in the same demographic groups that have been found to be generally under-represented on the Internet as a whole?

Since the authors were unable to locate any research literature devoted to answering the above questions, the authors prepared this study to provide researchers with information that can be used to appropriately account for, and manage, patterns of online survey non-response if and when they exist.

Two separate hypotheses will be tested:

H1: Patterns of non-response within an online panel will tend to exist in the same demographic categories that have experienced patterns of non-response in traditional research modes, such as mail and telephone.

H2: Patterns of non-response within an online panel will tend to exist in the same demographic groups that are known to be under-represented on the Internet vs. the general U.S. population.

Non-response patterns in traditional research modes

Several key findings have been observed in studies investigating patterns of non-response among phone and mail survey respondents of differing demographic characteristics. Dillman noted in 1978 (corroborating Suchman and McCandless, 1940) that non-respondents generally tend to have less education and are older. A number of recent studies have also shown that gender has an effect on response propensity. Specifically, it has been found that females are more likely to respond to a mail questionnaire than males (Collins et al, 2000). One study reports that only 31 percent of males responded to a mail questionnaire as compared to 49 percent of females (McCabe et al, 2002). Other studies, such as Wardle, Robb and Johnson (2002) have found that affluence is a factor; the more affluent the respondent’s household, the greater the percentage of survey questions answered. To summarize these observations among traditional research modes, non-responders were found more likely to be: less educated; older; less affluent; male.

The online population vs. the general population

A substantial amount of research literature exists that is devoted to how the U.S. Internet population compares with (and differs from) the overall U.S. general population. Five separate studies are cited below.

In 1996, two different studies found that younger, better-educated, and wealthier males were over-represented in the Internet-based population (Bonchek et al, 1996; Kehoe and Pitkow, 1996). It is important to note, however, that the Internet population has normalized substantially since 1996, with some vestiges of over-representation fading in subsequent, more recent studies.

A nationwide Harris Interactive survey conducted by telephone with a sample of 2,038 adults in February and March of 2002 showed that the profile of the U.S. online adult population can differ from the overall general adult population in the following ways (Figure 1): the online population is biased toward the more affluent; the online population is biased toward the better-educated; the online population profile is looking more like a cross-section of all adults, up to, but not including, those over 65, who comprise 16 percent of all adults but only 5 percent of those online.

A fourth study was based on the U.S. Census Bureau’s Current Population Survey conducted in 2001 which included approximately 57,000 households and more than 137,000 individuals across the United States (Figure 2). The study findings were presented by the National Telecommunications and Information Administration (NTIA) and the Economics and Statistics Administration in a paper titled, “A Nation Online: How Americans Are Expanding Their Use of the Internet” (Washington , D.C. , February 2002):

  • The online population is under-represented in the 50+ age group.
  • The online population is under-represented in the black and Hispanic ethnic groups.
  • The online population is under-represented in the “less than high school” and “high school diploma/GED only” educational groups, while over-represented in the “bachelors degree and beyond” educational groups.
  •  The online population is under-represented in households earning less than $25,000 of income annually, while over-represented in households earning over $50,000 of income annually.
  • Since August 2000, males and females have had virtually identical rates of Internet use, while in September 2001, the Internet use rate was 53.9 percent for males and 53.8 percent for females.
  • There was no significant under-representation found in rural vs. urban dwellers.
  • There were only slight differences in representation regarding household types such as the number of children in the household.

The fifth study cited comes from the 2003 U.S. Census Bureau’s Current Population Study (www.bls.census.gov/cps/computer/sdata.htm ). The following items and graphic in Figure 3 summarize its conclusions related to demographic differences that exist between the online and offline populations:

  • The online population is the most under-represented in the 65+ age groups.
  • The online population is under-represented in the black/African-American and Hispanic ethnic groups.
  • The online population is under-represented in the “less than high school” and “high school only” educational groups, while over-represented in the “some college” and the “bachelors degree or higher” educational groups.
  • The online population is the most under-represented in households earning less than $25,000 of income annually, while over-represented in households earning over $50,000 of income annually.

Methodology

Our firm, e-Rewards, Inc., a Dallas-based online sample provider that provides a currency-based incentive to survey respondents sourced from its panel of approximately 1.3 million members, analyzed survey response rate data from over six million survey invitations that were e-mailed to its panel members during 2004. To account for any seasonality, the months of January, April, August, and October were arbitrarily selected for survey response rate analysis across eight separate consumer demographic dimensions (e.g., gender, age, annual household income, education level, ethnicity, marital status, urban/rural residence, number of children in household) and two business/occupational demographic dimensions (e.g., occupation and business title). Survey response rates were defined as the percentage of outbound e-mail survey invitations sent to active panelists (e.g., equally eligible panelists with deliverable e-mail addresses) that resulted in fully completed online research survey interviews (e.g., surveys where respondents completed all non-optional survey questions presented to them). Survey instrument lengths varied, but the average survey length was approximately 10 minutes. Since this type of analysis requires unrestricted access to internal panel response rate data, the authors used the e-Rewards online panel for this test. Other online panels may experience significantly different absolute response rates, but the authors believe that the directional data should be consistent with our findings. The survey response rates observed in the overall demographic groups studied ranged from 22.7-31.3 percent.

Analysis and findings

The key findings are summarized below (and in Figures 4, 5 and 6):

  • Males respond at a lower rate than females (although not significantly).
  • Those who have not obtained a high school diploma respond at a lower rate than those who have.
  • The response rates increase with the level of education that a respondent has achieved.
  • Those who report $200K+ in annual household income respond at a lower rate than those of other income brackets.
  • Those aged 65+ responded at the lowest rates compared to other age groups.
  • African-Americans, Hispanics and Native Americans respond at a lower rate than other ethnic groups, while Asian-Americans respond at a significantly higher rate versus the mean.
  • Those who are currently divorced, separated or widowed respond at a lower rate than other marital status groups.
  • Those with four or five children in the household respond at a lower rate than other groups.
  • There were no significant response rate differences among rural vs. urban dwellers.
  • Executives/upper management and sales professionals respond at the lowest rates versus other occupational groups, while homemakers and teachers respond at the highest rates versus other occupational groups.
  • The workers with the most seniority (e.g., chairman/board member, president/CEO/COO, executive vice president/senior vice president, vice president) responded at the lowest rates, while developer/programmers, CPAs, and doctors responded at the highest rates. (Note: the unique presence of higher than average financial incentives/honorariums for doctors may be responsible for the higher than average doctor response rates.)

 

Conclusions and discussion

  • The data provides support for H1.

H1: Patterns of non-response within an online panel will tend to exist in the same demographic categories that have experienced patterns of non-response in traditional research modes, such as mail and telephone.

Similar to observations among traditional research modes (e.g., direct mail and telephone), non-responders were also found in the online research mode to more likely be: less educated; older; less affluent; male (but to a much lesser degree).

  • The data provides support for H2.

H2: Non-response rate biases within an online panel will tend to exist in the same demographic categories that are known to be under-represented on the Internet vs. the general U.S. population.

Similar to observations among demographic categories that are known to be under-represented on the Internet vs. the general U.S. population, non-responders were also found in the online research mode to more likely be: less educated (in the some high school category, but not in the high school graduate category); older (especially in 65+ age categories); less affluent (although significant response rate drop-off was detected only at the $200K+ annual HHI level); African-Americans, Hispanics, and Native Americans (while Asian-Americans were found to respond at a rate higher than average).

More studies are clearly needed from other online panels to compare with the results of this study since there may be factors particular to the e-Rewards panel, its recruitment methodology, and its use of incentives. In addition, comparison data from outside sources was only available for consumer demographics across research modes, and more B2B research is needed for valid comparisons to take place in the future. However, the magnitude of the response rate data that was observed (over six million survey invites) provides the authors with a high degree of confidence in the directional validity of the findings.

In summary, researchers need a clearer understanding of the tendency of different demographic groups to under-respond to online survey instruments, so that any resultant patterns of non-response can be appropriately identified and managed during the course of a project. Appropriate ways for researchers to respond to lower than desired response rates by certain demographic groups include: 1) sending additional survey invitations and/or reminders, 2) increasing financial incentives, 3) adjusting or weighting data to account for groups with lower response rates, and/or 4) testing for non-response bias using an alternate research mode. The areas where under-response was observed appeared to be consistent with respondent behavior seen in traditional modes of research and that which is currently understood about the U.S. online population. We hope that the current study’s findings are an aid to researchers and help generate more interest and research on this topic. 

References

Bickart, B., and Schmittlein, D. (1999). “The Distribution of Survey Contact and Participation in the United States: Constructing a Survey-Based Estimate.” Journal of Marketing Research, Spring, 286-294.

Bonchek, M.S., Hurwitz, R., and Mallery, J. (1996, Summer). “Will the Web Democratize or Polarize the Political Process?” World Wide Web Journal. Retrieved October 2001 from the World Wide Web: www.w3j.com/3/s3.bonchek.html.

Church, A.H. “Incentives in Mail Surveys: A Meta Analysis.” Public Opinion Quarterly. 57. 62-79.

Collins, R.L., Ellickson, P.L., Hays, R.D., McCaffrey, D.F. “Effects of Incentive Size and Timing on Response Rates to a Follow-Up Wave of a Longitudinal Mailed Survey.” Eval. Rev. 2000;24(4):437-63.

Dillman, D.A. (1978). Mail and Telephone Surveys: The Total Design Method. New York : John Wiley and Sons.

Goetz, E.F., T.R. Tyler and F.L. Cook. (1984). “Promised Incentives in Media Research: a Look at Data Quality, Sample Representativeness, and Response Rates.” Journal of Marketing Research. 21. 148-154.

Kanuk, L. and Berenson, C. (1975). “Mail Surveys and Response Rates: A Literature Review.” Journal of Marketing Research, Vol. XII, 440-453.

Kehoe, C.M., and Pitkow, J.E. (1996, Summer). “Surveying the Territory: GVU’s Five WWW User Surveys.” World Wide Web Journal. Retrieved October 2001 from the World Wide Web: www.w3j.com/3/s3.kehoe.html.

King C. V., and Delana, P. (2002). “Web Data Collection: Reminders and Their Effect on Response Rates.” Journal of e-Commerce and Psychology, 3 (1 and 2), 12-34.

McCabe, S.E., Boyd, C.J., Couper, M.P., Crawford, S., D’Arcy, H. “Mode Effect for Collecting Alcohol and Other Drug Use Data: Web and U.S. Mail.” J. Stud. Alcohol 2002;63(6):755-61.

Mason, R., Lesser, V., and Dillman, D. “Effect of Financial Incentives on Unit and Item Non-Responses in Mail Surveys.” Paper presented at the Workshop on Item Non-Response and Data Quality in Large Social Surveys, Basel , Switzerland, October 9-12, 2003.

McDaniel, S.V. and C.P. Rao. “The Effects of Monetary Inducement on Mailed Questionnaire Response Quality.” Journal of Marketing Research. 17 (1980). 265-268.

National Telecommunications and Information Administration, and the Economics and Statistics Administration. “A Nation Online: How Americans Are Expanding Their Use of the Internet,” Washington , D.C. February 2002.

On the Definition of Response Rates (1982) A Special Report of the CASRO Task Force on Completion Rates. www.casro.org.

Sheehan, Kim. “E-mail Survey Response Rates - A Review,” Journal of Computer Mediated Communication. Vol. 6, No. 2, January 2001. http://www.ascusc.org/jcmc/vol6/issue2/sheehan.html.

Shettle, C. and G. Mooney. (1999) “Evaluation of Using Monetary Incentives in a Government Survey.” National Science Foundation. Arlington , Va.

Suchman and McCandless (1940). Respondents are more educated than the general population, as cited in “Nonresponse” by Peter D. DeMichele, www.responsivemanagement.com/download/reports/Nonresponse.pdf.

The Harris Poll #18, April 17, 2002.

Wardle, J., Robb, K., and Johnson, F. “Assessing Socioeconomic Status in Adolescents: the Validity of a Home Affluence Scale.” Journal of Epidemiology and Community Health 2002;56:595-599.

2001 and 2003 U.S. Census Bureau’s Current Population Study. www.bls.census.gov/cps/computer/sdata.htm