Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

By the Numbers: 10 online sample integrity tips



Article ID:
20140306
Published:
March 2014, page 26
Author:
Jerry Savage

Article Abstract

From inspecting IP and e-mail addresses to post-survey telephone interviews, here are strategies for analyzing the quality of online survey samples.

Editor's note: Jerry Savage is the proprietor of Seattle-based ResearchSight.

The widespread availability of online survey technology has opened important doors for product developers and marketing managers who seek to develop customer-centric strategies that will help them succeed in a challenging marketplace. However, the Internet is still a largely uncharted social space that harbors the well-intentioned as well as others who participate in surveys merely to make money or to get inside information on new product and marketing concepts. Indeed, the Internet could be likened to the Wild West of primary market research: a place of great opportunity and innovation, yet one which also poses many unique challenges.

It seems fair to conclude that there are two critical elements of any quality survey: 1) a sample that is representative of the population being studied and 2) survey questions or “items” that garner valid and reliable responses from participants. The quality of the sample is arguably the more important of the two and it is precisely this issue which poses some of the biggest challenges in online survey research. While online surveys using established consumer panels as a sample source are an enticing option for companies that want to survey various segments of the marketplace, cheaters (i.e., individuals who lie in surveys to acquire an incentive or get information) can pose a serious threat to data integrity and undermine the generalizability of results.

As such, questions of sample integrity are at the core of methodological questions about how to use the Internet to acquire accurate and actionable business intelligence. While not all companies are faced with exactly the same questions, there are strategies that can be applied by most analysts and managers when analyzing the integrity of online survey samples:

1. Be diligent. The first step to ensuring sample integrity in any online survey is committing to a diligent review of all responses. While automated approaches to catching cheaters with straight-line and other pattern syntax can be useful, it is important that each survey response be manually inspected to help ensure the study includes only respondents who provide honest answers and are representative of the population under study. While diligent analysis at this level can be a tedious matter, the time spent inspecting data is well worth it

2. Predefine sample parameters. While achieving a truly random sample of any large population is often difficult or impossible due to typically low response rates in surveys as well as time and budget constraints, it is possible to gather a sample which matches some known population parameters. When possible, use extant data such as the census or sales data to create sample plans. This helps ensure that the final sample is representative of at least some key parameters of the population being studied, thereby reducing sampling error.

3. Carefully develop screening criteria. Screening questions are measures at the beginning of surveys that are targeted to specific market segments. Such questions are used to determine whether a potential respondent is appropriate for the study and, in some cases, what questions they should be asked. For key screening criteria, it is sometimes useful to include “dummy levels” in survey questions so that it’s not easy for would-be cheaters to guess which response must be selected to proceed through the survey. For instance, if you wanted only people who use a certain model of car in your survey, you could include five models and terminate everyone who did not select the model of interest. This helps ensure that the final sample is representative of the market segment being studied.

4. Inspect IP addresses. Determine whether respondents have common IP addresses or are from a common subnet. Depending on the nature of the panel or list being used to field the survey, respondents who come from a common IP address or subnet should probably be flagged as suspicious in the database. Suspicious IP addresses can be looked up on the Internet to determine the country of origin and that information can be especially important when conducting international surveys.

5. Conduct time-based integrity checks. Quality online survey tools allow one to determine the number of seconds it took each respondent to complete a survey. Cheaters tend to complete surveys much faster than authentic respondents. After all, they are often only taking the survey to make money. If a given respondent took the survey in less than the median number of seconds for the sample as a whole, that person should be flagged as suspicious. Note that some respondents also take an unusually long time to complete a survey simply because they are preoccupied with other tasks. These people may be qualified respondents who are simply busy but they should be excluded as outliers from analyses of time-to-completion.

6. Inspect e-mail addresses. Cheaters sometimes have unusual e-mail addresses from domains like yahoo.com or hotmail.com. (Consider the e-mail address ddfytc@yahoo.com.) Suspicious e-mail addresses should generally be removed from the panel and/or dataset if the e-mail address is associated with a common suspect subnet, odd verbatims or response times that fall below the median time-to-survey-completion measure.

7. Gather qualitative data. Cheaters often give very short responses to open-ended questions and/or responses that would not be given by a qualified respondent. Bear in mind that past studies have shown respondents to online surveys tend to provide longer, more detailed responses than those responding to telephone surveys. Short responses that don’t seem relevant to the study are an important indicator of cheating behavior.

8. Telephone control. While online surveys are a far more efficient and flexible way to do survey research, telephone surveys offer an important advantage in that interviewers are able to speak with respondents and ensure they are indeed part of the target population. When conducting a study of the general population or a large market segment using the Internet, develop a telephone control and use this as a benchmark in analysis. This can be challenging at times but is an important element of an effective sample integrity strategy. For instance, in a study of 500 consumers, 50 interviews could be done by phone to help ensure that all questions are being properly interpreted by respondents and that segments of the population who are under-represented in online panels are properly represented in the final sample. That control – especially when combined with other controls developed with reliable extant data – can be used in when developing sample plans and weighting strategies. This multi-method approach to data collection helps ensure that the final sample is generalizable.

9. Post-survey telephone interviews of online respondents. Conduct brief, semi-structured telephone interviews with a random sample of online respondents after they have taken the survey. This process can yield useful data and also allows one to spot problems that may have occurred during the course of online data collection.

10. Careful interpretation. While market analysts and research firms tend to tout the “latest and greatest” in statistical modeling, customers don’t always think and behave in ways that can be understood with statistical models. A careful analysis of survey data that includes input from researchers, managers and stakeholders – together with a commitment to putting the customer at the center of development efforts – is a key element of developing an accurate and actionable interpretation of survey results.

Ever more important

Without a doubt, online survey research will continue to grow in the future and will provide many more companies with the opportunity to get to know their customers and create better relationships with them. As the use of online survey technology becomes more widespread, it will become ever more important that researchers and managers take steps to maintain the integrity of survey samples. The work can be tedious but in the long run – especially in highly competitive consumer markets – it’s well worth the effort. 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

RIVA COURSE 201: FUNDAMENTALS OF MODERATING
September 15-17, 2014
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on September 15-17 in Rockville, Md.
RIVA COURSE 201: FUNDAMENTALS OF MODERATING
Sept. 29 - Oct. 1, 2014
RIVA Training Institute will hold a course, themed 'Fundamentals of Moderating,' on Sept. 29 - Oct. 1 in Rockville, Md.

View more Related Events...

Related Articles

There are 1250 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

A look at corporate and brand images in Eastern Europe, the Caucasus and Central Asia
A study of corporate and brand images across several Eastern European and neighboring countries shows that while Western products have made significant inroads, there is still a strong sense of nationalism that can drive success for regional and local brands.
Trade Talk: Notes from the MRA conference in Boston
Quirk's Editor Joseph Rydholm reflects on his time and what themes he noticed at the Marketing Research Association's annual conference.
It's the user satisfaction, stupid
Though corporate Web sites are not products in the literal sense, they serve as powerful brand image communicators. While companies conduct large amounts of research on their actual products and services, many err by failing to seek user input when developing or refining their Web sites - which can lead to alienated or frustrated users. The author provides a brief overview of the steps companies should take to gauge their site’s usability.
In Case You Missed It... May 2014
News and notes on marketing and marketing research: Twitter free-coffee offer finds success; free-from eating trends
Interviewing: The impact of predictive dialing
Today’s telephone interviewing industry faces several obstacles, including rising refusal rates, leaner client budgets and fewer hours when consumers are at home. This article discusses predictive dialing, a tool that can benefit companies in terms of both productivity and expenditures.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

Online Focus Group sessions
10/28/2013 by Sally Hooper
Market research report
08/20/2013 by Aarkstore Store
SSI - Panel Quality
07/19/2012 by Paul Ponsford
E-mail reminders
04/03/2011 by Ruben Alcaraz
E-mail reminders
02/03/2011 by Kyle Burnam

View More