Skip to: Main Content / Navigation

  • Facebook
  • Twitter
  • LinkedIn
  • Add This

By the Numbers: 10 online sample integrity tips



Article ID:
20140306
Published:
March 2014, page 26
Author:
Jerry Savage

Article Abstract

From inspecting IP and e-mail addresses to post-survey telephone interviews, here are strategies for analyzing the quality of online survey samples.

Editor's note: Jerry Savage is the proprietor of Seattle-based ResearchSight.

The widespread availability of online survey technology has opened important doors for product developers and marketing managers who seek to develop customer-centric strategies that will help them succeed in a challenging marketplace. However, the Internet is still a largely uncharted social space that harbors the well-intentioned as well as others who participate in surveys merely to make money or to get inside information on new product and marketing concepts. Indeed, the Internet could be likened to the Wild West of primary market research: a place of great opportunity and innovation, yet one which also poses many unique challenges.

It seems fair to conclude that there are two critical elements of any quality survey: 1) a sample that is representative of the population being studied and 2) survey questions or “items” that garner valid and reliable responses from participants. The quality of the sample is arguably the more important of the two and it is precisely this issue which poses some of the biggest challenges in online survey research. While online surveys using established consumer panels as a sample source are an enticing option for companies that want to survey various segments of the marketplace, cheaters (i.e., individuals who lie in surveys to acquire an incentive or get information) can pose a serious threat to data integrity and undermine the generalizability of results.

As such, questions of sample integrity are at the core of methodological questions about how to use the Internet to acquire accurate and actionable business intelligence. While not all companies are faced with exactly the same questions, there are strategies that can be applied by most analysts and managers when analyzing the integrity of online survey samples:

1. Be diligent. The first step to ensuring sample integrity in any online survey is committing to a diligent review of all responses. While automated approaches to catching cheaters with straight-line and other pattern syntax can be useful, it is important that each survey response be manually inspected to help ensure the study includes only respondents who provide honest answers and are representative of the population under study. While diligent analysis at this level can be a tedious matter, the time spent inspecting data is well worth it

2. Predefine sample parameters. While achieving a truly random sample of any large population is often difficult or impossible due to typically low response rates in surveys as well as time and budget constraints, it is possible to gather a sample which matches some known population parameters. When possible, use extant data such as the census or sales data to create sample plans. This helps ensure that the final sample is representative of at least some key parameters of the population being studied, thereby reducing sampling error.

3. Carefully develop screening criteria. Screening questions are measures at the beginning of surveys that are targeted to specific market segments. Such questions are used to determine whether a potential respondent is appropriate for the study and, in some cases, what questions they should be asked. For key screening criteria, it is sometimes useful to include “dummy levels” in survey questions so that it’s not easy for would-be cheaters to guess which response must be selected to proceed through the survey. For instance, if you wanted only people who use a certain model of car in your survey, you could include five models and terminate everyone who did not select the model of interest. This helps ensure that the final sample is representative of the market segment being studied.

4. Inspect IP addresses. Determine whether respondents have common IP addresses or are from a common subnet. Depending on the nature of the panel or list being used to field the survey, respondents who come from a common IP address or subnet should probably be flagged as suspicious in the database. Suspicious IP addresses can be looked up on the Internet to determine the country of origin and that information can be especially important when conducting international surveys.

5. Conduct time-based integrity checks. Quality online survey tools allow one to determine the number of seconds it took each respondent to complete a survey. Cheaters tend to complete surveys much faster than authentic respondents. After all, they are often only taking the survey to make money. If a given respondent took the survey in less than the median number of seconds for the sample as a whole, that person should be flagged as suspicious. Note that some respondents also take an unusually long time to complete a survey simply because they are preoccupied with other tasks. These people may be qualified respondents who are simply busy but they should be excluded as outliers from analyses of time-to-completion.

6. Inspect e-mail addresses. Cheaters sometimes have unusual e-mail addresses from domains like yahoo.com or hotmail.com. (Consider the e-mail address ddfytc@yahoo.com.) Suspicious e-mail addresses should generally be removed from the panel and/or dataset if the e-mail address is associated with a common suspect subnet, odd verbatims or response times that fall below the median time-to-survey-completion measure.

7. Gather qualitative data. Cheaters often give very short responses to open-ended questions and/or responses that would not be given by a qualified respondent. Bear in mind that past studies have shown respondents to online surveys tend to provide longer, more detailed responses than those responding to telephone surveys. Short responses that don’t seem relevant to the study are an important indicator of cheating behavior.

8. Telephone control. While online surveys are a far more efficient and flexible way to do survey research, telephone surveys offer an important advantage in that interviewers are able to speak with respondents and ensure they are indeed part of the target population. When conducting a study of the general population or a large market segment using the Internet, develop a telephone control and use this as a benchmark in analysis. This can be challenging at times but is an important element of an effective sample integrity strategy. For instance, in a study of 500 consumers, 50 interviews could be done by phone to help ensure that all questions are being properly interpreted by respondents and that segments of the population who are under-represented in online panels are properly represented in the final sample. That control – especially when combined with other controls developed with reliable extant data – can be used in when developing sample plans and weighting strategies. This multi-method approach to data collection helps ensure that the final sample is generalizable.

9. Post-survey telephone interviews of online respondents. Conduct brief, semi-structured telephone interviews with a random sample of online respondents after they have taken the survey. This process can yield useful data and also allows one to spot problems that may have occurred during the course of online data collection.

10. Careful interpretation. While market analysts and research firms tend to tout the “latest and greatest” in statistical modeling, customers don’t always think and behave in ways that can be understood with statistical models. A careful analysis of survey data that includes input from researchers, managers and stakeholders – together with a commitment to putting the customer at the center of development efforts – is a key element of developing an accurate and actionable interpretation of survey results.

Ever more important

Without a doubt, online survey research will continue to grow in the future and will provide many more companies with the opportunity to get to know their customers and create better relationships with them. As the use of online survey technology becomes more widespread, it will become ever more important that researchers and managers take steps to maintain the integrity of survey samples. The work can be tedious but in the long run – especially in highly competitive consumer markets – it’s well worth the effort. 

Comment on this article

comments powered by Disqus

Related Glossary Terms

Search for more...

Related Events

RIVA COURSE 303: ADVANCED MODERATING
October 27-29, 2014
RIVA Training Institute will hold a course, themed 'Advanced Moderating,' on October 27-29 in Rockville, Md.
MOBILE SHOPPING 2014
October 28-30, 2014
Worldwide Business Research will hold its mobile shopping conference on October 28-30 at The Wigwam, Litchfield Park, in Phoenix.

View more Related Events...

Related Articles

There are 1259 articles in our archive related to this topic. Below are 5 selected at random and available to all users of the site.

Avoid 'surprises' with a comprehensive research plan
Having a solid research plan in place can help companies stay prepared and avoid unwanted surprises.
The top 10 FAQs about online research
The author answers 10 frequent questions from people considering using the Internet for conducting marketing research. Some questions address recruitment issues, such as how to find potential participants, incentive options, and differences between e-mail and spam recruitment. Other questions focus on the survey design and delivery issues, such as types of online surveys available, equipment needed, cookies, security issues and timelines.
What's hot in conjoint: A recap from the 2013 Sawtooth Software Conference
SKIM's Chris Fotenos reports back from the 2013 Sawtooth Software Conference with an overview of the themes and popular topics covered regarding conjoint analysis, including how to make conjoint more engaging, how to take conjoint mobile and how to augment conjoint with other data sources.
By The Numbers: Does the survey sender’s gender matter?
The author and colleagues conducted experiments to measure the impact of the so-called Anita effect, under which the gender of the person sending an e-mailed survey invitation appears in some cases to affect response rates.
By the Numbers: Protecting online survey data integrity
The article outlines ways to handle various forms of undesirable respondents during the questionnaire design process, including the use of personal access codes, question-based verification and time-elapsed verification.

See more articles on this topic

Related Suppliers: Research Companies from the SourceBook

Click on a category below to see firms that specialize in the following areas of research and/or industries

Specialties

Conduct a detailed search of the entire Researcher SourceBook directory

Related Discussion Topics

Online Focus Group sessions
10/28/2013 by Sally Hooper
Market research report
08/20/2013 by Aarkstore Store
SSI - Panel Quality
07/19/2012 by Paul Ponsford
E-mail reminders
04/03/2011 by Ruben Alcaraz
E-mail reminders
02/03/2011 by Kyle Burnam

View More