Catch me if you can

Editor’s note: Kurt Knapton is executive vice president and chief revenue officer at Dallas research firm e-Rewards Market Research. Rick Garlick is director of consulting and strategic implementation at St. Louis-based Maritz Research.

Have you ever paid close attention to the labels on the consumer products you buy and use every day? For example, on a particular portable infant stroller, there is a warning label that reads, “Warning. Remove child before folding for storage.”! On an automobile windshield sun shade, there is a warning that reads, “Do not drive automobile with sun shade in place.”! Perhaps we can all agree that these consumer labels have gone a bit too far.

Well, if we had to create a warning label for our industry and the online market research studies that we conduct, it should probably state, “Beware. Sample may contain undesirable respondents.”

As online research continues to expand, there is growing scrutiny of and concern about online respondent quality. In fact, panel quality was the chief concern about online research in a March 2006 study conducted by Cambiar among 247 market research firms1. Furthermore, there have been numerous industry summits and expert panels assembled to discuss and debate how to address online respondent quality issues. While there are many different types of error that are the bane of researchers across all research modes (such as survey design error, hypothesis error and interviewer error), it is respondent error - or, more specifically, respondent falsification/undesirable respondents - that many agree can jeopardize the results of an otherwise well-designed and well-executed online study.

So how does one define the problem of undesirable respondents, how prevalent is it, how can it be detected, and what steps can the researcher take to minimize it? These are the key questions that the authors attempt to answer for you.

Definitional framework 

If you subscribe to the adage, “A problem well-defined is a problem half-solved,” then perhaps a definitional framework for undesirable respondents is in order. Adopting some definitions that Harris Interactive2 has used to classify certain types of professional respondents, these same classifications are useful to describe undesirable respondents:

  • Fraudulent respondents: those who intentionally misrepresent themselves or provide inaccurate information - often to maximize the incentives they earn.
  • Inattentive respondents: those who, because of survey length, time constraints or other reasons, do not appear to be providing thoughtful answers during the course of a specific study.
  • Hyperactive respondents: those who participate in numerous surveys - especially online surveys - and who often belong to multiple online panels.

The authors conducted a literature search to determine if others had investigated the prevalence of online survey cheating. In 2005, Harris Interactive reported that across 20 large studies, approximately 3 percent of respondents provided fraudulent age and gender answers that differed from previously profiled answers. Furthermore, it found that an additional 1 percent of respondents failed to differentiate their responses across a large set of attributes (e.g., responded “randomly”), while another 2.5 percent of respondents might have been responding randomly. However, it is important to note that the Harris study did not apply fraudulent traps beyond age and gender, and did not apply traps to identify inattentive respondents, so the percentage ranges may be underestimated.

In a white paper from research firm Doxus, “Satisficing Behavior in Online Panelists,”3 Theo Downes-Le Guin reported that for all commercial, Internet-recruited U.S. Web panels, Doxus specifies an overage of between 10 percent and 15 percent above sample size targets per quota to compensate for data cleaning required due to data quality issues. Furthermore, using a full variety of different screening traps (on the front end) and data cleaning strikes (on the back end), Doxus has observed an incidence of failures among different online panels ranging from 4 percent to 34 percent.

Lastly, e-Rewards Market Research uses a by-invitation-only enrollment methodology that permits only pre-invited members to join its panel. This prevention technique - as compared to an open recruitment technique where anyone can join - intends to lower the amount of potential undesirable respondents from being able to self-select and attempt to join the panel in the first place. However, even with this more stringent invitation-only recruitment approach, e-Rewards typically encounters (and flags these respondents) about 2 to 6 percent of new panel members who fail either a fraudulent, inattentive or hyperactive trap.

Therefore, to truly answer the question of “How big is the problem of undesirable respondents?”, one must evaluate the sample source that is chosen, the source’s recruitment methods and the level of stringency of the detection methods used by both the sample source and the researcher. With that said, drawing from sources previously cited, undesirable response by Fraudulents and Inattentives seems to range between 2 and 34 percent, and Hyperactives can add another 2 to 25 percent.

Embed traps

If you are open to the premise that there could be a significant problem, then you may wonder how we can detect undesirable respondents to eliminate them from our survey results or, better yet, avoid them altogether. The way to detect undesirable respondents is to embed within survey instruments respondent traps that are designed to expose the Fraudulent, Inattentive and Hyperactive respondents. Once detected, the bad survey responses can be thrown out so as not to pollute the study results, and the undesirable respondents themselves can be flagged and/or removed from the online survey panel altogether.

There are many effective ways to trap and flush out Fraudulents, Inattentives and Hyperactives. Here are a few examples of traps that e-Rewards Market Research has embedded during the panel enrollment process:

Fraudulents

  • “Red herring” traps - Catch respondents who choose answers from a list and claim, for example, to personally suffer from named diseases that do not exist.
  • Consistency of answer traps - Detect respondents who answer the same fact-based question differently when it is asked both at the beginning and the end of the survey instrument.
  • Mutually exclusive logic traps - Expose respondents who provide answers that are logically inconsistent with each other.

Inattentives

  • Straightlining traps - Detect response patterns that are too predictable to be credible.
  • Speed traps - Measure the overall time that respondents take to answer the survey instrument and detect those who do not take adequate time to be credible.
  • Simple instruction traps - Ask respondents to simply “check the third box” or “identify a simple picture” to ensure that they are paying attention.

Hyperactives

  • Panel/survey frequency traps - Ask respondents questions about whether they belong to other panels, how many, how frequently they participate, which ones, etc.

Lose privileges?

With the proper traps in place, the undesirable respondents are identified. But what should be their sentence? Should they lose all privileges of participating in future surveys? After all, what if the respondent was simply having a bad day? Is there any proof that detected undesirable respondents are more prone than others to continue their bad behavior on future surveys? Or in other words, are undesirable respondents, in fact, prone to be repeat offenders? If so, then the case is much stronger toward banning the one-time undesirable respondents from the panel altogether.

The authors could not find any data or studies that could answer this question. Therefore, the authors designed an experimental construct that set out to either prove or disprove the following:

Hypothesis: “Undesirable respondents detected during an online panel enrollment process are more likely than others to be undesirable respondents in subsequent follow-up surveys.”

If this is proven to be true, then the following corollary conclusions are logical:

  • Undesirable respondent behavior can be predicted to some extent.
  • Online sample providers who identify and eliminate undesirable respondents during panel enrollment will introduce a lower level of undesirable respondent behavior into subsequent online research studies.
  • Overall respondent error will be reduced.
  • Online research results will be more accurate and reliable.

The experiment

Here is how the joint e-Rewards/Maritz experiment was constructed and executed in August 2006. As previously described, e-Rewards Market Research screens enrolling panelists using a variety of traps to identify Fraudulent, Inattentive and Hyperactive respondents. Normally, e-Rewards simply flags these respondents as bad and they lose the privilege of taking future surveys. However, to facilitate the experiment - with Maritz’ pre-knowledge and consent4 - e-Rewards sent over a group of known bad panelists (who were previously caught in respondent traps) to a Maritz Poll study to see how they compared to other respondent groups sent over in equal proportions. In addition to sample that Maritz commissioned from another panel vendor as a control group, 3,000 total outbound e-mails were sent by e-Rewards to the Maritz Poll survey instrument - one third were the pre-identified undesirable respondents (labeled as bad):

  • 1,000 undesirable respondents (e.g., the bad panelists) were invited to take the Maritz Poll survey. An equal amount of Fraudulent (334), Inattentive (333) and Hyperactive respondents (333) were used.
  • 1,000 known good panelists (who passed all of the enrollment traps) were also invited to take the Maritz Poll survey.
  • 1,000 from a control group of mixed/random panelists (who were randomly selected without regard to the enrollment trap results) were invited too.

For objectivity purposes, Maritz was solely in control of the Maritz Poll subject matter and quality control measures (e.g., its own traps) which it did not reveal to e-Rewards in advance. The Maritz Poll, fielded August 2-6, 2006, had an overall theme of casual dining. Question categories included questions about: recent dining experiences; importance of restaurant attributes in the choice process; spending patterns; healthy dining; and other “fun” questions.

Chart 1 shows a summary of the specific traps that Maritz embedded in its Maritz Poll for experimental purposes.

The results

From the control group sample (e.g., the sample obtained from another, non-e-Rewards panel) 998 respondents were collected. This sample was labeled as control-group sample.   From the panelists invited from the e-Rewards panel, 1,003 responses were collected, representing a 33 percent overall response rate. Chart 2 shows the overall breakdown of the sample respondent results.

When the responses were analyzed by Maritz, overall 5.5 percent of the total sample failed at least one of the traps, with the most failing panelists sourced from the e-Rewards pre-labeled bad sample and the control group panel sample. In fact, previously-labeled bad panelists were five times more likely than good panelists to be bad again in the Maritz Poll survey, and were nearly twice as likely to be bad vs. the control group from the other panel provider.

The findings

Upon analyzing all of the respondent-provided answers, Maritz drew the following conclusion: Significant differences were observed in three areas - attribute ratings, speeding and professional respondents.

Attribute ratings

The attribute ratings questions in the Maritz Poll were analyzed:

  • The impact of the respondents caught in the Maritz Poll traps was most apparent in matrix sets of items:

- significant differences on 10 out of 14 importance attribute ratings;

- those caught in traps claimed to spend 32 percent more on a meal when dining out.

Speeding

Those completing the Maritz Poll the quickest (e.g., the fastest 10 percent of survey takers) were examined:

  • 60 percent of the fastest 10 percent to complete the survey were from the other (non-e-Rewards) sample vendor.
  • 23 percent were from the pre-identified bad e-Rewards sample.
  • 6 percent were from the pre-identified good e-Rewards sample.
  • 11 percent were from the pre-identified mixed/random e-Rewards sample.
  • Pre-identified Hyperactives produced five times as many fast completions compared to pre-identified Inattentives.
  • Fast completes provided significantly different ratings on 10 out of 14 importance rating attributes (and differed significantly on a number of other items as well).

Professional respondents

Maritz Poll questions relating to the respondent’s frequency of survey taking were analyzed:

  • One in 10 completed over 15 surveys within the past month. Maritz labeled these respondents as Professional respondents.
  • 14 percent of the other (non-e-Rewards) sample were Professional respondents.
  • 12 percent of the pre-identified bad e-Rewards respondents were Professional respondents. Almost all (88 percent) of the bad respondents were previously identified Hyperactives.
  • 4 percent of the pre-identified good e-Rewards respondents were Professional respondents.
  • 1 percent of the pre-identified mixed/random e-Rewards respondents were Professional respondents.
  • Professional respondents provided significantly different ratings on 11 out of 14 rating attributes (and gave significantly different response patterns on many other items as well).

The conclusions

Maritz drew the following conclusions from the Maritz Poll findings:

The hypothesis (“Undesirable respondents detected during an online panel enrollment process are more likely than others to be undesirable respondents in subsequent follow-up surveys.”) was found to be true.

  • Although the incidence of getting caught by any one specific trap is relatively small (1 to 3 percent), the cumulative effect of bad respondents may negatively influence survey results.

- Approximately one-in-20 (5.5 percent) online survey respondents showed some degree of inattentiveness or fraud at some point in the survey.

- While even good respondents slipped up occasionally, screening for bad respondents up front is useful for improving data validity.

  • Bad survey respondents appear to most likely affect results when:

- surveys involve lengthy lists of rating attributes;

- surveys have long, repetitive, or tedious questions.

  • There is some evidence of overlap between being a Professional respondent (>15 surveys per month) and the quickness with which respondents finish surveys (e.g., Fast Completers.)

- Response patterns are different among these individuals.

  • There is evidence that Hyperactives (those participating on multiple online research panels) are more prone to speed through surveys.

- Response patterns are different among these individuals.

  • There is evidence that Professional respondents (>15 surveys per month) answer differently than those who take surveys less frequently.

- Eliminating these respondents probably boosts data validity.

  • Although even the most well-intentioned respondent occasionally makes an error on a survey, pre-screening panel respondents using traps certainly improves the odds of a valid subsequent survey.
  • Pre-screening panelists is particularly important when surveys ask respondents to rate multiple attributes or questions more complex than a simple dichotomous (yes/no) option.

Effective ways

Will we ever be able to stop all of the undesirable respondents all of the time? Probably not. However, the joint e-Rewards/Maritz experiment shows that there are very effective ways that undesirable respondents can be stopped, positively impacting the integrity of study results. Additionally, the e-Rewards/Maritz experiment is a great example of how survey designers and survey fielders can work together to help solve a broader industry problem - in this case online respondent quality.

In summary, in order to take proactive measures to ensure the integrity and quality of your online research, it is important to consider these factors:

  • Undesirable respondents are apparent in online research and are a cause for concern, but they can be minimized using respondent traps.
  • Researchers are wise to utilize an arsenal of traps to prevent undesirable respondents from jeopardizing research results - and insist that their panel providers do too.
  • A partnership between survey designers and fielders (who are most often providing the online respondents) is required to minimize undesirable respondents.
  • Undesirable respondents detected during an online panel enrollment process, if not removed from the panel, are more likely than others to be undesirable respondents in subsequent surveys.
  • Studies are most susceptible when respondents are sourced from sample providers who do not prevent, detect or remove undesirable respondents at panel enrollment.
  • Insist on knowing what undesirable respondent precautions the sample vendor has taken.
  • Have your own methodology for identifying undesirable respondents in place.
  • Removing undesirable panelists can make a significant difference in the integrity of study results.

The authors wish to acknowledge the assistance of additional study contributors Cort Clark of e-Rewards Market Research and Allen Hogg, who was with Socratic Technologies at time of the study. Additional article contributions came from Ashley Harlan of e-Rewards Market Research. 

References

1 “The Online Research Industry - An Update on Current Practices and Trends,” Cambiar (May 2006).

2 Renee Smith and Holland Hofma Brown, “Assessing the Quality of Data from Online Panels: Moving Forward with Confidence,” Harris Interactive (2005).

3 Theo Downes-Le Guin, “Satisficing Behavior in Online Panelists,” Doxus (2005).

4 Note: Maritz was fully aware of and expecting these undesirable respondents for observation and did not use the responses from undesirable respondents to support any client work.

Â