Speed traps

Editor’s note: Aron Levin is associate professor of marketing at Northern Kentucky University, Highland Heights, Ky. Michael G. Holmes is president of EMI Surveys, Cincinnati.

  • n = 200;
  • target: adults 18 years and older residing in the Cincinnati metropolitan area;
  • 50/50 gender split;
  • incidence rate = 75 percent among the target;
  • survey length = 15 minutes;
  • survey topic: entertainment, politics, and social issues in Cincinnati.

Note: EMI did not leverage cookies, track by IP address, include any other duplication blockers or implement timing mechanisms in order to find the natural occurrence of duplication and speeders.

The study used both self reporting and survey traps to find the frequency of professional and hyperactive respondents. When queried directly, it appeared that most respondents answered quite candidly when asked how many panels they were members of and how many surveys they had taken within the past month. Self reporting results found that 25 percent of survey respondents indicated that they take more than 10 surveys per month and 25 percent noted that they belong to four or more online research panels.

Registered multiple times

EMI tracked both intra- and inter-panel duplication. Intra-panel duplication occurs when a single person has registered multiple times within one panel under different names and/or e-mail addresses. Inter-panel duplication occurs when a single person has registered multiple times across several different panels. This can be seen when multiple panels are utilized to fill a geographically-specific study, and the same person is invited from all the panels to which they belong. To track both, EMI matched respondents’ birthdays with their street addresses. If there was an exact match, that subject was considered to be hyperactive, meaning that they were either members of multiple panels or a duplicate within one panel and thus taking the same survey multiple times. Of the 1,161 total completes, 139 duplicate submissions were discovered. In a few rare instances, duplication occurred intra-panel, but more commonly duplication occurred because a single respondent belonged to four or more of the panels and was able to take the survey multiple times under each panel’s memberships.

The pervasiveness of professional respondents and hyperactives significantly differed across panels. Three out of the seven online panels had 40 percent or more of their respondents indicating they had taken more than seven surveys within the past 30 days versus only 18 percent from another panel.

Duplicate answers were analyzed to see if those respondents would change their actual “identity” from survey to survey. Interestingly, answers did not vary much across the same panels for repeaters. If someone indicated he was a 32-year-old Xavier University graduate who lived on Hillcrest Lane, and was born on September 4, his answers were consistent each time he took the survey. This finding was the first indication that speeders and hyperactives may not be as negative as many industry professionals think. This study showed that they answer questions with thought and consistency versus straight-lining their responses. It is noteworthy that the survey topic was focused on local news and interests, perhaps causing the respondent to give more thoughtful, detailed and consistent answers.

The survey included a question that asked respondents to indicate their favorite local ice cream shop; 31.4 percent listed Graeters Ice Cream shop while 33.4 percent chose United Dairy Farmers (UDF). In order to find the potential impact of duplication on the statistical significance for this answer, the information was reanalyzed and duplicate answers were eliminated. After removing the 139 duplicate submissions, EMI found that the results varied only slightly, with 31.8 percent selecting Graeters and 32.6 percent selecting UDF. However, a larger incidence of duplication in other studies could significantly impact data and negatively affect a company’s decisions surrounding their products or services.

Answers from an attitudinal and frequency question were also analyzed to determine the effect of duplication on the study. Results from the entire sample show 19.5 percent chose responses from “agree” to “strongly agree” with an attitude statement about a highly publicized local land development project; 19.1 percent after repeaters were thrown out. Of the entire sample, 9.9 percent of respondents had visited a local casino one to two times during the last six months; 10.2 percent after repeaters were thrown out. In all three instances, the results did not change significantly for attitudinal, forced-choice or frequency-determination questions.

The firm further scrutinized results to compare the answers of professional respondents versus less active survey takers and found that the difference between their answers were not statistically significant. For example, 32 percent of those who had taken fewer than 10 surveys in the past 30 days preferred Graeters vs. 29 percent of those who had taken 10 or more. Result differences for UDF were also not significant. Of respondents who had taken fewer than 10 surveys, 18.8 percent agreed to strongly agreed with the attitudinal question about the land development project versus 21 percent of those who had taken more than 10 surveys. Finally, 9.6 percent of those who have taken fewer than 10 surveys have been to a local casino one or two times versus 11 percent for those who had taken more than 10. In addition, the study did not uncover any noteworthy differences in the answers of those who belong to one to three panels and those who belong to four or more panels.

Labeled as speeders

EMI analyzed the quality of the data that it obtained from speeders. The firm expected the average respondent to take approximately 15 minutes to complete the study, which included only one skip logic question. All respondents who answered the survey in less than seven minutes were labeled as speeders.

The study included several traps to see if these speeders were in fact mental cheaters or people who did not pay close attention to the questions asked. These traps included questions such as “Please type the number 49 into the box below.” Interestingly, these validating questions were answered accurately 98 percent of the time. EMI also found that the speeders’ responses did not differ significantly from non-speeders on attitudinal, forced-choice or frequency-determination questions: 30 percent of speeders preferred UDF, 34 percent of non-speeders; 29 percent of speeders preferred Graeters, 32 percent of non-speeders. When asked attitude questions about the land development project, 18 percent of speeders agreed or strongly agreed versus 19.8 percent of non-speeders. Of speeders, 6.6 percent had been to a local casino versus 10.8 percent of non-speeders. Speeders’ attention to trap questions and the quality of their open-ended answers for this study indicates that speeders are not necessarily mental cheaters.

Duplication significant

The study found that duplication is more significant in some panels than others. This duplication challenge must be addressed directly when leveraging multiple panels for a study in order to avoid duplication of respondents.

It also found that from a demographic standpoint (other than age), speeders do not differ significantly when compared to non-speeders. Specifically, younger respondents were significantly more likely than older respondents to be speeders (p<.001). One may conclude that younger respondents are more technologically savvy and are able to take surveys faster than their older and less technically adept counterparts, allowing them to provide the same quality of answers in less time.

Analyze more variables

More important than any demographic similarities or differences, the study revealed   that the answers and opinions of speeders and hyperactives do not vary significantly from those of their counterparts. With that being said, organizations should analyze more variables than the duration it took for a respondent to finish the interview when scouring the data for cheaters or similar respondent types.

While all of these potential issues do exist, they did not appear to greatly impact the results of this particular research. To ensure that these issues to do not impact future studies, researchers must carefully scrutinize their sample partners because not all panels are created equal. Those panels that take leadership roles in addressing quality issues are the partners that you can rely on for high-quality data.