Editor's note: Ron Sellers is president of Grey Matter Research. He can be reached at ron@greymatterresearch.com.
Do you want to rely on a study where nearly half the respondents aren’t giving you valid answers? Without the right steps, that’s what you could be getting from your online panel research.
“Whoa – nearly half? That’s a pretty wild claim,” is probably your first thought, followed closely by, “I know online panel has some bad respondents, but we use CAPTCHA and eliminate speeders, so we should be good.”
If only it were that simple…
Grey Matter Research (a consumer insights consultancy) and Harmon Research (a field agency) joined forces to determine just how many disengaged, fraudulent or otherwise bogus respondents are in a typical online panel study. The result is our report Still More Dirty Little Secrets of Online Panels (e-mail me for a copy). We tried to make the cover really attractive, because the content is pretty ugly for anyone who uses online panel research.
Don’t get the notion this is an anti-panel screed or a promotion of some “proprietary” methodology over online panels. Every insights methodology has challenges and quality issues. Good researchers recognize this and do everything they can to reduce the problems and mitigate the challenges without throwing out the methodology. But from our observation, nowhere near enough researchers are taking the necessary steps to address the problems in online panel studies.
We fielded a questionnaire of about 10 minutes, with age and gender quotas, through five of the 10 largest U.S. opt-in consumer research panels. Nearly 2,000 panel members attempted to complete the questionnaire.
We used a variety of techniques to identify respondents who were giving poor-quality or fraudulent answers, such as: asking the same demographic question at the beginning and end of the questionnaire to compare response...