Editor's note: Tom Burdick is vice president at Eleven Market Research. He can be reached at tom@elevenmr.com.
When I first started out in the market research industry in the late 1990s we collected all of our data using pencil-and-paper surveys, which were packaged in large boxes and mailed out to our executive interviewers. They were used to collect data for high-level financial consulting which was used to rate institutional investors, plan sponsors and banking executives, among others. The ratings impacted these financial professionals’ compensation and bonuses, so data quality and accuracy were critical.
Back then, the task of printing the thousands of color-coded questionnaires, bank lists and supporting materials and then packing the boxes for shipments sent around the world was physical and tedious work. The executive interviewers who scheduled and administered each interview were critical for ensuring they spoke to the correct respondents and collected accurate and reliable data. There was little doubt about the authenticity of the respondent or the data they were providing. But it was expensive and time-consuming from start to finish. Fast-forward 30 years and much has changed in terms of how data is collected, but the importance of high-quality, accurate data is as important as ever.
Because the go-to methodology for data collection has changed dramatically since the pencil-and-paper days (and especially over the last decade), the biggest challenge we face in 2025 is trusting the authenticity of the respondents in our dataset. This is especially true with regard to B2B research. Online methods have eliminated the built-in principal validation mechanisms of in-person (and to some degree, phone methodologies). The ease of joining and the ubiquity of online panels has lowered trust and fueled an entire sub-industry of data quality fraud detection platforms and software. They can detect bots, mismatched geography, duplicate entries, keystroke anomalies, AI-generated open-ends and much more! The funny thing though, is that data quality seems to be only getting worse. That’s not to say these tools aren’t important to the data quality landscape. They have some limitations and can’t always stop intentional fraud (hello Op4G).
Fighting data fraud and providing high-quality online data is the most important issue the market research industry is facing today. And as I’ve written previously, good online data quality takes a community approach (“Data fraud: A threat to market research,” Quirk’s e-newsletter, May 5, 2025). All is not lost. I’m hopeful that this community approach, with key stakeholders involved, and the continued introduction of new technologies will ultimately prevail. In the meantime, let’s highlight a few things that we can all do now to improve the likelihood of collecting reliable, good-quality data.
- Stop expecting that VP- and C-level B2B executives are going to participate in your survey for $20 CPIs and it’s going to yield quality responses. After the panels take their cut and prices are marked-up, how much do you think the incentive going to the respondent really is? $5? $10? That is just not going to move the needle for these busy professionals making six-to-seven-figure salaries. If you are trying to get CEOs at enterprise-size companies for $20, that should raise suspicions. But completing interviews with them at that rate should be considered Defcon 1 for your research.
- If budgets and timelines allow, don’t be afraid to use telephone for your data collection. It exponentially increases the likelihood that the right person is answering your survey and a well-trained interviewer can deliver better data than a self-administered online survey. This human element can’t be replicated online.
- Stop with the poorly designed screeners that telegraph to the respondents how to qualify, basically escorting fraudsters right into the survey by making it obvious what the topic is. Construct an intelligent screener that disguises who you want to interview, thus increasing the chances that only qualified respondents make it in and not those just trying to earn an incentive.
- Stop with the long and repetitive surveys that take too much time for busy professionals to complete and that ask the same questions over and over again. Convince your clients that they can get the most important data within 15 minutes. Shorter surveys lead to better conversion rates, fewer dropouts and higher-quality data. Then implement techniques within the survey to identify and terminate fraudulent respondents.
- Do utilize fraud detection software, both before respondents hit your link and within the survey itself, to detect and block things like bots, automation software, duplicate devices, duplicate IP addresses, location mismatches and other distinctive rogue issues.
- Do use multiple sample sources on each project to reduce the risk of a single source tainting the results. Multiple sources improve the speed, reduce bias and allow for more flexible and proactive sampling (i.e., remove/replace bad apples). Reputable sample aggregators can accomplish this very effectively.
- Do make sure you are using an experienced and knowledgeable field manager/field team who can often help save a project from disaster due to poor design, technical issues or the like. The industry seems awash with young field project managers, who although well-meaning, often lack the experience needed to proactively identify red flags and take quick corrective actions.
- Always do a soft launch and then thoroughly review the data to make sure areas like the survey logic, brand awareness, revenues, etc., are all making sense. Look for fraudulent responses and tell the field team so they can remove the source. Do not wait until the end to review data. Anomalies or irregularities in the data can most effectively be addressed before fully launching a study so that the foundation is stable. Waiting to the end only creates stress and delays for all parties involved.
Reduce the potential for disaster
While these actions won’t always solve the problem of poor data quality and fraud, together they help move things in the right direction and reduce the potential for disaster. Being diligent and realistic about project specs and expectations will also help. Construct your survey with fraudulent actors and lazy respondents in mind so your foundation is strong. Use common sense. If you want high-quality data, expect to pay for it. The other alternative is to go back to the quaint world of pencil-and-paper surveys for face-to-face interviewing. Not me! I’m happy to leave the 1990s in the past where they belong – although I still enjoy watching reruns of “Friends” and “Seinfeld”!