Research has changed
Editor’s note: Bob Fawson is founder and CEO of Data Quality Co-Op. He’s committed to promoting transparency and data quality for consumer insights. An experienced executive, strategic advisor and consumer insights expert, Fawson has contributed to the evolution and improvement of consumer data through roles at Numerator, Dynata, SSI and Opinionology. As vice board chairman at SampleCon, Fawson enjoys contributing to the next wave of insights innovation. Find Fawson on LinkedIn.
Two years ago, I argued that the industry's biggest quality challenges weren't simply the result of better fraudsters or weaker quality checks. The market itself had become more interconnected, making it more difficult for any individual supplier, platform or buyer to understand respondent quality using only the information available within a single project or organization.
At the time, those ideas were based largely on observation and theory. I argued that fragmented quality signals and increasingly complex sourcing networks would make it harder for any one organization to understand respondent quality in isolation. I believed a broader, shared view of quality would reveal patterns that individual organizations couldn't see, but we didn't yet have enough evidence to prove it.
Today, we have enough data to revisit those early assumptions. Many of the questions researchers ask today simply couldn't have been answered with the information available when we started out.
A longer history changes the questions we ask
When participation history is available, researchers can evaluate patterns that develop over months rather than minutes. A respondent who appears unremarkable within one study may have a long track record of successful participation across dozens of projects. Another may repeatedly trigger quality concerns despite appearing acceptable in any individual survey. Looking across a larger body of activity makes it easier to distinguish isolated incidents from recurring behavior.
That broader perspective can change the way many organizations think about quality, because the conversation is no longer limited to who should be removed from a study. Researchers are (or should be) interested in understanding which respondents consistently contribute reliable data and how those participants can be retained over time.
Multiple quality signals reveal patterns that individual checks miss
Last year, we published a research-on-research study (registration required) that combined technical fraud indicators, in-survey behavior and source-level quality signals. We found that no single measure explained respondent quality on its own. Device checks identified some issues, survey behavior identified others and participation history added another layer of context. Looking across all three produced conclusions that none of those signals could support individually.
Since then, we've seen the same pattern in practice across the industry, with other firms incorporating layered quality processes. In some cases, this looks like working to combine identity verification, technical fraud detection, behavioral monitoring and more to identify issues before questionable traffic reaches a study, reducing the need for removals, replacements and other corrective actions later in the process.
Rather than relying on a single quality check or expecting one technology to solve the problem, organizations are building processes that bring multiple signals together and use them from recruitment through fieldwork.
Reliable respondents are becoming easier to identify
In the past, the insight industry's focus on identifying problematic respondents often left little room to discuss the people who consistently contribute reliable data.
As more participation history becomes available, those respondents are easier to identify. Some participate frequently, and others do not. But it is important to take note of the consistency of their behavior over time. They complete studies successfully, avoid recurring quality issues and continue to perform well across projects and suppliers.
In our research-on-research study, the segment we labeled professional panelists performed well: They were seen frequently, generally came from reliable panel sources and produced consistent responses in that specific project. That finding was useful precisely because it ran counter to the easy assumption that frequent participation is always a quality problem. It does not mean every frequent survey taker is reliable, but it does suggest that participation history can help distinguish professional bad behavior from people who understand the process and take the task seriously.
A few questions worth asking:
- Are quality decisions being made using information from a single project, or is respondent history part of the process?
- Do your quality controls rely on one primary signal, or are technical, behavioral and source-level indicators evaluated together?
- How much attention is devoted to identifying trustworthy respondents compared with identifying respondents who should be removed?
None of these questions eliminate the need for strong project-level quality controls. They do encourage organizations to look beyond individual surveys and consider how quality is measured across respondents, suppliers and studies over time.