Building trust in marketing research data
Editor’s note: Ricky Odello is EVP of sales at Rep Data, where he advises customers on the best solutions to meet their needs across the company’s suite of research services and technology. Prior to joining Rep Data, Odello was the EVP of Americas at Cint and spent time in a variety of commercial leadership capacities at Lucid and SSI. Odello has a depth of experience in data collection methods and best practices, including offline and online, managed services, DIY interfaces and APIs. Find Odello on LinkedIn.
Kurt Knapton of Realm Resources has watched online sample develop from an emerging alternative to telephone research into a primary source of data for the market research industry.
Knapton joined Steve Snell of Rep Data on stage at the 2026 Quirk’s Event – New York for a conversation about his recent e-book, “From Access to Trust,” discussing what the industry learned during that transition and how those lessons apply to the data quality issues researchers are dealing with today. The talk ranged from the growth of sample aggregation and changes in respondent recruitment to fraud detection, inattentiveness, AI agents and synthetic respondents.
On stage, Knapton recalled a time when sample quality was evaluated as the three Rs: how people were recruited, whether they were representative of the population and whether response rates were healthy.
Today, researchers may be working with respondents coming through multiple panels, exchanges and traffic sources, with quality assessed through many different identity, device and behavioral signals.
“As much as the technology has changed, we still have the same fundamental question: Are the respondents trustworthy?” Knapton said during the New York session. “We’re going to be making really big decisions based on the data. Can it be trusted?”
How sample quality got more complicated
When many research companies began in 2000, the approach was built around permission and a value exchange with respondents. People agreed to participate, provided information that could be used to match them with relevant surveys and they received rewards for their time. A company could recruit heavily through airlines, hotels, rental car companies and customer databases, which provided information about where panelists had come from and helped establish that they were real people.
“We didn’t really have a fraud problem,” Knapton said. “They [recruited respondents] all had an ID to get on an airplane. They had to show an ID to get in a hotel. We knew these were real people.”
As online research grew, researchers needed access to larger and more varied groups of respondents. Aggregators, exchanges and marketplaces brought together supply from multiple sources and made that access easier. They also changed how directly researchers could see where an individual respondent originated.
“There was something lost, I think, along the way,” said Knapton. “Something between the sample sources, the traffic sources, the vetting and the accountability to how all the respondents originated.”
As those sourcing models developed, quality checks also spread across more stages of the research process. Knapton described an approach that starts with vetting sample sources and continues with checks before, during and after the survey.
“You have to vet the sources. You have to track the sourcing of all the panels and traffic that you aggregate, and you need a pre-survey, in-survey and post-survey set of layers and signals that can catch any bad actors along the way,” Knapton said. “You can really evaluate a sample source by how much you catch in those three layers after the sourcing.”
When AI-generated responses look good
The signals used to assess quality now include device and browser information, typing speed, mouse movement and copy-and-paste activity. Any one of those signals may be inconclusive on its own, which makes it important to look at them together. AI-generated responses add to that challenge because they can appear coherent and thoughtful.
“We can have nonhumans taking a survey that look really good,” said Knapton. “In fact, they can even look better than humans because humans are fickle, they’re flawed, they’re responding out of emotion, not just complete rationality.”
During the Quirk’s discussion, Knapton shared that he sees uses for synthetic respondents and AI representations of real people, with the research question determining how useful those approaches can be. In the session, he also imagined an AI model trained to represent him. It could know that he likes history and movies and use that information to answer some questions.
But research about a specific experience would require information the model does not have. He said, “I think we can use synthetic data or an AI agent representative of a person or a twin or a clone, but only to a certain point for certain types of research.”
Agentic AI currently represents a smaller problem than some of the more familiar sources of bad data, according to Knapton. He pointed first to people misrepresenting themselves to qualify for a survey, followed by respondents who participate without giving the questions adequate attention. For Knapton, respondent quality comes down to whether people are both qualified to participate and attentive when they do.
“I think ultimately it comes down to we need qualified respondents that are attentive,” Knapton said. “They’re who they say they are, and they’re where they say they are.”
Snell described qualified respondents as people who have the identity, location, experience or knowledge required for a particular study. Attention comes down to whether they engage with the questions and provide thoughtful responses, with researchers using multiple signals to evaluate both.
“Is that fraud or inattention?” Snell said. “It doesn’t matter. It’s a low-quality respondent.”
AI is also being used to examine those signals and identify suspicious behavior. Knapton pointed to fraud detection alongside survey design, programming, open-end coding and analysis as areas where AI is already changing how research is conducted.
“The history is we have to adapt as an industry, and we’ve done that along the way,” said Knapton. “The problem is we tend to adapt kind of more slowly than the threat is replicating.”
What researchers should ask about their sample
Knapton said researchers should know where their respondents came from, how the sample was sourced and what happened to respondents who were filtered out along the way. He encouraged clients to ask about data sources, request percentage breakdowns and find out how many respondents were excluded for each quality issue.
He also expects the tools used for data collection and quality control to keep changing, making those questions worth asking regardless of the technology is being used.
“Technological change is a constant. You’ve got to be proactive at putting tools in place to make sure you can be confident in the data,” Knapton said, emphasizing the importance of working with companies that prioritize data quality and transparency.
“I wouldn’t just say, ‘Oh, they’re making sausage and it tastes good. I’m going to trust it.’ I wouldn’t do that,” Knapton said. “I would want to know step by step what’s their recipe and what’s their process.”