Editor's note: Bob Wood is director of data science at J.D. Power with more than 20 years of experience in analytics, market research and consumer behavior. He has led data science organizations at Lands’ End, Resonate and Merkle, supporting major global brands. He holds advanced degrees in mathematics, statistics, business and psychology, including a Ph.D. focused on consumer decision-making. Toshi Yumoto is a machine learning engineer at J.D. Power and Associates, with over 20 years of experience in data science, AI, quantitative research and psychometrics. His work focuses on applied machine learning, LLM systems, Bayesian networks, anomaly detection and fair evaluation methods. He holds a Ph.D. in measurement, statistics and evaluation from the University of Maryland, College Park.
Few issues worry insights professionals more than bad survey data. We can design elegant questionnaires, recruit carefully and analyze rigorously, but if a meaningful share of respondents is not answering each question thoughtfully and truthfully, everything downstream is at risk. Models become noisy, segments blur and decisions drift away from reality.
The uncomfortable truth is that bad data is not rare. Multiple industry studies over the past decade suggest that anywhere from 15% to 30% of online survey responses may be partially or wholly unreliable (Callegaro, 2014) (Gao, 2016). The exact number varies by panel, incentive structure and survey length, but the problem itself is well established. As research continues to move faster and cheaper, the economic incentives that produce low-effort responses have not gone away (Kennedy, 2020).
The harder question is not whether bad data exists but how we should detect it without damaging the data we are trying to protect.
The cost and prevalence of bad survey data
Low-quality responses create two kinds of damage. The first is obvious: respondents who click randomly, straightline grids or answer without reading dilute the signal in the data. The second is more subtle and often more dangerous. Standard cleaning rules tend to remove certain types of respondents more than others, introducing bias even as they reduce noise. For example, removing speeders tends to bias results toward higher education and older populations (Zhang, 2014).
Researchers have long documented this tension. For example, speeders do not always provide poor data and slow respondents are not necessarily careful ones. Attention checks can identify some careless respondents, but they can also frustrate engaged participants and change how they answer later questions. Pattern detection catches obvious straightlining but misses more sophisticated forms of satisficing. Self-reported effort checks tend to identify only a small fraction of problematic cases.
In other words, the industry knows bad data is common, but the tools we use to find it are blunt. They often work best for the most extreme cases and worst for the gray area where real harm occurs.
Standard approaches and their limits
Most survey quality controls fall into a few familiar categories.
Timing-based rules flag respondents who complete surveys “too quickly,” assuming speed implies inattention. In practice, completion time can be helpful, but it can be noisy and needs to be combined with other checks. Some kinds of respondents just generally go faster, especially experienced panelists or mobile respondents (Berry, 2019).
Instructional manipulation checks and trap questions test whether respondents are reading carefully. These can work, but they are also easy to learn and avoid. More importantly, they interrupt the natural flow of the survey and can change respondent behavior after the check is encountered by interrupting the interview flow (Paas, 2018).
Consistency checks repeat or mirror questions to see whether answers align. While intuitive, these checks lengthen surveys and can feel punitive to attentive respondents. They also assume that inconsistency is always a sign of carelessness, which is not necessarily true in attitudinal research.
Pattern detection looks for straightlining or simple response sequences. This is useful for the most obvious fraud but fails when respondents give random-looking answers that still pass superficial pattern tests.
Each of these methods can remove some bad data. None of them scale well to the more common problem of respondents who answer plausibly but without much thought. And nearly all of them rely on external rules imposed on the data rather than on the structure of the data itself.
A different idea: measuring internal consistency
An emerging class of approaches starts from a different premise: thoughtful respondents tend to give answers that hang together, even when the survey does not explicitly enforce consistency. Someone enthusiastic about their job is more likely to recommend their company. A satisfied customer is more likely to buy again. An experienced hiker will more likely have clear opinions about camping equipment.
This idea leads to what we can broadly call an internal consistency measure (ICM). Instead of testing respondents against predefined traps or timing thresholds, ICM methods examine how well each respondent’s answers conform to the natural relationships present in the survey data.
At a high level, the logic is simple. In most surveys, some questions are statistically related to others. These relationships may not be obvious on the surface, but they are detectable in the aggregate data. If a respondent answers thoughtfully, their responses should generally respect those relationships. If they answer randomly or carelessly, their answers are more likely to violate them.
The key is that the method does not require us to specify those relationships in advance. They are learned empirically from the data itself.
How internal consistency is modeled
In practice, an ICM approach typically works in three stages.
First, the system identifies which survey questions share meaningful statistical relationships. This is often done using measures such as mutual information, which captures how much knowing the answer to one question reduces uncertainty about another. Mutual information is more flexible than simple correlation and works well with categorical survey data.
Second, these relationships are organized into a probabilistic structure, often using a Bayesian network or something similar. Each node represents a question and connections represent learned relationships between questions. The structure reflects how responses tend to co-occur across the full sample. These connections between questions aren’t assumed to be causal, but the answers thoughtful people give between these questions are related.
Third, each individual respondent is evaluated against this structure. The system computes how likely that respondent’s full set of answers is, given the learned relationships. Respondents whose answers consistently violate expected dependencies receive lower internal consistency scores.
When these scores are plotted across the sample, a familiar pattern often emerges. Most respondents form a smooth distribution of relatively good consistency. A smaller group often forms a distinct tail or secondary cluster with lower scores. These are the responses most likely to reflect low effort or fraudulent and random behavior.
Crucially, this evaluation happens after data collection and does not depend on special preset survey design features. The survey can be written naturally, without trap questions or artificial repetition.
Why this approach is different
The internal consistency method addresses several limitations of the more traditional quality controls.
ICM is content-agnostic. The system does not need to “understand” the meaning of the questions, only the statistical relationships among answers.
It is adaptive. The relationships are learned from each dataset, making the method robust across topics, audiences and survey designs.
It is non-disruptive. Because no attention checks or traps are required, respondents are not prompted to manipulate their behavior mid-survey.
And, instead of a simple pass-fail rule, respondents receive a consistency score. Researchers can decide whether to remove the lowest scorers, down-weight them or simply flag them for sensitivity analysis.
In large, real-world surveys, this class of methods has been used to identify problematic responses that traditional rule-based checks likely would have missed, while leaving engaged respondents much more likely to end up in the analysis data set. The intuitive appeal is strong: We are judging answers by how well they fit with each other, not by how fast someone clicked or whether they noticed a trick.
Practical considerations and caveats
Internal consistency is not a silver bullet. Legitimate respondents with genuinely unusual attitudes can also produce low-probability answer patterns. For that reason, it is more practical to treat ICM as a diagnostic tool, not an automatic deletion engine.
It also works best with surveys that contain enough interconnected questions for relationships to emerge. Very short surveys or collections of entirely unrelated items offer less structure to learn from.
Finally, as with any advanced method, transparency matters. Researchers should understand how scores are generated and should test the impact of exclusions on key outcomes.
A stronger foundation for cleaner data
The industry has spent years trying to outsmart careless respondents with faster checks and cleverer traps. Internal consistency approaches suggest a quieter, more robust addition or alternative: let the data itself tell us when something does not add up.
By focusing on how answers relate to one another, rather than on how respondents behave externally, these methods align more closely with how thoughtful responding actually works. As survey data continues to underpin high-stakes decisions, that shift in perspective may be one of the most important advances in data quality in years.
Bad data may be inevitable but letting it damage the good data no longer needs to be the price we pay.
References
Berry, K. R. (2019). “Factors associated with inattentive responding in online survey research.” Personality and Individual Differences, 149, 157-159.
C. Kennedy et al. (2020, 2 18). “Assessing the risks to online polls from bogus respondents.” Retrieved January 22, 2026, from Pew Research: https://www.pewresearch.org/methods/2020/02/18/assessing-the-risks-to-online-polls-from-bogus-respondents/
Callegaro, M. e. (2014). “Online Panel Research: A Data Quality Perspective.” John Wiley & Sons.
Gao, Z. L. (2016). “Online survey data quality and its implication for willingness-to-pay: A cross-country comparison.” Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie 64(2), 199-221.
Paas, L. J. (2018). “Instructional manipulation checks: A longitudinal analysis with implications for MTurk.” International Journal of Research in Marketing 35.2, 258-269.
Zhang, C., and Conrad, F. (2014). “Speeding in web surveys: The tendency to answer very fast and its association with straightlining.” Survey Research Methods. Vol. 8. No. 2, 127-135.