Are we writing surveys for ourselves?
Editor's note: Nicky Marks is CEO at Censuswide, London, which she has spent nearly 14 years building into the research partner for global B2B and B2C brands. She holds a degree in business and management from Exeter University and Censuswide is a member of the British Polling Council. Find Marks on LinkedIn. Charles Moore is head of compliance at Censuswide, London, with 20 years of experience in market research and data analysis. He holds an economics and marketing degree from the University of the West of England and Censuswide is a member of the British Polling Council. Find Charles on LinkedIn.
When we came across the Scottish government's recent research on survey nonresponse bias,1 something clicked – not because the findings were particularly surprising but because they were unexpectedly close to home. It raised an important question about how we engage with people and the limitations we might have as researchers.
The Scottish government's work highlighted a problem that market researchers rarely discuss: The people designing surveys may be unconsciously creating barriers for the very respondents we're trying to reach. For our team at Censuswide, this realization led us to investigate whether we were doing the same thing.
Trapped in our bubble?
The market research industry could be accused of having an intentional but deeply structural literacy bias. Virtually all market research roles require a university degree. The result is that we spend our professional lives surrounded by people with a similar level of education to us and often our personal lives follow the same pattern. It's a self-selecting ecosystem that quietly shapes how we think about language and comprehension.
The numbers suggest that we might be trapped in our bubble, whether we know it or not. For example, slightly less than half the population of England and Wales has a degree, and close to one in five people hold no qualifications at all. And yet we're trying to talk to everyone and make our research open and fair.
This illustrates a fundamental disconnect between the researchers writing surveys and the broader population responding to them, a gap that can subtly influence every choice we make about wording, phrasing and what we assume constitutes common knowledge.
The literacy landscape
The national picture reinforces these concerns in stark terms: 18% of Britons and 28% of Americans aged between 16 and 65 can only read at or below the level expected of a 10-year-old, according to a major study of literacy rates2 across the developed world. For these respondents, encountering unfamiliar wording and phrasing in a survey creates genuine barriers to participation.
To understand the scale of the issue, we conducted our own research, testing comprehension of words commonly used in market research: academic, hypothetical, redundant. The results were revealing and when we asked respondents about their experiences with survey confusion, we discovered that 75% had felt confused by a survey question at least once.
Perhaps more striking was that 28% of respondents didn't know how many countries make up the U.K., which tells us something about the assumptions researchers make about what constitutes common knowledge.
These are accessibility concerns of course, as well as potential sources of systematic bias that can skew our data in ways we never intended.
How comprehension becomes bias
The mechanism is straightforward, even if the consequences are not. When respondents encounter language they struggle to understand, several things can happen: some drop out entirely; others click through quickly to move on while disengaging from the survey but remaining technically present in the data; and others misinterpret the question but answer anyway, confident they've understood.
For the survey results to make sense, you have to trust that people interpret the question as you expected they would. Otherwise you have a whole different set of factors coming into play that can undermine your entire study.
The dropout issue is relatively manageable and we, like most research agencies, simply exclude incomplete responses. The clicking-through problem is harder to address because these respondents are incentivized, with a financial benefit to completing the survey, which means they might stay in even when they're annoyed by it or confused.
But the most insidious issue is the silent misinterpretation that happens when respondents think they understand a question but actually don't, providing data that looks valid but isn't. Unlike speeding through or straightlining, which can be detected through quality checks, genuine misinterpretation is nearly impossible to catch.
The result is sample skew that happens when confusion drives away or disengages respondents with lower literacy levels, leaving a sample that tilts toward those who navigate academic language easily. The bias isn't in who you recruit but in who makes it through to the end with their engagement intact.
If they misinterpret and disengage, that's probably a smaller subgroup of those who have misinterpreted, but it's almost impossible data to rectify because they're giving a view they don't actually hold. Unless it contradicts something else, it would be almost impossible to pick up.
A quality issue, not a simplicity problem
We're careful about how we frame the solution, because if you take it to the lowest common denominator, you're going to have very simplistic wording that makes it difficult to analyze complicated things.
There's a point here that matters: Survey questions need to be precise and get to the point succinctly. Oversimplification can create its own problems, particularly when you're trying to capture nuanced attitudes or complex behaviors.
The answer isn't to make everything simpler but to make everything clearer, which means we need to reframe plain language as a research quality tool rather than a compromise. You want to succinctly get to the point but we are sometimes guilty of using words unnecessarily. When writing questions, that creates problems we could avoid.
Even when we recognize the issue, implementation isn't straightforward, partly because people skim online surveys in ways that challenge our best intentions. Eye-tracking research suggests respondents scan question text rather than reading it thoroughly, focusing more on answer options than on the carefully crafted question wording we've labored over.
Longer questions don't get more attention but just get more skimming, which creates pressure from both sides: We need to keep questions brief enough that people will engage but clear enough that everyone understands.
Definitions seem like an obvious solution and if a complex term is necessary, we can define it – but only if the definition itself is understandable. That seems like an elegantly simple solution but again, only if it's actually read and comprehended.
Balance is also a consideration and at Censuswide we have strict rules about balanced scales to prevent leading respondents toward particular answers. If positive options are included, there must be an equal number of negative options and five-point scales must have a midpoint and a “don't know” option.
The focus on not leading people gets much more attention than making sure people understand what's happening, partly because that's also a lot more difficult to know with any certainty.
Where education and expertise diverge
B2B surveys present a particular challenge because when you're interviewing executives or industry professionals, there's a tendency to assume high literacy levels and deep subject knowledge. Often that assumption holds, but not always, and that's where problems can creep in.
At our firm, we conduct a lot of B2B surveys, interviewing people who hold university degrees and are accomplished in their careers. Our wording can get technical because we assume these people hold deep industry knowledge but that assumption isn't always fair – even if they have a senior job title, there's always a possibility they might be newer to the industry.
An example is a health care survey we conducted for a U.S. client that highlighted the gap between researcher knowledge and respondent reality. The survey targeted a specific audience of B2B health care benefits brokers, navigating the Byzantine complexity of the American health care system in ways that required constant translation.
The way health care functions in the U.S. is so different from the U.K. that our project manager kept coming back to the client (the researcher in this example) asking for clarification: What does this mean? What are you trying to get at? The fact that he didn't have the cultural context essentially tested the client's assumptions of how easily the questions could be understood. The process made the client explain it in plain language, which led to clearly worded questions.
That back-and-forth, we would argue, should be a crucial part of the process, because research agencies aren't experts in every industry we survey. That's far from a weakness – if we embrace it, it forces a translation from specialist language to something accessible that serves the research better.
We're experts in research, while our clients are experts in their industries. The combination of that knowledge and the humility to ask clarifying questions before a project goes into the field is what leads to good surveys. Researchers who are open to that collaborative process are likely to have a lot more success than those who just field what they think will be understood.
An industry-wide conversation
Data quality is rarely discussed openly in market research, making it somewhat a dirty secret of the industry. There's understandable nervousness about raising it with clients when it acknowledges that there's invalidity in the responses we collect.
At our firm, we take data quality seriously and remove respondents as standard for speeding, straightlining or inconsistent answers. We use red-herring questions and we deploy sophisticated digital fingerprinting technology along with checks to catch automated responses that might otherwise contaminate our data.
But comprehension bias is different in that it can't be caught with quality checks, doesn't show up as speeding or straightlining and could shift the composition of your final sample in ways that are nearly impossible to detect.
When comparing literacy bias to other sources of bias like social desirability or sampling error, we're candid about the challenge: it's very difficult to detect because the incentivization bias is easier to remove with data-quality checks. But if respondents are misinterpreting questions and that's leading them to disengage, that's almost impossible to pick up with our current tools.
What good looks like
So what would change if the industry took this seriously? Fundamentally, it would just be more accurate data, where the question we think we asked would be the question we actually asked. That would mean we're better at answering what the client wants and ultimately would make research better at serving its purpose.
Implementation comes down to awareness first, then incremental change. This article is not meant to be prescriptive about specific solutions, because we don't claim to have solved the problem entirely ourselves. While we can't really say what the end looks like, starting to think about it already gives you a better way to do research.
Our advice to fellow researchers is to recognize your own biases – not just wording or educational background but all the ways your personal and professional experience shapes how you interpret the world and assume others do the same.
It's a very difficult thing to do but by being aware of the problem you start to solve it. Realizing your own biases is a process we should go through as researchers if we want to improve the validity of our work.
A live example
To test our hypothesis, we ran a survey of 2,000 people where we asked two pairs of questions with a very similar sentiment:
Q1: Have you ever concealed a misunderstanding you had about a topic?
Q3: Have you ever hidden a misunderstanding you had about a topic?
The results:
When we changed "concealed" in Q1 to "hidden" in Q3, we found that 38% of people who had responded "I don’t know" for Q1 were then able to answer in Q3.
Q2: How onerous do you find completing surveys?
Q4: How hard do you find completing surveys?
The results:
When we replaced "onerous" in Q2 with "hard" in Q4, we found that 39% of people gave a different answer.
But there was no clear switch from one answer to another. Rather, there were varied different answers in Q4 compared to Q2.
We suspect the changes may be due to "onerous" and "hard" having slightly different meanings. However, the high percentage of different answers still show the impact wording can have on results – overly complex language and questionnaire design (in this case, by not including an opt-out) can exacerbate the problem.
Beyond individual question wording
This conversation extends beyond individual question wording and touches on who gets to participate meaningfully in research, whose voices are captured accurately and how unconscious assumptions shape what counts as valid data in ways we rarely examine.
This dynamic plays out in research through the potential for slight differences in how things are done, in upbringing or in education to lead to vastly different conclusions about what seems normal or what needs explanation. The challenge is bringing all those perspectives in and ensuring everyone is interpreting questions the same way, which requires us to step outside our own assumptions about what's clear.
What makes the world interesting also makes our work difficult when we're trying to capture views across such diverse experiences and backgrounds.
The solution isn't complex, even if implementation is challenging. Start with awareness, question your assumptions, test comprehension and write for clarity rather than academic convention. Treat plain language not as dumbing-down but as a quality control measure that ensures your data actually captures what you're trying to measure.
Because if respondents can't understand the question, you're not measuring their views. Instead, you're measuring who had the educational background to navigate your wording and that's not the same thing at all.
References
1 “Understanding survey nonresponse behaviours: evidence and practical solutions.”
2 “Do adults have the skills they need to thrive in a changing world?”