Editor’s note: Emily Geisen is senior XM scientist, Qualtrics, a Provo, Utah, and Seattle-based software company.
Eliminating bias in study questions, especially when we are not aware of it, is challenging but essential. Why? Biased questions lead to inconsistent data, offensive questions, lower response rates and misleading findings.
Given that we all come from different backgrounds, cultures and life circumstances, we all have something called cognitive bias. That means we have biases we’re not usually even aware of that impact the way we process and interpret information and that influence our judgments and decisions. As researchers, we try to account for it as much as possible in our designs.
How to reduce bias
Cognitive testing is one way we can reduce bias. As researchers, it’s essential that we pretest our studies to help us identify bias before we launch them.
To do that, we use inclusive research design. Inclusive research design is the idea that the question can be understood and interpreted in the same way for all respondents regardless of differences in background or experiences. It does not mean that people will answer the questions in the same way. In fact, different backgrounds and experiences will often lead people to have very different answers to the questions, and that is part of what we want to measure.
Too often, people think that simply adjusting demographic questions is equivalent to making their studies inclusive. For example, while adding non-binary or gender-fluid response options in a demographic question about gender is indeed more inclusive, that does not solve the issue of inclusive research design. By ensuring your study meets the standards of inclusive research design, you will reduce the instance of biased questions, have greater fidelity in your data, avoid offensive questions and likely increase response rates.
Intention does not equal interpretation
All of us have faced issues with respondents interpreting questions differently from time to time. That hurts all of us and thankfully, it’s something we can avoid. Let me share one example of a time I failed and the lesson I learned.
While working on a survey for the Census Bureau, we pretested the following question, “On [date], were you experiencing homelessness?” Our intention with this question was to find out how many people were living on the streets, residing in homeless shelters, did not have a home that day or were sleeping in public areas.
We asked respondents who answered “yes” how they had come up with their answer and were surprised to find out a few of them answered that way because they didn’t have a home of their own (moved in with parents, couldn’t afford their mortgage, renting month to month, crashing on a friend’s couch, etc.). The way we understand things ultimately depends on our own experiences. These “yes” responses were considered false positives because the situations did not match the intention of the question.
The future of research design: cognitive testing
The first step in solving this issue is acknowledging how important it is to solve it. Step two is accepting that it’s never going to be perfect. We cannot account for the perspective of every single person in the world. But we can improve it.
Cognitive testing is not new in the research space, and it has never been more important. If we can’t ensure that our questions are interpreted the same way by all respondents, then the validity of our data will be suspect and our research will not be inclusive. Prior to launching any study, pretest questions with a group that is representative of the diversity of your target population, across race, gender, age and other factors. The goal is to understand the cognitive processes people use when they answer a question to determine if there is a mismatch between question intent and how people understand and answer the question. After respondents have answered a question, researchers can follow up by asking them probes such as: “How did you come up with this answer?” and “What do each of these words mean to you?”
Through cognitive testing, we can learn the reasoning behind an answer and take steps to ensure that questions are being interpreted consistently across all respondents. When questions are not working as intended, it requires rewording questions to be clearer or removing questions that are biased toward certain populations.
As we apply the principles of inclusive research design, we can reduce our own cognitive bias, gather more reliable and consistent data and conduct research that is more representative of the world around us.