Understanding demographic questions through implicit testing

Gaining insights through implicit testing 

Editor’s note: Sara Yang, InsightsNow, is a member of the Insights Association IDEA Council. The Insights Association IDEA Council would like to thank the sponsors of this study, including InsightsNow, Logica Research and Escalent for the design, execution and analysis of the study; Cint, Dynata, Kantar, InnovateMR and Tap Research for providing sample; Nuance for coding open-ended questions; G3 Translate for translation; and Confirmit for hosting.

In a recent study, we employed implicit testing to help our industry create more inclusive demographic questions by creating deeper insights into human behavior. The first wave of this research looked at race and ethnicity questions in survey design. By utilizing implicit and explicit testing, we were able to better understand people’s emotional reactions and subconscious associations with their experiences when asked race and ethnicity questions. The incorporation of this approach, along with scaled responses and forced choice exercises, allowed the research-on-research team to gather the insights needed to tell a stronger, more well-rounded behavioral story about each person’s preferences for inclusive questions.   

The objective of the study is to develop approaches where researchers can ask questions about race and ethnicity in a way that is inclusive and sensitive, so it is important to understand the reactions evoked by asking the questions in different ways. Using the implicit testing technique, the study was able to quantify how many people reading and answering a race or ethnicity demographic question have fast implicit positive emotional reactions, such as happy, proud, confident, loved – and how many felt angry, embarrassed or disappointed. These fleeting emotions are quickly captured by projecting onto a metaphorical image of the person’s choice, and then completing a short series of reaction tasks. The percentage of people who have positive or negative implicit emotional reactions is the key to understanding their gut reaction.  

For more complex sentiment reactions, such as “it is quick and easy to answer,” “it is politically correct” or “it makes me feel uncomfortable to answer,” the same testing technique (called The Implicit/Explicit Test) is used but the data is weighted based on whether their reaction was implicit or explicit. This provides an overall aggregate score which accounts for both the fast implicit reactions as well as the slower more pondered explicit reactions. The timing for fast implicit versus slow explicit is specific for each individual and determined with a calibration exercise at the start of the survey. 

The results show that most people (about 70%) experienced at least one positive implicit emotion when reading and answering the race and ethnicity question types included in this study. In particular, those who were tagged in the panels as Black/African American experienced implicit emotions of pride, confidence, excitement and feeling loved more than other groups. Some qualitative responses to the question included, “I am proud to be a beautiful black woman, I have never been ashamed of my race. I was always taught to be proud of it.” 

 On the other hand, about one in four experienced one or more negative implicit emotions, across all question formats tested. One in seven people were implicitly offended, and this population segment is more likely to identify with multiple races or another race/ethnicity not typically listed, younger people or those who take fewer surveys. This is a subset of our audience whom we should pay careful attention to if we want to make our surveys more inclusive and sensitive for all. With the knowledge that many feel negatively about answering race and ethnicity questions in surveys, it is worth considering on a case-by-case basis whether it is truly necessary to include the question in your specific study design. To learn more about steps to take to be more inclusive, read the article “How to ask race and ethnicity questions in a more inclusive way,” and view the “On Good Terms” presentation here (registration required). 

The next phase of research will be a study on gender and sexual orientation identity. 

If you’re interested in supporting this ongoing research initiative, e-mail the IDEACouncil@insightsassociation.org or visit the IDEA Council on LinkedIn.