Editor's note: Margaret Roller is principal at Roller Research, Gloucester, Va.
Researchers of all types care about bias and how it may creep into their research designs, resulting in measurement error. This is true among quantitative researchers as well as among qualitative researchers who routinely demonstrate their sensitivity to potential bias in their data by way of building interviewer training, careful recruitment screening and appropriate modes into their research designs. It is these types of measures that acknowledge qualitative researchers’ concerns about quality data; and yet, there are many other ways to mitigate bias in qualitative research that are often overlooked.
Marketing researchers (and marketing clients) in particular could benefit from thinking more deeply about bias and measurement error. In the interest of “faster-cheaper-better” research solutions, marketing researchers often lose sight of quality design issues, not the least of which concern bias and measurement error in the data. If marketing researchers care enough about mitigating bias to train interviewers/moderators, develop screening questions that effectively target the appropriate participant and carefully select the suitable mode for the population segment, then it is sensible to adopt broader design standards that more fully embrace the collecting of quality data.
Think about assumptions
An example of a tool that serves to raise the design standard is the reflexive journal. The reflexive journal has been the subject (in whole or in part) of many articles in the blog Research Design Review as well as scholarly journals. A reflexive journal is simply a diary of sorts that is utilized by the qualitative interviewer or moderator (as well as the observer in ethnographic research) to think about (reflect on) how his/her assumptions or beliefs may be affecting the outcomes (i.e., the data). It enables the researcher to reassess (if necessary) his/her behavior, attitude, question wording or other aspects of data collection for the purpose of mitigating distortions in the data. A reflexive journal (as depicted in Figure 1) asks the researcher to reflect on questions such as, “What do I think I ‘know’ from this/these participant(s)?” and “How do I think I ‘know’ it?” as well as, “What assumptions did I make about the participant(s)?” and “How did my personal beliefs shape the questions I asked?”
The reflexive journal appears to be a particularly vague or foreign concept among qualitative marketing researchers (and marketing clients) given the absence of discussions concerning this tool in their research designs. Why is this? Is there an acceptance that interviewer/moderator training sufficiently guards against potential bias? Is there a belief that all qualitative research is biased to some degree – because, after all, it isn’t survey research – so any attempt at mitigation is futile (which, of course, begs the question: Why bother with qualitative research at all?)? Is there a head-in-the-sand (i.e., not-wanting-to-know) mentality that refuses to think of the interviewer/moderator as someone with assumptions, beliefs, values and judgments but rather as a “super human” who is able to conduct a semi-structured in-depth interview (IDI) or focus group discussion devoid of these human qualities?
The humanness in all of us is worthy of reflection. And in qualitative research design this reflection can be put to good use mitigating bias in our data. As the interviewer considers how certain behavior may have elicited responses that were not true to the participant, or the moderator reflects on how his/her favoritism and attention towards a few focus group participants over others shifted the course of conversation and the outcomes of the discussion, these researchers are using their introspection to improve the research by moving data collection (and data outcomes) to a higher standard. This is how interviewers learn to adjust the interview guide or consciously alter their behavior during an IDI to gain more accurate data, or the moderator comes to understand his/her own prejudices and finds corrective techniques to become a more inclusive moderator and ensure an evenhanded approach to the discussion.
Two important and unique attributes to qualitative research methods are the “researcher as instrument” component – i.e., the researcher is the data collection tool – and the participant-researcher relationship. These attributes speak to the humanness that both enriches and complicates the social-exchange environment of the IDI and focus group discussion. And it is this humanness – embedded in qualitative research – that should obligate marketing researchers to consider its import in achieving quality outcomes. If marketers care enough about the integrity of their data to adopt high standards in training, recruiting and mode, why not care enough to mitigate bias in data collection by utilizing tools – such as a reflexive journal – to seriously examine the human factors that potentially increase inaccuracies and error in the final data?