Editor's note: Michael Lieberman is founder and president of Multivariate Solutions, a New York research consulting firm. Jon Montgomery is a partner at Hudson Group, a New York research firm. This article appeared in the September 13, 2010, edition of Quirk's e-newsletter.
Marketing researchers face a number of measurement challenges when interviewing their audiences. Certain factors limit subjects' ability to provide accurate or reasonable answers to all questions: they may not be informed; may not remember; or may be unable to articulate certain types of responses. And even if respondents are able to answer a particular question, they may be unwilling to accurately disclose sensitive information at the risk of it causing embarrassment or threatening their prestige.
Research projects that investigate socially-risky behavior (e.g., "Have you used illegal drugs in the past week?") or, conversely, socially-expected behavior (e.g., voting, religious attendance, charitable giving, etc.) are subject to what are referred to as social desirability pressures. These sorts of questions will almost certainly yield inaccurate results if not administered correctly.
Social desirability distortion is the tendency of respondents to answer questions in a more socially-desirable direction than they would if the survey were administered anonymously. A form of measurement error, it is often referred to as bias, socially-desirable responding or response distortion.
Over the past few years, we have employed a questionnaire technique called the item-count method to reduce the effects of social desirability distortion. As used, respondents report only the number of items on the list in which they have engaged, not which behaviors. If the average number of non-stigmatized behaviors is known for the population, one can estimate the rate of the sensitive behavior for the population by the difference between the average number of behaviors reported for the population including and excluding the stigmatized behavior
Item-count method
The item-count method allows survey respondents to remain anonymous when reporting a sensitive behavior. This is accomplished by including the sensitive behavior of interest in a list of other relatively non-stigmatizing behaviors. It is also used to estimate socially-positive behaviors as well, such as voting.
In the example below, the researcher is attempting to identify the percentage of college students who tried or used cocaine in the past. One of the groups received a set of five items. The participants were told that this questionnaire was designed to encourage honest responding and were asked not to respond directly to whether any particular item is true; rather, they were asked to list how many of the four items were true.
How many of the following have you done in the past six months?
While indicating how many items were true, they never directly endorse any particular item. Someone who responded three, for instance, was indicating that three of the five items were true for him or her.
Another group of respondents were given the same list, plus one additional behavior - the one we are interested in measuring.
How many of the following have you done in the past six months?
Subtracting the average number of behaviors reported by the first group from the average number of behaviors reported by the second group estimates the proportion of people given the longer list who said they performed the requested behavior.
There are a few guidelines when designing an item-count method experiment:
- The behavior used on the item-count method list should be such that few respondents have performed all or none of them. Reporting one activity negates the anonymity.
- Behaviors should be within the same category. For example, if one is investigating risky sexual behavior, then other risky behaviors should be included on the list. If the goal is to
estimate voter turnout, other civil activities should be included. - A separate sample for direct reporting may be included for comparison.
- Larger samples enhance estimate stability and accuracy.
Below are preliminary results, by gender.
The base rate estimate for the behavior of interest is found by subtracting the two means: mean (Group 2) - mean (Group 1). In this example, in total Group 1 conducted 2.35 behaviors; Group 2 conducted 2.72 behaviors. Thus, the base rate estimate in this population for cocaine use is
2.72 - 2.35 = 0.37, meaning roughly 37 percent of our college student sample has used cocaine at one time or another.
Broken down by gender, men have slightly higher means of risky behavior. Not surprising, given that, for example, riding a bike without a helmet or walking through a dangerous neighborhood is viewed as less risky by male students than female. Men also had a slightly higher estimated cocaine usage.
An ongoing challenge
Reducing measurement error is an ongoing challenge for marketing researchers. Respondents are generally unwilling to respond, or respond truthfully, to questions considered inappropriate for the given context; not seen as serving a legitimate purpose; or sensitive, embarrassing or threatening to their own self-image. Utilizing the item-count measure can be an effective way of reducing misreporting caused by social desirability pressures associated with interviewer administration.