Editor’s note: Lev Mazin is CEO and co-founder of research firm Ask Your Target Market.
If you’ve ever taken a Rorschach inkblot test, you know the quandary. You’re shown what is, undeniably, a spot on the paper, but your mind starts drawing conclusions about what it “really” looks like. If you show it to someone else, there’s a good chance they’ll see something different. Same “data,” different conclusions.
Well, sometimes that happens with data from online surveys, too. Two people can look at the same data and draw different conclusions. This is particularly likely to happen if some of the data appears to be contradictory. Two common examples:
• Preference-ownership conflict — You do a survey and the majority of respondents say that they prefer Brand X but responses to another question indicate that they’re actually more likely to own Brand Y.
• Price-point conflict — Respondents report that their price sensitivity when purchasing your product category is high but they later report purchasing items known to be expensive.
Our challenge as market researchers is to recognize that there are potentially conflicting data points and to seek out the story behind them. For us, it isn’t just a gut reaction to an inkblot; we need to dig deeper.
Solving preference-ownership conflict
In many cases — and some product categories are more subject to this than others — intent simply gets derailed. What I intend to do and what I actually do can be very different. Impulse, coupons, in-store promotions, recent word-of-mouth and social media buzz can all conspire to change planned consumer behavior. I might be planning to purchase a Samsung cell phone but at the last minute I’m swayed by a promotion offered by my preferred carrier. I had the intent but my behavior was very different.
In this case, a great strategy is to be as precise as possible. For example, if you’re looking for accurate brand-related behavior, you could ask the question in two ways.
Version A: What brand of cell phone do you prefer?
Version B: The last time you bought a cell phone, what brand did you buy?
In this example, Version B is obviously more accurate because it removes the generalization from the equation. By asking a precise question about actual behavior, you’re likely to get a much more objective view. Space permitting, asking both can be very illuminating; you may have a customer group that “aspires” to own Brand A but in reality ends up often owning Brand B.
Another strategy for collecting accurate-as-possible purchase behavior is to focus on near-term time frames. Consider the following options:
Version A: How likely are you to purchase a new laptop in the next 12 months?
Version B: How likely are you to purchase a new laptop in the next three months?
Asking Version A will be more risky; it is just too long a time frame for people to predict a future purchase. If you want accurate information on planned behaviors, three-month or short time frames will get the most accurate information.
Solving price-point conflict
People can have a hard time accurately reporting their selection criteria, especially pricing-related criteria. After all, in a survey, people tend to report price is important to them because they don’t want to be telling retailers or manufacturers, “Hey, charge me what you want. I’ll pay it!” In some categories, people also simply have a hard time self-reporting their price sensitivity; they may perceive themselves as price-sensitive but not behave that way consistently (or vice versa).
How can we address this in an online survey? Understand that there is a gap and collect both the self-perception information and the actual behavioral data. For example, continuing with the laptop example, certainly include questions on selection criteria, including price. But also ask questions to assess actual price-sensitivity, which may be reflected by: brands they buy; specifications of current laptop (if they have one); and how much research they do before they buy (real bargain-hunters tend to do a lot of price comparisons before they buy).
All of the above can be good proxies for price-sensitivity.
Context and precision
You always have the possibility of conflicts in your research data. By adding some good contextual questions and being precise in your questioning, you will be able to get to the real story behind those discrepancies and collect more objective data in the process. Be careful studying those inkblots, though; sometimes your first conclusion is not the real story.