Editor's note: Chris Szczepanski is assistant vice president at New York research firm Weinman Schnee Morais Inc.
As researchers we sometimes encounter clients who focus on the score on a single measure within a study as if it represents the gestalt of respondent reactions to a concept, product or advertisement and its potential. This single score, be it purchase interest, recall or an attitude, is forced to carry an unjustified weight of responsibility as a critical determinant of a go/no-go decision. Achieve a score that satisfies the critical need and continue; fall short and a promising innovation might be cancelled.
Focusing on a single score, in most circumstances, overlooks and undervalues the supporting measures of a study. Even worse, such simplistic distillation is myopic, especially when a few points are the difference between crossing and not crossing the threshold.
When clients focus on the score to a single measure (“measure of focus”) within a study, there are often two types of reactions: take the number and run or stop everything. Such reactions are often akin to Type II (failure to reject) and Type I (incorrectly rejecting) errors, respectively.
Take the number and run
The score on the measure of focus crosses the threshold. The client, satisfied, especially if it is high, becomes unconcerned about the supporting numbers (i.e., all of the other diagnostic questions that were asked). There is no problem if both the score on the measure of focus and the data for the supporting questions are in sync. However, if the supporting numbers suggest that real-world reactions may not be as promising as the measure of focus suggests, there may be a problem.
For example, a concept, product or advertising test could yield a high purchase interest score yet comparative measures, especially among non-loyal and competitive users, may suggest low motivation to switch. Product X is a wonderful product that category users would have no problem buying/using but it is not perceived to be an upgrade/replacement for what they currently buy/use. In other words, it is an acceptable product but, lacking consumer need or switching motivation, real-world purchasing of this product is likely to be much less enthusiastic.
Another problem is ignoring potentially critical shortcomings, such as a surface cleaner that works great but leaves behind residue or damages surrounding areas. These kinds of issues make consumers think twice about purchasing a product. The more likely they are to think twice, the less likely they are to buy the product.
Stop everything
The measure of focus yields a near-miss, a poor score, or, in some instances, a score that crossed the desired threshold but not convincingly. The client, engrossed by the score, decides it is time to pull the plug. Unfortunately, tunnel vision in situations like this prevents the client from seeing that the project may have suffered a false start.
The only way to know if it has is by exploring the supporting measures, such as reasons why/why not, perceived/actual performance ratings, etc. For example, a concept, advertisement or product can be undermined by muddled communication or benefits unstated which are as important as those that are stated. Such communication can leave a consumer confused, uncertain and not knowing what to think or expect.
Price is another, usually obvious, factor, but more important than the fact that it undermines purchase interest is why it undermines purchase interest. Is it too expensive relative to the competition, available alternatives or for what the consumer gets (if they understand what they are getting)?
By closely examining the supporting numbers, you can uncover reasons why the concept, product or advertisement underperformed and why it may still hold promise. More importantly, how the promise may be realized can be discovered, too. That’s not to say the promise is achievable (e.g., cost-of-goods may indicate otherwise) but at least the path can be uncovered.
Look at all the numbers
These examples represent potentially costly mistakes that can be guarded against by helping a client to look at all of the numbers within a study. Deciding to move forward with a potentially crippling issue can lead to lost sales, lost usage occasions, slow consumption, disadoption and/or, worse, damage to the brand. Likewise, prematurely abandoning concepts, products or advertisements with promise could cause a client to miss out on a winning idea that just needs more development/fine-tuning.
Our clients are generally prudent guardians of their brand/product but some occasionally become consumed by a single measure of focus. It is our job to keep them from staring at the bark and, instead, to help them see the whole tree and, perhaps, how it may look in the forest.