Watch, don't listen

Editor’s note: Douglas Ryan is president of Ryan Consulting, a Chicago research firm.

“Listen to your customer” is an adage invoked to both explain successful companies and to chide poor market performers. Many industry analysts point to customer-based innovation as the one true differentiator for companies in a world where technology and competition have reduced the time and returns from any particular product or service enhancement. That is because successful new features in almost any industry, whether it is in automobile design, online shopping or insurance products, are quickly assimilated, imitated or leapfrogged by competition. Thus, companies wishing to generate continued growth cannot rely on the advantage of any particular feature, but on a process by which a product or service is continually improved more quickly and more in line with evolving market needs than the competition. This recognition drives a rush to include words like “market-driven” and “customer-centered” in their mission statements, along with a stepped up effort to gather increasing amounts of market feedback in the forms of surveys, focus groups, telephone interviews and other traditional research mechanisms.

In many companies, this enthusiasm for market feedback turns into cynicism as the various research studies pile up without any significant improvement in the product development process. In some cases, the increased depth of data fails to translate into deeper or more accurate consumer insights. In other cases, the data becomes a tool for no-fault decision making rather than a platform for better decisions (e.g., “We didn’t include that feature because it didn’t score well with our customer panel.”). In still other cases, the new emphasis turns the company into the trappings of a bad political candidate, shifting from one approach to another in order to conform to the latest poll results.

The problem is that listening to your customers can often be a bad idea. This is because what customers say is not always a good indication of what they do. This is a well-established though well-ignored fact human behavior. A study in 19501 of telephone users asked whether they would prefer lighter headsets. The general answer was no. Yet, when the same subjects were presented with identical looking headsets that were of different weights, the lighter weights were widely preferred. An international study of international retail brands by McKinsey provides a more recent example of this same dissonance:

By comparing 1,500 consumers’ ratings of how well the stores performed with the store choices these consumers actually made, we found that what they say and what they do are not always identical. Customers tend, for example, to say that they don’t shop in particular stores because their friends do, but their friends’ shopping choices turn out to be powerful motivators. Customers also overstate the importance of certain issues. In choosing grocery stores, for example, German shoppers are less influenced by the range of products stocked than they claim to be, so retailers that spend heavily to offer a wide product range might achieve better results by investing, for example, in more targeted marketing to boost a store’s attractions for affinity customers.2

Why traditional listening tools may not work

There are numerous reasons why traditional marketing research techniques may not work. Some of them have to do with poor technique, such as using poorly structured or leading questions, or drawing quantitative projections from qualitative tools. But even if the techniques are properly used and executed, there are some inescapable pitfalls in the most common tools used for customer feedback. Among these are the following:

Self-selection bias
The very nature of responding to a survey or an invitation to a focus group reflects some ingoing degree of bias. The bias can be large or small depending on the circumstances. As an extreme example, a survey on marital sexual practices is likely to have a significant self-selection bias. Most people are hesitant to talk openly about intimate details of their lives with strangers, so those willing to respond are likely to have a particularly untypical mindset on the subject. Less dramatically, a company trying to market a service to busy C-level executives should be aware that anyone willing to take the time to show up to a focus group may not represent their core target. Self-selection biases show up in every circumstance when you require someone to raise their hand to participate. These biases will make the results of the research less reliable.

Social codes
People tend to observe the standard social codes even in commercial transactions. It is typical practice for a survey to begin by asking for the respondents to give their honest responses, but it is hard to put aside established behaviors that may dilute that honesty. In social situations, we learn to employ a range of behaviors to avoid conflict or to avoid offending someone. It is hard for people to suddenly put aside those ingrained behaviors. That difficulty is multiplied in group situations where people are not only interacting with a moderator or questionnaire but with other respondents. These behaviors tend to gloss over differences in opinion, both negative and positive, and make it harder to identify the most significant issues in an interview subject’s response.

Avoidance of negative judgments
None of us likes to admit our failings, or to admit to traits that what we know others perceive to be failings. People tend to underreport the things that they are not supposed to do (e.g., the amount of television they watch) and overreport the things that they know they are supposed to do (e.g., how much of exercise they get). One of my nieces works as a babysitter, and estimates that the bedtimes parents tell her are “typical” for their children are usually between one-half hour and one hour before their actual bedtimes. Because most parents believe earlier bedtimes are better for children, they are prone to give the babysitter a more idealized version of their parenting discipline. This is a common problem for doctors, who often get misleading information from patients who don’t want to reveal unhealthy parts of their lifestyle. So research based on self-reported data is skewed by people’s desire to appear either more intelligent, organized, healthy, hip or generally better than they actually are.

Rational projection
People like to think that they make sensible choices. In truth, we make so many decisions each day about so many different things that most of us don’t do a rational analysis for each choice. Nonetheless, in our desire to appear sensible to either ourselves or others, we often come up with reasoned explanations for our behavior, if pressed. In truth, we are rarely the logical creatures we pretend to be and emotions play a significant role in our decisions. Behavioral economists cite numerous instances when people systematically and consistently act irrationally. One common irrational behavior is referred to as the “framing effect,” in which the same person will make a different decision based on how a choice is framed. For example, a problem about whether to administer a new drug to a group of 1,000 sick people can be presented as a gain (200 of 600 threatened people will be saved) or as a loss (400 of 600 threatened people will die). Although the outcomes are the exact same in each case, people tend to decide for the vaccine when presented as a gain and against it when it is presented as a loss.3

Consumer studies4 in the beer industry provide an equally common if less academic less example of irrationality. Casual beer drinkers who drink mostly national brands are hard-pressed to identify their brands in a blind taste test. Yet when these same consumers are asked why they select a certain brand, they usually will refer to the taste, or ingredients related to taste, as their primary criterion. Rarely will drinkers refer to more emotional factors such as the brand image or the influence of their friends. That same scenario is played out over many categories because people look to provide a rational explanation for their choices when challenged to explain their behavior. This is problematic for researchers trying to determine the most relevant purchase criteria, as it often leads to either a mischaracterization of the main influences or a misprioritization of those influences.

Artificial environments
Some of the inherent difficulties of market research have an analogy in aspects of the Heisenberg Uncertainty Principle in physics. The Uncertainty Principle concludes that it is impossible to determine both the precise position and the momentum of a particle. In unscientific terms, the reason for this results from the fact that you can’t measure something without bumping it a little, so the very act of measurement affects the results. The famous Western Electric time and motion studies in 1939 bore this out. In those studies, researchers were trying to determine the optimal environment for enhancing worker productivity. The experiments were confounded, however, by the workers’ efforts to make a positive impression. Knowing they were being observed, the workers upped their output independently of the variables being controlled.

Whenever you put people into an artificial environment, be it physically or mentally, you affect the way they behave and think. The more artificial the environment, the more likely you are to get artificial responses.

Evaluation vs. projection
A good portion of market research is flawed from the beginning because it asks its subjects for expertise that they don’t have. It asks them to act as product designers, or worse yet, to predict the future. Some of the most profound thinkers of the business world, armed with a trove of data and years of training, try to do this with limited success, yet there is an almost mystical belief that a person off the street can perform that function. It is unrealistic to expect consumers to be futurists or new product designers. Humans tend to think linearly, and to project from what they already know. So if you ask a typical consumer to describe the ideal features of a new car, for example, you are likely to get a description of a car with more of one thing or less of another. In a story that typifies many product successes, Sony developed the Walkman in 1979 based on its own observation of consumer trends. Users of transistor radios, tape recorders, and boom boxes were telegraphing the desire for a more personal transportable sound system. Even the warning signs on buses and subway trains banning the playing of radios were a signal of the strong desire for people to listen to do just that as they commuted.5 Yet no survey, phone interview or focus group ever resulted in the suggestion for the Walkman.

In his neo-classic book Innovator’s Dilemma,6 Clayton Christensen describes the futility of expecting consumers to provide new product direction in the case of disruptive technologies. Christensen describes disruptive technologies as those that change the accepted value proposition of a product category, as opposed to sustaining technologies that provide an improvement on current performance attributes. He cites examples across numerous industries of customers explicitly rejecting products that were to become the future mainstream leaders.

Consumers are excellent at evaluating how well something does or does not meet their needs. They are not proficient at designing better ways to meet those needs. In the words of the Web usability maven Jakob Nielsen,7 “users are not designers, and designers are not users.” Just as the skills required to be a first-rate film critic are different from the skills necessary to be a first-rate film director, so are the abilities of a consumer to evaluate a product different than the ability to create a product.

Watching instead of listening

Both figuratively and literally, watching is superior to listening. Watching implies observing choices that consumers make with a minimum of interference. In most every circumstance, observing how consumers actually behave provides more useful feedback than having them talk about how they behave. Having a record of how many times a consumer purchased your product is worth more than having that same consumer describe herself as a heavy or loyal user. In Aristotelian fashion, behavior reveals character in a way that speech does not. As observed in the examples in the introduction, what people say is often at odds with what they do.

To ensure the most actionable information, the careful marketer should observe these guidelines:

Watch the right people
Avoid the self-selection bias by focusing on the people who are in the best position to give you the information you need rather than the people who are easiest to access. Technical product or service companies often make the mistake of enlisting consumers eager to talk about new gadgets. As a result, their interactions are dominated by early adopters who may or may not represent the true target group. Savvy researchers should avoid marketing research lists, which can be dominated by “professional” responders. Instead look to organizations or situations where your target is likely to be. For example, a company wishing to observe working moms might approach a local day care center for subjects.

Part of making sure you’re getting both the right people and the right information is to use a control group. If you want to learn more about a group, study their contrasting counterparts as well. You’ll learn more about heavy users by including light users, more about loyal customers by including non-loyal, etc. Because existing users have already demonstrated some affinity for a product’s value proposition, it is non-users who are more likely to offer overlooked perspectives. The importance of control groups was brought to light by recent developments in medical treatments for people with back problems8. Doctors had been routinely recommending back surgery for years to treat patients complaining of back pain and whose CT or MRI scans revealed damaged disks. But then a recent study of otherwise healthy people found it was not unusual to find degenerated disks in people who are fully functioning. As a result, the benefit of these frequent surgeries is now being called into question. The lesson is to make sure a control group is present to guide your insights and conclusions.

Observe choices instead of eliciting judgments
Actions speak louder than words. Observing or recording the choices consumers make provides more reliable and objective data than asking them for their judgments and opinions. This fact was a key force behind television rating service AC Nielsen’s accelerated move to automated people meters. Early trials with the meters had the unintended effect of highlighting the flaws in the household diary process that the company had been relying on families to fill out for years. Memory, habit, and other human traits created a gap between what “Nielsen” families filled out in their diaries and what it turned out they actually were doing.

In this vein, watching consumers shopping or asking them to bring in store receipts is far more effective than asking them what they have purchased in the last week. Watching or measuring respondents’ choices in a natural environment without overt interference is more effective than asking them to list their preferences. A clever local travel agency ran a promotion last year that asked contest entrants to select from five equally priced U.S. travel destinations for their grand prize. That promotion provided better data about the respondents’ vacation preferences than a general survey because entrants had to consider their destination choice in the context of actually being sent there. Research that is built around real choices generates more meaningful consumer feedback.

Strive for invisibility
The ideal situation is to collect information with a minimum of intrusion. The more that people are aware of being monitored, the more likely they are to alter their patterns of behavior. Web sites provide a close to ideal model of being able to unobtrusively measure where people visit and how long they stay. Total invisibility is difficult to achieve, but anything that reduces the effect of the measurement increases the validity of the measure.

The invisibility goes both ways. If the users can respond anonymously, their choices are likely to be less self-censored. True anonymity is hard to construct if you have to track the results over time, or want to go back to the same source for follow-up research. In this case, allowing subjects to use pseudonyms or impersonal usernames can provide them with more comfort in expressing strong or critical opinions.

Meet them in their environment
This goes hand-in-hand with the goal of invisibility. In the spirit of anthropological field studies, whenever possible, the research should go to the subject and not the other way around. Consumers should be allowed to sample new products in the place and situations where they would be expected to use them. New features for a PDA would be better examined by giving them to an executive heading off on a business trip rather than sitting in a contrived usability lab. Similarly, TV ads are more effectively evaluated if inserted among the clutter of programs and other ads of an evening at home, than if reviewed by themselves in a research lab with a moderator. A more realistic context leads to more reliable responses.

Avoid abstraction
The more abstract and conceptual you ask consumers to be, the more you move away from their expertise as research subjects. To avoid this, marketers should strive to provide concrete stimuli rather than conceptual stimuli. When dealing with new products, it may not be possible to have existing products for consumers to experience. In this case, even rough prototypes are far more valuable than positioning statements or product descriptions. Web sites should appear on a computer screen (even if no links actually work), print ads should appear in a magazine, and products should appear in as close to a form as people would find them on the store shelf. In early development stages, these prototypes will often be rudimentary. That is perfectly fine as part of an iterative process that leads to more robust prototypes. The goal should be to make something as real as possible for the consumer in order to ensure the most useful feedback.

Researchers also have to be careful to avoid eliciting predictive leaps from consumers. Watching instead of listening helps keep the learning focused in the present and not wandering into the future tense. Referring to the Walkman example from above, questions such as “What new features would you like?” or “How could this be improved?” ask the consumer to go beyond their interest and expertise. That is akin to a doctor asking a patient what medication to prescribe. The focus should be on understanding how an existing product or prototype meets or fails what the consumer is using it for.

Connecting the dots

There are issues of cost involved with these recommendations. In fact, many of the reasons the traditional tools are constantly (mis)used is that they are relatively easy and cheap to implement. While it does take more effort to observe people in their everyday lives than to mail out surveys or gather around a focus group table, watching does not have to be more expensive than listening. The improved richness and accuracy of the watching can justify a smaller pool of subjects, and thereby cost no more than traditional tools.

The larger issue around these recommendations may be around existing mindsets. This approach refutes the notion that the secret to market success lies in the mouths of every consumer, and the resulting belief that if you just talk to them, those secrets will be revealed. If this were true, the most valuable marketing tool would be a camcorder. The more accurate model is to think of market research as a game of connect-the-dots. Each consumer only has enough information to provide you with one dot. The first burden of marketing research is to ensure that each dot is mapped as accurately as possible. All the points above make the case that watching customers provides a more accurate map than just listening to them. When enough dots are in place, the second challenge is to then connect the dots. That is dependent on the insight and attentiveness of the marketer, not of the consumer. It is the failure to understand these responsibilities that leads to the common frustrations expressed in the introduction. A truly market-driven company not only understands how to get better market feedback, but also understands the important interpretive role they must play to transform that feedback into meaningful improvement.

References

1 Ergonomics: Harness The Powers of Human Factors in Your Business, J.E. Karlin and E.T. Klemmer, Ablex, 1989.

2 “Do Retail Brands Travel,” McKinsey Quarterly, 2002 Number One, Peter Child, Suzanne Heywood, Michael Kliger.

3 “The Framing of Decisions and the Psychology of Choice”, Tversky, A. & Kahneman, D., Science, 211, 1981.

4 “Could You Tell? Bud vs. Miller,” Consumer Reports, August 1996.

5 Don Norman, a leading authority on human-centered design, likes to note that signs instructing people not to do something usually indicate the need for a better design. The signs are only necessary because some natural desire or tendency is being thwarted. For an introduction to his views, see The Design of Everyday Things, Donald Norman, Basic Books, 2002.

6 The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Clayton M. Christensen, McGraw-Hill, 1997.

7 Usability Engineering, Jakob Nielsen, Morgan Kaufmann Publishers, 1994. Also by Nielsen, among numerous helpful essays on www.useit.com, “First Rule of Usability? Don’t Listen to Users,” AlertBox, August 5, 2001 and “Field Studies Done Right: Fast and Observational,” AlertBox, January 20, 2002.

8 “A Knife in the Back,” Jerome Groopman, New Yorker, April 8, 2002.