Editor's note: Joanne Ulnick is a partner in Ducker Research Co., Bloomfield Hills, Mich.

The first "don't" to keep in mind is that it is not customer satisfaction for which companies should strive, it is market satisfaction. We believe that the term "customer satisfaction" is a misnomer, as any such project should include non-customers and former customers. In addition, your company's performance should be measured vis-a-vis your competitors'. The essence of a satisfaction effort should be to assess market perceptions of your company and to determine ways in which your company can improve the products and services that it offers to all market segments.

Based on our experience involving market satisfaction studies, we offer the following guidelines. (The following list of do's and don'ts has been developed over the last 10 years and is based on actual trial-and-error experience.)

Do's:

1. Use an unbiased sample. When designing the sample, it is essential that once potential respondent lists are developed (i.e., customers, non-customers, former customers) a random selection process is used. This will ensure that the results are not biased by any particular group of respondents. For example, if top-tier customers are targeted for interviewing, the results surely would be biased by the level of service often provided to such customers.

2. Use of the voice of the customer. It is essential that the evaluation criteria against which supplier's performance will be measured are based on the terminology used in the marketplace. "Do not speak research language," says Mark Daniels, market research analyst with Motorola's Paging Products Group, "speak the customer's language." The criteria should reflect all the elements that are part of the purchasing process and reflect the thought process of the respondents. Internal company terminology should not be used during the interviewing process.

3. Pre-test all interview outlines. Part of developing the evaluation criteria is the pre-testing phase. Such pre-tests allow the researchers to modify and finalize interview outlines in order to incorporate the voice of the customer. The results of pre-test interviews include:

  • Proper identification of evaluation criteria;
  • Definition of criteria using the words of the customer;
  • Validation of research methodology;
  • Test of interview outline flow and questioning technique

4. Define performance targets. To develop and implement improvement strategies that address the concerns of the marketplace, specific quality targets for key evaluation factors must be sought. All respondents should be queried about their definitions of ideal and acceptable levels of performance. This information will help develop actionable steps for improvement.

5. Use experienced researchers. To obtain specific and accurate performance targets, use experienced researchers. By this we mean that researchers should be experienced in both market satisfaction and in the industry under study. These researchers are able to probe in depth as well as accurately and quickly assess respondent input (and probe further, if necessary). "It's critical to use a firm that knows your business or industry," says Hughes Petteway, project manager of Allied-Signal Aerospace Co.'s Market Analysis Group.

Gretchen Ackerman, marketing manager with Petrolite Corp., agrees. "Using interviewers who are trained to pay attention to and record important comments made by respondents, even when they are outside the scope of a structured questionnaire format, allowed us to obtain greater value from our quantitative measurement of customer satisfaction."

6. Ask for improvement suggestions. Whether the project is being conducted internally or by an outside firm, we suggest that specific questions about areas of needed improvement be included. These questions should address product as well as service improvement areas. In addition, seek suggested priority of improvement strategies. Regardless of whether the name of the study sponsor is revealed, these questions can be incorporated into the latter part of an interview outline.

7. Include market-driven questions. Questions on the respondent's likelihood to continue purchases with a supplier or even recommend current or past suppliers to colleagues are essential. These types of market-driven questions lend insight into the overall satisfaction of respondents.

8. Include quantitative and qualitative information. Quantitative data is critical to the overall satisfaction effort. Qualitative information is equally important. Such information lends insight into the quantitative measures, bringing dynamics and causal data to the analysis.

Qualitative information typically is defined as the reasoning behind respondent perceptions. It gives you the "why" and "why not," the "how" and "please explain." Of course, qualitative data represents a challenge for analysis as respondents' thoughts need to be captured and reported in a concise manner without losing the depth of the information. "We have learned the value of paying attention to verbatim comments made by respondents while they are rating evaluation criteria on customer satisfaction surveys," says Petrolite Corp.'s Ackerman. "Getting this information adds dimension to and helps validate the respondent's interpretation of the survey's evaluation criteria. This was extremely useful in giving us a clearer picture of the customer's mindset when they think about our company's offerings in comparison to the competition's."

9. Use techniques that emphasize improvement. When analyzing data, the most useful multivariate techniques are those that assist in identifying and prioritizing improvement requirements. Simple scorecard techniques are just that - simple. They provide minimal dynamics and strategic input. Rather, techniques such as the reward-penalty analysis depict where the most impact of an improvement strategy on overall satisfaction will be yielded. For example, in the diagram shown below, Company A will increase its overall satisfaction rating from a 3.90 to a 4.45 if all its efforts are placed in the area of delivery. In other words scoring a 5 on delivery, while all other issues remain constant, (e.g., the current performance) would improve the overall grand mean. Conversely, the penalty numbers reflect the effect if Company A were to fail (receive a 1 rating) in the specific criteria.

This type of technique not only shows the priority of improvement strategies but also the impact of such strategies on overall performance. And many companies concur that improved performance results in increased market share and/or revenues.

10. Market satisfaction is part of a continuous improvement strategy. Although an essential part of the process, market satisfaction is only one part of a continually evolving improvement process. The information should be used to augment and add dimension to an on going information gathering effort. Market satisfaction is one element in a multi-faceted process.

Don'ts:

1. Don't assume anything about your customer. "Don't assume anything about your customers; let them tell you their perceptions," says Motorola's Mark Daniels. The ability to objectively evaluate research findings and set aside internal paradigms is paramount and directly impacts the ability to develop and implement effective improvement strategies.

2. Don't rely on mean scores. When evaluating performance ratings, companies often consider mean scores as reflective of overall performance. Although a reliable technique, it provides minimal dimension to the analysis. A simple distribution of response has more impact and lends more insight to the analysis. Other, more complex techniques such as discriminant analysis and cluster analysis will add significant dimension to the effort.

3. Don't rely on stated importance. The debate between stated importance and derived importance continues. Although each technique has merit, we suggest that if stated importance is preferred that a derived importance also be conducted. If the results are consistent then you can feel comfortable with the fact that the stated importance is reflective of the true buying process.

4. Don't fatigue the respondents. A classic mistake that researchers often make in the field is to ask every question that they ever wanted to know. In addition, lists of evaluation criteria often exceed 30 factors. This type of interviewing surely fatigues the respondents and such fatigue certainly makes the data suspect.

A rule of thumb in market research is to maintain specific time frames for interviewing: 30 minutes for telephone interviewing and 50 minutes for personal interviewing. Evaluation criteria should not exceed 26.

To develop timely and concise interview outlines, role playing and pre-testing are helpful. When developing a list of criteria, a factor analysis is beneficial. These approaches will enable researchers to develop effective and reliable interview outlines to be used for the full-scale effort.

5. Don't do research for research's sake. The most critical guideline to keep in mind is that a satisfaction effort is only as effective as the implementation strategy proceeding it. Throughout any such effort, researchers should be asking, "How can we use this information? How can we make a change?" If a specific set of questions does not result in useful information, don't include it.