Improve the customer experience

Editor’s note: D. Randall Brandt is vice president, customer experience and loyalty, at Maritz Research, St. Louis.

Mystery shopping is one of many techniques used to develop an understanding of the customer experience. People who are either real customers or who fit the desired customer profile go through actual product or service encounters. Afterwards, these shoppers systematically record the incidence of specific events or employee behaviors and often make additional measurements or observations regarding the product/service experience as well.

Mystery shopping can provide a variety of benefits and insights, including:

  • Enabling an organization to monitor compliance with product/service delivery standards and specifications.
  • Enabling marketers to examine the gap between promises made through advertising/sales promotion and actual service delivery.
  • Helping monitor the impact of training and performance improvement initiatives on compliance with or conformance to product/service delivery specifications.
  • Identifying differences in the customer experience across different day parts, locations, product/service types and other potential sources of variation in product/service quality.

Mystery shopping also has some important limitations:

  • Limited projectability - traditionally, while mystery shoppers are viewing product/service experiences from the customer’s side of the table, shoppers may not be genuine customers. (Increasingly, due to the sophistication of client customer management systems and the scenarios that require testing, real customers need to be recruited to conduct the evaluations. Such an approach adds credibility to any client program and is increasingly essential within certain market sectors [e.g., financial services].)
  • The criteria used by mystery shoppers to monitor the quality of product/service encounters will only be useful in effective management of customer experiences to the degree that these criteria are based upon actual customer expectations, needs and requirements.
  • Continuous efforts to keep criteria consistent with evolving customer needs and requirements must be undertaken.

Understanding the strengths and weaknesses of alternative methods of capturing the voice of the customer - and how they impact an organization’s ability to address alternative managerial objectives and applications - is a key to building a comprehensive and effective customer experience management system. If mystery shopping is to be a part of that system, there are several methodological considerations that must be addressed. In particular, the following issues should be considered carefully in designing and executing a mystery shopping program:

1. Selection of the attributes and evaluative criteria used by the mystery shopper.

Ensure that the attributes/criteria used by mystery shoppers reflect things that are important to customers. What do customers want or expect from the product/service experience? What does the customer see, experience, scrutinize and evaluate? Answers to these questions let the organization act to actually improve customer satisfaction and loyalty.

The relative importance of the attributes also should be taken into account. This will ensure that any weighting scheme used to compute mystery shopping scores accurately reflects the differential impact of each attribute, and gives the greatest weight to the attributes that are most influential in determining overall customer satisfaction and loyalty.

Measures and criteria used by mystery shoppers should be aligned with those used in surveys, comment cards and other methods of gauging the customer experience, so that convergence of findings obtained from these different data sources may be determined. In some instances, this means making an effort to match mystery shopping measures/criteria with those used in other methods. This makes it possible to determine if results of mystery shopping point to the same action items indicated by results obtained from surveys, inbound customer communications and other data sources. Such convergence, when established, tends to validate conclusions regarding priorities for effectively managing and improving customer experiences.

In some cases mystery shopping measures and criteria cannot or should not be matched to those used in other methods. However, an effort should at least be made to ensure that measures may be meaningfully linked based upon relevance or interdependence. For example, in industries such as banking, retail and hospitality, “waiting time” is an element that commonly appears in both customer surveys and mystery shopping instruments. The key is to ensure that results obtained from each of these data sources may be linked. If a mystery shopper gauges actual wait time, while surveys ask customers to report how long they perceived to have waited in line, data analysis should focus on how customer perceptions vary with increases or decreases in actual wait time.

Alternatively, it can be equally insightful to also ask both mystery shoppers and real customers if their wait time was “satisfactory.” Such information makes it possible to calibrate shopper and customer perspectives and also enables the organization to set operational performance targets for wait time that are most likely to translate into positive customer experiences.

2. Adequate sampling or coverage of events and/or sites via mystery shopping.

As is the case with customer and market surveys, the quantity and frequency of observations directly impacts an organization’s ability to draw projectable conclusions from mystery shopping results. Results are more likely to be representative if a bank branch or customer service center is shopped on multiple occasions within a relatively tight time frame than if that branch/center is shopped once within a relatively wide and arbitrary time frame. This is another way of saying that the results of mystery shopping are subject to the effects of both sampling and non-sampling error. Therefore, take steps to ensure adequate sampling or coverage of the events, channels, and/or sites about which conclusions will be drawn based on mystery shopping.

Some specific steps to minimize sampling error and/or bias in mystery shopping results include:

  • Carefully defining the entity or entities for which statistically accurate conclusions must be formulated and developing sample size requirements at the entity level. This becomes especially critical when results are going to be used in a manner that demands a high degree of statistical precision (e.g., if store managers’ compensation is to be based at least partially on results of mystery shopping).
  • Using results of mystery shopping as one of several performance indicators regarding the customer experience. When results of mystery shopping, customer surveys and customer comment cards all point to the same areas for improvement, managers are less likely to question the accuracy or validity of any single data source/method.
  • Matching the timing, location and/or frequency of mystery shopping to actual customer behaviors, channel usage and related patterns. To illustrate, in the case of quick-service restaurants, if 60 percent of business is conducted at the drive-through window while 40 percent comes from walk-in customers, allocate the total number of mystery shops to be conducted in a manner that mirrors these proportions.

Execution of the preceding steps probably won’t eliminate sampling error and/or bias altogether, but it will go a long way toward minimizing the impact of such error/bias.

3. Reliability and consistency of the mystery shopping criteria and procedures.

The mystery shopping method must be reliable. This means: (a) providing shoppers with clear and explicit guidelines for using mystery shopping criteria; (b) ensuring that mystery shopping criteria and procedures are applied consistently by each shopper from one site/event to the next; and (c) ensuring that the criteria and procedures are used similarly by different shoppers.

There are well-established methods for determining the reliability of mystery shopping procedures, and these should be applied in a pilot test of the shopping method before its full-scale implementation. For example, suppose that shoppers are asked to observe whether a store has floors that are “clean and free of debris.” This item should be supported by clear definitions and descriptions (including photos or illustrations, if available), along with guidelines that explain any allowances and thresholds. Similar support should be provided for all measures and criteria to ensure that shoppers are as objective and consistent as possible across sites and events.

It is also critical to establish consistency among shoppers. This is achieved by having multiple shoppers evaluate a common set of events or transactions to establish correlations and levels of agreement. For example, in the case of a call center, multiple shoppers should listen to and evaluate a common set of recorded calls. Results may then be compared and, if necessary, efforts undertaken to maximize consistency and agreement among shoppers. This process, sometimes referred to as calibration, is another way to ensure the reliability and consistency of mystery shopping procedures.

All of the steps above promote consistency and objectivity in mystery shopping procedures. A cautionary note, however, is in order: Emphasizing evaluative criteria and/or aspects of product/service delivery that are readily defined - or that reflect the organization’s priorities - facilitates objectivity and consistency. However, too much emphasis on these elements may preclude shoppers from recording unique or incidental things that might furnish important insights and opportunities for product/service improvement. Therefore, it is critical to leave room in the shoppers’ experiential checklist for these additional elements and insights. The result will be a mystery shopping approach that is disciplined and reliable but also opportunistic.

4. Keeping shoppers and customers connected.

Mystery shopping enables an organization to evaluate the degree to which product/service delivery complies with performance targets, standards, specifications, policies and desired behaviors that are intended to ensure a positive customer experience. Presumably, such targets, standards, specifications and policies are established on the basis of a clear understanding of customer needs, expectations and requirements. However these are moving targets that continually change and evolve. Therefore, it is critical to check periodically the alignment of performance standards, etc., with changing customer expectations, and to make adjustments in mystery shopping criteria and procedures accordingly. This will increase the odds that shoppers and customers stay connected.

Comprehensive and panoramic

There is no perfect method of learning all that is needed to manage customer experiences effectively. Thus, it is critical to employ and integrate multiple methods of measurement and observation to develop a comprehensive and panoramic perspective.

While it does not always capture the voice of the customer directly, mystery shopping can be designed and implemented in ways that make it a very good method of customer-driven quality assurance. Generally speaking, mystery shopping is most useful for gaining insights that enable an organization to address two key questions about the customer experience:

  • Do we actually deliver products and services in ways that are consistent with our performance goals, standards, specifications and/or policies?
  • Do we actually deliver products and services in ways that are consistent with promises and claims made in advertisements, promotions and other customer and market communications?

By addressing the methodological considerations discussed above, an organization can go a long way toward ensuring that mystery shopping fulfills this purpose.  | Q