Mystery shopping reports: Are quality and timeliness competing priorities?

Editor’s note: Elaine Buxton is executive vice president of Confero, a Cary, N.C., mystery shopping firm.

Clients of mystery shopping companies are increasingly demanding speedy reports. Without question, the widespread availability of the Internet, e-mail and interactive voice response (IVR) have forever changed mystery shopping reporting. But is faster necessarily better? Are sacrifices in quality being made for the sake of speed? Are clients even aware of the sacrifices being made?

Quality mystery shopping programs result from appropriate design and implementation of each program component. Assume that a well-designed program would include clearly defined objectives, a mystery shopper evaluation form addressing measurable criteria or behaviors over which employees have control, appropriate shoppers and instruction, an appropriate visit schedule and timely reporting. The final component - reporting - is the focus here. Reporting includes communication from shopper to shopping company, from shopping company to client and from client to the field.

If your shopping program’s goal is to provide measurement and incentives for specific front-line employee behaviors, then employees must be able to trust the validity and quality of the reports. Companies that use mystery shopping services are demanding -- they want quality reports in a timely manner. Sometimes, the quality priority competes with the timeliness priority. Which one should win? Here are some issues to consider as you weigh each of them, so that the method of reporting does not interfere with the objectives of your shopping program.

• Report validation. Each shopper visit and report must be validated. Most companies validate using a cash register receipt showing the location shopped, the date, time, items purchased, etc. A reputable shopping company will consider a shopper report to be fiction until it is validated. Having shoppers report to the shopping company via Internet or IVR speeds up the reporting process, but reports should not be 100 percent accepted until validated. Internet and IVR reporting raise the question "Could shoppers sit in their living rooms and make up their answers?" Waiting for a mailed or faxed receipt is a small price to pay for preserving the quality and integrity of the shopping program. A receipt can prove that the shopper visited at the right time of day, the right day of the week, purchased precisely what was required and visited the correct location.
• Program value. If the objective of your program is to encourage employees to serve customers well at all times, then how instantly should you report to the store level? For example, if all locations are to be shopped monthly and reports are due precisely at month end, then the shopping company must schedule visits early in the month in order to provide valid on-time reports. Employees soon learn that shops take place at the beginning of the month and tend to relax for the remainder.

An example: At a retail store recently, a customer asked an employee for assistance in finding a certain item. The employee asked "Is today the 10th or the 11th?" When the customer responded that it was the 11th, the employee said, "OK, then it’s on aisle nine." The customer asked, "Why did you ask me the date?" and the employee replied, "Because today is the 11th of the month and we always get mystery shopped by the 10th. If you’re the shopper, I have to walk you over to the item you’re looking for so I’ll get all the points on the shop report. Our store manager is a real stickler about that. But today’s the 11th, so I can just tell you where to find the item."

Obviously, this employee’s perception of the mystery shopping program is not what the company intended and the company is not maximizing program value by reporting so quickly.

• Program quality. Validation is just one part of the screening and quality review of mystery shopping reports. Some companies provide screening services only, simply making sure all questions are answered and commentary, if required, is complete. Some companies provide full quality review services, reviewing and reading each individual report to check for consistency of commentary with scores provided. A screened report can be delivered to the client faster, since the shopping company need only check that all questions are answered. A quality-reviewed report requires added delivery time, since the full review itself takes more time.

Whichever method is selected, the shopping company must be expected to reject reports that don’t meet the criteria required and send another shopper to visit the location. That process takes time. Expect a screened report to be rejected if answers are missing; expect a quality-reviewed report to be rejected if answers are missing, inconsistent with scores or if comments seem biased or unfair. Full quality review will result in more rejections from the shopping company itself, before the report hits the field. Screening will result in more requests to re-shop after the reports hit the field. Reports may seem slower if they are quality reviewed by the shopping company, but that may not be the case, since requests for re-shops from the field can ultimately slow up the end-of-period tabulations. Quality will take time -- whether on the front end or back end.

• Speed to the field. It is not speed of reporting to the client corporate office but speed of reporting to the field which will most likely change front-line behavior in the short term. Working with a reputable mystery shopping company allows a client the comfort to permit reports to be sent directly to the field, rather than to a central corporate office for disbursement. This level of comfort requires validation plus, at minimum, a screening process. Otherwise, employees will spend their time refuting the reports rather than acting on the information they receive. Ask yourself if you want corporate management spending time answering questions such as "Suzie works at the location down the street, but the shopper report says she works here. Did the shopper visit the wrong place? Doesn’t somebody look after this?" A fully reviewed shopper report results in fewer questioned reports than a simply screened one. If you plan to speed reports to the field, motivation can be maintained by more random shopping periods, so employees will not know exactly when to expect the next shopper visit.

• Cost considerations. For companies requiring very specific shopper scenarios, speedier reporting may actually cost more. Why? The specific shopper scenario will likely enable employees to spot the shopper based on the shopper’s report. So, instead of rotating the shopper to that location again in a few months, the shopping company must send a new shopper each and every time - a costly move. Employees at quick-service restaurants serve hundreds of customers per day and are not likely to recall the circumstances of individual customers over the previous day or two. By contrast, employees selling big-ticket items such as furniture, appliances, computers or fitness equipment may interact with only a few customers each day, so they may more readily recall the shopper’s visit based on the scenario, the day of the visit or the time of day. Specifics increase the probability of shopper recognition. The less specific the shopper scenario, the faster the reporting can be without identifying the shopper.

The speed afforded by the Internet, e-mail and IVR should not be ignored. Even if there is an initial rate of 8 percent rejection from failed validity, the ability to capture data now, pending later validation, has its advantages. For example, if 1,000 reports are conducted today and reported via Internet by tomorrow, then 1,000 reports can be at least screened tomorrow and the following day. If it takes two days for mail to arrive (i.e., validation via receipt or otherwise) then 92 percent of the reports, on average, could be released to the client within 48 hours of the shop date.

Faster reporting opportunities afforded by Internet, e-mail and IVR may also reduce costs, since they are usually less costly for shopping companies to set up and administer. Bypassing postage, printing, and telecommunications costs significantly reduces the cost of conducting shopping programs, putting them within the reach of a wider range of corporate budgets.

An appropriate balance between timeliness and quality results from a series of well-considered trade-offs. If the shopping company will be conducting complete quality reviews of the reports, then sending reports straight to the field is an appropriate choice, leaving the corporate office to look at trends shown in tabulations at the end of each reporting period. If your corporate office plans to designate a staff member to perform the quality review, then consider an up-to-date benchmarking program to take advantage of speed and accommodate quality concerns. For example, a monthly program might produce a weekly communication to stores stating last month’s aggregate scores along with aggregate month-to-date scores to keep motivation in the field high, since employees will not know where their own scores fall until month end. The process of sharing trend analysis among stores creates opportunities for creative and innovative incentive programs. Stores can compete against their own previous scores, against the aggregate for the month or year-to-date or against shops conducted at competitor sites, all while the corporate office is receiving extremely timely, validated electronic data about specific visits. The corporate office may then disseminate specific shop performance data after the end of the shopping period.

Timeliness or quality?

Experienced mystery shopping companies have faced the quality-versus-timeliness issue with a variety of clients and can tailor programs to suit your unique requirements. In fact, a shopping company with integrity will tell you up front that this issue may arise from time to time. Well-defined objectives for the program will guide both you and the shopping company in determining the best approach.

It all comes down to this: if faced with a choice, which would your shopping company choose - timeliness or quality? If your company has a monthly shopping program and the last report for the month arrives at the shopping company and its validity is questioned, would you expect the company to send it on to you and hope it gets by? Or would you expect the company to contact you, state the problem and explain that the final tabulation report would be delayed by a day?

Marketing research people often quote the old adage "garbage in, garbage out." All of us involved in the industry must be careful not to give undue credence to reports simply because they are fast or electronically reported. The source of a shopping report should be a well-designed program. Clear objectives, a mystery shopper evaluation form addressing measurable criteria, appropriate shoppers and instruction, an appropriate visit schedule and reasonable timeliness are more important to an overall shopping program than the method of transmitting its results. Perhaps an updated version of the adage should read "fast garbage in, fast garbage out."