Editor’s note: Peter Thorwarth is president of BMA Mystery Shopping, Phoenixville, Pa.

When the client says…“Good news – the mystery shopping program has just been approved and now people here are very excited about seeing results as soon as possible. How soon can you have the shoppers in the stores? Can it start full-force on Monday?”

…what should you do? Jump ahead and get the mystery shopping in the field right away? Or strongly recommend a brief delay, in order to test and fine-tune the survey form, guidelines, scenarios and Internet online elements?

Firms like ours have faced this temptation time and again: a full-service or field-service client has a draft of a mystery shopping form that looks right to the people there and they want to start a large mystery shopping project right away. It would be easy to give in and throw it out in the field. But it’s really not in anyone’s best interest to do so. We know from hard-won experience that we shouldn’t mystery shop hundreds or thousands of locations without first testing the form.

One key reason to do a pretest is expressed well by Bill Green of Shoppers Critique International: “Mystery shopping providers rely in many cases on the information our clients divulge with respect to the nature of their business and method of operations. Many times, the client [unintentionally] does not provide details that are critical to us and unfortunately we may not find out until the program is underway.”

A related reason is that the mystery shopping experience may not flow as expected. As a result, some questions on the survey form may need “N/A (not applicable).” Looking at selling/retail situations, for example, we’ve learned that some clients assume that salespeople in their stores will at least follow the general training guidelines. The truth we discover, thanks to the pretest, is that some sales associates make almost no effort at all to assist customers, even those who seem very interested.

Associates like that make it impossible for the mystery shopper to answer some of the questions. Faced with that situation, individual mystery shoppers nationwide would have been stumped and calling our 800-number for instructions.

I’m sure the same is true in restaurants, hotels, banks, and other mystery shopping venues: testing and correcting the form in advance prevents headaches and delays.

How to conduct a pretest

A pretest is a very simple endeavor. In essence, it is just sending top-level shoppers to actual stores in order to act out the scenarios, follow the guidelines, and fill out the forms (paper forms and, where applicable, online forms). After that, debrief the pretest shoppers and review their results. Use those findings to improve (or at least fine-tune) the scenarios, guidelines, and form.

Simple though it may be, a pretest can make the difference between 1) a research project that is successful and 2) a haphazard comedy that frustrates everyone and has few actionable results. It requires a small effort and it pays significant dividends.

Pretest elements

First, consider the survey instrument, i.e., the mystery shopping form. Is there anything wrong with any of the following questions?

  • “Were all associates wearing nametags?”
  • “Was the sales associate friendly and helpful?”
  • “Did you see (manufacturer name) rebate forms?”
  • “Was the video display showing the demo tape?”
  • “How many tires were missing from the display wall?”

Yes, it turns out they were all flawed.

1) During the pretest, some sales associates were wearing nametags that were not meeting the client’s needs, because they were under aprons or sweaters and could barely be seen.

Solution: “Were all associates wearing nametags that were plainly visible?”

2) During the pretest some sales associates we encountered were friendly but not helpful.

Solution: We made it into two questions – “Was the sales associate friendly?” “Was the sales associate helpful?”

3) During the pretest, expired rebate forms were found in some stores, which would have given the client a “false positive.”

Solution: “Did you see (manufacturer name) rebate forms that were current/not expired?”

4) During the pretest some stores had no video display at all.

Solution: Again, we volunteered that two questions would better serve the client: “Was there an operating video display? If yes, was it showing the demo tape?”

5) During the pretest, it was noted that some stores also had tires on displays on columns and on freestanding units, in addition to the display wall. In some cases, those displays were missing tires.

Solution: “How many tires were missing from the display wall or other tire displays?”

The most common improvement, as noted above, is to take one question and break it into two separate questions. Every question on every form should be examined and tested to see if this would improve the accuracy of the data gathering.

Second, consider who should actually go to the stores to perform the pretest. We recommend using two or three very experienced mystery shoppers from diverse parts of the country. A New York shopper might spot something that won’t flow as expected in the New York stores, while a Mississippi shopper might spot something that doesn’t apply to small-town stores.

The pretest shoppers should only be given the training materials, guidelines, and survey form, without extra background information. They should be no more informed about this particular survey than the average shopper would be, using the materials provided. As part of the debriefing, the pretest shoppers should be encouraged to point out anything they feel is confusing or insufficiently specific. Preventing misunderstandings is an important key to a successful mystery shopping program.

Third, have the pretest shoppers use the actual systems that the rest of the shoppers will use to deliver the results data to the mystery shopping provider. If the shoppers will send the results via a Web site, for example, have the pretest shoppers use and critique that Web site.

Fourth, test the methods by which you will deliver the results to the client. This is not an issue if you will be delivering the data in Excel or on paper, but it matters a great deal if the client will be accessing results online.

Rob Schneider, president of Archon Development, a firm which provides online services for mystery shopping companies, says, “From an online technical standpoint, pretests give you tremendous advantages and resolve important questions.” Such as:

1. Is the end-client behind any firewalls or proxy servers that need to be addressed prior to launch?

2. Is the end-client using an ISP or operating system that isn’t fully supported by your online system and needs to be?

3. Are all end-user logins, triggers, and other capabilities fully functional prior to mass use?

4. Is all form scoring and report scoring perfect?

5. Has the end-client given all of the correct information relating to locations (i.e., zip codes, names of locations, management hierarchy, etc.)

“The biggest benefit and reason for conducting a pretest from a technical view is simple,” Schneider says. “If you first test a handful of users within a company, you will find that the majority of technical questions and problems can and will be addressed prior to thousands of users having those same questions and problems with a full-blown project launch.”

Accept the importance?

Will clients around the world accept the importance of a pretest? For everyone’s sake, we hope they do. Showing them this article might help them see the light. Bill Green points out the economic benefits: “It is more cost-effective to start the program right the first time than to make modifications to programming, guidelines, and forms after the fact, which certainly permits shopping companies to provide lower-cost programs.”

Non-U.S. clients will benefit just as much from taking the time to allow a pretest, says Per Bjorkman of Retail Services in Stockholm, Sweden. “We always include a pretest in our offerings/quotations, to avoid any misunderstanding and to improve the results of the mystery shopping for our clients. Most clients accept the idea of pretest when we explain why. We are currently mystery shopping the public transport system in Stockholm [all subway trains, other trains, and buses plus three stations]. The client agreed to do a pretest, after some convincing, and now the whole project is now working 99 percent perfectly.”