Adding a little advance work
Editor’s note: Alan S. Kornheiser is research director and principal of sophisticated market research, North Salem, N.Y.
In conducting qualitative research, especially when doing a sizable number of focus groups, there is always the temptation to treat some of the data as quantitative. Thus, a report may contain such phrases as “a strong majority of respondents preferred the new to the old formulation,” even if the researcher is concerned that such a “majority” has little meaning in the real world. Client expectations, and the tendency to want to substitute focus groups for projectable surveys, make this problem worse.
In fact, however, the total sample available from a set of eight - or even, pushing the point slightly, a set of four or six groups or a set of 50 in-depth interviews - is often sufficiently large to provide some measure of reliability. A sample of almost 100 people is usually large enough, for example, to ensure that the new version is not seriously worse than the old version and indeed - because sensitivity in a survey only increases as the square root of the sample size - may represent a perfectly reasonable number of interviews if one only needs a general sense of the market’s reactions.
The problem is not the sample size; the problem is the sample selection and the interviewing process. Those attending focus groups are rarely typical of the market as a whole, and the discussion process is rarely typical of real-life purchasing behavior. Thus, ending a focus group by asking “Who prefers A and who prefers B?” is unlikely to yield much except a dangerous self-confidence. Still, because many clients will insist on asking such a question and because it seems foolish to avoid asking the question if there is some way to make the answer useful, we have developed a methodology that both improves on the quality of what we normally want to get from focus groups and provides a better “quantitative” result.
There are three caveats:
1. This procedure is not cheap; plan on increasing your cost-per-respondent by at least $25 and perhaps as much as $50, depending on how you value your time.
2. These results are only meaningful if your respondents represent your target population. With simple consumer goods, focus group recruiting often does not do this as well as one might wish.
3. Accordingly, we suggest using this procedure primarily for complex financial and business-oriented goods and services. When you wish only to evaluate a new flavor of toothpaste or the like, it is probably better to bite the bullet and pay for a fast and inexpensive mall intercept or omnibus survey.
Has an impact
We typically utilize a pre/post testing formulation. When they are recruited, respondents are informed that they will be sent a questionnaire to be filled out and returned before the group begins. We send this via Federal Express, and we enclose a prepaid Federal Express envelope for the return. This has an impact all out of proportion to its modest cost: our respondents become aware that we are serious and take themselves and their task seriously. We use this questionnaire to obtain all the useful but time-consuming baseline data that so clutter up the first 20 minutes of most qualitative research and to prepare the respondent to think seriously about the topic at hand.
We have attempted in the past to save time and money by using either e-mail or faxes to distribute the questionnaires. Neither works well. Fax distribution is surprisingly unreliable for such studies; additionally, it limits the size of the questionnaire and inhibits the questionnaire’s return. E-mail should work but doesn’t; mail programs are sufficiently variable that formats are destroyed and what begins as an elegant questionnaire can often become misaligned gibberish. We have abandoned attempts to use either one. Both also lack the sheer impact of a FedEx envelope and thus earn lower response rates.
(Another approach is setting up an interactive Web site, with built-in questionnaires and presentations, and sending out e-mails with instructions for logging onto the site. Technology for doing this is just now becoming inexpensive enough and easy to use to make this viable and we will be testing such tools in our next projects. Properly done, the cost should be competitive with express service delivery and the increased time to set up the site will be compensated for by the reduced time needed to analyze the results.)
One recent study sent via FedEx dealt with a brokerage product designed for a company’s best clients. In the questionnaire we asked these clients to rate their satisfaction with the company’s present product and to provide, if possible, examples of other companies who had better products.
We have found that properly motivated respondents will easily fill out a four-page (or even a six-page) questionnaire. (We have also found out that many will accidentally skip over the middle pages of a questionnaire, so we provide multiple reminders that there are inside pages.) We keep these questions as simple and direct as possible and minimize open-ended responses...although we always include a final open-ended question to allow respondents to feel comfortable that their opinions are being heard. A particularly effective procedure involves letting respondents circle as many words as they think apply to a product, ending up with a multivariate response grid of rich complexity.
As in all self-administered surveys, it is important to avoid ambiguity or excessive subtlety. Many respondents proved unable to distinguish between a product offering being “simple” and being “simplistic.” None of this is news to those who regularly use such questionnaires but may be overlooked by qualitative researchers used to being able to discuss answers with their respondents.
The answers must be submitted a few days before the actual group. Yes, I know...sometimes they’re not even recruited until the day of the group, but you want these questionnaires back in sufficient time to integrate what they tell you into your sessions. Typically, a questionnaire will contain questions about reasons why something can’t be done, barriers to acceptance and related problems. By having these in place before the group starts, one can focus on how to overcome barriers rather than spending time learning what the barriers are.
In many ways, this is the most useful aspect of this technique. Rather than spending time determining reasons for rejecting a new concept - which, to be frank, are usually obvious before the group has begun - the group can be presented with a list of objections drawn from their own comments. Because these objections are indeed from the group, they are accepted without difficulty and time is more productively spent discussing ways to overcome such objections. In addition, often unexpected objections may surface in the questionnaires, and time can be allocated to exploring these objections in detail before examining how best to overcome them.
Thus, one goes into the focus groups knowing how the respondents now feel, what issues face you, and what must be done to overcome internal and exterior barriers to acceptance. Such specifics provide excellent places for effective group discussion, and a session will typically end with the group having both a good understanding of what is being evaluated and a cogent set of arguments for that good or service.
The “post” work
At the end of a discussion session, respondents are sent off with a final questionnaire and an addressed FedEx return envelope. Again, response rate is very high, and the few who fail to return the envelope will almost always do so if called and reminded by telephone.
The “post” questions fall into two categories: we re-ask several of the key questions from the first questionnaire, to determine if exposure to whatever has been shown in the group has effected a change in perception. And we ask key purchase and interest questions: Would you buy this? For how much? Is there a price at which you would buy it and a price at which you wouldn’t? What are they? These answers, prepared after thought and at the respondents’ leisure, uninfluenced by others in the group, provide the best measure possible of true purchase interest...always with the caveat that the respondents actually be representative of the target population as a whole.
The final result, in addition to improved efficacy in the groups themselves, is a solid measure of changes that have occurred because of exposure to new products or ideas and a not-unprojectable measure of new product acceptance.
It is worth emphasizing that this technique does more than just provide a reasonably projectable form of mini-survey; it also greatly enriches the focus group itself. It does this by concentrating the respondents’ minds on the subject at hand, eliminating much extraneous discussion at the start of each group, and - most of all - allowing the moderator to tailor each group to its unique wants and needs.