Editor’s note: Ron Sellers is president of Ellison Research, Phoenix, Ariz.
Flip through any research publication and there’s likely to be at least one article expounding the virtues of one methodology over another. Focus groups are passé because there’s too much peer pressure bias (or a dozen other reasons cited by the anti-focus-group crowd). Online studies are superior to telephone interviewing because of immediacy…or cost…or falling phone response rates…or lack of interviewer bias…or (fill in the blank). A few issues later will come the articles extolling the virtues of focus groups and telephone interviewing.
And let’s not even talk about mail surveys or mall intercepts.
Those who like to claim that a particular methodology is “the best” often have a vested interest in having more clients choose that methodology. I know the head of one small research company who decided he didn’t want to travel and moderate any more. Rather than simply telling clients that, he decided it was time to promote the benefits of telephone in-depth interviews over focus groups, and claim that focus groups were no longer a viable methodology.
Think about it: How many articles have you read claiming online focus groups are superior to traditional focus groups that weren’t written by someone from a company that offers online focus groups to clients?
The real truth is, there’s room for all these toys in the sandbox. One of the most overlooked keys to being a good researcher is not just using each tool properly, but choosing the proper tool.
Each methodological approach has fairly well-defined advantages and disadvantages. The trick is figuring out how each one of these applies to the project at hand and then selecting the right methodology.
Are in-depth interviews superior to focus groups or vice versa? Is telephone interviewing better than online surveys? Should you migrate your traditional focus groups to online focus groups? The real answer is that it depends on the project.
Let’s look at one simple example: in-depth interviews versus focus groups. If the project requires reaching a small population that is scattered across the U.S. or in rural locations, or there are serious concerns about privacy issues or group bias, then in-depth interviews are a very viable option for gathering qualitative information. If group interplay can add value to the discussion, or you need to show various materials for respondents and get their reaction, and/or your client would really benefit from watching the project unfold, then focus groups are a better methodology.
For that matter, there’s no rule that says you can’t conduct some focus groups and supplement them with telephone IDIs.
Take any two legitimate research methodologies and you can make that same type of comparison. Telephone versus online? A significant portion of the population still is not online, while everyone has a phone. Online not only has sampling issues, but also concerns about a self-selecting sample and response rates. Of course, telephone has falling response rates, unlisted numbers, blocked calls and cell phone-only households.
Online interviewing allows you to show complex graphics, play audio clips or even show streaming videos to people and immediately gauge their reaction. It’s faster and often cheaper than telephone work and has no interviewer bias. Using panels, low-incidence populations can often be reached far more efficiently because the panel operator already knows which panel members fit the study qualifications.
But telephone reaches a broader sample of the population than does online and it allows good interviewers to probe open-ended questions, which can’t be done easily online. In many cases, it’s also more secure than an online questionnaire, because you have more control over who is actually providing the data - it’s pretty hard for a 12-year-old to complete a survey meant for dad about beer advertising over the phone. A good interviewer can also screen out respondents who aren’t taking it seriously and keep a respondent on the phone who might have quit in the middle of an online survey.
Finally, phone surveys do not keep recycling the same panel members over and over. Consider an e-mail I recently received at work: “How do I become on the panel group (sic)? I currently participate in online studies at Go Zing, Survey Savvy, NPDOR, Harris Poll, SurveySpot, American Consumer panel, PSB, and many more groups. I’m looking for other groups that I have not been on yet. Thanks!!” You don’t run into that with a phone survey.
So which is the better option? Again, it depends on the project. A good researcher will analyze the options for each project, and decide which methodology has the fewest risks and downsides while providing the most advantages in that particular situation. Even out-of-favor methods such as mall intercepts and mail surveys definitely have their place.
When clients come calling (whether they are internal or external clients), researchers are fairly well-trained to ask questions that help the client determine whether qualitative or quantitative research is needed (or both). But too many times, once that determination is made, the researcher falls back into his or her comfortable habits. “Qualitative? Okay - that means focus groups.”
Well, maybe it does and maybe it doesn’t. Maybe it means in-depth interviews or ethnography. Or, even if it does mean focus groups, maybe it means some creative approaches to those groups - giving respondents a disposable camera and having them take pictures of what a brand means to them, or having them create collages that illustrate how they feel about a particular product. Maybe it’s focus groups with parents and children in the same room, or groups held on-site at a store location so the group can shop together and then retire to a back room to talk about their experience. Maybe it means doing something you as a researcher haven’t done before - building something from scratch instead of dusting off last year’s discussion guide and making a couple of quick revisions.
This can be particularly challenging when your client calls with an opening line of, “We need to do a phone survey.” Trying to convince the client there might be a better way runs the risk of making them feel they weren’t bright enough to come up with the best answer on their own. Often, what we resort to is the approach of, “Sure, we can cost that out for you. But I’d like to discuss an alternative with you that might be even more useful for this particular project. If you think it’s appropriate, we can give you a cost on that approach, as well.”
The point is, it’s time we stopped always reaching for the standard answers. It’s also time to put to rest criticism that certain methodologies are innately superior. One methodology is not better than another for every project. Online panels will not solve every quantitative need (nor will mail surveys, IVR or outbound phone surveys). Online focus groups are not going to replace traditional focus groups (nor should they be ignored as a possible approach).
Only when we stop worrying about what is better, and start worrying about what is most appropriate for this particular project, are we truly bringing to our clients all of the strengths that research can offer.