Best practices for online qualitative research

Editor’s note: Berni Stevens is CEO of TechFocus Research Inc., Exton , Pa. This is the second of a two-part series on best practices for online qualitative research. Part one appeared in the May issue.

In the first article of this series we presented 16 best practices for online qualitative moderating. Here we consider ways to best recruit and provide incentives for your online qualitative and how to construct the most effective groups for participation.

Best practices for recruiting

Recruiting for online is different than for in-person interviewing because there are no restricting geographical boundaries and those recruiting aren’t working off their local databases. Here are four tips for how to effectively manage your online qual recruiting:

1. Evaluate your source of recruits carefully.

For those instances in which you aren’t using a list of your own customers, you should choose your recruiting source carefully as there is quite a bit of variance in quality of sample sources. For in-person qualitative, focus group facilities have typically handled the recruiting, relying on their database of potential participants (fraught with problems, but that’s another story).

With online, where there are no geographical boundaries, recruiting organizations rely on purchased lists from firms like Survey Sampling International or online consumer or business panels like those from Greenfield Online or eRewards. Lists are fine, so long as they have been randomly generated from a sample frame that is representative of the population. These, though, aren’t often used because they are more expensive than sourcing the now-ubiquitous panels available to us for recruiting for both quantitative and qualitative. Besides being less expensive, panels also allow us to recruit more quickly and provide us with respondents who are comfortable with online research.

Panel recruiting and retention practices vary quite a bit and their practices have a major impact on respondent quality. Factors to evaluate include:

  • How new panelists are recruited

- Are panelists recruited through open recruiting on Web sites or by invitation only? By invitation-only allows for verification against available databases to ensure the panelist is who he says he is and also prevents the professional respondents from joining at will.

- Do they use a combination of online and offline methods for a more representative panel?

  • How panel members are managed and rewarded.

- Is profile information updated regularly?

- What incentives are they given to participate? Sweepstakes are inexpensive but typically result in professional gaming and low retention rates. Appropriate rewards for a respondent’s time will lead to high retention rates and cooperation rates.

- On average, how many panels are its members involved with?

  • How often do panelists participate in surveys? Look for low participation rates, under once per month.
  • How are panelists screened for each survey? Using a double-blind screening process will protect against respondents providing false screener answers in order to qualify for the study.

2. Confirm and re-confirm using both telephone and online methods.

Ensuring a high show-rate for your interviews requires gaining commitment from your recruits. After the initial contact and screen, whether by e-mail or phone, you need to confirm your respondents via telephone, listening for cues that would suggest they are hostile or iffy. Also tell them on the phone who subsequent e-mail will be from so they can keep an eye out for your e-mails that may get stuck in spam filters. Then, the day before the interview begins, send another e-mail confirmation with instructions on participation. If it’s a multi-day interview, an e-mail should be sent at the end of each day.

3. Over-recruit.

Even though your participants won’t have to travel and have the comfort and convenience of participating from wherever they want, it’s unlikely that everyone is going to show up, despite your best efforts. People forget or have something come up that’s a higher priority. Our general rule of thumb is over-recruit by 10-20 percent (higher if for a synchronous interview).

We all have a responsibility to maintain research industry goodwill. So, should the unlikely happen that more participants show than what you really need, pay the dismissed participants at least a third of the total incentive if for a multi-day interview and half if a single-day, short-duration interview.

4. Ensure participants are going to be active.

Just as with in-person qualitative, you want respondents who are going to have something to say. Other than using moderating techniques that will draw respondents out (see below), your screening techniques can also heighten the likelihood that you are going to have active participation. One criterion is the individual’s interest in the topic and in participating in the interview. Another is their ability to be express themselves clearly in writing. Both can be assessed with a few open-ended questions in a preliminary online recruit or in an e-mail before inviting them to participate.

Best practices for incentives

Many of the guidelines for online qualitative incentives are the same as for in-person. However, best practices for incentive payments for online qualitative are somewhat different because the burden is less for participants relative to the hassles and time involved with central-location interviews and there is added work and commitment when participating in multi-day asynchronous interviews.

5. Offer an incentive high enough to ensure a high participation rate.

If you want a good show rate and a sample that is not skewed to those who are less busy or of lower incomes, you are going to have to fairly compensate participants for their time. What is fair and enticing is going to vary by: how valuable someone’s time is; the amount of effort that will be required; the convenience of participation; and how relevant the topic is to them.

The higher-paid executive, for example, needs to be compensated at a higher rate than the stay-at-home mom or retiree. If you don’t offer an incentive that reflects participants’ opportunity cost, you risk getting a non-representative sample if income has anything to do with what you’re investigating.

One of the advantages of online is that individuals can participate from their office or home, thereby eliminating the hassle and time involved with travel to a central location. Even better is when you use an asynchronous method, because there’s the added flexibility of participating from not just a convenient location but a convenient time as well. So, overall, incentives offered for online qualitative can be lower than those for in-person interviews.

Finally, you should consider how personally or professionally relevant the topic is and whether the participant will potentially benefit from the findings. Product users are willing to participate for a lower incentive if they think the research will result in improvements that will benefit them. Under other circumstances, you may be able to offer participants a summary of the findings in lieu of a monetary incentive if they would find that to be of value to them.

6. Don’t pay more than is required to recruit quality respondents.

Offering more isn’t going to necessarily increase your participation rate or give you a better sample. In fact, if you offer a very high incentive, you’ll have a lot of people saying “yes” who aren’t interested in the topic and are just in it for the money (the “professional respondent”).

Also, you can offer different incentives to different individuals. For example, you can try recruiting a professional who meets a threshold level interest or involvement with the topic at $100 for a 90-minute interview and if they say no, ask them if it would be worth their while for $150, etc. Be careful, though, to avoid a situation where respondents become aware of varying payments. If they do become aware, be prepared with solid justification for the discrepancies.

7. Require full participation for incentive payment.

Those who agree to participate in multi-session/-day discussions must be clearly instructed that they will only be paid for participation in all sessions. A participant’s input won’t be of much value if they drop out after day two of a three-day interview. If you expect a high dropout rate you should over-recruit to compensate for this, or better yet, redesign the discussion guide so that it’s more engaging and enjoyable.

8. Reward promptly.

We need to preserve our respondent pool. To do so, we need to make the research process enjoyable and rewarding. Paying/delivering incentives immediately following completion of an interview strengthens the positive association. With in-person interviews, participants typically receive a check as they leave the facility. Panels often provide points for online quantitative survey participation. With online qualitative, though, we are sometimes tempted to move on to other more pressing items than processing incentives. Paying respondents promptly, however, will go a long way to reinforcing a positive experience.

Incentives are an important component of gaining respondent cooperation. Developing and implementing a balanced and fair system will get you the respondents you need for a successful outcome.

Best practices for group composition

Online group interviewing presents some unique possibilities relative to the in-person approach.

9. Take advantage of the ability to efficiently interview large numbers of respondents but don’t overdo it.

One of the key benefits of online interviewing is the ability to conduct either large-group interviews or very large numbers of in-depth interviews simultaneously. If you want group interaction in a synchronous session, stick to the max of eight to 10 that is conventional for in-person groups.

For asynchronous groups where respondents have the time to digest more information, you can safely increase this to 20-25. And for asynchronous groups in which respondents cannot see others’ responses and are not interacting at all with one another, you can have as many as your software will allow.

Our ability, though, to digest such large volumes of data is limited and you have to ask why you are doing a qualitative exploration among so many in the first place. The exception is when we are collecting information from a number of segments or conducting a hybrid quantitative/qualitative session.

10. Feel free to mix up non-interactive groups.

If you are allowing participants to interact, then it’s no different than for in-person groups: Keep them homogenous on key variables that influence needs, attitudes, behaviors, etc. However, there’s no need to do so if you are interviewing a number of individuals who don’t see each others’ responses. On the other hand, make sure the moderator and participants can keep track of key characteristics that influence questioning and interpretation.

The right group

Seasoned researchers know how vitally important it is to have the right group of participants for the research to produce valuable results. No matter how nice the environment (your online platform, in this case) and how brilliant the discussion guide, you’ll walk away with little of use if you don’t interview the right individuals and groups. Solid recruiting practices, including the incentives offered and paid, and assembling the right group of people together, will contribute to a successful online qualitative experience.