Best practices for online qualitative research

Editor’s note: Berni Stevens is CEO of TechFocus Research Inc., Exton, Pa. This is the first of a two-part series on best practices for online qualitative research. Part two will appear in the July issue.

You may have tried an online qualitative method and were disappointed or maybe you haven’t taken the leap to online yet at all. Or perhaps you’re already a convert and just want to find ways to improve on a good thing. Regardless of where you are now, this two-part series of articles will give you a guide to how to get the most out of it.

In total, we’ll give you 26 best practices across a number of critical areas that are central to conducting online qualitative research. In this first installment we will consider 16 ways to improve online moderating.

1. Provide an engaging introduction and clear instructions.

Your participants’ first impressions are important for setting the tone and expectations. Just as for in-person interviewing, you need to immediately establish rapport and make your participants feel welcome. Recognize, too, that even though they agreed to participate, your respondents may not be entirely comfortable with participating in an online interview and may be uncertain about what is going to be required of them.

Put everyone at ease by letting them know that spelling and grammar don’t matter - what matters is what they have to say. Provide clear, easy-to-read instructions in both their e-mail confirmation as well as in the introduction once they’ve logged in.

2. Warm up your respondents before diving into the topic at hand.

Your first few questions should further work to develop rapport and make participants comfortable. Start with asking them to introduce themselves, just as you would in an in-person group. Respond with similar information about yourself so long as it’s not germane to the primary discussion topics.

Give them an open-ended question that allows them to share broadly about the topic at hand, even if you won’t include it in the analysis. For example, if you are going to be evaluating advertising for a new digital camera, ask them: "Tell us about what kinds of pictures you’ve been taking lately and what you’ve been doing with them."

3. Use a chatty, informal style for your questions and include a variety of question formats.

Written discussion guides can seem cold, formal and sometimes repetitive. Go out of your way to make yourself seem human by using a style that’s similar to the way you talk. Don’t ask the same question over and over again, even if you’re trying to get at the same thing across a number of concepts, ads, features, etc. After all, this isn’t a quantitative survey, so controlling for variation in question wording isn’t really necessary. And sometimes, we need to ask a question in a variety of ways to get at what we’re looking for. If, for a series of concepts you find that a particular question style is eliciting a richer response, then you can probe using the variation that is most evocative for the respondent.

4. Ask oodles and oodles of probing questions.

This is exploratory qualitative, so ask as many questions of your respondents as you would in an in-person group. Asynchronous groups allow for greater consideration of responses and probing than face-to-face groups. Probing questions will show that the moderator is engaged and interested and will often inspire respondents to be likewise engaged.

5. Infuse a little humor.

If you’d use humor in an in-person group, then use it online, too, especially in probes and additional requests for information! Humor can help to establish empathy and defuse tough situations. Studies have also shown that when one uses or experiences positive humor, the whole brain is involved, not just one side, and that there’s more coordination between both sides. Just keep it good-natured and don’t make it too personal or at all vicious.

6. Encourage interaction with other participants.

If you’re opening up the discussion so that all participants can see others’ responses, then make sure all know that you expect them to read what’s been posted by others and react. As a moderator, you should point out particular responses and seek to gain agreement or dissension, just as you would in an in-person environment. You’re only going to get out of it what you put in.

7. Include visuals that are easy for participants to view.

Images can be included as response stimuli and they make the interview more interesting, too. Be careful, though, to make images and video easy for participants to view. Respondents need to be instructed on how to change the size of the whiteboard to view the entire image. For long text descriptions, make sure your participants know to scroll down if needed to read the entire description. Sometimes, it may be advantageous to have multimedia open in a separate browser window, particularly when respondents are to interact in some way with the stimuli.

8. Incorporate suitable projective techniques.

If you would use projective techniques for in-person qualitative interviews, why not use them online, too? Many, like free associations, visualizations, storytelling and personification, can easily be employed online. Even the ones that require physical sorting or drawing could be accomplished with a little creativity.

9. Deal with problem respondents offline.

Don’t deal with issues concerning a participant’s lack of participation, dominance, inappropriate responses or other troublesome behavior within the discussion format. Even if the interview format is set up so that participants can’t see others’ responses, your interviewee may forget that and be resentful that you chastised them. An offline, personal e-mail that is tactful and sensitive is your first line of action. If that doesn’t work, then you may just have to dismiss the participant.

10. Encourage expressions of emotion.

The perceived difficulty of capturing emotion in an unfiltered way is a primary concern for many when it comes to online qualitative. And, as discussed above, there may be some times when it’s better to utilize in-person interviewing when it’s critical to gauge visceral response. Emotion, though, will be expressed if respondents care about the topics being explored and they may be more willing to share their emotions because of their visual anonymity.

In an online environment, there are a number of ways that you can encourage and effectively capture emotion. First, tell your respondents that you want them to convey emotional reactions when they have them! They can do this through:

  • words, e.g., "This makes me very angry!";
  • emoticons, e.g., >:-O (provide a key for respondents’ easy reference);
  • CAPS, underlining and italicizing.

Also, your own questioning can go far in soliciting emotional response. For example, ask participants "How does this make you feel?" or "Tell me about the emotions this brings up in you."

11. String related questions into one.

Avoid choppy questions that will yield short, choppy responses. If you ask a series of related questions, participants will respond with fuller responses, mirroring the moderator’s style. Further, you will actually get more information from this approach as participants will typically respond first to the issues that are most important to them.

12. Don’t just ask "why."

I know, you think that understanding why is a major component of what qualitative is all about. But if you just ask why, especially in an online environment, you’re likely to be met with a short, superficial answer. If you ask "Why did you purchase a Sony camcorder instead of a Panasonic one" you may hear "Because it had better features." If you instead ask "Tell me some of the reasons that you purchased a Sony camcorder instead of a Panasonic?" you might get "Because it has a longer battery life, better zoom function and it costs about the same." Additionally, you risk putting a respondent on the defensive when you merely ask why because they feel like they are being challenged or attacked.

13. For multi-day asynchronous interviews, pace the questioning and leave room at the end for emergent issues.

Many have a tendency to front-load questioning. If you demand too much up front, you’re going to overwhelm your participants, and they may hold back on how much they say because you’re asking too much and you haven’t yet gained their trust and committed involvement. You should have already given your participants an idea about the amount of time they can expect to dedicate each day. Now you need to make good on your promise if you are going to keep people coming back.

Additionally, questions will arise during the first days’ interviews that you’d like to get additional insight on. If you’ve built a discussion guide that’s equally divided over the interview duration, then you may not have time to address the issues that emerge over the course of the first days’ input.

14. Keep clients engaged and involved.

It’s critical in any study to involve the ultimate users of the research throughout the process. This is especially true of qualitative, where the process is more fluid and observers can have a strong impact on the direction the interviewing takes should interesting or unexpected responses arise.

The good news is that it’s more convenient to observe online qualitative. Client teams don’t have to travel to be involved, so more should be able to participate. On the other hand, because clients haven’t isolated themselves from the office and all its interruptions and demands, it’s more likely they will get distracted and not log in. So, if you’re going to keep your client team engaged you need to make it easy for them to do so. Incorporate a practice login on a demo site to get everyone comfortable with the venue and how to post comments. Then, send frequent reminders and, if it’s an asynchronous group, send some of the more interesting quotes to pique their interest.

15. Create screen names that make it easy to identify specific segments in non-interactive groups.

It’s hard enough keeping track of all the information that’s being shared in large, asynchronous sessions, but add on top of that various segments which you want to analyze separately and the moderator’s and client observers’ heads will be spinning trying to keep up with it all. One helpful method is to assign screen names/handles that are coded to reflect their segment membership. For example, a current product user is Sarah (C) and a non-user is Andrew (N).

16. Choose a moderator who understands the unique challenges and nuances of online qualitative.

Many skills used in in-person interviewing easily translate to an online environment, like knowing how to ask questions and the right probes and how to keep everyone participating. However, the moderator you’ve used for years for in-person qual may not conduct online qualitative in a way that makes the most of the medium. Ask them what the differences are and if they can’t give you at least a handful of the tips in this article, then you should look elsewhere.

Complexity is greater

Moderating any kind of qualitative interview requires numerous learned and innate skills. The complexity is even greater when you move your interviews to an online environment. You can, though, effectively moderate online interviews with the right guidance. Take advantage of a few of the tips we’ve provided above and you’ll see a sharp improvement in the quality of your experience.

Look for the second part of this series, where we’ll look at best practices for recruiting, incentives and group composition, in the July issue of Quirk’s.