FAQs for first-time clients of online qualitative

How to make sure ‘new’ is not ‘too new’

Editor’s note: Judy Langer is president of New York-based Langer Qualitative.

Online methods have become pretty standard today in survey and quantitative market research. In qualitative research, too, more studies are being done totally or partially online, yet many clients remain hesitant to try this approach. (And, for that matter, so do a number of qualitative researchers/moderators.)

As someone who believes that the research methodology should fit the purpose, I use a variety of interview methods - face-to-face, phone and online - depending on the study situation and goal. In the online area, I’m a “regular user,” as we market researchers would say, of bulletin boards; some other online methods I either don’t care for (a “rejector”) or don’t have experience with yet (“non-user”). The bulletin boards offer a rich way of interacting with and getting to know participants, sometimes making it possible to do research that we couldn’t have done in other ways. No, I don’t think online is superior overall to other forms of qualitative, despite claims some make, but I do think it’s worthwhile using when appropriate.

Proposing online methods to non-user clients is sometimes the classic double bind for researchers: clients are impressed by and may even ask for “new” methods, yet they reject online qualitative because it’s “too new” for them.

Here are some top FAQs for clients hesitant to use online qualitative and my answers.

What is online qualitative anyway?

The “online” part refers to the participant interviews/engagement, which typically use Internet access through specially-designed software. Recruiting may be at least partially online too, but regular telephone screening is often used as well. A number of companies now offer software platforms that enable moderators and participants to enter a virtual room, accessed through the Internet, where they can engage and clients can observe. This “facility” is both wide open because people can participate from anywhere and restricted because access is password-protected for each study.

Some basics about on how online qualitative works, whatever the particular method:

Anywhere. Physical boundaries disappear with the Internet, opening up the research to anyone with Internet access. Rather than spending time with a few consumers in one suburb, for example, we can interact with people all around the country (or even in different countries). Online qualitative can be a good alternative when potential respondents are far-flung or in remote areas, making in-person interviews impractical or impossible. In one study, we talked about air quality with a mix of urban, suburban and rural residents.

Forms of interaction. Among the different types are the question-and-answer format we’re familiar with from other market research; closed-ended questions; participant-created visuals, such as their photographs, videos, collages; self-directed exercises (written or video diaries, blogs); Webcam video of participants and their environment (what some call “ethnography lite”). These activities can be done in face-to-face research too but not as easily or on spur of the moment.

Group or private discussion. Researchers can choose how much of the online discussion participants can see. A question or comment relevant to a subgroup, for example, can be directed just to them. The moderator can send a note publicly to all participants, send it only to certain individuals, or can write privately one-on-one. A particularly valuable feature is the “masked” question: participants see one another’s responses (e.g., to a new product concept) only after submitting their comments, eliminating any issue of group influence. In one study I conducted, all responses were private - participants thought they were the only person being interviewed, but observers and I could view all responses together on the chat stream.

Showing materials. Still and animated materials (concept statements, ads, commercials, etc.) can be shown on a whiteboard. In some platforms, participants can use mark-up tools so they can circle and X-out different parts of concept. Participants can be given links to the client’s or other Web sites. And, of course, participants’ own visuals can be displayed.

Planned yet flexible. The topic guide is pre-loaded into the virtual room so that questions and instructions will be posted at specified times or, in live discussions, if/when the moderator chooses. In the case of live sessions, various probes come with some software (“Why do you say that?,” “Please tell me more.”) or can be written in advance by the moderator, then used when desired. These pre-planned time-savers do not confine the researcher, however. We can easily shift the direction of the discussion based on issues that arise.

Client role. Clients can observe the entire process as it develops. They have their own viewing room where they can talk with the moderator and one another. Communications with participants have to go through the moderator - that is, unless the decision is made to involve clients in the conversation. At the end of one magazine study, several editors came into the participant room; the back-and-forth dialogue greatly enriched the research.

Of course, online qualitative eliminates travel expenses, time and hassles, an advantage it shares with telephone qualitative. But other expected advantages are not always realized. Clients are sometimes surprised to find that online qualitative compared to other forms of qualitative is not necessarily:

Faster. Recruiting time is about what it is for in-person and phone qualitative study. Fieldwork time, of course, depends on the length of the engagement, which may or may not be different from other qual studies. One part that is definitely faster: raw transcripts are instantly available since participants type and post their pictures. (And, yes, there has been a decline in the public’s spelling, grammar and punctuation.)

Cheaper. Online surveys are significantly less expensive than door-to-door (remember those?) and phone studies, but online qualitative is, at best, slightly lower. Recruiting costs and incentives are often about the same; while people aren’t asked to travel to a research facility, studies can take a substantial amount of time. Moderator fees may be higher because some online studies are so labor-intensive, even without travel. (More about that later.)

Easier. Sure, it’s easier for clients and researchers to sit at their computers than to traipse through airports, but designing, conducting and, most of all, analyzing online qualitative projects can be more work.

What online qualitative method alternatives are there?

A wide range of online qualitative methods (products) already exists, and undoubtedly the number will continue to grow.

Real-time. Participants interact with the interviewer and one another live. Online focus groups or chats typically have six to eight respondents and last 60-90 minutes - that is, have about the same number of participants as in-person focus groups with somewhat shorter duration. Mini-groups using Webcams can bring together people around the country; there tends to be a short time lag between the audio and video, which can be distracting at first. One-on-one interviews done by phone with a Webcam enable the interviewer and respondent to see and hear each other, while visuals (e.g., ads) and text can be shown.

Asynchronous. A somewhat fancy way to say that people are in the virtual room at different times, usually when it is convenient for them in their time zone. Bulletin boards, which I personally like, are a popular example.

Words and pictures. Methods used to be divided between (primarily) text-based tools and image/video-based platforms, but increasingly these are coming together. Today’s software systems now support a wide variety of formats and ways of interacting. Bulletin boards, originally mainly question-and-answer, for example, now include more of the participant-created photographs and videos mentioned before.

Blogging. Participant journals can be used as a separate tool or integrated into other formats. These secure blogs, available only to study participants and observers, are an easy way for participants to record their thoughts and experiences.

Mobile. Text messaging enables participants and researchers to interact from anywhere. Rather than trying to recollect their feelings and actions later, they can keep notes or answer questions at the time. From what I’ve seen and heard, this approach seems to work better as a supplement to other online qualitative rather than as a standalone method.

Hybrid quant/qual studies. One approach is described as “incorporating moderated interview sessions into an online survey,” where respondents’ answers to closed-ended questions are probed. Another method uses live focus groups with anywhere from 25 to 200 people. Participants answer both open- and closed-ended questions; answer tallies and randomly selected verbatims are displayed in real time. (I worry about showing numbers on such small samples, particularly in percentages, and find the verbatims skimpy.)

Communities. These ongoing groups are typically large, with hundreds of people. Some communities consist of brand fans/customers for what one research company calls “insight and co-creation.” Research is often only one part of the community’s purpose. Here too the method is positioned as “transcend[ing] the qual/quant divide.”

What are bulletin boards and their advantages?

Bulletin boards, in one form or another, are the most popular online qualitative method. Asynchronous, the boards take place over an extended period of time. Participants may be asked to answer questions, read and discuss one another’s comments in a threaded discussion, perform different exercises and homework assignments, create and upload photos, and so on.

Most commonly, a study has between one and three bulletin boards (more is definite overload), with 18-20 active participants each, logging on once or twice a day over a period of three to six days. That size group, though too large in person, provides substantial interaction online without overwhelming participants.

The days or day parts are usually divided into different topics. Here’s an example of a four-day study: day one, general context; day two, specific products/brands; day three, reports (including photographs/videos) of store visits; day four, reactions to new product concepts. Or, there might be a week’s break after day three, then another day or two with the same participants on new product concepts developed in phase one.

The beauty part is that bulletin boards are so flexible. The answer to “How many people for how many days?” is, “Do what works for your study.” The size of a board can range from a small group (six to eight) to a large group (I’ve heard up to 80 mentioned). Longer studies can go for a year or more, for example, following the decision-making process; in these cases, posting is less frequent. Participants can be added during the board, can be selected for follow-up work, or (this moderator’s dream) discreetly dropped through the magic trap door without disrupting the group. We can subdivide a group for separate conversations or bring bulletin boards together. In one study of a Western state, for instance, one board was with long-term residents, the other with newer residents. After a few days, we created one board, encouraging dialogue between these presumably hostile segments.

Bulletin boards have several advantages as a qualitative tool, in addition to geographic spread.

Participation engagement. Bulletin boards offer the convenience of participation from home or office without being locked into a discussion at one particular time. (While we ask for daily participation in a shorter board, participants can catch up if they miss a day or two.) As a result, it’s often easier to recruit busy people, including higher-level executives, than for in-person interviews. Interacting with and learning from others also attracts some to the group discussions.

Rich responses. With more time to think, write, read, explore, post pictures and so on, participants often give us much more input. I much prefer bulletin boards to the live online chats, where everyone has to rush to type, and moderators researchers rush to read, retain and redirect. Compared to in-person qual, the bulletin board researcher doesn’t have to cut off articulate respondents who take airtime in a group, throw off the moderator guide schedule or simply annoy observers; on a board, participants can “talk” at the same time without depriving anyone else of their opportunity. (“Here are 10 reasons to buy organic” was the beginning of a 469-word post in one study.) Obviously, some people “talk” more than others, but the overall level of engagement tends to be quite high.

Little group influence. Because they do not interact face-to-face, participants tend to be candid, not worrying about offending others or being swayed by dominant respondents. With the masking feature, respondents react to materials initially on an individual basis, before seeing others’ comments. Screen names can disguise participants’ identity, freeing them to answer anonymously; employees who would normally be concerned about speaking up, for example, can give honest feedback. Another advantage is that a broader range of people can feel comfortable together in a board than they might in person. For example, in face-to-face focus groups on skincare, we usually wouldn’t mix women 21-50 years old since age can be a touchy issue. Online, because they don’t see one another, this works well.

Creative/interesting activities. Assignments during the boards can enrich the conversation. A few examples:

• Go to at least one store, look at products in the category (not just your regular brand).
• Talk to your friends to find out what they think of the issue.
• Visit a makeover Web site, upload your photo and try out different cosmetics.
• Send in a photo of yourself with a favorite treat and tell us why it’s special.

When are bulletin boards a good choice?

Bulletin boards can work well for many different kinds of studies - attitudes and usage, deep-dive exploratory, new concepts, advertising and so on. The choice of online qual and other qual doesn’t have to be either/or, however. Often they make a great combination in hybrid projects with other qual methods:

Before. Prior to in-person focus groups on happiness - a broad, perhaps intimidating subject - respondents were invited to a bulletin board with questions on when they experienced moments of being happy and unhappy. Their private answers served as warm-up for the sessions. In another study, prospects for an automotive brand participated in a journal-type board (no interaction); the most interesting people were selected for follow-up home visits and drive-alongs.

Simultaneous. Usually these are done together in the interest of time, rather than sequentially.

After. In-person interviews in one market with the client observing (often in their home town) help the team refine the topic guide and concept materials. The subsequent interactions with participants around the country provide a wider view. Online can continue the dialogue with participants, for instance, in probing their reflections on the in-person experience or having them try a product at home.

In the study in which we asked women to visit a makeover Web site, the in-person depth interviews and focus group enabled us to see what the women did on the sites, their body language in response to the sites and their appearance/style (relevant to cosmetics). In the focus group, we also observed how strangers bonded using the software. In the bulletin board, women around the country could upload their own photograph and play with a Web site as long as they wanted rather than being rushed; most sent us their photographs. Reactions across methods were quite consistent, giving us confidence that the face-to-face interviews in one market were not anomalous.

Needless to say, online doesn’t work for all kinds of studies. In-person is still better for tasting and touching products (vs. using the products on their own), for showing or sorting a large number of objects and for seeing how customers look and behave.

What should I look for in a moderator?

Should your bulletin board moderator be an online specialist or a qualitative researcher with some online skills? They don’t have to be a techie, although that can help. Several online qualitative software companies offer very good training for newbie online moderators and updates on their constantly-evolving technology. Some of these companies also provide excellent support during the studies, patiently explaining how-tos on posting the topic guide, sending probes and so on. Needless to say, it takes a while for most moderators to feel comfortable using the software.

Bulletin boards are different from other qual from a research standpoint as well. Topic guides need to be more fleshed out yet avoid being long-winded. Moderator posts have to be very clear on what we’re looking for since we’re not with the participant to explain, reword or probe. Participants may give simple yes/no answers if we don’t repeatedly emphasize we’re interested in why they feel a certain way. What would be a follow-up probe in live qual becomes part of the question online.

With all the differences, it is important to keep in mind that bulletin board moderators should fundamentally be good researchers who can design and analyze studies.

What are the problems/issues of bulletin boards?

Boards present a number of challenges for moderators and clients.

Maintaining interest. I’ve never had a participant walk out in the middle of an in-person interview, but overburdened or bored board members do complain or drop out. Graphics, stimulating questions, feedback, reminders and upping the incentives are among the ways to keep participants actively involved. In a business-to-business three-day board, we showed 17 concepts and asked the same questions for each. Explaining that we needed consistency and interspersing some refresher questions kept them with us.

Partial participation. Not everyone on a board answers every question, even with e-mail and phone reminders. In face-to-face focus groups this also happens but directly addressing an individual usually gets a response. Some board software can be programmed to require answering a question before proceeding. A related challenge is engaging “quieter” participants. An issue in any qual interview, these people can be more difficult to engage and draw out online.

Problems probing. Follow-ups by e-mail are generally less effective than in in-person or phone interviews. Because the probes are not immediate, they’re easier to ignore, even with reminders. Software improvements help in showing participants the questions and messages to which they have not yet responded.

Engagement overload. We need to remember that participants have a life outside the boards. We should stick to the time demands promised, rather than burdening participants with too many assignments and questions. Homework overwork is an issue, especially if the moderator underestimates (or fudges) the time commitment during recruiting. This doesn’t mean that we can’t be flexible; if ideas for new tasks arise, raise the incentives in recognition of the extra work.

Getting to know you. Following individuals is more difficult, although personalities do emerge and interesting/colorful people stand out. Strategies for dealing with this include sorting the transcript by segment (user/non-user), keeping notes during the board and running the transcript through Excel to see a person’s posts straight through.

What does it all mean? It’s great that boards are so rich - and in many ways this is the biggest challenge for every moderator I’ve talked with. A three-day board on home decorating resulted in a transcript of nearly 400 pages, plus four-to-six photos of each participant’s home. That’s an awful lot for researchers and clients to wade through. The software platforms have tagging features that enable researchers to classify verbatims, highlighting good ones, along with comments/analysis; some researchers also use special qualitative analysis software. A number of researchers use these but I prefer my own informal note-taking.

What role do clients play in bulletin board studies?

Clients are involved in the usual role of briefing the researcher, having input into and approving the screener, the topic guide and report. Observation of the boards, however, tends to be very different, and not necessarily in a good way. Real-time interviews, whether in-person or online, are “appointments” clients usually make time for on their calendars. With bulletin boards, though, client involvement seems to range from super-intense to non-existent. On one end of the spectrum, clients log on multiple times a day, sending multiple comments to the moderator on new probes, changes in the guide, respondents to get rid of, etc. Comments can come from a team of clients, who don’t necessarily agree with one another.

More common seems to be the other extreme, the absent client. In a client team of 10, for example, just one or two may log on just once over the three days, sometimes only at the beginning of the board before discussion has gotten underway. (The moderator view of a board shows us exactly when and how often observers log on.) From the client’s point of view, the fact that boards are less demanding of their time than in-person groups can be a plus. Moderators I’ve spoken with, however, want some direction during the study rather than trying to fix problems afterwards. Techniques that researchers use for engaging clients include sending daily updates, transcripts with starred quotes, intriguing quotes, ideas for new approaches and (trying to) schedule debriefs.

Is online qual real qualitative?

The basic goal of qualitative research is to get closer to people, to understand them more deeply. So how, non-users ask, can interacting with them online be true qualitative research? We need to observe non-verbal behavior, to see and hear people in person - their expressions, the way they dress, their overall appearance, their body language, their tone of voice, their responses to tasting a product, the way they relate to others in a focus group, and so on.

Each medium has its advantages, I find. Telephone interviews for qualitative studies are common with business professionals and consumers who are geographically scattered or unwilling or unable to come into a facility. Relying on sound only, researchers can establish rapport and learn a good deal about the person on the other end of the line.

Online qualitative enables us to get to know people in other ways. Emotions do come through in what they write (with or without emoticons), especially because they are free to write as much as they wish. Participants’ observations and their pictures of themselves and their world can be highly revealing. Especially in the asynchronous methods, every participant has an equal opportunity to weigh in. In discussions that take place over a period of time, people also get to know one another and open up more.

What kinds of people participate in online qualitative?

Are they odd, atypical, super techie, all young, all upscale? No, I don't think so. Internet penetration has risen pretty much across demographic groups (including in the 50+ group, whose participation is often underestimated). Most online qualitative studies only require Internet access (sometimes high-speed access) and skills no greater than typing, even with two fingers. Essentially, people who can use e-mail can easily participate. Motivations to be involved are the same for other research studies, with the added attraction of participating from home or office, at a convenient time (asynchronous studies). Incentives are higher than for quantitative online studies. And qualitative studies are more interesting, more fun - that’s what participants tell us; at the end of bulletin boards, they often spontaneously talk about how much they have enjoyed sharing their views and having someone interested in their feelings.

Clients also wonder where online qualitative participants come from. Are they online panelists who participate over and over, and, most importantly, are they who they say they are? This is an issue for all online research and, to some extent, in-person qual too, though of course seeing people enables us to be pretty sure of at least some things about them.

Good recruiting services do not rely solely on online panels but, instead, use a variety of sources - national or local databases, client lists, referrals, and directed campaigns (e-mail, ads, phone calls), etc. These services limit participation frequency. Recruiters also make a point of talking with participants in screening and/or confirmation. This provides reassurance of participants’ identity and consistency of their answers; it makes the research more tangible to participants as well. On an intuitive level, none of the participants in studies I've conducted have seemed fishy to me.

So, how to get started?

For first-time clients, a toe-in-the-water approach may be best: use one or two bulletin boards as a complement to the qual method you’re comfortable with, and use your regular researcher, if possible. Let the bulletin board method free you. Imagine a focus group with people from just about anywhere; that’s almost as long as you want; where you can ask follow-ups days later, even after the research is officially over (you have to ask permission during the board); where participants can go beyond the virtual room - to the store, to talk with their friends and family, to check out Web sites, to try a product in their own home; where articulate people don’t have to be cut off in the interest of time; where you get to see people’s real-life world; see their thoughts and feelings represented visually. Remember that you’re not locked into an online interaction - you can arrange for a phone, Webcam or face-to-face interview too. Exciting, isn’t it?