At my own pace in my own place

Editor's note: Theo Downes-Le Guin is principal of Doxus, a Portland, Ore., research firm. Ted Kendall is vice president of innovation and development at QualTalk, a Castle Rock, Colo., qualitative software company. Ruchira Gupta is a researcher at Doxus.

As a method for conducting qualitative research, online bulletin boards (or bulletin board focus groups, as they are sometimes called) continue to grow in popularity. Online bulletin boards are similar to Web-based focus group or chat sessions, except that they occur asynchronously. The moderator posts a question on a secure Web site; participants log on at their convenience throughout a one- or multi-day period and respond to questions. Each participant can see the answers that others have given (usually after posting his or her own comments) and is encouraged to interact with other participants, not just with the moderator. The ideal result is a rich, developed dialogue about the given topic.

QualTalk, a Castle Rock, Colo., provider of bulletin board research software, has seen demand for bulletin boards quadruple from a year ago, and we believe demand will continue to grow as researchers find that the combination of convenience, depth of discussion, and group dynamics proves useful for certain qualitative research situations. To date, however, most articles on online bulletin boards have described the method from the researcher's perspective - how the bulletin boards differ from traditional focus groups, tips and tricks, and so forth. But how do participants feel about this new approach to giving qualitative feedback?

Over the past year, Atlanta research firm Doxus has included questions in several bulletin board discussions to learn how participants react to the method. (Doxus is a licensee of the QualTalk bulletin board software.) Typically, at the end of a bulletin board discussion, participants were asked about their experience of participating in this format as compared to other forms of research in which they may have taken part, including surveys and focus groups. Keep in mind that the research populations and topics for these projects vary widely, but for the most part we talked to professionals and a few consumers regarding technology products and services. We chose the bulletin board method for these projects because of its appropriateness to the population and topic, so this article in no way represents feedback from a stringent experimental "research on research" design.

We reviewed the responses of 110 participants taken from a dozen bulletin boards to assess the experience from the participants' perspective. Our questions about the bulletin board experience were open-ended, but generally, we talked to participants about their experience in terms of: convenience, discussion and interaction quality, and software interface/usability.

Convenience: the wherever, whenever factor

Most of the participants we interviewed had participated in traditional, in-person focus groups or standardized surveys in the past, with a smaller subset having participated in other online qualitative methods such as online focus groups. Participants expressed a preference for bulletin boards in terms of convenience compared to in-person research methods, and to some extent compared to online, real-time focus groups because of their flexible, asynchronous approach. Many comments regarding convenience underscored the general appeal of the methodology:

"I liked it quite a bit actually. Flexible schedule, lots of opinions, and good questions. On line provides [the] best value of time."

"I liked this format. I have done online focus groups and I have done in person research groups. I hope more research groups will take this approach in the future."

The bulletin board methodology allows respondents to participate from anywhere, according to preference and their ability to get the most reliable or fastest Internet access. Participants said the method also offers flexibility in terms of timing as they can "pop in and out of the discussion" over a relatively long period of time.

In sum, participants appear to recognize and appreciate that bulletin board discussions are very different from the traditional approach to research, which involves a contained interaction that may or may not happen at the respondent's convenience.

Discussion quality: thoughtfulness versus spontaneity

To some extent we can infer that bulletin boards' flexibility and lack of time pressure lead to increased discussion quality. Participants remarked that self-pacing allows them to post more thoughtful ideas and to reply more thoroughly to other participants' posts. In the final analysis, this benefit is fairly subjective, and varies substantially by individual. Across many studies we have noticed that some participants indeed take the time to consider and edit their responses, while others treat the method with the immediacy and relative casualness of a chat session.

As with traditional focus groups, we have observed two elements that drive the overall quality of bulletin board group discussions:

  • the quality of individual responses or posts, meaning the ability to understand the moderator's questions and articulate valuable responses; and
  • the quality of the discussion based on group interaction, and extent to which the group dynamic enhances or goes beyond the moderator's initial questions.

A few participants mentioned the lack of dominance of strong personalities as a merit of bulletin boards compared to an in-person focus group. Because of dominant respondents, "some in-person group interviews don't give enough opportunity for everyone to express their opinions," while in a bulletin board all participants have (theoretically) equal opportunity to post their comments without being "dominated by...persons that love to hear themselves talk."

Despite these advantages, however, we found that a subset of participants is keenly attuned to the trade-offs between participating in an in-person group versus a bulletin board. Many participants recognize that the bulletin board discussion is not always able to achieve the richness and participatory gratification of an in-person focus group - a point with which, at least for some topics and populations, we wholly agree. The concern is not so much the lack of spontaneity of an online discussion as it is the absence of the liveliness of an in-person focus group. But, as one respondent neatly summarized, "You're trading off the spontaneity of real-time interaction for responses that are perhaps more thought-out."

A related concern regarding bulletin boards - and one that is obvious to researchers who've used the method - is that the price of a leisurely discussion pace is the difficulty of maintaining participants' attention over a longer period of time and through multiple interactions rather than just one. Some participants we interviewed commented that the online experience simply didn't grab and hold them as an in-person group would. In terms of sheer physical stimulation, most participants with a basis for comparison agree that in-person groups are optimal for keeping participants engaged:

"It's easy to drop out of an online exchange."

"In-person...you're in a controlled environment without the distractions of the office."

All in all, we find that participants reflected nearly all the pros and cons we have found in using the bulletin board method. Participants recognize that a certain interpersonal quality (as well as the simple pleasure of meeting new people) is lost, but in place they may gain a chance to express themselves more fully or consistently. As with other methods, the researcher is ultimately responsible for maintaining participation rates by making sure that the appropriate respondents are recruited, the discussion is relevant and interesting, and reasonable means are employed to keep participants engaged throughout the process.

Software interface and usability

We heard mixed reactions to the QualTalk user interface. Some felt that the "software was extremely easy to use and lends itself more to discussion than debating." Others experienced problems such as run-time errors and navigability issues. (The interface was updated during the year these discussions were conducted, in part as a response to participant comments and needs. Some of the bulletin board discussions from which this article is drawn were simple text posts while others used whiteboard graphics, embedded links, and other tools that proved challenging for some participants.)

On the whole, most first-time participants quickly grasped the asynchronous nature of a bulletin board and were able to easily take part in the discussion. The technical problems some faced usually came as a result of the unpredictable nature of technology - idiosyncrasies of browser version, connection speed, and Internet congestion, or Windows registry conflicts. Reactions to the software interface thus depended largely on the unique experience and technical sophistication of the user. Many of the participants we interviewed during this period were IT managers, resellers or relatively sophisticated end users, which on the whole resulted in a lack of serious usage problems - but also created high expectations for a glitch-free experience ("It was disappointing to be ready to participate and have to wait until the technical problem was addressed. Other than that I thought this program was easy to use.").

Remember, however, that our analysis only reflects the comments of those who "hung on" through a full session. Initially, our bulletin board discussions demonstrated significant drop-off in participation over time. Anecdotal evidence we gathered while trying to convert non-responders into engaged participants showed that technical difficulties can be a real factor in this drop-off. Technically literate participants may be more likely to see the process through. However, changes to the interface and the underlying programming during the year, which resulted in greater technical stability and fewer technical problems, resulted in a lower drop-out rate due to technical issues over time.

Practical implications for researchers

From a participant's point of view, the bulletin board group experience is clearly different from a face-to-face interaction, and on balance the differences are complementary to the method. The convenience of participating in a bulletin board suggests high suitability of the method for research among professional populations who are hard to recruit for a synchronous two-hour focus group, whether in-person or online. The apparent likeability of the method for most participants further suggests that, while in-person or online chat-style groups may sometimes offer a more fun and participatory experience, the bulletin board approach can be equally rewarding for participants. The most important inference we can draw from this feedback is that asynchronous discussions may actually lead to greater cognitive involvement for some topics, while erasing barriers to equal participation such as typing skills or verbal articulateness.

Participants' comments regarding quality of discussion and interaction underscore our belief that the method is not appropriate for every topic or every participant. Attempting to uncover deep personal values and emotional motivations, for which moderators typically rely heavily on non-verbal cues and patterns of group interaction, is difficult in bulletin board discussions. An interesting exception to this may be topics of extreme sensitivity or where there are major social desirability biases. Bulletin board groups offer a real - but in our experience untested - opportunity for candid feedback due to the relative anonymity and low pressure of the setting. As the method catches on, we look forward to contributing to - and benefiting from - an emerging "best practices" knowledge base for how and when to use this method.