Watch me as I buy

Editor’s note: Matt Schroder is director of communications for the Qualitative Research Consultants Association, St. Paul, Minn.

Imagine the ethnographic research technique of task observation - say, perhaps, observing a consumer while they shop for clothing in a department store. Before the Internet, consumers were likely seeing their clothing choices for the first time, unless they happened to see the item in a circular or catalog.

Now imagine doing that same task in the online era. You observe your subjects as they enter the store, but they go right to the new dress they want, grab it from the rack and take it to the checkout counter and leave. That creates a big gap in the understanding of how a purchase decision was made. The problem, of course, is that the customer did their research and made their comparisons, color choices and style decisions somewhere else: in front of their computer.

To get the ethnography insights needed in the Internet Age, researchers are realizing the need to supplement their methodologies with observational techniques that take usability testing into the respondent’s environment, with the researcher in a remote location. “Research tools are changing rapidly, and we expect them to keep changing,” says Abby Leafe, president of the Qualitative Research Consultants Association (QRCA), St. Paul, Minn. “No one method is always right every time - we always try to balance cost against the client’s objectives. A skilled researcher can suggest multiple approaches to achieve business goals.”

Ethnographic studies were founded on the idea that consumers are best understood in the fullest possible context, including: the places where they live, work, play and shop; how they are making a living and providing food, housing and energy for themselves; the language(s) they speak and so on.

What’s changing is the information and experiences that consumers are exposed to before, during and after they’re being observed in those environments. The proliferation of the Internet, especially hand-held wireless devices, is changing the game. And the qualitative research industry is keeping pace.

Users who had purchased their existing car from another site were asked to buy the same car on the Honda (UK) site and compare the experience. A recording of their actions was then played back to them with the moderator and client teams asking about points of interested tagged during the session.

More intuitive

In early 2009, Honda (UK) rethought the part of its Web site aimed at used-car buyers. The previous site used the common dropdown menu page design where users pick their model and price range to come up with available used cars. A new design was created to make the search experience easier and more intuitive to use, and to distinguish Honda from competitor sites.

QRCA member Jamie Hamilton, managing partner of U.K.-based research firm Nqual, worked with U.K. research agency Simpson Carpenter to conduct the qualitative research for Honda (UK). The team combined Internet-based research and application-sharing technology with the aim of unobtrusively observing and capturing natural online behavior.

The methodology is simple. Participants are given a single or set of online tasks to perform on their home computer. Immediately prior to the session, the moderator calls the participant to get a sense of how they expect they will perform the task. The participant is then left to perform the task(s) uninterrupted while the moderator, usability expert and client team remotely observe their journey. Immediately following the task(s), while the participant’s experience is fresh in the mind, an online recording of the journey is played back to them with the moderator and client teams navigating to, and asking questions about, points of interest they have tagged during the session.

Of the sessions Hamilton’s team conducted for Honda, half of the participants were Honda owners who had purchased their used car using the old Honda Web site. The other half were people who had purchased any car from another Web site. Users who had purchased their last car on the site were asked to “buy” their old car on the new site and compare the user experience. Users who had purchased their existing car from another site were asked to buy the exact same car on the Honda (UK) site and compare the experience.

“We call our approach ‘e-thnography’ because although our context is online, we have the same goal as classic ethnography - to non-intrusively observe as close to natural behavior as possible,” Hamilton says. “Unlike standard usability testing, participants were not in an unfamiliar research facility or testing lab - they were in the comfort of their own home, at their own computer. Unquestionably, observer effects, moderator interjections and unnatural environments and procedures distort findings. When seeking to understand any natural performance - skiing, dancing, stand-up comedy - it is counterproductive to conduct the study out-of-context and continually interrupt the performer mid-flow to ask questions. It is for this reason that we capture the participants’ journeys and reactions remotely, whilst tagging points-of-interest en route. Only when they’re done do we explore events and experiences with them by reference to a shared online recording of their journey.”

Honda (UK) is getting essential feedback on the new used-car selection tool. It is making a range of usability tweaks and attaining an understanding of the higher-level preferences, motivations and dispositions of a variety of Web site visitors and users. “To give one example: amongst participants who set their own starting point for the exercise there emerged several categories of behavior based on the level of patience and method they demonstrated,” Hamilton says. “These categories employed different search engine techniques, reached the tool by various routes and landing pages, and had distinctive approaches to absorbing site information and learning the ropes. Findings such as these showed that improvements to search engine optimization, and tweaks to Web site design and usability, should be catered to different behavioral categories operating in a wider context, as well as expected user-end goals, and statistical findings.”

This technique can offer cost efficiencies over doing a study in a usability lab, which requires the renting of venues and equipment and potentially includes travel costs, Hamilton says. “Doing everything remotely, online, is also very fast. Within four days of commissioning the project, we found the participants and we were conducting our first sessions. Transcripts and recorded files were available immediately for analysis, and the process was further accelerated by being able to trial suggested improvements and theories with later participants,” he says.

Same benefits apply

Those who practice these online observational techniques report that all the same benefits of traditional ethnography apply, including holistic observation, real-life insights and the ability to minimize the “research effect.” The difference of course is the ability to take advantage of the Internet, with two billion people online worldwide. “An online presence is a prerequisite for modern businesses. In fact, Web sites are increasingly the main communications channel as well as often being a purchasing channel,” says QRCA member Kasia Gandhi, associate director of U.K. research firm RS Consulting Ltd., who co-presented with Hamilton on this topic at a recent QRCA annual conference.

However, while many companies often spend an enormous amount of money on their digital presence, they often do not truly understand their consumers’ behavior and how to keep them interested in their Web site. Site hits, page views, visitors and referrers are important metrics, but they are not all-telling. Web analytics offer only aggregate data: they tell us the what but not the who and why. They reveal little about visitor motivations and needs, which underpin decision-making, or how each Web site fits into the larger landscape of the Internet. “What we’re talking about is showing the participant a recording of what they did while it is fresh in their memory, and in most cases they are able to articulate why they did it,” Gandhi says. “In fact, the increasing importance of Internet assets, combined with the limitations of lab usability testing and analytics, were directly responsible for this breakthrough.”

QRCA researchers suggest there are several other applications where this solution could apply, including:

Understanding online behaviors and attitudes - such as the online decision-making process, online purchase habits, how information is gathered, and understanding “disengaged surfing.”

Qualitative Web usability testing - getting subjective insights on the user experience, evaluating Web design, and examining insights derived from quantitative Web usability testing.

Advertising tests - online ad evaluations and viral marketing tests.

Analysis - conducting competitive Web site analysis or problem discovery.

Important distinctions

Hamilton stresses that the specific methodology used in the Honda (UK) case, referenced earlier as “e-thnography,” should be distinguished from online ethnography, a term which has been employed to describe an online diary/blog/immersive methodology which involves participants writing about their behavior in the real world, and using the Internet merely as a reporting mechanism. Ethnography in its classic sense is an observational methodology which aims at objectivity and therefore “online ethnography” can be confusing to research buyers.

What was achieved in the Honda (UK) project was the study of human behavior within the virtual world of the Internet. It is therefore much closer to the traditional definition of ethnography, a time-tested and proven technique within the qualitative research industry. “We’ve found terminology can be a very interesting topic to discuss among researchers within QRCA,” Leafe says. “The needs of research buyers are always changing, and the rules of the virtual world are moving so quickly. While it’s important to clearly define what our terms mean, it’s that much more important to focus squarely on the needs of the company doing the research and finding the most credible, effective qualitative solution for them. That’s what was done in this case, and that’s what will keep the solutions we as qualitative researchers can offer viable into the next decade.”