Editor’s note: Jacob Brown is principal of In-Depth Research, Tiburon, Calif.

At the beginning of the 21st century, Web site usability research was probably the fastest growing area within the market research industry. Everyone was pouring money into developing sophisticated sites, e-commerce was booming, new site concepts were being developed on a daily basis. Then the market crashed, budgets shrank, and the resources available for developing and testing new sites dried up.

But research moves in cycles. Companies have recently begun to remember the importance of evolving, expanding and refreshing their Web presence. And as Web development dollars have begun to grow again, so have the resources for Web site research.

While it is clear that companies are returning to usability research, the jury is still out on how usability research should be conducted. This article is meant to provide professional researchers and clients with a perspective on conducting usability studies. It does not attempt to define the “best” way. Rather, it is meant to provide a point of view and some suggestions that may help shape the discussion.

At the simplest level, there are two basic approaches to Web site usability testing and each has its own strengths and weaknesses. In this article, we’ll focus primarily on qualitative usability interviews.

Qualitative usability interviews

Qualitative Web site usability interviews are traditionally done with one respondent — or site visitor — and one interviewer sitting together at a computer. The interviewer prompts the respondent with a series of open-ended questions as he directs his or her attention to different areas of the site (e.g., home page, nav bars, product layout) and asks the respondent to attempt a variety of tasks (e.g., sign up for an e-mail newsletter).

By watching how respondents navigate the site and by listening to their comments, patterns emerge in how visitors respond to both the site as a whole and to specific elements. As the interviews progress, the moderator is able to drill down deeper and deeper into these areas. Research questions that can best be addressed by this methodology include:

  • What is the visitor’s overall impression of the site?
  • Is the site easy to navigate and does it follow a logical design?
  • What are the obstacles to completing important site tasks?

Because the interviews are qualitative, a large number of interviews are not necessary. We typically recommend 10-15 interviews per user segment. Therefore, if you are just testing one segment consider doing about 12 interviews, but if you’re testing two segments (e.g., design engineers and line of business managers) try a total of 18-20 interviews.

Clients are often attracted to this idea of conducting usability research in a focus group setting; it increases the efficiency of the process and lowers the per-interview cost. However, it also dramatically reduces the effectiveness of the research. In order to truly assess the user’s experience, the respondent must be free to navigate the site as they might at their own desk, not while trying to keep up with a group. The group setting creates a distance between the visitor and the site and the interaction between group members interferes with the respondent’s ability to clearly communicate his experience with the interviewer.

A group setting can be an appropriate choice when exploring design concepts and getting feedback on new ideas. In that case, the group is used in the same way that one might do an advertising concept test or creative exploration. The focus in that case is on the strengths and weaknesses of different design strategies rather than understanding how it feels to navigate through the site.

Quantitative usability surveys

Quantitative Web site evaluations are great when you want to test alternative design elements or choose between different site layouts, or select navigation iconography. But quantitative surveys are generally not very helpful in understanding the visitor’s experience.

Use a quantitative approach when you need a statistically reliable answer on which design alternative visitors will prefer. Examples of the types of questions you can answer best with a quantitative approach include:

  • Do visitors prefer lots of small product pictures on one page or a few larger pictures on two pages?
  • Which icon style most clearly communicates its meaning?
  • What percentage of respondents can successfully find a specific item from the product inventory?
  • Which navigation strategy results in the fewest dropouts?

Assessing site usability

Web designers often believe that the fewer the clicks the more usable the site. Nothing could be further from the truth. Visitors will report a much more positive experience for a page that uses five clicks that are logically designed and easy to find than for one that only takes two clicks, but requires them to hunt through a page of content to find the right link.

Try to resist the urge to focus on the number of clicks it takes to find a specific page or how many seconds it takes to complete a task. Instead, investigate the respondent’s experience finding the material or completing the task.

  • What did you expect when you clicked on that link?
  • How did you decide where to click?
  • Did the path feel logical, fast, easy to find?

The answers to these questions will be much more useful than just learning that it took three clicks to find the account sign-in page.

Visit other sites

Research participants cannot design a Web page for you. If you ask a respondent “How would you like to see the product page laid out?” they will rarely provide a useful answer. After all, they are Web surfers not Web designers.

At the same time, respondents are excellent at telling you what they like or don’t like about a Web page. That’s why it is often a good idea to visit other sites during the usability interview. If you’re not sure which approach to take when designing the sign-in page, take the respondent to two or three other sites and have them compare the different approaches. That way you can quickly discern some of the advantages and disadvantages that go with any design decision.

The goal is not to rip off someone else’s design; you are just giving the respondent something to look at and respond to. This will stimulate the discussion and provide more actionable insights than by asking them to come up with the design on their own.

Similarly, it may be useful to have them visit some of your competitor’s sites. This is an opportunity to understand how your site stacks up against your competitors and understand the different imagery that each site communicates.

Negotiating turf wars

Large corporate Web sites often suffer from ongoing turf wars. The site is “owned” by multiple business units, each of which lays claim to a portion of the site’s footprint and navigation structure. As a service department, without budget or authority, the Web group often has a hard time laying down the law and managing the conflicting needs of the business units.

Three signs of this type of internal conflict are: one menu for the corporate site and one for the business units; page layouts that change from section to section; a homepage that is packed with information but has no overall organization.

In these cases the usability research has often been commissioned as a means of forcing the different site owners to cooperate and impose design rules. It is often best to resist the temptation to have the research be the judge of the best site design style or graphics. Instead, focus the research on how the lack of consistency impacts the user’s experience. For example, do respondents:

  • Know where they are on the site?
  • Understand the roles of the multiple navigation menus?
  • Realize when they’ve moved from one division’s site to the next?
  • Get lost and not know how to retrace their steps?

Testing imagery

It is very easy to become so focused on the mechanics of a site that you forget that your site is also an expression of your brand. In addition to the more functional or task-driven research activities, have the respondent take a step back and respond to questions like:

  • What is the first impression you get from the home page?
  • What kind of person is this site designed for?
  • How does the site fit with your perceptions of the company?
  • How does the site make you feel about the company?
  • What is the company trying to say about themselves?

These questions are most effective early in the interview before the respondent has become too educated about the site and the company. Their initial reactions to the site will give you the best indicator of the image the site is communicating.