Useful usability testing

Editor’s note: Steve Ellis and Pamela Ellis are founders/partners at Inverse Ratio, Inc., an Irvington, N.Y., research firm.

First some numbers: Net Effect reports that 67 percent of Web shoppers click out of the shopping cart before completing a purchase. Accenture says that 35 percent of people who experienced problems on a particular site will immediately leave that site for another. Boston Consulting group has shown that 45 percent of users abandon Web sites with poor navigability, slow download times, or confusing content.

Fortune 1000 companies will spend an average of $2 million this year on a single Web site redesign, without actually knowing if the redesign will even begin to address these critical customer experience problems. How has this happened?

We think it’s because marketers aren’t paying enough attention to the most important aspect of any successful Web site - its usability. Bottom line: Can your customers accomplish quickly and easily what they are supposed to do on your site? Do you really know? How would you find out? Surveys won’t tell you. Focus groups won’t tell you. All of these methods involve self-reported data that only measures preferences. It measures saying, not doing. And usability is all about watching what people do and learning from it.

Marketers need to know more about usability research. They need to know how to do it, when to do it, and how much it costs. We think the natural role of the marketer on a Web team is to become the user advocate. Who else knows the customer better? Who has the budget to spend on research? Let’s get Machiavellian: Who’s likely to be held accountable for a poor customer experience in the long run?

Making the business case for usability research

If it isn’t scary enough that your Web site might make it hard to accomplish a basic task, like finding product information or making a purchase, there are hard numbers you can point to to make the business case for usability research. Conversion rate, or the number of shoppers who become buyers, is something on everyone’s mind these days. Sites that have been improved through usability testing typically have higher conversion rates and less customer churn.

Let’s look at conversion rate. With an industry average of around 1.8 percent there would seem to be nowhere to go but up. By increasing a conversion rate of 2 percent to 3 percent, your revenues will increase by 50 percent. Studies have shown that the cost-benefit ratio of a usability study can be as high as 1/400 (for every dollar spent $400 is saved or made). If you’ve done usability research in your organization (or a naysayer has) and didn’t see a significant improvement in your conversion rate, a reduction in churn, or efficiency improvements, you may have serious problems with how that research was done.

What usability is and is not

First, a few words about what usability research is not. It is not focus groups with computers on the table, where participants are prompted to spout off opinions about what they like or don’t like. It is not a post-hoc survey tacked onto a Web site with questions like “Did you find this feature usable?” It is not grabbing people from the accounting department and sitting them down in front of the developer’s workstation and asking them what they think (although this would probably be better than nothing!). Usability research is not about preferences or opinions. It is not “I like this design” or “This color scheme really stinks!” But don’t worry. People will naturally tell you what they like and don’t like.

All good usability research is aimed at discovering what people are able to accomplish based on cognitive and physiological ability. If someone is not able to find the download button, typically it is because he or she cannot see the button or the button is difficult to click. It is for this reason that narrowly defined usability studies are valid with as few as eight (some would say five) participants. As humans our cognitive and physiological abilities are not as variable as, say, our personalities, preferences, and opinions. So usability research does not require larger samples, as do opinion studies, to be predictive.

Demographics and geography are not as relevant in usability research as they are for opinion-based research. However, if the site is designed for automotive engineers, you are best off if you find those people. If you don’t, it won’t necessarily invalidate your data, because engineers (despite what some say) are still human, after all.

What is important is that all of the users you get have a baseline of skill for using a mouse and a keyboard. Time spent teaching someone how to double click will not be time well spent. A good deal of our recruiting effort is directed toward screening for these often taken-for-granted skills. Almost nothing can undermine your credibility more than testing an application for usability on someone who doesn’t know how to use a mouse.

Most of the usability research we do takes place one-on-one with a moderator who walks (but doesn’t lead!) a user through a set of predetermined tasks. We log what users do onscreen with video of their posture and facial expressions. Usually sessions last about an hour. Each session produces a rich dataset, which is analyzed and distilled so findings can be communicated effectively.

The way usability studies are conducted is often a source of confusion for the uninitiated. Watching a usability study for the first time can be variously funny, excruciatingly boring, depressing, and revelatory. The confusion arises from the fact that we are interested in trends across multiple users. To get evidence of a major problem you have to sit still and be attentive for more than five hours at a time, something most people are simply not capable of.

The different types of usability research and the development process

All usability research is not created equal, which is not to say that some methods are necessarily better than others. Rather, different research situations and goals require different methods. The trick is knowing which method works best in what situation. Slightly more free-form and open sessions make more sense early in the project lifecycle when you want to know just as much about what users want to do as you do about how they do it. Here you can swap out different prototypes representing different tasks and get immediate feedback about which is better. Keep in mind, as with all usability testing, we’re not as concerned with features that users say they want as we are with watching and learning from what they do. So having them interact with prototypes is key.

As the project progresses and new versions are created a more rigorous form of usability testing is appropriate. Here it makes sense to focus almost exclusively on tasks and people’s ability to complete them. And here comparing an old version with a new and “improved” version will have the greatest impact. Sessions like these can be extremely effective in resolving differences of opinion about the elements of the user interface that actually work. If the sessions are done well, it should also be possible to provide hard numbers that show one design element is better than another.

How Web development processes work and where the marketer fits in

Marketers are frequently in the position of hiring usability researchers. They have the research budgets and product development teams often go to them for help. More and more marketers are taking a proactive role and actively participating in product development acting as the chief customer advocate. If you choose this role (and we think you should!) be aware that you may be stepping into the lion’s den.

When you bring a usability specialist in make sure they meet all the right people from the team on day one. Your credibility is at stake and nothing makes a programmer unhappier than the prospect of someone telling him he is going to have to change his code. You and your usability researchers have to earn everyone’s trust with your knowledge and confidence that what you are doing is critical to the project’s success.

How much usability research costs

Like any professional service, the cost of usability research varies depending upon the project requirements. Contrary to what some believe you do not need a $100,000 usability laboratory to get results. It is preferable that the research firm have its own means of logging the data it collects in some sort of video format. We use a setup that fits in a suitcase that can be checked as baggage. Expect to pay between $10,000 and $20,000 per day of testing with eight to 10 participants a day. This should cover every aspect of the testing, from recruiting to reporting, with costs per day going down for each successive day. Using an informal setting like an office or conference room can save money. Your goal should be to keep costs low and to focus on testing successive iterations of the design rather than blowing it all on the all-encompassing perfect study.

How to use the results of usability research to make real improvements

Good usability researchers don’t just point out what is wrong with a Web site or application. They have an intimate knowledge of the technology they are studying and are fully capable of making concrete recommendations about how to make it right once problems are discovered. If they don’t have this knowledge, or waver for a moment when presenting their findings when asked a question by an engineer or developer, their credibility all but flies out the window, and so does the validity of the research. Developers have a strong tendency to disrespect anyone who is going to tell them how to do their job who doesn’t know their job inside out (and is able to display that knowledge in touchy situations).

All of this is to say that the worst usability research, no matter how expertly done, is the kind that has no impact on design. The goal of all usability research should be to make changes that make a real difference to the customer experience. Everyone on the team needs to be dedicated and motivated to make this happen. Lead the way and reap the rewards!

Establishing metrics and measuring success over time

Having hard numbers to point to when things get tough doesn’t hurt. Numbers come in handy the most when two designs are being compared. If a study is designed correctly, it should be possible to show with numbers why one is better than the other.

Success rate is a number that we often use when quantifying the results of a usability study. Simply stated, success rate is the percentage of people who were able to complete a given task. Each task under consideration (such as purchasing a hat or finding product information) can have a success rate associated with it. What qualifies as “success” is up to the usability researcher. The key is that the standard be applied consistently across participants and design iterations.

Success rate captures information about tasks, but let’s say you were interested in things that aren’t directly related to a task, like the success of your navigation scheme, how many times a user encounters an error, or even whether users express a feeling of distrust while filing out your registration form. For things like this we have invented a metric called, appropriately enough, the pain factor. Unlike success rate, which you would like to see increase, pain factor always needs to be reduced.

It’s the subtleties

It may sound completely crazy but we find that the average Web site has more than 100 usability problems prior to undergoing a thorough review. This isn’t to say that all of the problems are major problems. Things like an incomprehensible error message or poorly defined navigation category would be included in this long list. Most sites have three to five major problems that severely impact the overall usability. Here we would include things like task flow for critical tasks like registration and the location and style of global navigation.

The point is that we have to recognize that creating a usable site is just as much about the little things as it is about the big things. The fact that the Web is an extremely mutable medium is both its blessing and its curse. The only thing that Web sites necessarily have to have in common are things like radio buttons, check boxes and text input areas. After that, all bets are off. Your navigation can be floating off to the left or in the upper right. Your link colors can be any one of 216 colors. While allowing for a great deal of creative freedom, it ultimately confuses users. The goal is to become expert in what doesn’t confuse people and be effective communicating that to the project team. Implementing a customer experience program, of which usability testing is a part, will be a major step in the right direction.