Editor's note: Cheo Massion is qualitative research manager at Socratic Technologies, a San Francisco research firm specializing in interactive and computer-based techniques.

With the rapid increase in software program releases and daily additions to the World Wide Web, software developers and publishers are becoming increasingly aware of customers' reactions to and acceptance of their products. In order to understand how customers will respond to new software versions, many companies are undertaking user interface (UI) research and usability testing.

This testing allows a company's developers and marketing staff to view real-time customer reaction to their prototype software. Once the testing is completed and analyzed, the development team can alter critical areas of the software program making it more user-friendly before the product is shipped, and thereby greatly enhancing the product's overall acceptance and success.

Traditionally UI testing has been done in formal UI labs where multiple unit audio and video equipment is permanently installed. These types of laboratories require large capital investments and usually employ human factors engineers. If you're wondering how much it costs to use such a laboratory, be prepared to pay up to $20,000 per day. For most research budgets, this kind of investment is well out of reach. The good news is that user interface testing has changed considerably over the last few years with the PC revolution and accessibility to portable UI equipment. Now not only can the development team go where the customers are, but overall research costs are significantly less; for about $4,500 per day clients receive a full day of UI testing.

So what would one find as part of a portable test lab? The portable UI equipment setup includes a personal IBM or Macintosh computer, a professional quality video camera connected to a video-out splitter, a viewing monitor and recording VCR. This system creates two images: the first image is of the software interface (full screen image), and the second image is a "head-shot" of the respondent. By using the video-out splitter a single video image is ultimately created which shows both the live action on the PC screen and then superimposes in a smaller floating window the head-shot of the respondent. The effect of this technique allows clients to see and hear both the screen activity and the emotional state of the respondent as they interact with the software program. Output from the system is fed to a viewing monitor for observers following the live action and also to a VCR for permanent video recording. With this setup virtually any focus group facility or spare office can be turned into a UI Lab overnight for interface testing.

Very useful

Companies who have used this technology find the live action testing very useful in understanding what portions of the program work well for the customer and which areas need more refinement by the development team. Even if only six to eight in-depth interviews are conducted usually one or two key areas of confusion are identified which baffle most respondents. Incorporating this feedback into final programming code has been invaluable to the ultimate success of their products according to software manufacturers, Web site developers, game producers and multimedia publishers.

A basic UI test interview is direct and employs standard in-depth personal interviewing techniques. Normally, respondents are pre-recruited with the help of a fielding agency and are offered a cash incentive ($35-$45). The respondent is brought into a viewing room with a one-way mirror for client observation. The moderator introduces the discussion topic and puts the respondent at ease. After general impressions about the application are verbally explored, the respondent is shown the software, usually through a brief demonstration.

In the next step, the respondent is given one or more tasks or activities to complete on their own, unaided by the moderator yet with encouragement to identify what areas of the program work well or are easy to use. Equally important, the respondent should indicate what areas do not work well and explain why these areas are problematic from their perspective. The moderator's role here is to capture the respondent's internal dialogue while working with the program. As with most qualitative studies, it is critical that the moderator remain neutral about the respondent's commentary and avoid answering the usual series of questions about how the software works versus their expectations of it.

Test concepts and content

Specific areas of UI testing include concept and content testing, where the overall concept of the software or Web site is evaluated by the respondent. Here questions explore what functional areas of the program the respondent would use first and why, which areas may be appealing or unappealing and why, and possible areas that they would avoid. The latter is especially interesting to developers because the feedback may lead to a simple alteration in the program design or copy which greatly improves the initial impression a customer has about the program.

A project for a financial services company revealed in early prototype UI testing that respondents were confused about the process they were supposed to go through while using the application. Apparently the more knowledgeable the respondent was about this financial activity the less problematic. However, the company developing the software wanted to reach both beginners and sophisticated users. Because of the UI test results the software developers created a simple new introduction screen which informed the user of the steps they were about to complete with brief explanation. This new addition to the program met with favorable response in subsequent testing.

Another specific area of UI testing is functionality testing where a respondent is asked to complete an assigned task. From this testing developers can determine if the program's flow matches the respondent's natural strategy in approaching a task for the application. And should the respondent make an error, valuable information can be collected by observing their recovery process, use of the Help function, and general emotional reaction to the error involved. Functionality testing can result in simple adjustments to the program, such as adding confirmation dialogue boxes, which greatly improve the program's functionality and increase users' acceptance.

Lastly, navigation testing can help determine the customer's ability to move from area to area within the program. This testing can determine if a respondent navigates easily or with difficulty, understands where they are within the program at all times, and corrects mistakes easily. For example, in a recent UI project, a travel club was designing its Web site for its customer base and also with the intention of signing on new members. To make the site visually appealing and to create an exciting site experience, the developers were experimenting with some avant-garde art images for the icon buttons and hot links. The test results showed that for two or three of these images the respondents could only guess at what the image intended to convey. With this information, new art images were constructed which illustrated the topic or function so that users could easily understand them.

The UI interview normally concludes with asking the respondent's overall reaction to the program. At this time the moderator can summarize points already made and ask for further clarification. Also general questions about probability of purchase, degree of usefulness and comparison with other software products can be explored. It's usually a good idea to ask the respondent for a bottom line message to the development team; this simple question allows the respondent to impact the design and development of software that they will potentially be using. There is no more convincing feedback for the development team than to hear it in the customer's own words.