Be on the safe side

Editor’s note: Robert Schumacher and Gavin Lew are managing directors at User Centric, Inc., a Rolling Meadows, Ill., usability consulting firm.

An FDA study from 1985 to 1989 showed that almost 50 percent of medical device recalls came from poor product design. Congress, disturbed by this result, passed the Safe Medical Devices Act in 1990. It gave the FDA authority to mandate good manufacturing practices (GMP) that ensure proper device design. The FDA, acting on this mandate, revised and released the GMP requirements relating to the “methods used in and the facilities and controls used for designing, manufacturing, packaging, labeling, storing, installing and servicing of medical devices intended for human use” (according to the Association of Medical Diagnostics Manufacturers). The requirement goes on to state that manufacturers must ensure that design requirements address the intended use of the device, including the needs of users and patients. These new regulations became effective in June 1997.

Essentially, this is an imprimatur to user experience researchers and designers to get involved. But how? User research, notably usability testing, is ideally suited for this kind of work. The pull-through for traditional product development and product marketing is that validated ease-of-use is not only a “nice-to-have” but is actually required in the product development process.

While the obvious, highly visible research at hospitals and clinics is what comes to mind, there are less obvious, and perhaps more important, areas where testing should be done. For example, a random walk through a pharmacy yields dozens of home health diagnostic tests and over-the-counter devices. What is the incidence of error with these devices? It is probably impossible to estimate. But even opening a package of allergy medication can leave one exasperated. We can only speculate on the overwhelming frustration, if not serious harm, that occurs because packaging is very difficult to open and/or instructions can only be read by a pharmacist with an electron microscope.

Even if not mandated by government, a user-centered approach to health care products and devices is good business.

The core of the problem

The question is, why do users and patients make errors? Here are a few reasons:

  • labeling is incoherent, difficult to read or ambiguous;
  • controls are difficult to set or program;
  • documentation is incomplete, poorly written;
  • connections (from leads or between devices) are improperly made or leads are accidentally misconnected;
  • devices are not properly assembled or maintained;
  • practitioners often use devices in unintended ways;
  • interfaces designed to alert the user or sound an alarm provide poor feedback;
  • failure to anticipate possible sources of error;
  • automated features are inappropriately relied upon;
  • devices lack proper affordances (e.g., buttons do not look like buttons or lack of obvious places to grasp or push);
  • failure to account for the ambient environment or state of mind of the user;
  • lack of training - device usage should be self-evident without requiring repeated consultation of manuals.

Note that none of these reasons speak to the functionality or reliability of the device itself, yet more than half of medical device errors are due to reasons stated above.

User researchers have the tools and techniques to reduce device errors resulting from these causes. We will focus on the primary technique, usability testing, and walk through some applications or tactics to uncover errors and suggest remedies.

Usability testing

Testing products with users is a critical component to success. All too often, those who work to develop a product find themselves too close to the process to have the objective viewpoint necessary to improve the design. Robust product information is found through one-on-one usability testing. Testing involves naïve users who are recruited from the target demographics and asked to use the product. As users complete the common tasks expected of the device, the results are carefully observed.

For example, if the product is a self-monitoring device, tell the user the situation (e.g., it is time to check your blood sugar level) and hand them the box. Watch what they do to complete the task. The emphasis should be to observe behaviors. Do not engage in discussions until the user has worked through the task on their own. Use a “think aloud” protocol where the user verbalizes their thoughts while using the device.

Usability testing should be both pragmatic and iterative. The goal is to identify potential usability issues that include dimensions like ease-of-use, error or failure, comprehension of instructions, learnability and satisfaction. With even a few users, major problems rise up and are identified very quickly. The general rule of thumb is to test six to eight users for every target user group. Thus, testing can be completed in days not weeks.

Results should be evaluated for potential design changes, and perhaps, depending on the nature or number of errors and their corrections, another round of usability testing is required. This iterative process ensures that problems are resolved before the product is launched. Performing comparative or head-to-head usability tests can also provide valuable data on features, functions and usability differences across products and your competition. These lessons can be exploited for future product development or marketing.

The following are good candidates for usability tests:

Device: Test to ensure that both first-time and experienced users can operate the device effectively and efficiently. Assess affordances of device controls, such as “Does this look like a button or switch?” Note errors and how users recover from errors. Ensure that feedback from the device provides sufficient information that users know when the process is complete and successful. In one usability test, we found that users often held the device in a manner inconsistent with intended use, one that could have resulted in serious injury and/or expensive product replacement. Affordances for proper handling were not obvious. A second test showed that users failed to recognize a device’s handling cues (e.g., hand grips). Yet this turned out to be a benefit — the device was more usable because of its flexibility with regard to handling. So rather than strengthening the handling cues, the lack of a strong cue enabled us to understand how flexible the device truly needed to be.

Labels and instructions: Test these elements early to remove jargon. Terms that are heard everyday in product development or marketing are often gibberish to your users. In many cases, these documents and terms are left until just before launch and the only review occurs with the legal department. Why do we wonder that user guides are never read? Labels and icons are critical and should be tested objectively. In another usability study, we found that device labels were not as obvious to users as product developers thought they would be. The result was operation failure and a hazardous condition where the patient placed himself in danger.

Longitudinal experience: Let the device “soak” for a period of time and assess the user experience over time. Often, initial hurdles can be overcome and novel practices and techniques arise. Moreover, letting the device become part of a user’s “routine” over time may show different uses and potential problems due to interactions with other devices or device errors. In our practice, we have found future product enhancements in interviews during longitudinal testing. In another study, we found intermittent device failure that went undetected in production testing and most likely would have resulted in high churn and poor customer satisfaction.

Some problems do not present themselves in a test setting but do appear in the ambient environment. For example, consider a medical device in an ICU that has an alarm that sounds at 1000Hz. There may be other devices that have alarms at the same frequency. Moreover, testing with users in close-to-typical environments may uncover the need for multimodal cues (e.g., alarms trigger lights and tones).

“Out-of-the-box” experience: Test the product as it will be given to the user. The user experience involves more than just the device. Exterior and interior packaging are prime candidates for testing (i.e., opening the box and removing the device). This is especially important for users who may be elderly, have impaired dexterity (e.g., rheumatoid arthritis patients), vision problems or find themselves using the device in less than optimal environments (e.g., low-light conditions). These usability issues are not easily uncovered during product design meetings, but they surface immediately to users. In one study, nurses turned off the lights in the room to simulate how they would prefer to use of the device to avoid disrupting sleeping patients.

Testing with prototypes

Modification and flexibility are essential to achieve good design. Recognize that the product need not be 100 percent functional to learn critical usability lessons. Test with prototypes. Test with foam mock-ups. Test with paper. Be pragmatic, but test with users. Obtain feedback to make design changes at the beginning of the development cycle - before too much programming or die molding have occurred. It is better and cheaper to identify a design problem early, rather than late, in the design process. For example, we have had success using touch-screens laid on a bed to simulate the use of a handheld device on a patient. We found that testing instructions and icons in draft form provide robust information that made the product easier and more intuitive to use.

Serve the need

For technology or products (or even packaging) to be successful both in patient care and in the marketplace, the devices have to serve the intended need in functionality and in usability. While this point may seem obvious, as pointed out above, multiple research studies show that user error still accounts for more than half of all serious medical device failures.

User research is a growing need in health care as more emphasis is placed on technology and, culturally, more care is designed to be in the home. The burden of learning a plethora of new devices in the milieu of a clinic or doctor’s office is one that, for patient and care-giver safety, must be lessened by increased reliance on user research.