Worst Usability Issue?
by Tom Tullis
Posted on July 4, 2011
|
Worst Usability Issue?by Tom Tullis |
|
When I recently learned that I would be receiving the Lifetime Achievement Award from the Usability Professionals Association (UPA), it caused me to reflect on my 35+ years in this field. One of the things I started wondering about was the worst usability issue I've personally encountered in my work. The one that pretty quickly came to mind is something I worked on many years ago (early 1980's) when I was at Bell Laboratories. Specifically, I was working in an area that was designing systems and equipment for telephone repair personnel to use in testing and maintaining the phone lines running from the central offices to individual homes or offices. (This was back in the days when there was such a thing as the Bell System.)
One day I was asked to review a prototype of a new device being designed for the field repair personnel to use in testing for possible high voltage on something they needed to work on. Keep in mind that telephone lines and eletrical lines are often run on the same poles, and telephone technicians are often called out during or after a major storm to repair downed phone lines where there might also be downed power lines. So a portable device that can easily test for high voltage is very important.
Here's a mockup of that original prototype:
Unfortunately, I don't have an actual photo of the original prototype. (Note to self and others: Take photos of the things you work on, whenever you can.) The probe is hand-held and vaguely resembles a large flashlight with a pointed tip. The tip is designed to break through any paint that might be on the surface to be tested. The user would depress the trigger, touch the tip to surface to be tested, and then look inside the base of the unit at the two green LEDs imbedded there. If the LED on the "trigger" side was lit that means it is safe to proceed. If the LED on the other side was lit, that means there is hazardous voltage here that could kill you.
Hmmm... Does anyone think there just might be a usability issue here? Let's see... A single green LED is being used to indicate either "OK" or "Hazardous Voltage". And keep in mind that the user can twist and rotate their hand and the probe in various ways to see inside the bottom of the unit.
So I gave my feedback outlining in no uncertain terms my concerns about the design. (The engineers did not want to use a red LED for the hazardous condition because at the time they were not sure they could find one bright enough.) The decision-makers weren't convinced yet. They believed the users would always be able to tell which green LED was which. They wanted proof that a problem existed. So I designed a quick usability test.
Brief historical side-note: At this time, these tests were not widely referred to as "usability tests" but rather "human factors tests" or something similar. The term "usability test" started being used in the mid-1980's, particularly in the context of computer systems, when Rick Spencer published the book Computer Usability Testing and Evaluation (in Jan. 1985) and Ben Shneiderman published his book Designing the User Interface: Strategies for Effective Human-Computer Interaction (in Nov. 1986).
For my test, I recruited 10 participants. They were definitely a sample of convenience, consisting mostly of co-workers at Bell Labs who were not involved in the project. The prototype we were testing didn't actually work, so the engineers rigged it such that when the tip was touched to a surface one of the LEDs would come on. They designed it so that the "OK" LED came on about 80% of the time and the "Hazardous" one about 20% of the time. (Once lit, the LED stayed on until it was reset by touching a metal stud on the side of the probe.)
I set up five different stations, outdoors, for the participants to do their tests on. Some even involved climbing on a step-ladder (to simulate working on a telephone pole). Each participant would go to a station, do the test, and announce to me what the result was ("OK" or "Hazardous Voltage"). They would then hand the probe to me, I would record which LED was lit, reset it, and hand it back to the participant. After going through the five stations once, I had them go through again. So each person did 10 tasks. The testing was done over 3 days, and the weather ranged from clear to a light rain. A few of the sessions were done late in the day, at sunset. Overall, there were 100 "participant-tasks".
Of these 100 participant-tasks, about 20 signalled the "Hazardous Voltage" condition. The key finding was that in one of those cases, the participant failed to recognize it as the "Hazardous Voltage" condition. No one ever reported the "OK" condition as the "Hazardous" condition. So technically the overall task success rate was 99%. In most usability studies, you would probably be thrilled to get an overall task success rate of 99%. But not in a case like this. That one error would have been life-threatening in the real world.
Bottom line: The decision-makers were convinced and the design was changed significantly. The two LEDs were moved from inside the base to the top of the device, and the "hazardous" LED was changed to a flashing red LED. The device, basically as it was redesigned, is still being made today. Here's a photo:
You can also see a YouTube video of one of these probes being used to detect hazardous voltage behind a flat-panel TV.
So here's a case where just one participant making one error was enough to convince people of the need to make a significant change in the design. And I think this easily qualifies as the worst usability issue I've encountered in my career.
Comments? Contact Tom@MeasuringUX.com.