| CPSC
681 Research Methodologies in HCI Saul Greenberg, Instructor |
 Contents |
| CPSC
681 Research Methodologies in HCI Saul Greenberg, Instructor |
Contents |
Usability Testing is a product test approach that is used to assess the usability of an interface, system or product. The emphasis in usability testing is on uncovering problems as well as verifing the product's existing usablity. It is one of the many user-centered design techniques. The user-centered design approach is based on designing products so that the users of the product will be able to complete the use, operation, service and support tasks with minimal stress and maximal efficiency. This type of approach in design of products is based on the belief that the products of use should be made to fit the user rather than the user making the fit to use the product. It is practical and economically sound that most of the major problems that the user is going to face while using the product be rectified before the product is sent out into the market place. User-centered design techniques provide designers with the information they need to solve some of these usability problems.
See also: User-Centered Design Techniques
Usability testing uses techniques, such as thinking aloud, to collect empirical data while observing and recording representative users perform representative tasks with the product. Usability testing can be performed at any level of the product’s design. This product may still be in the paper stage, or at an early prototype, or at a later near completion prototype. The goals of usability testing can be either formative or summative. According to Rubin (1994) the overall goal of usability testing is to identify and rectify usability deficiencies, along with an intent to create products that are easy to learn and to use, satisfying to use and provide high utility and functionality for the users. Thus, most usability testing with end user products is formative in nature and can be divided into four types : exploratory, assessment, validation and comparison (Rubin, 1994). These are different from the controlled experiments which employ rigorous control of variables and use strict statistical analysis techniques. Usability tests then range from product testing with rigorous controls to more informal exploratory studies. The type of test to use depends on the objective of the product test, along with the resources that are available to the experimenter.
Formative types of usability tests allow the experimenter to test an assortment of features in a relatively fast and efficient manner that is easy to set up. The study can be performed in a laboratory setting or simply on a desktop. Videotaping the process is recommended as it captures a wide range of information, both visual and auditory. When the product is tested with one individual, the participant is encouraged to think aloud during the test. In co-discovery learning, two participants interact with each other and the product. This information collected can then be applied in another iteration of the product's design, leading to another usability test. Pre and post test questionnaires, along with post test interviews, are also sources of usability information.
The tasks for each test can be a representation of the the tasks that the eventual users are going to perform. It is not recommended that the participants be immersed into testing the product with the specified tasks as soon as the test begins. It is important that the test users become familiar with the product and gain practice with it. These tasks can be designed to strengthen the participant's confidence in conducting the task and providing them with the belief that the test of the product is a participatory process that is dependent on their support. Using more formalized tasks, the usability test then allows the project team to test a product with prespective users in search of some of the more subtle flaws that may not have risen out of the previous evaluations. Instead of several isolated tasks, a complete scenario can be given to the user and the tasks can be embedded into this scenario. Scenarios make test sessions more natural and help the test users to understand the context and to concentrate on the current task.
It is recommended that the test users be a representative sample of the eventual users of the end product. This becomes important as the product is nearing the final design phase. However for pilot testing and during early design, a subset of available participants is satisfactory. During more formal testing, it is not recommended that the designers of the product be the experimenters in the usability testing due to a sometimes obvious attachment to some of the features of the product. The costs associated with each usability vary with the stage of the product's design and the number of participants chosen for the test.
The data collected via thinking aloud is analysed using protocol analysis. Quantitative data can be analysed using statistical measures. Results of the protocol analysis, problems with the product, quantitative statistics and preference information are provided in a report that also includes recommendation of improvements to the design of the product, and once these recommendations have been met then another usability test can be performed. Once a usability program has been set up within an organization it can follow a rudimentary test process, that can guide future usability tests. Specific ethical guidelines can also be developed to ensure proper treatment of participants.As the visibility and awareness of a usability test program increases in a company, it can develop as a critical process in product design, and a formal usability program can be established as in at Microsoft. This will take time and may require convincing higher level management that the usability program is effective, credible and will reduce long term costs (Rauch, Kahler and Flanagan, 1996).
How to Perform a Usability Test ?
As mentioned previously a usability test can be performed at any level of design. This is usually after an analysis of the characteristics of the end users and the tasks they are going to perform has taken place. Cognitive walkthroughs and heuristic evaluations will have hopefully identified most of the major usability problems. Usability testing then integrates the user's input into the design process.
These five stages can be used in conducting a usability test. Rubin (1994) provides extensive examples of each of these stages, along with his extra stage of Debriefing the Participant, which I have included in Conducting the Test. While I have added a sub stage of Running a Pilot Test.
1) Developing a Test Plan
2) Selecting and Acquiring Participants
and Experimenter
3) Preparing the Test Materials
3a) Running a Pilot Test
4) Conducting the Test
5) Transforming Data into Findings
and Recommendations
When to Perform Usability Tests ?
Usability tests can be performed at all levels of product design, from paper concepts to first cut prototypes to later more physically stable prototypes. It is always easier and cost effective to make any changes in the product during early rather than later phases of the design. It is therefore recommended that strucuted focus groups, market surveys, cognitive walkthroughs, heuristic evaluations and repeated usability tests be used early during the design process, all encompassing a user-centered design approach. As mentioned above, usability tests usually follow cognitive walkthroughs and heuristic evaluations. It is not necessary however, that if the design team is unable to get usability experts to perform heuristic evaluations, then a usability test is not to be performed. The major focus of user-centered design is to obtain input from the users and thus it is always better to test rather than not to test. Usability testing is most powerful and most effective when implemented as part of an iterative design process. Even if important product flaws or deficiencies are missed during one test, another testing cycle offers the opportunity to identify these problems or issues.
It is recommended that usability testing be performed rather early in the design phase with paper prototypes as modifications to the design are relatively easy to implement during early phases rather than later phases when the product is nearing completion. If such an approach is used, then the cost of the usability test is relatively low and the benefits gained from the participant comments, if implemented, can reduce some of the major usability problems with the final product.
Strengths of Usability Tests
Usability tests not only evaluate the use of the product, they also improve the process of iterative design, as designers are provided with critical usability information that can guide the design during the process, rather than at the end of the design phase. They reveal the larger usability problems for tests performed in the early phases of the design, while more fine usability issues are resolved during later iterations. Early tests can assist in modeling the structure and tasks of the product, while later tests assist in fine tuning the characteristics of the features of the product. Frequent use of a usability tests can establish a quick and easy way for designers to test the use of their product, and this forms the underlying goal of usability testing.
The usability test is not confined to test with a physical product as during early testing, a paper design of the product would be sufficient for the test. As the product takes a more regimented physical form during later phases of the design, a prototype can be used. The cost effectiveness of usability tests is in the result of flaws and problems in usability found by product users during the early stages of product design, when modifing product features is relatively easy, rather than when the product is nearing completion for when modifing product features is more difficult. Once these problems have been resolved then another usability test can be performed for another assessment of the product's usability.
Limitations of Usability Tests
Even though usability testing uncovers most of the products deficiencies in usability, it does have its limitations. Usability testing does not always guarantee success or a usable product because usability testing is performed in artificial laboratory settings and the test participants are rarely fully representative of the end user population. These limitations are accentuated if other user-centered techniques are not used in addition to usability testing.
Research Issues | Annonated Bibliography | More Usability Sites on the Net