These types correspond with formative/summative goals of usabiliy testing. The are usually used sequentially as the product is developing, with exploratory studies being performed during earlier phases, leading to assessment studies and validation studies towards the end. Comparison studies between product features can be performed at most levels of product design, but do not involve the more strict control as controlled experiments. Most of these test studies can be set-up rather cheaply and simply or can be performed in more formalized laboratory settings.
These tests are conducted after user profiles and task analyses have taken place and the design team is formulating the functional specification and early models of the product. The objective of such a test is to explore the effectiveness of preliminary design features and concepts. This high-level test of the product, requires exploration of the cognitive thought process of the test user and a high degree of thinking aloud in encouraged. To this end, the test experimenter interacts extensively with the test user via guiding and exploring the conceptual features of the product. This interaction is quite informal and the participant tests the product in a proactive participatory manner to create a collaboration between the test experimenter and the test user. The physical design of the product is about to begin and the design teams needs confirmation on the primary functions of the product. Test users are asked for their ideas on how to improve confusing areas, and the test experimenter strives to understand why the user performs as he or she does.
Assessment type tests are the most common usability tests as they are the most simple and straightforward to do. These tests are performed after some of the higher level issues surrounding the product have been expounded by exploratory tests and incorporated into the product by the design team. The purpose then is to expand exploratory test findings by evaluating the lower level operations and aspects of the product. These tests then examine how effectively the concepts from the exploratory testing have been implemented into the product. For this examination, the participant is asked to complete actual tasks with thinking aloud, and the focus is on gathering information. The test process in the How To Perform A Usability Test on the main page is based on an assessment usability test.
For validation tests, the focus is on how well the product's usability compares to standarized usability performance measures. These measures can be prespecified internally or set up as part of the product planning process. This test is usually performed before the test is ready for release to the market. If a history of such tests has not developed within a company, then such a test can establish a benchmark comparison for future products. The test can also be used to evaluate how the many features of the product work together. In most cases, this would be the first time when the product, documentation and help services are available for test at the same time. There is a greater focus on experimental rigor and quantitative information, than in assessment testing, as the product's usability must pass some prespecified standard. These tests should be performed as early as possible, to allow time for the design team to modify some of the features that did not perform well.
Comparison tests can be used at any level of product design in conjunction with the above tests. It is used to compare two or more designs of the product in terms of which is more usable. It involves collection of both performance and preference data, and the participant is asked to perform similar tasks with both designs. The use of this technique forces the design team to consider alternatives in design, and this process can sometimes lead to ingenious design features.
An example of a usability test that employs both a comparison
test as well as an assessment test is the Evaluation
of the Effects of Representation Type and Transparency Level on
Overviews in Carl's PipeLine System (CPLS).