Reliability : Reliable results in usability testing are obtained again and again when the test is repeated with different users. Reliability is a problem with usability tests due to large individual differences between users. This is accentuated as usability tests do not exercise the same degree of control as controlled experiments. The best user is sometimes as much as 10 times faster than the slow user, and the fastest 25 % of users are twice as fast as the slowest 25 % of users.
Even though the speeded task test results are not highly reliable with usability testing, usability testing does have its benefits. In most cases, it is the only mechanism available to test the product with representative users. If the focus is more on finding usability problems and develop recommendations for product improvement in iterative design then usability testing is the method to use. Approximately 80 percent of the usability problems in a product can be detected with 6 test users per test grouping. Even though usability tests are not highly reliable in terms of speed tasks due to high participant variablity, they do provide some critical information for product design.
Validity : Valid results in usability testing are obtained if a particular test is testing what it is supposed to be testing. Highly valid usability tests are a result of test goals, techniques and measures that correspond with representative users and tasks. Typical validity problems involve using the wrong users or giving them the wrong tasks. For example, a tourism travel booking system might be tested with chemistry students as test users, but it is likely that the results would have been different with real tourism students.
Generalizability of Results : As each product, in each domain, is constrained in use by its own tasks and users, results from one usability testing with one product are not generalizable to another product. As the usability test is focused on testing the product, generalizability would be in the form of between one product to another.