Strictly controlled behavioral experiments usually require extensive theoretical considerations, apriori statements on independent and dependent variables, a control comparison group, a means to control confounding variables, and a concervative statistical base for conclusions. They also require extensive literature reviews and explanations on how the conclusions of the study either agree or disagree with previous literature. In controlled experiments the test is performed with a manipulation of a select number of variables, a strict dependent measure and an apriori decision on the hypothesis. The focus is on quantitative proof in support of one alternative or the other, and a theoretical explanation as to why the data supports the theory or refutes it. Controlled experiments require a high number of subjects to attain proper power in the results and conclusions. This is especially true for between subjects designs. The researcher experimenter requires extensive training on conducting such experiments, learning how to control the confounding variables, developing ecologically sound tasks, and asking the right question. Controlled experiments can be used for testing the usability of products, but this process would require rigorous control and adherence to the above mentioned criteria. It is not possible for an experimenter, in controlled experiments, to modify the variables midway if they seem to be showing interesting trends, due to the rigidity of the technique as well as ethical concerns regarding fishing in the data. A different experiment would be required for such a change. Each controlled experiment is also usually restricted to a single set of variables with corresponding dependent measures.
Most usability tests are unlike controlled experiments as they do not entail rigorous control in the comparison of two or more conditions of a variable. Usability testing is more of an assessment of the usability of a product as it undergoes iterative design. The focus is also on qualitative information on how to fix problems and redesign products. During such testing the product designers can manipulate the design of the product to allow them to look at the features that encourage and/or discourage usability. Participants can test several tasks with the product, that might not be related, leading to a within subject design that is more cost effective than between subjects. The product itself can be changed around as the test is taking place. If pilot testing reveals that certain problems are going to be prevelant for all participants, then the product can be modified midway. Unlike controlled experiments, usability tests can be performed for features that are unrelated to each other but support the overall usability. This allows the usability test to perform several controlled experiments in one. A completely different usability test that needs to undergo another set of ethical approval is not required. This aspect of usability testing allows it to be performed on several iterations of a product design, fairly inexpensively and within short time frames. Even though speed and accuracy data are useful in usability testing, information that entails high utility is primarily in the form of errors made, difficulties risen in product use and subjective preference measures. Differences and difficulties in usability tests are measured on severity with the product being tested in terms of ease and efficiency of use. With usability tests such analysis are made in terms of practical signficance rather than statistical significance.