Are heuristic evaluations more effective when conducted individually or in teams?
Nielsen recommends performing heuristic evaluation by having each individual evaluator inspect the interface alone [12,13]. This allows the individual evaluator the opportunity to form his/her own evaluation about an interface and without being influenced by others. This ensures results are independent and unbiased from each evaluator with greater variability in the kinds of errors found. Plus, with single evaluators there is no overhead required to organize group meetings. Only after all evaluations have been completed are the evaluators allowed to communicate and have their findings aggregated.
However, others have suggested using multiple evaluators to look at the interface together because it results in finding more problems and suggesting better solutions than if the evaluators work alone. People can reinforce each other and help each other to notice problems that may go unnoticed individually. Also, multiple evaluators helps to filter out problem predictions that are not plausible end-user problems [7]. One recommendation is to use a maximum of 2 evaluators during a heuristic walkthrough [15]. It is found that more than two causes a bottleneck with the observer taking notes with the risk of the evaluators becoming bored.
Karat et al [5] compared walkthroughs (i.e. heuristic evaluations) performed individually against those done in pairs. It was found that team walkthroughs achieved better results than individual walkthroughs in some areas. They identified more usability problems; however, the total number of "significant problem areas" was consistent across the two types of walkthroughs.
The trade-offs between individual and group heuristic evaluations are time and the cost of not finding usability problems.