Heuristic evaluation was originally developed as a usability engineering method for evaluators who had some knowledge of usability principles, but were not necessarily usability experts. However, subsequent research has shown the method is more effective when the evaluators are usability experts. When evaluators, with a range of expertise, identified problems in an interface, Nielsen [11] found the following results:
As can be seen, the performance of novice evaluators was fairly poor relative to the other types of inspectors. Obviously, it is preferable to use usability specialists as evaluators and even better would be to use double specialists. However, as the expertise of the evaluators increases so does the cost of performing a heuristic evaluation. It is advantageous from a discount usability engineering (i.e. cost) perspective to perform heuristic evaluations with people having little or no usability expertise. Usability experts are also sometimes hard and expensive to come by, especially if they also must have expertise in a particular kind of application. Given the choice between lower costs or increased productivity, Nielsen recommends using the costlier UI specialists as a minimum since it is believed that heuristic evaluation draws much of its strength from the skilled UI professionals who use them [4]. It is thought that the importance of these peoples' knowledge and experience cannot be underestimated [4]. Realistically, trade-offs must be made between the cost of not finding usability problems and the cost of using expert evaluators to perform a heuristic evaluation.
Nielsen [11] also noticed that double specialists found more user interface problems than regular specialist because they were able to identify those problems which were specifically domain related. Double specialists are not necessarily better than usability specialists, they just have more specific experience with the usability issues for the kind of user interface that is being evaluated.
Nielsen [13] does not recommend performing heuristic evaluations with users as evaluators because of their poor understanding of usability principles. It is much better to use available users for user testing. They are normally harder to find, more difficult to schedule in large numbers and and are not very good at analyzing systems according to abstract principles. In contrast, users are perfectly suited to using the system, and one can then observe what actually happens instead of asking the users to guess what might happen.