When evaluators perform a heuristic evaluation, they can either step through the interface using representative end user tasks (i.e. scenarios) or self-guided exploration. Both methods have their advantages.
Scenarios are intended to represent how the system is intended to be used by the end-users. They list the various steps a user would take to perform a few realistic tasks. Scenarios are constructed on the basis of a task analysis, involving consultation with end users and observation of these users in their work setting.
The main advantage of scenarios is to help supply evaluators, with little domain expertise, the knowledge needed to operate the system [6,12]. Plus, the use of well-established scenarios can provide some assurance that real world problems will be identified [11]. Scenarios can also be constructed to ensure that specific features of interest to the UI specialists are evaluated. This is especially useful if certain areas of a system require more attention than others. However, limiting the scope of the evaluation to specific features in an interface causes the evaluators range of exploration to be reduced. This in turn could cause potential usability problems to be missed. Scenarios also have the disadvantage in that real users will not necessarily be given explicit task scenarios.
Evaluators who are not given scenarios to aid in the evaluation of a user interface will use self-exploration. This usually involves the evaluators generating their own goals with meaningful tasks to achieve them [6]. This depends on how much the inspectors are encouraged to explore and how task-oriented they are [6]. This freedom allows the evaluators more opportunities to explore more diverse aspects of the interface. They are not limited to the interface features that are focused on by the scenarios.
Karat et al. [5] provide a comparison between performing walkthroughs using prescribed scenarios and those using self-guided exploration. Walkthroughs in this study consisted of evaluators utilizing 12 usability guidelines (8 of which come from Nielsen's original set of heuristics) to find usability problems. Essentially, the evaluators performed a heuristic evaluation on the system. All walkthrough groups favored the use of scenarios over self-guided exploration in identifying usability problems.
Ultimately, there is a trade-off between ensuring that evaluators inspect an interface in terms of a specific focus versus allowing more freedom in the exploration.