The Cognitive Walkthrough Inspection Method:  An Overview

Jason Laberge, Department of Psychology, University of Calgary

    Interface evaluation has become increasingly important in our modern world as technology revolutionizes the way we do things.  However, with a surge of new products, technologies, and improved designs there is a multitude of questions that must be addressed.  In addition to the ever debatable question of whether we are advancing too quickly is the idea that in light of all the new developments, how many are actually usable for us.  This introduces the idea of evaluating these technological revelations, and more specifically the computer interface.  Since a great deal of our time is spent using computers and interacting with them in all settings, it seems logical to begin the evaluation at this point.  The interfaces are varied in both their design, goals, usability, and learnability.  Therefore there exists a need for an evaluation technique that can help distinguish a good interface from a poorer one.  Unfortunately the inherent variability in these interfaces makes it difficult to create one evaluation technique that can be applied universally across all settings.  The result has been a multitude of techniques that are best suited for different applications and interface types (see Nielsen & Mack, 1994 for an overview).  These techniques have been separated into either formal usability testing of usability inspection methods.  The later focuses on individual judgments as a source of evaluative feedback and de-emphasizes the need for highly controlled and empirically valid methodologies.  One such inspection technique is the cognitive walkthrough (CW) and will be the focus of this paper.  The reason this technique has been chosen is twofold; first although this method is thorough, it is often misunderstood in its application and second by understanding what the strengths and weaknesses are of this method, it can be better applied to interface evaluation situations that warrant its use.

OVERVIEW

Brief Description of the Method

    The cognitive walkthrough is a user interface (UI) technique that examines how certain aspects of design may impair or enhance the learnability for the user.  How a system is learned is thought to occur primarily by exploration (Polson, Lewis, Rieman, & Wharton, 1992).  The rationale behind this approach is that many observations have been made showing that users typically prefer to learn by exploration rather than by formal training or using manuals (Carrol & Rosson, 1987).  Therefore as one develops their knowledge base of a system the ease of learning is governed to a large part by the immediate benefit to the user.  The aim of the CW is to understand how the design characteristics influence the creation of goals and lead to the generation of action.  This is accomplished by evaluating the tasks that are needed to use the interface and determine how these tasks increase or decrease the likelihood of the correct action being performed.  The goal is to identify problematic tasks and task sequences that can later be re-designed in order to make the system more learnable and subsequently more usable.  What the CW is looking for is those tasks that may break the goal-action stream and cause difficulties in learning the system.
 
    The CW process often involves a small group of analysts (although it can be done by ones person), is very formal, and involves both preparation and evaluation phases.  The CW is an inspection method whereby the designer presents a design to the CW team and they evaluate it based on the appropriate cognitive processes that are needed to perform the given tasks.  This evaluation is done using criteria that are appropriate to the system design and its issues.  Using the information from a CW, an evaluator is in a better position to identify usability/learnability problems and attribute potential causes for these problems.  Once presented with these findings, the design team is in a better position to re-design the product and make it more usable and learnable.

    As a basis for determining the usability problems, the CW uses current cognitive theory to examine particular tasks that may be problematic in the design.  This theory is comprised of understanding how a user makes decisions as they navigate through the system and perform various tasks.  The method uses standard CW forms as a template for discovering the problematic tasks and task sequences.  The CW team then interprets the results obtained from the forms and presents the key findings to the designers (who may or may not be part of the CW team).  This approach is useful and important because it provides for a basis of understanding how a user learns a system and helps identify design features that both help and hinder the ease of learning for a system.  The CW team itself is typically composed of actual designers or implementors of the software and may include a usability specialist (although not necessary).  This team becomes the analysts for the CW being tested and are the individuals who interpret and present the results of the inspection.

Brief Description of the Theory

    The importance of examining how easily a user learns a system is based on the idea that if a system is difficult to learn, then its usability will be greatly compromised. When a system is difficult to learn, it will be difficult to use and will require a great deal of experience to accomplish the desired tasks.  This can ultimately lead to increased costs for training and frustration while decreasing satisfaction.  Since the ultimate goal for any interface is to allow users to obtain the skills necessary to use the system and complete the necessary tasks as quickly and efficiently as possible, these learnability issues are extremely important.  However, theories of skill acquisition (Anderson, 1987) identify that learning by exploration helps to facilitate skill acquisition.   As a result, having a system that is easy to learn allows a user to more easily learn the necessary skills to use the interface and become highly productive as soon as possible.

    The building blocks of the CW come from Norman’s (1986, 1988) theory of action that specifies a series of stages from the initial goal, generation of action plans, execution of the action, evaluation of the action, and goal revision.  The CW also incorporates the construction-integration model developed by Kintsch (1988) that describes the way users integrate a representation of external stimulus with their background knowledge to construct a representation  that facilitates the accomplishment of a task.  The CW incorporates both these ideas into the theory such that it examines how various tasks and interface characteristics influence the development goals, subgoals, and the execution of action while interacting with the user's current background knowledge to accomplish desired tasks.

Refer to figure 1 for an outline of the theory for the CW.

THEORY IN DETAIL

    The CW uses a problem solving process as described by Polson and Lewis (1990) to inspect the user interface (UI).  This model identifies how users first start with a crude description of the task they want to accomplish, second the user explores the interface and selects actions they link with the accomplishment of the desired goal, third they use the interface feedback to evaluate their actions, and finally determine which action to take next.  This process is based on the cognitive model composed of viewing the user as having cognitive representations of goals in their goal structure. These goals are linked to other goals, background knowledge, representations of objects in the environment and to actions.  However, for an action to be executed, it must be sufficiently activated. This is accomplished by having a clear and uninterrupted link between the goal and the execution of action.  Responses by the interface are interpreted and used to evaluate the action and the user then makes a decision as to whether the current goal needs to be deactivated, another developed, or whether the same goal requires a different action for successful completion.  Each link that is created indicates successes and failures in the achievement of goals and is integrated into an ever developing network of goals..  New goals must go through the increasingly complex network before the proper action will be sufficiently activated.  This illustrates why simple tasks are easier and quicker to accomplish than more complex ones.

    The CW attempts to identify those areas of the interface that would interrupt or break the path between the goal and the action.  However, in most designs it can be assumed that the user has an incomplete goal structure and a rather simplistic mental model of the UI.   The user creates fragments of their goal structure as they interact with the interface and try to accomplish the tasks required. The task itself is very important for understanding the theoretical basis for the CW.  When completed by the execution of the appropriate action, those tasks that are simple and physical achieve a subgoal.  A sequence of subgoals results in the achievement of more complex and higher goals. Higher goals represent task sequences and can vary in complexity depending on the nature of the tasks required.

    The transition from one subgoal to another and ultimately to the completion of the overall goal is though to occur via "and-then" structures.  These structures connect subgoals and indicate when one subgoal has been completed and the user can proceed to the next one in the goal structure.  The higher level goal is accomplished by completing the first subgoal, and then the second, and then the third.  This process if continued until the higher goal is completed and a new higher goal is generated.  The CW focuses on these steps as fundamental units of analysis.  Specifically it examines the goals required before the execution of action, whether these goals, given the interface, will lead to the correct action, and how the interface feedback influences the user's goals.  When the early subgoals resemble the higher level goal, and when the subgoal "and-then" structure is activated, a connection may be made that causes the user to think they have completed the higher overall goal.  This results in the deactivation of the original goal when only the first subgoal has been accomplished.   This phenomenon is called "supergoal kill off" and it is imperative that the CW identifies the tasks whose goals are subject to this problem.

Refer to figure 2 for an overview of the cognitive problem solving model.
 
  The generation of new goals are driven almost exclusively by evaluating the action based on the interface feedback.  This feedback is fundamental for the development of the goal structure since the user uses the feedback as a basis for determining when goals have been successfully completed, when they must be deactivated, and when new goals must be added to the structure.   The user must interpret the system response or feedback and determine the state of their goal structure.  It is the feedback that hinders the development of new goals that the CW also tries to identify.   The CW then can determine which prior knowledge the user must have in order to develop both the appropriate goals and create subsequent subgoals that will result in the execution of the appropriate action.  As the interaction between the user and the interface continues, the features of the interface interact with the user's background and existing goals within their network to create more explicit goals which compose the complete task sequence.  This theory is extended to every task in the interface and the results obtained are interpreted  and discussed by the design team.  These results consist of prioritized problems and are presented to the designers for the purpose of redesign or other means of interface improvement.

THE COGNITIVE WALKTHROUGH PROCESS

    The CW process involves two different phases.  The first phase is the preparation phase and focuses on having preliminary information ready in order to conduct the actual walkthrough.  The most important part of this first phase and subsequently the entire process is the development of the tasks that are going to be inspected.  Without suitable tasks to analyze, the CW would provide little useful information that could be used to improve the design of the UI.  The development of the task list is most easily accomplished using five steps; choosing the tasks to analyze, providing task descriptions, determining the correct sequence of tasks, identifying anticipated user population, and describing the user's initial goals. The first phase is described in more detail below.

Phase 1:  Preparation
 
    In order to choose the tasks to analyze, one should resort to the designers as they are typically the most resourceful source of  information regarding the UI.  Presumably they designed the interface knowing what the users would be trying to accomplish.  This approach typically leads to a large list of tasks (depending on the interface of course) and the next step is to determine which tasks are the most appropriate for testing.  This is usually a collaborative effort by the CW team as they must determine which tasks are the most important and may lead to the emergence of important design issues.  The team should pay special attention to those tasks who have a collection of subgoals associated with them that have to be executed in the right order.  Focusing on these types of tasks forces the CW to identify where in the goal structure there may be problems between selecting the proper goal and executing the appropriate action.

    The next step in the development of a testable task list is to provide a task description of those tasks that have been chosen by the design team.  This is where a walkthrough form can help to describe the task from a first time user's perspective using realistic terms and few technical terms.  This step is very important for identifying any assumptions that are being made about either the users or the state of the system.  Any background knowledge that is required by the users must also be noted for each task.  This helps during the later phases of the CW when helping to understand why particular tasks may be problematic.  For more complex tasks, this step must be sure to include the task decomposition for each subgoal.

    In order to determine the correct sequence of tasks, again one should resort to the designers as they have presumably identified the path which the interface was designed to support for any given task.  It is this sequence of subgoals, goals, actions, and tasks that the CW critiques and therefore this sequence must be the best offered by the interface so that the team can be sure they are criticizing the interface design rather than a poorly defined task sequence.  The best way to approach this step is to think about the actions as being very finely defined and then collapse them into larger units or tasks only when it is clear that there will be no problems that will be embedded within the sequence itself.  This is important so that the CW can identify all design problems and is not being limited from discovering potential problems by the task sequence.  This step is also very important for identifying what interface feedback will be provided at each task and how this might change throughout the sequence.

    The next step is identifying the user population and making detailed descriptions about what they will be doing with the interface.  This includes what goals they will form, what actions they will find difficult, and what changes in the interface will be regarded as appropriate given the tasks.  The most important aspect of this step is to make clear what the anticipated user population's background knowledge is and to focus on their relevant experience.  This information is absolutely critical for the CW as it bases most of its analysis on what background knowledge the users have and how this influences the development of their goal structure.  It is important for the team to identify a relatively uniform population in terms of background so that the influence of individual differences is minimized when performing the CW.  This ensures that the team doesn't find themselves performing different analyses because they are dealing with different subgroups within the overall population. Again this step is most easily accomplished using a user profile form to identify the important information for analysis (see Form 1 below).

    The final step in the first phase of the CW is describing the user's initial goals.  This step is important for starting the CW and determining how the initial tasks will affect the development of the goal structure.  The analyst must also note those initial goals that will likely be generated from the initial tasks so that  they can set up and conduct the rest of the analysis.  In this step it is important to focus on those goals that the user is likely to have and not the ones they should have.  Following the later would invalidate the process as the analysis would be based on an improper initial goal structure. An outline of Form1 is provided and includes space for descriptions of the task, action sequence, anticipated users, and the user's initial goals.

Form 1:  Cognitive Walkthrough Preparation Form

The above form is completed for each task in the task sequence.

Phase 2:  Evaluation

    The second phase of the CW is the evaluation phase.   Now that all the information has been provided in phase one, the actual CW inspection can begin.  In this phase, the analysis is done by looking at each of the required actions that have been identified in the preparation phase and looks for problems in the development of the goal structure within the exploration process model.  At each step, the analysts make predictions based on the previous actions of the percentage of users that are likely to have problems in developing the appropriate goals and execute the right actions.  This phase is driven almost exclusively by CW forms.  There are separate forms for each step of the task and attempts to identify the implicit and explicit assumptions made by the developers and designers of the system.  This phase of the CW consists of examining each action in the task sequence and attempting to determine why the expected user populations would choose to perform the actions (whether appropriate or inappropriate).
 
    The purpose of the forms is to identify which problems a user may have with regards to the goals and actions they must develop.  The use of forms provides for a more standardized way of testing the task sequence and provides for both easier comparisons and the identification of interface problems.  The forms help address three main issues with regards to the UI evaluation; they are:  goals needed and user's prior knowledge and existing goals, problems selecting actions given proper goals, and how goals will likely change following actions and feedback  An example template is provided for each form as well as a detailed explanation of each step in the evaluation.

Form 2:  Goals Needed, User's Prior Knowledge and Existing Goals
Form 3:  Problems Selecting Actions Given Proper Goals
Form 4:  How Goals will Likely Change Following Actions and Feedback

These forms are completed for each action of a task and the aggregation of interpretations fro each form allows the analysts to identify the important interface problems with regards to each task.

IMPORTANT QUESTIONS FOR DESIGNERS AND MANAGERS

1.  How does the cognitive walkthrough compare to other walkthroughs?
2.  What is the turnaround time for a CW process?
3.  How cost effective is the CW?
4.  What degree of external and internal validity does the CW possess?
5.   What degree of qualitative and quantitative information can the CW provide?
6.  What kind of equipment is needed for a CW?
7.  What kind of personnel is needed to perform a CW?
8.  Where in the design process does the CW fit?
9.  How does the CW compare to other methods of interface evaluation?

 A WORKING EXAMPLE

One from the literature

STRENGTHS AND WEAKNESSES OF THE COGNITIVE WALKTHROUGH
 
Strengths

#1    Standardized and consistent
The forms and underlying theory provide for consistent inspection of all the tasks required for an interface.

#2    Trained UI specialist is not needed
Although trained specialists are not required, they are an asset (see Question #7).

#3    No special equipment is needed
The CW requires no special equipment to be applied effectively (see Question #6).

#4    Very detailed information
The information provided by the CW is very detailed and allows for in depth discussion, interpretation, and feedback.  By focusing on a fundamental unit of analysis, the CW team can be sure to provide information that describes the heart of the problems.

#5    Numerous variations exist
Variations of the the CW include a jog through approach and an automated version (see Question #2).

#6    The CW is applicable anywhere in the design process
Although more suitable for application in the early design phases, the CW can be applied to any interface.  The interface developmental stage could potentially range from a paper prototype to a fully implemented system (see Question #8).

#7    Cost
The CW can be less costly than either formal usability testing or other inspection techniques,  this depends on numerous factors  (see Question #3).

Weaknesses

#1    Validity Issues
The CW exhibits questionable internal and external validity problems (see Question #4).

#2    Timely
When done formally, the CW is very time consuming.  This can be overcome by introducing variations of the CW as previously mentioned (see Question #2).

#3    Limited problem identification
Compared to other methods, the CW identifies fewer usability problems.  The information that is provided may be more suitable for system design rather than interface evaluation (see Question #8 and Question #9).

#4    Training required
The CW requires a in depth understanding of both the process and the theory, therefore those individuals who are not UI specialists may require training on the method (revisit cost issues above).

 RESOURCES FOR MORE INFORMATION

Online
Literature Reference List

CONCLUSIONS

    In contrast to other inspection methods, the CW is useful at all stages of interface development (Nielsen and Mack, 1994).  Clearly the CW is useful for identifying interface problems with regards to the development and execution of action by the user.  However, the CW process can be quite long and requires extensive knowledge of the underlying theory.  Therefore, the use of the CW should be reserved for situations where the designers are seeking to know how the system allows the user to make decisions and execute actions rather than for identifying recurring usability problems.  The use of goals, actions, and task breakdowns has the advantage of considerable face validity for addressing UI problems.  The CW is therefore useful from a theoretical perspective and lends itself to deep and thorough inspection of an interface.  However, the CW is not limited to the formal and standardized method that is described in detail here, but rather is flexible and adaptive.  This requires the analysts to determine which inspection method may be the most useful given their particular situation.  Therefore, depending on specific interface goals and inspection objectives, the use of the CW can be more or less appropriate.

Created Fall, 98, by Jason Laberge, as part of the course requirements for CPSC 681.