Evaluating Interfaces with Users: Qualitative Methods
This section introduces qualitative methods for evaluating interfaces with users'
involvement. These form the basis for usability studies. I (and many other HCI people)
strongly believe that evaluation should occur continually through the design and
implementation process. Students should remember that these and other evaluation methods
are applied as the interface is being developed, rather than at the end. A good evaluation
process means that designers will catch major problems (and successes!) early on, with
lesser problems being ironed out as the interface is being refined.
An assignment on usability studies can
be used to complement this section by providing students with hands-on practice in various
usability study methods.
Overheads
Topics Covered
- Evaluating interfaces with Users
- Why bother?
- Natural vs experimental approaches
- Reliability and validity concerns
- Ethics
- Qualitative evaluation methods
- Introspection
- Direct observation: think aloud and co-discovery learning
- Recording results
- Query techniques: interviews and questionnaires
- Continuous evaluation: user feedback and field studies
Required Readings
- Nielsen, J. (1993) Chapter 6: Usability testing. In Usability
Engineering, p165-205, Academic Press.
- Kathleen Gomoll
& Anne Nicol (1990) Discussion of guidelines for user
observation. From User Observation: Guidelines for Apple Developers, January
[Available as HTML].
Optional Readings from Baecker Grudin Buxton and Greenberg
- Chapter 2: Design and Evaluation, p.73-91, gives an overview of the life cycle
and methodologies of design and evaluation. This chapter is really an introduction to the
rest of the course, as it touches upon themes that are covered later. The sections
directly relevant to qualitative evaluation are: a) Design and evaluation considered
together; b) Usability testing.
- How to Design Usable Systems, p.93-121, is another overview to the process of
system design, and shows how evaluation fits within many parts of the design cycle.
- Methodology Matters: Doing Research in the Behavioral and Social Sciences,
p.152-169, covers many fundamental issues in empirical research methodologies.
- Using Video in the BNR Usability Lab, p.182-185, discusses the value of
co-discovery learning and video as a means to record observations within a product
environment setting.
- Optional reading: Conducting and Analyzing a Contextual Interview, p.241-253,
introduces and describes the process of contextual interviews, an interviewing approach
that acquires information on users' work and tool use, and that can actively involve them
in co-design of the system.
Videos
Ghostbusters (commercial video, available from any video store) has a very funny
clip early on that shows an example of poor ethics. In it, two subjects are in a
parapsychology experiment, with shock treatments being administered for a subject who
guesses wrong on a card. The experimenter, however, is totally oblivious to how the
subjects are performing, and always administers shocks to the "nerdy guy"
subject (who happens to start getting it right), while encouraging the beautiful woman
subject (who gets it wrong)! Ok, not a huge educational value here, but the students like
it as a break from the normal routine!
- Ghostbusters (commercial video, available from any video store)
In-Class Teaching tips
I have found that performing usability studies in class is an excellent use of time. I
do several of them, each emphasizing slightly different methods of performing a usability
study.
- Initial conceptual model formation plus think aloud of a paper prototype. I place
a stylized picture of a Cannon fax machine on the overhead (included in the qualitative
evaluation overheads) and ask them to explain the meaning of labels and controls. I then
asked them to pretend they are sending a one page fax, and have them use think aloud to
say what controls they are selecting and why.
- Think aloud on a physical artifact. I have a student volunteer do a think-aloud
exercise as they try to display a slide on an overhead projector rigged with a burnt
bulb---it usually takes the student 20 minutes to discover the problem and realize that
the projector has a spare bulb that they can switch to. I also have them try to change the
bulb, and most have difficulties figuring out how to open the projector to reach the bulb.
The class, who are are taking notes, then critically analyze the design of the overhead
projector (relating back to Norman's design principles of everyday things) and suggest
improvements. As most recommendations are simple changes to the plastic overhead case, a
"better" projector could probably be built for the exact same price. The class
often wonders why the manufacturer (3M) had never bothered doing this simple exercise!
- Constructive interaction/co-discovery learning on an existing computer system. A
pair of students are given a PC (the display is linked to a large screen so the class can
see what is going on). One student is usually familiar with conventional GUI file systems,
while another is a novice to it. The novice is the driver, while the 'expert' is the
coach. Students are asked to do a few basic tasks i.e., one year we used Windows 95
and had them copy files around,
finding files, and so on. While the scenario is somewhat rigged to bring out the
interfaces worst features, its surprising how much difficulty students have doing even the
most basic things. I also have done this for the IBM RealPhone System.
Major sources used to prepare lecture material
- Gould's reading in Chapter 2 lays much of the foundation for evaluating interfaces
- Usability Engineering, by Jakob Nielsen, helped structure the idea of evaluating
systems with a user's involvement.
- For qualitative evaluation, the readings How to design usable systems and Discussions
and guidelines for user observation are particularly valuable.
- This book provides complete coverage: Dumas, J. and Redish, J. (1993) A
Practical Guide to Usability Testing. Ablex. I
All Topics