CPSC 681 Research Methodologies in HCI Saul Greenberg, Instructor

Contents

Tentative Reading List

This list will change! All references are required reading except those marked with a *.

General discussions of methodologies

Baecker, R., Grudin, J., Buxton W. and Greenberg, S. (eds) Extract: Chapter 2: Design and evaluation. In Readings in Human Computer Interaction: Towards the Year 2000, p73-91, Morgan-Kaufmann.
This overview summarizes the fit between design and evaluation.

Gould, J.D. (1996) How to design usable systems. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p93-121, Morgan-Kaufmann.
This delightful paper, originally written in 1988, presents many informal methods and how they are applied to the usability design process.

Dix, A., Finlay, J., Abowd, G. and Beale, R. (1993) Extract-Chapter 11: Evaluation Techniques. Human Computer Interaction, p363-400, Prentice Hall.
This chapter briefly surveys a variety of methodologies, and is a useful overview.

McGrath, J. (1996) Methodology matters: Doing research in the behavioural and social sciences. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p152-169, Morgan-Kaufmann.
A general discussion and comparison of fundamental concepts in evaluation methods.

Card, S. (1996) Pioneers and settlers: Methods used in successful user interface design. In M. Rudisill, C. Lewis, P. Polson and T. McKay (eds) Human-Computer Interface Design: Success Stories, Emerging Methods, and Real-World Context, p122-169, Morgan-Kaufmann.
Successful systems are reconsidered against a variety of design and evaluation methods as well as real deployment requirements.

Ethics

APA (1992) Ethical Principles of Psychologists and Code of Conduct. American Psychologist. December, p1597-1611
The sections dealing with experimentation are relevant for this course.

Ethical Review Form (Human subjects), Department of Computer Science.

Rogers, T. (1996) Research Participant's Bill of Rights. (based on the Canadian Psychological Association's Code of Ethics for Psychologists. This version was adapted to be used on the Dept of Computer Science web site. Note: Rogers has an updated version of all this: see the Department of Psychology Ethics Page

Nielsen, J. (1993) Extract-Chapter 6.4: Ethical aspects of tests with human subjects. In Usability Engineering, p181-185, Academic Press. I have a few slides based on this.

User Observation

Experimental methodologies

Eberts, R.E. (1994) Extracts from User Interface Design, Prentice Hall.
A general and high level introduction to experimental design. Extracts are:
Chapter 4: Experimental methodology;
Chapter 5: Experimental designs and analysis;
Chapter 6: Hazards to conducting and interpreting HCI experiments

Lecture Notes:
o Quantitative methods: [Adobe PDF]
o Analysis of Variance [Adobe PDF]

*Greenberg, S. and Witten, I. (1984) Comparison of menu displays for ordered lists. In Proc Canadian Information Processing Society National Conference, Calgary, Alberta, May
This paper is an example of how statistical methods (Anova) can be applied to comparing particular system features and how results are interpreted.

Usability Testing

Nielsen, J. (1993) Extract-Chapter 6: Usability testing. In Usability Engineering, p165-205, Academic Press.

Dumas, J.S. and Redish, J.C. (1993) A Practical Guide to Usability Testing. Various chapters:

• 2: Introducing usability testing
• 6: Establishing a usability program in your organization
• 7: Planning a usability test
• 9: Deciding who should be participants
• 11: Selecting and organizing tasks to test
• 12: Creating task scenarios
• 15: Preparing the testing environment
• 18: Caring for the test participants
• 19: Conducting the test
• Appendix A: Setting up a usability laboratory

Gomoll, K and Nicol, A. (1990) Discussion of guidelines for user observation. From User Observation: Guidelines for Apple Developers, January, 1990. [HTML].
This reading is an excellent recipe for performing usability studies.

Ehrlich, K., Butler, M. and Pernce, K. (1994) Getting the whole team into usability testing. IEEE Software, p89-90.

O'Malley, C., Draper, S. and Riley, M. (1984) Constructive interaction: A method for studying user-computer-user interaction. From Proceedings of Interact '84, p1-5.

Neal, L. (1989) The use of video in empirical research. SIGCHI Bulletin, 21(2), p100-101, ACM Press.

Kennedy, S. Using video in the BNR usability lab, SIGCHI Bulletin, 21(2), p92-95, ACM Press.

Harrison, B. and Baecker, R. (1992) Designing video annotation and analysis systems, Graphics Interface, p157-166, Morgan-Kaufmann.

Interface Inspection

Overview

Mack, R. and Nielsen, J. (1994) Chapter 1: Executive summary. In J. Nielsen and R. Mack (eds) Usability Inspection Methods, p1-23, Wiley and Sons.
An exectuive summary and discussion of inspection methods.

Usability Heuristics and Heuristic Evaluation

Molich, R. and Nielsen, J. (1990) Improving a human-computer dialogue, Communications of the ACM 33(3), March, p338-348, ACM Press.
This article lists the heuristics and presents a working example, solutions, and alternatives.

Nielsen, J. (1993) Extract-Chapter 5: Usability heuristics. In Usability Engineering, p115-163, Academic Press.
Describes the heuristics in detail plus how and why it can be used to evaluate interfaces.

Nielsen, J. (1994) Enhancing the explanatory power of usability heuristics. In Proceedings of the CHI'94 Conference on Human Factors in Computing Systems, p152-158.
This article takes usability guidelines developed by different sources and sees which ones conribute most the the explanation of actual usability problems drawn from a database.

Nielsen, J. (1994) Chapter 2: Heuristic evaluation. In J. Nielsen and R. Mack (eds) Usability Inspection Methods, p25-62, Wiley and Sons.
A more in depth discussion of how heuristic evaluation works and its reliability

Cognitive, Pluralistic, and Task-Centered Walkthroughs

Rieman, J. and Lewis, C. Getting to Know Users and their Tasks. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p122-127, Morgan-Kaufmann.
Describes task-centered system design and how tasks are used to define walkthroughs.

Bias, R.G. (1994) Chapter 3: The pluralistic usability walkthrough: Coordinated Empathies. In J. Nielsen and R. Mack (eds) Usability Inspection Methods, p63-76, Wiley and Sons.
Describe the steps in the pluralistic walkthough process.

Wharton, C., Rieman, J., Lewis, C. and Polson, P. (1994) Chapter 5: The cognitive walkthrough method: A practitioner's guide. In J. Nielsen and R. Mack (eds) Usability Inspection Methods, p105-140, Wiley and Sons.
Describe the steps in the cognitive walkthough process.

Preece, J. et. al., (1994) "Excerpts-Box 33.2: Checklist for doing a cognitive walkthrough." in Human Computer Interaction, p679-684, Addison-Wesley.
Includes a form containing cognitive walkthrough instructions.

Prototyping: A Tool for Design and Evaluation.

Nielsen, J. (1993) Extract-Chapter 4.8: Prototyping. In Usability Engineering, p93-101, Academic Press.
Describes different styles of prototypes and how they can be used within scenarios.

Rettig, M. (1994) Prototyping for tiny fingers. Communications of the ACM, 37(4), ACM Press.

Muller, M.J. (1991) Pictive: An exploration in participatory design. In Proceedings of the ACM Conference on Human Factors in Computing Systems, p225-231, ACM Press.

Rudd, J., Stern, K. and Isensee, S. (1996) Low vs. high fidelity prototyping debate. Interactions 3(1), p76-85, ACM Press.

Gould, J., Conti, J., and Hovanyecz, T. (1981) Composing letters with a simulated listening typewritter. In Proceedings of the ACM Conference on Human Factors in Computing Systems, p367-370. ACM Press.

Field Study Methodologies (Note: This will change!)

Contextual Inquiry

Holtzblatt, K. and Jones, S. (1996) Contextual Inquiry: A Participatory Approcah. In D. Schuler and A. Namioka (eds) Participatory Design: Principles and Practices, p177-210.
Describes contextual inquiry

Holtzblatt, K. and Beyer, H. (1993) Contextual Design: Principles and Practice. In D. Wixon and J. Ramey (eds) Field Methods Casebook for Software Design, p303-333.

Incontext, Inc. Getting Started with Contextual Techniques. http://www.incent.com/connection.indx/techniques.html

Ethnography

Bentley, R., Hughes, J.A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D. and Summerville, I. (1992) Ethnographically-informed systems design for air traffic control, in Proc CSCW'92, p123-129, November.

Hughes, J., King, V., Rodden, T. and Andersen, H. (1994) Moving out of the control room: Ethnography in system design. In Proc CSCW'94, p429439, ACM Press.

Randall, D. (1996) Ethnography and Systems Development: Bounding the Intersection. Tutorial notes presented at CSCW'96. Excerpts: Sections 3, 4,5,7

*Bowers, J. (1994) The work to make the network work: Studying CSCW in action. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, p287-298, ACM Press.

*Shapiro, D. (1994) The limits of ethnography: Combining social sciences for CSCW. In Proc CSCW'94, p417-428, ACM Press.

*Suchman, L. and Trigg, R. (1996) Understanding practice: Video as a medium for reflection and design. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p233-240, Morgan-Kaufmann.
Describes how video records can be used for ethnographic and interaction analysis.

Other Possibilities

General

Nielsen, J. (1993) Extract-Chapter 7: Usability assessment methods beyond testing, p207-2261, Academic Press.
Briefly describes other evaluation approaches including questionairres, interviews, focus groups, logging, and user feedback.
Section 7.6 also compares usability methods.

Questionnaires and surveys

Preece, J. et. al., (1994) "Excerpts-Chapter 30.4: Users opinions: Interviews and questionairres." in Human Computer Interaction, p628-638, Addison-Wesley.
Provides an introduction..

Comparing Methods

Olson, J. and Moran, T. (1996) Mapping the method muddle: Guidance in using methods for user interface design. In M. Rudisill, C. Lewis, P. Polson and T. McKay (eds) Human-Computer Interface Design: Success Stories, Emerging Methods, and Real-World Context, p269-300, Morgan-Kaufmann.
The authors associate a variety of methodological approaches to specific interface design activities.

Preece, J. et. al., (1994) "Excerpts-Chapter 34: Comparing methods." in Human Computer Interaction, p691-707, Addison-Wesley.
Provides an overview of previous comparitive studies of evaluation methods.

Jeffries, R., Miller, J., Wharton, C. and Uyeda, K. (1991) User interface evaluation in the real world: A comparison of four techniques. In ACM Conference on Human Factors in Computing Systems. p119-124, ACM Press.

Karat, C., Campbell, R. and Fiegel, T. (1992) Comparison of empirical testing and walkthrough methods in user interface evaluation. In ACM Conference on Human Factors in Computing Systems. 397-4044, ACM Press.

Landauer, T. (1991) Lets Get Real: A Position Paper on the Role of Cognitive Psychology in the Design of Humanly Useful and Usable Systems, In J. Carroll (ed) Designing Interaction, Cambridge University Press.

Cognitive Modeling (Not done)

Fitt's Law

*Mackenzie, I.S. (1996) Movement time prediction in human computer interaction. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p483-493, Morgan-Kaufmann.

Keystroke-level model

*Preece, J. et. al., (1994) "Excerpts-Chapter 33.4: Modelling: The keystroke level model." in Human Computer Interaction, p685-687, Addison-Wesley.
Includes a working example of a GOMS analysis.

*Card, S., Moran, T., and Newel, A. (1980) The keystroke level model for user performance time with interactive systems. Communications of the ACM, 23(7), p396-410, ACM Press.
The original paper describing the model.

Goms

*Preece, J. et. al., (1994) "Excerpts-Chapter 20:Task analysis." in Human Computer Interaction, p409-413, 417-424, Addison-Wesley.
Includes a working example of a GOMS analysis.

*Kieras, D. (1988) Towards a practical GOMS Model methodology for user interface design. In M. Helander (ed.) Handbook of Human-Computer Interaction, Elsevier, North-Holland.

*John, B. and Vera, A. (1996) A GOMS analysis of a graphic machine-paced, highly-interactive task. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p626-633, Morgan-Kaufmann.
A case study of GOMS in use.

*Gray, W., John, B., Stuart, R, Lawrence, D. and Atwood, M. (1996) GOMS meets the phone company: Analytic modeling applied to real world problems. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p634-639, Morgan-Kaufmann.
Another case study of GOMS in use.

General

• *Olson, J. and Olson, G. (1996) The growth of cognitive modelling in human-computer interaction since GOMS. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p603-625, Morgan-Kaufmann.

  Card, S. (1996) User technology: From pointing to pondering. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p587-602, Morgan-Kaufmann.