CPSC 681 Research Methodologies in HCI Saul Greenberg, Instructor

Contents

Project

Purpose

CPSC 681 is about learning how to apply methodological techniques to problems in human computer interaction. The purpose of the project is to give you experience

1. defining an interesting (but modest) problem,
2. learning how to apply a particular methodological technique to that problem
3. planning a study
4. conducting both pilot and the main study
5. analyzing and interpreting the results
6. critically appraising the limitations of the study
7. presenting your work in a professional manner, both through writing a technical report and through oral presentation.

Alternatively, your project can give you experience crafting a software tool that supports a methodological process (e.g., on-line support of heuristic evaluation). The experience you will gain is:

1. defining an interesting portion of a methodology that is amenable to software support
2. defining the audience and when and where it would be useful
3. developing a design rationale and a design process to bring to bear on that methodology
4. actual iterative system design
5. conducting a study using your software tool to see how well it performs in practice (note that this involves doing a mini-study!)
6. critically appraising the limitations of the system
7. presenting your work in a professional manner, both through writing a technical report and through oral presentation.

Project Teams

Projects can be team work (prefered) or individual. The ideal team size is two to three people. While these teams can be self-selecting, I will insist that stronger (experienced) team members include novice researchers within their groups. Each group is responsible for carrying out all aspects of the project (e.g., finding subjects if necessary, booking rooms and equipment); while I will guide you through the work, I will not hold your hand.

Most of the work will have to be done outside of class time.

Example Milestones

The schedule contains required milestones and when they are due. I suspect that most projects will be based upon some type of user observation, and the milestones are crafted around that. Milestones will be tailored to fit projects that do not require users (e.g., some of the analytic methods) and projects that involve software design .

Problem statement and suggested methodological approach

The problem must be one relevant to human-computer interaction and of interest to you and your team. Ideally, it will be related to:

• your thesis work,
• systems being worked on at your job site (if you are working),
• an on-going research project.

Note: Problems selected must be amenable to one of the methodological approaches discussed in class. This is somewhat backwards; while the class philosophy is to choose a methodology that is appropriate to a problem and the needs of the researcher, we may have to craft the problem so that it is appropriate to a selected methodology!

Deliverable. Concise problem statement, why it is interesting to you and your team, what methodology you will bring to bear on this problem, and why you feel that methodology is appropriate.

Detailed description of problem and methodological setup

You should describe the problem in detail and say exactly how your group will conduct the study. Methodological steps should be detailed enough so that someone else could actually run the experiment! You should also describe the data you will be collecting, how it will be analyzed, and what results you expect.

Deliverable: The deliverable is really a draft of the first 3rd of your final report. It will contain

• Detailed problem statement, motivation, and background
• Methodology
• Example of your expected data / results (made up)
• Example analysis

Pilot study

Experiments, like computer programs, are full of bugs. Save yourself time and effort by running yourselves (and your class mates) through your procedure. It would be nice if you could practice your analysis on the data produced by these subjects. This is a great time to catch the bugs and get rid of them. It is also a good time to get a sense of what the real results will be like, and perhaps to consider what changes should be made to the problem statement and methodological approach.

Deliverable: The deliverable is a variation of the previous one

• Detailed problem statement, motivation, and background
• Methodology used for pilot study
• Data and results observed
• Initial analysis of results
• Expected trends
• Changes to problem statement/methodological approach

Running the study

For the actual study, you will probably have to:

• get approval from the ethics committee (see me about this)
• book rooms if needed
• prepare any necessary equipment and/or software
• solicit and schedule subjects
• run subjects and collect data/observations
• analyze the data and interpret the results

Final report and presentation

Deliverables:

• Professional quality (CHI format) report that integrates all the above, and that includes background, conclusions, and suggestions for further work. An MSWord style sheet is included here.
• Archival and well-structured binder that details all your research (this will include everything mentioned above)
• Class Presentation in conference format