CPSC 681 Research Methodologies in HCI Saul Greenberg, Instructor

Contents

Ethnography

Written by:

Amy McCleverty for partial requirements for Computer Science 681 : Research Methodologies. March, 1997

What is Ethnography?

Background

A significant problem in system design (including the design of Computer Supported Cooperative Work (CSCW) systems), is the system's failure to provide or support features that are central to its real world use. Part of this failure stems from the fact that many developers of systems often have only partial awareness of the "real world" setting where the end system will be used. This is not entirely the fault of the designer: in most cases designers are developing systems to fulfill a prescribed set of requirements -the specification. However, simply fulfilling the requirements specification does not ensure that a good system design will result: the requirements must capture the needs of the end-users.

This seems like an obvious point: to have a well-designed system, you need to know the system requirements. However, it is a far more complicated issue than it appears at first glance. A simple example is an employee leaving the office for a few days. Invariably there will be issues that need to be dealt with in the employee's absence; so s/he will have to bring a colleague "up to speed" on the issues and explain what needs to be done. Despite the employee's best efforts, it is likely that some of the information provided will not be adequate to address a particular issue. This is because there are certain aspects of peoples' day-to-day jobs that are so familiar to them that they become unremarkable. If people can't even describe their own jobs precisely, how can they describe other peoples' tasks, especially for domains in which they are not familiar, or for systems that are revolutionary?

Even formal task analysis techniques present problems:

• stakeholders, even within the same group, all have different opinions.
• one stakeholder group (management) has a lot more influence than other groups.
• task analysis lends itself to designing the operator interface to support the necessary skill and expertise among groups at work instead of designing a system to support the way groups of users like to work (why should users have to adapt to suit the technology instead of technology adapting to suit users?).

Since the system requirements must capture the true needs of the system's end-users, it is imperative that a reliable requirements capture method be implemented. Enter ethnography…

Ethnography is a traditional method of sociology and cultural anthropology. It involves the study of people performing activities and interacting in complex social settings in order to obtain a qualitative understanding of these interactions.

While the results of ethnographic studies provide information that is vital to requirements capture, the qualitative and non-judgemental nature of these results make it difficult to apply them directly to requirements capture and/or system design. In order to address this problem (thus making ethnography a feasible alternative), the method has been adapted to several forms. The most prevalent form of ethnography is "ethnomethodological (sometimes called "informed) ethnography". In fact, ethnomethodological ethnography is often referred to simply as "ethnography" within the context of system design. However, other variations of this method are used and are summarized in Table 1.

Table 1: Ethnography Methods

Ethnographic Studies

The fundamental difference between a task analysis and an ethnographic study is that the purpose of an ethnographic study is see how work is truly accomplished, while a task analysis reveals how work "should" be accomplished. This difference enables ethnographic studies to provide a realistic understanding of labour division and "work-arounds" that people implement without being aware that they're fixing a problem (such as posting a list of error codes and their meanings beside the photocopier - the photocopier should provide meaningful error information, but because someone has published a list of meanings associated with the codes, thus allowing people to do their jobs, they may not even be aware that the system is poorly designed.)

The goal of an ethnographic study is to identify routine practices, problems and possibilities for development within a given activity or setting. The study can produce powerful results, including:

• Identification of undocumented transfer of information (e.g. order in which information is passed);
• Identification of true "working division of labour", since this division is often organized dynamically, and therefore cannot necessarily follow a prescribed form. For example, if a colleague has to present a report and is running behind schedule, you may offer to make a few copies and bind the reports - even though this is not in your job description, and if asked to describe the types of duties that you are responsible for, photocopying and binding other people's reports would not be one of the duties.
• Debriefing meetings let developers gain an intuitive impression of system interactions.
• The ethnographer can act as the "end-user" or user advocate because of her/his in-depth knowledge of the system from significant amounts of time spent conducting ethnographic study. This gives the designers the advantages of having participatory design without requiring end users to be available at all times in the design process (usually not feasible to have users available).

There are four major features of ethnography that make it distinct as a method. Table 2 provides a summary of these features.

Table 2: Features of Ethnography

How do I do it?

It is important to note here that the actual ethnographic studies are performed by sociologists trained in ethnography. This section merely describes the process, and is not intended to be a detailed "how to" instruction. In addition, only ethnomethodological (informed) ethnography is discussed here. The other methods described in Table 1 are merely variations of these primary method. More details regarding the other methods can be found in Hughes et. al.

Step 1: Innocent ethnographic study

The first step for the ethnographer is to do an "innocent" or "quick and dirty" ethnographic study. The purpose of this step is to gain an understanding of the work environment and interactions without paying attention to designers' specific areas of interest. This is necessary to ensure that the ethnographer has an understanding of the current process, allowing time to learn terminology and establish a rapport with the system users. The procedures for this step are the same as those for an "informed" or "ethnomethodological" study, but merely on a smaller time scale. The length of time for this step can vary from a few weeks to a few months.

Step 2: Debriefing meeting

The results of the initial study are presented to the design team. The ethnography provides pointers to appropriate design decisions, not specific detailed system requirements. At this point, designers can focus the ethnographers on particular design issues that may arise from findings presented. These meetings occur at regular intervals throughout the ethnographic study, and are the primary means of impacting the design. An important result is that they let developers gain an "intuitive" impression of the system interactions; so even though explicit system requirements are not proposed by the ethnographers, the information can aid in the creation of the requirements.

After the first meeting, designers will develop a system prototype, which will be refined after each debriefing meeting until the "final system" is complete. There are several difficulties associated with these meetings which are discussed in the "When do I do this" stage.

Step 3: Informed ethnographic study

The first task in an ethnographic study is to understand peoples' current work practices, using the technologies that are available to them. There are several methods for accomplishing this:

1. record participants in their environment;
2. invite participants to do part of their real job in a proposed location that is equipped with recording equipment and existing tools (such as a white board or telephone system) that participants use to perform their job duties; or
3. have participants use a prototype of a new system to perform specific job-related tasks in a location that is equipped with recording equipment.

In addition, there are also different recording methods:

• videotape (described in more detail in Table 3);
• audiotape; and
• hand-written notes.

Table 3: Videotaping Methods

The first analysis undertaken after recording an extended sequence of activity is a rough content log of the videotape. The content log is used to describe observed events and index them chronologically. This log is essential for searching for particular examples further on in the analysis phase. The log also identifies issues raised while viewing the tape again. Such issue-based logging might include:

• marking an event as being of inherent interest and possibly representative of a new theme or category;
• labeling an event as being representative of a given theme or category; or
• generating proposals for further fieldwork.

Frequently, a sequence of recorded activity is picked out as being of particular interest and deserving of more systematic analysis. A first step toward such analysis (called "interaction analysis") is a careful transcription of recorded data. This transcription involves:

• recorded dialogue;
• description of non-vocal aspects of interaction, including gestures and body language; and
• description of interaction between people and machines (including machine records to show results of interaction).

With the transcription of all recorded activity complete, the interaction analysis can begin. The purpose of the analysis is to uncover the regularity and efficiency of peoples' interactions with each other and their use of the resources that their environment affords. This is accomplished through a set of general observations and analyses of specific instances. As these analyses progress, it becomes necessary to create "collections": instances of interactions that can be grouped into a particular class. These collections are used to illustrate both common and distinct features of a particular type of interaction. An example of this is illustrated in Suchman and Trigg (1996):

[From analysis of videotapes of hospital births, it became apparent that,] in the presence of high-technology equipment (e.g. electronic fetal monitor) the attentional focus of medical personnel as well as that of non-staff attendants shifts from the patient to the machinery (p. 234).

A benefit of using video-based interaction analysis is that it allows researchers to ensure that they are analyzing what actually occurred, not what they expected to occur. The tendency to see what one expects to see is powerful. Observers frequently report occurrences of a particular action, even though playback of the video tape reveals that the action did not occur at all.

Suchman and Trigg (1996) argue that it is not easy to delegate the interaction analysis work, in fact, they indicate that it is "precisely in the repeated, careful working through of the primary materials that theoretical insights arise." (p. 235). It is in this way that the analysis is similar to iterative design, where the discovery of themes/categories/similarities occurs due to familiarity with materials and constant reevaluation.

Randall and Rouncefield describe three Qualitative Data Analysis packages (NUDIST: URL: http://www.qsr.altrobe.edu.au, The Ethnograph and the Designer's Notepad (DNP)) that have two primary functions:

1. more flexible storage, coding, retrieval and analysis of greater amounts of qualitative data; and
2. consistency of presentation. In principle, everyone has access to the same data presented the same way.

However, they strongly caution the reader not to rely too heavily on these packages because they tend to hide the very facts that ethnography uncovers. That is, to use the system, data must be categorized. However, in categorizing the data, the process that went into determining what the appropriate category is (the essence of the ethnography) can be hidden.

Findings from this study are presented at a subsequent design briefing, and the ethnographers, together with the designers, reevaluate their focus. In addition, designers may have a working prototype available, and this can be used in the next iteration of the ethnographic study.

This step (including iterations) usually takes at least one year, and could take up to three years.

Summary

To summarize, ethnography is an iterative process and is detailed in Figure 1.

Figure 1: Iterative Design

When should I use this technique?

What types of systems?

Ethnography can be used for designing new systems, where the difficulty is often in determining what user needs and uses will be, as well as improving existing systems. This is because ethnography looks at work as it is being done, not as it is "supposed" to be done (as in task analysis). For example, in examining the processes in an office environment and documenting them (for ISO 9000 certification), these processes should be repeatable merely by following the steps outlined in the task. However, there are intangible things such as who the "right" person is to approach (e.g. who is known around the office as someone who "gets things done"). The intangibles are seldom apparent from a task analysis, but are often the critical aspects of the task. The basis here is that systems which are being investigated involve subtle cooperative activity, and the nature of that cooperation must be understood in order to build effective system (computer) support.

For product improvement, it is also imperative to do ethnographic studies because of peoples' incredible ability to "make do" with inferior products. For example, operators may use scraps of paper to capture information obtained from a search that will be needed later on but will not be available again without redoing the search. Using the scraps of paper, operators are able to get their tasks done; so at first glance it seems that the process is efficient. Ethnographic study makes it possible to identify areas of improvement in the flow of work.

When in the design process?

Initial efforts to use ethnography to improve system design made the application of the method seem impractical. Ethnography came in before system requirement generation. That is, the results of the ethnographic study were used to create a system model, which in turn was used as a basis for system requirements, and ultimately, contract award. Since designers are usually in a hurry to get to the contract award stage, the fact that ethnographic studies can take years doesn't really sell the method well.

However, an iterative approach allows ethnography to adapt well to system design, because the initial findings can be used to help designers make an initial prototype. This initial prototype can be refined as domain and process knowledge increases (as the study progresses, ethnographers have more and more knowledge of the users and their environment).

Problems with Ethnography

Cross-disciplinary Differences

1. The fundamental approach of each discipline is totally different. Sociology is analytic. It is concerned with gathering and interpreting data about some social situation or process and drawing some conclusions from that interpretation. By contrast, software engineering/system design is concerned with synthesis: designing and building new abstract models of the real-world. Thus, sociology focuses on and pays great attention to detail; software engineering strives to hide detail through abstractions.
2. Social analysis is a fairly prolonged activity. Typically, ethnography will take place over a period of several months with at least the same amount of time spent in analysis and interpretation of the observations. To inform the systems development process, however, software engineers need quick results and make demands on the sociologists to provide rapid assessments of their work.
3. The roots of ethnography are in anthropology so ethnographers are trained to avoid making judgements about a social situation or process and, as far as possible, to avoid letting their own prejudices interfere with their observations. By contrast, engineers must make judgements, often with inadequate information, as to what is and is not significant. Engineers therefore may find the more remote attitude of sociologists difficult to understand.

Table 4 provides a summary of the differences described above.

Table 4: Ethnographer and System Designer - Different Approaches

In addition to cross-disciplinary differences, there are implementation problems for the actual field study. These are summarized in Table 5.

Table 5: Field Work Problems

How well does this method generally perform for the following attributes? Why?

Can I see a working example?

The following examples are excerpts from a report described in Randall/Rouncefield. (1996, p. 42). The words in italics indicate conversation that was recorded, and the words in plain text are the conclusions that were drawn from the examples.

Cashier 1: "What do I do about this *** account … it's got nil written on … you can't open an account without any money in it, can you?"

Cashier 2: "It's Mr. … just put it to one side until he pays the 100 … he's got over 30,000 in his other account … don't actually open the account, just hold it … he doesn't want to open the Norcash account unless he gets the Mortgage he's applied for …

Mortgage processing is manifestly a cooperative activity, at least in this branch. Files ready to be 'sent upstairs' are always checked and the deployment of highly specialized knowledge becomes most evident during the checking process. Two brief examples demonstrate this:

"Flats like these … it's usually a situation where one of them holds the freehold. You have to have a Special conditions that the freehold applies to, say, Flat 1 …"

"This isn't a tax reference … it's a personal tax number … if we send that to the Inland Revenue they'll just send it straight back again … we'll have to ring him … "

Experienced members of the team are a resource for the less experienced and for those undergoing training . Policy is decided on the basis of the mutual deployment of knowledge:

Cashier, having dealt with a mortgage inquiry over the phone:

"that's the trouble with the Repayment ones … you've got to take the Tax Relief off, take the discount off … and when you don't deal with them every day …"

Where a team is at work in the *** branch, solutions are often quickly found by the pooling of expertise:

"If the advance is dead on 90%, would you times it by three or by 2.75?"

"Does she need to have a multiple of three?"

"Yeah, its dead on now …"

"Put it in as 89 then and times it by three … you're alright … just … doing it by 2.75 …. Do it by 2.75"

Who can tell me more (References)?

Bentley, R., Hughes, J.A., Randall, D., Rodden, T., Sawyer, P., Shapiro, D. and Summerville, I. (1992) Ethnographically-informed systems design for air traffic control, in Proc CSCW '92, p123-129

• Describes the iterative approach to ethnography with specific reference to an ATC application.

Hughes, J., King, V., Rodden, T. and Anderson, H. (1994) Moving out from the control room: Ethnography in system design. In Proc CSCW '94, p429-439, ACM Press.

• Provides a description of different ethnography methods (concurrent, quick and dirty, evaluative and re-assessment).

Randall, D. (1996) Ethnography and Systems Development: Bounding the Intersection (Parts 1 & 2). Tutorial notes presented at CSCW '96.

• Provides extensive background, motivation and process information for ethnography.

Richards and Richards (1991a); Richards and Richards (1991b); URL: http://www.qsr.latrobe.edu.au

NUDIST Software

Shapiro, D. (1994) The limits of ethnography: Combining social sciences for CSCW. In Proc CSCW '94, p417-428, ACM Press.

• Discusses difficulties using ethnography in system design (sociological versus engineering approach).

Suchman, L. and Trigg, R. (1996) Understanding practice: Video as a medium for reflection and design. In R. Baecker, J. Grudin, W. Buxton and S. Greenberg (eds) Readings in Human Computer Interaction: Towards the Year 2000, p233-240, Morgan-Kaufmann.

• Describes the use of video in an ethnographic study.