Working Examples
Example # 1 : Usability evaluation of Electric Book at Georgia Tech School of Computing.
Example # 2 : An evaluation of representation type and transparency level on overviews in Carl's PipeLine System.
This is an assessment/comparitive usability study that is currently testing for the effects of overview representation type and transparency levels on participant performance and preference, with a pipeline construction system. The contraints of the system are that the main view screen is smaller than the workspace, such that if only using a main view screen, the user is unable to attain global awareness and is unsure of where other pipeline structures are located as well as how the main view fits into the overall workspace. Overviews of the entire workspace can rectify this constraint. There is however a disadvantage with overviews as they require desktop space in the already cluttered main view. CPLS allows overviews to be layered on top of the main view such that it is possible to see pieces in the main view as well as in the overview. There is however an interference effect as the user is sometimes unable to distinguish which pieces are layered on top and when overview pieces cover up main view pieces. The present usability study wishes to compare between three overview representation types and transparency levels, with participants completing pipeline construction tasks. Three types of overviews were selected for the usability test : scaled, outline, and schematic. Three transparency levels were also selected for the test : 30 %, 50 % and 70 % transparent.
We are currently running a pilot test, and have encountered some software problems that are under repair. At the start of the study there is a verbal procedural protocol that we have developed that is read to each participant, along with the signing of a consent form. The participant then fills out a pre test questionnaire. Two sets of tasks have been developed for each of the six conditions, with the first task as a practice task. The participants are encouraged to think aloud and their interaction with the pipeline system is video recorded. A post test questionnaire and interview are also being used to obtain usability information.
It has not cost us much (yet) to run the test. The CPLS was already up and running before the idea for the test. This is thanks to Mr. Carl Gutwin. Donald has done a great job in modifing the CPLS to fit our study. We did a task analysis on the kinds of tasks that can be performed with the CPLS. A usability lab, with a HI8 video recording system, is currently being set up in the Computer Science department, so we didn't really have to worry about a location to run the test in. Our pilot participants are graduate students and professors who are part of the computer science and psychology departments. So even though most of these usability experts and books harp about getting representative users, it is not really that essential during early phases of design. Our pilot tests are somewhat familiar with the system, but that is not a big concern, as pilot testing is a test of the test process, and is not dependent on test users not being familiar with the product. For the real usability test, we will recruit individuals from the computer science and psychology departments who are not familiar with the product. We have not and are not planning on providing renumeration for our participants. We are asking them for favors and we hope to repay them in kind. We are planning on running 4 pilot tests with 8 main usability test participants.
One of our pilot test participants has already commented that the scaled version should turn a different color when a pipeline piece is grabbed in the overview if it is not inside the main view. This recommendation might be followed up if the system develops further, resulting in another usability test.