|
Article Excerpt
INTRODUCTION
Usability evaluation has grown to be part of every good effort to produce a software product that consumers can easily learn and use. The motivation to deliver products that satisfy consumer demand for usable applications has led to the development of large usability testing environments and associated methods to develop effective interaction designs. Despite increased focus on usability and on the processes and methods used to increase usability, a substantial amount of software is unusable and poorly designed. This is attributable, in part, to the lack of a sufficiently broad set of cost-effective usability evaluation tools. For example, traditional lab-based usability evaluation is often too expensive and can be less applicable early in the development cycle.
Our goal was to develop and evaluate a usability inspection method and supporting tool that has a theory-based framework for usability problem analysis and classification, usability data management, and design guidelines. This paper reports on the development of this usability evaluation method and its evaluation in a comparison study with two other usability evaluation methods (UEMs). Although other tools exist for supporting usability development activities (e.g., tools to support low-fidelity or paper prototyping, high-fidelity or programmed prototyping, usability problem classification and analysis, and usability problem data maintenance), these are beyond the scope of this paper.
The Need for More Useful Metrics
Practitioners wishing to select a UEM best suited for their needs look to comparison studies of UEMs. However, researchers performing those studies have not always provided commonly used and understood metrics for knowing if a method is effective. Lund (1998) pointed out that no single standard exists for direct comparison, resulting in a multiplicity of different measures used in UEM studies, capturing different data defined in different ways. Because usability is highly contextual, it is unlikely that standards can exist except at a very high level of abstraction.
Practitioners and researchers still need measures that can be commonly used and understood for determining whether a usability method is effective. However, few studies have clearly identified the target criteria against which success of a UEM is measured. As a result, the body of literature reporting UEM comparison studies does not support accurate or meaningful assessment of, or comparisons among, UEMs. Gray and Salzman (1998) highlighted this problem when documenting specific validity concerns about five popular UEM comparison studies. A key concern noted by Gray and Salzman is the issue of using the right measure (or measures) to compare UEMs in terms of effectiveness. Part of our goal in this work was to contribute to the development of more useful metrics for UEM comparison studies.
Model-Based Approaches
As an integrating framework and theory for our usability inspection tool and our other usability engineering support tools, we adapted and extended Norman's stages-of-action model of interaction (Norman, 1986). Our work is not the first to use Norman's model as a basis for usability inspection, classification, and analysis. Kahn and Prail (1994) used a task performance model that is similar to Norman's stages of action in that it has steps for planning, selecting, acting, perceiving, and evaluating with respect to the user's goal. Several approaches (e.g., Cuomo & Bowen, 1994; Garzotto, Matera, & Paolini, 1998; Lim, Benbasat, & Todd, 1996; Rizzo, Marchigiani, & Andreadis, 1997; Sutcliffe, Ryan, Springett, & Doubleday, 1996) have used Norman's model and found it helpful for communicating about usability problems, identifying frequently occurring problems within the model, and providing guidance for diagnosis of usability problems. In particular, Cuomo (1994) used Norman's stages-of-action model with some success to assess the usability of graphical, direct-manipulation interfaces. Cuomo concluded that the model shows promise, especially for problem classification in the usability testing environment, but that more work is needed to modify Norman's stages-of-action model for use as an inspection technique.
USER ACTION FRAMEWORK AND USABILITY PROBLEM INSPECTOR
Out of the needs expressed by practitioners and researchers, we developed an inspection tool and method, called the usability problem inspector (UPI), to support usability practitioners within the interaction development process for ensuring usability. We began by extending Norman's (1986) stages-of-action model into what we call the interaction cycle. Like Norman's model, the interaction cycle describes cognitive, perceptual, and physical actions users make as they interact with any kind of machine, especially computers. Our extension of Norman's model primarily involved some terminology changes as well as more detail and focus on user physical actions. Norman's model focuses primarily on cognitive and perceptual actions and has less emphasis on physical actions.
We then built the user action framework (UAF), a hierarchically structured knowledge base of usability issues and concepts residing in a relational database, using the parts of the interaction cycle (planning, translation, physical actions, and assessment) as the top-level UAF categories. Under the planning category, for example, we developed a structure of successively more detailed subcategories about how interaction designs support user planning (e.g., the user's system model, goal decomposition, user's knowledge of system state and modalities, user and work context). The UAF content came from interaction design guidelines, human-computer interaction issues in the literature, and from our own experience with more than 1000 real-world usability problems. Figure 1 shows how the interaction cycle is used as the top-level organizing structure for usability concepts and issues to form the UAF.
[FIGURE 1 OMITTED]
The UPI is part of a suite of usability engineering support tools built on the same UAF content and structure. Each tool is a mechanism for applying the UAF contents in a certain way for a certain purpose. Each tool governs the way in which UAF contents are navigated to serve the purpose of the tool. Each tool governs the context in which the tool user sees UAF content and the expression of that content. In the case of the UPI, for example, the UAF content is presented as a set of usability inspection questions. The UPI also supports reporting of answers that indicate possible usability problems, along with full usability problem classification within the UAF structure.
Using the underlying UAF to provide the inspection questions for the UPI ensures that the scope of the inspection questions will represent the full breadth across all the cognitive and physical user actions that a user does while interacting with a computer. These include questions on how well the interaction design being inspect ed supports the user in planning what to do, determining how to do it, doing it, and assessing whether the outcome was successful. The UAF as a foundation for UPI questions also ensures that the inspection questions cover the full range of issues, from the most general to the most specific.
The UPI is implemented as a Web-based tool using HTML and Active Server Pages to connect to a back-end relational database. The choice of which inspection questions in the UAF to present to the inspector to drive an inspection is based on answers to questions at previous nodes in the framework. The inspector has a choice between two modes: task-based inspection or free-exploration inspection; in the latter, the inspector has no particular task in mind. Table 1 shows some of this structure in a graphical map of the UAF, including the major parts of the interaction cycle (i.e., planning, translation, physical actions, and assessment) and one or two levels under each part of the interaction cycle.
This map is implemented as a "fast-access tree" in each UAF tool. The fast-access tree allows the user to traverse the UAF hierarchy by using an expanding directory structure. During the inspection process, the UPI presents the inspector with questions from each UAF node visited about how well the target (being inspected) interaction design supports usability from the specific perspective of that node content. For example, suppose the current UAF node content is about the meaning of cognitive affordances for determining actions to carry out an intention. The inspector will be asked to evaluate the effectiveness of, for example, visual cues such as button labels and menu choice wordings with respect to how well they convey the meaning of the functionality behind the button or menu choice in the context of the task being considered. For the UPI tool user, there is additional information to explain the meaning of each UAF node. In addition to the node name (e.g., "task structure and interaction control"), each node contains much more information to help users understand and distinguish concepts to make choices, including node content, choice descriptions at each node, and "look-ahead" information previewing lower-level contents of each subtree.
As the user of the UPI traverses the hierarchical UAF structure node by node, the inspector is thus presented with a series of questions about a broad range of usability issues in the target system. The question traversal process starts with the inspection session screen, where the first node of the UAF (i.e., planning) is presented to the inspector, as shown in the bottom half of the screen in Figure 2. The example shown in Figure 2 is a sample screen shot of a version of the UPI used to evaluate an electronic address book application. The top half of the screen displays the task so that the inspector can easily remember the context of the inspection. Inspectors examine the current UAF node content (labeled in the screen of Figure 2 as "problem statement") as an inspection question and decide whether a potential issue exists in the context of the current task.
[FIGURE 2 OMITTED]
As an example of how control of the flow of inspection questions is based on inspector answers, consider a question from the UAF node about the meaning of a feedback message (under "assessment > issues about feedback > content, meaning of feedback" in Table 1). The question would ask if the inspector thinks there could be problems with users understanding the meaning of the message. If the inspector says "yes," questions from nodes at the next lower level are considered. Questions from lower-level nodes are related to the topic of their "parent" nodes but are more detailed and more specific. For example, the inspector may see more detailed questions about possible causes of problems in understanding a feedback message. These include a sequence of questions about clarity, precision, completeness, correctness, relevance, user-centeredness, and consistency of that feedback message. If the inspector says "yes" to the question about a problem with clarity, the UPI will present questions about possible causes of problems with clarity, such as precision and conciseness of wording.
If the inspector gives a "no" answer to a question--for example, to the question about understanding the meaning of the feedback message--that topic is abandoned and the subtree containing more specific issues about that node is pruned off. The next question, instead, comes from moving laterally to a "sibling" node (e.g., the next topic about feedback messages). If there are no siblings left, the UPI moves back up to the parent and over to its next sibling, until a usability problem is detected and the answer to the inspection question is "yes."
When a problem is identified, inspectors are presented with a problem report form, as shown in Figure 3. When a problem is detected and the inspector has traversed the full depth of the corresponding area in the UAF database, the inspector reaches one or more end nodes, where very specific usability attributes are listed for selection. As shown in Figure 3, inspectors select one or more usability attributes relevant to the problem and provide a problem name and narrative description to complete the report form. The problem report form also records relevant information, such as the current task and the inspection path taken by user to reach the end node, as shown in the top portion of Figure 3.
[FIGURE 3 OMITTED]
Once inspectors document a problem, they continue traversing the remaining structure of the UAF database, answering inspection questions while examining potential usability issues related to each node applied to the current task. After inspectors traverse the UAF database by selecting "yes" or "no" for each usability problem question, they can then return to the beginning of the UAF to investigate the next task or they can finish the inspection. Inspectors can also use the UPI in a free-exploration mode to investigate an interface design.
The process in the free-exploration mode remains the same as with the task-based approach, except that task information is not recorded. Free-exploration mode allows the inspector to report on potential usability issues without a specific task in mind. In addition, free-exploration mode allows the inspector to review in more detail interface areas that were encountered only briefly on the way to completing a task.
Our approach to the UAF and UPI shares some similarities to the work done by Grammenos, Akoumianakis, and Stephanidis (2000) and by Henninger (2000). Grammenos et al. and Henninger have found a hierarchical structure of guidelines to be useful for providing context-specific usability problem descriptions. Grammenos et al. provided a limited automated inspection of the static components of a user interface design using the Visual Basic Integrated Development Environment. Henninger (2000) implemented a hierarchical structure of guidelines using cases and rules to examine the object attributes of the user interface. Both of these approaches show significant value in the use of guidelines, especially if they are combined with a theory-based approach using Norman's stages-of-action model of interaction (Norman, 1986).
THE THREE USABILITY INSPECTION METHODS OF THE COMPARISON STUDY
Expert-based usability inspection methods have emerged as an alternative to lab-based usability testing, being applicable earlier in the development process as well as less costly (Nielsen & Mack, 1994). Usability inspection methods (e.g., heuristic evaluations, cognitive walkthroughs, formal usability inspections, guideline reviews, and usability walkthroughs) can also be used in circumstances where lab-based usability testing is impractical or to help focus later lab-based testing efforts.
Of the usability inspection methods, heuristic evaluation (Nielsen & Molich, 1990) and cognitive walkthrough (Polson, Lewis, Rieman, & Wharton, 1992; Wharton, 1992; Wharton, Rieman, Lewis, & Polson, 1993) are the ones for which researchers have provided of the most data in comparative...
|
