|
Article Excerpt
Introduction
The importance of geovisualization for exploration and analysis of spatial data has been widely recognized (Andrienko and Andrienko 2006), and the potential and limitations of information visualization tools have been explored in numerous controlled experiments (Plaisant 2004). Yet there exists certain skepticism towards visualization techniques among data analysts. One of the reasons for this is that visual data exploration is essentially a very complex process, and at present not much is known about how visual tools actually support humans in data exploration (Andrienko and Andrienko 2006). While usability testing in controlled conditions using the principles of human-computer interaction remains the main approach to evaluating visualization tools (Plaisant 2004), such experiments are not always sufficient in the domain of geovisualization, due to their exploratory and interactive nature (Andrienko and Andrienko 2006). One way to approach this issue is to combine formal and exploratory usability evaluation methods in one experiment to assess how data exploration is performed with geovisualization tools. This paper presents an experiment that combines both approaches. Besides investigating how users explore a spatial data set with a visual data-mining system, the study also introduces a low-cost methodology for usability evaluation, developed after discount usability principles by Nielsen (1994).
Usability is the extent to which a computer system supports users to achieve specified goals and does so effectively, efficiently, and in a satisfactory way (Ivory and Hearst 2001). In human-computer interaction, which investigates interaction between human users and information systems (Preece et al. 2002), usability forms a small part of the larger issue of system acceptability. System acceptability is the answer to the question of whether the system is good enough to satisfy all the needs and requirements of the users. The term combines social and practical acceptability, and the state of acceptability is further subdivided into several categories, such as usefulness, cost, reliability, and compatibility with existing systems.
Usefulness denotes whether the system can be used to perform some defined task in order to achieve some desired goal. It is divided into utility and usability. Utility describes whether system functionality can do what is needed for the defined task, whereas usability refers to how well the users can use the functionality. The outline of the system acceptability model is presented in Figure 1 (Nielsen 1993).
[FIGURE 1 OMITTED]
Usability evaluation is the process of systematically collecting data on how a particular user or a group of users uses the system for a particular task in a particular environment. There are four main methodologies for performing the usability assessment: "quick and dirty" evaluation, usability testing, field studies, and predictive evaluation (Preece et al. 2002).
In a "quick and dirty" evaluation the designers receive informal feedback from users at any stage of the design process. Usability testing includes methods that measure the performance of the users and their experience with the tool. Field studies are conducted in natural settings with the aim of understanding how technology impacts the users in their daily routines, using qualitative techniques, such as interviews, observation, participant observation and ethnography. Predictive evaluation predicts usability problems by asking the experts to apply their knowledge of typical users to a particular usability issue, while the users do not have to be present. The most commonly used method for this purpose is heuristic evaluation (Hackos and Redish 1998; Preece et al. 2002).
The experiment performed in this study attempts to combine formal evaluation methods with exploratory usability methods. Formal evaluation methods are tests that measure the performance of the users on carefully prepared tasks. Exploratory usability on the other hand investigates how users use the tools. Data collected in a formal evaluation are observational and consist of controlled measurements of the number of errors made and the time used to complete the tasks. Typically, data on user satisfaction are also collected. Exploratory usability produces descriptive data in the form of verbal protocols and observational notes (Preece et al. 2002; Griffin 2004).
During usability testing, a number of usability indicators can be observed and measured in order to assess the five main usability factors: learnability, efficiency, memorability, error rate and satisfaction (Nielsen 1993). Sweeney et al. (1993) suggest a taxonomy of typically measured usability indicators. The indicators are grouped according to the evaluation approach into indicators for a user-based approach, expert-based approach, and theory-based approach. Table 1 shows the usability indicators for the user-based approach--which are of interest for this study--and the data relevant to each of the indicators.
The issues of usability testing in geovisualization are not exactly the same as those in human--computer interaction. The problem lies in the exploratory and interactive nature of geovisualization, which makes it difficult to clearly define tasks that should be evaluated. Traditional usability methods borrowed from human-computer interaction therefore need to be adapted accordingly (Fuhrmann et al. 2005; Andrienko and Andrienko 2006).
Most of the recent attempts of usability research in geovisualization fall into the category of formal usability evaluations. There are numerous evaluations in literature (Edsall et al. 2001; Andrienko et al. 2002; Suchan 2002; Tobon 2002; Edsall 2003; Haklay and Tobon 2003; Slocum et al. 2003, Ahonen-Rainio 2005; Koua 2005; Robinson et al. 2005).
A disadvantage of a formal usability evaluation is that the predefined tasks used during such an evaluation must be very specific in order to be able to measure the performance. They must also not be too long or too complex, so that they can be accomplished in a short time. While differences in time, error rate, or user satisfaction help the developers to estimate the potential and limitations of the tools, formal evaluations can not answer such questions as: what would have happened if the test participants had more time to explore the data on their own, to use their own data, or to choose and define the exploration tasks themselves (Plaisant 2004). For this purpose, test participants need to be engaged in free data exploration, where they are given a free hand to do whatever they want with the tool (Andrienko et al. 2002).
An experiment using a free exploration session gives the test participants the chance to look at the data from different perspectives and to formulate and answer questions that did not occur to them before using the tools. As Plaisant (2004) puts it, "Information visualization is sometimes described as a way to answer questions you didn't know you had." Test participants should therefore be allowed to explore the data freely on their own, while the observer collects data on what they are able to see and which observations they make about the data set. A disadvantage of such experiments is that they take a long time to conduct, because the participants need a training session before they can start the free exploration. Another issue is that the results of such evaluations might not be replicable (nor generalizable) and are in general difficult to interpret (Plaisant 2004). These are perhaps the reasons that only a few exploratory usability assessments have been performed for exploratory geovisualization (Tobon 2002; Griffin 2004; van Elzakker 2004).
Formal and exploratory methods complement each other. Ideally, a convergence of both types of methods should be used in the same test for an optimal evaluation. Rather than conducting only controlled experiments with pre-defined tasks, or observing only natural situations, both approaches should be used in parallel to better understand the data exploration process. An analogy for this combination of research methods can be found in the research practices in other fields. In biological research, for example, scientists use "in vitro" methods to investigate an organism in controlled laboratory settings. In parallel, they observe how the same organism behaves "in vivo," in its natural environment. Using both approaches at the same time helps to understand a certain phenomenon better. An analogy to this "in vitro/in vivo" approach in usability research is the combination of formal evaluation and exploratory usability (Dunbar 2001).
As mentioned, a problem with exploratory usability is that the users need to be very familiar with the tools and all their functionalities. Extensive training is therefore necessary. This paper presents an experiment that combines formal and exploratory approaches, where formal evaluation serves as the training session for the exploratory usability assessment. In the formal usability evaluation, users were required to perform a number of pre-defined tasks in order to assess their performance. Only after completing this part did the participants get an opportunity to freely explore the data in an exploratory usability session. The experiment was conducted for a visual data-mining system developed for emergency response data. The rest of the paper is organized as follows: the following section presents the application of visual data mining on emergency response data, which was evaluated in this study. Next comes an evaluation of the methodology used. Results are presented in the following section, and they are further discussed in the last but one section. The last section presents conclusions and recommendations for further work.
Exploration System and Its Application for Emergency Response
The visual data mining system used in this study was developed for exploration of...
|
