|
Article Excerpt
ABSTRACT
Decision Support Systems (DSS) frequently have multiple decision aid (DA) features, causing users to engage in exclusive choice behavior; i.e., choice between alternative DA features that results in one feature being used to the exclusion of all others. We hypothesize that: (1) users choose the least effective (least accurate) DA feature in certain predictability environments; (2) users choose the DA feature that they believe they are most competent with; and (3) choice between DA features improves performance compared to those assigned the same DA feature. We test these hypotheses in an experiment in which 164 participants act as loan officers who chose between two decision aids (a database aid and a regression aid). The results support our hypotheses. Users employed a choice heuristic that caused them to choose the least effective DA feature for the task more than or as often as the most effective DA feature. Results also indicate a positive relationship between perceived competence and DA feature choice, and the positive effect of DA choice. We conclude by describing the insights provided by the results into the heuristics of information technology choice.
Keywords: user choice; decision aids; decision support systems; competence; heuristics.
Data Availability: Data are available upon request.
I. INTRODUCTION
Accountants and other computer users must choose which features to utilize in their computers. Computer systems have become so versatile and user friendly that they now present the user with a myriad of options. Toolbars, task panes, and drop-down menus are everyday examples of the choice-rich environment confronting users. Although programs may allow the user to employ multiple features in a single work session, the very richness of the choice environment often results in users choosing to use only a subset of the available features. Users are thus responding to this rich environment by engaging in "exclusive choice," i.e., a choice between alternative features that results in one feature being used to the exclusion of all other features. Users may make exclusive choices for several reasons--time constraints, cognitive effort minimization, heuristics, habit, or the belief that additional tools will results in only marginal gains. Nonetheless, little is known about the nature of exclusive choice. To what exte nt, for example, is it rational? To what extent is it driven by heuristics that may lead to suboptimal performance? (1)
One environment in which accountants and accounting information users engage in exclusive user choice is that of decision support systems (DSS) containing multiple decision aid (DA) features. DSS are widely used in business, going under such names as online analytical processing (OLAP), software agents, business intelligence software, executive information systems (EIS), geographic information systems (GIS), executive support systems (ESS), group DSS, and knowledge discovery systems (Power 1997; Turban et al. 2001). One aspect that these systems share and that allows them to be highly versatile is the presence of multiple DA features. DSS typically include some mix of DA features that are model-driven, data-driven, communications-driven, document-driven, and knowledge-driven (Power 2001). Software designers and vendors have long recognized the difficulties involved in designing DSS with the proper mix of DA features, the goal being to provide neither too few nor too many features (Dix et al. 1993; Sprague an d Carlson 1982). (2)
Research suggests that exclusive user choice of DA features can affect decision process and accuracy. Jones and Schkade (1995) found that when users have to choose which representation will be used for solving a problem, they make this choice early and heuristically. Dilla and Stone (1997) demonstrated that user choice of response formats impacts DSS usage. Prior research has not, however, investigated the effect of exclusive choice between multiple DA features in a DSS on user performance, specifically accuracy and choice strategies. Accordingly, this paper investigates the issue of how exclusive user choice affects decision aid usage; i.e., the accuracy and choice behavior of users provided with multiple DA features.
In a behavioral experiment, exclusive choice between two DA features in a DSS is forced on subjects via the design of the DSS in order to proxy for environmental and psychological factors that cause real-world users to engage in exclusive user choice. Our contention is not that systems would be designed to force exclusive choice (although this may be the best solution in some cases), but that these exclusive choices constantly occur in computer usage, that they impact user effectiveness (e.g., accuracy) and that, like many behavioral problems in IT usage, the solution may not be straightforwardly technological (i.e., a design issue).
We organized the remainder of this paper as follows. The next section reviews the literature on DSS and user choice. We then develop hypotheses concerning the impact of exclusive user choice on judgment quality and decision making. The subsequent section explains the experimental method. We then present the results. In the final section, we discuss limitations, contributions, and future research.
II. LITERATURE REVIEW AND HYPOTHESIS DEVELOPMENT
Of necessity, users of multiple DA features must engage in some type of choice behavior. Users must either choose: (1) to rely on all of the DA features available as fully as possible, (2) to rely on the DA features to varying degrees less than fully, or (3) to use one or more DA features to the exclusion of the others. Case studies indicate that users engage in type (3) (exclusive user choice) and, consequently, do not always use DSS with multiple features optimally (Stabell 1983; Swisher 1996). Stabell (1983) found that several DA features in a portfolio management DSS were never used. Swisher's (1996) investigation of the Medicaid Management Information System (MMIS) DSS indicated that some features were used infrequently. Similarly, Silicon Graphic's Mineset[TM] allows users to choose from a menu of datamining techniques to analyze the data set and it is common to see users using the selected technique/feature exclusively for an extended period of time (Rathjens and Vanderberg 1998).
Researchers have concentrated on two areas of user choice in DSS. First, research has been done on the extent to which users choose to employ a single DA feature; i.e., decision aid reliance (Arkes et al. 1986; Ashton 1990; Brown and Jones 1998; Davis and Kotteman 1994; Dos Santos and Bariff 1988; Jones et al. 2001). This research, however, does not address the issue of exclusive choice between alternatives. The second stream of research has investigated different types of exclusive user choice in DSS, but not the choice between competing decision aids (Becker 1997; Jones and Schkade 1995; Dilla and Stone 1997; Whitecotton and Butler 1998). The studies in this second stream are important because they shed light on the general nature of exclusive user choice in DSS. They indicate that user choice is a significant type of behavior among IT users, although the results are somewhat mixed. In two studies (Jones and Schkade 1995; Whitecotton and Butler 1998), user choice had a predominantly negative effect on perf ormance, while in the other two studies (Becker 1997; Dilla and Stone 1997) it had a primarily positive impact. However, these studies have important differences in both theoretical focus and experimental design from each other and from this study. Accordingly, the present study makes a contribution by investigating the impact of exclusive user choice between multiple DA features on decision making.
Hypothesis 1: Choice Heuristic Hypothesis
To the extent that decision makers choose rationally, choice should improve judgment quality (e.g., accuracy). However, choice may cause a decline in judgment quality if users choose inappropriately, which may occur if decision makers choose the DA feature using a choice heuristic. Heuristics tend to introduce biases in decision making because they are often used across diverse conditions for which they are not equally effective (Baron 1994; Kahneman et al. 1982).
Prior research indicates that the heuristic specific to choosing between DA features involves a tendency toward suboptimal use of information technology (IT) under a wide variety of circumstances. Arkes et al. (1986) and Ashton (1990) found that incentives that were normally effective at improving judgment quality in unaided conditions became ineffective, if not detrimental, in conditions with mechanistic aids. Arnold and Sutton (1998) find evidence that users become overly dependent upon technology-based decision aids. Peterson and Pitz (1986) conversely found users rejecting advice from models based on the users' own implied decision-making model when that model is automated.
The heuristic suggested by this research is one in which human cognition, adapted toward certain responses to decision-making problems under nontechnological situations, becomes dysfunctional by the relatively recent addition of IT, such as decision aids. Modularity theory in cognitive science postulates that the mind confronts reality with a finite number of specialized modules consisting of various decision-making and problem-solving heuristics (Fodor 1983). Because such modules and heuristics develop slowly and meticulously, there is probably not currently a fully functional module or heuristic for incorporating modem IT effectively into human cognition (Buss 1998). The choice heuristic we are suggesting is, therefore, one in which, when confronted with choices between multiple IT features, the "rule of thumb" humans use is either: (1) to resist the technology or (2) to inadequately incorporate it into an incompletely developed module/heuristic or into one developed for other types of problems. The first reaction (resistance) helps explain decision aid underreliance. The second reaction (inadequate incorporation) is the one predicted for the situation in the current study because a choice between two technologies forces the inclusion of technology, no matter how ineffectively.
We hypothesize that users employ a heuristic for choosing between DA features and subsequently applying the chosen DA feature. We also hypothesize that exclusive user choice will have differing effects...
|
