|
Article Excerpt
INTRODUCTION
The study of decision making has always been motivated, at least in part, by the desire to improve it. Cognitive engineering is a burgeoning subspecialty within the human factors and ergonomics community that is focused on the design, development, and testing of interventions for that purpose. Of necessity, it requires criteria and models to determine when decision making needs improvement, what kinds of interventions are appropriate, and the interventions' degree of success. Such models and criteria provide warrants for prescription.
A large number of present-day decision researchers frame the question of warrants, implicitly or explicitly, in terms of a distinction that first became pervasive in the 1950s (e.g., Savage, 1954) between descriptive and normative decision-making models. Descriptive research is concerned with how people actually make judgments and choices, and it aims at empirical accuracy and (if possible) explanatory adequacy. Normative research explores how people ought to make decisions, by specifying principles and constraints derived from formal or mathematical systems such as deductive logic, Bayesian probability theory, and decision theory (also known as rational choice theory or the subjectively expected utility model). According to researchers in this tradition, the need to improve decision making arises because human decision makers systematically violate normative constraints. Because of their limited computational capacity, however, human decision makers cannot directly implement ideal normative models.
To bridge the gap between the normative and the descriptive, Bell, Raiffa, and Tversky (1988) introduced the term prescriptive to refer to a third category of research, the "engineering side of the pure (normative) theory" (Raiffa, 1994, p. 4; von Winterfeldt, 1999). A prescriptive intervention (such as decision training, decision analytic consulting, or decision aiding) aims to help real people more nearly satisfy the normative ideal within the constraints of their cognitive abilities. Experimental work in behavioral decision making helps identify targets for prescriptive intervention (e.g., Gilovich, Griffin, & Kahneman, 2002; Kahneman, Slovic, & Tversky, 1982; Kahneman & Tversky, 2000), and experiments in cognitive engineering determine whether decision training, advice, or decision aids successfully reduce departures from formal normative standards (e.g., Baron & Brown, 1991; Nisbett, 1993). Because formal normative models provide the warrants for intervention, its content, and the ultimate criterion of success, this widely shared paradigm is called analytically based prescription.
Are analytically based normative models the only source of warrants for prescription? Should analytically based models play a dominant role in the design and evaluation of prescriptive practices? One alternative to analytically based prescription is descriptive research on the way proficient or expert decision makers accomplish real-world tasks. Such research is conducted by experimental psychologists (Ericsson & Smith, 1991), researchers in judgment and decision making (Shanteau, 1992), and researchers in naturalistic decision making (NDM; Cohen, Freeman, & Wolf, 1996; Klein, Orasanu, Calderwood, & Zsambok, 1993; Lipshitz, Klein, Orasanu, & Salas, 2001). Researchers in NDM, in particular, have designed training and decision aids based on discrepancies in the knowledge representations and cognitive strategies used by more and less experienced decision makers (Cohen & Freeman, 1997; Cohen, Freeman, & Thompson, 1997, 1998; Cohen & Thompson, 2001; Crandall & Getchell-Reiter, 1993; Klein, 1997; Pliske, McCloskey, & Klein, 2001). Because the warrants for intervention, the contents of prescriptions, and the standards of success are based on empirical research rather than formal models, this approach is called empirically based prescription. Our purpose in this paper is to compare analytically based and empirically based approaches and to present the broader implications of the latter for prescriptive applications such as system design, decision aiding, and training. These implications have not been discussed systematically in the decision-making and cognitive design literatures.
The value and indeed the existence of empirically based prescription is controversial. For example, LeBoeuf and Shafir (2001) asked how NDM can "presume to offer prescription without a normative framework" (p. 374). A related question is, How can prescription be based on description without confusing "what is" with "what ought to be" (the naturalistic fallacy)? Bell et al. (1988) reserved exclusive use of the term normative for formal systems, and researchers sometimes assume that evaluative criteria must, by definition, be based on such models. This is hard to reconcile, however, with the widespread use of empirically based warrants for prescription--for example, wherever people learn through apprenticeship with more experienced colleagues and, even more broadly, wherever it has been discovered that certain decision processes achieve objectives more reliably than others.
In this article, we will compare analytically and empirically based prescription in both theory and practice. We will argue that formal normative models are not unique as warrants for prescriptive intervention and, as a result, that the way is open for cognitive engineers to explore alternative, empirically based avenues of prescriptive research and intervention. In addition, formal normative models are not sufficiently specific to warrant actual prescriptive practice (e.g., in decision aiding, training, or consulting). Thus cognitive engineers must seek alternative sources of warrant. Finally, we will describe how empirically based prescription works, respond to criticisms, and illustrate its application in systems design and the development of real-world training. We argue that empirically based normative models are a relevant and pragmatically useful source of warrant for prescriptive intervention--without denying the merits of analytically based interventions, such as decision analysis, under suitable conditions.
ANALYTICALLY BASED PRESCRIPTION
Von Winterfeldt (1999, p. 134) specified four steps for analytically based cognitive engineering: (a) Identify a normative model for a judgment or decision-making task; (b) identify obstacles in the way of implementing the normative model (e.g., systematic errors or cognitive illusions in unaided decision behavior; Kahneman et al., 1982); (c) develop tools to overcome the implementation problems; and (d) test the tools and implement them if successful. These steps highlight the independence of normative modeling (Step a) from description (Step b) and the role of both in prescriptive modeling (Step c). It also makes it clear that description has no independent justification; its purpose is simply to identify deviations from the normative model. The following discussion will focus on decision analytic consulting, but the issues we discuss are also likely to arise in training and decision-aiding applications.
Decision analysis offers a repertoire of techniques for modeling choice among options with uncertain outcomes, choice among options that differ on multiple evaluative dimensions, multilevel inference from evidence to uncertain conclusions, allocation of scarce resources, and negotiation (Keeney & Raiffa, 1976: Raiffa, 1968). A decision analysis for choice under uncertainty might include the following steps: (a) modeling the problem, which includes generating mutually exclusive options and identifying a mutually exclusive and exhaustive set of possible outcomes of the options; (b) eliciting inputs, which includes assessing the probability that each outcome will occur conditional on each of the options and the utility or degree of preference of each option-outcome combination; (c) calculating the conclusion--that is, mathematically aggregating the probabilities and utilities for each option to calculate its subjectively expected utility; (d) testing and improving robustness, which includes an optional sensitivity analysis and iterations of modeling and elicitation steps to firm up parts of the model that are found to have a strong influence on the conclusion; and (e) communicating results--that is, displaying and explaining the conclusion in such a way that decision makers are willing and able to select the option with the highest subjectively expected utility (SEU). A similar process can be used to create a Bayesian inference model (Schum, 1994) or a multi-attribute utility model (Keeney & Raiffa, 1976).
The role of decision theory (i.e., the SEU model) is to provide an analytical normative basis for the prescriptive methods just described--but to what extent does it succeed? We will pursue this question in two parts. First, in what sense is decision theory itself uniquely normatively justified? Second, how much does decision theory dictate (hence, warrant) actual decision analytic practice? In both respects it turns out that the analytically based warrant for decision analytic methods is less than meets the eye. Decision analysis, like empirically based prescription, must ultimately be warranted by descriptive considerations.
Is SEU Uniquely Normative?
Proponents often state that SEU theory uniquely defines rationality (e.g., von Winterfeldt & Edwards, 1986, p. 19). Formulation in the language of logic and mathematics sometimes lends SEU a false air of inevitability. As one commentator said, "Since mathematics [of probability and statistics] is true ... we must accept the mathematics" (Rubin, 1988, p. 293). Frequent reference in the literature to "logical consistency" for compliance with SEU constraints and "logical inconsistency" for violations adds to this impression, as does the misleading language that can be found in some of the normative arguments for SEU. According to Skyrms (cited by Kaplan, 1996, p. 159), for example, acceptance of SEU is necessary to avoid the "literal inconsistency" of deciding differently in the same gambling situation based merely on how the gamble is described.
These characterizations are not correct, however. When SEU is violated, what is in question is not compliance with standard systems of logic or mathematics but with specific axioms, or alleged first principles, of decision making, which have been adopted without logical or mathematical proof. In particular, the situations in which people "ought" to behave in the same way are not literally the same. They are different situations that are analyzed as the same from the perspective of the SEU theory, which stipulates that only the probabilities and utilities of ultimate consequences "should" matter to a decision maker (Kaplan, 1996; Schick, 1997). Slovic's (1982) defense of SEU identifies the issue more fairly: "Maximization of expected utility commands respect as a guideline for wise behavior because it is deduced from axiomatic principles that presumably would be accepted by any rational person" (p. 172).
The analytically based warrant for prescription, then, depends on the normative status of the axioms. The traditional (and perhaps still the most common) justification for them is that they appear compelling to decision makers after considered reflection, even if the same individuals do not initially behave in accordance with them. In his classic foundational book about the SEU model, Savage (1954) argued for its normative status by asking the reader to judge whether the model's axioms were compelling:
I am about to build up a highly idealized theory of the behavior of a "rational" person with respect to decisions. In doing so I will, of course, have to ask you to agree with me that such and such maxims of behavior are "rational".... So, when certain maxims are presented for your consideration, you must ask yourself whether you try to behave in accordance with them, or, to put it differently, how you would react if you noticed yourself violating them. (p. 7)
A variant of Savage's (1954) idea does not treat intuitions as the last word on rationality, but is based instead on the idea of a reflective equilibrium (or coherence) between intuitions and behavior (Goodman, 1965). Individuals adjust their decision-making behavior to fit principles they find intuitively compelling, but they may also adjust their principles to fit behavior they regard as rational. A normative model is justified for an individual if it reflects an equilibrium point in such a process of mutual adjustment.
MacCrimmon (1968) put Savage's (1954) proposition to a test, asking upper-middle-level executives to solve problems with known decision theoretic solutions and also to indicate the extent to which they endorsed Savage's axioms. After being shown the discrepancies between the axioms and their behavior and hearing a rationale for the axioms, many of the participants wanted to conform their behavior to the axioms. However, MacCrimmon did not test whether SEU axioms were different from other principles of decision making in this regard and, thus, failed to establish their unique normative status. Slovic and Tversky (1974) repeated MacCrimmon's experiment but provided rationales for either the acceptance of Savage's axioms or their rejection, and sometimes for both. The result was that decision makers were equally likely to be persuaded in the "normative" or the "nonnormative" direction. Slovic and Tversky concluded that both concrete decision-making behaviors and "uneducated" abstract judgments were irrational. In doing so, they simply assumed the normative...
|
