|
Article Excerpt
Determining the nature of work activities is necessary before occupational and organizational psychology can design interventions that improve the performance of both individuals and organizations. The role and importance of using job and task analysis techniques are well understood and have been reviewed extensively (e.g. Harvey, 1991; Kirwan & Ainsworth, 1992). Despite this, there has been an upsurge of publications over the last decade that refer to 'cognitive task analysis' (Militello & Hutton, 1998; O'Hare, Wiggins, Williams, & Wong, 1998; Schraagen, Chipman, & Shalin, 2000; Vicente, 1999). This term is used to emphasize the importance of identifying the cognitive activities underpinning work performance, as opposed to traditional approaches that specify the 'prescribed task' (Leplat, 1987), the logically optimal means of carrying out the task, or the observable task activities. One reason for the increased interest in analysing cognitions is that in this knowledge-intensive society there is a need to elicit knowledge from subject-matter experts in order to design and develop new training and automatic manufacturing control systems. (For reviews of knowledge elicitation techniques, see Cooke, 1994; Fowlkes, Salas, Baker, Cannon-Bowers, & Stout, 2000; Hoffman, Shadbolt, Burton, & Klein, 1995.) Also, despite the increasing use of technology at work and the downsizing of the workforce in manufacturing industry, the role of cognition has increased in many of those jobs that remain. A few personnel are now responsible for monitoring, controlling and troubleshooting complex plant and equipment. Ironically, despite automation, their jobs have become more difficult because they are less involved directly with the industrial processes and are required to intervene infrequently but in more critical ways (Bainbridge, 1987). Therefore the recent literature on cognitive task analysis reflects a desire to develop more powerful techniques capable of identifying workers' cognitions, such that the risk of potentially costly and catastrophic human error in industrial and medical contexts can be reduced. This need is recognised by Bainbridge (1990):
There are many complex jobs in which the outcome of thinking does not emerge in observable action. For example, one can think out a plan of action, assess it, and decide it is inadequate for the purpose, or one can work out the implications of a situation, and memorise the decision for use later, If we want to train and support these types of work, then we need information about these processes. (p. 161)
The aim of this paper is to discuss one form of cognitive task analysis, termed 'process tracing', that has been employed to examine performance in a wide range of work situations, including managerial decision making (Marmaras, Lioukas, & Laios, 1992); the use of hypertext (Mao & Benbasat, 1998); the reasoning of medical physicians (Hasselbrock & Prietula, 1992); the problem-solving strategies of process control operators (e.g. Decortis, 1993; Patrick, Grainger, et al., 1999); and decision making in the control of a blast furnace (Hoc, 1989), a paper mill (Schaafstal, 1991), and a submarine (Kirschenbaum, 1992).
Despite the increasing popularity of process tracing, serious concerns and criticisms need to be raised about this methodology. First, there is so much variability in how process tracing is carried out that it is difficult to delineate the nature and boundaries of this methodology. This is reflected in a review by Woods (1992) that distinguishes verbal reports and behavioural protocols as two main process-tracing methodologies. However, these do not constitute complete methodologies, but rather refer only to types of data, as they do not indicate how the data are collected, coded, structured and represented. The question therefore arises as to what are the stages or steps that constitute a process-tracing method. A second criticism, made by Blackman and Nelson (1988) and reiterated by Doherty (1993) in the context of research into naturalistic decision making, is that many studies do not describe fully the procedures used to infer underlying cognitive processes, and in some cases, even the data acquisition methods. Consequently, this paper attempts to specify the generic stages of process tracing, discussing how each may be accomplished by citing examples from the literature. However before this, the overall aim of process tracing needs to be defined and the type that is the subject of the present paper needs to be differentiated from other forms.
The nature and goal of process tracing
The aim and nature of process tracing are described by Woods (1992) as:
To map out how the incident unfolded including available cues, those cues actually noted by participants, and participants' interpretation in both the immediate and in the larger institutional and professional contexts. This is called a process tracing or protocol analysis method because it focuses on how a given outcome came about. (p. 5)
The unique feature of this form of process tracing is that the analyst is developing a dynamic account of a person's thinking and reasoning in relation to an evolving work situation or incident that may occur in the control of, e.g., an industrial plant or fire. In these situations not only does information become available to, or is requested by, the person/team, but information also changes due to variations in the work environment. Identifying the person or team's cognitions in these situations is both important and difficult (Bainbridge, 1990; Moray, 1998; Woods, 1992) because these work situations occur infrequently, and when they do, there is little overt behaviour to record (Sanderson, Verhage, & Fuld, 1989). Also within a complex environment, such as a process control plant, there may be a variety of equally valid work strategies that can be employed to achieve the same work goal. Process tracing adopts an ideographic analytical approach that produces a qualitative description of how a person tackles some dynamic work task, which can be compared against how others deal with similar situations. In addition to this qualitative approach it is often desirable to develop generalizations about work performance that can be quantified. This can be achieved by selecting measures or categories from these qualitative descriptions on which data from a number of persons can be collated, analysed and evaluated using traditional statistical techniques. This is discussed later in this paper.
The form of process tracing in scope to the present paper can be differentiated from two other variants of this methodology. First, process tracing is cited frequently in the decision-making literature (e.g. Covey & Lovie, 1998; Ford, Schmitt, Schectman, Hults, & Doherty, 1989; Svenson, 1979). In these studies the sequence in which the decision maker searches an information board, sometimes supplemented by verbal report and eye movement data, is used to infer the nature of decision making and to test theoretical models. Although the information unfolds over time, it is not dynamic and subject to change, unlike the dynamic work situations that are the focus of this paper. Secondly, Chi (1997) discusses the steps involved in analysing verbal reports from persons involved in learning and comprehension situations. Although there is some similarity with the generic stages discussed below, as Chi points out, her goal is to identify the nature of the explanations and types of knowledge used in these situations. In contrast, we are concerned with making a person's chain of reasoning explicit rather than the representation of knowledge underpinning this process. Therefore the type of process tracing in scope to the present paper produces a temporal account of the nature of a person's major cognitive activities plotted against the available information from an evolving work situation. This not only enables these covert processes to become more transparent but also facilitates their evaluation together with the development of interventions such as training, selection or redesign.
The specific purpose and nature of each process-tracing application can vary even though the overall aim of process tracing is the same. Frequently the purpose is to identify the reasoning characteristics and strategies of either experts or experienced personnel in a range of work situations. For example, Reed and Johnson (1993) examined diagnosis of faults in computer hardware by using think aloud protocols, observation and directed interviews; Kirschenbaum (1992) employed a submarine search scenario to understand the decision making of naval officers, particularly with respect to their information gathering activities; Eekhout and Rouse (1981) studied the performance of seven crews in coping with failures in a high fidelity simulator of a supertanker control room using a mixture of verbal protocols, computer log of events, interviews, questionnaires and observation. The purpose of some process-tracing studies is to not only pinpoint workers' strategies but also to identify their lack of effectiveness or efficiency in order to develop training programmes. Thus Patrick, Gregov, Halliday, Handley, and O'Reilly (1999) used concurrent verbal reports and video recording of actions to identify process controller's strategies for dealing with unusual plant situations and to implement and evaluate subsequent training. Alternatively the purpose may be to design cooperative work and aiding systems (e.g. Holmstrom, Volden, & Endestad, 1993; Roth, Bennett, & Woods, 1987).
Many studies that involve process tracing are multi-purpose. Thus the purpose of Pew, Miller, and Feehrer's (1981) classic retrospective analysis of four nuclear incidents at Prairie Island, North Anna, Oconee and Oyster Creek was to improve decision making via personnel changes, training and computerized support systems. Process tracing was used to develop a timeline of an operator's knowledge/belief state, intention, expectation, and decision/action in relation to evolving work situations for each nuclear incident. However to produce recommendations for improvement, other techniques were used including an expert panel to assess decisions and actions and the potential impact...
|
