|
Article Excerpt
Measurement reliability represents a major issue in assessment for decision-making in employment. Inherent in much of the literature on rater training, assessment, and performance appraisal, is the notion that ratings should reflect the norms and values of a particular organization (Lievens, 2001) such that the training of assessors should be tailored to the needs and values of specific organizations. One particular method of assessment that has been the subject of concern for over two decades is the assessment centre (AC) process (see Arthur, Day, McNelly, & Edens, 2003; Woehr & Arthur, 2003, for reviews). ACs commonly exhibit poor construct validity (Hough & Oswald, 2000), and several authors have suggested that measurement inadequacies may contribute to this (e.g. Lievens, 1998). Therefore, reliability should be a primary focus in efforts to maximize the potential for ACs to exhibit construct-valid ratings.
Given that reliability is fundamental to measurement generally, further explorations of methods that encourage improved precision in AC ratings could help to strengthen the bases on which employment decisions are grounded. Frame of reference (FOR) training has received much attention in terms of its potential to increase the dependability of ratings in performance appraisal settings, in comparison to more traditional behavioural or rater-error oriented approaches (see Pulakos, 1986). Lievens (2001) and Schleicher, Day, Mayes, and Riggio (2002) found improvements in rater reliability using this method. However, these improvements were generally small in terms of overall magnitude.
The use of signal detection theory (SDT) has been suggested by Lievens (2001) following Lord (1985). SDT, although well known within cognitive or perceptual research, presents a paradigm that is largely unexplored with respect to the assessment of the performance of raters in an AC. An SDT approach may help to supplement strategies, for example FOR training, used to isolate idiosyncratic rater behaviour, and may provide a more in-depth analysis of rater peculiarities than traditional indices such as interrater agreement. In this study, we introduce and investigate the usefulness of SDT for the evaluation of the performance of assessors in a real world AC, albeit in a somewhat different manner to that suggested by Lievens (2001) and by Lord (1985).
Theoretical overview
When applied to the realm of human performance, SDT quantifies the sensitivity, or discriminability, of a person making a judgment about the presence or absence of, or a change in, some item or quality of interest. In the parlance of SDT, the item or quality to be detected is referred to as the 'signal'. This nomenclature harks back to the roots of SDT in statistical decision theory applied to the detection of faint auditory (sonar) or visual (radar) signals during the Second World War (Peterson, Birdsall, & Fox, 1954; Swets, 1973; Swets & Green, 1978). SDT also allows for an independent and simultaneous evaluation of the response tendencies of the observer, who may be more or less conservative or liberal; that is, more or less inclined to take risks or to avoid risks. In line with the application of the theory to the performance of electronic receivers detecting radio signals against a noisy background, anything that is not the signal, but which may potentially be mistaken for it, is classified as 'noise'. In order to apply detection theory it must be possible to classify the observations upon which judgments are offered as being instances of either signal, or of noise.
There must also be occasions when an observer is uncertain as to whether an observation is an instance of 'signal' or 'noise'. The detection of suitable candidates from amongst a pool of job applicants potentially fits these requirements.
Readers unfamiliar with SDT will find information concerning its use in applied settings in a comprehensive classic text by Swets and Pickett (1982), and in a recent article by Harvey (2003). An in-depth consideration of various SDT models is provided in Macmillan and Creelman (2005). A barrier to the use of SDT in applied settings has been the investment of time and computational expertise required to obtain dependable measures of discriminability and response bias, although several algorithms have been published (Appendix 6, Macmillan & Creelman, 2005, provides a basic program). Now, however, user-friendly software is freely available on the World Wide Web, including RcorePlus 5.3.2. (Harvey, 2002) which is well-documented and builds upon earlier software by Dorfman and Aft (1969). Further, Harvey (1992) showed that, once measures of discriminability and response bias have been obtained, an extension of SDT allows for the calculation of the likelihood that an expert's (e.g. an AC assessor's) future classification of an event as 'signal' will be correct. Such knowledge can be used to decide what weighting to give to proffered advice, and whether to attempt to increase the expert's discriminability, or to moderate the expert's response tendencies.
Assessors in ACs are confronted with job applicants who may, or may not, subsequently prove to be a good fit to the needs of the organization. During an AC, candidates present a variety of evidence in favour of their suitability (Thornton, 1992). The two aspects of assessor performance previously alluded to are of importance under these conditions. The first is sensitivity to the evidence of suitability provided by the applicants, which underlies an ability to discriminate between suitable and unsuitable candidates. The second is the nature of the decision about what to do on the basis of the perceived evidence. Two assessors with identical discriminability to the qualities displayed by an applicant may nevertheless differ as to whether those qualities are of a sufficient magnitude to warrant a particular rating on the assessment. When evaluating the performance of assessors it would be useful to have objective measurements of both the accuracy of their perceptions, and the degree to which...
|
