|
Article Excerpt
Despite two recent meta-analyses (Latham & Sue-Chan, 1999; Taylor & Small, 2002) that have investigated the criterion-related validity of the situational interview (SI) and the past-behaviour interview (PBD, there remains debate as to the criterion-related validity of each of these structured interview types in predicting job performance--particularly with respect to the domain of managerial job performance (see Huffcutt, Weekley, Welsher, DeGroot, & Jones, 2001; Pulakos & Schmitt, 1995; Taylor & Small, 2002). There are three problems with past research that have given rise to the debate surrounding the criterion-related validity of the SI and PBI. First, there are few studies that have concomitantly examined the relation of the SI and PBI with managerial performance in an actual selection context--a fact that has, of course, attenuated the ability to build consensus regarding the relative predictive validity of the SI versus the PBI. Second, similarly designed studies that have investigated the SI, the PBI or both continue to produce conflicting results. For example, some past research has found that the SI is a valid predictor of job performance (Campion, Campion, & Hudson, 1994; Day & Carroll, 2003; Latham & Skarlicki, 1995; Latham & Sue-Chan, 1999) and is superior to the PBI in predicting job performance (Latham & Sue-Chan, 1999). Yet other past research has shown that the SI is a less valid predictor of job performance than the PBI for a range of job types (Taylor & Small, 2002) and that the SI actually shows zero validity in predicting performance in higher-level jobs (Huffcutt et al., 2001; Pulakos & Schmitt, 1995). Third, close inspection of the studies that have investigated the validity of the SI and the PBI in predicting managerial job performance shows that there are several methodological deficiencies and design limitations present that may have affected the accuracy or generality of the observed results (see Latham & Sue-Chan, 1999; discussed further below).
Additionally, questions persist regarding the construct validity of both the SI and the PBI (see Conway & Peneno, 1999; Cook, Vance, & Spector, 2000; Huffcutt, Roth, & McDaniel, 1996). Indeed, few studies have comprehensively examined construct differences between the SI and the PBI and very few studies have examined a priori-specified construct differences between the SI and the PBI as means of explaining observed criterion-related validity differences. More generally, Posthuma, Morgeson, and Campion (2002) support the need for additional research on the constructs measured by the employment interview.
Given the issues outlined above, the overall goals of the current study were to (a) give a clearer picture as to the criterion-related and incremental validity of the SI versus the PBI in predicting managerial job performance by improving upon methodological flaws and design limitations in past research and (b) examine relations of the SI and the PBI with numerous facets of cognitive ability, personality and work-sample tests in order to illuminate construct differences between the SI and the PBI that may help to explain any observed differences in criterion-related validity. The potential to provide more accurate and generalizable findings regarding the relative criterion-related and incremental validity of the SI and the PBI is a contribution to a relatively thin literature comparing the validity of differently structured interview types in predicting managerial job performance. Moreover, within the context of a more rigorously conducted test of criterion-related validity, any potential construct differences that are observed between the SI and the PBI may allow substantive inferences to be drawn with regard to why criterion-related validity differences were observed between the SI and the PBI--particularly, in the present case, with respect to managerial positions. The current study went a step beyond past research not only in examining a wide range of construct relations, but also in examining the relation of specific managerially relevant facets of these constructs with the SI and the PBI. As stated by Huffcutt et al. (1996), research that serves to explicate the constructs that are captured by structured interview types may provide 'the next major breakthrough in construct research' (p. 470). In terms of applied contributions, possessing information about which constructs are being tapped by different interview types may help organizations to choose the most appropriate interview format, depending on the specific predictors the organization is interested in assessing (Jell', 1999; Taylor & Small, 2002).
Description of the SI and the PBI
The SI and PBI are two of the most commonly used structured interview types for selecting employees (Campion, Palmer, & Campion, 1997). The SI requires applicants to provide responses to hypothetical dilemmas that might be encountered in the job in question (Latham, Saari, Pursell, & Campion, 1980), whereas the PBI requires that applicants recall past experiences relevant to the types of tasks and situations likely to be encountered on the job (e.g. Janz, 1982). The SI has roots in goal-setting theory in that behavioural intentions (i.e. goals), as expressed in response to hypothetical scenarios, are thought to be the immediate precursors to actions (Latham & Sue-Chan, 1996), whereas the PBI is based on the premise that the best predictor of future behaviour in a given context is past behaviour exhibited in a similar context (Janz, 1989). It is recognized that Janz labelled his interview format the 'patterned behaviour description interview' (PBDI); however, as has been the norm in past research, we use the label PBI to refer, more generally, to structured interviews using the past-behaviour format. Similarly, although we try to be faithful to Latham et al.'s (1980) format for designing future-oriented situational interview questions, we recognize that some may argue that we are only employing situational interview 'types' of questions.
Comparing the predictive validity of the SI and PBI
In directly comparing the validity of the SI and PBI, care must be taken to ensure the respective interviews are developed, administered and scored in as similar a manner as possible while still preserving the essential and distinctive aspects of these two formats. Holding constant such elements of the research design helps increase one's confidence that any observed differences in validity are due to the presumed core differences in the SI and PBI formats (i.e. the focus on hypothetical responses [SI] versus actual past behaviours [PBI]) rather than developmental or procedural artifacts. Indeed, Latham and Sue-Chan (1999) noted that past research comparing the criterion-related validity of the SI and the PBI has been confounded by (a) a failure to rate each SI question separately as opposed to giving a global rating after the completion of the interview (e.g. Pulakos & Schmitt, 1995), (b) a failure to include a scoring guide for the PBI (e.g. Latham & Scarlicki, 1995), (c) a failure to tap the same content areas (e.g. the interviewee's willingness to learn, team orientation etc.) in SI and PBI questions (e.g. Campion et al., 1994) and (d) a failure to properly operationalize the SI by neglecting to use a true situational 'dilemma' that helps to disguise the most correct answers (e.g. Campion et al., 1994; Pulakos & Schmitt, 1995).
In order to address limitations of past research that resulted in inequitable comparisons of the SI and the PBI, the current study sought to create SI and PBI measures that were as similar as possible. Accordingly, the PBI and SI interviews were strictly developed as prescribed in the literature. Further, the SI and the PBI were designed to assess identical managerial skill dimensions. Measuring the same managerial skill dimensions in each interview facilitated the use of the same behaviourally anchored scoring guide for each interview. Finally, the current study required that assessors rate each managerial skill dimension in each interview immediately after the relevant question, as opposed to making ratings after all the managerial skill dimensions have been assessed.
In addition to the above-mentioned methodological problems, past research that has directly compared the SI and the PBI has used concurrent rather than predictive validity designs (Campion et al., 1994; Huffcutt et al., 2001; Pulakos & Schmitt, 1995), non-employee (Conway & Peneno, 1999; Day & Carroll, 2003) or non-managerial (Campion et al., 1994; Huffcutt et al., 2001; Latham & Scarlicki, 1995; Pulakos & Schmitt, 1995) populations, or small sample sizes (Huffcutt et al., 2001; Latham & Scarlicki, 1995). As is well known, the use of concurrent validity designs and non-employee populations can limit the ability to generalize to applicant samples, whereas extremely small sample sizes are more likely to be unrepresentative and result in reduced power. Accordingly, the current research used a predictive validity design to directly compare the validity of the SI and the PBI within a relatively large sample of applicants to high-level managerial positions.
Predictive validity hypotheses
Although past studies have shown that the PBI predicts job performance more strongly than the SI in samples consisting of applicants for higher-level positions (Huffcutt et al., 2001; Pulakos & Schmitt, 1995; Taylor & Small, 2002), methodological flaws or design limitations may have contributed to the less than impressive results for the SI (see Latham & Sue-Chart, 1999). Because the current study attempts to correct for methodological flaws and design limitations of past studies, it is expected that:
Hypothesis 1a. Both the SI and PBI will show a positive and significant relation with managerial job performance.
However, with regards to examining the relative predictive validity of the SI and the PBI, it is instructive to consider the distinction between 'will do' and 'can do' determinants of job performance. 'Will do' determinants of performance refer to individuals' motivation or willingness to expend effort in their jobs, whereas 'can do' determinants of performance refer to individuals' capacity for performing the job (for reviews, see Blumberg & Pringle, 1982; Campbell, 1990). Taylor and Small (2002) suggested that, because of its focus on hypothetical, future-oriented scenarios rather than actual past performance, the SI predominantly taps the 'can do' performance determinants and may not adequately reflect the 'will do' determinants. In contrast, by requiring applicants to provide examples of actual past behaviours that are reflective of their day-to-day motivation to apply relevant knowledge, skills and abilities to the job, the PBI is purported to more adequately capture both the 'can do' and 'will do' determinants of job performance (Taylor & Small, 2002). Thus, it is expected that:
Hypothesis 1b. The PBI will...
|
