Home | Business News | Browse by Publication | A | Academic Exchange Quarterly

Building inter-rater reliability into performance assessment.

Article Excerpt
Abstract

The purpose of this paper is to examine issues ranging from strategic aspects of assessment planning to the more fundamental aspects of assessment such as measuring student evaluation processes. Assessment activities in the capstone course of a college of business were used as the focus of a study on inter-rater reliability among instructors and as a foundation for a discussion on assessment processes. The details of establishing inter-rater reliability for one instrument are provided, and conclusions are drawn concerning the value and limitations of this type of performance assessment.

Introduction

The main driver behind outcomes assessment in schools of business has been the accountability movement (Apostolou, 1999; Burke & Modarresi, 1999; Henninger, 1994; Kimmell, Marquette, & Olsen, 1998). Employers have been unhappy with the basic skills, particularly communication skills, of university graduates. Legislatures then have sought to hold universities accountable for student learning. Accreditation agencies, such as AACSB, are consequently held responsible for ensuring that business schools engage in assessment. Much effort has gone into what exactly business school administrators must do with assessment in order to get accredited (Kerby & Weber, 2000; Sinning & Dykxhoorn, 2001).

Accountability v. Assessment While accountability has provided a motivation behind assessment, it is important to distinguish the two concepts. Accountability refers to indicators used as a basis for resource allocation across divisions or for individual incentives. Assessment refers to indicators used for continuous improvement. Accountability indicators provide a means for outsiders to evaluate the quality of an institution or program, while assessment indicators provide a means for insiders to strengthen quality within the institution. Outcomes assessment has been lauded by some as a means to simultaneously provide a basis for both program improvement and accountability (McCoy & Chamberlain, 1994). More recently, however, there has been a recognition of the danger of confusing assessment processes with accountability indicators. To be effective, the tools required for each are different (Wellman, 2000). Accreditation bodies are looking for evidence that assessment data is being systematically collected and, more importantly, being widely disseminated and used effectively (Kimmell, Marquette & Olsen, 1998). Alternatives, such as seeking quantitative criteria from all universities on a series of standardized tests, for example, would probably do little to improve student learning and might instead actually impair student preparation for future careers (Henninger, 1994).

Apostolou's (1999) thorough review of the assessment literature relevant to accounting programs suggests that there is no one best assessment measure and that multiple measures of outcomes should be used to obtain feedback on programs. Most research on the validity and reliability of assessment measures has focused on general education outcomes. Very little has been published on the successful development of instruments to measure learning in business programs specifically....