Home | Business News | Browse by Publication | M | Measurement and Evaluation in Counseling and Development

Estimating the reliability of a test battery composite or a test score based on weighted item scoring.

Article Excerpt
In some settings, the validity of a battery composite or a test score is enhanced by weighting some parts or items more heavily than others in the total score. This article describes methods of estimating the total score reliability coefficient when differential weights are used with items or parts.

**********

In many moderate-sized and smaller school districts, counselors are often the most knowledgeable persons regarding the technical aspects of educational and psychological measurement. Their training and experience with standardized tests typically exceeds that of classroom teachers and most administrators. Therefore, if technical questions arise with respect to the reliability and validity of grades, selection procedures for gifted education, summarizing district-wide data for school board consideration, and similar issues involving test data, counselors must often provide technical expertise.

This dependence on counselors has been magnified by the measurement demands of the No Child Left Behind (2002) legislation. In a number of states, school districts are encouraged to supplement state-mandated, standardized measures of reading comprehension and achievement in mathematics with nonstandardized local measures. Presumably, these local measures will conform more closely to the local curriculum. Also, they may break away from the pervasive use of multiple-choice questions in standardized tests, using instead a variety of constructed-response exercises, essays, and more creative forms of measurement. When this is the case, someone--often the director of counseling--must determine how scores on these heterogeneous measures should be weighted and combined. Some kind of composite score must be defined at each grade level, 3 through 8. This is necessary in order to classify each student's achievement level as "basic," "proficient," or "advanced" in reports to the state department of education.

This process of weighting and combining measures that are quantified on heterogeneous score scales is implicitly involved in course grades. Most teachers use a variety of test types, item types, and assignments in their student evaluations (Qualls, 1995). One formula or another is inherently applied in reducing student data to a course grade, whether it is numerical, alphabetical, or verbally descriptive. Very few teachers are concerned about the reliability of such a composite. However, a counselor may be called on to investigate reliability issues when a student's grade is challenged by parents or the student. If the use of standardized tests for college admission purposes is de-emphasized, greater emphasis will inevitably accrue to high school grades. The demand for studies of the trustworthiness of grades may then be expected to increase.

These observations suggest that, in the future, counselors may be called on more frequently to address reliability issues. In past decades, these issues have been the concern primarily of standardized test publishers and state testing agencies. Much of the literature bearing on reliability estimation procedures in unconventional settings has been directed toward specialists and has appeared in journals read by this group. But as our society moves toward ever-greater reliance on tests of one kind or another, it behooves counselors to become more familiar with techniques that heretofore were the province of test specialists.

One such area that is relatively simple, mathematically, is reliability estimation when the scores of interest are weighted composites of several numerical bits of information. We explicate these methods in the context of measurement of achievement because the issues and the techniques are easily illustrated in this setting. However, as noted in the foregoing discussion, these methods carry over to a variety of local settings. In whatever setting weighted scoring of items or subtests within a composite may arise, counselors may well have to provide the expertise in the technical analysis of the data.

WEIGHTING...