|
Article Excerpt
Rising concern about the quality of medical care and preventable medical errors has increased interest in how systems of care operate. Health care organizations can shape the quality of care through the selection of clinical staff or educational programs for patients. Influencing clinician behavior, however, is arguably the most important way in which organizations affect care (Flood 1994; Landon, Wilson, and Cleary 1998). Organizations can influence clinicians using financial incentives, management strategies (e.g., utilization review, guidelines, profiling), structural arrangements (e.g., presence of particular facilities or domains of expertise, governance structures), and normative practice styles or organizational cultures.
Studies of organizational influences on the quality of care require measures of organizational characteristics that are rarely, if ever, recorded in a standardized way. Organizational data are commonly collected by surveying informants about their organizations. Surveys often ask for factual data such as the number of FTE medical staff or whether professionals with particular specialties are on site. They can also ask about subjective phenomena, such as an organization's culture or mission. Recent examples include Kralewski et al. (2000), who gathered data on revenue sources and methods of physician compensation from clinic medical directors or administrators, and Meterko, Mohr, and Young (2004), who measured hospital culture by surveying hospital employees.
Lazarsfeld and Menzel (1980) distinguish "global" and "analytical" organizational survey measures. Global measures refer to organization-level properties such as size or centralization of decision making. "Analytical" measures are organization-level averages of respondent-level data, such as the proportion of clinicians who are board certified in infectious diseases.
High reliability is necessary but not sufficient for the validity of measurement (Bohrnstedt 1983). Imprecise measurement (low reliability) will sometimes lead investigators to incorrect conclusions about relationships between an organizational factor and outcome measures of interest. Nonetheless, few organizational studies examine the reliability of informant reports.
If informant reliability is low, relying on a single informant per organization may be unwise. Just as using multiple-item scales can improve respondent-level survey measures, combining reports from multiple informants may raise reliability for organizational measurements. Assessing measure reliability can offer guidance about the number of informants needed to adequately measure different organizational properties.
When organizations are the objects of measurement, studies usually can select among several possible informants, so researchers must decide which informants to approach. Standard advice is to seek out informants who are knowledgeable, motivated, and unbiased (Huber and Power 1985). Managerial or administrative informants are often chosen on the assumption that they have good access to information. Such informants, however, also may tend to present the organization positively (Seidler 1974). Studies rarely examine differences in descriptions of an organization between types of informants (e.g., medical directors and physicians).
This article addresses issues of measure reliability and differences across informant types using data from a national study of medical clinics, the Evaluation of Quality Improvement for HIV (EQHIV) study. That study gathered data about clinic characteristics from the clinic director and several clinicians in each practice studied. It asked about implementation and assessment of improvement initiatives, HIV care priorities, and barriers to improvement. We examine the reliability of single-informant organizational measures based on individual survey items as well as multiple-item scales, and how reliability can be improved by using multiple informants. We also calculate the number of informants required to obtain reliable organization-level measures, and assess clinician-director differences in descriptions of a clinic.
ASSESSING RELIABILITY
Several health care studies have used surveys or interviews with informants to measure organizational characteristics. Studies relying on data from a single informant per organization have examined effects of group practice and payment methods on costs of care (Kralewski et al. 2000) and effects of care management processes on the quality of care (Casalino et al. 2003). Other studies used multiple informants in assessing organizational characteristics and performance in intensive care units (Shortell et al. 1991), long-term care teams (Temkin-Greener et al. 2004), and hospitals (Shortell et al. 1995; Aiken and Sloane 1997; Aiken and Patrician 2000; Meterko et al. 2004). Studies have used both single-item organizational measures (e.g. Aiken and Sloane 1997), and multiple-item scales (e.g. Shortell et al. 1991).
Multiple-informant studies often present one-way analyses of variance (ANOVA) of informant reports classified by organization to support combining informant assessments into organization-level measures. A statistically significant Fratio in such an analysis indicates a nonrandom resemblance in reports by informants within a given organization, but does not directly measure the extent of resemblance. The F ratio is sometimes supplemented by the correlation ratio [[eta].sup.2], equivalent to the coefficient of determination ([R.sup.2]) for regressing an informant report on a set of indicator variables for organizational differences. Like [R.sup.2], [[eta].sup.2] can be misleadingly large when there are many indicator variables relative to the total number of reports.
Bohrnstedt (1983) generically defines the reliability of a measure as the ratio of true-score variance to total variance, or alternately as the complement of the ratio of error to total variance:
[[rho].sub.Measure] = [[sigma].sup.2.sub.Trus]/ [[sigma].sup.2.sub.Measure] = [[sigma].sup.2.True]/ [[sigma].sup.2.sub.True] + [[sigma].sup.2.sub.Error] = 1 - [[sigma].sup.2.sub.Error]/[[sigma].sup.2.sub.Measure]. (1)
The last expression in (1) shows that reliability is low when error variance is large relative to total variance. The next-to-last expression shows that reliability is also low if variation in a phenomenon ([[sigma].sup.2.sub.True]) is limited within a given study population.
When several informant reports are available, it is common to use their average
[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (2)
as an organization-level measure. In (2) [r.sub.jh] is the report of informant h about organization j and [n.sub.j] is the number of informants for organization j; J is the number of organizations and N = [[summation of].sup.J.sub.j=1] [n.sub.j] is the total number of informants. If [n.sub.j] = 1, (2) is the report [r.sub.jh] of a single informant. The measurement [r.sub.jh] may be a scale averaging K items [x.sub.kjh]; if K = 1, [r.sub.jh] is a single item.
When [r.sub.jh] is a scale score, two potential sources of error variation in (2) are distinguishable, measurement error in [r.sub.jh] and error because of informant differences in [R.sub.jh]. Since the object of measurement is the organization, (2) is reliable when organizational variability is high relative to these sources of error. Likewise, the informant-level measure [r.sub.jh] is affected by organizational and informant differences as well as errors of measurement. Assuming that these sources are independent, the variance [[sigma].sup.2.sub.r] of [r.sub.jh] is
[[sigma].sup.2.sub.r] = [[sigma].sup.2.sub.o] + [[sigma].sup.2.sub.i + [[sigma].sup.2.sub.e] (3)
where [[sigma].sup.2.sub.o], [[sigma].sup.2.sub.e] and [[sigma].sup.2.sub.e] refer, respectively, to organizational, informants-within-organizations, and error components of variance. The variance [[sigma].sup.2.sub.r] of the...
|
