|
Article Excerpt
Clustering effect and non-zero intraclass correlation produce the variability that can be observed in meta-analytic data. Statistical modeling is shown to provide a flexible approach to capturing and describing this empirical variability. The most common multilevel approach to analyzing data from meta-analysis is illustrated in this article with a set of 177 studies of behavioral therapies.
In this article, we present a multilevel (or hierarchical linear) model that illustrates issues in the application of the model to data from meta-analytic studies. In doing so, several issues are discussed that typically arise in the course of a meta-analysis. These include the presence of non-zero between-study variability, how multiple effect sizes per studies can be combined into a single analysis, and an examination of model misfit and robustness in an attempt to understand the effects of treatments on multiple outcomes. Our purpose is to explain the model and illustrate analytic strategies with a well-known data set. Examples of computer code are provided in the Appendix to facilitate the work of researchers who have an interest in meta-analysis.
The results reported in this study are based on a set of 177 studies (containing 454 effect sizes) of systematic desensitization, behavior modification, and cognitive-behavioral (CB) comparisons from the original study by Smith, Glass, and Miller (1980). (Data were obtained from G. E. Matt, San Diego State University, and used with the permission of Gene Glass, Arizona State University. Only effect sizes that could be reconstructed from recorded means and standard deviations were used. Data analyses are for the purpose of illustration only and are not intended to constitute a reanalysis of the Smith-Glass data.) Specifically, the tools of multilevel linear and nonlinear modeling are shown (a) to provide a more useful description of the data than do traditional techniques and (b) to bring the data (more specifically residuals) more into line with theoretical expectations. On the basis of this analysis of the Smith et al. data, it is concluded that interventions based on CB theory have been moderately effective in improving outcomes, whereas interventions based on behavior modification and systematic desensitization may have had negligible effects.
THEORETICAL FRAMEWORK
Cronbach (1982) conceptualized empirical sampling variability in terms of four population domains: persons (U) assigned to the treatments, treatments (T) themselves, measurements of outcome (O), and the historical-social setting (S) in which a treatment takes place. As observed by Becker (1996), this theoretical framework has a great deal of potential to inform the process of generalization in meta-analysis as well as to guide the development of statistical models. Classical statistical inference is based solely on taking samples of individuals (u) from a population (U). Generalizations about treatment efficacy are thus made to U on the basis of u (for short, this is designated as u [right arrow] U). Yet any type of actual treatment intervention would show some variability in implementation (t, for treatment), and measured outcome (o, for observation), and historical setting (s). In meta-analysis, studies typically represent unique utos combinations that must be combined to make a generalization; that is, studies represent different samplings from the UTOS facets. Given this framework, choosing a hypothesis to examine is equivalent to a choice of how to generalize to UTOS. For example, if global conclusions are sought about the efficacy of psychotherapy intervention, then the domain of generalization is defined by utoS [right arrow] UTOS for a set of studies conducted in the same setting. (It is probably true that for studies conducted across broader social contexts and longer periods of time, the setting itself changes; yet the desired inference is most likely to the current setting.) More specific questions can also be asked, such as what the effect of an intervention is for a particular class outcome of outcome measure (sub-O) or a particular type of psychotherapy (sub-T).
Note that choosing a relatively homogenous set of treatment interventions is synonymous with reducing the empirical sampling variability in t and may also have the consequence of reducing variability in u and o as well. With a broad range of conditions within facets, it would be more difficult to make the case for combining effect sizes. For example, Hedges and Olkin (1985) argued that effect sizes "that are derived from dissimilar measures [o] should rarely, if ever, be combined" (p. 211). In a nutshell, these are apples-and-oranges problems. Should outcomes of self-esteem be combined with outcomes of anxiety? However broadly or narrowly the boundaries of UTOS facets are drawn, the question of global effect is nonetheless persistent among both researchers and policy makers....
|
