|
Article Excerpt
Abstract
One result of the increasing sophistication and complexity of MIS theory and research is the number of studies hypothesizing and testing for moderation effects. A review of the MIS and broader management literatures suggests researchers investigating moderated relationships often commit one or more errors falling into three broad categories: inappropriate use or interpretation of statistics, misalignment of research design with phenomena of interest, and measurement or scaling issues. Examples of nine common errors are presented. Commission of these errors is expected to yield literatures characterized by mixed results at best, and thoroughly erroneous results at worse. Procedures representing examples of best practice and reporting guidelines are provided to help MIS investigators avoid or minimize these errors.
Keywords: Tests of moderation, contingency models, PLS
Introduction
Lee (2001) argued that the contribution many university researchers make to the MIS field is "scrupulous attention" to scientific methods, using largely quantitatively and statistically based approaches. MIS researchers have recently focused on improving the quantitative methods employed.
For example, MIS researchers investigated methodological issues in experiments (Jarvenpaa et al. 1985), highlighted problems of statistical power (Baroudi and Orlikowski 1989), questioned model complexity (Lee et al. 1997), and examined the rigor with which instruments are validated (Boudreau et al. 2001). The goal of this paper is to sensitize MIS researchers to methodological issues surrounding tests of moderated relationships.
Three types of relationships dominate MIS research: simple linear or additive relationships, mediated relationships (typically sequences of linear relationships), and moderator relationships. Moderator relationships are the most interesting and perhaps the most difficult of the three to establish empirically (McClelland and Judd 1993). A review of recent MIS research reveals an increasing interest in moderated relationships. From 1991 through 2000, MIS Quarterly, Information Systems Research, and Journal of Management Information Systems published 26 articles directly testing moderated relationships (see Appendix A). MIS Quarterly and Information Systems Research had 17 articles suggesting but not testing moderation in the same 10-year period.
The increasing interest in moderated relationships reinforces a notion that MIS researchers are increasingly addressing: context matters in MIS research. Relevant contexts include organizational, technological, and individual. For example, researchers investigating technology acceptance have incorporated individual contexts such as personal innovativeness (Agarwal and Prasad 1998), work experience and gender (Venkatesh and Morris 2000) and yielded a richer understanding of the phenomenon of interest.
This paper critically assesses moderation tests performed by MIS researchers. We hope to raise awareness about common errors and enhance the craftsmanship of moderation testing by providing a central summary of nine common errors. While these errors have been separately identified elsewhere, this is the first attempt to synthesize and assess the extent to which MIS researchers are prone to their commission. Some of these errors, while generally understood, still occur frequently. Others are less well understood and occur with great regularity. Importantly, the increasingly popular use of partial least squares (PLS) applications (Gefen et al. 2000) has been accompanied by an introduction of a new error as well as reintroduction of some old errors.
We critically assess moderation tests in the sample of 26 articles published from 1991 through 2000, identifying three general types of errors labeled inappropriate statistics, misalignment of phenomena and research design, and measurement issues. Nine specific errors were distinguished, although not all studies reported enough information to determine whether an error occurred. Descriptions of these errors and methods of avoiding them should help MIS investigators advance theory and practice by minimizing Type I and Type II errors in tests of moderation.
We first review various conceptual definitions of moderation, then present three sets of common difficulties encountered when searching for moderation in MIS research and ways to avoid them. Analysis of select articles is presented to demonstrate error commission, potential consequences, and illustrations of best research practice. We conclude by recommending reporting guidelines to improve the thoroughness with which authors report moderation-related evidence and enhance the ability of readers and reviewers to evaluate tests of moderation.
Definitions
Review of moderation definitions revealed what at first appeared to be an unsettlingly high level of variation. Fortunately, evidence supporting the presence of virtually all conceptualizations of moderation in applied behavioral field research can be assessed using hierarchical moderated multiple regression (MMR, Saunders 1956) to test [H.sub.0]: [delta][R.sup.2.sub.mult]-[R.sup.2.sub.add]=0 using at least squares procedures (ordinary or PLS), where:
Equation 1 Y = [b.sub.0] + [b.sub.1]X + [b.sub.2]Z; [R.sup.2.sub.add]
Equation 2 Y = [b.sub.0] + [b.sub.1]X + [b.sub.2]Z + [b.sub.3]XZ; [R.sup.2.sub.mult]
An F statistic derived using Equation 3 that is significantly greater than 1.00 leads to rejection of [H.sub.0]: [delta][R.sup.2] = and the conclusion that either Z moderates the X [right arrow] Y relationship or X moderates the Z [right arrow] Y relationship.(2) Using this procedure, large values of [delta][R.sup.2] occur when any one of a number of conceptualizations of moderation occurs.
Equation 3
[F.sub[df.sub.mult]-[df.sub.(add.sup.1)]N-[df.sub.mult]-1]= [delta][R.sup.2]/([df.sub.mult]-[df.sub.add])/(1-[R.sup.2.sub.mult])/ (N-[df.sub.mult]-1)
Definitions of moderation provided in the literature are summarized in Table 1. Of particular note is Arnold's (1982, 1984, amplified by Baron and Kenny 1986) distinction between circumstances where the strength of the X [right arrow] Y relationship varies as a function of Z versus the nature of the X [right arrow] Y relationship varies as a function of Z. The former is often referred to as differential validity while the latter is referred to as differential prediction.(3) The distinction between these two types is important as differential prediction is the form of moderation appropriately tested for using MMR. The definition of moderation applied in this study is that of differential prediction, where the nature of the X [right arrow] Y relationship varies as a function of Z.
MIS researchers are not consistent in their moderation conceptualizations. For example, a number of MIS investigators incorrectly use differential validity and differential prediction interchangeably. Four articles in our sample included language describing moderation as differences in strength of the X [right arrow] Y relationship and differences in the nature of the X [right arrow] Y relationship (Devaraj and Kohli 2000; Hardgrave et al. 1999; Harrison et al. 1997; McKeen et al. 1994). By way of illustration, McKeen et al. (1994) stated they examined whether "the strength of the participation-satisfaction relationship depended on the level of" (p. 427) task complexity and other moderators. However, these authors did not report differences in strength of participation-satisfaction (i.e., [r.sub.participation-satisfaction]) across levels of task complexity, instead reporting differences in the nature or slope of the participation-satisfaction relationship across levels of task complexity.
Importantly, insight into underlying processes behind moderation is most likely to result from qualitative research efforts aimed at adding meaning to abstract relationships found in quantitative research. Such efforts will be most justified when empirical evidence suggests the presence of an underlying moderation process. In field studies using random effects designs (by far the dominant research design used in applied behavior research), MMR procedures and recent PLS variants constitute the dominant method of detecting moderation effects (Aiken and West 1991). The nine common errors discussed below address interpretations of MMR and PLS results used to test the definition of moderation described above.
Nine Common Errors
Unfortunately, even a casual reader of research in MIS, organizational behavior, human resources management, organizational theory, and strategy can find examples of ill-advised or outright inappropriate research methods in studies examining moderation effects. Examination of the MIS research generated a list of nine common errors that cause severe problems. These are summarized in Table 2 and grouped into three categories based on our views of underlying similarities: (1) inappropriate use or interpretation of statistics, (2) misalignment of phenomena and research design, and (3) measurement or scaling issues.
In identifying illustrations from our sample, we soon discovered that reporting standards in MIS do not routinely include enough information to assess commission of these errors. For most errors we summarize information reported that contributed to our evaluation of the likelihood an error was committed. Appendix A summarizes each article's assessment.
Inappropriate Use or Interpretation of Statistics
Solutions to problems in this first category are fairly straightforward: investigators should appropriately use and interpret statistical procedures. Examples from the literature are used to describe two problems and solutions in this category.
Error 1: Interpreting [b.sub.3] Instead of [delta][R.sup.2]
Arithmetically, test statistics regarding [H.sub.0]: [b.sub.3] = and [H.sub.0]: [delta][R.sup.2] = parallel one another and always yield the same conclusions. While this is true about the test statistics, the population parameters [delta][[rho].sup.2] and [[Beta].sub.3] are generally not parallel or equal representations of moderator effect size. In fact, [delta][R.sup.2] and [b.sub.3] are only equal when the XZ interaction is measured without error and the variance of Y [S.sup.2.sub.y] is equal to the variance of the product term [S.sup.2.sub.xz]. [delta] [R.sup.2] and [b.sub.3] are not generally even linearly related. Only the sample estimate [delta] [R.sup.2] is a reflection of moderator effect size.
Chin et al. (1996) recently noted that,
in addition to the change in [R.sup.2], the estimated beta for the interaction term provides additional information regarding the interaction effect. This estimate informs us as to how much a unit change in the moderator variable Z would change the regression relationship of Y on X. (p. 22)
Unfortunately, when X, Y, and Z are measured on interval scales, the units of measurement are arbitrary. Change in the X-Y relationship associated with a unit change in Z can be artificially inflated or deflated by simply changing Z's scale of measurement. Further, multicolinearity between X, Z, and the XZ product term causes additional [b.sub.3] distortion.(4)
After making this incorrect assertion, Chin et al. focused on [b.sub.3] estimates in reviewing 70 MIS studies reporting tests of moderation since 1980. In their Table 2 summarizing studies using regression and path analytic techniques, Chin et al. reported [b.sub.3] terms as evidence of moderator effect size and concluded that
the literature consistently reported moderators with a small effect size, beta averaging 0.10, suggesting that moderating terms play only a small part for understanding information systems issues. (p. 23)
In fact, as [b.sub.3] is not an indicator of moderator effect size, no conclusion can be drawn about the role moderators play in understanding information systems issues. Chin et al. could have formed a conclusion about the role of moderators if they had summarized [delta][R.sup.2] across studies.
Unfortunately, Chin et al. may have been limited by the information reported in their studies and unable to draw strong conclusions about the role of moderators in MIS research. Only seven articles (27 percent) in our sample actually reported [delta][R.sup.2]. In one best practice example, Harrington (1996) investigated moderating effects of denial of responsibility on codes of ethics and their relationship to computer abuse judgments and intentions. Her analysis included not only a calculation but a discussion of [delta][R.sup.2] effect size.
Solution. Investigators must use [delta][R.sup.2] to draw conclusions about relative moderator effect sizes; use of [b.sub.3] will lead to spurious conclusions.
Error 2: Interpreting [b.sub.1] and [b.sub.2] When X and Z are Interval Scale Measures
Error 2 occurs when X and Z are measured on interval scales and investigators attempt to interpret [b.sub.1] and [b.sub.2] in Equation 2. There are two potential problems with interpreting these statistics: variability due to linear transformation and/or confounding main and moderating effects.
To our knowledge, Schmidt (1973, footnote 4) first noted [b.sub.1] and [b.sub.2] could vary greatly after linear transformations of X and Z. If X and Z are measured using interval scales, the information contained in those measures remains unchanged when a constant is added to or subtracted from them or they are multiplied or divided by a constant--all linear transformations of X and Y are equally legitimate...
|
