|
Article Excerpt
Abstract
Case research has commanded respect in the information systems (IS) discipline for at/east a decade. Notwithstanding the relevance and potential value of case studies, this methodological approach was once considered to be one of the least systematic. Toward the end of the 1980s, the issue of whether IS case research was rigorously conducted was first raised. Researchers from our field (e.g., Benbasat et al. 1987; Lee 1989) and from other disciplines (e.g., Eisenhardt 1989; Yin 1994) called for more rigor in case research and, through their recommendations, contributed to the advancement of the case study methodology. Considering these contributions, the present study seeks to determine the extent to which the field of IS has advanced in its operational use of case study method. Precisely, it investigates the level of methodological rigor in positivist IS case research conducted over the past decade. To fulfill this objective, we identified and coded 183 case articles from seven major IS journals. Evaluation attributes or criteria considered in the present review focus on three main areas, namely, design issues, data collection, and data analysis. While the level of methodological rigor has experienced modest progress with respect to some specific attributes, the overall assessed rigor is somewhat equivocal and there are still significant areas for improvement. One of the keys is to include better documentation particularly regarding issues related to the data collection and analysis processes.
Keywords: Case study research, methodological rigor, positivism, research design
Introduction
Research methods are at the basis of the production of knowledge in any given field (Pinsonneault and Kraemer 1993b). "Research methods shape the language we use to describe the world, and language shapes how we think about the world" (Benbasat and Weber 1996, p. 392). Different trends in research topics and philosophical perspectives have led to a wider diversity in research methods and, more specifically, to the emergence of qualitative methods in information systems (IS) research (Lee and Liebenau 1997; Trauth 2001; Wynn 2001). There is a growing tradition to use qualitative research approaches to study information technology (IT) phenomena (e.g., Romm and Plinski 1999; Trauth and Jessup 2000), and case study research figures among those qualitative methods that have been recognized as having gained acceptance over the past decade in the IS field (Benbasat et al. 1987; Benbasat and Weber 1996; Klein and Myers 1999; Orlikowski and Baroudi 1991).
Although there are numerous definitions, Yin (1994) defines the scope of a case study as follows:
A case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident (p. 13).
Case research is, therefore, useful when a phenomenon is broad and complex, when a holistic, in-depth investigation is needed, and when a phenomenon cannot be studied outside the context in which it occurs (Benbasat et al. 1987; Bonoma 1985; Feagin et al. 1991; Yin 1994). The case research strategy allows for a great deal of flexibility and individual variation (Cavaye 1996a). Case research, in its versatility, can be used with any philosophical perspective, be it positivist, interpretivist, or critical. It typically combines several qualitative data collection methods such as interviews, documentation, and observations, but can also include quantitative data such as questionnaires and time series.
Case research gained respect in our field for several reasons. First, the case research method is particularly well-suited to IS research, since the object of our discipline is information systems in organizations, where interest shifted to organizational rather than technical issues (Benbasat et al. 1987). Second, having access to and reporting on real-life IT experiences, case researchers allow both academia and practice to keep up with the rapid changes occurring in the IT world as well as in organizations. Third, holistic investigation, which represents a key characteristic of case research, suits well our need to understand the complex and ubiquitous interactions among organizations, technologies, and people. In this regard, the access to and use of a wide range of data collection methods, both qualitative and quantitative, bring richness and flexibility to the overall research process, making case research particularly well designed for the study of a complex phenomenon such as IT. Fourth, in-depth case investigations open the way to new ideas and new lines of reasoning and pinpoint the opportunities, challenges, and issues facing IT specialists and managers. Finally, case research is widely used for exploration and hypothesis generation, but can also be used for providing explanations and for testing hypotheses (Benbasat et al. 1987; Cavaye 1996a; Yin 1994), all of which contribute to the development of knowledge in our field.
Notwithstanding the relevance and potential value of case research in the IS field, this methodological approach was once considered to be one of the least systematic (e.g., Stone 1978). Toward the end of the 1980s, a few IS researchers, preoccupied with the rigor (or lack thereof) of case research in our field, started to evaluate the usefulness and methodological soundness of IS case studies. One of the earliest contributions was that of Benbasat et al. (1987), who surveyed the case study literature published in four journals and one major conference proceedings for the period 1981 through 1985 and evaluated the cases based on a number of guidelines associated with the case design and data collection process. They recommended that case researchers should provide clearer descriptions of where their topics fit into the knowledge building process; detail the case selection criteria; and provide more information about the data collection process. Another key contribution was made shortly after by Lee (1989), who provided an overview of, and responded to, the methodological problems involved in the study of a single case and summarized what a scientific methodology for IS case studies does, and does not, involve. Lee also demonstrated how to make controlled observations and deductions as well as how to allow for replicability and generalizability with the use of a single case.
Researchers from other disciplines have also contributed to the growth of qualitative research in general, and to case research in particular. For instance, Eisenhardt (1989) proposed a roadmap for building theories from positivist case study research and IS researchers have illustrated how this methodological framework can serve as a useful guide in conducting case research in our field (e.g., Pare and Elam 1997; Peterson et al. 2000; Sarker and Lee 2002). Other researchers have offered a series of methods to help fieldwork cycle back and forth between thinking about existing data and generating strategies for collecting new, often better, data (Kuzel et al. 1994; Mays and Pope 1995; Miles and Huberman 1994; Yin 1994). Several approaches have also been proposed to code, organize, and analyze qualitative data (Miles and Huberman 1994; Seidel and Kelle 1995). Finally, a number of software packages have been developed and commercialized to support the work of qualitative researchers (Fielding and Lee 1998; Kelle 1995; Weitzman and Miles 1995).
In this article, the primary question that we attempt to answer is, "To what extent has published positivist case research in IS actually adopted and implemented the attributes that leading case research methodologists have identified as contributing to rigor in such research? The extent to which actual, published positivist case research has, or has not, embraced these attributes is a necessary factor for us to consider when assessing the extent to which published positivist case research in IS has, or has not, achieved rigor. Achieving a higher level of rigor is required in scientific research. If positivist case research in IS wants to be considered a premier-class methodology and to add to the growth of knowledge in our field, it has to pass the tests of scientific rigor.
Since standards of quality vary with the assumptions of each philosophical tradition (Anderson et al. 2001; Jensen and Rodgers 2001; Klein and Meyers 1999), it was decided to restrict our assessment to positivist case studies. Two main reasons motivated this decision. First, positivist case research, which includes both descriptive and theoretically grounded case studies (Orlikowski and Baroudi 1991), represents the dominant paradigm in IS case research (see the next section). Second, early recommendations formulated by researchers are most suitable to the positivist paradigm. In this regard, Klein and Myers (1999, p. 68) posit that the principles or guidelines formulated by Benbasat et al. (1987), Lee (1989), and Yin (1994) have become de facto standard in positivist case studies.
The present review helps identify trends and patterns in our use of case study research and, hence, serves as an instrument to reflect, as a research community, on our progress. It also allows us to pinpoint areas where more work needs to be done. In short, our intent is not to achieve any goal as grandiose as a definitive assessment of any specific case article or positivist case research in general, but instead to gauge the extent to which positivist case research in IS is taking advantage of, or ignoring, the valuable methodological insights or guidelines of leading case methodologists.
A major finding of our exhaustive, empirical examination of published positivist case studies over the period 1990 through 1999 is that a large portion of them have actually ignored the state of the art of case research methods that have been readily available to them. Notable examples are that only 42 percent of all case study articles in our database have posed clear research questions; only 58 percent have provided information about their data collection methods; and only 23 percent have elucidated their data analysis process. These and other important attributes of rigorous positivist case research are no secret. We believe the widespread practice in which many positivist case researchers have ignored fundamental attributes of the state of the art of case research methods to be nothing short of disappointing.
The paper is organized as follows. The following section describes the research method, namely how the journals and articles were selected along with the basic characteristics of the articles assembled. Next, we present the attributes used to assess the rigor of positivist case research; the coding process is then explained. This is followed by a presentation and discussion of the results of our analysis. The last section summarizes our findings and presents our final recommendations.
Journal and Article Selection
Journal selection was undertaken with careful consideration. Our intent was to assure a wide representation of case research conducted in IS. In making this selection, we used those journals considered in the review by Benbasat et al. (1987) as well as the evaluations of top-ranked journals by Hardgrave and Walstrom (1997) and by Whitman et al. (1999). As a result, we selected the following seven major IS journals: European Journal of Information Systems; Information & Management; Information and Organization (formerly Accounting, Management and Information Technology); Information Systems Research; Information, Technology & People; Journal of Management Information Systems; and Management Information Systems Quarterly. (2, 3)
Each journal review started with the table of contents, and extended to the abstracts, the keywords, and the articles themselves. Each candidate article was compared to a set of characteristics of case research aligned with both Yin's (1994) definition (presented above) and Benbasat et al.'s list of key characteristics of case study research:
* a contemporary phenomenon is examined in a real-life context or setting
* one or few entities (person, group, organization, technology) are examined
* the complexity of the unit is studied intensively
* the phenomenon of interest is not isolated from its context, especially at the data analysis stage
* no controlled observation that involves manipulation is involved
Although case research shares many characteristics with other qualitative methodologies (e.g., natural setting as source of data; researcher as key instrument of data collection; data collected as words; focus on participants' perspectives), the objectives and challenges inherent to each tradition, as well as the criteria by which to judge their quality, are quite different (Cresswell 1998; Lau 1997). For this reason, we carefully excluded those studies that made use of other related qualitative methodological approaches, such as grounded theory (e.g., Macredi and Sandom 1999), ethnography (e.g., Orlikowski 1991), and action research (e.g., Braa and Vidgen 1999).
Field studies conducted in one or a few organizational settings (e.g., Rowe and Struck 1999; Wijayanayake and Higa 1999) where quantitative data (e.g., questionnaires, time series, Q-sort) are gathered and analyzed in isolation from the organizational context were also excluded from the present review.
The initial selection process produced a total of 261 articles using case research methodology for the period 1990 through 1999. A preliminary assessment of this group of articles proves itself to be interesting and, hence, deserves some attention. All 261 studies were classified according to journal names and publication dates. As shown in Table 1, the number of case articles differs widely from journal to journal over the period 1990 through 1999. A relatively small number of case study articles were published in Information Systems Research (6 percent) while about one out of ten articles (11 percent) published in Information and Management4 and Journal of MIS used the case research approach. On the other hand, in Information Technology & People, MIS Quarterly, European Journal of Information Systems, and Information and Organization, more than one out of five published articles used the case study methodology (29 percent, 26 percent, 23 percent, and 21 percent, respectively).
Despite the fact that the actual number of case research articles published in the selected journals slightly increased over the last decade, we observe in Table 2 that the actual proportion of published articles in IS journals using case research has been relatively stable (around 15 percent), with the notable exception of 1996. The proportion of case study articles published in the 1990s has also been stable based on comparisons with previous reviews in the field. For instance, Hamilton and Ives (1982) observed that 12 percent of all articles published in 15 journals (IS and non-IS) between 1970 and 1979 used case research methodology. For their part, Vogel and Wetherbe (1984) reported that approximately 15 percent (5) of the articles published between 1977 and 1983 in 15 journals (IS and non-IS) were case study articles. More recently, Orlikowski and Baroudi (1991) analyzed a total of 155 articles from four major IS journals and observed that approximately 14 percent of all articles published between 1983 and 1988 used a case study approach.
Among the total number of case articles included in our database, 51 articles did not use case research as a primary methodology. In most of these studies, authors typically propose new and innovative systems development approaches and briefly illustrate them in real-life settings (e.g., Clemens and Weber 1998; Elofson and Konsynski 1991; Zviran 1990). In other studies, a field survey or an experiment is first conducted (considered as the primary method) followed by the presentation of one or several short illustrative case studies (e.g., Cox and Ghoneim 1996; Reinig et al. Nunamaker 1997-1998). Given the small amount or complete lack of information related to the case method in these studies, it was decided not to consider them in subsequent analyses, leaving us with a population of 210 "pure" IS case study articles (see Tables 1 and 2).
Next, we classified these articles as positivist, interpretive, or critical using criteria proposed by Cavaye (1996a), Devers (1999); Orlikowski and Baroudi (1991), and Yin (1994). The three philosophical traditions differ mainly in terms of their assumptions about the sources and development of knowledge, the nature of physical and social reality, and the relationship between theory and practice (Orlikowski and Baroudi 1991). Following Orlikowski and Baroudi, we first found it useful to distinguish within the positivist paradigm those studies where researchers were working within a theoretical perspective from those where the researchers' intentions were merely descriptive. On the one hand, in descriptive case research, investigators attempt no theoretical interpretation of the phenomena; rather, they present what they believe to be straightforward, objective, factual accounts of events to illustrate some issue of interest. Similar to Orlikowski and Baroudi, we used a single criterion for considering a case study as descriptive or illustrative based on what the investigators had implicitly or explicitly stated in their exposition.
Based on the work of Devers (1999), of Lincoln and Guba (1985), and of Orlikowski and Baroudi (1991), we then identified the criteria to classify a theoretically-grounded case study as being positivist. The adoption of a positivist perspective is accompanied by a broad commitment to the idea that the social sciences should emulate the natural sciences (Lee 1989). Epistemologically, positivist studies are premised on the existence of a priori fixed relationships within phenomena capable of being identified and tested via hypothetico-deductive logic and analysis. Causal relationships, which are the basis for generalized knowledge, can predict patterns of behavior across situations. Ontologically, positivist research assumes an objective physical and social world that exists independently of humans. The researcher is seen to play a passive, neutral role, and does not intervene in the phenomenon of interest. Keeping in spirit with this set of beliefs, a theoretically grounded positivist case study is likely to be conducted with the ideas of establishing appropriate measures (qualitative and/or quantitative) for the constructs being studied; establishing or testing causal relationships; determining the domain to which the study's findings can be generalized; and demonstrating that the inquiry is value-free. The criteria for judging the quality of such positivist studies, in opposition to interpretive and critical case studies, are related to the traditional validity and reliability tests used in the natural sciences (Yin 1994). Specifically, the primary criteria for classifying a theoretically-grounded case article as positivist were the following:
* adoption of a positivist perspective clearly stated in the study
* evidence of formal research hypotheses or propositions
* evidence of qualitative and/or quantitative measures of variables or constructs
* explicit purpose of theory testing or theory building
* concern for validity...
|
