|
Article Excerpt
ABSTRACT
THIS ESSAY PROVIDES AN OVERVIEW of journal evaluation indicators. It highlights the strengths and weaknesses of different indicators, together with their range of applicability. The definition of a "quality journal," different notions of impact factors, the meaning of ranking journals, and possible biases in citation databases are also discussed. Attention is given to using the journal impact in evaluation studies.
The quality of a journal is a multifaceted notion. Journals can be evaluated for different purposes, and hence the results of such evaluation exercises can be quite different depending on the indicator(s) used. The impact factor, in one of its versions, is probably the most used indicator when it comes to gauging the visibility of a journal on the research front. Generalized impact factors, over periods longer than the traditional two years, are better indicators for the long-term value of a journal. As with all evaluation studies, care must be exercised when considering journal impact factors as a quality indicator. It seems best to use a whole battery of indicators (including several impact factors) and to change this group of indicators depending on the purpose of the evaluation study. Nowadays it goes without saying that special attention is paid to e-journals and specific indicators for this type of journal.
INTRODUCTION
Few model-based approaches to journal evaluation can be found in the literature. A descriptive, but not explanatory model is the one used by the Leiden-based Centre for Science and Technology Studies (Tijssen & van Raan, 1990). Perhaps this overview will inspire fellow scientists to construct an overall model explaining observed journal citation scores, and hence lead to a better understanding of their role in institutional and national evaluations. Theoretical issues dealt with in this article are restricted to giving precise formulations of indicators, in particular of the journal impact factor. No input-output model or explanation of dependent variables, such as journal citation counts, as a function of one or more independent variables (e.g., number of journals in the field or number of active scientists) is provided.
The study of the use and relative impact of scientific journals is an important application of citation analysis. Yet citations are only one aspect of a journal evaluation exercise. Indeed, journal evaluation can be performed with many purposes in mind. Impact factors measure only the (international) use of journals on the research front. Hence, they are of little direct use to a (special) librarian, because, as Line (1977) notes: Users of journals read, but many actually publish little or nothing at all. In this context, it is important to investigate the relation between in-house use and citation use. This has been done, for example, by Ming-yueh Tsay (1998, 1999) in a medical library. Numerous studies have shown that older volumes of scientific journals are less frequently used (read as well as cited) than more recent volumes. This phenomenon is generally described by the term "obsolescence" (Brookes, 1970; Line, 1993). A mathematical model describing the relation between the growth of the literature and obsolescence can be found in Egghe & Rousseau (2000).
It should also be pointed out that scientists read not only as a step in their scientific investigations, but also to keep informed of the latest findings in their field, or simply out of general interest. Further, the importance of scientific journals is not restricted to use (local or international). Geographic penetration in the sense of geographical distribution patterns of subscribers, authors, and citers, as well as the correlations between them, is still another indicator. Irene Wormell (1998) performed such an investigation of geographical distributions for the following journals: College & Research Libraries, Computer Journal, Information Processing & Management, Journal of Documentation, Journal of the American Society for Information Science, Libri, and Scientometrics. Studies like this one tell us whether international journals are really international in scope and impact. Among the journals considered by Wormell, Libri turned out to be the most international one, while College & Research Libraries is a very nationally oriented (i.e., U.S.) journal.
Many people are interested in journal evaluations: Librarians, scientists, science evaluators, publishers, etc. Librarians are interested in journal evaluations and local circulation data for selection and deselection purposes, and ill the relation between impact and price (Van Hooydonk et al., 1994; Van Hooydonk, 1995; Abbott, 1999). Scientists want to find the most appropriate journal in which to publish their results. Funding agencies and governments want their grantees to publish in the most prestigious journals (Pao & Goffman, 1990; Lewison & Dawson, 1997). Editors and publishers may relate high citation scores to a successful editorial practice and policy. Commercial publishers are interested in subscription data and sales. Information brokers are interested in finding those sources that have the most potential of satisfying their clients' needs. University research councils use journal impact and prestige scores as elements in local research evaluation studies in view of enlarging the visibility of the university's research.
Because economic indicators such as subscription data are essential for commercial publishers, an investigation, such as Peritz's (1995), of the relation between these and citation data is of great value. Let us just mention that, in most instances, Peritz found correlations between 0.25 and 0.5.
Besides serving as an archive for research findings, scholarly printed journals also provide professional, institutional, and disciplinary visibility, as well as recognition and prestige, to scientific authors. This, in turn, provides prestige to the journals themselves. Complex systems of "pecking orders" are based on the ranking of journals and a journal's position in them. The quality of the editorial board counts for much, of course, but the typography, quality of the paper used, quality of the illustrations, etc. all play their role. A truly excellent journal regularly garners papers from well-established authors and secures a larger number of institutional and individual subscriptions, thus making for a solid financial (economic) base.
The next sections cover the following topics: The definition of a quality journal, different definitions of impact factors, a general model for the citation distribution, electronic journals, the meaning of ranking journals, possible biases in citation databases, and how to use the journal impact in evaluation studies.
QUALITY JOURNALS
HOW has a quality journal been defined, what are the elements in such a definition, and how have they been used in practice? As early as 1970, Zwemer published the following list of characteristics of a "good journal":
1. High standards for acceptance of manuscripts (results must be based on new scientific information, reliable methods, adequate controls, and statistical treatment of data);
2. Having a broadly representative editorial board with appropriate representation of subdisciplines;
3. The editor uses a critical refereeing system;
4. Promptness of publication;
5. Being covered by major abstracting and indexing services;
6. Scientists using the articles published in the journal have a high confidence level in its contents;
7. Having a high frequency of citation by other journals.
These seven criteria are also among those used by the Philadelphia-based Institute of Scientific Information (ISI) to determine inclusion (or exclusion) of journals in their database (Garfield, 1990; Testa, 1998). The ISI management further mentions the following requirements:
8. Including abstracts or summaries in English;
9. Including authors' addresses;
10. Providing complete bibliographic information.
For new journals the reputation of the publisher and of the main editor is a good indicator of the possible importance or quality of the journal. If, for example, Elsevier, the American Chemical Society, or the IEEE launches a new journal, this will probably be a more important one than the newly established "Research Reviews of the Department of ... of the ... University."
Panels of (subject) experts have acted as judges to determine the value of journals and to draw formal ranked lists (Van Fleet, McWilliams, & Siegel, 2000). This approach is especially useful in the social sciences and humanities where the Science Citation Index (SCI) and Journal Citation Reports (JCR) cannot be used, and where local journals are often important. This is due to the local character of the investigations, as is the case in (national) law, or the literature or linguistics of small languages (Luwel et al., 1999).
Depending on the purpose and the type of journal, different journal indicators may be determined. Popular science journals, such as Scientific American, Dr. Dobb'sJournal, and the New Scientist, are only marginally interested in impact factors. Besides practicing good (science) journalism, the number of subscriptions and corresponding revenues is what really counts for such journals.
The number of interlibrary lending (ILL) requests is still another local "use" indicator. Indeed, if a library does not subscribe to a journal, the librarian cannot directly determine its local use. In that case the number of local ILL requests for that journal can act as an indicator of its importance for the community served by the library.
Finally, a quality journal is indexed by many databases. Hence, the number of databases indexing this journal can be used as an indicator of its importance. However, as sheer numbers are not very important here, it is probably more relevant to investigate whether a scientific journal is covered by the most important database (s) in the field.
CITATION...
|
