|
Article Excerpt
ABSTRACT
THIS PAPER DESCRIBES A general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application-oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on our study of scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The study illustrates that bibliometrics is much more than conducting citation analyses based on the indexes produced by the Institute for Scientific Information (ISI), since citation data do not play a role in the study. Interaction with scholars in the fields under consideration and openness in the presentation of the quantitative outcomes are the basic features of the methodology. Bibliometrics should be used as an instrument to create a mirror. While not a direct reflection, this study provides a thorough analysis of how scholars in the humanities and social sciences structure their activities and their research output. This structure can be examined empirically from the point of view of its consistency and the degree of consensus among scholars. Relevant issues can be raised that are worth considering in more detail in followup studies, and conclusions from our empirical materials may illuminate such issues. We argue that the principal aim of the development and application of bibliometric indicators is to stimulate a debate among scholars in the field under investigation on the nature of scholarly quality, its principal dimensions, and operationalizations. This aim provides a criterion of "productivity" of the development process. We further contend that librarians are not infrequently requested to provide assistance in collecting data related to research performance assessments, and that the methodology described in the paper aims at offering a general framework for such activities, and can be used by librarians as a line of action whenever they become involved.
1. INTRODUCTION
The study presented in this paper focused on the fundamental questions: How does one recognize a "good" scholar? How does one recognize an "important" scholarly contribution? The approach adopted in this study can be defined as bibliometric. It aims at identifying characteristics of scholarly publications that can validly be assumed to reflect the "quality" or "importance" of a scholar or a scholarly work. Therefore, a first answer to the question "How does one recognize a `good' scholar?" is: One should examine his or her scholarly publications. In other words, in a bibliometric approach, it is assumed that important contributions to scholarly progress are sooner or later communicated in scholarly publications. This is considered to be a universal characteristic of scholarly development in natural sciences, life sciences, social sciences, and humanities.
A bibliometric approach is a quantitative approach. It attempts to calculate statistics of quantitative aspects derived from scholarly publications. Bibliometric indicators result from the statistical analysis of bibliographic information retrieved from the scholarly literature. This determines both their strength and their limitations. The strength of the bibliometric method is that, once established, it can be applied in a uniform or objective manner, eliminating the influence of subjective or personal factors. On the other hand, being a statistical method, it cannot take into account all particularities or special features of the objects to be assessed. As a consequence, bibliometric data should always be applied in combination with qualitative knowledge about the scholars involved and the subdisciplines in which they are active.
Bibliometric indicators have been successfully applied in many subdisciplines in the natural and life sciences. Data from the Science Citation Index (SCI), produced by the Institute for Scientific Information (ISI), play an important role in analyses of research performance in these subdisciplines (e.g., van Raan, 1996; Van Den Berghe et al., 1998). Thus far, the social sciences and humanities have not often been subjected to such analyses. At the same time, the academic authorities of many universities have expressed the need to obtain an insight into the research performance of all faculties and in all fields of scholarship.
Fundamental differences exist between the natural and life sciences, on the one hand, and the humanities and social sciences, on the other hand, with respect to the research object, the methodologies applied, and the structure of scholarly communication. As a result, those who are involved in the development of performance indicators for the humanities and social sciences are confronted with the following situation.
Firstly, they need to develop methodological tools to assist evaluation agencies or policymakers in carrying out their tasks, in the same way that the current SCI-based methodologies provide supplementary research assessment tools in the natural and life sciences. Secondly, this methodology should take into account the characteristics of the field of scholarship, the nature of the scholarly research object, and particularly the communication practices among scholars and the structure of the communication system in their fields. A study dealing with these challenges can indeed be considered an endeavour.
This paper attempts to describe a general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on a study on scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The background, setup, and methodological framework are presented in Section 2. It is followed by a concise review in Section 3 of earlier studies on research performance in this field of scholarship.
Section 4 presents a number of characteristic outcomes of the study. Its principal aim is to illustrate the methodology outlined in Section 2. A detailed overview of the study is presented in a research report by Luwel et al. (1999). The study illustrates that bibliometrics is much more than conducting citation analyses based on the ISI citation indexes, as citation data do not play a role in this study.
Finally, Section 5 gives a critical discussion of the methodology, in the light of the experiences collected in the study. This discussion includes a short overview of the comments of scholars and of the followup of our study, and summarizes the main features of our methodology.
2. BACKGROUND, SETUP, AND METHODOLOGICAL FRAMEWORK
2.1 Background and Setup of the Study
The study presented in this paper was a pilot study commissioned by the Flemish Inter-University Council (VLIR) for developing a methodology to assess research performance in the social sciences and humanities. The disciplines selected by the VLIR for this study were Law and Linguistics. This paper discusses only the study on Law. The Catholic University of Leuven, the University of Gent, the Flemish-speaking Free University of Brussels, and the University of Antwerp decided to participate in the study, which was partially funded by a grant approved by the Flemish Minister-President, who is also in charge of science and technology policy.
At the start of the project, in early 1997, a project team was set up, consisting of the research staff of the VLIR, researchers of the Centre for Science and Technology Studies (CWTS) at Leiden University (the Netherlands), and a staff member of the Science and Innovation administration of the Ministry of the Flemish Community.
The activity of the project team was supported by a university expert group, set up for each of the two disciplines, and composed of senior academic staff members of the departments of Law and Linguistics at the four participating universities. The expert groups assisted the project team during the elaboration of the project, and played an active role in their respective universities in a series of activities, such as data collection, the development of classification systems, and commenting on drafts of the final report.
The first stage of the project work plan was a clear boundary setting of the two disciplines. For the Law faculties, this operation was relatively simple. Academic staff data were extracted from the universities' central administration databases, including year of birth, gender, starting and ending date of their appointments, rank, length of appointment (e.g., 40 percent, that is, two days a week), funding source, and year of Ph.D. granting.
In a subsequent phase, a questionnaire was prepared to collect quantitative data. It was sent to all researchers, both junior and senior, active in one of the four Flemish universities at the end of the year 1996. The members of the two expert groups were of the opinion that the study should not be limited to research activities only, but that all academic activities should be taken into account, analyzing also the fraction of work dedicated to research. The most important data, analyzed in this paper, were lists of publications.
In tandem with the analysis of the first questionnaire, a second questionnaire was elaborated, again in collaboration with the expert groups, in order to collect more qualitative information on leading publications,journals, publishers, and Flemish scholars in the two disciplines. The main objective was to obtain insight into scholars' perceptions on scholarly work quality, and to assess to what extent the scholars' opinions corroborated the outcomes of the quantitative indicators. For Law, this questionnaire was sent to professors working at Flemish, Dutch, and Belgian French-speaking universities, as well as to Belgian senior magistrates. Respondents were asked to indicate "outstanding," "good, yet not outstanding," and "less...
|
