Citation Analysis In Research Evaluation

This book is written for members of the scholarly research community, and for persons involved in research evaluation and research policy. More specifically, it is directed towards the following four main groups of readers: – All scientists and scholars who have been or will be subjected to a quantitative assessment of research performance using citation analysis. – Research policy makers and managers who wish to become conversant with the basic features of citation analysis, and about its potentialities and limitations. – Members of peer review committees and other evaluators, who consider the use of citation analysis as a tool in their assessments. – Practitioners and students in the field of quantitative science and technology studies, informetrics, and library and information science. Citation analysis involves the construction and application of a series of indicators of the ‘impact’, ‘influence’ or ‘quality’ of scholarly work, derived from citation data, i.e. data on references cited in footnotes or bibliographies of scholarly research publications. Such indicators are applied both in the study of scholarly communication and in the assessment of research performance. The term ‘scholarly’ comprises all domains of science and scholarship, including not only those fields that are normally denoted as science – the natural and life sciences, mathematical and technical sciences – but also social sciences and humanities.

This book is written for members of the scholarly research community, and for persons involved in research evaluation and research policy. More specifically, it is directed towards the following four main groups of readers: – All scientists and scholars who have been or will be subjected to a quantitative assessment of research performance using citation analysis. – Research policy makers and managers who wish to become conversant with the basic features of citation analysis, and about its potentialities and limitations. – Members of peer review committees and other evaluators, who consider the use of citation analysis as a tool in their assessments. – Practitioners and students in the field of quantitative science and technology studies, informetrics, and library and information science. Citation analysis involves the construction and application of a series of indicators of the ‘impact’, ‘influence’ or ‘quality’ of scholarly work, derived from citation data, i.e. data on references cited in footnotes or bibliographies of scholarly research publications. Such indicators are applied both in the study of scholarly communication and in the assessment of research performance. The term ‘scholarly’ comprises all domains of science and scholarship, including not only those fields that are normally denoted as science – the natural and life sciences, mathematical and technical sciences – but also social sciences and humanities.

Traducción parcial de la Introducción: "En la actualidad, la evaluación de la investigaciones es una cuestión que se está replanteando en todo el mundo. En algunos casos, los trabajos de investigación están generando resultados muy buenos, en la mayoría de los casos los resultados son mediocres, y en algunos casos negativos. Por todo esto, la evaluación de los resultados de la investigación se convierte en una condición sine qua non. Cuando el número de investigadores eran menos, eran los propios colegas de profesión quienes evaluaban la investigación. Con el paso del tiempo, el número de investigadores aumentó, las áreas de investigación proliferaron, los resultados de la investigación se multiplicaron. La tendencia continuó y después de la Segunda Guerra Mundial, la investigación comenzó a crecer exponencialmente. Hoy en día, incluso en una estimación moderada hay alrededor de más de un millón de investigadores y producen más de dos millón de trabajos de investigación y otros documentos por año. En este contexto, la evaluación de la investigación es una cuestión de primera importancia. Para cualquier promoción, acreditación, premio y beca puede haber decenas o cientos de nominados. De entre éstos, seleccionar el mejor candidato es una cuestión difícil de determinar. Las evaluaciones inter pares en muchos casos están demostrando ser subjetivas. En 1963 se crea Science Citation Index (SCI) que cubre la literatura científica desde 1961. Unos años después, Eugene Garfield, fundador del SCI, preparó una lista de los 50 autores científicos más citados basándose en las citas que recibía el trabajo de un autor por parte de los trabajos de otros colegas de investigación. El documento titulado "¿Pueden predecirse los ganadores del Premio Nobel? 'Fue publicado en 1968 (Garfield y Malin, 1968). En el siguiente año es decir, 1969, dos científicos que figuran en la lista, por ejemplo, Derek HR Barton y Murray Gell-Mann recibieron el codiciado premio. Esto reivindicó la utilidad del análisis de citas. Cada año, varios científicos pertenecientes al campo de la Física, Química, Fisiología y Medicina reciben el Premio Nobel. De esta manera el análisis de citas se convirtió en una herramienta útil. Sin embargo, el análisis de citas siempre tuvo críticas y múltiples fallas. Incluso Garfield comentó - "El Uso del análisis de citas de los trabajos de evaluación es una tarea difícil. Existen muchas posibilidades de error '(Garfiled, 1983). Para la evaluación de la investigación, se necesitaban algunos otros indicadores. El análisis de citas, junto con la revisión por pares garantiza el mejor juicio en innumerables casos. Pero se necesita algo que sea más exacto. La llegada de la World Wide Web (WWW) brindó la oportunidad; pues un buen número de indicadores se están generando a partir de los datos disponibles en la WWW". (Trad. Julio Alonso Arévalo. Univ. Salamanca).

Citation analysis—the exploration of reference patterns in the scholarly and scientific literature—has long been applied in a number of social sciences to study research impact, knowledge flows, and knowledge networks. It has important information science applications as well, particularly in knowledge representation and in information retrieval. Recent years have seen a burgeoning interest in citation analysis to help address research, management, or information service issues such as university rankings, research evaluation, or knowledge domain visualization. This renewed and growing interest stems from significant improvements in the availability and accessibility of digital bibliographic data (both citation and full text) and of relevant computer technologies. The former provides large amounts of data and the latter the necessary tools for researchers to conduct new types of large-scale citation analysis, even without special access to special data collections. Exciting new developments are emerging this way in many aspects of citation analysis. This book critically examines both theory and practical techniques of citation network analysis and visualization, one of the two main types of citation analysis (the other being evaluative citation analysis). To set the context for its main theme, the book begins with a discussion of the foundations of citation analysis in general, including an overview of what can and what cannot be done with citation analysis (Chapter 1). An in-depth examination of the generally accepted steps and procedures for citation network analysis follows, including the concepts and techniques that are associated with each step (Chapter 2). Individual issues that are particularly important in citation network analysis are then scrutinized, namely: field delineation and data sources for citation analysis (Chapter 3); disambiguation of names and references (Chapter 4); and visualization of citation networks (Chapter 5). Sufficient technical detail is provided in each chapter so the book can serve as a practical how-to guide to conducting citation network analysis and visualization studies. While the discussion of most of the topics in this book applies to all types of citation analysis, the structure of the text and the details of procedures, examples, and tools covered here are geared to citation network analysis rather than evaluative citation analysis. This conscious choice was based on the authors’ observation that, compared to evaluative citation analysis, citation network analysis has not been covered nearly as well by dedicated books, despite the fact that it has not been subject to nearly as much severe criticism and has been substantially enriched in recent years with new theory and techniques from research areas such as network science, social network analysis, or information visualization.

Why bibliometrics is useful for understanding the global dynamics of science but generate perverse effects when applied inappropriately in research evaluation and university rankings. The research evaluation market is booming. “Ranking,” “metrics,” “h-index,” and “impact factors” are reigning buzzwords. Government and research administrators want to evaluate everything—teachers, professors, training programs, universities—using quantitative indicators. Among the tools used to measure “research excellence,” bibliometrics—aggregate data on publications and citations—has become dominant. Bibliometrics is hailed as an “objective” measure of research quality, a quantitative measure more useful than “subjective” and intuitive evaluation methods such as peer review that have been used since scientific papers were first published in the seventeenth century. In this book, Yves Gingras offers a spirited argument against an unquestioning reliance on bibliometrics as an indicator of research quality. Gingras shows that bibliometric rankings have no real scientific validity, rarely measuring what they pretend to. Although the study of publication and citation patterns, at the proper scales, can yield insights on the global dynamics of science over time, ill-defined quantitative indicators often generate perverse and unintended effects on the direction of research. Moreover, abuse of bibliometrics occurs when data is manipulated to boost rankings. Gingras looks at the politics of evaluation and argues that using numbers can be a way to control scientists and diminish their autonomy in the evaluation process. Proposing precise criteria for establishing the validity of indicators at a given scale of analysis, Gingras questions why universities are so eager to let invalid indicators influence their research strategy.

This book is an authoritative handbook of current topics, technologies and methodological approaches that may be used for the study of scholarly impact. The included methods cover a range of fields such as statistical sciences, scientific visualization, network analysis, text mining, and information retrieval. The techniques and tools enable researchers to investigate metric phenomena and to assess scholarly impact in new ways. Each chapter offers an introduction to the selected topic and outlines how the topic, technology or methodological approach may be applied to metrics-related research. Comprehensive and up-to-date, Measuring Scholarly Impact: Methods and Practice is designed for researchers and scholars interested in informetrics, scientometrics, and text mining. The hands-on perspective is also beneficial to advanced-level students in fields from computer science and statistics to information science.

Policy makers, academic administrators, scholars, and members of the public are clamoring for indicators of the value and reach of research. The question of how to quantify the impact and importance of research and scholarly output, from the publication of books and journal articles to the indexing of citations and tweets, is a critical one in predicting innovation, and in deciding what sorts of research is supported and whom is hired to carry it out. There is a wide set of data and tools available for measuring research, but they are often used in crude ways, and each have their own limitations and internal logics. Measuring Research: What Everyone Needs to Know(R) will provide, for the first time, an accessible account of the methods used to gather and analyze data on research output and impact. Following a brief history of scholarly communication and its measurement -- from traditional peer review to crowdsourced review on the social web -- the book will look at the classification of knowledge and academic disciplines, the differences between citations and references, the role of peer review, national research evaluation exercises, the tools used to measure research, the many different types of measurement indicators, and how to measure interdisciplinarity. The book also addresses emerging issues within scholarly communication, including whether or not measurement promotes a "publish or perish" culture, fraud in research, or "citation cartels." It will also look at the stakeholders behind these analytical tools, the adverse effects of these quantifications, and the future of research measurement.

This book analyses and discusses the recent developments for assessing research quality in the humanities and related fields in the social sciences. Research assessments in the humanities are highly controversial and the evaluation of humanities research is delicate. While citation-based research performance indicators are widely used in the natural and life sciences, quantitative measures for research performance meet strong opposition in the humanities. This volume combines the presentation of state-of-the-art projects on research assessments in the humanities by humanities scholars themselves with a description of the evaluation of humanities research in practice presented by research funders. Bibliometric issues concerning humanities research complete the exhaustive analysis of humanities research assessment. The selection of authors is well-balanced between humanities scholars, research funders, and researchers on higher education. Hence, the edited volume succeeds in painting a comprehensive picture of research evaluation in the humanities. This book is valuable to university and science policy makers, university administrators, research evaluators, bibliometricians as well as humanities scholars who seek expert knowledge in research evaluation in the humanities.