New Data, New Knowledge? Infrastructures of Performance Measurement in Science (NEDA)

Start of the project: 2020-Nov-01 - End of the project: 2023-Oct-31

In cooperation with the Berlin-Brandenburg Academy of Sciences and Humanities (BBAW), the BMBF-funded junior research group is investigating the increasing technical possibilities of data production and data assessment of scientific performance and their consequences for the understanding of scientific practice.

Research information and the associated measurement of performance are an important factor in the debate on quality assurance in science. However, the generation of empirically proven knowledge about research performance is ridden with prerequisites. Technical infrastructures are increasingly being used for this purpose - above all by commercial providers - which transform existing data into databases and make instruments available which are intended to generate relevant knowledge for the governance of science.

The project focuses on the development of these "research analytics". It examines the question of which data are used in what ways. The understanding of scientific practice that is inscribed in these continuously developing technical infrastructures of data production and data assessment is analysed.

The project goes beyond a critique of the construction and use of bibliometric databases and their indicators. Instead, it looks at the technical infrastructures of scientific performance measurement as a whole and asks how the data production and usage infrastructures help to shape what can be observed and evaluated as scientific performance.

The project is divided into a qualitative and a quantitative sub-project. In the quantitative sub-project at the German Centre for Higher Education and Science Studies (DZHW), bibliometric analyses are conducted via the infrastructure of the Competence Centre for Bibliometrics to investigate how and in what ways the infrastructures of data providers differ in their statements on scientific performance.

The qualitative sub-project at the Berlin-Brandenburg Academy of Sciences and Humanities focuses on the user orientation of the providers of these infrastructures by analysing the particular usage scenarios offered to potential users.

Taken together, the aim of the project is to examine the assumptions made in the databases and corresponding usage instruments about what constitutes scientific performance.

Show more Show less
Publications

On the performativity of SDG classifications in large bibliometric databases.

Ottaviani, M., & Stahlschmidt, S. (2024).
On the performativity of SDG classifications in large bibliometric databases. ArXiv (online first). https://doi.org/10.48550/arXiv.2405.03007
Abstract

This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data by exploring five SDGs. We build a LLM that is fine-tuned in parallel by the diverse SDG classifications inscribed into the databases' SDG classifications. Our results show high sensitivity in model architecture, classified publications, fine-tuning process and natural language generation. The wide arbitrariness at different levels raises concerns about using LLM in research practice.

The State of Unpaywall: Analyzing the Consistency of Open Access Data.

Sanford, H. (2022).
The State of Unpaywall: Analyzing the Consistency of Open Access Data. Proceedings 26th International Conference on Science, Technology and Innovation Indicators (STI 2022).

From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers.

Stahlschmidt, S., & Stephen, D. (2022).
From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers. Scientometrics, 2022(127), 2413-2431. https://doi.org/10.1007/s11192-022-04309-6

From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers.

Stahlschmidt, S., & Stephen, D. (2021).
From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers. Ithaca: arXiv.
Presentations

Neue Daten, neues Wissen? Infrastrukturen der Leistungsbewertung in der Wissenschaft (NeDa).

Stephen, D. (2023, Oktober).
Neue Daten, neues Wissen? Infrastrukturen der Leistungsbewertung in der Wissenschaft (NeDa). Vortrag auf dem Workshop Abschlusstagung der BMBF-Förderlinie Qualitätsentwicklung(en) in der Wissenschaft, Dortmund.

On the performativity of SDG classifications in large bibliometric databases.

Ottaviani, M., & Stahlschmidt, S. (2023, Oktober).
On the performativity of SDG classifications in large bibliometric databases. Poster auf der Konferenz Nordic Workshop on Bibliometrics and Research Policy, Gothenburg, Sweden.
Abstract

Large bibliometric databases, such as Web of Science, Scopus and Dimensions, are performative and affect the visibility of scientific outputs and impact measurement. However, these databases do not match the UN's SDGs classifications. This work proposes using large language models (LLM) to learn about data bias injected by diverse SDG classifications. The results show high sensitivity to diverse classifications in model architecture, classified publications, fine-tuning process and natural language generation. The wide arbitrariness raises ethical concerns. Qualitative text analyses can help inform science policy on diverse SDG classifications.

The State of Unpaywall: Analyzing the Consistency of Open Access Data.

Sanford, H. (2022, September).
The State of Unpaywall: Analyzing the Consistency of Open Access Data. Vortrag auf der Konferenz 26th International Conference on Science, Technology and Innovation Indicators (STI 2022), Universidad de Granada, Granada, Spanien.

Szientometrie als Gegenstand von Wissenschaftsforschung.

Stahlschmidt, S. (2022, Juli).
Teilnahme an der Podiumsdiskussion Szientometrie als Gegenstand von Wissenschaftsforschung auf dem Symposium Wissenschaftsforschung im Fokus - Potentiale und neue Perspektiven, VolkswagenStiftung, Hannover.

Challenges for bibliometrics in the era of Open Science and non-academic impact assessment.

Stahlschmidt, S. (2022, Juni).
Teilnahme an der Podiumsdiskussion Challenges for bibliometrics in the era of Open Science and non-academic impact assessment Frontiers of the Evaluation. Workshop in Honour of the Memory of Henk F. Moed, Sapienza University, European Summer School for Scientometrics, Rome, Italy.

Performativity of PIDs: Observing or shaping research?

Stahlschmidt, S., & Krüger, A. (2020, Januar).
Performativity of PIDs: Observing or shaping research? Vortrag auf der PIDapalooza 2020, 29.-30.01.2020, Lissabon, Portugal.

Contact person

Stephan Stahlschmidt
Dr. Stephan Stahlschmidt +49 30 2064177-18

External Contact

Dr. Anne K. Krüger

In cooperation with

Berlin-Brandenburgische Akademie der Wissenschaften

Funded by

Bundesministerium für Bildung und Forschung