Motivation
Peer review lies at the center of academic quality control. Yet, in many fields of science, researchers pre-publish their article drafts on preprint servers like arxiv or bioarxiv to disseminate their research findings to by-pass the typically lengthy reviewing process. However, the resulting vast body of gray literature, i.e. research articles of unverified quality, poses a great challenge to the day-to-day research work by putting the burden of quality assessment on the reader. This issue becomes particularly pressing in crisis situations like the COVID-19 pandemic where the societal consequences of unfaithful scientific work can be tremendous and the number of published preprint articles within a field may explode.
While expert peer review is impossible to substitute, a partially automated and scalable solution that would provide some level of quality control would significantly increase the utility of the preprint literature. In this project we intend to address these challenges to help involve less experienced researchers in the reviewing process, thereby scaling up the reviewing effort on the community level and providing the first line of quality control for the ever-increasing publication pool.
Goals
Four key challenges hindered the development of practically useful assistance tools for scaling-up peer reviewing:
- Diverse peer reviewing data is only scarcely available to research.
- The human-centric design of peer reviewing assistance systems is challenging but fundamental for the sensitive application domain of peer review.
- Due to its gatekeeper role in science, assistance tools for peer review need to foster fairness and high-quality reviews.
- Peer reviewing is inherently subjective and pluralistic; assisting reviewers demands explicit modeling of these aspects.
Data
As a first step towards peer reviewing assistance, we collect a multi-domain and cross-temporal peer reviewing dataset. Peer reviewing data is confidential in most scientific communities and hereby a scarcely available resource to research. However, a diverse and representative sample of reviewing data is essential for the development of fair and general assistance tools, due to the diversity of reviewing habits and paradigms across science.
We compile these datasets into a unified resource for the computational study of peer review and to enable all further steps of the project.
Approach
At the center of the project lies the development of a practical assistance tool for junior reviewers. Peer review requires significant expertise and experience. We developed CARE, an open-source web service that embeds Natural Language Processing models into a reading environment to support readers and specifically peer reviewers during their assessment process. For peer reviewers, CARE offers extensive functionality to annotate and comment on the article under review which allows the direct transfer into a review report. For researchers, CARE offers a generic and scalable infrastructure for coupling machine learning models and for the systematic collection of human text interaction data.
Subsequently, we integrate machine-supported reviewing workflows. As inexperienced reviewers are typically not familiar with peer reviewing guidelines and conventions, we developed a structured step-by-step guide together with the domain experts for the assessment of a research article in the biomedical field. We embed this workflow into CARE and the coupling of natural language processing models that support the discovery of information during the execution of such a workflow.
Team
- Prof. Dr. Iryna Gurevych (PI)
- Prof. Dr. Beatrix Süß
- Prof. Dr. Heinz Köppl
- Nils Dycke
- Fengyu Cai
Funding
This research work is funded from 2020 – 2024 by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.
