Automatic Peer Review Quality Assessment from Self-reported Feedback

Master Thesis

Peer review is the core mechanism of scientific quality control: a study is evaluated by multiple anonymous researchers – peers – who independently decide if the work is methodologically sound, novel, and meets the quality standards of the field. Despite its many advantages, peer review is prone to bias, strategic and heuristic behaviour, and it is not uncommon for scientifically valid work to be rejected and for spurious findings to be accepted and published. To study the quality of peer reviews, the ACL-2018 conference asked the authors of submissions to rate the quality of the reviewing feedback they received, resulting in a unique dataset of reviews coupled with review quality scores. Yet, since this evaluation of peer reviews was done by the very authors whose work was evaluated, the review quality scores are themselves biased. This thesis will explore review quality assessment from theoretical perspective, perform in-detail data analysis of the ACL-2018 dataset, and develop prototype models for automatic review quality assessment using state-of-the-art NLP.