Dagstuhl-Seminar — Reviewer No. 2: Old and New Problems in Peer Review

Peer review is the best mechanism for assessing scientific validity of new research that we have so far. But this mechanism has many well-known issues, such as the different incentives of the authors and reviewers, difficulties with preserving reviewer and author anonymity, confirmation and other cognitive biases that even researchers fall prey to. These intrinsic problems are exacerbated in interdisciplinary fields like Natural Language Processing (NLP) and Machine Learning (ML), where groups of researchers may vary so much in their methodology, terminology, and research agendas, that sometimes they have trouble even recognizing each other's contributions as "research". This Dagstuhl Seminar will cover a range of topics related to organization of peer review in NLP and ML