Resources

Further reading on peer review research in ACL venues

On this page, you will find some resources related to collection of, and research on, peer review data in the ACL community. The initiative was pioneered by Gurevych, Kuznetsov and Dycke from UKP Lab, TU Darmstadt.

Description	Link
ARR-2024 data release: This new release includes a substantial addition of new review data from ARR-2024. The release includes 1063 reviews, as well as 480 meta-reviews, on 1807 papers from diverse domains. The release is included in NLPEER v2 for ease of use, alongside freshly crawled data from the open reviewing journals ELIFE and PLOS.	The ARR-2024 data can be found here. The extended NLPEER v2 is available here.
What can NLP do for peer review? (Kuznetsov et al. 2024) is a whitepaper originating from the Dagstuhl seminar on peer review. It includes a roadmap for future research, and an extensive literature list that is continually updated on Github.	Find the paper here
Donation-based Peer Review Data Collection at ARR (Dycke et al. 2023) is a blogpost describing the importance of peer review data collection and how people can contribute.	Read the blogpost
NLPeer: A Unified Resource for the Computational Study of Peer Review (Dycke et al. 2023) is the first ethically sourced, publically available multi-domain corpus of papers plus peer reviews from five different venues.	Find the paper here
Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond (Dycke et al. 2022) describes the workflow of peer review data collection as implemented in ARR.	Find the paper here
The original paper by Gurevych and Kuznetsov, proposing an initiative on collecting and publishing anonymized numerical data from the ACL conferences.	Get the PDF