ACL Rolling Review Data Collection (ARR-DC)

Welcome!

This is the overview website for the ACL Rolling Review Data Collection (ARR-DC) project. The ARR-DC project is a community-driven effort to collect and curate a large-scale dataset of peer reviews and associated metadata from the ACL community. The dataset will be used to study the peer review process in the ACL community, and to develop new tools and methods to improve peer reviewing in our conferences.

While peer review data collection is currently happening in the *ACL venues (Dycke et al. 2023), the resulting data is skewed towards papers with more positive reviewer scores. By adapting the data collection protocol to create more balanced data resources, we can improve the quality of this line of research. For the current data collection protocol, as well as the changes made in comparison to the earlier protocol, see the Protocol page. Some resources to get started with the subject of NLP research on peer review are listed on the Resources page.

Considering the recent developments in natural language processing, we believe it is time to ramp up our efforts in studying peer review in our community. The use of tools like ChatGPT for assistance during reviewing is just one example of the changes that are happening in the peer review process. We believe that the data collected in the ARR-DC project will be a valuable resource for the community to study these changes and to develop new tools and methods to improve the peer review process.

How can I contribute?

You can contribute by donating your data to the ARR-DC corpus by signing the respective license transfer agreements when submitting to or reviewing for *ACL affiliated venues. For more information, see the Protocol page. It is always possible to withdraw your consent for future donation of your data - however, note that retraction of individual data points is infeasible after publication.

You can find answers to frequently asked questions on the FAQ page.

News

The new data collection protocol is implemented in the April 2024 ARR cycle.

Apr 4, 2024	The data collection protocol is approved by the ACL Ethics Committee.
Feb 2, 2024	The ARR data collection project is born.
Jan 28, 2024	Dagstuhl seminar on peer review.