Welcome

This is the overview website for the ACL Rolling Review Data Collection (ARR-DC) project. The ARR-DC project is a community-driven effort to collect and curate a large-scale dataset of peer reviews and associated metadata from the ACL community. The dataset will be used to study the peer review process in the ACL community, and to develop new tools and methods to improve peer reviewing in our conferences.

While peer review data collection is currently happening in the *ACL venues (Dycke et al. 2023), the resulting data is skewed towards papers with more positive reviewer scores. By adapting the data collection protocol to create more balanced data resources, we can improve the quality of this line of research. For the current data collection protocol, as well as the changes made in comparison to the earlier protocol, see the Protocol page. Some resources to get started with the subject of NLP research on peer review are listed on the Resources page.

Considering the recent developments in natural language processing, we believe it is time to ramp up our efforts in studying peer review in our community. The use of tools like ChatGPT for assistance during reviewing is just one example of the changes that are happening in the peer review process. We believe that the data collected in the ARR-DC project will be a valuable resource for the community to study these changes and to develop new tools and methods to improve the peer review process.

How can I contribute?

You can contribute by donating your data to the ARR-DC corpus by signing the respective license transfer agreements when submitting to or reviewing for *ACL affiliated venues. For more information, see the Protocol page. It is always possible to withdraw your consent for future donation of your data - however, note that retraction of individual data points is infeasible after publication.

You can find answers to frequently asked questions on the FAQ page.

News

From peer review nuances to best practices

Research on peer review is growing fast. As more studies appear, it becomes both necessary and timely to agree on how we handle the small but consequential “nuances” in peer review data. These nuances are easy to overlook, yet getting them wrong can undermine the rigor of a study.

Using the ARR data, we took a first look at the three most common nuances and tried to quantify how much they matter:

  • Paper version. A manuscript exists in several versions, most notably the initial draft and the camera-ready version.
  • Score version. Multiple review score versions exist, such as the initial score and the post-rebuttal score.
  • Input format. A manuscript can be fed to a model as Text, JSON, Markdown, or Image.

For the full analysis and our best-practice recommendations for both data providers and data users, see the detailed report here.

The ARR data collection effort exists precisely to support this kind of analysis. By preserving multiple review data artifacts, it gives the community a solid foundation for studying peer review. This year we have further updated the author and reviewer license agreement, which will make richer data types available, such as author reported review issues (see Data Collection Protocol). Stay tuned.

Notice on ARR 2024 non-accepted submissions

Due to a communication error, data from non-accepted submissions in the ARR 2024 April and June cycles were released earlier than intended. Under the current policy, these data were expected to remain confidential until June 2026.

If you are an author or reviewer of a submission from the ARR 2024 April or June cycles and do not have concerns about this early release, you do not need to take any action. If you have any concerns or would prefer your data to be retracted, please see Can data be withdrawn? and How to reach out to us? for more details.

Update to the data collection protocol

The data collection protocol has been updated to include additional types of review data, including the before-and-after-rebuttal versions of review scores and author-flagged review issues. Please see Data Collection Protocol for more details.

Jul 28, 2026 From peer review nuances to best practices.
May 29, 2026 ARR-2026 data released! More info here.
Mar 25, 2026 Notice on ARR 2024 non-accepted submissions.
Mar 25, 2026 Update to the data collection protocol.
Dec 13, 2025 ARR-2025 data released! More info here.
Feb 7, 2025 ARR-2024 data released! More info here.
Apr 4, 2024 New data collection protocol implemented in ARR.
Apr 4, 2024 The data collection protocol is approved by the ACL Ethics Committee.
Feb 2, 2024 The ARR data collection project is born.
Jan 28, 2024 Dagstuhl seminar on peer review.