Data Collection Protocol for ARR-DC
Information for authors, reviewers and meta-reviewers
The data collection protocol for ARR-DC builds on the initiative by Gurevych, Kuznetsov and Dycke (2021). A general overview of the updated data collection protocol for each role in the peer review process can be found below.
The ARR-DC organization invests reasonable effort to allow reviewers and authors to withdraw their consent and license agreement at any time. The data collection protocol is approved by the ACL Ethics Committee.
Note that full consent by authors and reviewers is always required before any data is published.
Important changes
From April 2024 onwards, the new protocol described on this page will be in place. Below are the key novelties with respect to the pre-April 2024 protocol:
- Authors now decide on data donation during submission and can revise this decision throughout the reviewing process.
- Reviewers will now also be requested to donate their metadata (i.e. scores), in addition to the review texts.
- Meta-reviewers will be asked to sign a license agreement to donate their meta-reviews of a cycle. The collection of meta-reviews is a new addition to the protocol.
The pre-April 2024 protocol is described in detail in this blogpost on the ARR website. For a short overview, refer to the respective section below.
Overview of current protocol
For authors
Authors are requested to consent to the collection of their data at submission time. This decision can be revised during the review process. Collected data includes the draft paper and associated peer review texts.
Any paper is eligible for donation regardless of final decision, as long as the authors agree.
For reviewers
Reviewers are asked to sign a license agreement for all their reviews of an ARR cycle for each cycle individually, or they can grant the license for all future ARR cycles until consent is withdrawn.
The data collected includes the review texts, metadata and discussion threads.
For meta-reviewers
Area Chairs (ACs) are asked to sign a license agreement to donate all their meta-reviews for the ARR cycle, or they can grant the license for all future ARR cycles until consent is withdrawn. Additionally, ACs can specify if they prefer their meta-reviews to be included as-is or paraphrased.
Overview of pre-April 2024 protocol
For authors
Authors are requested to consent to the collection of their data after the review process is completed and acceptance decisions are released. Collected data includes the submission draft and the donated peer reviews of their draft.
For reviewers
Reviewers are asked once per cycle if they are willing to donate their peer review reports. The collected data consists of the review texts only.
For meta-reviewers
ACs are asked to consent to the sharing of reviews of papers that are eventually accepted at an *ACL conference. Meta-review texts are NOT collected.