JASA Editors Talk Reproducibility
In September of last year, JASA began requiring authors to submit their code and data sets as a condition of publication. A new editorial role—associate editor for reproducibility (AER)—was added to ensure each paper met a standard of reproducibility. Christopher Paciorek, Victoria Stodden, and Julian Wolfson make up JASA’s first group of AERs, while Montse Fuentes is JASA editor.
Why did JASA add reproducibility editors?
Montse Fuentes: JASA is committed to enhancing rigor and reproducibility in our scientific research. This is essential to build the credibility of our published work and the impact of our statistical literature.
On parallel, we are committed to managing the reviews promptly. As a step in the direction of reproducibility, we have changed our review process and our submission requirements. Computer code is required and is reviewed by a code reviewer. This is done while other reviewers are evaluating the methods and applications, so it does not delay the review process. At the final stage, and prior to final acceptance, the AER promptly evaluates all the material to determine if it meets our reproducibility expectations.
JASA Applications and Case Studies has three AERs with statistical computing expertise. Their responsibility is to efficiently, effectively, and promptly determine whether the paper is ready for publication, meeting our high expectations of quality and reproducibility.
What has been the most challenging part of being a reproducibility editor?
Christopher Paciorek: So far, I think the most challenging part has been trying to anticipate the different workflows researchers might use and to create appropriate requirements and guidelines. We are just getting to the point where we are seeing provisionally accepted manuscripts and determining how this is working in practice.
You have a list of criteria for authors. Have you had any problems with authors unable to comply?
Christopher Paciorek: We don’t really know yet. We’ve actually not gotten any provisionally accepted papers to look at yet because of the cycle time for publication in JASA. So, the only interactions we’ve had with authors have been a small number of queries by email, none of which has indicated serious problems.
Do you have any tips to help researchers prepare code and data sets to submit to JASA?
Christopher Paciorek: We tried to include some suggestions in the reproducibility guidelines to authors, but I think the main point is that this will evolve as we get more experience with the process. The biggest suggestion we could make so far would be to have the data, code, and workflow information available in public repositories.