Skip to content

Latest commit

 

History

History
114 lines (84 loc) · 7.71 KB

SC-reproducibility-initiative.md

File metadata and controls

114 lines (84 loc) · 7.71 KB
Raw

SC Reproducibility Initiative

TL;DR — After 3 years of increasing momentum, the SC Reproducibility Initiative makes Artifact Description (AD) Appendices mandatory for all papers submitted to the SC19 Technical Program (but optional for workshop papers and posters). Artifact Evaluation (AE) is still optional, but highly encouraged. AD/AE Appendices will now be auto-generated from author responses to a standard form, embedded in the SC online submission system. In 2019, the SC Reproducibility Initiative introduces three new tracks/committees, under the SC Reproducibility Chair: (1) AD/AE Appendices, (2) Reproducibility Challenge, (3) Journal Special Issue.

New SC Reproducibility Tracks/Committees

AD/AE Appendices.

This committee has the following charge:

  • Implement a program of actions to mentor authors in the preparation of the appendices. This may include webinars, blog posts, expanding the reproducibility FAQs, etc.
  • Inspect the appendices of submitted papers, follow artifact URLs to check their contents, and support authors with appendix improvement, if needed.

Reproducibility Challenge.

This committee has the following charge:

  • Select a paper accepted to the previous conference, to be used as source of the Reproducibility Challenge in the Students Cluster Competition (SCC) of the upcoming conference.
  • Work with the authors of the selected paper to build the challenge benchmark for the SCC teams.
  • Continue working until the time of the conference in crafting the rules and requirements for the challenge, and ensuring compatibility with the various hardware used by the SCC teams.

Journal Special Issue.

This committee has the following charge:

  • Review full papers submitted by the SCC teams for publication in the journal special issue, reporting on the results of the efforts to reproduce the challenge benchmark.
  • Work with student teams to assist them in revision of their papers, for a successful final publication of the reports in the journal special issue.

Artifact Description / Article Evaluation (AD/AE) Appendix Form

All SC19 submitting authors to the Technical Program must complete this form to describe the computational artifacts their results rely on: software, data, and hardware. The form is standard, and embedded in the conference submission system.

Note: If your paper used no computational artifacts, you will respond "No" to the first question and be done.

Submissions will be double-blind reviewed, and reviewers will have all the information in the AD/AE form except for the URLs pointing to the artifacts. The paper and the appendix, including URLs to artifacts, will be available to the AD/AE Appendices Committee. The committee may provide feedback to authors in a double-open arrangement (authors and committee members are known to each other). The AD/AE Appendices Committee will not share any information with the PC other than to confirm whether artifacts indicated as available are in fact available. "Available" means that the provided URLs properly resolve, and that the author-created artifacts are persistently archived and have a global, unique identifier. Accepted papers with available artifacts will display an ACM badge.

Additional notes:

  • The form targets both Artifact Description (required) and Evaluation (optional).
  • Availability of software, data, and hardware artifacts must be explicitly indicated.
  • Papers should still be self contained: they should describe the experimental methodology enough for the PC to evaluate the contribution.
  • Sample scripts for machine-generated environment data will be provided at: https://github.com/SC-Tech-Program
  • Authors are encouraged to familiarize themselves early with the AD/AE Appendix requirements, and to direct questions to: [email protected]

History of the SC Reproducibility Initiative

2015 — The SC steering committee approved the reproducibility initiative. Authors of SC15 papers were invited to submit an AD Appendix after the conference: one paper did so, became the source for the SC16 Student Cluster Competition reproducibility challenge and the first SC paper to display an ACM badge.

2016 — Authors submitting to the SC16 conference could optionally submit an AD Appendix: 9 authors submitted one, 3 were finalists and one was selected to become the source for the SC17 Student Cluster Competition reproducibility challenge.

2017 — SC made the AD Appendix a requirement to be considered for the Best Paper or Best Student Paper awards. SC17 also introduced the Computational Results Analysis (CRA) appendix. 40% of submitted and 50% of accepted papers included an AD appendix; 9 submitted papers (6 accepted) included a CRA appendix.

2018 — SC extended the option of submitting AD appendices to workshops and posters. The CRA appendix was renamed Artifact Evaluation (AE) appendix, and limited to 4 pp. AD appendices were limited to 2 pp. and remained optional but required for consideration as Best Paper/Best Student Paper, and also Best Poster/Best Student Poster.

2019 — AD appendices will be mandatory for all submissions, AE appendices are still optional, and both will be submitted via a standard form in the conference submission system. Three new TP tracks, with their respective committees and chairs, are introduced in support of the SC Reproducibility Initiative.

Rationale and results of Reproducibility at SC

The SC Conference Series is committed to introducing activities that highlight, enhance, and reward participant efforts to improve reproducibility. Each year, authors have engaged more with the Reproducibility Initiative, and the benefits have become more clear to our community.

The AD/AE Appendices improve the quality of SC papers, and increase efficiency of our collective endeavors:

  • The AD appendix provides information (details of software, hardware and data) that should be part of any publication with computational results.
  • The new AD/AE Appendices form (starting 2019) standardizes the items that authors should include.
  • If an SC paper used no computational artifacts (software, hardware or data), the author will click “No” in answer to the first question, and be done.
  • In the AE appendix, authors describe how they verified and validated their computational results, supporting their trustworthiness and providing best-practice descriptions to the community that can be adopted by others who want to improve their own efforts.
  • AD/AE Appendices are reviewed in a double-open format by a dedicated committee, independent of the Technical Program committee (which maintains a double-blind process).

Engaging the Student Cluster Competition

The SC Reproducibility Challenge committee, now a part of the Technical Program, works with the SC Reproducibility Chair to select a paper from the previous conference, which will be used as source of the reproducibility challenge the next year. This committee will work with the authors of the selected paper to develop the challenge benchmark, and ensure the SCC teams have the best chance at succeeding. The SCC teams write extended reports of their results and their efforts with the reproducibility challenge, which are submitted for a journal special issue (where they are peer reviewed before acceptance). The SC paper selected for the reproducibility challenge receives a special recognition at the SC Awards Ceremony.

SC Reproducibility Resources for Authors

Please visit the GitHub repository to find resources and answers to FAQs. https://github.com/SC-Tech-Program

— By the SC19 Reproducibility Chair, Lorena A. Barba, PhD, derived in part from the SC18 Reproducibility Initiative web page.