| layout | title | permalink |
|---|---|---|
page |
AD/AE Appendices Author FAQ |
/ad-ae-appendices-author/ |
If you still have questions, you can email: [email protected]
Artifact Description (AD) Appendices are required for all papers submitted to the Technical Program at SC19. Artifact Evaluation (AE) Appendices are optional, but strongly encouraged.
AD and AE Appendix Requirements
Impact of AD and AE Appendices
ACM Artifacts Available and Artifacts Evaluated Badges
Are AD and AE appendices required in order to submit to SC19?
An AD appendix is required for all Technical Program submissions. An AE appendix is optional but strongly encouraged.
Do I need to make my software open source in order to complete the AD appendix?
No. You are not asked to make any changes to your computing environment in order to complete the appendix. The AD appendix is meant to describe the computing environment in which you produced your results. Any author-created software does need to be open source, however, to be eligible for the ACM Artifacts Available badge (see below).
How should I format my AD Appendix
You don't need to worry about formatting the Appendices. You will be presented with an online form during the paper submission with questions that you will answer directly on the submission site. After answering the questions, the system will automatically generate a PDF of the Appendix for you.
What information do I need to provide in the AD/AE Appendix online form?
A printout of the questions included in the AD/AE Appendix online form is provided in the Author-Kit repository. Be sure to familiarize yourself with these before writing your paper, and ideally before or while you are producing your results.
Who will review my appendices?
The AD & AE Appendices will be reviewed with your paper by the Technical Program committee, but the artifact URLs will be removed from the version they review, as a precaution in support of double-blind review. In addition, the AD/AE Appendices Committee will review the unredacted appendices, and will check that artifacts are indeed available in the URLs provided. They will also help authors improve their appendices, in a double-open arrangement.
How will review of appendices interact with the double-blind review process?
The AD appendix should describe the data, software and hardware artifacts involved in producing the results. Reviewers could discover the author's identity if they embark on an online search, but they will be asked not to, in support of double-blind review. Author-provided artifact URLs will be redacted from the appendices provided to the reviewers.
What's the impact of an Artifact Description appendix on scientific reproducibility?
Reproducibility depends on, as a first step, sharing the provenance of results with transparency, and the AD appendix is an instrument of documentation and transparency. A good AD appendix helps researchers document their results, and helps other researchers build from them.
The paper text explains why I believe my answers are right and shows all my work. Why do I need to provide an Artifact Evaluation appendix?
There are many good reasons for formalizing the artifact evaluation process. Standard practice varies across disciplines, and SC is an international, multi-disciplinary conference. Labeling the evaluation as such improves our ability to review the paper and improves reader confidence in the veracity of the results when approaching the work from a different background.
What are "author created" artifacts and why make the distinction?
Author created artifacts are the hardware, software, or data created by the paper's authors. Only these artifacts need be made available to facilitate reproducibility. Proprietary, closed source artifacts (e.g. commercial software and CPUs) will necessarily be part of many research studies. These proprietary artifacts should be described to the best of the author's ability but do not need to be provided.
What about proprietary author-created artifacts?
The ideal case for reproducibility is to have all author created artifacts publically available with a stable identifier. Papers involving proprietary, closed source author-created artifacts should indicate the availability of the artifacts and describe them as much as possible. Note that results dependent on closed source artifacts are not reproducible and are therefore ineligible for most of the ACM's artifact review badges. See https://www.acm.org/publications/policies/artifact-review-badging.
Are the numbers used to draw our charts a data artifact?
Not necessarily. Data artifacts are the data (input or output) required to reproduce the results, not necessarily the results themselves. For example, if your paper presents a system that generates charts from datasets then providing an input dataset would facilitate reproducibility. However, if the paper merely uses charts to elucidate results then the input data to whatever tool you used to draw those charts isn't required to reproduce the paper's results. The tool which drew the chart isn't part of the study, so the input data to that tool is not a data artifact of this work.
Help! My data is HUGE! How do I make it publically available with a stable identifier?
Use Zenodo (https://help.zenodo.org/). Contact them for information on how to upload extremely large datasets. You can easily upload datasets of 50GB or less, have multiple datasets, and there is no size limit on communities.
What's the impact of an Artifact Evaluation appendix on scientific reproducibility?
An artifact-evaluation effort can increase the trustworthiness of computational results. It can be particularly effective in the case of results obtained using specialized computing platforms, not available to other researchers. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the supercomputing community. Access to these systems is typically limited, however. Thus, most reviewers cannot independently check results, and the authors themselves may be unable to recompute their own results in the future, given the impact of irreversible changes in the environment (compilers, libraries, components, etc.). The various forms of Artifact Evaluation improve confidence that computational results from these special platforms are correct.
By the ACM Badging definitions:
ACM Artifacts Available badge: This badge is applied to papers in which associated artifacts have been made permanently available for retrieval.
The AD/AE Appendix form (new for SC19) will automatically determine eligibility for an ACM Artifacts Available badge on the basis of the answers to questions about the availability of author-created software, hardware or data products. The conditions of eligibility are:
- All author-created software artifacts are maintained in a public repository under an OSI- approved license.
- All author-created hardware artifacts are available and comply with the Open Source Hardware Definition.
- All author-created data artifacts are maintained in a public repository with a stable identifier, such as a DOI.
Artifacts Evaluated badge: This badge is applied to papers whose associated artifacts have successfully completed an independent audit.