Reproducibility Status
Reproducibility status labels tell readers how much trust to place in a computational artifact. They are not badges of prestige; they are maintenance information.
Status Labels
Section titled “Status Labels”Use these labels for notebooks, generated figures, benchmark reports, and computational tables:
| Status | Meaning | May Support Claims? |
|---|---|---|
reproduced | rerun successfully in the recorded environment with declared checks passing | yes |
reproduced_with_warnings | rerun succeeds, but warnings, tolerances, or platform differences need explanation | yes, with caveat |
needs_update | dependency, API, data, or convention drift is likely | no new claims |
broken | expected run or validation fails | no |
archived | retained for history but no longer maintained | historical only |
conceptual_only | explains an idea but is not an executable evidence artifact | no numerical evidence |
If the status is not recorded, treat the artifact as unreviewed.
Evidence Required for Reproduced
Section titled “Evidence Required for Reproduced”To mark an artifact reproduced, record:
- environment or lockfile,
- run date,
- command or workflow,
- benchmark target,
- tolerance,
- pass or fail result,
- reviewer or automation source when available.
For notebooks, the validation cells should remain in the notebook. Do not strip the evidence and keep only the final figure.
Status Changes
Section titled “Status Changes”Move an artifact to needs_update when:
- a dependency has a breaking API change,
- a canonical formula or convention changes,
- a source dataset changes,
- a benchmark tolerance is questioned,
- an exported figure no longer matches the source notebook,
- the page using the artifact is substantially revised.
Move an artifact to broken when validation fails or the notebook can no longer run in its declared environment. Do not quietly loosen tolerances to recover a passing status.
Warnings
Section titled “Warnings”Use reproduced_with_warnings for cases such as:
- minor floating-point differences across platforms,
- dependency deprecation warnings that do not affect results yet,
- a benchmark that passes but only at a looser tolerance than desired,
- a known finite-size or finite-grid limitation that is documented.
The warning should state whether the physics conclusion is affected.
Reporting Format
Section titled “Reporting Format”A compact report can use:
| Field | Example |
|---|---|
| artifact | harmonic-oscillator-eigenstates.ipynb |
| status | reproduced |
| environment | Python and package versions |
| benchmark | oscillator energies and parity |
| tolerance | relative energy error below stated threshold |
| last run | date and commit identifier |
| notes | finite-box tails negligible for reported states |
Cross-Links
Section titled “Cross-Links”- Approximation and Scattering Reproducibility Checklist collects the release evidence needed before assigning one of these labels.
- Notebook Index
- Many-Body Reproducible Notebooks — inventory states and release gates applied without treating planned files as artifacts.
- Environments
- Benchmark Problems
- Numerical Benchmarks
- Citation Style
References
Section titled “References”- The Turing Way Community, The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Data Science.
- Project Jupyter, Jupyter Documentation.
- Python Packaging Authority, Python Packaging User Guide.
- C. R. Harris et al., “Array programming with NumPy,” Nature 585, 357-362 (2020), DOI: 10.1038/s41586-020-2649-2.