What makes a scientific-computing deliverable reproducible

1. Define the target before labelling the package

Terminology varies across fields, so the contract should define it. Here, repeatability means the original team reruns the same artefacts under controlled conditions. Computational reproducibility means another operator obtains consistent results with the original inputs, code, methods, and analysis conditions. Replicability means a new study with independently obtained data addresses the same scientific question.[1] We use auditability for the weaker but valuable ability to inspect inputs, provenance, decisions, logs, and result lineage when execution is blocked by privacy, licensing, or compute constraints. These levels are not interchangeable: a repeatable run may be non-transferable, and a perfectly reproduced calculation may still encode a flawed model.

2. Deliver an evidence graph, not a results directory

A report says what the team concluded; it rarely establishes which input, commit, configuration, and run produced each figure. W3C PROV models entities, activities, and responsible agents, while RO-Crate packages data, methods, software, people, and outcomes as linked, machine-readable research objects.[5,6] The minimum lineage should answer where every consequential input came from, which activity consumed it, what intermediate and final objects were generated, who executed and reviewed the work, and where failures or manual interventions occurred. Human-facing structure and documentation make the project understandable; manifests, hashes, and structured provenance make it checkable.

3. Preserve data identity and every consequential transformation

An input manifest should record source, acquisition date, release or accession, format, size, checksum, licence, and access class. Raw inputs remain immutable. Cleaning, exclusion, renaming, unit conversion, and manual correction are generated by scripts or explicit change tables rather than silently applied to a spreadsheet. FAIR does not require unrestricted public release: controlled objects can retain discoverable metadata, an access procedure, and authorisation rules.[2,3] When clinical, proprietary, or very large data cannot travel with the package, provide a verifiable inventory, accountable custodian, retention period, and de-identified or synthetic test fixture that exercises the same interfaces.

4. Make the software version identifiable, citable, and usable

Pin custom code to a commit or signed release, not a moving repository page. Identify patches, submodules, model weights, and reference databases separately. Documentation should cover entry points, parameters, layout, expected outputs, and failure handling; a licence defines permitted reuse; CITATION.cff or equivalent metadata supports credit. Software Citation Principles call for unique identification, persistence, accessibility, and version specificity.[8] A versioned DOI supports scholarly discovery and citation, while a content-derived SWHID can identify an exact source-code object; neither substitutes for the other or for complete dependency capture.[9]

5. Capture the resolved environment and its portability limits

Language and package version labels omit transitive dependencies, system libraries, compilers, build flags, and hardware-sensitive components. Record lockfiles or full environment exports, OS and architecture, CPU/GPU, drivers and accelerator libraries, locale and encoding, plus thread counts, seeds, determinism settings, and precision where they matter. Containers improve isolation, but mutable tags, unlocked repositories, missing recipes, and unrecorded image digests still permit drift.[10,11] Stronger packages retain the recipe, base-image digest, dependency lock, archived image or binaries, and a test command. Functional systems such as Guix or Nix can describe more of the build graph, yet upstream archives, firmware, hardware, licences, and maintenance remain external constraints.[12]

7. Freeze, archive, cite, and then ask someone else to run it

A development repository is not automatically a preservation service. At release, freeze an immutable package with authorship, version, date, licence, related data and publications, and persistent identifiers. Large assets may be tiered, but the inventory, location, checksum, retention rule, and deletion policy must remain explicit. A person who did not build the workflow should then run a smoke test or reproduce the key result in a clean environment and deposit the comparison log. Before that event, the accurate status is “reproduction-ready,” not “independently reproduced.” A successful rerun still needs a date, platform, input identity, tolerance, and coverage statement because services, licences, formats, and hardware age.

L0 — Minimum auditable package

• README: research question, scope, layout, entry point, owner, limitations, and support window.
• Input manifest: provenance, release/date, licence/access, format, size, and checksums; raw inputs remain immutable.
• Exact code revision, configurations, seeds, manual steps, and figure/table generation code.
• Actual commands, timestamps, resource and software/hardware identity, logs, exit status, warnings, and interventions.
• Raw or essential intermediate outputs, final tables/figures, interpretation, and claims the package does not support.

L1 — Rerunnable package for the originating team

• One entry point or workflow DAG, relative paths, dependency locks, environment recipe, and container digest.
• Minimal test fixture, expected outputs, schema/integrity checks, and numerical or statistical tolerances.
• A smoke test passes in CI or an isolated environment, with failures attributable to a specific layer.

L2 — Transferable package for another operator or institution

• Versioned archive with DOI, SWHID, or institutional PID; complete version, authorship, licence, and citation metadata.
• Data, code, environment, workflow, runs, and outputs linked through RO-Crate/PROV or an equivalent manifest.
• Restricted assets have an access path, accountable custodian, substitute test data, and an explicit disclosure boundary.

L3 — Independently reproduced package

• A non-developer reruns it on a named platform and date; the run log and environment identity are retained.
• Key outputs are compared using predeclared bitwise, numerical, or statistical criteria; discrepancies and omissions are reported.
• A maintainer, preservation horizon, update policy, and revalidation triggers are named without promising perpetual execution.

Common misconceptions

• Misconception: “The code is on GitHub, so it is reproducible.” · Better interpretation: A moving URL does not pin inputs, revision, environment, run, or archived release.
• Misconception: “A Docker image freezes everything.” · Better interpretation: Record the digest, recipe, locked sources, host/hardware assumptions, and preserved image.
• Misconception: “One seed guarantees identical numbers.” · Better interpretation: Parallel scheduling, GPU kernels, compilers, and floating-point order can still differ; declare a scientific tolerance.
• Misconception: “FAIR means every dataset must be public.” · Better interpretation: FAIR permits authentication and authorisation; controlled access still needs rich metadata and a route to request use.
• Misconception: “Only the successful command belongs in the record.” · Better interpretation: Failures, retries, warnings, and manual interventions are part of the provenance of the accepted result.
• Misconception: “Matching figures replicate the scientific claim.” · Better interpretation: Recomputing original data and code supports computational reproducibility, not independent replication with new evidence.

Conclusion

A credible scientific-computing delivery turns a result into an inspectable, executable, and preservable evidence chain. L0 explains what happened; L1 removes dependence on one workstation; L2 lets another institution locate and understand the objects; only an independent rerun with a comparison record justifies an “independently reproduced” label. The professional promise is not “one-click forever.” It is a precise account of who verified which outputs, when, with which inputs and environment, under what tolerance, and with what unresolved data, model, compute, and governance constraints.