Computational Reproducibility

Assesses stability, traceability, and agreement of results when physical simulations are repeated.

Conceptual context

A physical simulation is not reproducible merely because it can be run again. It must produce compatible results when repeated under documented, comparable conditions. Computational reproducibility studies how much results vary, which part of that variation comes from numerical randomness, and which part reveals changes in the model, algorithm, precision, or environment. The central question is how $\epsilon _rep$ decreases as $N$ grows. More repetitions may reduce statistical fluctuation, but they do not automatically remove a bias shared by every run. Seeds, versions, parameters, mesh, time step, and output criteria must therefore be recorded.

🟢 Essential level

Computational reproducibility compares equivalent runs of a physical simulation. When results cluster tightly, $s_x$ is small and $\epsilon _rep$ is usually small relative to $x_ref$.

As $N$ increases, the mean becomes more stable and $u_xbar$ tends to decrease, although each additional run contributes less improvement than the previous one. This behavior does not prove that the simulation is correct, because all runs may agree while sharing the same modeling error. The key distinction is among repeatability, reproducibility, and accuracy. Repeatability concerns consistency in one environment; reproducibility requires agreement under comparable documented conditions; accuracy concerns agreement with the correct physics. $C_rep$ summarizes the variability of a particular setup and helps estimate how many runs may be needed. A reproducibility study therefore asks not only whether values agree, but whether the whole computational chain can explain that agreement and reproduce it under controlled changes.

Reporting only a repetition count without recording seeds, versions, mesh, and time step therefore says little: the same count can hide very different amounts of genuinely independent information.

🔵 Formal level

Basic formulas

The first step is to measure relative spread across runs. $s_x$ is compared with a common physical scale, $x_ref$:

Relative reproducibility error

Thus $\epsilon _rep$ is dimensionless and nonnegative. A reference close to zero makes this normalization unstable, so an absolute error is preferable in that situation.

The fluctuation of a mean obtained from independent runs is estimated through:

Statistical uncertainty of the mean

$u_xbar$ retains the unit of the observable and decreases as $N$ increases. This relation assumes finite variance, independent runs, and no dominant drift.

Real student questions

Do more repetitions always improve the result? They improve the estimate of the random component, but they do not remove shared bias. Once a plateau appears, more runs may be ineffective.

Is a perfectly repeatable simulation correct? No. It may reproduce the same wrong solution because of a model, unit, or implementation error.

Why are two runs not enough? Two runs may expose large discrepancies, but they estimate spread poorly and cannot establish the convergence law reliably.

What does failure of the inverse-root law mean? Runs may be correlated, drifting, heavy-tailed, differently configured, or limited by a systematic floor.

What must be saved for another person to reproduce the result? Code, version, dependencies, data, parameters, seed, relevant hardware, numerical method, and analysis criterion.

Final synthesis

Computational reproducibility measures stability under comparable, traceable repetitions. Random error usually decreases as the run count grows, but with diminishing returns. A plateau, deviation from the ideal law, or poor normalization reveals model limits. Repeating is useful; understanding what is repeated and why it changes is essential.