A reviewer opens an earlier report beside a new calculation. The input values appear unchanged, but the third quartile is different. Before using either number, they need to know whether the data, the method, or the software changed.
In this fictional example, two implementations receive the same four values and return different answers. Neither raises an error. Both pass their existing tests. The discrepancy is legitimate because they use different definitions of a sample quantile.
That is a small example of a large engineering problem. A function name can conceal choices about interpolation, missing values, weighting, preprocessing, and numerical comparison. A package version helps identify an implementation, but it does not explain the mathematical question the caller intended to ask.
The reviewer needs enough context to reconstruct the old result and understand what the new one means. A reproducibility contract captures those choices so a changed output can be investigated and explained, especially when several products use a shared calculation.
Begin with a disagreement that can be calculated by hand
Use the sorted sample [1, 2, 3, 4] and the probability p = 0.75. The R documentation describes nine sample-quantile algorithms following the Hyndman-Fan taxonomy. The examples below use its named types. R's quantile documentation supplies the definitions.
For type 1, the inverse empirical distribution selects the third observation at this probability, producing 3. Type 2 averages at the discontinuity, producing (3+4)/2 = 3.5.
Type 7 uses the interpolation position h = 1 + (n-1)*p. With four observations, h = 3.25, producing 3 + 0.25*(4-3) = 3.25.
Type 8 uses h = (n+1/3)*p + 1/3. Here the position is 43/12. Interpolating between the observations 3 and 4 gives a quantile of 43/12, approximately 3.583333.
Third quartile for the synthetic sample [1, 2, 3, 4]
| Convention | Result for this sample |
|---|---|
| Type 1 | 3 |
| Type 2 | 3.5 |
| Type 7 | 3.25 |
| Type 8 | 43/12, approximately 3.583333 |
The discrepancy is meaningful before floating-point precision enters the discussion. A team investigating it should first ask whether the two implementations are attempting the same calculation.
Name the mathematical contract
A calculation's public contract should identify the quantity being estimated and the conventions used to estimate it. That can include interpolation, weighting, missing-value handling, boundary behavior, and the meaning of units.
“Percentile” is insufficient when clients rely on a particular convention. “Correlation” can be ambiguous about rank or linear association, weighting, and how incomplete pairs are handled. A calculation that discards missing records answers a question about the retained subset, which may differ from the user's intended population.
Some of these choices belong in explicit parameters. Others can be fixed in a named algorithm or policy version. The important point is that they are discoverable and stable under the declared contract.
A parameter's type is only part of that contract. A method can retain the same signature while changing the interpretation of its output. For compatibility-sensitive consumers, those semantic changes can matter as much as a removed field.
Preserve the context that can affect the result
Input identity is the first requirement. A reproducibility record should identify which data were used, including the source version or retained artifact needed to distinguish a later correction.
Preparation matters too. Units, ordering, filtering, normalization, and missingness policy can change the effective input even when the original file is identical. These transformations should be identified rather than hidden in an informal preprocessing step.
The algorithm and statistical convention identify the requested computation. The execution context can add numerical type, relevant library versions, and platform details when they affect the reproducibility commitment.
The level of commitment should be explicit. Reproducing an output within a declared numerical tolerance differs from requiring identical serialized bytes. A system should not promise universal bitwise portability merely because its business logic contains no random-number generator.
For floating-point arithmetic, execution order can matter. In ordinary binary floating-point evaluation, (1e16 + -1e16) + 1 can produce 1 while 1e16 + (-1e16 + 1) produces 0. A different reduction order can therefore matter to the result even when the symbolic expression seems equivalent.
This is a reason to define and test the promised behavior, including where exact equality is appropriate and where it is not.
Use reference results for the calculation being attempted
A useful numerical test set should include cases that can be checked independently. Hand-computable examples expose convention errors. Boundary cases expose assumptions about empty, constant, tied, or incomplete inputs. Difficult reference datasets examine numerical behavior that ordinary examples can miss.
NIST's Statistical Reference Datasets provide certified computational values for specified calculations. Its guidance emphasizes matching the calculation and considering difficulty and precision. The StRD FAQ supports using reference results as evidence about numerical accuracy, without implying product certification.
A reference vector is most useful when it states what it is testing. The four-number quantile example checks convention behavior. It does not establish the accuracy of an unrelated nonlinear optimizer or the scientific validity of an interpretation built on top of the quantile.
Independent implementations can provide another check, but agreement is not infallibility. Two implementations can share a definition, a dependency, or the same mistaken assumption. The test record should explain the basis of the expected result rather than treat agreement as a universal proof.
Choose a tolerance that means something in the task
A numerical comparison needs its own semantics. Exact equality can be appropriate for discrete results or a deliberately fixed representation. Approximate comparison may be appropriate for floating-point calculations, under a tolerance justified by the task.
One reference-scaled criterion has the form:
abs(actual - reference) <= absolute_tolerance
+ relative_tolerance * abs(reference)NumPy documents this form for allclose, including the fact that its arguments are not treated symmetrically and that the default absolute tolerance may be unsuitable near zero. The allclose documentation makes the comparison policy worth reading rather than borrowing by habit.
For a reference value of zero, the relative term vanishes. Comparing 1e-9 with zero under an absolute tolerance of 1e-8 passes this test; under 1e-12, it fails. Neither choice is universally correct. Units, expected scale, and the consequence of the discrepancy determine the useful tolerance.
Special values need an explicit policy as well. A missing value, a non-finite result, and a valid zero should not become interchangeable merely because a test helper permits a convenient option.
Version the behavior consumers rely on
A library can handle a convention change in several ways. It can retain the previous named behavior, introduce a new opt-in convention, or migrate a default under an explicit compatibility policy.
The choice depends on the declared contract. If consumers were promised a particular convention, changing that convention is a change to the promise even when the function name remains the same. A bug fix can also have significant downstream consequences when historical outputs are retained or decisions depend on a boundary value.
Semantic Versioning assigns meaning to version increments relative to a declared public API. The specification is useful for communicating compatibility intentions. It does not test numerical behavior or decide how a product should migrate an interpretation policy.
A serious migration record should identify which behavior changed, why, which inputs expose the change, and how a consumer can reproduce the old result when that remains necessary. If the product chooses to stop supporting an old behavior, that decision should be visible rather than an accidental consequence of installing a package.
Explain changed outputs at the point of use
A user comparing an old report with a new one needs to know whether the input changed, the calculation changed, or the interpretation changed. A package release note may be too far removed from that task.
The result record can link to the relevant convention and algorithm version. A comparison view can explain which dependents were recalculated and whether a new value supersedes an earlier output for the current use.
Equal outputs also deserve careful handling. Two runs can return the same number under different rules or source context. They are numerically equal while remaining distinct production records. That difference may matter when a future update affects one set of dependencies but not the other.
The same care applies to hashes and signatures. They can help establish which retained representation was examined. They do not establish that the selected statistic answers the user's scientific or business question.
Caches must identify the same calculation
A cache can preserve an old result while hiding the fact that the requested calculation changed. A key based only on a method name and input values can return the type-7 quantile after a default has moved to another convention. The cached number is valid for the previous request and misleading for the new one.
The cache identity should include the context that distinguishes those requests: the relevant input version, preparation policy, statistical convention, and algorithm contract. The exact representation depends on the system. A digest is useful only if the retained representation contains the distinctions the caller relies on.
This is another reason to make semantic versions explicit rather than infer them from deployment time. A cache hit should mean the stored result satisfies the requested computation, with enough context to explain that decision. Clearing a cache during every deployment can conceal an incomplete identity contract without solving it for historical replay or independently deployed consumers.
Make compatibility a reviewable engineering decision
A bounded first milestone can focus on one calculation that several consumers rely on. Document its mathematical definition, preparation rules, execution context, expected results, and comparison policy.
Verify a small set of hand-computable and boundary cases, then add difficult references appropriate to the method. Walk a proposed behavior change through a historical output and a current consumer. Ask whether the change can be explained, reproduced, and adopted without hidden drift.
A reviewer can then identify the question the software answered, reproduce its numerical behavior, and explain why a later output differs. The people responsible for the result can decide how that change affects its use, with the calculation and its limits available for inspection.
Protabyte helps teams build scientific and data-intensive software with explicit calculation behavior and explainable change. Talk with us about a result your team needs to reproduce, validate, or migrate.