Testing Observable Behavior

Equivalence is the contract your tests define. Besides return values, consider exceptions, input mutation, receiver state, ordering, and numerical tolerances. Use independent expected results to check the reference itself, as in An Optimization Workflow; agreement alone does not prove either implementation correct.

Fresh Inputs for Mutating Functions

Both implementations receive the same argument objects if you pass them the same objects. Neither pytest parametrization nor measure() copies inputs. Create independent inputs and compare the resulting state as well as the return values. Use fresh instances for stateful methods, and fresh iterators for single-use iterable inputs.

import alternative


@alternative.reference
def sort_values(values: list[int]) -> None:
    """Sort the provided list in place."""
    values.sort()


@sort_values.add
def sort_values_copy(values: list[int]) -> None:
    """Replace its contents with a sorted copy."""
    values[:] = sorted(values)


@sort_values.pytest_parametrize_pairs()
def test_sort_state(reference, implementation):
    """Compare independently initialized mutable inputs."""
    left, right = [3, 1, 2], [3, 1, 2]
    assert reference(left) is implementation(right) is None
    assert left == right == [1, 2, 3]

For nested mutable inputs, construct independent nested data or copy it to the depth required by the contract. Reference caching is disabled in this example; caching mutable results can cause one test case to change another case’s oracle.

Exceptions Are Part of the Contract

Check success and failure cases separately. Compare exception classes and only compare messages when the API promises their exact wording:

import pytest


@alternative.reference
def parse_decimal(value: str) -> int:
    """Parse a stripped decimal integer or raise ValueError."""
    return int(value.strip())


@parse_decimal.add
def parse_decimal_explicit(value: str) -> int:
    """Use an explicit decimal base with the same failure contract."""
    return int(value.strip(), 10)


@parse_decimal.pytest_parametrize_pairs()
def test_parser_errors(reference, implementation):
    """Both implementations reject invalid decimal input."""
    for function in (reference, implementation):
        with pytest.raises(ValueError):
            function("not a number")

Numerical Comparisons

Choose absolute and relative tolerances from the application’s error budget and input domain, then compare using pytest.approx. Decide explicitly how to handle NaN, infinity, overflow, signed zero, and empty inputs. A tolerance should not hide a semantic difference such as changed ordering, precision, or exception behavior. Include cancellation and extreme-value cases relevant to the domain.

For example, these vector-length implementations use different floating-point operations. The test chooses a relative tolerance of 1e-12 and an absolute tolerance of 1e-15 for small, finite vectors:

import math


@alternative.reference
def vector_length(values: tuple[float, ...]) -> float:
    """Calculate length by squaring and summing components."""
    return math.sqrt(sum(value * value for value in values))


@vector_length.add
def vector_length_hypot(values: tuple[float, ...]) -> float:
    """Use hypot for the same vector-length contract."""
    return math.hypot(*values)


@vector_length.pytest_parametrize_pairs()
@pytest.mark.parametrize(
    ("values", "expected"), [((), 0.0), ((3.0, 4.0), 5.0), ((0.3, 0.4), 0.5)]
)
def test_vector_lengths(reference, implementation, values, expected):
    """Check expected lengths and compare within the chosen error budget."""
    reference_result = reference(values)
    assert reference_result == pytest.approx(expected, rel=1e-12, abs=1e-15)
    assert implementation(values) == pytest.approx(
        reference_result, rel=1e-12, abs=1e-15
    )

These example tolerances are not a general recommendation. For very large components, squaring can overflow while hypot still returns a finite length. If that input range is part of your API, the implementations need an explicit contract for it and tests beyond the small vectors above.

Different but Valid Outputs

Some algorithms produce different representations of an equivalent result. Check their meaning or invariants separately from cost. For compression, compare decoded bytes with the original input, then use measure(len, data) to compare compressed sizes. For query plans or circuits, validate the result before measuring estimated cost, depth, or gate count. measure() orders scores; it does not establish semantic validity or automatically select a default.

The compression example in Measure Implementations demonstrates validation before scoring. Use separate benchmarks when elapsed time is also part of the tradeoff.