Tests prove one observable mathematical or transport contract at a time.
Prefer one test at the highest useful boundary, in this order:
- Full end-to-end behavior through a real caller-visible path, with real application components and no mocked data or API boundaries.
- Integration behavior across schemas, APIs, and producer-consumer seams.
- Golden cases using realistic inputs and exact expected results.
These forms can overlap: an end-to-end test can also be an integration golden. Keep a lower-level unit test only when it proves a distinct mathematical invariant, admission rule, or failure mode that the higher-boundary test cannot exercise. Do not mirror implementation methods or rerun the same fixture in multiple lanes to satisfy a checklist. Classify evidence by the boundaries it actually crosses; an owner-local operation call is not an end-to-end test.
test-focused, quick-scoped, and handoff-scoped serve the edit loop.
The handoff commands add static checks to one declared owner test path;
their names do not imply complete branch evidence. Use make affected for
final branch validation.
During edits, use make test-focused LANE=... TESTS=... for the changed owner
and relevant composition tests. Use make quick-scoped with explicit PATHS
when scoped Ruff checks are also useful. These existing commands are the normal
edit loop; do not rerun the complete branch plan after every small change.
Run planner-selected affected validation on the final tree before sharing a code change:
make setup
make affected AFFECTED_BASE=origin/mainmake affected resolves AFFECTED_BASE...HEAD together with staged, unstaged, and
untracked paths, then uses the checked-in pull-request planner to run only the
selected owner, catalog, and boundary lanes. Ruff and mypy run only on changed
Python files in their normal repository scopes. It
prints the immutable plan and its reasons before execution, streams each command's
stdout and stderr, and reports per-command elapsed time and a final success summary.
Failures stop execution and identify the command, status, and exit code.
make affected-plan prints the selection without running it. Earlier shared-runtime
changes on the branch can still select broad lanes; focused edit-loop checks do
not narrow the final plan. Installed-wheel evidence remains
CI-owned. For a deliberately narrower one-owner loop, use make handoff-scoped
with explicit PATHS. Add the narrowest named boundary lane when a change
crosses its real boundary:
| Change | Additional check |
|---|---|
| MCP tool schema or transport | make test-mcp |
| Focused mathematical edit loop | make handoff LANE=math TESTS=tests/math/logic/test_tools.py |
| Cross-owner behavior | make test-integration |
| Advertised invocation examples | make test-catalog-examples; Singular-owned examples use make test-singular |
| Child-process behavior | make test-process |
| Singular-backed ideals, polynomial maps, projective singularity profiles, or marked catalog examples | make test-singular |
| QEPCAD plane-component backend | make test-qepcad |
Shared process runner (src/jacobian/process.py) |
make test-process, make test-singular, and make test-qepcad |
| Documentation | make docs-linkcheck |
make quick LANE=... TESTS=... omits mypy but retains the repository-wide Ruff
check. In a shared checkout, use make quick-scoped LANE=... TESTS=... PATHS="src/... tests/..." to scope both Ruff and the test path. The explicit
lane preserves its configured timeout and worker count. Supported
focused lanes are math, catalog, dispatch, cli, tooling,
integration, process, and mcp; Singular and QEPCAD retain dedicated commands.
Use the Make entry points rather than reconstructing owner-wide or broad
pytest -n N commands. Setting MATH_WORKERS beside a direct pytest command
does not control pytest's -n option or acquire a validation lease. For one-owner
edits, keep explicit TESTS on make test-focused or make handoff-scoped;
use make affected for final-tree evidence.
The local math lane defaults to two workers. Before a broad math run, set
MATH_WORKERS deliberately on make test-math or make affected to suit the
host's available memory and concurrent workload, not just its CPU count.
The setting also reaches focused math commands
and preserves their selectors, markers, and timeout. Use MATH_WORKERS=1 for
the conservative isolated-worker path, or MATH_WORKERS=0 to run in the main
process, which can avoid worker startup for a small focused test. Each extra
worker collects the suite and imports its backends separately; account for
concurrent solver children and exact intermediates as well as worker memory.
Do not derive a worker count from CPU count alone. Hosted math shards retain
their separate one-worker configuration because each shard already has its own runner.
MATH_WORKERS also reaches the math lane in make check-all and
make test-full; other lanes retain their own worker settings. In particular,
make check uses the fixed two-worker mixed-owner test-fast lane, so
MATH_WORKERS=1 does not reduce that command's concurrency. If that mixed lane
does not fit, use the affected plan or, when broad ordinary evidence is actually
needed, make check-all MATH_WORKERS=1 after accounting for its other lanes.
The validation lease coordinates
selected broad commands within one worktree; it does not budget host memory or
coordinate other worktrees and desktop applications.
In a shared checkout with unrelated static drift, declare the source and test paths you own instead of waiting on unrelated files:
make handoff-scoped \
LANE=math \
TESTS=tests/math/graphs/test_graph_distance_matrix.py \
PATHS="src/jacobian/math/graphs tests/math/graphs/test_graph_distance_matrix.py"PATHS is required and is passed directly to Ruff and mypy; use repository
paths only, without pytest node IDs. This is additive local evidence, not a
replacement for final planner-selected validation or CI's full static gate.
make check is an explicit broad local gate: it runs lint, types, and the math,
catalog, dispatch, CLI, and tooling owners, excluding property, exhaustive,
and scale evidence. make check-all adds the ordinary integration owner.
make test-full is the exceptional complete local reproduction. Do not use a
broad or full suite as a substitute for a focused regression test.
The scale lane defaults to two workers because its exact boundary cases can
consume substantially more memory than ordinary tests. Hosted CI explicitly
sets SCALE_WORKERS=4; contributors can keep the default or override it for a
host with less capacity.
Use make affected for normal branch-local validation and
make handoff-scoped LANE=... TESTS=... PATHS="..." for a one-owner edit loop.
Use make check on a frozen tree when broad ordinary evidence is requested or
the changed scope needs it. A completed affected plan does not automatically
require another broad run. make check-all adds ordinary integration tests;
make test-full includes all advertised invocation examples plus the named
process, MCP, and external-backend lanes. It routes non-Singular examples through
test-catalog-examples and Singular-owned examples through test-singular.
Neither command reproduces hosted packaging, coverage, or all CI jobs.
make check, make check-all, and make test-full take a worktree-local
non-blocking validation lease. make validation-status identifies a competing
broad run immediately; focused, affected, and standalone make test-math
commands do not take that lease. Coordinate concurrent runs and apply the
math worker policy even when using Make. Use
ALLOW_PARALLEL_VALIDATION=1 only for an intentional parallel broad run on a
host with known capacity; this override applies to check and check-all,
not test-full.
CI lanes using the shared run-test-lane action emit a JUnit artifact and a
worker-timing sidecar retained for 90 days. Download both to identify slow
testcases, xdist call-time skew, and the non-call wall remainder before changing worker counts, fixtures, or shards:
make test-timings JUNIT=pytest.xml TIMING=timing.jsonPull requests normally run static validation. The checked-in CI planner selects changed mathematical owners and changed dispatch, CLI, tooling, integration, process, MCP, Singular, QEPCAD, and installed-wheel boundaries. A public operation, model, admission, or canonical contract change also selects catalog conformance, the direct-dispatch lane, and the advertised-example integration test. Shared runtime, CI, dependency, and unmapped paths without a narrower ownership rule fail closed to the complete ordinary suite.
The linear-optimization source owner also selects the non-catalog integration
lane, including its existing source-coordinate LP tests in
tests/integration/linear. This relation covers private models, admission,
normalization, and kernels as well as public entry points. It matches the direct
src/jacobian/math/optimization package; independent nested owners such as
optimization/submodular retain their own evidence selection. Keep source-owner
integration relations in tools/ci_test_plan.py, and regress them with source-only
changes so edits to integration tests cannot conceal a missing dependency.
The planner selects Singular and QEPCAD independently. Changes to
src/jacobian/process.py select both runtime suites and the process boundary
lane because both backends use the shared supervisor. Backend-specific changes
select their corresponding runtime suite. When adding or moving a backend
consumer, update its source/test ownership in tools/ci_test_plan.py and include
its installed-backend tests in the corresponding Make target. Add a planner
regression using the actual repository-relative source path with no other
changed files; a test-file change or broad fallback can otherwise hide missing
source ownership.
When automatic linting adds a formatting commit, the default GitHub token does
not emit the ordinary pull-request synchronization event. The lint workflow
therefore dispatches the same CI workflow with the exact pull-request base and
head revisions. A pull request may briefly show a withheld, queued, or duplicate
event record while that dispatched run is starting; identify the evidence by
its head SHA and the required result, not by an older event record. A token
with permission to emit pull-request events follows the normal synchronization
path instead.
Merge-group candidates run the complete ordinary suite plus optional
near-envelope scale evidence. main runs the landed-tree ordinary suite and
coverage. Scheduled validation owns exhaustive evidence, repeated property
checks, optional scale evidence, and randomized order/provider variation; the
deferred exhaustive and scale lanes run as independent jobs so they do not
serialize one another.
Each ordinary CI lane retains JUnit timing evidence for 90 days. The complete
math lane uses four timing-balanced jobs; focused mathematical owners use the
planner's narrower selection. Use node and worker timings to distinguish
expensive evidence from collection, setup, and scheduling overhead before
changing a lane's resource policy. Collection inspection remains a manual
diagnostic, not a prerequisite for every test lane.
Markers are execution tiers, not synonyms for slow tests:
| Marker | Evidence | Owner |
|---|---|---|
property |
Repeated or generalized invariant checks | Scheduled validation; one run in make test-full |
exhaustive |
Broad finite reference sweeps | Scheduled validation; one run in make test-full |
scale |
Optional exact near-envelope execution proof | Merge-group and scheduled validation; one run in make test-full |
Keep a small ordinary regression for the same public behavior when moving a
near-envelope case to scale.
A test_exhaustive_* name is not a marker claim. That prefix is a
repo-wide convention for a test that enumerates a complete small finite domain
rather than sampling it, and it is orthogonal to the exhaustive marker: a
marked sweep may or may not use the prefix, and most tests that use it are
cheap enough to belong in the ordinary lane. Add the marker when a sweep is
genuinely expensive to run repeatedly, and not merely because it enumerates
everything in a small domain. Renaming an unmarked sweep to remove the prefix
would make it inconsistent with the rest of the suite without changing which
lane it runs in. See issue #4362.
The marker-lane Make recipes derive their default collection roots with
tools/marker_test_roots.py and then retain pytest's -m selection inside those
files. Marker ownership must therefore use direct pytest.mark.property,
pytest.mark.exhaustive, or pytest.mark.scale syntax; aliases and dynamically
constructed marks are not collection owners. This narrows collection without
replacing markers as the evidence-tier source of truth. Explicit TESTS=...
overrides remain available for focused reproduction.
singular_catalog_example is a runtime-ownership selector, not an evidence-tier
marker. test-catalog-examples excludes it, while test-singular selects it so
those advertised examples execute with the required system backend.
For representation or admission changes, pair an expensive-case rejection test with a structurally simple, larger accepted case. Exercise the public operation and verify both the result and retained mathematical context, such as ambient axes or source-coordinate mappings. Larger input size alone must not imply greater computational cost.
Use the same newly accepted result for the boundary round trip. For example, if inertia admission grows from dimension four to sixteen, computing the 16-dimensional result and round-tripping a separate 2-dimensional example is insufficient. Round-trip the 16-dimensional result and check it with the actual consumer. Trusted kernel construction can bypass model validation, so a correct native result may still violate the published result schema. Keep canonical carrier limits separate from operation budgets instead of copying a kernel cap into result fields.
For a resource-refusal regression, use a mathematically valid source or true
serialized claim that exceeds the admitted work or growth bound. Verify that
the public wrapper and consumer preserve the resource error. Returning False
would incorrectly refute the claim; relabeling it as invalid mathematics would
hide the actual limitation.
A review finding is closed only when the behavior is fixed at the public boundary and a behavioral regression proves the failure can no longer recur. For findings about malformed or over-sized input, cover the smallest valid near-boundary case, the malformed boundary form, and deeply nested data when canonicalization or parsing is involved. For findings about exact results, cover the defining invariant. Add schema/runtime parity, native/MCP parity, producer-consumer round trips, or late serialization and deadline cases when the finding affects those boundaries. Prefer tests that exercise the supported interface; do not assert private helper names or copied source text.
For a widened execution envelope, cover three distinct cases: the smallest formerly rejected valid request, a structurally simple request at the full carrier boundary, and a request near the derived output or storage bound. Assert every schema-representable limit, runtime acceptance and rejection, result shape, and serialization separately. For derived cross-field admission, inspect the schema-visible description and valid example, then exercise the exact accepted and rejected relation through owner admission. When measurements support an algorithmic repair, freeze the workload, environment, completion condition, and measured phase; report repeated before/after results from that harness. Timings are diagnostic evidence and must not become machine-sensitive correctness thresholds.
tests/math boots kernels. A module under tests/math must not import
jacobian.catalog.builtins, jacobian.catalog.catalog, jacobian.dispatch,
jacobian.cli, or jacobian.mcp; those start the catalog, bootstrap every
declaration, and cost
far more than the math it would exercise. Drive the kernel directly instead, and
pass canonical values rather than a request model when the kernel's signature
takes them. The rule covers direct imports and package-form imports such as
from jacobian.catalog import builtins, including aliases. Contract-only imports
from jacobian.catalog.models remain allowed.
Place new tests according to the boundary under test and its existing Make target:
tests/catalog/covers immutable catalog membership, declarations, schema checks, and discovery.make test-cataloguses a 30-second per-test timeout.tests/dispatch/covers strict parsing and direct operation-ID execution, including regressions for individual operations and native-versus-dispatch parity checks.make test-dispatchuses a 120-second per-test timeout.tests/integration/catalog/contains advertised-example execution coverage, including the aggregate example runner, and broader operation-composition checks. An advertised example used only as a regression fixture does not change the owning boundary: a direct-dispatch regression still uses the dispatch lane.
The metadata-only authored-phrase recall probe is
uv run python tools/discovery_phrase_recall.py --output phrase-recall.json.
Its default five-result cutoff is stricter than the ordinary MCP math.find
query page, which defaults to ten results; --limit can select another
supported page size. The report groups case/whitespace-equivalent
phrases, retains their authored spellings and distinct owners, and records each
owner's first-page rank and aggregate recall. It does not execute operations or
measure mathematical correctness or agent selection. The catalog conformance
sweep has an explicit 120-second budget because it queries every authored
multiword phrase rather than one ordinary lookup.
Some existing catalog modules still contain direct
dispatch and example-execution checks. These are legacy placements that have
not been comprehensively migrated. Put new execution regressions in the lanes
above; when moving legacy tests, preserve their fixtures and select the owning
Make target so they receive its execution budget. Keep every catalog-backed
check outside tests/math/.
test_math_tests_do_not_boot_complete_product_boundaries enforces the math
lane's import boundary. It does not enforce placement among catalog, dispatch,
and integration lanes. Treat a failure of that import guard as a lane placement
problem: move the test, do not relax the gate.
For an operation, prefer one representative public-path test that checks its typed request and observable result. Add a separate request-boundary case only for an accepted or rejected limit not exercised by that path. Test a caller-visible transport only when that projection changed. The catalog integration test already executes each advertised example and validates its request and result round trips; add an exact golden when the expected mathematical value is the claim. When one result feeds another operation, serialize the producer result and pass its canonical value unchanged through the consumer's typed payload. The test should fail if a caller would have to reconstruct mathematical context or translate between parallel representations.
When a consumer relies on a stronger mathematical property than its input's structural type establishes, add an adversarial boundary fixture: construct a structurally valid value that violates the required property, and prove that the consumer rejects it or returns the declared typed non-applicability outcome. Pair that fixture with a positive producer → serialization → consumer test using the canonical value unchanged. Shape validation and a separate producer test do not by themselves establish safe composition.
Also submit a forged serialized subtype directly to the public consumer without
calling its producer first. The test must prove that the consumer establishes
the mathematical property it relies on. A nominal subtype tag, validated
field, or producer-shaped payload must not cross the stateless boundary as
proof.
Where native callers can bypass Pydantic validation with model_construct(),
model_copy(), or direct mutation, submit an equivalent forged claim to the
real consumer or verifier. Preserve valid container shape and scalar bounds so
the test reaches the mathematical relation rather than stopping at structural
parsing.
When an operation is added or changed because of a source-backed gap, preserve at least one minimally reduced motivating request as a behavioral regression. Run it through the final public request model and operation, and assert both admission and the typed mathematical result. A smaller happy-path example does not replace the request that established the need. Include the degenerate producer case most likely to erase ambient information, such as an empty basis, zero-row matrix, empty trace, or zero count. For source-bound decisions whose public contract accepts independently supplied conclusions or certificates, mutate the source and conclusion independently and require the consuming operation to reject both forgeries. For ordinary computed results, test the defining invariant of the returned value; do not introduce a replay path solely for that test.
For an operation that publishes an isolating interval, rectangle, or box, the
defining invariant is separation: each object contains exactly one root of the
relevant complete axis polynomial, and objects on one axis are pairwise
disjoint. Check both with an independent oracle such as strict_root_count or
count_roots over the returned box, not by re-reading the producer's own
refinement. Do not narrow the obstacle set to the values that are convenient to
compare: a guard that separates only rational, real, or same-factor siblings is
a partial-obstacle-set bug even when every accepted bound passes.
Generate near-degenerate structure deliberately instead of sampling uniformly,
because a uniform draw will not hit a gap smaller than a naive enclosure. A
property-based strategy should construct the degeneracy, for example a surd
pair sqrt(a) and sqrt(a + 1/q) with a large q, a Pell rational
approximation of a quadratic irrational, or a reflected root. Then assert the
defining invariant on every generated case. Pair that property with one
minimally reduced motivating regression through the final public operation, and
cross-check any parallel refinement against an independent oracle the way a
base, Karatsuba, and generic algorithm are compared.
For every exact-success branch, assert the defining equation or preservation law and reject every representable status/diagnostic combination that would claim success while admitting that the invariant failed. Checking only that a step list, factor list, witness, or certificate-shaped object is nonempty is not correctness evidence. For non-success branches, assert that exact witness fields are absent unless the contract explicitly assigns them another meaning. When branches have different fields or field meanings, inspect the generated schema for a discriminated union and validate every branch through both the declared result type and final transport projection. Construct contradictory payloads directly; proving only that the producer does not emit them is insufficient.
Center correctness tests on regression evidence when fixing a concrete
regression, defining identities, consistency checks, and property-based tests
against actual symbolic or mathematical behavior. Use maintained libraries in
their owning domain tests rather than mocking their algorithms. Exercise real
mathematical and numerical assertions at public typed interfaces with
deterministic inputs; keep fixtures sparse and name the evidence
each one contributes. Do not monkeypatch or use test fakes for mathematical
values, validators, serializers, or backend correctness. Keep
unavailable-environment and transport-failure tests in separate boundary
cases; they must not stand in for mathematical correctness. A timeout,
cancellation, unavailable external executable, or solver UNKNOWN is never a
positive mathematical conclusion.
When a test does use monkeypatch, it must assert something about the patched
behavior: an ordinary assert, a typed-error check (pytest.raises or an
owner-local raises helper), an assert_*/_assert_* helper, or a sentinel
replacement whose body raises (the "this must not run" pattern). A patch with
none of these proves nothing about the behavior it replaces. make test-hygiene
enforces this over tests/; a test may waive the rule with a
# monkeypatch-evidence: <reason> comment when the replacement itself is the
observable behavior. Prefer explicit dependency seams so most code can be tested
without patching a global name.
Public typed operations own the authoritative mathematical evidence. Test convention-sensitive known answers, defining identities, reconstruction, metamorphic laws, producer-consumer composition, and bounded independent oracles through that interface. A private-kernel test may supplement this evidence when an internal algorithm or admission formula has its own defining invariant, but it must not replace the representative public test for the mathematical behavior.
When a change affects more than one calling surface, exercise the same canonical fixture through each affected surface. Do not repeat unchanged surfaces solely because the operation is callable there. A success on one surface does not establish changed contracts on the others: transport and shaped validation establish projection and schema agreement, not mathematical truth.
Keep scalar codec conformance in its owner: native construction, Python dump and revalidation, JSON round trips, schema agreement, malformed encodings, and large exact values. Domain tests add representative nested, empty, and producer-consumer cases rather than duplicating the entire scalar suite for every operation. A raw preflight change must exercise both Python and JSON validation modes; passing one does not establish the other. See the codec boundary contract.
Treat a verifier as independent evidence only when it checks the claimed relation by a different route. If it recomputes the producer result with the producer's own helper or backend, pair it with an independently defined identity, relation, or oracle; a forged claim that genuinely traverses the check; and a valid claim produced independently of that producer. A same-code verifier checks transport, deserialization, and forgery handling; it is not the defining invariant.
Keep process and transport evidence focused on their boundaries. Prove worker startup, isolation, cancellation, timeout, codec, and result projection with the smallest discriminating cases; do not make a broad mathematical sweep pay for repeated worker or transport setup merely to duplicate evidence already owned by those cases. Moving a slow case into an optional lane changes coverage scheduling, not its computational cost; report it as such rather than as an algorithmic speedup. When startup dominates an otherwise useful sweep, retain a deliberately varied public corpus and use direct kernel cases only for additional internal evidence.
Before trimming or accelerating a slow test, identify whether its cost comes from mathematical coverage, backend computation, process isolation, or transport. Consolidate repeated setup or duplicate execution only after naming the evidence each retained fixture contributes. Do not trade public mathematical evidence for a faster private implementation test. Golden examples anchor conventions and regressions; pair them with a defining invariant, metamorphic property, reconstruction, or independent oracle when they support a broader correctness claim.
Boundary rejection tests do not replace a realistic admitted workload. For a new or repaired operation, run at least one motivating interior request through the final public boundary and assert its typed mathematical result. When the operation has multiple mandatory phases, test that they share one deadline and work ledger rather than receiving independent wall budgets. Test caller-side timeouts separately in the MCP lane; their only conclusion is that transport aborted the request.
Dispatch rejection tests must name the boundary they exercise. A malformed or
structurally invalid payload raises OperationRequestValidationError. A
structurally valid payload rejected by native mathematical admission raises
OperationDomainValidationError; where practical, assert that
native and dispatch calls preserve the same structured owner error code. MCP
projects both validation classes through the model-visible tool-error channel
with bounded structured diagnostics. Timeout, cancellation,
host or worker failure, and unexpected backend failures remain
operational errors and project as agent-visible tool failures. Test that these
paths establish no mathematical result and do not become invalid-parameter
responses. Do not update a semantic admission test to expect request-model
validation merely because both validation failures appear as an
invalid-parameter response over MCP.
Native boundary tests must include empty, mismatched, and malformed values that could otherwise reach indexing, tuple unpacking, helper arithmetic, or result construction. Assert the structured domain error and stable owner code rather than accepting a raw built-in or Pydantic exception.
Add a backend or verifier call-count regression only when repeated work is a demonstrated cost or correctness risk. Exercise the owning public boundary, assert its mathematical result, and keep the call-count assertion limited to that risk; do not pin internal call structure by default. If independently supplied claims are supported, prove a forged claim fails through the explicit verifier. Timing tests must state whether serialization is included in the reported duration or exposed as a distinct named phase; no mandatory phase may be absent from all timing evidence.
Examples prove that a documented path works; they do not establish an algebraic claim. Select evidence by the claim being made:
| Contract claim | Required evidence |
|---|---|
| Request domain | Accepted and rejected boundary cases |
| Backend domain | Supported edge and an immediately unsupported case |
| Exact decomposition | Reconstruction property |
| Canonical value | Normalization and round-trip property |
| Theorem-bearing subtype | Invalid structural candidate and forged validated payload rejected, then recognizer → serialization → consumer success |
| Exact-success state | Defining invariant holds, schema exposes discriminated branches, and contradictory combinations are rejected |
| Parent identity | Incompatible-parent rejection and explicit-map success |
| Algebraic operation | Defining identities or an independent oracle |
| Isolating interval, rectangle, or box | Exactly one root by an independent count, pairwise disjointness, and a constructed near-degenerate sibling |
| Enclosure or distance certificate | Independent containment oracle and the defining inequality at its boundary |
| Public operation | Accepted-request mutation conformance |
| Process backend | Codec, version, timeout/output, and typed failure tests |
When correcting an admission or boundedness rule, inspect the affected owner lane for assertions that pin the previous accept/reject outcome and rewrite or remove them in the same change. Every retained rejected boundary case must fail for the guard it names: keep the rest of its payload valid and assert the owner's structured error code. A request rejected by an earlier field bound is not evidence that a later cross-field admission guard ran.
For example, an ideal radical needs containment and radicality evidence or an independent bounded oracle. A factorization needs reconstruction, retained unit, and positive-multiplicity properties. Known answers remain useful regressions, but they do not replace these defining properties.
The recording domain does not determine the evidence; the mathematical result family does. The recurring patterns below name the invariant, the independent oracle, and the degenerate structure a generator must construct. Start from the family and adapt the owner's carrier and admitted bounds.
| Result family | Defining evidence | Independent oracle | Constructed degeneracy |
|---|---|---|---|
| Exact arithmetic and modular identity | Reconstruction or reduction identity | Independent formula, or exhaustive enumeration inside a small admitted bound | Zero, unit, negative, and maximum-height values |
| Polynomial factorization and decomposition | Product reconstruction, retained unit, positive multiplicity | Maintained backend factorization | Repeated factors, near-equal irreducible factors |
| Root identity and isolation | Exactly one root per object and pairwise disjointness on the axis | strict_root_count, count_roots, or eval_rational on the support |
Surd or Pell sibling below the naive enclosure; purely real/imaginary and repeated roots |
| Exact linear algebra | Defining identity (A A^-1, L U, characteristic polynomial) |
Exact elimination or a maintained exact backend | Rank-deficient, singular, and identity-shaped matrices |
| Finite algebra and group structure | Closure, associativity, inverse, and order axioms | Exhaustive finite model, independent order computation | Zero ring, singleton group, non-associative and non-abelian cases |
| Finite graph or labelled linear-algebraic claim | Complete edge relation, degree or regularity invariants, rank as the largest nonsingular minor, and pivot coherence | Independent edge-difference, minor, or determinant computation | Boundary orders, singleton or degenerate axes, transposed or relabeled axes, and forged edge directions |
| Combinatorial enumeration | Equality with brute force and a bijection or symmetry law | Exhaustive generation on small sizes | Empty, singleton, and complete families |
| Order and lattice structure | Order axioms and a Galois or adjunction law | Exhaustive finite poset | Empty, antichain, chain, and repeated-element cases |
| Exact probability and generating functions | Exact identity, sum-to-one, or coefficient reconstruction | Independent exact summation or coefficient extraction | Empty support, point mass, and maximum precision |
| Analytic enclosure | Certified containment of the true value | High-precision independent evaluation | Endpoint, zero-width, and branch-cut cases |
| Sign, orientation, or branch convention | Metamorphic law in both directions (antisymmetry, parity, row or argument swap, product identity) | Independent definition such as a Sylvester determinant or evaluation off the branch cut | Odd and even degrees, equal-degree and reordered operands, branch-cut and negative-assumption inputs |
| Geometric or topological invariant | Incidence, orientation, or reconstruction equation | Independent coordinate or boundary computation | Degenerate simplex, repeated point, and boundary cycle |
When a listed oracle is unavailable for a carrier, a weaker invariant does not replace it: either strengthen the oracle or report the narrower claim and the unresolved case rather than presenting the weaker check as the required evidence.
A typed rejection's stable contract is its error code, not its
human-readable message. Assert the code. The message is prose: it may be
reworded, and a different internal bound may trip first, so a match= on the
message breaks opaquely even while the rejection is still correct.
For coded errors (OperationDomainValidationError,
OperationResourceAdmissionError, and other typed owner errors), capture the
exception and assert its code:
with pytest.raises(OperationDomainValidationError) as exc_info:
invoke_operation("polytope.facets.compute", payload, Catalog.open())
# The contract is a typed admission rejection with this stable code; the message
# wording (and which internal bound trips first) may change without changing it.
assert exc_info.value.errors()[0]["type"] == "polytope.facet_profile_not_admitted"Do not pin the message with
match=f"{MAX_COMPUTED_FACETS}-facet result bound". Interpolating the constant
tracks the number but still pins the surrounding wording, so a reword or a
different bound tripping first fails the test even though the contract holds.
This is assertion debt: it couples the test to prose instead of behavior.
Some failures have no stable, reason-specific code. A bare built-in ValueError
has no structured error identifier. That does not extend to the owner errors
that subclass it: OperationDomainValidationError and
OperationResourceAdmissionError are ValueError subclasses, so a
pytest.raises(ValueError) block catches them too, and they do expose a stable
owner code through errors()[0]["type"]. Check for that structured interface
first and assert the code. A Pydantic ValidationError may expose a custom,
owner-specific errors()[0]["type"], or only a generic type such as
value_error. Prefer the specific type whenever it exists. Do not treat a
generic type as identifying a particular bound when the message is the only
thing that distinguishes it.
For a code-less failure, first make the test input otherwise valid and isolate the boundary it is meant to exercise. If rejection alone establishes the contract, assert the exception class and keep the message out of the test. A message match is justified only when the exact failure reason or ordering is itself part of the test and there is no stable identifier for it. Prefer adding a stable owner error code when callers or tests need to distinguish such reasons. When a message match is justified, derive any embedded bound from the owning source constant rather than hardcoding it, so a scale-cap change does not break the assertion:
from jacobian.math.number_theory.diophantine_approximation._models import (
_convergent_component_digit_cap,
)
# A bare built-in ValueError, which has no errors()[0]["type"] to assert.
cap = _convergent_component_digit_cap(4)
with pytest.raises(ValueError, match=rf"{cap}-digit"):
...Prefer adding a stable code to an owner error over growing message-text matches. Boundary test inputs (the value that triggers the error) should likewise be computed from the constant, not hardcoded:
beyond = "9" * (_MAX_MULTIVARIATE_COEFFICIENT_DIGITS + 1)When the owning constant is private (underscore-prefixed), importing it from its owning module is acceptable in tests — the test and the source share one definition.
Before implementing an exact decomposition, certificate, or authoritative derived value, state its defining invariant: the reconstruction equation, preservation law, or independently checkable property that makes a returned value mathematically valid. Then name the smallest set of tests that establishes that invariant and rejects plausible results that satisfy only a weaker mathematical claim.
When authoring or changing an operation's contract or backend adapter, derive at least one expected value from the mathematical definition rather than from the maintained backend before merging: a worked computation, brute-force enumeration on a small admitted slice, or an independently constructed object such as a determinant assembled directly from coefficients. Agreement with the backend is not independent evidence — every code path in one backend can share one defect. Where the operation already advertises an identity, such as reconstruction, a swap law, or invariance under a change of basis, test that identity against the same change. Before trusting backend output for a new claim, consult the known backend defects registry; add an entry whenever an adapter compensates for backend behavior.
Do not obtain expected answers by copying the implementation's branch or snapshotting its current output. A passing test can preserve the same mistake as the code. Derive the expectation separately and explain the convention when it matters. For matrix formulas, write down dimensions and whether bases are rows or columns before choosing fixtures; square diagonal matrices often hide wrong multiplication order, transpose, and orientation errors.
Use transformations and counterexamples that discriminate between plausible implementations:
| Risk | Discriminating evidence |
|---|---|
| Wrong matrix orientation | Rectangular and nonorthogonal inputs; for a row dual basis, check D B^T = I and its returned Gram matrix. |
| A proxy replaces a mathematical property | Check the defining predicate: covolume rationality depends on whether the integer Gram determinant is a perfect square, not on codimension. |
| Ordering affects a mathematical conclusion | Permute constraint rows or apply a valid change of basis; compare objectives, statuses, or equivalent mathematical results, not incidental witness ordering. |
| Candidate data is treated as established truth | Alter source and certificate fields independently and exercise the actual public consumer. |
| A safety bound makes the operation unusable | Execute zero, identity, sparse, and representative dense inputs at useful orders, including immediately across an admission threshold. |
| A producer outgrows its consumer | Serialize expanded outputs at the supported boundary and pass them unchanged to their actual consumers. |
Select the rows relevant to the change; this is not a requirement to add every fixture to every operation. Redundant constraints are another useful meaning-preserving probe when both forms are admitted. If a transformation changes admission, record that separately from mathematical correctness.
Classify each fixture by the evidence it contributes:
| Fixture role | What it establishes |
|---|---|
| Defining-invariant | Independently checks the reconstruction equation, preservation law, or certificate relation in tests. Production recognition of caller-supplied claims belongs to the consuming operation. |
| Convention/known-answer | Fixes terminology, normalization, indexing, signs, or another convention-sensitive value. |
| Adversarial weaker-semantics | Rejects a tempting result that has the right shape or satisfies only a weaker claim. |
| Metamorphic or equivalence | Checks invariance under a meaning-preserving transformation or compares noncanonical outputs by mathematical equivalence. |
| Producer-consumer | Supplies one operation's serialized result unchanged to a downstream operation and checks the composed meaning. |
| Scale/stress | Exercises a source-backed size or representation regime near a justified admission boundary. |
A fixture may fill more than one role, but the evidence plan should name those roles rather than treating fixture count as coverage. A large certificate should normally yield a small discriminating CI fixture, with the full source-scale case retained only as an optional stress or benchmark fixture when it remains reproducible and useful. Agreement on a scalar summary is insufficient when correctness concerns a set, partition, or family of canonical representatives: compare the exact objects or their declared equivalence class. For example, two tree generators can return the same count while duplicating one representative and omitting another.
Accepted and rejected boundary cases prove that a stated envelope is enforced; they do not prove that the envelope admits realistic work. Pair boundary tests with a source-backed accepted case even when its public dimensions are small. A compact request can still be the decisive scale fixture when private normalization, expansion, or backend representation makes it expensive.
For an admission-precision defect, preserve the motivating valid request as an accepted public regression with a defining-invariant check. First compare the estimate against a bounded real-backend diagnostic; do not infer expense from the estimate being tested. If the computation is cheap, a test expecting resource rejection would freeze the defect rather than fix it. Exercise related sizes, nontrivial structures, and equivalent presentations so the repair scales a mathematical regime instead of recognizing one fixture. Keep timing measurements out of brittle unit-test thresholds and separate measured output sizes from proved intermediate bounds. See the scale-first admission review.
Use a source-backed reference fixture when a standard example helps fix terminology, normalization, or another convention-sensitive output. Cite the specific theorem or example and record the convention the fixture depends on. Pair that fixture with the relevant boundary and adversarial cases and with a reconstruction, defining-identity, bounded exhaustive, or independent-oracle test. A reference fixture is an anchor, not the correctness argument.
When a valid result is not unique, compare its mathematical equivalence class or validate its reconstruction. Do not require incidental backend ordering, temporary identifiers, tree roots, or one particular witness unless the public contract makes that choice canonical.
Singular testing follows the same ownership split as other child-process
backends. The shared bounded-process supervisor owns process-group termination,
including descendants that ignore termination or retain inherited pipes. The
Singular lane includes mathematical consumer tests, its marked advertised
catalog examples, and adapter-specific behavior: timeout and execution-outcome
projection, supported-version enforcement, strict codec behavior, output limits,
and request-scoped cleanup. The remaining advertised examples belong to
test-catalog-examples; this keeps them in test-full without serializing them
behind the Singular runtime. Do not duplicate the supervisor's termination suite
for each mathematical backend.
The commutative-algebra domain checks Singular's mathematical results against an independent combinatorial oracle on bounded monomial ideals, in addition to containment, idempotence, identity, and colon-law properties. SageMath may be used as a development-time differential oracle, but it is not a runtime or required-CI dependency.
Documentation should describe current behavior rather than refactor history. Link to the product blueprint for product philosophy, and keep tool/reference pages focused on the contracts they own.
Run make docs-linkcheck after changing Markdown. It validates relative links,
documented Make commands, and documented test paths.