test(conformance): adequacy criteria for a vector set, applied to every set here (#186)

A conformance vector set is a claim that a non-implementing verifier will fail it.
Nothing checked that claim. #169 and #170 were both found by asking it of a set
rather than of a vector, and both were sets that were passing.

Four criteria, each from a defect on a real set rather than from first principles:

- a set must fail both unconditional implementations, accept-everything and
  reject-everything, or it pins nothing
- each boundary needs more than one vector, since a single vector cannot
  distinguish a check that reads the head of a list from one that reads all of it
- every set on disk is measured here or named with the test that measures it
- shortfalls are recorded exactly, so they cannot widen unnoticed and the entry is
  deleted when someone closes the gap

Applied to every set in this repository. `build-provenance-depth` carries a margin
at every boundary. `canonicalization-boundary`, which I wrote, expects acceptance in
every vector and so cannot tell a conformant verifier from one that accepts
unconditionally; that is recorded rather than skipped, and the record is asserted so
it cannot grow. `action-receipts` is named as covered by test_vector_completeness.py
rather than graded twice.

The completeness guard is on the instrument itself for a reason. SETS is a
hand-maintained list of what gets graded, which is the defect these criteria exist
to catch, and the one place it would otherwise be invisible: a set added later would
simply not be graded and nothing would fail. Adding an unlisted set directory turns
the guard red, as does a stale entry, as does naming a test that does not exist.
Each was checked by making the change and watching the specific test fail.

Signed-off-by: lywinged <louie.lunz@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: lywinged

docs/conformance-method.md

@@ -0,0 +1,187 @@
+# How the conformance vectors are built, and how the set is checked for holes
+
+*Informative. This page describes method, not requirements, and carries no RFC 2119
+keywords.*
+
+A conformance suite makes a promise: an implementation that passes it has done the
+things the specification asks for. That promise is only as good as the suite's
+coverage, and coverage is usually assumed rather than established. A suite that never
+exercises a rule certifies implementations that skip that rule entirely — silently,
+and with a green badge.
+
+This page describes how the action-receipt vectors in
+`examples/action-receipts/conformance/`
+are built, and how the set itself is checked. The checking part found seven rules in
+the receipt verifier with no vector behind them.
+
+## 1. Vectors are data, not tests
+
+Each vector is a JSON file stating the inputs, the trust anchors, and the outcome any
+conformant verifier must produce:
+
+```jsonc
+{
+  "context":  { /* what the verifier is told: session, call, freshness policy */ },
+  "action":   { /* the action and its canonical reference */ },
+  "trusted_issuer_keys": { /* the pinned key set — nothing self-authenticates */ },
+  "receipt":  { /* the artifact under test, genuinely signed */ },
+  "expected": { "status": "...", "failures": [...], "warnings": [...] }
+}
+```
+
+Nothing in a vector names a language, a function, or an API. An implementation writes a
+small adapter that feeds the JSON to its own verifier and compares against `expected`;
+the vectors themselves are portable.
+
+This is worth stating because it is easy to get wrong. One set I wrote began as tests
+against a single library's function signature. Those tests were correct, and they were
+not conformance vectors: no second implementation could run them. The shape matters
+more than the assertions, and the two are easy to confuse while the assertions are
+passing.
+
+**Every artifact in a vector is genuinely signed, including in the negative cases.**
+A vector that expects a rejection must reject for the reason it names. If its signature
+were malformed, the rejection would come from the signature check instead, the vector
+would pass, and it would have stopped testing the rule in its own filename.
+
+## 2. The set is checked against the verifier, automatically
+
+The interesting question is not whether the vectors pass. It is whether they cover
+what the verifier does. Three questions, in increasing strength:
+
+**Are there dead expectations?** A vector naming a failure code that nothing can emit
+holds an assertion that can never fail. Usually it means a rule was renamed or removed
+and the vector was left behind.
+
+**Is every rule exercised?** For each code the verifier can emit, is there a vector that
+expects it? A rule with no vector is a check an implementation can omit while passing.
+
+**Is every rule load-bearing — twice?** The strongest form: *delete the rule and count
+the vectors that notice.* A rule can be named by a vector and still not be load-bearing —
+if another rule fires on the same input, removing it changes no outcome and nothing
+distinguishes an implementation that performs the check from one that skips it. And one
+load-bearing vector is existence, not margin: any change that weakens or retires that
+single vector silently removes the rule's coverage. So the floor is two
+([#124](https://github.com/agentrust-io/trace-spec/issues/124)), and a ratchet keeps
+anything above the floor from quietly decaying back toward it.
+
+**Are the two vectors different tests, or two copies?** Two identical vectors have
+margin two and prove nothing extra. #124's definition: vectors are independent if a
+single implementation defect causes one to pass and the other to fail. That is made
+executable by declaring, for every rule, at least one *weakened* variant of its check —
+a plausible implementation shortcut: comparing digest prefixes, case-normalising
+identifiers, granting clock tolerance, validating signature structure without
+cryptography. The suite fails unless some declared defect deviates one of the rule's
+vectors while leaving another undisturbed. The declaration is itself fail-closed: a
+rule with no declared defect fails, so "what bug would your second vector catch that
+your first would not?" is answered when the rule is added.
+
+The strong criteria subsume the first two questions, which are kept because they are
+cheap and their failure messages are more direct.
+
+### The inventory is the registry the verifier consumes
+
+The obvious implementation is a list of rules kept next to the tests. That list is
+guaranteed to drift: it is correct only until someone adds a rule and forgets it, and
+the failure is silent in exactly the direction that matters.
+
+The first replacement recovered the inventory from the verifier's source with `ast` —
+every string literal appended to a failure or warning list. The
+[#124 review](https://github.com/agentrust-io/trace-spec/issues/124) identified that
+this has the mirror failure mode: a rule written as `extend([...])`, `+=`, an f-string
+or a named constant is invisible to the walk, and the suite reports complete coverage
+over an inventory that is quietly missing entries.
+
+The resolution, from the same review: the verifier itself consumes an explicit registry
+of named rules, and the registry *is* the inventory. A check that is not registered
+never runs, so it cannot exist outside the inventory; a residual guard fails on any
+code that would emit around the registry. Mutation follows the same principle — a rule
+is deleted by rebuilding the registry without its entry, or weakened by substituting
+its check, never by pattern-matching source text. If no vector's outcome changes under
+a deletion, that rule has no vector standing behind it.
+
+## 3. Fixtures and their checker cannot vouch for each other
+
+A green run proves the vectors and the verifier agree. Both are usually written by the
+same person in the same sitting, so agreement is close to guaranteed and says little.
+The failure it cannot see: a shared helper that canonicalizes or decodes incorrectly,
+used both to generate the fixtures and to check them. Everything agrees, and agrees with
+nothing else in the world.
+
+So every signature is re-derived through a path that shares no code with either: it
+imports nothing from the library, reuses no helper from the vector modules, and
+reconstructs each signing input from the JSON directly. Any vector whose signature is
+not re-derived by that path is a failure unless it is explicitly declared
+signature-free with a reason — the "missing receipt" case is the only one, since the
+absence of a receipt is the thing it tests.
+
+## 4. What this found
+
+Seven rules in the receipt verifier had no vector at all:
+
+| Rule | What an implementation could have skipped |
+|---|---|
+| `action_ref_invalid` | Recomputing the action reference instead of trusting the declared value |
+| `call_id_mismatch` | Checking that the receipt is bound to *this* call |
+| `session_id_mismatch` | Checking that it is bound to *this* session |
+| `evidence_hash_mismatch` | Recomputing the evidence digest, so a swapped evidence body passes |
+| `issuer_key_unknown` | Consulting the pinned key set at all |
+| `receipt_from_future` | Rejecting a receipt issued after the verification time |
+| `decision_invalid` | Refusing to read an unrecognised decision verb as accept or reject |
+
+Each is a check a conforming implementation could have omitted entirely while passing
+the published suite. Two are load-bearing for the trust model rather than merely tidy:
+without `issuer_key_unknown` a receipt authenticates itself — a signature verifies
+against whatever key it names, and only the pinned set says which keys the verifier can
+check, so a receipt under an unpinned key is surfaced as unverified rather than valid or
+invalid — and without `evidence_hash_mismatch` the signature covers a digest whose
+document may have been replaced.
+
+## 5. The checker's own false positives
+
+Three of its first findings were bugs in the checker, not in the suite. They are
+recorded because a completeness checker that reports the wrong thing is worse than
+none: it converts attention into noise, and the next person stops reading it.
+
+| Symptom | Cause |
+|---|---|
+| Every rule reported as non-load-bearing | Mutants were executed into a bare namespace, so `@dataclass` raised before a single vector ran. It read as "every rule matters" while testing nothing. |
+| A real outcome reported as unreachable | The scanner walked an entire conditional expression and collected the literal being *compared against* as if it were an outcome. |
+| A real warning reported as emitted by no rule | It scanned inline `failures=[...]` arguments but not `warnings=[...]`. |
+
+The pattern in all three: the check failed *open* — it reported a problem where there
+was none, which is survivable, rather than passing where there was one, which is not.
+That direction was not designed in, and a checker of this kind should be built so that
+its own breakage is loud. Hence the guard that asserts the inventory was non-empty
+before any conclusion is drawn from it.
+
+## 6. What this does not establish
+
+- **Not that the rules are right.** Completeness is a property of the vectors relative
+  to the verifier. If the verifier implements the wrong rule, a complete set pins the
+  wrong rule down precisely.
+- **Not that the specification is complete.** A rule absent from both the verifier and
+  the vectors is invisible to this method. Only reading the specification finds it.
+- **Not that an implementation is correct.** Passing shows it agrees on these inputs.
+  Behaviour on inputs no vector describes is unconstrained.
+- **Not that the vectors are adversarial enough.** Two independent vectors per rule is
+  a floor, not a proof of adversarial coverage: independence is demonstrated against
+  the *declared* defects, and a shortcut nobody thought to declare is a shortcut the
+  pair may still share. The declaration requirement makes the blind spot enumerable,
+  not empty.
+
+The method answers one question — *could an implementation skip this check and still
+pass?* — and answers it mechanically. That is narrower than "is the suite good", and it
+is the part that was previously left to assumption.
+
+## Reproducing
+
+```bash
+pip install -e ".[dev]"
+pytest tests/test_vector_completeness.py -v          # the completeness checks
+pytest tests/test_fixture_signatures_independent.py  # the independent signature path
+```
+
+Signing keys are derived from published seeds, so every vector set regenerates
+byte-for-byte and only public JWKs appear in the files. They are deliberately
+deterministic test keys with no standing.

tests/adequacy.py

@@ -0,0 +1,113 @@
+"""Criteria a conformance vector set has to meet to be worth running.
+
+A vector set is a claim about what an implementation must do. The claim is only
+as strong as the set's ability to fail an implementation that does not do it, and
+a set can pass every one of its own assertions while failing that. These are the
+ways it happens that we have actually seen, each with the vector set it was found
+on:
+
+1. **No control, in either direction.** If every vector expects rejection, an
+   implementation that rejects unconditionally satisfies the whole set. If every
+   vector expects acceptance, one that accepts unconditionally does. Both halves
+   are needed and the second is the one that gets forgotten: a set written to show
+   that certain inputs *do* verify reads as complete while pinning nothing, which
+   is the shape `canonicalization-boundary` has here.
+
+2. **No margin.** One vector per boundary has no margin: a single implementation
+   shortcut that happens to reject that one vector passes the boundary. Two vectors
+   introducing distinct failure codes force the boundary itself to be implemented.
+   Established in agentrust-io/trace-spec#124.
+
+   Margin is counted per *boundary*, never per failure code, and the two are not
+   the same unit. A boundary covered the way #124 asks for produces two vectors
+   carrying two different codes, so counting by code reports the correct design as
+   a shortfall. This was written the wrong way round first and the set that caught
+   it was the one that had done it right. A set therefore has to say what its
+   boundaries are; where it does not, the code is used and that assumption is the
+   set's to justify.
+
+3. **A rule that nothing pins.** If deleting a rule from the verifier changes no
+   expected verdict, the set does not cover it, whatever its name suggests. Only
+   deletion establishes this; reading does not.
+
+4. **A weakness shared across a boundary's vectors.** Distinct failure codes are
+   not sufficient. Both dependency vectors of the build-provenance set placed
+   their defect last in a list, so a verifier that read one entry of three passed
+   both while still rejecting the absent-list vector, presenting as having
+   implemented the rule. Every check was present; the shortcut was in how many
+   entries each one ran over, so no code went missing and no code-level guard
+   could see it. Found in #169, on a set that already satisfied 1 through 3.
+
+Criteria 1 and 2 are decidable from the fixtures alone and are implemented here.
+Criteria 3 and 4 need to run the set's own verifier under mutation, so each set
+implements them in its own module; this file states them so that a set which omits
+them omits something named rather than something nobody thought of.
+"""
+from __future__ import annotations
+from collections import defaultdict
+from collections.abc import Iterable
+
+ACCEPTING = frozenset({"accept", "verified", "pass"})
+
+
+class Vector:
+    """One fixture, reduced to what adequacy is decided on."""
+
+    __slots__ = ("name", "outcome", "codes", "boundary")
+
+    def __init__(self, name: str, outcome: str, codes: Iterable[str] = ()):
+        self.name, self.outcome = name, outcome
+        self.codes = tuple(c for c in codes if c)
+        self.boundary: str | None = None
+
+    @property
+    def accepts(self) -> bool:
+        return self.outcome in ACCEPTING
+
+
+def trivially_satisfied_by(vectors: list[Vector]) -> str | None:
+    """The unconditional implementation this set cannot fail, if there is one.
+
+    Returns ``"reject"`` when no vector expects acceptance, ``"accept"`` when none
+    expects rejection, and ``None`` when the set pins both directions.
+    """
+    if not any(v.accepts for v in vectors):
+        return "reject"
+    if all(v.accepts for v in vectors):
+        return "accept"
+    return None
+
+
+def boundaries_without_margin(
+    vectors: list[Vector], boundary_of=None
+) -> dict[str, list[str]]:
+    """Boundaries covered by exactly one vector, each with that vector named.
+
+    *boundary_of* maps a vector to the boundary it separates. It defaults to the
+    vector's failure codes, which is right only where one code means one rule; a set
+    whose boundaries are coarser than its codes must supply it, or every correctly
+    covered boundary reads as a shortfall.
+
+    The return is the shortfall itself rather than a boolean, because a set short in
+    a named place is in a different position from one nobody has measured.
+    """
+    key = boundary_of or (lambda v: v.codes)
+    by_boundary: dict[str, list[str]] = defaultdict(list)
+    for v in vectors:
+        for b in key(v):
+            by_boundary[b].append(v.name)
+    return {b: names for b, names in sorted(by_boundary.items()) if len(names) < 2}
+
+
+def report(label: str, vectors: list[Vector], boundary_of=None) -> str:
+    thin = boundaries_without_margin(vectors, boundary_of)
+    lines = [f"{label}: {len(vectors)} vectors, "
+             f"{sum(v.accepts for v in vectors)} accepting, "
+             f"{len({c for v in vectors for c in v.codes})} distinct failure codes"]
+    trivial = trivially_satisfied_by(vectors)
+    if trivial:
+        lines.append(f"  NO CONTROL: a verifier that answers {trivial!r} to everything "
+                     f"passes this set")
+    for boundary, names in thin.items():
+        lines.append(f"  NO MARGIN: {boundary} is covered only by {names[0]}")
+    return "\n".join(lines)