From 62fea76d08363a2bc5b52c66ffd3a83a9e395167 Mon Sep 17 00:00:00 2001 From: LouieLuNZ <48041247+lywinged@users.noreply.github.com> Date: Thu, 10 Sep 2026 18:47:21 +0000 Subject: [PATCH 1/3] test(vectors): pin which vector carries which discrimination `test_margins_have_not_thinned` stores a count and `_margin` returns a set, so both are blind to identity. A vector rewritten into a copy of its partner leaves the rule's margin at two, and `test_vectors_for_each_rule_are_independent` still passes whenever some other declared defect happens to separate the pair. What is lost in that case is the specific property the vector was written to exercise, and nothing recorded which vector carried it. `vector_roles.json` records, per fixture, the rules it is load-bearing for and the declared defects it separates. `test_no_vector_has_lost_its_role` fails by name when a fixture drops one, on the same ratchet terms as the margins file: gaining a role is an ordinary PR, losing one is updated in the same commit with a reason. This lands ahead of the #178 reissue rather than with it, because a guard that arrives after the thing it guards protects nothing. Of the nine fixtures that reissue moves, exactly one carries a discrimination: `04` is the corpus's only structurally sound signature that does not verify, so it separates `checks_structure_only` while `24`, whose signature is the wrong length, does not. That shape exists only when the signer is a key the verifier does not hold, so a single-key reissue that corrupts `04` to keep it failing turns it into a second `24`. The reissue therefore takes two published deterministic keys, and the README and the `signature_or_key_mismatch` defect comment now say so. Measured on both mutations rather than argued: - Collapsing `04` to a wrong-length signature: the new guard names it, and `test_vectors_for_each_rule_are_independent[signature_or_key_mismatch]` also fails, though its message says only that the pair is redundant. - Swapping which of `06` and `25` sits inside the sixty-second grace window, touching only unsigned context so both stay stale: the independence test and the margin ratchet both pass, and the new guard is the only thing that fails, naming `25` and the defect it stopped separating. - Renaming `04`, which is the likeliest thing a reissue actually does to a file: the independence test and the margin ratchet both pass, because the renamed vector still carries the same coverage under its new name, and again the new guard is the only thing that fails. Suite is 1403 passed and 1 skipped, with ruff, mypy and check_dashes.py clean. Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com> Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Gt2dsnVufgd4VyPFThx3JV --- examples/action-receipts/README.md | 8 ++ tests/test_vector_completeness.py | 112 +++++++++++++++++++++++++++- tests/vector_roles.json | 114 +++++++++++++++++++++++++++++ 3 files changed, 233 insertions(+), 1 deletion(-) create mode 100644 tests/vector_roles.json diff --git a/examples/action-receipts/README.md b/examples/action-receipts/README.md index 5f77e6f2..327e9ce6 100644 --- a/examples/action-receipts/README.md +++ b/examples/action-receipts/README.md @@ -134,6 +134,14 @@ Everything from `10` up pins its own deterministic test key, since the private h the key used by `01`–`09` is not published; `gen_rule_coverage_vectors.py` regenerates the range byte-for-byte and only public JWKs appear in the files. +Reissuing `01`–`09` from published keys takes two of them, not one (#178). `04` is the +only vector whose signature is structurally sound and still does not verify, which is +the shape that separates it from `24`, and that shape exists only when the signer is a +key the verifier does not hold: under a single key the obvious way to keep `04` failing +is to corrupt its signature, which makes it a second `24`. `tests/vector_roles.json` +records which vector carries which discrimination, and +`test_no_vector_has_lost_its_role` fails by name if a reissue drops one. + `tests/test_action_receipt_fixtures.py` recomputes each digest, verifies each signature against the pinned key, checks session and call binding, enforces freshness and receipt-chain ordering, and compares the result with each diff --git a/tests/test_vector_completeness.py b/tests/test_vector_completeness.py index 77a5b8e1..62590f95 100644 --- a/tests/test_vector_completeness.py +++ b/tests/test_vector_completeness.py @@ -31,6 +31,7 @@ implementation defect that one catches and the other misses? Two copies of the same vector satisfy question 3 and fail this one. 5. Has any rule's margin dropped below what it was? (silent thinning) +6. Is each vector still the one doing the discriminating? (silent substitution) Independence is #124's definition made executable. For every rule, ``DEFECTS`` declares at least one *weakened* variant of its check, each modelling a real @@ -41,6 +42,12 @@ declaration is fail-closed: a registered rule with no defect entry fails the suite, so the question "what bug would your second vector catch that your first would not?" has to be answered when the rule is added, not after a regression demonstrates it. + +Question 5 counts and question 6 names. A margin is a set of fixture names reduced to +its size, so a change that keeps the size and moves the work is invisible to it: +rewriting a vector into a copy of its partner still leaves the count at two. +``vector_roles.json`` records which fixture carries which discrimination, so that +substitution fails by name instead of passing. """ from __future__ import annotations @@ -71,6 +78,7 @@ VERIFIER_MODULE = TESTS_DIR / "test_action_receipt_fixtures.py" FIXTURE_DIR = TESTS_DIR.parent / "examples" / "action-receipts" / "conformance" MARGINS_FILE = TESTS_DIR / "vector_margins.json" +ROLES_FILE = TESTS_DIR / "vector_roles.json" FIXTURES = discover_fixtures(FIXTURE_DIR) RULE_CODES = tuple(rule.code for rule in RULES) @@ -157,7 +165,10 @@ def _recomputed_action_ref(fixture: dict[str, Any]) -> str: ].lower() != "none", }, - # A signature check that stops at well-formedness. + # A signature check that stops at well-formedness. `04` is the only vector in the + # corpus whose signature is structurally sound and still does not verify, a shape + # that exists only when the signer is a key the verifier does not hold. Reissuing + # that fixture therefore takes a second published key, per #178. "signature_or_key_mismatch": { "checks_structure_only": lambda f: _trusted_jwk(f, f["receipt"]) is not None and _sig_malformed(f["receipt"]), @@ -468,3 +479,102 @@ def test_margins_have_not_thinned() -> None: f"rules that had recorded margins no longer exist: {vanished}. If they were " f"removed on purpose, drop them from {MARGINS_FILE.name} in the same commit." ) + + +# --------------------------------------------------------------------------- +# 6. The role ratchet +# --------------------------------------------------------------------------- + + +def _roles() -> dict[str, dict[str, list[str]]]: + """Per fixture: the rules it is load-bearing for, and the defects it separates. + + The same measurement questions 3 and 4 make, keyed by fixture rather than by rule. + `_margin` returns a set and the ratchet stores only its size, so neither can see + which fixture is doing the work. This asks whether a given vector still does the + job it was written for. + """ + roles: dict[str, dict[str, list[str]]] = {} + for code in RULE_CODES: + bearing = _margin(code) + for name in bearing: + roles.setdefault(name, {}).setdefault(code, []) + for defect, weakened_check in DEFECTS[code].items(): + for name in sorted(_deviating(_weakened(code, weakened_check)) & bearing): + roles[name][code].append(defect) + return roles + + +def test_no_vector_has_lost_its_role() -> None: + """A ratchet on identity: a vector may not quietly stop discriminating. + + Margins count, and a count cannot see work moving between vectors. Rewriting a + vector into a copy of its partner leaves the rule's margin at two, and + `test_vectors_for_each_rule_are_independent` still passes whenever some other + declared defect happens to separate the pair. What is lost is the specific + property the vector was written to exercise, and nothing recorded which vector + carried it. + + The case this is built for is #178, reissuing fixtures 01-09 from a key whose + private half is published. Exactly one of the nine carries a discrimination: `04` + is the corpus's only structurally sound signature that does not verify, so it + catches `checks_structure_only` while `24`, whose signature is the wrong length, + does not. That shape exists only when the signer is a key the verifier does not + hold, so reissuing `04` under the issuer key and corrupting its signature to keep + it failing turns it into a second `24`. The reissue therefore takes two published + deterministic keys, the issuer key and one that plays the wrong signer, and this + test is what says so at the moment the mistake is made rather than after. + + Six of the remaining eight record as load-bearing with no defect of their own, + which is what they are: their partners in 17-30 carry the discrimination. `01` and + `02` are the other two, and they are absent from the file altogether, because they + are the valid vectors and deleting a rule does not change the outcome of a fixture + that passes. + + Gaining a role is an ordinary PR. Losing one is a decision someone makes on + purpose, in the same commit, with a reason. + """ + current = _roles() + + if not ROLES_FILE.exists(): + ROLES_FILE.write_text( + json.dumps(current, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + pytest.skip(f"recorded initial roles to {ROLES_FILE.name}; re-run to enforce") + + recorded: dict[str, dict[str, list[str]]] = json.loads( + ROLES_FILE.read_text(encoding="utf-8") + ) + + # Each way of losing a role gets its own reason. They are three different + # mistakes, and a single closing sentence fits only one of them. + lost: list[str] = [] + for name in sorted(recorded): + if name not in current: + lost.append( + f"{name}: gone, and it was load-bearing for {sorted(recorded[name])}. " + "Retiring a vector retires whatever it was the only one to catch." + ) + continue + for code in sorted(recorded[name]): + if code not in current[name]: + lost.append( + f"{name}: no longer load-bearing for {code!r}. Deleting that rule " + "stopped changing this fixture's outcome, so it no longer " + "exercises it at all." + ) + continue + missing = sorted(set(recorded[name][code]) - set(current[name][code])) + if missing: + lost.append( + f"{name}: no longer separates {missing} for {code!r}. It still " + "fails for the rule, so what it lost is the margin rather than " + "the coverage: it has become a copy of its partner." + ) + + assert not lost, ( + "vectors lost the discrimination they were written for:\n " + + "\n ".join(lost) + + f"\nIf the change is intended, update {ROLES_FILE.name} in the same commit " + + "and say why." + ) diff --git a/tests/vector_roles.json b/tests/vector_roles.json new file mode 100644 index 00000000..d3227540 --- /dev/null +++ b/tests/vector_roles.json @@ -0,0 +1,114 @@ +{ + "03-missing-required-receipt.json": { + "receipt_missing": [] + }, + "04-signature-key-mismatch.json": { + "signature_or_key_mismatch": [ + "checks_structure_only" + ] + }, + "05-action-ref-mismatch.json": { + "action_ref_mismatch": [] + }, + "06-stale-receipt.json": { + "receipt_stale": [] + }, + "07-receipt-chain-gap.json": { + "receipt_chain_gap": [] + }, + "08-same-party-self-report.json": { + "issuer_not_independent": [] + }, + "09-unsupported-physical-completion.json": { + "unsupported_physical_completion_claim": [] + }, + "10-action-ref-not-recomputable.json": { + "action_ref_invalid": [] + }, + "11-call-id-mismatch.json": { + "call_id_mismatch": [] + }, + "12-session-id-mismatch.json": { + "session_id_mismatch": [] + }, + "13-evidence-hash-mismatch.json": { + "evidence_hash_mismatch": [] + }, + "14-receipt-issuer-key-unknown.json": { + "issuer_key_unknown": [] + }, + "15-receipt-from-future.json": { + "receipt_from_future": [] + }, + "16-decision-not-in-enum.json": { + "decision_invalid": [] + }, + "17-missing-receipt-explicit-null.json": { + "receipt_missing": [ + "treats_explicit_null_as_present" + ] + }, + "18-action-ref-tail-forged.json": { + "action_ref_invalid": [ + "compares_truncated_digest" + ] + }, + "19-action-ref-mismatch-in-tail.json": { + "action_ref_mismatch": [ + "compares_truncated_digest" + ] + }, + "20-call-id-case-mismatch.json": { + "call_id_mismatch": [ + "compares_case_insensitively" + ] + }, + "21-session-id-case-mismatch.json": { + "session_id_mismatch": [ + "compares_case_insensitively" + ] + }, + "22-evidence-hash-mismatch-in-tail.json": { + "evidence_hash_mismatch": [ + "compares_truncated_digest" + ] + }, + "23-receipt-issuer-key-case-variant.json": { + "issuer_key_unknown": [ + "looks_up_keys_case_insensitively" + ] + }, + "24-receipt-signature-malformed.json": { + "signature_or_key_mismatch": [] + }, + "25-stale-receipt-boundary.json": { + "receipt_stale": [ + "grants_sixty_seconds_grace" + ] + }, + "26-receipt-from-future-boundary.json": { + "receipt_from_future": [ + "tolerates_sixty_seconds_skew" + ] + }, + "27-receipt-chain-gap-in-tail.json": { + "receipt_chain_gap": [ + "compares_truncated_digest" + ] + }, + "28-physical-completion-claim-case.json": { + "unsupported_physical_completion_claim": [ + "compares_case_insensitively" + ] + }, + "29-same-party-self-report-rejected.json": { + "issuer_not_independent": [ + "warns_only_on_accepted" + ] + }, + "30-decision-case-variant.json": { + "decision_invalid": [ + "matches_enum_case_insensitively" + ] + } +} From 154e7e6ada5b727bc25dffbaf394de2840841863 Mon Sep 17 00:00:00 2001 From: LouieLuNZ <48041247+lywinged@users.noreply.github.com> Date: Thu, 10 Sep 2026 21:49:04 +0000 Subject: [PATCH 2/3] test(vectors): a ratchet with no baseline file fails instead of writing one Both ratchets in this module, the margins count and the new roles record, wrote their baseline file when it was missing and reported skipped. So a change that degraded a fixture and deleted the file in the same commit recorded the degraded state as the baseline: one run skipped, the next passed, and nothing was red. Measured on 62fea76 with 04 collapsed to a 32-byte signature and vector_roles.json removed: first run skipped and rewrote the file with 04's discrimination gone, second run passed. A missing file now fails by name. The message says to restore it from history, and that rebaselining on purpose means writing the file in the same commit that changes the fixtures, with the reason. With either file deleted the test fails; with both present the module passes 37 and the suite 1403 with 1 skipped on 3.11 and 3.12. Found by rajnisht7 on the pull request. Co-Authored-By: Claude Fable 5.1 Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com> --- tests/test_vector_completeness.py | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/tests/test_vector_completeness.py b/tests/test_vector_completeness.py index 62590f95..5c44634d 100644 --- a/tests/test_vector_completeness.py +++ b/tests/test_vector_completeness.py @@ -457,9 +457,12 @@ def test_margins_have_not_thinned() -> None: """ current = {code: len(_margin(code)) for code in RULE_CODES} - if not MARGINS_FILE.exists(): - MARGINS_FILE.write_text(json.dumps(current, indent=2, sort_keys=True) + "\n") - pytest.skip(f"recorded initial margins to {MARGINS_FILE.name}; re-run to enforce") + assert MARGINS_FILE.exists(), ( + f"{MARGINS_FILE.name} is missing, so there is nothing to ratchet against. A guard " + "that rebuilds its own baseline from the current tree records whatever the tree " + "has just lost. Restore the file from history; to rebaseline on purpose, write " + "the current margins to it in the same commit that changes the fixtures, and say why." + ) recorded: dict[str, int] = json.loads(MARGINS_FILE.read_text(encoding="utf-8")) thinned = { @@ -536,11 +539,12 @@ def test_no_vector_has_lost_its_role() -> None: """ current = _roles() - if not ROLES_FILE.exists(): - ROLES_FILE.write_text( - json.dumps(current, indent=2, sort_keys=True) + "\n", encoding="utf-8" - ) - pytest.skip(f"recorded initial roles to {ROLES_FILE.name}; re-run to enforce") + assert ROLES_FILE.exists(), ( + f"{ROLES_FILE.name} is missing, so there is nothing to ratchet against. A guard " + "that rebuilds its own baseline from the current tree records whatever the tree " + "has just lost. Restore the file from history; to rebaseline on purpose, write " + "_roles() to it in the same commit that changes the fixtures, and say why." + ) recorded: dict[str, dict[str, list[str]]] = json.loads( ROLES_FILE.read_text(encoding="utf-8") From f9841a487498ffd7fa080fbf13eb911ab09bda5e Mon Sep 17 00:00:00 2001 From: LouieLuNZ <48041247+lywinged@users.noreply.github.com> Date: Thu, 10 Sep 2026 21:58:52 +0000 Subject: [PATCH 3/3] test(vectors): say what the role guard holds, the shape and not the signer The README paragraph, the defect comment and the docstring all said that a structurally sound signature that does not verify exists only when the signer is a key the verifier does not hold, and so a single-key reissue of 04 would turn it into a second 24. Measured, that is false: flipping one bit of 04's signature at its full 64 bytes keeps the issuer's kid trusted and the signature invalid, every guard in this module passes, the fixture suite passes 35, and 04 still records checks_structure_only. The guard sees the shape of a signature, not who made it. All three now say that. 04 is the only vector whose issuer key the verifier holds and whose 64-byte signature still fails to verify; 14 and 23 carry 64-byte signatures the verifier has no key for, 24 is 32 bytes. The second published key that #178 asks for stays right, on the fixture's meaning rather than on a shape the guard can enforce, and the text says it is a decision the reissue makes on purpose. No code path changes; the suite is 1403 passed and 1 skipped on 3.11 and 3.12. Co-Authored-By: Claude Fable 5.1 Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com> --- examples/action-receipts/README.md | 14 ++++++++------ tests/test_vector_completeness.py | 26 +++++++++++++++----------- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/examples/action-receipts/README.md b/examples/action-receipts/README.md index 327e9ce6..595804a4 100644 --- a/examples/action-receipts/README.md +++ b/examples/action-receipts/README.md @@ -135,12 +135,14 @@ the key used by `01`–`09` is not published; `gen_rule_coverage_vectors.py` reg the range byte-for-byte and only public JWKs appear in the files. Reissuing `01`–`09` from published keys takes two of them, not one (#178). `04` is the -only vector whose signature is structurally sound and still does not verify, which is -the shape that separates it from `24`, and that shape exists only when the signer is a -key the verifier does not hold: under a single key the obvious way to keep `04` failing -is to corrupt its signature, which makes it a second `24`. `tests/vector_roles.json` -records which vector carries which discrimination, and -`test_no_vector_has_lost_its_role` fails by name if a reissue drops one. +only vector whose issuer key the verifier holds and whose 64-byte signature still fails +to verify, which is the shape that separates it from `24`. A single key can keep that +shape, by corrupting the signature at fixed length; what it cannot keep is the fixture's +meaning, a signature made by a key the verifier does not hold. The guard below sees the +shape and not the signer, so the second key is a decision the reissue makes on purpose +rather than one a test enforces. `tests/vector_roles.json` records which vector carries +which discrimination, and `test_no_vector_has_lost_its_role` fails by name if a reissue +drops one. `tests/test_action_receipt_fixtures.py` recomputes each digest, verifies each signature against the pinned key, checks session and call binding, enforces diff --git a/tests/test_vector_completeness.py b/tests/test_vector_completeness.py index 5c44634d..fe6bb18d 100644 --- a/tests/test_vector_completeness.py +++ b/tests/test_vector_completeness.py @@ -165,10 +165,13 @@ def _recomputed_action_ref(fixture: dict[str, Any]) -> str: ].lower() != "none", }, - # A signature check that stops at well-formedness. `04` is the only vector in the - # corpus whose signature is structurally sound and still does not verify, a shape - # that exists only when the signer is a key the verifier does not hold. Reissuing - # that fixture therefore takes a second published key, per #178. + # A signature check that stops at well-formedness. `04` is the only vector whose + # issuer key the verifier holds and whose 64-byte signature still fails to verify; + # `14` and `23` fail for want of a key, `24` for length. A single key can produce + # that shape by corrupting a signature at fixed length; what it cannot produce is a + # signature by a key the verifier does not hold, which is what `04` models. This + # check sees the shape, not the signer, so the second key #178 asks for is a + # decision the reissue makes on purpose. "signature_or_key_mismatch": { "checks_structure_only": lambda f: _trusted_jwk(f, f["receipt"]) is not None and _sig_malformed(f["receipt"]), @@ -520,13 +523,14 @@ def test_no_vector_has_lost_its_role() -> None: The case this is built for is #178, reissuing fixtures 01-09 from a key whose private half is published. Exactly one of the nine carries a discrimination: `04` - is the corpus's only structurally sound signature that does not verify, so it - catches `checks_structure_only` while `24`, whose signature is the wrong length, - does not. That shape exists only when the signer is a key the verifier does not - hold, so reissuing `04` under the issuer key and corrupting its signature to keep - it failing turns it into a second `24`. The reissue therefore takes two published - deterministic keys, the issuer key and one that plays the wrong signer, and this - test is what says so at the moment the mistake is made rather than after. + is the only vector whose issuer key the verifier holds and whose 64-byte signature + still fails to verify, so it catches `checks_structure_only` while `24`, whose + signature is the wrong length, does not. What this test holds is that shape. A + single-key reissue that collapses `04` to the wrong length, or renames it, fails + here by name; one that corrupts the signature at fixed length passes, because the + shape survives even though the fixture no longer models a wrong signer. The second + published key #178 asks for is therefore a decision the reissue makes on purpose, + and this test says so about the shape, not about the signer. Six of the remaining eight record as load-bearing with no defect of their own, which is what they are: their partners in 17-30 carry the discrimination. `01` and