test: conformance vectors for seven unexercised receipt rules - #122
imran-siddique merged 2 commits into
Conversation
The receipt verifier applies rules that no fixture exercises. An implementation
can omit each of these checks entirely and still pass the published set, which is
the one thing a conformance suite is supposed to prevent.
action_ref_invalid recomputing the action reference rather than trusting
the declared value
call_id_mismatch that the receipt is bound to this call
session_id_mismatch that it is bound to this session
evidence_hash_mismatch recomputing the evidence digest
issuer_key_untrusted the signing key against a pinned set at all
receipt_from_future a receipt issued after the verification time
decision_invalid refusing to read an unknown verb as accept or reject
Two are load-bearing for the trust model rather than tidiness. Without
issuer_key_untrusted a receipt authenticates itself, since a signature verifies
against whatever key it names and only a pinned set decides whether that key was
entitled to speak. Without evidence_hash_mismatch the signature covers a digest
whose document can be swapped, because the receipt signs the digest and not the
evidence body.
One fixture per rule, each triggering exactly that rule and nothing else, so a
failure names the check that broke. They pin their own deterministic test key:
the private half of the key behind 01-09 is not published, and each fixture
already carries its own trusted_issuer_keys. gen_rule_coverage_vectors.py
regenerates the set byte-for-byte and only public JWKs appear in the files.
Found by walking the verifier's source for every failure code it can emit and
comparing that against the codes the fixtures expect, rather than by reading the
set and guessing what was missing.
Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
github-actions
Bot
added
the
needs-review:UNKNOWN
|
Flagging something against my own PR, found while reading §3.3 rather than the verifier.
The spec says, of a receipt whose issuer key the verifier does not hold:
The fixture verifier appends "status": "receipt_invalid",
"failures": ["issuer_key_untrusted"],
"warnings": []Advisory and invalid are different claims. §3.3.1 draws the line deliberately — an The divergence is not mine. §3.3.2 is informative and this fixture set declares its own test profile, so nothing here What I would suggest, though it is your callThree options, in the order I would rank them:
I have no view on which of the first two is correct — it is a question about intent, not The other six fixtures are unaffected: none of them touches a rule §3.3.1 or §3.3.2 speaks |
Spec section 3.3.1: "A receipt whose issuer key is unknown to the verifier is unverified, not invalid", surfaced as an advisory rather than silence. The receipt verifier appended issuer_key_untrusted to failures, resolving to receipt_invalid, and fixture 14 pinned that divergence as the expected outcome — flagged on the PR before this commit. The verifier now records an issuer_key_unknown advisory in warnings and, when nothing else failed, returns a fifth outcome, receipt_unverified: no trust conferred, no forgery proven. The structural checks still run first, and any positive failure still yields receipt_invalid. Fixture 14 is renamed to 14-receipt-issuer-key-unknown.json and regenerated against the corrected behaviour; the other six fixtures are byte-identical. The outcome table in docs/verification.md gains the new row. An unpinned key is an inability to check, not evidence of forgery. "Invalid" claimed evidence this path never had. Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
|
Pushed the correction for the flag above, as option 1 ( I said above that the behaviour change was not this PR's to make, and the call is still yours. But every reading I could construct came out the same way — an unpinned key is an inability to check, not evidence of forgery, and §3.3.1 spends its paragraph on exactly that line — and of the three options, merging a vector that pins the divergence looked strictly worst. So the branch now carries the spec-aligned behaviour as the default rather than the divergence. If you would rather take option 2 or 3, reverting is one commit: say so and I will push it. |
|
One more note, to make this easier to dispose of rather than to add to it. The six vectors other than I would rather that than have a question I raised become the reason six unrelated vectors sit unmerged. |
imran-siddique
left a comment
imran-siddique
left a comment
There was a problem hiding this comment.
Seven rules the verifier applies that no fixture exercised. An implementation could skip recomputing the action reference or the evidence digest, or never pin the issuer key, and still pass the suite. Closing that is worth doing before the spec moves and other-language verifiers start certifying against it.
Upstream merged agentrust-io#148, the last large piece this fork was carrying, which the maintainer rebased and opened himself under this fork's authorship. With agentrust-io#122, agentrust-io#125, agentrust-io#126, agentrust-io#136 and agentrust-io#137 already merged, most of the thirty-three commits here described work that now lives upstream with better provenance than this fork can give it: a PR number and a maintainer's review. `git rebase upstream/main` was tried first and abandoned. It stopped on the oldest commit in the set, a schema-and-version alignment whose content upstream has since taken, superseded and released three times over. Replaying thirty-three commits against fifteen of upstream's resolves early commits into shapes that no longer mean anything. So: main reset to upstream/main, fork-only material re-applied. The old history is tagged `archive/pre-576507b` and pushed rather than discarded. What is held here, and why it is held: - agentrust-io#117 gap disclosure: design note, two normative drafts, 18 vectors, generator - agentrust-io#116 verifier compatibility: 8 vectors, generator, normative draft - the normative crosswalk, mapping every RFC 2119 statement to whom it binds - `docs/conformance-method.md`, `coverage-report/` (historical), DECISIONS.md - the independent signature path and the package-consistency test Each is an unaccepted proposal or a method write-up, not unfinished work. Four files needed a real merge, and not in the same direction. `models.py` and `__init__.py` are upstream's plus this fork's two profile constants, because upstream had moved on with `origin` (agentrust-io#135) and the `declared` enforcement mode (agentrust-io#143) and a wholesale copy would have dropped both. `sign.py` and `test_sign.py` are this fork's `accepted_profiles` version, which supersedes the minimal cutover check upstream took from agentrust-io#125 -- a supersession CLAUDE.md predicted when agentrust-io#125 was offered. All four merged cleanly three-way against a817621, the last commit the two histories agree on. The crosswalk guard earned its place in the same run: upstream added two normative statements this fork had never seen, and `test_normative_crosswalk.py` failed until both had rows. A source-derived inventory noticing its subject moved is the property that document exists to have. 432 passed, 1 skipped. ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
Upstream merged agentrust-io#148, the last large piece this fork was carrying, which the maintainer rebased and opened himself under this fork's authorship. With agentrust-io#122, described work that now lives upstream with better provenance than this fork can give it: a PR number and a maintainer's review. `git rebase upstream/main` was tried first and abandoned. It stopped on the oldest commit in the set, a schema-and-version alignment whose content upstream has since taken, superseded and released three times over. Replaying thirty-three commits against fifteen of upstream's resolves early commits into shapes that no longer mean anything. So: main reset to upstream/main, fork-only material re-applied. The old history is tagged `archive/pre-576507b` and pushed rather than discarded. What is held here, and why it is held: - agentrust-io#117 gap disclosure: design note, two normative drafts, 18 vectors, generator - agentrust-io#116 verifier compatibility: 8 vectors, generator, normative draft - the normative crosswalk, mapping every RFC 2119 statement to whom it binds - `docs/conformance-method.md`, `coverage-report/` (historical), DECISIONS.md - the independent signature path and the package-consistency test Each is an unaccepted proposal or a method write-up, not unfinished work. Four files needed a real merge, and not in the same direction. `models.py` and `__init__.py` are upstream's plus this fork's two profile constants, because upstream had moved on with `origin` (agentrust-io#135) and the `declared` enforcement mode (agentrust-io#143) and a wholesale copy would have dropped both. `sign.py` and `test_sign.py` are this fork's `accepted_profiles` version, which supersedes the minimal cutover check upstream took from agentrust-io#125 -- a supersession CLAUDE.md predicted when agentrust-io#125 was offered. All four merged cleanly three-way against a817621, the last commit the two histories agree on. The crosswalk guard earned its place in the same run: upstream added two normative statements this fork had never seen, and `test_normative_crosswalk.py` failed until both had rows. A source-derived inventory noticing its subject moved is the property that document exists to have. 432 passed, 1 skipped. ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
2 participants
What
Seven conformance fixtures, one per receipt rule that the verifier applies and that no
fixture previously exercised.
action_ref_invalidcall_id_mismatchsession_id_mismatchevidence_hash_mismatchissuer_key_untrustedreceipt_from_futuredecision_invalidWhy these two matter beyond tidiness
Without
issuer_key_untrusted, a receipt authenticates itself. A signature verifiesagainst whatever key it names, and only a pinned set decides whether that key was ever
entitled to speak.
Without
evidence_hash_mismatch, the signature covers a digest whose document can beswapped. The receipt signs
evidence_hash, not the evidence body, so only recomputationcatches a substituted body.
The other five are ordinary binding and freshness checks, and the same argument applies
in weaker form: a suite that never exercises a rule certifies implementations that skip
it.
How they were found
By walking the verifier's source for every failure code it can emit and comparing that
against the codes the fixtures expect. The seven above were emitted by the verifier and
expected by nothing.
I am happy to contribute the check itself as a follow-up if it is wanted — it recovers
the rule inventory from source with
astrather than from a hand-maintained list, so arule added without a fixture fails rather than passing quietly. Keeping it out of this PR
so the fixtures can be judged on their own.
Shape
One fixture per rule, each triggering exactly that rule and nothing else, so a failure
names the check that broke rather than a set of them.
They pin their own deterministic test key. The private half of the key behind
01–09is not published, and each fixture already carries its own
trusted_issuer_keys, so aseparate key changes nothing about how the set is verified.
gen_rule_coverage_vectors.pyregenerates the set byte-for-byte; only public JWKs appearin the files. Worth stating plainly that the keys are deliberately deterministic test
keys rather than leaving a reviewer to notice the pattern.
Scope
Additive. No existing fixture changes, no verifier changes, no schema or spec changes.
The only edits to existing files are the filename list in
test_fixture_set_is_completeand a table in the examples README.
Under
GOVERNANCE.mdthis is a conformance-test addition: no normative text, so nosponsor and no comment window. Please tell me if you read it differently.
Checks
ruff check src testsandmypy src/agentrust_traceclean