From 845215a44464ac491cb86220e8acdf5aa05ef1dd Mon Sep 17 00:00:00 2001 From: Sneha Dalvi Date: Mon, 24 Aug 2026 22:52:08 +0530 Subject: [PATCH 1/4] docs: clarify ADR-0003 selects only two Merkle hash operations ADR-0003 listed the two approved RFC 9162 Merkle hash operations (leaf hash and interior-node hash) alongside rejected alternatives (plain concatenation, BLAKE3, flat hash) without clearly separating the two. This could be misread as four supported constructions. Also removed outdated leaf-content wording that no longer matches the current specification, and pointed readers to spec sections 4.1.1, 3.2.3, and 3.2.5.1 as the source of truth for those details. Documentation only. No code, hashing, or API changes. --- docs/adr/0003-rfc9162-merkle-domain-separation.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/adr/0003-rfc9162-merkle-domain-separation.md b/docs/adr/0003-rfc9162-merkle-domain-separation.md index 5b9bea61..851f3c4a 100644 --- a/docs/adr/0003-rfc9162-merkle-domain-separation.md +++ b/docs/adr/0003-rfc9162-merkle-domain-separation.md @@ -15,8 +15,14 @@ Use the RFC 9162 (Certificate Transparency v2) Merkle tree construction with exp - Leaf nodes: `SHA-256(0x00 || leaf_data)` - Internal nodes: `SHA-256(0x01 || left_hash || right_hash)` -Leaf data for tool entries: RFC 8785 canonical JSON of the tool descriptor (schema + description, sorted by tool name). -Leaf data for corpus documents: RFC 8785 canonical JSON of the document descriptor (hash + identifier + ingested_at). +These are the only two Merkle hash operations selected by this ADR. The entries in +"Alternatives considered" are rejected constructions, not additional supported +operations. + +Section 4.1.1 of the specification is the normative definition of the shared +construction. Sections 3.2.3 and 3.2.5.1 normatively define each artifact's leaf +data and ordering; they take precedence over this ADR for those details. RFC 8785 +applies only where those sections define JSON as an input to a hash. ## Rationale From 49ac815678425bc970c034f80465c525723f02da Mon Sep 17 00:00:00 2001 From: Sneha Dalvi Date: Mon, 24 Aug 2026 22:58:22 +0530 Subject: [PATCH 2/4] fix: make canonicalize() RFC 8785 conformant (#322) canonicalize() is the single RFC 8785 pre-image used for manifest signatures, COSE signing, delegation chains, revocation records, memory deltas, plugin bundles, and TRACE envelopes. It was not RFC 8785 conformant in four ways: 1. Sorted object keys by Unicode code point instead of UTF-16 code unit (RFC 8785 section 3.2.3) - diverges when a key contains a supplementary-plane character. 2. Escaped U+007F, U+0080-U+009F, U+2028 and U+2029, which are outside ECMAScript JSON.stringify's escape set (RFC 8785 section 3.2.2.2). 3. Float formatting used ad-hoc shortcuts instead of ECMAScript Number::toString (RFC 8785 section 3.2.2.3): wrong integer/exponential cutover, wrong exponent padding, wrong small-magnitude threshold. 4. Silently serialized integers beyond the IEEE-754 safe range instead of refusing them. Fix: UTF-16 code-unit sort key for object keys; escape set corrected to quote, reverse solidus, and U+0000-U+001F only; full ECMA-262 Number::toString algorithm via Decimal(repr(f)).normalize(); integers with abs(value) > 2**53-1 now raise ValueError. Tests: removed the test that asserted the old non-conformant escaping behaviour; added tests for UTF-16 key order, the number-formatting table, and the safe-integer bound in test_canonicalize.py. Added test_trace_canonicalization_boundary.py, a cross-repository regression guard that verifies canonicalize() against four real signed vectors vendored from agentrust-io/trace-spec (fetched via git clone, not retyped). Validation: pytest tests/test_canonicalize.py -q -> 57 passed. pytest tests/interop/test_trace_canonicalization_boundary.py -v -> passed against all 4 real trace-spec vectors. Load-bearing check: reverted _canonicalize.py only, reran the guard - 03-utf16-key-order.json failed exactly as reported in #322; restored, passes. Full suite: 1096 passed, 6 skipped, 0 failed. Compatibility: this changes canonical bytes for the affected value classes (supplementary-plane keys, values needing >2^53 or specific exponent ranges, and the four literal characters above). Anything already signed at one of those values stops verifying against itself under the corrected canonicalizer, though it was already unverifiable against any conformant implementation, which is the point. Signed-off-by: Sneha Dalvi --- python/src/agent_manifest/_canonicalize.py | 80 ++++++++++++---- .../test_trace_canonicalization_boundary.py | 60 ++++++++++++ .../01-non-ascii-values.json | 54 +++++++++++ .../02-non-bmp-values.json | 54 +++++++++++ .../03-utf16-key-order.json | 56 +++++++++++ .../04-utf16-key-order-nested.json | 58 +++++++++++ .../canonicalization-boundary/README.md | 26 +++++ python/tests/test_canonicalize.py | 96 ++++++++++++++++++- 8 files changed, 465 insertions(+), 19 deletions(-) create mode 100644 python/tests/interop/test_trace_canonicalization_boundary.py create mode 100644 python/tests/interop/vectors/canonicalization-boundary/01-non-ascii-values.json create mode 100644 python/tests/interop/vectors/canonicalization-boundary/02-non-bmp-values.json create mode 100644 python/tests/interop/vectors/canonicalization-boundary/03-utf16-key-order.json create mode 100644 python/tests/interop/vectors/canonicalization-boundary/04-utf16-key-order-nested.json create mode 100644 python/tests/interop/vectors/canonicalization-boundary/README.md diff --git a/python/src/agent_manifest/_canonicalize.py b/python/src/agent_manifest/_canonicalize.py index 44960864..de42e0a6 100644 --- a/python/src/agent_manifest/_canonicalize.py +++ b/python/src/agent_manifest/_canonicalize.py @@ -19,6 +19,7 @@ """ from __future__ import annotations +import decimal import hashlib import math import unicodedata @@ -26,6 +27,7 @@ _MAX_DEPTH = 64 # DOS-006: prevent RecursionError from deeply nested JSON +_MAX_SAFE_INTEGER = (1 << 53) - 1 # ECMAScript Number.MAX_SAFE_INTEGER def canonicalize(obj: Any, *, exclude_none: bool = True) -> bytes: @@ -83,6 +85,12 @@ def _serialize(obj: Any, *, exclude_none: bool, depth: int) -> str: # bool check must come before int — bool is a subclass of int in Python return "true" if obj else "false" if isinstance(obj, int): + if abs(obj) > _MAX_SAFE_INTEGER: + raise ValueError( + f"Integer {obj!r} exceeds the IEEE-754 safe integer range " + "(+/-2^53-1). RFC 8785 numbers must round-trip through a " + "double; encode larger values as a string instead." + ) return str(obj) if isinstance(obj, float): return _float_to_str(obj) @@ -98,11 +106,12 @@ def _serialize(obj: Any, *, exclude_none: bool, depth: int) -> str: def _serialize_dict(d: dict[str, Any], *, exclude_none: bool, depth: int) -> str: - # RFC 8785 §3.2.3: sort keys by Unicode code point order. - # Python's str comparison uses Unicode code point order by default — no - # special locale or collation needed. + # RFC 8785 §3.2.3: sort keys by UTF-16 code unit, not code point. Python's + # default str comparison is code-point order, which disagrees with this + # exactly when a key contains a supplementary-plane character (see + # _utf16_sort_key) — so sorted(d.keys()) alone is not conformant. parts: list[str] = [] - for k in sorted(d.keys()): + for k in sorted(d.keys(), key=_utf16_sort_key): v = d[k] if exclude_none and v is None: continue @@ -110,10 +119,35 @@ def _serialize_dict(d: dict[str, Any], *, exclude_none: bool, depth: int) -> str return "{" + ",".join(parts) + "}" +def _utf16_sort_key(s: str) -> tuple[int, ...]: + """Return *s* as the UTF-16 code unit sequence RFC 8785 §3.2.3 sorts by. + + A supplementary-plane character (code point > U+FFFF) is represented in + UTF-16 as a surrogate pair starting at 0xD800-0xDBFF, which sorts below + every BMP character above 0xD800 even though the character's own code + point sorts above them. Comparing code units instead of code points is + the only way to reproduce that ordering. + """ + units: list[int] = [] + for ch in s: + cp = ord(ch) + if cp > 0xFFFF: + cp -= 0x10000 + units.append(0xD800 + (cp >> 10)) + units.append(0xDC00 + (cp & 0x3FF)) + else: + units.append(cp) + return tuple(units) + + def _quote(s: str) -> str: """Serialize a Python string as a JSON string per RFC 8785 §3.2.2.2. - Applies NFC normalization (spec Section 4.3) before escaping. + Applies NFC normalization (spec Section 4.3) before escaping. RFC 8785 + defers to ECMAScript JSON.stringify, which escapes only the quote, the + reverse solidus, and U+0000-U+001F. U+007F, the C1 controls (U+0080- + U+009F) and the line/paragraph separators (U+2028, U+2029) are emitted + literally — they are not part of that escape set. """ s = unicodedata.normalize("NFC", s) buf: list[str] = ['"'] @@ -133,8 +167,7 @@ def _quote(s: str) -> str: buf.append("\\r") elif ch == "\t": buf.append("\\t") - elif cp <= 0x001F or 0x007F <= cp <= 0x009F or cp in (0x2028, 0x2029): - # Control characters and ECMAScript line terminators + elif cp <= 0x001F: buf.append(f"\\u{cp:04x}") else: buf.append(ch) @@ -143,18 +176,33 @@ def _quote(s: str) -> str: def _float_to_str(f: float) -> str: - """Serialize a float per RFC 8785 §3.2.2.3 (ECMAScript number formatting). + """Serialize a float per RFC 8785 §3.2.2.3 (ECMAScript Number::toString). + + Implements the ECMA-262 Number::toString algorithm directly rather than + reformatting Python's `repr`: `repr(f)` already gives the shortest decimal + digit string that round-trips to *f* (what the spec calls `s`), and + `Decimal(repr(f)).normalize()` recovers that digit string and its exponent + without the two shortcuts (integers bounded at 1e15, exponential notation + switching over at the wrong magnitude) the previous implementation used. Raises: ValueError: If *f* is NaN or Infinity (not permitted by RFC 8785). """ if math.isnan(f) or math.isinf(f): raise ValueError(f"RFC 8785 does not permit NaN or Infinity ({f!r})") - # Integers stored as floats: no decimal point - if f == math.floor(f) and abs(f) < 1e15: - return str(int(f)) - # Use Python's shortest-round-trip repr, then normalize exponent notation - s = repr(f) - if "e" in s and "e+" not in s and "e-" not in s: - s = s.replace("e", "e+") - return s + if f == 0.0: + return "0" + sign = "-" if f < 0 else "" + _, digits, exponent = decimal.Decimal(repr(abs(f))).normalize().as_tuple() + digit_str = "".join(str(x) for x in digits) + k = len(digit_str) + n = exponent + k + if k <= n <= 21: + return sign + digit_str + "0" * (n - k) + if 0 < n <= 21: + return sign + digit_str[:n] + "." + digit_str[n:] + if -6 < n <= 0: + return sign + "0." + "0" * (-n) + digit_str + e = n - 1 + mantissa = digit_str[0] if k == 1 else digit_str[0] + "." + digit_str[1:] + return sign + mantissa + "e" + ("+" if e >= 0 else "-") + str(abs(e)) diff --git a/python/tests/interop/test_trace_canonicalization_boundary.py b/python/tests/interop/test_trace_canonicalization_boundary.py new file mode 100644 index 00000000..da8723f8 --- /dev/null +++ b/python/tests/interop/test_trace_canonicalization_boundary.py @@ -0,0 +1,60 @@ +"""Cross-repository RFC 8785 conformance guard (issue #322). + +trace-spec's canonicalization-boundary vectors are signed Trust Records whose +signature verifies only over the RFC 8785 canonical bytes of every field +except ``signature``. They are a black-box check on +``agent_manifest._canonicalize.canonicalize`` from an independent producer: +a non-conformant canonicalizer computes different signing bytes here and the +signature stops verifying, which is exactly the failure issue #322 reported. + +The vectors are not vendored in this repository yet -- see +``tests/interop/vectors/canonicalization-boundary/README.md`` for exact fetch +commands. This test skips cleanly with those instructions until the files +are present, and is not required for the rest of the suite to pass. +""" +from __future__ import annotations + +import base64 +import json +from pathlib import Path + +import pytest +from cryptography.exceptions import InvalidSignature +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey + +from agent_manifest._canonicalize import canonicalize + +_VECTORS_DIR = Path(__file__).parent / "vectors" / "canonicalization-boundary" +_README = _VECTORS_DIR / "README.md" + + +def _b64url_decode(value: str) -> bytes: + return base64.urlsafe_b64decode(value + "=" * (-len(value) % 4)) + + +def _vector_files() -> list[Path]: + return sorted(_VECTORS_DIR.glob("*.json")) + + +def test_canonicalize_matches_trace_spec_signature(): + files = _vector_files() + if not files: + pytest.skip(f"vectors not vendored yet; see {_README}") + + for path in files: + vector = json.loads(path.read_text(encoding="utf-8")) + record = vector["record"] + jwk = vector["trusted_key"] + body = {k: v for k, v in record.items() if k != "signature"} + + pre_image = canonicalize(body) + public_key = Ed25519PublicKey.from_public_bytes(_b64url_decode(jwk["x"])) + signature = _b64url_decode(record["signature"]) + + try: + public_key.verify(signature, pre_image) + except InvalidSignature: + pytest.fail( + f"{path.name}: canonicalize() pre-image does not verify " + "against trace-spec's own signature over this record" + ) diff --git a/python/tests/interop/vectors/canonicalization-boundary/01-non-ascii-values.json b/python/tests/interop/vectors/canonicalization-boundary/01-non-ascii-values.json new file mode 100644 index 00000000..cd77d577 --- /dev/null +++ b/python/tests/interop/vectors/canonicalization-boundary/01-non-ascii-values.json @@ -0,0 +1,54 @@ +{ + "name": "non-ascii-values", + "description": "String values outside ASCII, all in the Basic Multilingual Plane. RFC 8785 emits them as literal UTF-8; a serializer that escapes to \\uXXXX signs different bytes and rejects this valid record.", + "spec": "trace-v0.2 section 3.2.2 — implementations MUST use an RFC 8785-conformant library", + "profile": "trace.canonicalization.boundary.v0", + "trusted_key": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA" + }, + "record": { + "eat_profile": "tag:agentrust-io.com,2026:trace-v0.2", + "iat": 1785000000, + "subject": "spiffe://factory.example/agent/payments/prod", + "model": { + "provider": "anthropic", + "model_id": "claude-sonnet-4-6", + "version": "modèle-géant-4.6" + }, + "runtime": { + "platform": "software-only", + "measurement": "sha256:0000000000000000000000000000000000000000000000000000000000000000" + }, + "policy": { + "bundle_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "enforcement_mode": "enforce" + }, + "data_class": "机密", + "build_provenance": { + "slsa_level": 0, + "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "appraisal": { + "status": "affirming", + "verifier": "https://verifier.example/v1" + }, + "transparency": "https://rekor.example/api/v1/log/entries/0", + "cnf": { + "jwk": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA" + } + }, + "signature": "WehNEF0FqgUa_c85Hw7jbbz4_d_kg2GEyo4r4p242CNGjTkmmRNvVPuwTfjtKJwbOCuNspqEyrMNgZMOTh-OAA" + }, + "expected": { + "outcome": "verified" + }, + "diverges_under": [ + "sort_keys_default", + "sort_keys_compact" + ] +} diff --git a/python/tests/interop/vectors/canonicalization-boundary/02-non-bmp-values.json b/python/tests/interop/vectors/canonicalization-boundary/02-non-bmp-values.json new file mode 100644 index 00000000..46e9222b --- /dev/null +++ b/python/tests/interop/vectors/canonicalization-boundary/02-non-bmp-values.json @@ -0,0 +1,54 @@ +{ + "name": "non-bmp-values", + "description": "String values above U+FFFF, encoded as four UTF-8 bytes each. Under ASCII-escaping they become surrogate pairs; either way the bytes differ from RFC 8785's literal UTF-8.", + "spec": "trace-v0.2 section 3.2.2 — implementations MUST use an RFC 8785-conformant library", + "profile": "trace.canonicalization.boundary.v0", + "trusted_key": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA" + }, + "record": { + "eat_profile": "tag:agentrust-io.com,2026:trace-v0.2", + "iat": 1785000000, + "subject": "spiffe://factory.example/agent/payments/prod", + "model": { + "provider": "anthropic", + "model_id": "claude-sonnet-4-6", + "version": "4.6-🤖" + }, + "runtime": { + "platform": "software-only", + "measurement": "sha256:0000000000000000000000000000000000000000000000000000000000000000" + }, + "policy": { + "bundle_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "enforcement_mode": "enforce" + }, + "data_class": "confidential-🔒", + "build_provenance": { + "slsa_level": 0, + "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "appraisal": { + "status": "affirming", + "verifier": "https://verifier.example/v1" + }, + "transparency": "https://rekor.example/api/v1/log/entries/0", + "cnf": { + "jwk": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA" + } + }, + "signature": "62CaOUWmDFPmgthTUkJ4cdwxmDQXzYg9hN6KaCB3EHjeDzeLiB_rdVFIRQrDVTzt-clmIoxNs7UxzJMFvWB_Bw" + }, + "expected": { + "outcome": "verified" + }, + "diverges_under": [ + "sort_keys_default", + "sort_keys_compact" + ] +} diff --git a/python/tests/interop/vectors/canonicalization-boundary/03-utf16-key-order.json b/python/tests/interop/vectors/canonicalization-boundary/03-utf16-key-order.json new file mode 100644 index 00000000..26309488 --- /dev/null +++ b/python/tests/interop/vectors/canonicalization-boundary/03-utf16-key-order.json @@ -0,0 +1,56 @@ +{ + "name": "utf16-key-order", + "description": "Two object keys whose order under RFC 8785's UTF-16 code-unit sort is the reverse of their code-point order. This is the record that distinguishes a true RFC 8785 serializer from json.dumps with every option set carefully: compact separators and ensure_ascii=False survive vectors 01 and 02, and fail here.", + "spec": "trace-v0.2 section 3.2.2 — implementations MUST use an RFC 8785-conformant library", + "profile": "trace.canonicalization.boundary.v0", + "trusted_key": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA" + }, + "record": { + "eat_profile": "tag:agentrust-io.com,2026:trace-v0.2", + "iat": 1785000000, + "subject": "spiffe://factory.example/agent/payments/prod", + "model": { + "provider": "anthropic", + "model_id": "claude-sonnet-4-6" + }, + "runtime": { + "platform": "software-only", + "measurement": "sha256:0000000000000000000000000000000000000000000000000000000000000000" + }, + "policy": { + "bundle_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "enforcement_mode": "enforce" + }, + "data_class": "confidential", + "build_provenance": { + "slsa_level": 0, + "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "appraisal": { + "status": "affirming", + "verifier": "https://verifier.example/v1" + }, + "transparency": "https://rekor.example/api/v1/log/entries/0", + "cnf": { + "jwk": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA", + "zk😀": "sorts-first-under-rfc-8785", + "zk�": "sorts-second-under-rfc-8785" + } + }, + "signature": "CjOuPwCnxnwegFjguiSCi-_xPg3iOwnCgyKuKYnV0OorofjPJrkOLn3dUFa-6tVf0z8EDiHaczl6AN46MuBtCQ" + }, + "expected": { + "outcome": "verified" + }, + "diverges_under": [ + "sort_keys_default", + "sort_keys_compact", + "sort_keys_compact_utf8" + ] +} diff --git a/python/tests/interop/vectors/canonicalization-boundary/04-utf16-key-order-nested.json b/python/tests/interop/vectors/canonicalization-boundary/04-utf16-key-order-nested.json new file mode 100644 index 00000000..b7b9714a --- /dev/null +++ b/python/tests/interop/vectors/canonicalization-boundary/04-utf16-key-order-nested.json @@ -0,0 +1,58 @@ +{ + "name": "utf16-key-order-nested", + "description": "The divergence of vector 03 moved inside a nested object, so that a canonicalizer sorting by UTF-16 code units at the outer levels and by code points below them passes 03 and fails here. Without it the closest non-conformant form is caught by one vector, and the boundary disappears with that vector.", + "spec": "trace-v0.2 section 3.2.2 — implementations MUST use an RFC 8785-conformant library", + "profile": "trace.canonicalization.boundary.v0", + "trusted_key": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA" + }, + "record": { + "eat_profile": "tag:agentrust-io.com,2026:trace-v0.2", + "iat": 1785000000, + "subject": "spiffe://factory.example/agent/payments/prod", + "model": { + "provider": "anthropic", + "model_id": "claude-sonnet-4-6" + }, + "runtime": { + "platform": "software-only", + "measurement": "sha256:0000000000000000000000000000000000000000000000000000000000000000" + }, + "policy": { + "bundle_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "enforcement_mode": "enforce" + }, + "data_class": "confidential", + "build_provenance": { + "slsa_level": 0, + "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "appraisal": { + "status": "affirming", + "verifier": "https://verifier.example/v1" + }, + "transparency": "https://rekor.example/api/v1/log/entries/0", + "cnf": { + "jwk": { + "kty": "OKP", + "crv": "Ed25519", + "x": "Be97jkxfFpVXzj9B-gwpMzv5t8PH30Edd-J7AIlrdoA", + "zmeta": { + "zk😀": "sorts-first-under-rfc-8785", + "zk�": "sorts-second-under-rfc-8785" + } + } + }, + "signature": "yXsht9nU--Hvr8K7xHq72MOU6xyVhsCKw0_YcAdDff641JNlPG1d2qAZ_zwXaLe48agijvRk3MVZioG85aAiBg" + }, + "expected": { + "outcome": "verified" + }, + "diverges_under": [ + "sort_keys_default", + "sort_keys_compact", + "sort_keys_compact_utf8" + ] +} diff --git a/python/tests/interop/vectors/canonicalization-boundary/README.md b/python/tests/interop/vectors/canonicalization-boundary/README.md new file mode 100644 index 00000000..df70bd27 --- /dev/null +++ b/python/tests/interop/vectors/canonicalization-boundary/README.md @@ -0,0 +1,26 @@ +# Vendored trace-spec canonicalization-boundary vectors + +Empty until fetched. `test_trace_canonicalization_boundary.py` skips with +fetch instructions when no `*.json` files are present here. + +Source: https://github.com/agentrust-io/trace-spec/tree/main/examples/canonicalization-boundary + +These are signed fixtures — fetch the raw bytes directly rather than +retyping them, since the guard exists to catch canonicalization differences +that a retyped copy could silently hide. + +```powershell +git clone --depth 1 https://github.com/agentrust-io/trace-spec C:\Temp\trace-spec +Copy-Item C:\Temp\trace-spec\examples\canonicalization-boundary\*.json . +``` + +or per file: + +```powershell +curl.exe -o 01-non-ascii-values.json https://raw.githubusercontent.com/agentrust-io/trace-spec/main/examples/canonicalization-boundary/01-non-ascii-values.json +curl.exe -o 02-non-bmp-values.json https://raw.githubusercontent.com/agentrust-io/trace-spec/main/examples/canonicalization-boundary/02-non-bmp-values.json +curl.exe -o 03-utf16-key-order.json https://raw.githubusercontent.com/agentrust-io/trace-spec/main/examples/canonicalization-boundary/03-utf16-key-order.json +curl.exe -o 04-utf16-key-order-nested.json https://raw.githubusercontent.com/agentrust-io/trace-spec/main/examples/canonicalization-boundary/04-utf16-key-order-nested.json +``` + +Run `pytest tests/interop/test_trace_canonicalization_boundary.py -v` afterward. diff --git a/python/tests/test_canonicalize.py b/python/tests/test_canonicalize.py index c2e499d7..ffdef9d2 100644 --- a/python/tests/test_canonicalize.py +++ b/python/tests/test_canonicalize.py @@ -62,6 +62,27 @@ def test_unicode_key_ordering(): assert result == ('{"e":2,"' + chr(233) + '":1}').encode("utf-8") +def test_utf16_code_unit_key_order_not_code_point_order(): + # RFC 8785 §3.2.3 sorts by UTF-16 code unit. U+10000 (a supplementary- + # plane character) is a surrogate pair D800 DC00 in UTF-16, which sorts + # below the BMP character U+FFFF (code unit FFFF) — the reverse of code + # point order, where U+10000 > U+FFFF. Issue #322. + obj = {chr(0x10000): 1, chr(0xFFFF): 2} + result = canonicalize(obj) + expected = ('{"' + chr(0x10000) + '":1,"' + chr(0xFFFF) + '":2}').encode("utf-8") + assert result == expected + + +def test_utf16_code_unit_key_order_nested(): + # Same divergence one level deep, matching trace-spec's + # 04-utf16-key-order-nested.json boundary vector. + obj = {"outer": {chr(0x10000): 1, chr(0xFFFF): 2}} + result = canonicalize(obj) + assert result == ( + '{"outer":{"' + chr(0x10000) + '":1,"' + chr(0xFFFF) + '":2}}' + ).encode("utf-8") + + # --------------------------------------------------------------------------- # Whitespace # --------------------------------------------------------------------------- @@ -132,9 +153,24 @@ def test_tab_newline_escaped(): assert canonicalize({"v": "\t\n"}) == b'{"v":"\\t\\n"}' -def test_line_separator_escaped(): - # U+2028 LINE SEPARATOR must be - assert b"\\u2028" in canonicalize({"v": chr(0x2028)}) +def test_line_separator_not_escaped(): + # RFC 8785 §3.2.2.2 defers to ECMAScript JSON.stringify, which escapes + # only the quote, the reverse solidus, and U+0000-U+001F. U+2028 LINE + # SEPARATOR is a hazard when JSON is pasted into JavaScript *source*, not + # a JSON serialization rule — it MUST be emitted literally. This test + # previously asserted the opposite (non-conformant) behaviour. Issue #322. + assert canonicalize({"v": chr(0x2028)}) == ('{"v":"' + chr(0x2028) + '"}').encode("utf-8") + + +def test_paragraph_separator_not_escaped(): + assert canonicalize({"v": chr(0x2029)}) == ('{"v":"' + chr(0x2029) + '"}').encode("utf-8") + + +def test_delete_and_c1_controls_not_escaped(): + # U+007F (DELETE) and the C1 control range (U+0080-U+009F) are outside + # ECMAScript JSON.stringify's escape set and must be emitted literally. + assert canonicalize({"v": chr(0x7F)}) == ('{"v":"' + chr(0x7F) + '"}').encode("utf-8") + assert canonicalize({"v": chr(0x85)}) == ('{"v":"' + chr(0x85) + '"}').encode("utf-8") def test_regular_unicode_verbatim(): @@ -201,6 +237,60 @@ def test_float_infinity_raises(): canonicalize({"v": math.inf}) +# --------------------------------------------------------------------------- +# ECMAScript Number::toString formatting — issue #322 differential table +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "value,expected", + [ + (1e-7, "1e-7"), + (3e-8, "3e-8"), + (-1e-7, "-1e-7"), + (1e-6, "0.000001"), + (1e-5, "0.00001"), + (1e15, "1000000000000000"), + (1e16, "10000000000000000"), + (1e20, "100000000000000000000"), + (1e21, "1e+21"), + (123.456, "123.456"), + (100.5, "100.5"), + (0.1, "0.1"), + (1.0, "1"), + (100.0, "100"), + (0.0, "0"), + (-0.0, "0"), + ], +) +def test_float_formatting_matches_ecmascript_number_tostring(value, expected): + assert canonicalize({"v": value}) == ('{"v":' + expected + "}").encode("utf-8") + + +def test_float_exponent_no_leading_zero_or_plus_padding(): + # Python's repr pads exponents to two digits and always signs them + # ("1e-07"); RFC 8785 has neither a minimum width nor a leading zero. + result = canonicalize({"v": 1e-7}) + assert result == b'{"v":1e-7}' + assert b"1e-07" not in result + + +def test_integer_within_safe_range_serializes(): + assert canonicalize({"v": (1 << 53) - 1}) == b'{"v":9007199254740991}' + + +def test_integer_beyond_safe_range_raises(): + # Integers past 2^53-1 cannot round-trip through an IEEE-754 double and + # must be refused rather than silently serialized. Issue #322. + with pytest.raises(ValueError, match="safe integer range"): + canonicalize({"v": (1 << 53) + 1}) + + +def test_negative_integer_beyond_safe_range_raises(): + with pytest.raises(ValueError, match="safe integer range"): + canonicalize({"v": -((1 << 53) + 1)}) + + # --------------------------------------------------------------------------- # @context / @type as ordinary fields # --------------------------------------------------------------------------- From 92d933c72ae0b55fa1a2fe138aeb64ef618df0af Mon Sep 17 00:00:00 2001 From: Sneha Dalvi Date: Tue, 25 Aug 2026 13:38:15 +0530 Subject: [PATCH 3/4] fix: remove non-JCS integer restriction and update ADR --- .../0003-rfc9162-merkle-domain-separation.md | 13 ++++++++++++- python/src/agent_manifest/_canonicalize.py | 6 ------ python/tests/test_canonicalize.py | 17 +++++++++-------- 3 files changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/adr/0003-rfc9162-merkle-domain-separation.md b/docs/adr/0003-rfc9162-merkle-domain-separation.md index fe9e38c0..833b0a9b 100644 --- a/docs/adr/0003-rfc9162-merkle-domain-separation.md +++ b/docs/adr/0003-rfc9162-merkle-domain-separation.md @@ -15,6 +15,7 @@ Use the RFC 9162 (Certificate Transparency v2) Merkle tree construction with exp - Leaf nodes: `SHA-256(0x00 || leaf_data)` - Internal nodes: `SHA-256(0x01 || left_hash || right_hash)` +<<<<<<< HEAD Leaf data for tool entries: RFC 8785 canonical JSON of the tool descriptor (schema + description, sorted by tool name). Leaf data for corpus documents: RFC 8785 canonical JSON of the document descriptor (hash + identifier + ingested_at). Leaf data for composite policy sub-bundles (Section 3.2.2): the **raw digest bytes** of each sub-bundle hash, not the `sha256:`-prefixed hex string and not a JSON descriptor. Unlike the two above, this leaf carries no structured descriptor, because the ordering rule already fixes which sub-bundle each leaf is. @@ -23,16 +24,26 @@ Leaf data for composite policy sub-bundles (Section 3.2.2): the **raw digest byt These are the only two Merkle hash operations selected by this ADR. The entries in "Alternatives considered" are rejected constructions, not additional supported operations. +======= +This ADR currently defines three Merkle hash operations. +The entries in "Alternatives considered" are rejected constructions, not additional +supported operations. +>>>>>>> a43da11 (fix: remove non-JCS integer restriction and update ADR) Section 4.1.1 of the specification is the normative definition of the shared -construction. Sections 3.2.3 and 3.2.5.1 normatively define each artifact's leaf +construction. Sections 3.2.2, 3.2.3, and 3.2.5.1 normatively define each artifact's leaf data and ordering; they take precedence over this ADR for those details. RFC 8785 +<<<<<<< HEAD applies only where those sections define JSON as an input to a hash. ======= Leaf data for tool entries: RFC 8785 canonical JSON of the tool descriptor (schema + description, sorted by tool name). Leaf data for corpus documents: RFC 8785 canonical JSON of the document descriptor (hash + identifier + ingested_at). Leaf data for composite policy sub-bundles (Section 3.2.2): the **raw digest bytes** of each sub-bundle hash, not the `sha256:`-prefixed hex string and not a JSON descriptor. Unlike the two above, this leaf carries no structured descriptor, because the ordering rule already fixes which sub-bundle each leaf is. >>>>>>> 3bb3ee5a46305bd639e8814308b66d1392a3eaf8 +======= +applies only where those sections define JSON as an input to a hash. The composite +policy sub-bundle leaf uses raw digest bytes and does not use JSON canonicalization. +>>>>>>> a43da11 (fix: remove non-JCS integer restriction and update ADR) ## Rationale diff --git a/python/src/agent_manifest/_canonicalize.py b/python/src/agent_manifest/_canonicalize.py index de42e0a6..92145937 100644 --- a/python/src/agent_manifest/_canonicalize.py +++ b/python/src/agent_manifest/_canonicalize.py @@ -85,12 +85,6 @@ def _serialize(obj: Any, *, exclude_none: bool, depth: int) -> str: # bool check must come before int — bool is a subclass of int in Python return "true" if obj else "false" if isinstance(obj, int): - if abs(obj) > _MAX_SAFE_INTEGER: - raise ValueError( - f"Integer {obj!r} exceeds the IEEE-754 safe integer range " - "(+/-2^53-1). RFC 8785 numbers must round-trip through a " - "double; encode larger values as a string instead." - ) return str(obj) if isinstance(obj, float): return _float_to_str(obj) diff --git a/python/tests/test_canonicalize.py b/python/tests/test_canonicalize.py index ffdef9d2..87409587 100644 --- a/python/tests/test_canonicalize.py +++ b/python/tests/test_canonicalize.py @@ -279,16 +279,17 @@ def test_integer_within_safe_range_serializes(): assert canonicalize({"v": (1 << 53) - 1}) == b'{"v":9007199254740991}' -def test_integer_beyond_safe_range_raises(): - # Integers past 2^53-1 cannot round-trip through an IEEE-754 double and - # must be refused rather than silently serialized. Issue #322. - with pytest.raises(ValueError, match="safe integer range"): - canonicalize({"v": (1 << 53) + 1}) +def test_large_integer_allowed(): + result = canonicalize( + {"v": 295147905179352830000} + ) + assert result == b'{"v":295147905179352830000}' -def test_negative_integer_beyond_safe_range_raises(): - with pytest.raises(ValueError, match="safe integer range"): - canonicalize({"v": -((1 << 53) + 1)}) +def test_large_integer_timestamp_allowed(): + assert canonicalize( + {"timestamp_ns": 1790000000000000000} + ) == b'{"timestamp_ns":1790000000000000000}' # --------------------------------------------------------------------------- From aa8867e3dfb05a1bb91b8a4a383de3b899f239a1 Mon Sep 17 00:00:00 2001 From: Sneha Dalvi Date: Tue, 1 Sep 2026 18:21:25 +0530 Subject: [PATCH 4/4] fix: rebase PR #338 per maintainer guidance - remove duplicate canonicalize code and resolve ADR conflicts Per imran-siddique's review (agentrust-io/agent-manifest#338): - Rebased onto current main (PR #352 merged with canonicalize fixes) - Removed python/src/agent_manifest/_canonicalize.py (now in #352) - Removed python/tests/test_canonicalize.py (now in #352) - Resolved 9 conflict markers in docs/adr/0003-rfc9162-merkle-domain-separation.md - Kept three Merkle operations (tool catalog, corpus, composite policy) - Sections 3.2.2, 3.2.3, 3.2.5.1 named as normative definitions - Preserved composite sub-bundle raw digest bytes (no JSON canonicalization) - Retained interop test vectors and test_trace_canonicalization_boundary.py This PR now carries only the unique value: ADR clarification and boundary test vectors. --- .../0003-rfc9162-merkle-domain-separation.md | 28 +- python/src/agent_manifest/_canonicalize.py | 202 ----------- python/tests/test_canonicalize.py | 326 ------------------ 3 files changed, 3 insertions(+), 553 deletions(-) delete mode 100644 python/src/agent_manifest/_canonicalize.py delete mode 100644 python/tests/test_canonicalize.py diff --git a/docs/adr/0003-rfc9162-merkle-domain-separation.md b/docs/adr/0003-rfc9162-merkle-domain-separation.md index 833b0a9b..d3a4ab32 100644 --- a/docs/adr/0003-rfc9162-merkle-domain-separation.md +++ b/docs/adr/0003-rfc9162-merkle-domain-separation.md @@ -15,35 +15,13 @@ Use the RFC 9162 (Certificate Transparency v2) Merkle tree construction with exp - Leaf nodes: `SHA-256(0x00 || leaf_data)` - Internal nodes: `SHA-256(0x01 || left_hash || right_hash)` -<<<<<<< HEAD -Leaf data for tool entries: RFC 8785 canonical JSON of the tool descriptor (schema + description, sorted by tool name). -Leaf data for corpus documents: RFC 8785 canonical JSON of the document descriptor (hash + identifier + ingested_at). -Leaf data for composite policy sub-bundles (Section 3.2.2): the **raw digest bytes** of each sub-bundle hash, not the `sha256:`-prefixed hex string and not a JSON descriptor. Unlike the two above, this leaf carries no structured descriptor, because the ordering rule already fixes which sub-bundle each leaf is. +This ADR defines three Merkle hash operations: tool catalog leaves, corpus document leaves, and composite policy sub-bundle leaves. -<<<<<<< HEAD -These are the only two Merkle hash operations selected by this ADR. The entries in -"Alternatives considered" are rejected constructions, not additional supported -operations. -======= -This ADR currently defines three Merkle hash operations. -The entries in "Alternatives considered" are rejected constructions, not additional -supported operations. ->>>>>>> a43da11 (fix: remove non-JCS integer restriction and update ADR) - -Section 4.1.1 of the specification is the normative definition of the shared -construction. Sections 3.2.2, 3.2.3, and 3.2.5.1 normatively define each artifact's leaf -data and ordering; they take precedence over this ADR for those details. RFC 8785 -<<<<<<< HEAD -applies only where those sections define JSON as an input to a hash. -======= Leaf data for tool entries: RFC 8785 canonical JSON of the tool descriptor (schema + description, sorted by tool name). Leaf data for corpus documents: RFC 8785 canonical JSON of the document descriptor (hash + identifier + ingested_at). Leaf data for composite policy sub-bundles (Section 3.2.2): the **raw digest bytes** of each sub-bundle hash, not the `sha256:`-prefixed hex string and not a JSON descriptor. Unlike the two above, this leaf carries no structured descriptor, because the ordering rule already fixes which sub-bundle each leaf is. ->>>>>>> 3bb3ee5a46305bd639e8814308b66d1392a3eaf8 -======= -applies only where those sections define JSON as an input to a hash. The composite -policy sub-bundle leaf uses raw digest bytes and does not use JSON canonicalization. ->>>>>>> a43da11 (fix: remove non-JCS integer restriction and update ADR) + +Section 4.1.1 of the specification is the normative definition of the shared construction. Sections 3.2.2, 3.2.3, and 3.2.5.1 normatively define each artifact's leaf data and ordering; they take precedence over this ADR for those details. RFC 8785 applies only where those sections define JSON as an input to a hash. The composite policy sub-bundle leaf uses raw digest bytes and does not use JSON canonicalization. ## Rationale diff --git a/python/src/agent_manifest/_canonicalize.py b/python/src/agent_manifest/_canonicalize.py deleted file mode 100644 index 92145937..00000000 --- a/python/src/agent_manifest/_canonicalize.py +++ /dev/null @@ -1,202 +0,0 @@ -"""RFC 8785 JSON Canonicalization Scheme (JCS). - -Reference: https://www.rfc-editor.org/rfc/rfc8785 - -Single canonicalization entry point for all signing, hashing, and Merkle -tree operations in the Agent Manifest SDK. Used for: - - - Manifest signature pre-image - - manifest_hash_in_report pre-image - - Memory snapshot hash input - - Evidence pack hash input - - Merkle tree leaf nodes containing JSON content - -Per spec Section 4.3: - - Null-valued optional fields are EXCLUDED from canonical form by default. - - @context and @type are treated as ordinary JSON fields (no JSON-LD normalization). - - Text artifact content (system_prompt, policy_bundle) is hashed as raw UTF-8 - NFC bytes, not as JSON — use hashlib directly for those, not this module. -""" -from __future__ import annotations - -import decimal -import hashlib -import math -import unicodedata -from typing import Any - - -_MAX_DEPTH = 64 # DOS-006: prevent RecursionError from deeply nested JSON -_MAX_SAFE_INTEGER = (1 << 53) - 1 # ECMAScript Number.MAX_SAFE_INTEGER - - -def canonicalize(obj: Any, *, exclude_none: bool = True) -> bytes: - """Return RFC 8785 canonical JSON bytes for *obj*. - - Args: - obj: Any JSON-serializable Python value. - exclude_none: When True (default, per spec Section 4.3), mapping - entries whose value is None are omitted from the output. - Set to False only when verifying round-trips with external - producers that include explicit null fields. - - Returns: - UTF-8 encoded bytes with no trailing newline. - - Raises: - TypeError: If *obj* contains a type that cannot be serialized. - ValueError: If a float value is NaN or Infinity, or nesting exceeds - the maximum depth. - """ - return _serialize(obj, exclude_none=exclude_none, depth=0).encode("utf-8") - - -def canonical_hash(obj: Any, *, algorithm: str = "sha256", exclude_none: bool = True) -> str: - """Canonicalize *obj* and return a prefixed hex digest. - - Returns: - String in HashValue format: ``"sha256:<64-hex>"`` or - ``"shake256:<64-hex>"``. - """ - data = canonicalize(obj, exclude_none=exclude_none) - if algorithm == "sha256": - digest = hashlib.sha256(data).hexdigest() - elif algorithm == "shake256": - digest = hashlib.shake_256(data).hexdigest(32) # 256-bit = 32 bytes - else: - raise ValueError(f"Unsupported algorithm {algorithm!r}. Use 'sha256' or 'shake256'.") - return f"{algorithm}:{digest}" - - -# --------------------------------------------------------------------------- -# Internal helpers -# --------------------------------------------------------------------------- - - -def _serialize(obj: Any, *, exclude_none: bool, depth: int) -> str: - if depth > _MAX_DEPTH: - raise ValueError( - f"JSON nesting depth exceeds maximum of {_MAX_DEPTH}. " - "The manifest contains deeply nested structures." - ) - if obj is None: - return "null" - if isinstance(obj, bool): - # bool check must come before int — bool is a subclass of int in Python - return "true" if obj else "false" - if isinstance(obj, int): - return str(obj) - if isinstance(obj, float): - return _float_to_str(obj) - if isinstance(obj, str): - return _quote(obj) - if isinstance(obj, (list, tuple)): - return "[" + ",".join(_serialize(v, exclude_none=exclude_none, depth=depth + 1) for v in obj) + "]" - if isinstance(obj, dict): - return _serialize_dict(obj, exclude_none=exclude_none, depth=depth + 1) - raise TypeError( - f"Object of type {type(obj).__name__!r} is not JSON-serializable under RFC 8785" - ) - - -def _serialize_dict(d: dict[str, Any], *, exclude_none: bool, depth: int) -> str: - # RFC 8785 §3.2.3: sort keys by UTF-16 code unit, not code point. Python's - # default str comparison is code-point order, which disagrees with this - # exactly when a key contains a supplementary-plane character (see - # _utf16_sort_key) — so sorted(d.keys()) alone is not conformant. - parts: list[str] = [] - for k in sorted(d.keys(), key=_utf16_sort_key): - v = d[k] - if exclude_none and v is None: - continue - parts.append(_quote(k) + ":" + _serialize(v, exclude_none=exclude_none, depth=depth)) - return "{" + ",".join(parts) + "}" - - -def _utf16_sort_key(s: str) -> tuple[int, ...]: - """Return *s* as the UTF-16 code unit sequence RFC 8785 §3.2.3 sorts by. - - A supplementary-plane character (code point > U+FFFF) is represented in - UTF-16 as a surrogate pair starting at 0xD800-0xDBFF, which sorts below - every BMP character above 0xD800 even though the character's own code - point sorts above them. Comparing code units instead of code points is - the only way to reproduce that ordering. - """ - units: list[int] = [] - for ch in s: - cp = ord(ch) - if cp > 0xFFFF: - cp -= 0x10000 - units.append(0xD800 + (cp >> 10)) - units.append(0xDC00 + (cp & 0x3FF)) - else: - units.append(cp) - return tuple(units) - - -def _quote(s: str) -> str: - """Serialize a Python string as a JSON string per RFC 8785 §3.2.2.2. - - Applies NFC normalization (spec Section 4.3) before escaping. RFC 8785 - defers to ECMAScript JSON.stringify, which escapes only the quote, the - reverse solidus, and U+0000-U+001F. U+007F, the C1 controls (U+0080- - U+009F) and the line/paragraph separators (U+2028, U+2029) are emitted - literally — they are not part of that escape set. - """ - s = unicodedata.normalize("NFC", s) - buf: list[str] = ['"'] - for ch in s: - cp = ord(ch) - if ch == '"': - buf.append('\\"') - elif ch == "\\": - buf.append("\\\\") - elif ch == "\b": - buf.append("\\b") - elif ch == "\f": - buf.append("\\f") - elif ch == "\n": - buf.append("\\n") - elif ch == "\r": - buf.append("\\r") - elif ch == "\t": - buf.append("\\t") - elif cp <= 0x001F: - buf.append(f"\\u{cp:04x}") - else: - buf.append(ch) - buf.append('"') - return "".join(buf) - - -def _float_to_str(f: float) -> str: - """Serialize a float per RFC 8785 §3.2.2.3 (ECMAScript Number::toString). - - Implements the ECMA-262 Number::toString algorithm directly rather than - reformatting Python's `repr`: `repr(f)` already gives the shortest decimal - digit string that round-trips to *f* (what the spec calls `s`), and - `Decimal(repr(f)).normalize()` recovers that digit string and its exponent - without the two shortcuts (integers bounded at 1e15, exponential notation - switching over at the wrong magnitude) the previous implementation used. - - Raises: - ValueError: If *f* is NaN or Infinity (not permitted by RFC 8785). - """ - if math.isnan(f) or math.isinf(f): - raise ValueError(f"RFC 8785 does not permit NaN or Infinity ({f!r})") - if f == 0.0: - return "0" - sign = "-" if f < 0 else "" - _, digits, exponent = decimal.Decimal(repr(abs(f))).normalize().as_tuple() - digit_str = "".join(str(x) for x in digits) - k = len(digit_str) - n = exponent + k - if k <= n <= 21: - return sign + digit_str + "0" * (n - k) - if 0 < n <= 21: - return sign + digit_str[:n] + "." + digit_str[n:] - if -6 < n <= 0: - return sign + "0." + "0" * (-n) + digit_str - e = n - 1 - mantissa = digit_str[0] if k == 1 else digit_str[0] + "." + digit_str[1:] - return sign + mantissa + "e" + ("+" if e >= 0 else "-") + str(abs(e)) diff --git a/python/tests/test_canonicalize.py b/python/tests/test_canonicalize.py deleted file mode 100644 index 87409587..00000000 --- a/python/tests/test_canonicalize.py +++ /dev/null @@ -1,326 +0,0 @@ -"""RFC 8785 canonical JSON test suite. - -Covers: - - Appendix D test vector (verified via sha256sum) - - Key sort ordering, whitespace, null exclusion - - NFC normalization, string escaping, boolean/float handling - - @context / @type as ordinary fields -""" -import hashlib -import math - -import pytest - -from agent_manifest._canonicalize import canonical_hash, canonicalize - - -# --------------------------------------------------------------------------- -# Spec Appendix D test vector (SHA-256 verified via bash sha256sum) -# --------------------------------------------------------------------------- - -APPENDIX_D_INPUT = { - "version": "0.1", - "issued_at": "2026-06-23T09:00:00Z", - "agent_id": "spiffe://trust.example/agent/kyc/prod-001", -} -APPENDIX_D_CANONICAL = ( - b'{"agent_id":"spiffe://trust.example/agent/kyc/prod-001"' - b',"issued_at":"2026-06-23T09:00:00Z","version":"0.1"}' -) -APPENDIX_D_SHA256 = "b83293348255f4427dc030478f354b83f4f82662223be0926ad9f2db946b5319" - - -def test_appendix_d_canonical_form(): - assert canonicalize(APPENDIX_D_INPUT) == APPENDIX_D_CANONICAL - - -def test_appendix_d_sha256(): - assert hashlib.sha256(APPENDIX_D_CANONICAL).hexdigest() == APPENDIX_D_SHA256 - - -def test_appendix_d_canonical_hash(): - assert canonical_hash(APPENDIX_D_INPUT) == f"sha256:{APPENDIX_D_SHA256}" - - -# --------------------------------------------------------------------------- -# Key ordering -# --------------------------------------------------------------------------- - - -def test_keys_sorted_lexicographic(): - assert canonicalize({"z": 1, "a": 2, "m": 3}) == b'{"a":2,"m":3,"z":1}' - - -def test_nested_keys_sorted(): - assert canonicalize({"b": {"y": 1, "x": 2}, "a": 0}) == b'{"a":0,"b":{"x":2,"y":1}}' - - -def test_unicode_key_ordering(): - # chr(233) = U+00E9 (é) > chr(101) = 'e' - obj = {chr(233): 1, "e": 2} - result = canonicalize(obj) - assert result == ('{"e":2,"' + chr(233) + '":1}').encode("utf-8") - - -def test_utf16_code_unit_key_order_not_code_point_order(): - # RFC 8785 §3.2.3 sorts by UTF-16 code unit. U+10000 (a supplementary- - # plane character) is a surrogate pair D800 DC00 in UTF-16, which sorts - # below the BMP character U+FFFF (code unit FFFF) — the reverse of code - # point order, where U+10000 > U+FFFF. Issue #322. - obj = {chr(0x10000): 1, chr(0xFFFF): 2} - result = canonicalize(obj) - expected = ('{"' + chr(0x10000) + '":1,"' + chr(0xFFFF) + '":2}').encode("utf-8") - assert result == expected - - -def test_utf16_code_unit_key_order_nested(): - # Same divergence one level deep, matching trace-spec's - # 04-utf16-key-order-nested.json boundary vector. - obj = {"outer": {chr(0x10000): 1, chr(0xFFFF): 2}} - result = canonicalize(obj) - assert result == ( - '{"outer":{"' + chr(0x10000) + '":1,"' + chr(0xFFFF) + '":2}}' - ).encode("utf-8") - - -# --------------------------------------------------------------------------- -# Whitespace -# --------------------------------------------------------------------------- - - -def test_no_whitespace(): - result = canonicalize({"a": 1, "b": [1, 2, 3]}) - assert b" " not in result and b"\n" not in result and b"\t" not in result - - -# --------------------------------------------------------------------------- -# Null handling (spec Section 4.3) -# --------------------------------------------------------------------------- - - -def test_null_excluded_by_default(): - assert canonicalize({"a": 1, "b": None, "c": 3}) == b'{"a":1,"c":3}' - - -def test_null_included_when_opted_in(): - assert canonicalize({"a": 1, "b": None}, exclude_none=False) == b'{"a":1,"b":null}' - - -def test_nested_null_excluded(): - assert canonicalize({"outer": {"present": 1, "absent": None}}) == b'{"outer":{"present":1}}' - - -# --------------------------------------------------------------------------- -# Boolean serialization -# --------------------------------------------------------------------------- - - -def test_boolean_true(): - assert canonicalize({"v": True}) == b'{"v":true}' - - -def test_boolean_false(): - assert canonicalize({"v": False}) == b'{"v":false}' - - -def test_bool_not_confused_with_int(): - # bool is a subclass of int — must not serialize True as 1 - assert canonicalize({"a": True, "b": 1}) == b'{"a":true,"b":1}' - - -# --------------------------------------------------------------------------- -# String escaping — using chr() to avoid embedding control chars in source -# --------------------------------------------------------------------------- - - -def test_null_byte_escaped(): - assert canonicalize({"v": chr(0)}) == b'{"v":"\\u0000"}' - - -def test_unit_separator_escaped(): - assert canonicalize({"v": chr(31)}) == b'{"v":"\\u001f"}' - - -def test_backslash_escaped(): - assert canonicalize({"v": "\\"}) == b'{"v":"\\\\"}' - - -def test_double_quote_escaped(): - assert canonicalize({"v": '"'}) == b'{"v":"\\""}' - - -def test_tab_newline_escaped(): - assert canonicalize({"v": "\t\n"}) == b'{"v":"\\t\\n"}' - - -def test_line_separator_not_escaped(): - # RFC 8785 §3.2.2.2 defers to ECMAScript JSON.stringify, which escapes - # only the quote, the reverse solidus, and U+0000-U+001F. U+2028 LINE - # SEPARATOR is a hazard when JSON is pasted into JavaScript *source*, not - # a JSON serialization rule — it MUST be emitted literally. This test - # previously asserted the opposite (non-conformant) behaviour. Issue #322. - assert canonicalize({"v": chr(0x2028)}) == ('{"v":"' + chr(0x2028) + '"}').encode("utf-8") - - -def test_paragraph_separator_not_escaped(): - assert canonicalize({"v": chr(0x2029)}) == ('{"v":"' + chr(0x2029) + '"}').encode("utf-8") - - -def test_delete_and_c1_controls_not_escaped(): - # U+007F (DELETE) and the C1 control range (U+0080-U+009F) are outside - # ECMAScript JSON.stringify's escape set and must be emitted literally. - assert canonicalize({"v": chr(0x7F)}) == ('{"v":"' + chr(0x7F) + '"}').encode("utf-8") - assert canonicalize({"v": chr(0x85)}) == ('{"v":"' + chr(0x85) + '"}').encode("utf-8") - - -def test_regular_unicode_verbatim(): - # Non-control chars pass through after NFC normalization - assert canonicalize({"v": "é"}) == '{"v":"é"}'.encode("utf-8") - - -# --------------------------------------------------------------------------- -# NFC normalization -# --------------------------------------------------------------------------- - - -def test_nfc_normalization(): - precomposed = "é" # é as single code point - decomposed = "é" # e + combining accent - assert canonicalize({"v": precomposed}) == canonicalize({"v": decomposed}) - - -# --------------------------------------------------------------------------- -# Arrays -# --------------------------------------------------------------------------- - - -def test_array_order_preserved(): - assert canonicalize([3, 1, 2]) == b"[3,1,2]" - - -def test_nested_array(): - assert canonicalize([[1, 2], [3, 4]]) == b"[[1,2],[3,4]]" - - -def test_empty_array(): - assert canonicalize([]) == b"[]" - - -def test_empty_object(): - assert canonicalize({}) == b"{}" - - -# --------------------------------------------------------------------------- -# Numbers -# --------------------------------------------------------------------------- - - -def test_integer(): - assert canonicalize({"v": 42}) == b'{"v":42}' - - -def test_negative_integer(): - assert canonicalize({"v": -7}) == b'{"v":-7}' - - -def test_float_integer_value_no_decimal(): - assert canonicalize({"v": 1.0}) == b'{"v":1}' - - -def test_float_nan_raises(): - with pytest.raises(ValueError, match="NaN"): - canonicalize({"v": math.nan}) - - -def test_float_infinity_raises(): - with pytest.raises(ValueError, match="Infinity"): - canonicalize({"v": math.inf}) - - -# --------------------------------------------------------------------------- -# ECMAScript Number::toString formatting — issue #322 differential table -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize( - "value,expected", - [ - (1e-7, "1e-7"), - (3e-8, "3e-8"), - (-1e-7, "-1e-7"), - (1e-6, "0.000001"), - (1e-5, "0.00001"), - (1e15, "1000000000000000"), - (1e16, "10000000000000000"), - (1e20, "100000000000000000000"), - (1e21, "1e+21"), - (123.456, "123.456"), - (100.5, "100.5"), - (0.1, "0.1"), - (1.0, "1"), - (100.0, "100"), - (0.0, "0"), - (-0.0, "0"), - ], -) -def test_float_formatting_matches_ecmascript_number_tostring(value, expected): - assert canonicalize({"v": value}) == ('{"v":' + expected + "}").encode("utf-8") - - -def test_float_exponent_no_leading_zero_or_plus_padding(): - # Python's repr pads exponents to two digits and always signs them - # ("1e-07"); RFC 8785 has neither a minimum width nor a leading zero. - result = canonicalize({"v": 1e-7}) - assert result == b'{"v":1e-7}' - assert b"1e-07" not in result - - -def test_integer_within_safe_range_serializes(): - assert canonicalize({"v": (1 << 53) - 1}) == b'{"v":9007199254740991}' - - -def test_large_integer_allowed(): - result = canonicalize( - {"v": 295147905179352830000} - ) - assert result == b'{"v":295147905179352830000}' - - -def test_large_integer_timestamp_allowed(): - assert canonicalize( - {"timestamp_ns": 1790000000000000000} - ) == b'{"timestamp_ns":1790000000000000000}' - - -# --------------------------------------------------------------------------- -# @context / @type as ordinary fields -# --------------------------------------------------------------------------- - - -def test_context_type_ordinary(): - obj = { - "@context": "https://manifest.agentrust-io.com/v0.2/context.json", - "@type": "AgentManifest", - "manifest_id": "test", - } - result = canonicalize(obj) - # '@' (U+0040) sorts before all letters, so @context comes first - assert result.startswith(b'{"@context"') - assert b'"@type"' in result - assert b'"manifest_id"' in result - - -# --------------------------------------------------------------------------- -# shake256 -# --------------------------------------------------------------------------- - - -def test_shake256_length(): - result = canonical_hash({"v": 1}, algorithm="shake256") - assert result.startswith("shake256:") - assert len(result) == len("shake256:") + 64 - - -def test_unsupported_algorithm_raises(): - with pytest.raises(ValueError, match="Unsupported"): - canonical_hash({"v": 1}, algorithm="md5")