agent-manifest rejects a TRACE record that trace-spec signed: the canonical form is not RFC 8785

[lywinged]

SDK version

0.11.0 (main at 2ce6c97)

Python version

3.11.15

What happened

_trace.py:210 computes the TRACE signature pre-image with this package's canonicalize():

return canonicalize({k: v for k, v in envelope.items() if k != "signature"})

canonicalize() is not RFC 8785 conformant. The clearest case is object key ordering — _serialize_dict, _canonicalize.py:100-105:

def _serialize_dict(d: dict[str, Any], *, exclude_none: bool, depth: int) -> str:
    # RFC 8785 §3.2.3: sort keys by Unicode code point order.
    # Python's str comparison uses Unicode code point order by default — no
    # special locale or collation needed.
    parts: list[str] = []
    for k in sorted(d.keys()):

RFC 8785 §3.2.3 sorts by UTF-16 code unit, not code point. The two agree across the BMP and part company once a key holds a supplementary-plane character, because its surrogate pair sorts below every high BMP character while its code point sorts above them.

trace-spec names this trap by name. spec/trace-v0.2.md §3.2.2:

Object keys are sorted by UTF-16 code unit (ascending), per RFC 8785 §3.2.3. [...] Sorting Python str values with sorted() gives code-point order and is wrong here; an RFC 8785 library gets this right.

and, in the same section:

Implementations MUST use an RFC 8785-conformant library.

What you expected

A TRACE record signed by trace-spec's own tooling verifies here. It does not.

Running trace-spec's published canonicalization-boundary corpus (examples/canonicalization-boundary/, four records, each valid against its own declared key over its own RFC 8785 bytes) through trace_signing_pre_image:

vector                     pre-image bytes   trace-spec verify   agent-manifest verify
non-ascii-values           identical         VALID               VALID
non-bmp-values             identical         VALID               VALID
utf16-key-order            DIFFERENT         VALID               REJECTED
utf16-key-order-nested     DIFFERENT         VALID               REJECTED

trace-spec's own verifier accepts   4/4
agent-manifest's pre-image accepts  2/4

Same records, same keys, same signatures.

Key order is not the only axis. A differential of canonicalize() against a reference JCS library also finds:

	this SDK	RFC 8785
exponent keeps repr's leading zero	1e-07, 3e-08	1e-7, 3e-8
exponential form starts too early	1e-06, 1e-05	0.000001, 0.00001
integral shortcut bounded at 1e15, ECMAScript's is 1e21	1e+16, 1e+20, 1000000000000000.0	10000000000000000, 100000000000000000000, 1000000000000000
integers past 2^53	serialized silently	refused — outside the IEEE-754 domain
over-escaping	\u007f, \u0085, \u2028	literal

_canonicalize.py's own docstring calls it the "single canonicalization entry point for all signing, hashing, and Merkle tree operations". It is also the pre-image for COSE signing, delegation chains, revocation records, memory deltas and plugin bundles — TRACE is just where a published corpus exists to demonstrate it.

One existing test asserts the non-conformant behaviour, which is part of why this survived: test_line_separator_escaped requires U+2028 to be escaped. RFC 8785 §3.2.2.2 defers to ECMAScript JSON.stringify, which escapes the quote, the reverse solidus and U+0000-U+001F and nothing else; U+2028 is a hazard when JSON is pasted into JavaScript source, not a JSON serialization rule.

Reproduction

import base64, json, rfc8785
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
from agent_manifest._trace import trace_signing_pre_image

# trace-spec examples/canonicalization-boundary/03-utf16-key-order.json
v = json.load(open("03-utf16-key-order.json"))
record, jwk = v["record"], v["trusted_key"]
b64 = lambda s: base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
body = {k: val for k, val in record.items() if k != "signature"}

assert trace_signing_pre_image(record) == rfc8785.dumps(body)   # fails
Ed25519PublicKey.from_public_bytes(b64(jwk["x"])).verify(
    b64(record["signature"]), trace_signing_pre_image(record))  # InvalidSignature

I have a patch: UTF-16 code-unit key order, ECMAScript Number::toString (verified byte-identical to a reference JCS library across 250,311 values — 200k random doubles, a 50k decimal grid, and the fixed corpus), the integer-domain refusal, the escaping set corrected, plus trace-spec's four vectors vendored as a cross-repository regression guard. Verified load-bearing by reverting: with the key-order fix removed, 4 of the guard's 9 cases fail; restored, 9 pass. Full suite is unchanged against baseline — same 25 pre-existing environment failures, 1040 → 1054 passed.

Two things worth your call rather than mine, which is why this is an issue and not a PR. It changes the canonical bytes for affected values, so anything this SDK already signed at one of those values stops verifying against itself — though it was already unverifiable everywhere else, which is the point. And you are mid CoSAI WS4 Phase 1 review, so the timing of a change at the signature layer is yours. Say the word and I'll open it.

[github-actions]

❔ Contributor Check: UNKNOWN

Check	Result
Profile	UNKNOWN
Credential	LOW
Overall	UNKNOWN

Automated check by AgenTrust Contributor Check.

[imran-siddique]

Confirmed, and it is worse than a conformance nit. Please open the PR.

I reproduced the key-order break directly rather than reading it off your table. At main 7858b81, _serialize_dict sorts with sorted(d.keys()) and its own comment states "RFC 8785 §3.2.3: sort keys by Unicode code point order". That comment is the bug: §3.2.3 requires UTF-16 code-unit order, and the two part company exactly where you say. For the keys U+FFFF and U+10000:

sorted()          -> U+FFFF, U+10000      (0xFFFF < 0x10000)
rfc8785.dumps()   -> U+10000, U+FFFF      (0xD800 < 0xFFFF)

Different pre-image, so the signature cannot verify. The comment asserting the wrong rule is why this survived review: a reader checking the code against the comment finds them in agreement.

What makes it a priority is the blast radius you name at the end. canonicalize() is the single pre-image for COSE signing, delegation chains, revocation records, memory deltas and plugin bundles. TRACE is only where a published corpus exists to demonstrate it, so 2 of 4 on the canonicalization-boundary vectors is a measurement of all of those surfaces at once. And trace-spec spec/trace-v0.2.md §3.2.2 names this exact trap, including that sorted() is wrong here, and says implementations MUST use an RFC 8785-conformant library. We wrote the warning and then did not follow it.

Take the full scope you described. Specifically:

• UTF-16 code-unit key order. The blocking fix.
• Number::toString, the integer domain refusal, and the escaping set. These change bytes for records that verify today, which normally argues for a separate PR. Not here. Every one of them is a case where we produce a pre-image no conformant verifier reproduces, so they are all the same defect and splitting them would ship a version that is conformant on one axis and not the others. Refusing integers past 2^53 rather than serializing them silently is right: outside the IEEE-754 domain there is no correct answer, and a quiet one is the worst of the three.
• Delete test_line_separator_escaped. You are right about it and it should go in the same PR with a comment saying why. RFC 8785 §3.2.2.2 defers to ECMAScript JSON.stringify, which escapes the quote, the reverse solidus and U+0000 through U+001F and nothing else. U+2028 is a hazard when JSON is pasted into JavaScript source, not a JSON serialization rule, and a test asserting non-conformant behaviour is how a wrong implementation gets defended.
• Vendor trace-spec's four vectors. Yes. A cross-repository regression guard is the thing that stops this recurring, and load-bearing verification by reverting the fix (4 of 9 failing without it, 9 passing with) is the evidence I want in the PR body.

Two things to include:

The 250,311-value differential against a reference JCS library is the strongest part of this and belongs in the PR description, with the generation method, so a later reader can rerun it rather than trust it.

Say in the PR whether any published artifact was signed with the non-conformant pre-image. If records exist in the wild that verify only under the old bytes, that is a migration question rather than a bug fix, and I would rather decide it in the open than discover it.

Also fix the comment, not just the code. # RFC 8785 §3.2.3: sort keys by UTF-16 code unit, not code point with a line saying why Python's default is wrong here.

The UNKNOWN on the contributor check is the bot, not you. It returns UNKNOWN on any transient failure with no retry and it lands on you repeatedly. It carries no information.

[imran-siddique]

Implementation is in open PR #338. Keeping this issue open until the canonicalization fix and its compatibility questions are resolved in that PR.