diff --git a/AGENTS.md b/AGENTS.md index 61f3ea8..4dc9f44 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,10 +23,10 @@ Core product traits right now: Treat these as core constraints unless the owner explicitly changes the product direction. - The app is a single exported client-side shell, not a backend product. -- Artifact payloads live in the URL fragment, using the compact `#` form where the single tag char identifies the codec: `p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4. Legacy `#agent-render=v1..` links (arx-family carry an extra `.` segment) still decode but are no longer emitted. +- Artifact payloads live in the URL fragment, using the compact `#` form where the single tag char identifies the codec: `p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4, `f` arx5. Legacy `#agent-render=v1..` links (arx-family carry an extra `.` segment) still decode but are no longer emitted. - The deployed host should not receive artifact contents as part of the initial page request. - Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`. -- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`. +- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, `arx4`, and `arx5`. Auto-emit prefers `arx5`; `arx3` and `arx4` remain decodable and explicitly encodable but are not auto-selected. - The product is zero-retention by host design, not secret-safe in an absolute sense. - Links may still leak through browser history, copied URLs, screenshots, and any future client-side analytics. @@ -84,10 +84,11 @@ The fragment transport is part of the product surface, not an implementation det Current rules: - fragment key: `agent-render` (legacy decode path only; the compact form has no key) -- emitted format: compact `#`, where the single tag char identifies the codec (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4); the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and refuses to decode a newer one) -- legacy format (still decodable, no longer emitted): `agent-render=v1..` for `plain|lz|deflate`, `agent-render=v1.arx..` for `arx`, `agent-render=v1.arx2..` for `arx2`, and `agent-render=v1.arx3..` for `arx3` -- codecs: `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4` -- `arx4` payloads carry one extra leading char after the tag, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the context mixer ran before the payload; `m`/`c`/`j` additionally need `/arx4-priors.json`, and an encoder that cannot load it emits `s` instead +- emitted format: compact `#`, where the single tag char identifies the codec (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4, `f` arx5); the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and refuses to decode a newer one) +- legacy format (still decodable, no longer emitted): `agent-render=v1..` for `plain|lz|deflate`, `agent-render=v1.arx..` for `arx`, `agent-render=v1.arx2..` for `arx2`, `agent-render=v1.arx3..` for `arx3`, `agent-render=v1.arx4..` for `arx4`, and `agent-render=v1.arx5..` for `arx5` +- codecs: `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, `arx4`, and `arx5` +- `arx5` is the emitted mixer codec (ARX 4.5): ARX4's context mixer on ARX2's tuple/overlay pipeline, with every wire scored by honest serialized transport length. `arx3` and `arx4` stay decodable but are no longer auto-emitted because they score dense Unicode by visible character count +- `arx4`/`arx5` payloads carry one extra leading char after the tag, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the context mixer ran before the payload; `m`/`c`/`j` additionally need `/arx4-priors.json`, and an encoder that cannot load it emits `s` instead - fragment size budget: `8192` characters - decoded payload budget: `200000` characters - Discord markdown link limit: `2000` characters for the full formatted `[label](url)` string; `createGeneratedArtifactLink*` returns `discordMarkdownLinkWarning` when exceeded @@ -124,7 +125,7 @@ If you change the payload contract, update the code, docs, examples, and the Ope - `src/lib/payload/schema.ts` - type surface, limits, fragment key, supported kinds/codecs - `src/lib/payload/fragment.ts` - encode/decode logic and transport behavior - `src/lib/payload/arx-codec.ts` - arx/arx2/arx3 codecs: dictionary substitution, tuple overlay, brotli, base76/base1k/baseBMP/base64url encoding -- `src/lib/payload/arx4-codec.ts` - arx4 codec: the arx3 stages with brotli replaced by a deterministic integer context mixer, plus the curated priors it primes on +- `src/lib/payload/arx4-codec.ts` - arx4/arx5 mixer: the arx2 stages with brotli replaced by a deterministic integer context mixer, plus the curated priors it primes on - `public/arx4-priors.json` - curated per-kind arx4 priming corpora (and `.br` pre-compressed variant), regenerated by `scripts/build-arx4-priors.mjs` - `public/arx-dictionary.json` - shared substitution dictionary for the arx codec (served as a static endpoint) - `public/arx-dictionary.json.br` - pre-compressed brotli variant of the dictionary diff --git a/CHANGELOG.md b/CHANGELOG.md index affba4f..fb39f6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and ### Added - In-viewer edit-and-reshare: open any artifact, correct its content, and generate a new fragment link without leaving the viewer. +- `arx5` (ARX 4.5): ARX4's context mixer on ARX2's tuple pipeline, scored by honest serialized transport length so Discord and WhatsApp no longer detonate Unicode wires. + +### Changed + +- Auto-emit no longer selects `arx3` or `arx4`. Existing `#c` and `#e` links still decode. ## [0.1.0] - 2026-05-05 diff --git a/README.md b/README.md index 61ebfc7..f009853 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ Built for the OpenClaw ecosystem, `agent-render` focuses on fragment-based shari ## Status - Markdown, code, diff, CSV, and JSON all render in the static shell -- Fragment transport supports `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`, with automatic shortest-fragment selection across available wire formats -- The `arx` substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the `arx2` tuple-envelope overlay is served at `/arx2-dictionary.json` with a pre-compressed `/arx2-dictionary.json.br` variant; `arx3` reuses those proven bytes and optimizes for compact visible Unicode fragments; `arx4` adds the curated context-mixer priors at `/arx4-priors.json` with a pre-compressed `/arx4-priors.json.br` variant +- Fragment transport supports `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, `arx4`, and `arx5`, with automatic shortest-fragment selection across available wire formats. Auto-emit prefers `arx5`; `arx3` and `arx4` remain decodable but are no longer selected +- The `arx` substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the `arx2` tuple-envelope overlay is served at `/arx2-dictionary.json` with a pre-compressed `/arx2-dictionary.json.br` variant; `arx5` (ARX 4.5) applies the context mixer from `arx4` on top of the `arx2` tuple pipeline and scores every wire by honest serialized transport length; `arx3`/`arx4` stay decodable and explicitly encodable but are not auto-selected, because their visible-length Unicode scoring detonates on Discord and WhatsApp. Mixer priors live at `/arx4-priors.json` with a pre-compressed `/arx4-priors.json.br` variant - The viewer toolbar copies artifact bodies to the clipboard, downloads them as files, (for markdown) supports browser print-to-PDF, and can edit the open artifact then reshare it as a new fragment link - Deployment target: static hosting, including Cloudflare Pages diff --git a/docs/architecture.md b/docs/architecture.md index e327b76..f50017f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -98,12 +98,13 @@ The fragment protocol keeps the JSON envelope stable and treats compression stri - `deflate` stores deflate-compressed UTF-8 JSON bytes when it outperforms other codecs - `arx` applies domain-dictionary substitution, brotli compression (quality 11), and binary-to-text encoding for best-in-class compression. Four wire shapes are candidates: base76 (ASCII, 77 fragment-safe chars), base64url (RFC 4648 `A-Za-z0-9-_` with a `B.` prefix for detection), base1k (Unicode, 1774 chars from U+00A1–U+07FF), and baseBMP (high-density Unicode, ~62k safe BMP code points from U+00A1–U+FFEF, ~15.92 bits/char). The async encoder tries all four and picks the shortest **transport** length (percent-encoded UTF-8 length for non-ASCII), so base64url can win over Unicode encodings on chat-style surfaces. baseBMP produces ~32% fewer characters than base1k and ~60% fewer than base76 for the same compressed bytes, achieving ~70% smaller fragments than deflate on typical payloads (~6.1x compression ratio for 8k markdown). Full pipeline timing is on the order of ~8–14ms for 8k payloads depending on the wire encoding. The substitution dictionary is served as a static file at `/arx-dictionary.json` so agents can fetch it for local compression; a pre-compressed `/arx-dictionary.json.br` variant is also available. The viewer tries the pre-compressed dictionary first on default ARX-family loads, falls back to the JSON file, and only loads external dictionaries when an ARX/ARX2/ARX3 encode or decode path needs them. - `arx2` keeps the arx compression stack but replaces the JSON envelope with a compact tuple envelope and applies `/arx2-dictionary.json` as an overlay before the shared arx dictionary. The viewer tries `/arx2-dictionary.json.br` first for default overlay loads and falls back to JSON. It is emitted with the compact `b` tag (which identifies the codec but does not carry a dictionary version — it implies the current pinned dictionary) and decodes back to the standard envelope before validation/rendering. -- `arx3` uses the same tuple envelope, overlay dictionary, shared arx dictionary, and brotli bytes as arx2, then allows the dense baseBMP wire to win by decoded visible character length. This deliberately optimizes copyable visible URL length for trusted Unicode-preserving surfaces; it is not a stronger compressed-byte format than arx2. -- `arx4` keeps the arx3 tuple envelope, overlay dictionary, shared arx dictionary and baseBMP selection rule, and replaces brotli with a deterministic integer context mixer (`arx4-codec.ts`). It is emitted with the compact `e` tag and carries one extra leading payload char, the prior id, naming the priming corpus both sides run before the payload. See `docs/payload-format.md` for the prior ids and the `/arx4-priors.json` asset. +- `arx3` is deprecated for automatic emission. It used the same bytes as arx2 but scored baseBMP by visible character count, which Discord and WhatsApp then percent-encode or mangle. Existing `#c` links still open; explicit `{ codec: "arx3" }` still encodes. +- `arx4` is deprecated for automatic emission. It is the context-mixer codec with the same broken visible-length policy as arx3. Existing `#e` links still open; explicit `{ codec: "arx4" }` still encodes. +- `arx5` (ARX 4.5) keeps the arx2 tuple envelope, overlay dictionary, shared arx dictionary, and arx4 context mixer, then scores every wire — including baseBMP — by honest serialized transport length. It is emitted with the compact `f` tag and the same prior-id prefix as arx4. See `docs/payload-format.md` for the prior ids, `/arx4-priors.json`, and the chat-safe alphabet research. - packed wire mode (`p: 1`) shortens transport keys before compression, then unpacks back to the standard envelope during decode -- automatic async codec selection tries `arx4 -> arx3 -> arx2 -> arx -> deflate -> lz -> plain`; arx compares packed + non-packed candidates, while arx2/arx3/arx4 use tuple envelopes +- automatic async codec selection tries `arx5 -> arx2 -> arx -> deflate -> lz -> plain`; arx compares packed + non-packed candidates, while arx2/arx5 use tuple envelopes. Explicit `{ codec: "arx3" }` or `{ codec: "arx4" }` still encodes for back-compat. - sync codec selection (used by examples and legacy paths) tries `deflate -> lz -> plain` -- decode enforces both visible fragment length and decoded payload size ceilings before UI rendering; arx/arx2/arx3 Brotli decompression uses a streaming output cap before final JSON or tuple parsing +- decode enforces both visible fragment length and decoded payload size ceilings before UI rendering; arx/arx2/arx3 Brotli decompression uses a streaming output cap before final JSON or tuple parsing; arx4/arx5 use the context mixer - invalid bundle state is normalized or rejected before renderers mount ## Zero-retention boundaries diff --git a/docs/dependency-notes.md b/docs/dependency-notes.md index c17ef1e..848a80c 100644 --- a/docs/dependency-notes.md +++ b/docs/dependency-notes.md @@ -33,7 +33,7 @@ - `@git-diff-view/*` fits review-style diffs better than a generic merge editor for the current viewer. Its pure CSS file is mirrored into `public/vendor/diff-view-pure.css` with a Brotli-compressed `public/vendor/diff-view-pure.css.br` copy by `npm run assets:compress`, and loaded only by the diff renderer; `tests/diff-style-asset.test.ts` keeps those assets in sync with the package copy. - `papaparse` handles CSV parsing; CSV rendering uses a native read-only table to avoid a data-grid dependency for the shipped static viewer. - `fflate` provides portable deflate/inflate support across iOS Safari and Android Chromium without relying on browser-specific compression streams. -- `brotli-wasm` provides the arx/arx2/arx3 Brotli compression layer, including streaming decompression used to cap expanded output before allocating oversized decoded payloads. +- `brotli-wasm` provides the arx/arx2/arx3 Brotli compression layer, including streaming decompression used to cap expanded output before allocating oversized decoded payloads. arx4/arx5 use the integer context mixer instead. - `mermaid` renders diagram definitions (flowcharts, sequence diagrams, etc.) to SVG client-side. Dynamically imported within the markdown renderer so it does not affect initial bundle size. - `better-sqlite3` provides synchronous SQLite access for the optional self-hosted server mode. Only used by `selfhosted/` code and not bundled into the static frontend export. diff --git a/docs/payload-format.md b/docs/payload-format.md index 698db11..653fe44 100644 --- a/docs/payload-format.md +++ b/docs/payload-format.md @@ -14,8 +14,9 @@ Payload contents are untrusted user content. Viewers, agents, and automations sh #d (deflate) #a (arx) #b (arx2) -#c (arx3) -#e (arx4) +#c (arx3, deprecated emit) +#e (arx4, deprecated emit) +#f (arx5) ``` The compact fragment is a single codec tag char followed by the payload. The tag encodes the codec so unsupported formats fail cleanly; the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and rejects a newer one). The legacy `#agent-render=v1..` form (arx-family carry an extra `.` segment) still decodes for back-compatibility but is no longer emitted. Fragment URLs can look long because they carry the artifact payload in the browser-only fragment instead of sending it to the host during the page request. @@ -26,9 +27,10 @@ Supported codecs: - `lz` - `lz-string` compressed JSON encoded for URL-safe transport - `deflate` - deflate-compressed UTF-8 JSON bytes encoded as base64url - `arx` - domain-dictionary substitution + brotli (quality 11) + binary-to-text encoding. The compact `a` tag identifies the arx codec but does not carry a dictionary version — it implies the build's current pinned dictionary (the build refuses to decode a forward-incompatible newer dictionary). Four wire shapes are tried and the shortest **transport** size wins (see `computeTransportLength` in `fragment.ts` — non-ASCII Unicode may count longer after percent-encoding): **base76** (ASCII-only, 77 fragment-safe chars), **base64url** (standard RFC 4648 alphabet `A-Za-z0-9-_`, no padding, prefixed with `B.` for detection), **base1k** (Unicode, 1774 chars from U+00A1–U+07FF), and **baseBMP** (high-density Unicode, ~62k safe BMP code points from U+00A1–U+FFEF, ~15.92 bits/char). BaseBMP produces ~32% fewer characters than base1k and ~60% fewer than base76 for the same compressed bytes. BaseBMP payloads are prefixed with a U+FFF0 marker for detection. The viewer’s `arxDecompress` auto-detects the wire shape (including the rare case where a base76 length prefix is also `B.` — it tries base64url first and falls back to base76 if Brotli fails). The substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the viewer tries the `.br` file first on default loads and falls back to JSON. The arx2 overlay dictionary follows the same `.br`-then-JSON default load pattern. -- `arx2` - tuple-envelope transport + arx2 overlay substitution + the shared arx dictionary + brotli (quality 11) + the same four binary-to-text wire shapes. The compact `b` tag identifies arx2 but does not carry a dictionary version — it implies the current pinned shared arx dictionary and arx2 overlay. Existing `arx` links remain valid; async auto-selection keeps arx2 as the conservative transport-measured tuple codec. -- `arx3` - the same tuple envelope, overlay substitution, shared arx dictionary, and brotli bytes as arx2, with a different selection rule: baseBMP may win by decoded visible character length instead of conservative percent-encoded transport length. This is the compact visible URL mode for trusted surfaces that preserve Unicode fragments. If a platform rewrites, truncates, or previews links aggressively, prefer arx2/base64url or UUID mode instead. -- `arx4` - the arx3 tuple/overlay/dictionary stages and the arx3 baseBMP selection rule, with brotli replaced by a deterministic integer context mixer (`arx4-codec.ts`). The payload carries one extra leading char, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the coder ran before the payload; an unrecognized prior id is a decode error. Every prior starts from the pinned dictionary slot text, so the `e` tag implies that dictionary twice over; the `m`, `c` and `j` priors append a curated per-kind corpus from `/arx4-priors.json` (pre-compressed `/arx4-priors.json.br` tried first), which the viewer fetches lazily on the first arx4 encode or decode. The `s` id is that shared dictionary text alone, and `n` runs the mixer cold with no prior at all. If the asset is unavailable the encoder falls back to the `s` prior and emits an `s` id, so link creation never blocks; decoding a fragment that names a curated prior fails cleanly instead of coding against a different corpus than the id names. It codes ~10% smaller than arx3 on the sample corpus and is roughly 100x slower, which is why the whole arx family is async-only. +- `arx2` - tuple-envelope transport + arx2 overlay substitution + the shared arx dictionary + brotli (quality 11) + the same four binary-to-text wire shapes. The compact `b` tag identifies arx2 but does not carry a dictionary version — it implies the current pinned shared arx dictionary and arx2 overlay. Existing `arx` links remain valid; async auto-selection keeps arx2 in the pool as the conservative Brotli tuple codec (needed for some CSV regressions). +- `arx3` - **deprecated emit.** Same compressed bytes as arx2. The only difference was scoring baseBMP by visible character count instead of serialized URL length, so Unicode won artificially and then Discord markdown / WhatsApp percent-encoding detonated the link. Existing `#c` links still decode. Do not mint new arx3 links. +- `arx4` - **deprecated emit.** ARX2's tuple/overlay pipeline with Brotli replaced by the deterministic context mixer, plus a prior id char, but it kept arx3's broken visible-length baseBMP policy. Existing `#e` links still decode. Do not mint new arx4 links. +- `arx5` - ARX 4.5: the sane mixer codec. Same context mixer, priors, and wire shapes as arx4 (`arx4-codec.ts`), scored with ARX2's honest serialized transport length for every wire including baseBMP. The compact `f` tag identifies arx5. The payload still carries one extra leading char, the prior id (`m`, `c`, `j`, `s`, or `n`); `m`/`c`/`j` need `/arx4-priors.json` (pre-compressed `/arx4-priors.json.br` tried first). If the asset is unavailable the encoder falls back to `s`. Auto-selection prefers arx5, then arx2. It is roughly 100x slower than Brotli, which is why the whole arx family is async-only. The encoder now also supports a packed wire representation (`p: 1`) that shortens key names before compression. Packed mode is transport-only; decoded envelopes normalize back to the standard shape. @@ -106,9 +108,9 @@ Tuple fields: - Supported decoded payload budget: 200,000 characters - Discord markdown link limit: 2,000 characters for the full formatted `[label](url)` string - Larger payloads should fail with a clear error before rendering -- Compression is selected automatically across packed/non-packed candidates; arx and arx2 optimize conservative transport length, while arx3 optimizes compact visible length for its dense Unicode wire +- Compression is selected automatically across packed/non-packed candidates. Live codecs (`arx`, `arx2`, `arx5`) optimize conservative percent-escaped transport length. Deprecated `arx3`/`arx4` still optimize compact visible length when explicitly requested - Default sync codec priority is `deflate -> lz -> plain` -- Default async codec priority is `arx4 -> arx3 -> arx2 -> arx -> deflate -> lz -> plain` +- Default async codec priority is `arx5 -> arx2 -> arx -> deflate -> lz -> plain` - Optional budget-aware encoding can target strict limits and returns the shortest fragment when none fit - `createGeneratedArtifactLink` / `createGeneratedArtifactLinkAsync` return `url`, `markdownLink` (ready to paste verbatim in chat), `markdownLinkLength`, and `discordMarkdownLinkWarning` so agents do not need to reconstruct `[label](url)` themselves @@ -125,7 +127,30 @@ Running `npm run bench:codecs` checks a fixed corpus across markdown, a real cod - `arx3` visible delta vs arx2: 60.48% fewer visible fragment characters - real code-bench report row: arx2 is 2,984 visible fragment characters; arx3 is 1,142 -The gate fails if arx2 is less than 0.5% smaller overall, if arx3 is less than 35% smaller by visible characters overall, or if any individual corpus row regresses by more than 0.5%. Use `npm run bench:codecs:update` only when intentionally refreshing the committed baseline. +The gate fails if arx2 is less than 0.5% smaller overall, if arx3 is less than 35% smaller by visible characters overall, or if any individual corpus row regresses by more than 0.5%. The arx3 visible-character row is historical: auto-emit no longer uses that policy. Use `npm run bench:codecs:update` only when intentionally refreshing the committed baseline. + +### Chat-safe alphabets + +Auto-selection measures every live wire with `computeTransportLength` in `fragment.ts`: RFC 3986 unreserved characters plus `=` stay 1, other ASCII punctuation counts as 3 (percent-escaped), and non-ASCII counts as 6/9/12 by UTF-8 width. That is the honest Discord/WhatsApp cost. + +Researched surface constraints: + +| Surface | What survives | What detonates | +| --- | --- | --- | +| Discord markdown `[label](url)` | RFC 3986 unreserved `A-Za-z0-9-._~` and `=`. 2,000-character limit on the whole formatted link. | Unicode is canonicalized and percent-encoded (a BMP char becomes 9 characters). `)` closes the destination. | +| Discord bare URL | Same unreserved set. Unicode is still percent-encoded in the client. | Dense BMP/base1k fragments explode past the message limit. | +| WhatsApp | Bare `https://` URLs only; no `[label](url)`. Unreserved ASCII in the path/fragment is typically kept. | `*bold*`, `_italic_`, and `~strike~` are formatting markers if URL detection fails. Unicode is often mangled or dropped from the tappable range. Trailing `.,!)` is stripped by linkifiers. | +| Browser fragment (WHATWG) | Unreserved plus most sub-delims. Fragment percent-encode set is only C0, space, `"<>\``. | Does not predict chat apps. A browser-safe Unicode fragment is not Discord-safe. | + +Largest alphabet that is safe on both Discord markdown and WhatsApp without mangling: **RFC 3986 unreserved `A-Za-z0-9-._~` (66 chars)**. Adding `=` (already treated as chat-safe) makes 67. That is only ~0.7–1.2% denser than base64url's 64-character `A-Za-z0-9-_`. + +Why a new base66/67 wire is not worth it: + +- base64url already uses 64 of those 66 characters and is proven in production `#bB.` / `#fB.` links +- base76's extra punctuation (`!$*()',;:@/`) is either fatal (`)`) or 3x after chat escaping, so honest scoring already rejects it +- base1k/baseBMP look shortest by visible character count and then explode 3–9x when a chat client percent-encodes them + +arx5 therefore keeps the existing four wires and lets honest transport length pick. In practice that is base64url (or base76 when its punctuation does not inflate). ## Active artifact behavior diff --git a/docs/testing.md b/docs/testing.md index 22f4639..b0285c8 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -48,7 +48,7 @@ The suite is intentionally split by responsibility: - component tests protect selector/disclosure UI contracts - unit tests protect transport codecs, envelope validation, diff parsing, and language inference - `npm run assets:compress` regenerates minified/precompressed public assets, including the ARX dictionaries and mirrored diff-view stylesheet -- `npm run bench:codecs` protects arx/arx2 compressed-byte ratios and arx3 visible-character wins against the committed `scripts/bench-baseline.json`; its corpus is fixed in `scripts/bench-codecs.mjs` so unrelated source, docs, or package metadata edits do not create false codec regressions +- `npm run bench:codecs` protects arx/arx2 compressed-byte ratios and the historical arx3 visible-character row against the committed `scripts/bench-baseline.json`; its corpus is fixed in `scripts/bench-codecs.mjs` so unrelated source, docs, or package metadata edits do not create false codec regressions. Auto-emit no longer uses the arx3 visible-length policy. - `npm run check:build-budgets` reads the generated `.next` manifests after `npm run build` and fails if the homepage shell or key deferred renderer chunks exceed their gzip budgets ## Self-hosted mode tests diff --git a/docs/url-fragments.md b/docs/url-fragments.md index 717aad1..81c0013 100644 --- a/docs/url-fragments.md +++ b/docs/url-fragments.md @@ -3,14 +3,14 @@ agent-render links carry the artifact in the URL fragment: ```text -https://agent-render.com/#c +https://agent-render.com/#f ``` Everything before `#` loads the static app. Everything after `#` is the artifact payload the browser decodes locally. ## What the parts mean -- The first character after `#` is a single codec tag. Here `c` means the `arx3` codec. (The tag does not carry a dictionary version; it implies the current dictionary.) +- The first character after `#` is a single codec tag. Here `f` means the `arx5` codec. (The tag does not carry a dictionary version; it implies the current dictionary.) - `` is the encoded artifact bundle. The tag char identifies the codec: @@ -21,11 +21,12 @@ The tag char identifies the codec: #d (deflate) #a (arx) #b (arx2) -#c (arx3) -#e (arx4) +#c (arx3, deprecated emit) +#e (arx4, deprecated emit) +#f (arx5) ``` -For `arx`, `arx2`, `arx3`, and `arx4`, the compact tag does not carry a dictionary version — it implies the current dictionary (the build pins the newest supported version and rejects a newer one). Only the legacy header below carries an explicit dictionary version. +For `arx`, `arx2`, `arx3`, `arx4`, and `arx5`, the compact tag does not carry a dictionary version — it implies the current dictionary (the build pins the newest supported version and rejects a newer one). Only the legacy header below carries an explicit dictionary version. Older links may use the legacy shape, which the viewer still decodes: @@ -33,11 +34,11 @@ Older links may use the legacy shape, which the viewer still decodes: #agent-render=v1.. ``` -where `` is `plain`, `lz`, or `deflate`, and the ARX-family legacy links include the dictionary version (`#agent-render=v1.arx..`, `arx2`, `arx3`). These legacy links are no longer emitted. +where `` is `plain`, `lz`, or `deflate`, and the ARX-family legacy links include the dictionary version (`#agent-render=v1.arx..`, `arx2`, `arx3`, `arx4`, `arx5`). These legacy links are no longer emitted. ## Why arx exists -Artifacts can be bigger than a comfortable URL. The ARX family keeps links shorter by applying agent-render substitution dictionaries, Brotli compression, tuple envelopes for arx2/arx3, and binary-to-text encoding. `arx3` favors compact visible Unicode fragments, so it can look especially strange even though the browser decodes it locally. +Artifacts can be bigger than a comfortable URL. The ARX family keeps links shorter by applying agent-render substitution dictionaries, Brotli or the context mixer, tuple envelopes, and binary-to-text encoding. Live links use chat-safe ASCII wires (usually base64url). Older `arx3`/`arx4` links may contain dense Unicode; the viewer still decodes them, but new links do not emit that form because Discord and WhatsApp percent-encode or mangle it. ## Privacy tradeoff diff --git a/skills/agent-render-linking/SKILL.md b/skills/agent-render-linking/SKILL.md index 437fdca..e7a21fc 100644 --- a/skills/agent-render-linking/SKILL.md +++ b/skills/agent-render-linking/SKILL.md @@ -31,11 +31,12 @@ character identifying the codec: #d (deflate) #a (arx) #b (arx2) -#c (arx3) -#e (arx4) +#c (arx3, deprecated emit) +#e (arx4, deprecated emit) +#f (arx5) ``` -The single tag char identifies the codec; for `arx`/`arx2`/`arx3`/`arx4` it implies +The single tag char identifies the codec; for `arx`/`arx2`/`arx3`/`arx4`/`arx5` it implies the current dictionary but does not carry a dictionary version. The payload follows immediately after it. The legacy `#agent-render=v1..` form (arx-family carry an extra `.` segment) still decodes, but the @@ -46,15 +47,17 @@ Supported codecs: - `lz`: `lz-string` compressed JSON encoded for URL-safe transport - `deflate`: deflate-compressed UTF-8 JSON bytes encoded as base64url - `arx`: domain-dictionary substitution + brotli (quality 11) + binary-to-text encoding (~70% smaller than deflate with baseBMP). Fetch the shared dictionary from `https://agent-render.com/arx-dictionary.json` to apply substitutions locally before brotli compression. Four wire shapes: baseBMP (~62k safe BMP code points, ~15.92 bits/char, best raw density), base1k (1774 Unicode code points U+00A1–U+07FF), base64url (ASCII `A-Za-z0-9-_`, `B.` prefix — good when Unicode would be percent-encoded), and base76 (77-char ASCII). The product encoder tries all four and picks the shortest **transport** length. -- `arx2`: tuple-envelope transport + `https://agent-render.com/arx2-dictionary.json` overlay (or pre-compressed `https://agent-render.com/arx2-dictionary.json.br`) + the shared arx dictionary + brotli (quality 11) + the same four wire shapes. Existing arx links remain valid; prefer arx2 when it is the shortest transport. -- `arx3`: same tuple envelope, arx2 overlay, shared arx dictionary, and brotli bytes as arx2, but the dense baseBMP wire may win by decoded visible character length. Use it for trusted surfaces that preserve Unicode fragments and strict visible URL budgets. Prefer arx2/base64url or UUID mode when the target platform rewrites, truncates, or previews long links aggressively. -- `arx4`: the same tuple envelope, arx2 overlay, shared arx dictionary and baseBMP selection rule as arx3, with brotli replaced by a deterministic integer context mixer and one extra leading payload char, the prior id (`m`, `c`, `j`, `s`, or `n`). Recognize and open `#e` links; do not hand-roll them. Reproducing the wire needs the exact frozen mixer plus `https://agent-render.com/arx4-priors.json`, so an agent encoding on its own should stop at `arx3` and let the app or library emit arx4. +- `arx2`: tuple-envelope transport + `https://agent-render.com/arx2-dictionary.json` overlay (or pre-compressed `https://agent-render.com/arx2-dictionary.json.br`) + the shared arx dictionary + brotli (quality 11) + the same four wire shapes. Existing arx links remain valid; prefer arx2 when you encode yourself and need a chat-safe ASCII wire. +- `arx3`: **deprecated emit.** Same bytes as arx2, but it scored baseBMP by visible character count. Discord and WhatsApp then percent-encode or mangle those Unicode fragments. Recognize and open `#c` links; do not mint new ones. +- `arx4`: **deprecated emit.** Context mixer plus the same broken visible-length Unicode policy. Recognize and open `#e` links; do not mint new ones. +- `arx5`: ARX 4.5 — arx4's context mixer on arx2's tuple pipeline, with every wire scored by honest serialized transport length. Compact tag `f`, same prior-id prefix as arx4 (`m`, `c`, `j`, `s`, or `n`). Recognize and open `#f` links; do not hand-roll them. Reproducing the wire needs the exact frozen mixer plus `https://agent-render.com/arx4-priors.json`, so an agent encoding on its own should stop at `arx2` (chat-safe ASCII) and let the app or library emit arx5. - packed wire mode (`p: 1`) may be used automatically to shorten transport keys Prefer: -1. shortest valid fragment for the target surface -2. codec priority `arx3 -> arx2 -> arx -> deflate -> lz -> plain` for links you encode yourself; the app itself tries `arx4` first +1. shortest valid fragment for the target surface, measured by serialized transport length (not visible Unicode count) +2. codec priority `arx2 -> arx -> deflate -> lz -> plain` for links you encode yourself; the app itself tries `arx5` first 3. packed wire mode when available +4. never emit baseBMP/base1k Unicode wires for Discord, WhatsApp, or any markdown-link destination ## Envelope shape @@ -199,8 +202,9 @@ https://agent-render.com/#l (lz) https://agent-render.com/#d (deflate) https://agent-render.com/#a (arx) https://agent-render.com/#b (arx2) -https://agent-render.com/#c (arx3) -https://agent-render.com/#e (arx4) +https://agent-render.com/#c (arx3, deprecated emit) +https://agent-render.com/#e (arx4, deprecated emit) +https://agent-render.com/#f (arx5) ``` For `plain`: @@ -244,7 +248,7 @@ To use the dictionary for local `arx` encoding: - Base76 uses 77 ASCII fragment-safe characters. ~6.27 bits/char 5. Prepend the tag `a` to form the fragment (the compact tag does not carry a dictionary version — it implies the current dictionary, so always substitute using the build's current dictionary) -The dictionary includes JSON envelope boilerplate patterns, JSON-escaped Markdown syntax, and programming-language patterns that are already present in the shipped corpus. The viewer tries the pre-compressed dictionary first on default ARX/ARX2/ARX3/ARX4 encode or decode paths, falls back to the JSON file, and falls back again to its built-in table if external fetches fail. +The dictionary includes JSON envelope boilerplate patterns, JSON-escaped Markdown syntax, and programming-language patterns that are already present in the shipped corpus. The viewer tries the pre-compressed dictionary first on default ARX-family encode or decode paths, falls back to the JSON file, and falls back again to its built-in table if external fetches fail. If the dictionary fetch fails, fall back to `deflate` codec. @@ -265,9 +269,9 @@ Then apply substitutions in this order: 4. Try baseBMP, base1k, base64url, and base76; choose the shortest transport representation 5. Prepend the tag `b` (the compact tag does not carry a dictionary version — it implies the current shared arx dictionary and arx2 overlay) -For `arx3`, use the same tuple, substitution, and brotli bytes as arx2, then try the same four wire shapes, but measure the baseBMP wire by decoded visible character length (rather than conservative transport length) and pick the shortest candidate — so the dense baseBMP wire can win on Unicode-preserving surfaces. Prepend the tag `c` (the compact tag does not carry a dictionary version — it implies the current shared arx dictionary and arx2 overlay). Do not invent a new dictionary entry unless it is backed by corpus evidence and improves the benchmark gate. +Do not encode `arx3` or `arx4`. Those tags remain readable so already-shared links open; their visible-length Unicode wires break on Discord and WhatsApp. -For `arx4`, there is no hand-rollable recipe: the payload is arithmetic-coded against a context-mixing model primed on a corpus that must match the encoder bit for bit, so encode arx4 only through the app or `encodeEnvelopeAsync`. Read the tag `e` and the prior id that follows it when parsing a link someone else produced. +For `arx5`, there is no hand-rollable recipe: the payload is arithmetic-coded against a context-mixing model primed on a corpus that must match the encoder bit for bit, so encode arx5 only through the app or `encodeEnvelopeAsync`. Read the tag `f` and the prior id that follows it when parsing a link someone else produced. When encoding yourself, stop at `arx2` with a transport-scored ASCII wire (usually base64url). ## Practical limits @@ -281,7 +285,7 @@ Before sharing on Discord, check `markdownLinkLength` or `discordMarkdownLinkWar When generating links programmatically via `createGeneratedArtifactLink` / `createGeneratedArtifactLinkAsync`, send `markdownLink` verbatim and inspect `discordMarkdownLinkWarning`. When it is non-null, surface that warning to the caller and split the payload before sharing on Discord. If a link is getting too large: -1. try `arx3` first for trusted Unicode-preserving surfaces; otherwise try `arx2`, then `arx`, then `deflate`, then `lz`, then `plain` +1. try `arx2` first (chat-safe ASCII), then `arx`, then `deflate`, then `lz`, then `plain`. Let the app emit `arx5` when you can use `encodeEnvelopeAsync` 2. allow packed wire mode 3. trim unnecessary prose or metadata 4. prefer a focused artifact over a bloated one @@ -291,7 +295,7 @@ If a link is getting too large: When the caller provides a strict budget (for example 1,500 chars): -1. encode using all available candidates (`arx3/arx2/arx/deflate/lz/plain`, packed and non-packed where applicable) +1. encode using all available live candidates (`arx5/arx2/arx/deflate/lz/plain` when the library is available, otherwise `arx2/arx/deflate/lz/plain`, packed and non-packed where applicable) 2. choose the shortest fragment that is within budget 3. if no candidate fits, return the shortest fragment plus a clear budget failure explanation diff --git a/src/app/globals.css b/src/app/globals.css index 9ccd331..55bcc2c 100644 --- a/src/app/globals.css +++ b/src/app/globals.css @@ -978,6 +978,10 @@ select { outline-offset: 2px; } +.artifact-action.is-deprecated { + color: var(--text-muted); +} + .artifact-action.is-primary { border-color: transparent; background: linear-gradient(135deg, var(--accent), var(--accent-strong)); diff --git a/src/components/home/link-creator.tsx b/src/components/home/link-creator.tsx index 8c3788f..7deed0f 100644 --- a/src/components/home/link-creator.tsx +++ b/src/components/home/link-creator.tsx @@ -9,7 +9,7 @@ import type { GeneratedArtifactLink, LinkCreatorDraft, } from "@/lib/payload/link-creator"; -import { artifactKinds, codecs, type ArtifactKind } from "@/lib/payload/schema"; +import { artifactKinds, codecPickerLabel, codecs, isDeprecatedEmitCodec, type ArtifactKind } from "@/lib/payload/schema"; import { cn } from "@/lib/utils"; type LinkCreatorProps = { @@ -341,11 +341,17 @@ export function LinkCreator({ onPreviewHash }: LinkCreatorProps) { className={cn( "artifact-action", (draft.codec ?? "auto") === option && "is-primary", + isDeprecatedEmitCodec(option) && "is-deprecated", )} aria-pressed={(draft.codec ?? "auto") === option} + title={ + isDeprecatedEmitCodec(option) + ? "Deprecated: Discord and WhatsApp detonate these Unicode wires. Use auto or arx5." + : undefined + } onClick={() => updateDraft("codec", option)} > - {option} + {codecPickerLabel(option)} ))} diff --git a/src/components/home/sample-link-data.ts b/src/components/home/sample-link-data.ts index 61b63c4..6c12644 100644 --- a/src/components/home/sample-link-data.ts +++ b/src/components/home/sample-link-data.ts @@ -9,8 +9,8 @@ export type SampleLinkCard = { description?: string; }; -// The ARX showcase uses a precomputed async ARX3 fragment so the static homepage -// can advertise the real compact transport without loading Brotli on first paint. +// The ARX showcase uses a precomputed async ARX5 fragment so the static homepage +// can advertise the real compact transport without loading the mixer on first paint. export const sampleLinkCards = [ { title: "Maintainer kickoff", @@ -42,11 +42,11 @@ export const sampleLinkCards = [ }, { title: "arx showcase", - hash: "#c￰¡ທ况㮡猋䘔噑ⶰ瓛ସ妔쓤ㆉ줾岱ꉩ鴍Ⅻ䀇煃쨏鯕ꐦ뚣䨝᭺荕쐖꥚蕲홀抣줶⧂恨㞜굚昒筞嘖곔㊨Ꞡ샱چ踉䘄⯦侽Ҕ柀땗졜鄆劖ᛧ썹ⅶ쑚枣⢢栣床둴偯ł♉깑榉⡦簚ꔁ⢼魶傽캖ừঊﺺ潑筆ڷ㦟瓆ⷔ鍋您擌붨ꋀ䒽蒐훉ϫ찟ﯓ꿂␒駄꜂허Ƥሎ턛鏞믥ڻ꾪㕺ﰢ㏑碖胆숩䕏펦띆⼈鄇䯸讓䪜ᙵ⭺Ṡ鱸癰ͱ쑧㏧ⴳꥁ寀䓗乌仰鿢姝䧌븴䰮⻉䈉䚦왰ɝ紎㍷폺䘶猕哃ࡸꉩ冟坩휢㘙뤔魼砿㩧䒗篐ඇ茗念䢯㖅艧뷗ﵥ敧覍ൿ匒옞閜福탇䈥쬨曪㠓櫚끫㯞祘멡胅㙤⪸騛尰譬拹չ淥ꆃ쁹뭻さ痭憕莼䏬赚櫣ꂛ⨎⌙歇釓Ố戵穠줽閆㚠틋옕鼜훉ᇢꔰ眾㻨搓욱⦈鋂ώ뷐史쨳抃萐죰雇쐷㳻䜖ᕝ∍⃇墫颱ẑ恘歬â逵ᖒᾁ╷ﵨȹ牎㈣উ貂奼쉞茸箋堵ꇏ壊ᡧẄ㴬☿ᮑ孪洮怒躡㹻鸷務Ӎ蒺筒간袨塷⯀⳸얁䎤틣拻Ṭ㪙㩷雋㾭ȡ彰뛠謺㋈嶭㷪ઠꂈڶ넪ﴧ咝宆雗䩍ᛧ༤馶姗爼污晶涙灭ᄴ鲄⾁缛ꚰ䆷쎼䈒홡躮ᄳ絡䦾ｐଛ취뀓糳觸젳뫜灆ɳ틟᥼趞䠇☔穮䮜ﭑ崒쇏盾휛﬏戡镵䠄ᵫ闦㯦悑䗹榃슌钾䖋彷㺶薢⧷圄餒옹⬌㦑Ê蒻䉽뮹ݿ솰駫還衅픠⨣㆒䮲倎䯧묗ﻧኾ䵣ꈬ旸ﾆ㭌彚ඤጥ㑯ꘇᏘ椛ꅺ菅ᶫ퐢营탼佉葃닌࡭栫ﳣ⭱숼쯋겢匘割䐡펒喀肨蕢뤇帲쒿嚇क़ꭏஓ䱊蛸螼ዚ領쁎鮴蝰﷮쐊蹱ꥬ䜱䖂﬿쫫訜ሁⓉ䦰ꤊ嶱뿝킷ᯑ芊㨏갾ᛇ茭홷텆酉觾쯩裁߿䶱骯놙糨랰뱋∺쳝⓿掲䳌렊䞡漂嚩夆⹎㍁㞢ᔃ傥鮟땢♯娹リ鹛垣క씥㞸猏鄷䰼聎瞿僞㩡䬄滪䯌稊ゅꆸ憸融픗ⵘ㽆찶憎ᕷ캿櫄쿄몰屎こ攎㑕ꞏ恋趤༠呥લ໾冎鋅닃䰖錣뱋䥏蓢ᛌ她좬増鼿㆑⩅骉혝䦧忥餒팤旎섢Ђ曓ㅦ贮ݠ邆ટ훔䝫湂㊍ߊ罸铟ဖ榽軗ﯞǍ툚㴼彞带꽻鞡薵齥曩⤙꩟럾᪜计墵쾈쥖鷁ﰮ叝앯䧹郖篷칵덒晶癤㱫螃핊ꛩ颔⛼᢯결螲䭛콹䏓䅐뺍릏㖬儸⍭쉥ậ㠟种凟봊篳≚纛⳹ẜ饔續掚䱁㔷籕㐥蒾숳嫲ᗁˊ翇䔶뎮ᗞ现ꅨ鼃㼪ᖫ☗皧ꬍ틈牪ﮟ㱎䨃㫭럋ఖᶠ僂㝰⸨ꚧ냳녡마崚꾨産૫ꪞ㞣鑳ꟺ૧兟㵂ť䮫莲閗䯃⏭뿋阡൧仙㕴乜誴㥅滠켼¬즛펃䁦鬜⣻똢ꗕ躇엑渌鿍攅䛳좥ᠧ黇䪍㠠꒳䳚囮⥿긓릒ﴲহ㷼現⪱돌庙૵掷ﮠ⺬㏴꩘஄ꧪ〨㧓ᢝ鲛溻膳麅殡墂޸밝∇꠨儱뽑庶᯴閔뉌᫬迢䚼娉暫᥾뗶柈ٻ㩶黔㖝ꟓ럭癭ṧ㓈岒墼䨞鈖⼦⿖礇ᾉʑ안擴鶱♾轑窔㠆ꎋ篸≷폪뽵Ұ烻멝䡹⛙픮⻐웃殡ᮄꤒᡞ啾屿⧻댳ḧ闣釀É㢞뷟丸贑ỻ麎꣉넽윞ମ섈좃塼힛贫蝏㭺␓첫䬱厼園⼟튐苽驪힪Ň퐐︐ꗒﱕꦓ癳吹켘훌桡鍎뉝顚练艫ង灤覮浏袟唂繥珆餻ﴹ㕸泵ⱦ姐迳鉡츢撸⢟䱁䠫핏䢢즜聵矸䊯櫋䑸袸솧ꪃ婴埦뽵ᓆ姄ऑ垿꾜刈ᛇ䯔뭲紫Ӥ襡禺㢮恐狚ⷉ᣺툊桾⒥쳱冰厃▋㣿ܛ셹뚞ﷲ栾蠋뜺㕣ꉣᛋ封䣸矴耀咆┎喯ꇗ喏ꎸ棾⦉䵐Ӣ怨골﫿ѵ姸滮횳颮匶怪놟鳭놫柫贊쎖ℕ㼰ꖂ䖽룲ﴬ㍱캠ﱫ鵃쾒Ẃ觋跻Ä닑婢锟Њꀯ׼䒨ꦅ隣㮇䋓ࢀᣥ㭦霐䴣⺪甘﬽ꈞ㭩㇘灥夀鹷턨꡹掾俩抂藋濯㹴糞쑔鉶⣷ḽ꿖风繰﷗唥蘏㔉崹⦜嘃៯ᒖ鹿쇉핗晄喝獡뎉ꡪ諦峫㔗觠␷홳翭㟍೼ⰹ嫶ﳺ쥞䋠䕾텄눎娳”Є䀾傗겱矡ꇳ튏㻁⳸粱ﯓ씂떴⥪ꔝ㧊癶뙏뾤碓탨㣪쐫䖯슸轝ت跃熲빗泮㤋ꄰꁋⓚ㽾焁畂팵恥⤯謎䋕蟬磺髏ⅽ䌊徳ㄆ广⡇筨顯㲽珃鈆餋⌁갖悞՚켶␄녌給鿦씨ꂋ鯢녤襡⵷갑䠱迫ዕ䯵◮䧛ꐠ䣯큪䯠纫⍍孓䘪㥘墳ᢡ돆ꅘ㜇僤ふ夺풂몮旽ኈ遷挱팊₠搘铙鷜激翬噊얆輹跞丰橚ᓧ豃Ễꤣ䢙ந៶⦖睾鲴ﺟࡁ꽢孋僩즥ꧧ첟⸁猨崂ᖪ׳䭿ᓕ腔䉤挧쫗멚茳ꢡ龀ꈱᆖ挩䲖ᆪ潴聪륎痧振蕠఍ᾥ氶뾫茹⩐艬⸑前ﮏ鞁춍࿤硻။퓝䦒䭏䫆ꆁ㟶ꫝ㏀䫋샚匾쏄鰊ـ술ꋂ楲삈摇艅횓盧꒦♕ꂝ렚樭ˊჼ寧↮ꞻ蔃᪓⤎턑⥗ㇳຉ摶⟍ō䊦莝პ鼅颶淞餢偠뽃灻襸첁⴦龂赸頉㓲닄곝更턍ⲋ⸽茿獥엻", - fragmentLength: 1465, + hash: "#fmB.9UT8WKbq5PLRGIWwBwP4U7ZcrIO7_3bszINfzuMZQ4SJkf8dvoRB-yANFGVV3H6Oc2jiivp9OXi4J9oyCIkNZV0Uj8hT-0yj23KiwYndmDl3Ztf6QJr7jTuhTYwXO-4yE2qvrUME7-SSGv91Y5Dlll6pi-GyItQsLlGGu8Ya4gdNemut3OcklputjrqZE-qp7sGG3ySYOdB0J8QQ8ujF1SP4B7stEQuGCCurhg3WZfD33U1AAdwe4UAN8PbZHTr2VJ_IWthl_XV28Rhha9oJxKPnQHmx3A7076kzLDmbhGpLQeUEn0wLMEjJzjr62nc2T1OSvkgYJqtCn_CJtecbiJDCrGWD2emhHkW6QqeItKkQKxMkBAIRvvRJEdyY26kBmP_ph9ecPQRo-hSQ-gVGSKpzeTRxYAHJoexpKoJD-1CdEWbzgudE4FETren53QNJqyzubY69FgdYQ8dPj6eo98fQWQR6XrcjuWhFGPSL9PlRUHFDjhPDe1TMmyc77yMJa8TV8TBaftZGNJUtdA9wMn4Z8q_5bCEN5hzUcF4cwTTmMSNe8traBkQY3fl-5B2_Ybr5aVOO-vCLdCgyBYffxZ9lOQfN6VVPQdRVFE6aeFTUfEIqXO1w-UrJPZKgYjpF6V21K8dKGi9HQWEGJtZk9Ll5hyTdREH4ZNydZyCQSYGqXV181hioc6O0AubKDHtHSleWmZdH-GocYgbSMHHfb9zNK1tDVAU_QuJH4NKZF35J8-lVBtfn1kRB9u99ccrHjhP_fyGeeJNP5EYwDS2-npmPZ9fbhq_eIhdbW1p6DZw-3looGe1Khbzn2rr0Kw3hFmu_jKBaIv-ao5_3CAa_QB9yKky3rOkR2GbuHutSt3-QjwdEf4qRwLrN7mVZovLi_x8xHzhg7hk5wUkxY4ZxawWuj4LyVduC9WsCsNFZd8wWVz4uWFB-EktKoJf9brRgll3BQslaGkKhNCFdoa0G81pupJFo7HaMDCtw6LRNq_PQI6DbQeNoUoAOBWMTkWk3vhO6WxwSzJNNYATDUhXMa341swHoi8xUoSTBLZFlYsxmqNmxDhKnXZUt_pFvRID0vdNlsHhlwzWsdFqFuhiEztM__PN_fp_k4RNJCvVlsK150_NxchVy8SRC5WDckMtjZIY2IS82eDagkzO4xj3APJ-dXxNBqajKlF9kKfKC08gqpiI5AELedXjAtg1KlbdSMDoGLJLepalRd6DzxaHlCOE6i_5hw2O1GnXVP4i8GT8IvGaJlTIcDhrn9XMBucxXmYIk_DgMPHccOvKO9bsE8z1dSzhyEjkW2vnSejrVzkq7y-aBKHefmtrqmtcoZsqDeNIyoeBNmbRBuKTcVk-4HSNWk1Zsezihol3zDjGE3rcq85FJIv3bi5jz1J7BLlZB8bV7VNLq3lVydDytz8HSTFFhZFEYG1HkxuYabAT26L10ih6VWk8j_8Z-Oz_zx2-nZXGpCj-yemjJAExwOS6X0J7OpIf4GqOBJXrWFoF_VhJWKqoJJo4-RqwzQRT_XP9Bm4KmCZo5nFe4oIRWMRlnB1l57JS6Cd8wO-lBaGP0OmwAtVh7U-KnYTEQ2Vk-lYnnX1XHGkZUm7PhbPO2ztXmHsH0mLCb0xJPFBLO5klTRXuIPDnpalfhHtXDTBWExOmN6hwfoVzWe9mQ7MBVaPeKiTmz_FgaOaUMlPXSqkSxy_J2epc0WFFqL7QNYV0QVHAoQDXgWy0s2_vVCyBcbd-p9qFhu1ti9MUOmI6QJXE2iILyBGDM02mfjxF3v6MxCu-i05_xcM4zl7arBvSKqcS_sZfX1WCJzFGiOZTd-VC7FmNTFY2JlF9LSE2aeJCT30zlO7nkQWnP6-So3cZJuqhOIKvjX0hRo-T_zSyf4cOBWF8EmP4FzhUCVO2yvO_BBF-YdROgg6bqrsDc1v_3_LTyADlVG6Sa60ktxmMvX3Vp7rMTenNT3r4qvNEr2P1XlFGgOXvaH1s7vRa9IvMwNveXElumc4TL9DwE4TGfL64Dwe-4NTYulwu_WLGlL0cts2xTQed0g78U9JzduEHec6pKsT93dYESigIvw9HlgcB3-A-u-oLK0h2sxWYlsG6vrlmqn4kapFtA5MvDRWHmyUzXU2J2To4iMXK-N1qh5JeSv2ZMpg4ixEzfrsoL0dyUCKgCLvg18LTnLstRKaxdl37cvjDAhH9LipCDp_i27G8r68s4NJ2KRcvyXOuJFyppyama2CDrr-fhD5YsBxwbZDSQeDmnSeaGIiq0dZiiq-44l3oFDvr0NHctR5eKuY1nWZk5zLDPGFhF6RP4N-yw6lXK8sc39VcSvblIot8NvToktQ6UyKytGn9Z5Xj4tICd5lnjYMKJoplbi4tqewuB5ON0rHpkOLaXFh4S_X8QTeW_jUw4kL9aFf721291-_HwROAXdCA_q_tx4i2oVrpATFRMHZ9FezzNCu2ax9BAKwpzPRAa6gvMrLnsfiE8v5_jAXJAaDWSw4iUNo2ddEKBN1FMFS6QsCs8jWTh_NB9ZGr9h8jSBV1zjOtv_EXs8AYreCdzZ5iWe3xFCCXnIudqNrDi5odC9e3Hp67OIp2dD6eCs2l2eF1NX4xojLV6DoZws3ipPi9Nf1TWJtDxZVQSBlgT6Awqlsn3T2NDNI22F4pJmJf3kFgvH5JP1KTOZd53ICfrZO-jLrfILfSgG-q9fsikEncB8fIIfGbzmtYLsrT1YLa6F6Pz43oDF0Pgd7VqagrMF1NoSL--l8g8iuVgHZSc96ruiknBWltb8exiVVRXxdhOQsiM_Zuu9cxMC_fLnKG3sb_LWV-fOX8XTIP3QcVOwG-m3agqnDYu59Jl7ZVszvGVAIjjZjOxh5lmMOGzOU2ksKyDNJ4o2eRTeNasvcrs0FHDLUUtbme5PaiScnPzi5G2Q7OqWyIk7VI2YMR778rMBkjDC3Ej_esbysurY_lxd8ZhF7Ohl1ssKI_UgkgNyYF_JCvSO-LmuxQISNO8llMrnnVasRiOLxxy-3DCKDFgi6ahSBOlpIhFnJMEOmTpBVjeHwTQ6lfyGUDQp5tMga9Y6NS3v_ewCr-W6VqmhO4TQm4mitjI38ew4uUZvaCRatjjE-qeG0sxz-xMW9aVeJaF7IBPiKIFLuNK7hwNAEKUWCK8rRiVWNs4u2F8jfpdZ63PfGJ0JEcGOZfgmpRTK4yev2t5EqZTL1M6g98jZZLFXc7arDgZ_5YWne9jIpY3vkwqOtonNGpX-5iYskjlbfqVxmhdAlMxjXpLnhyiFTMUxu0R-2j_Bu8jDGnhtsK9cqjp3fEgHqIkUE1yHwCwxh7gcEEt_8ipII-3VzO7fzhbNl-coHxcHKCyoXISOiVcyrRRuupXC6g36CKIM6Kdkxr_uZ-QMuitNng0HpxqWiI7iXrmbQJxqx2GGkDmkdYOFKVzraMYjDce3cC4eSTn4IS20dQ497gmX0z-tnpsy9fS4y7Ui2ICzVRD0JHKdrFbaP6pntTUHXdsO72uQYwiLiypYKYlcjXM4whSni3SB-AVV-vit3RC2Zi38T4cgwWdptGmOPj9z1irx6SDphOH4HrIN53cQszq7HMdsVYpIGEwLHy0mGWDOdKDZmb9NhHJqjK-uOeHNig7h7kqvnme5F2iaThqh4A5J4ZCXfKQpGO-IV8LyYA96UhUI-t1spn8bLcPFHhyvUqhmqYnAgA", + fragmentLength: 3711, kind: "json", artifactCount: 5, - description: "Tuple compression, Brotli, and visible-length Unicode encoding compress 5 rich artifacts into a single URL fragment.", + description: "Tuple compression and the context mixer, scored by honest transport length, compress 5 rich artifacts into a chat-safe URL fragment.", }, { title: "Malformed manifest", diff --git a/src/components/viewer/artifact-editor.tsx b/src/components/viewer/artifact-editor.tsx index a2becaf..c9bdb16 100644 --- a/src/components/viewer/artifact-editor.tsx +++ b/src/components/viewer/artifact-editor.tsx @@ -12,7 +12,9 @@ import { type GeneratedArtifactLink, } from "@/lib/payload/link-creator"; import { + codecPickerLabel, codecs, + isDeprecatedEmitCodec, type ArtifactKind, type ArtifactPayload, type PayloadEnvelope, @@ -373,11 +375,17 @@ export function ArtifactEditor({ className={cn( "artifact-action", (draft.codec ?? "auto") === option && "is-primary", + isDeprecatedEmitCodec(option) && "is-deprecated", )} aria-pressed={(draft.codec ?? "auto") === option} + title={ + isDeprecatedEmitCodec(option) + ? "Deprecated: Discord and WhatsApp detonate these Unicode wires. Use auto or arx5." + : undefined + } onClick={() => updateDraft("codec", option)} > - {option} + {codecPickerLabel(option)} ))} diff --git a/src/lib/payload/arx-codec.ts b/src/lib/payload/arx-codec.ts index 85e7ae4..3aa3ef1 100644 --- a/src/lib/payload/arx-codec.ts +++ b/src/lib/payload/arx-codec.ts @@ -604,7 +604,7 @@ function decodeArx2ArtifactTuple(value: unknown): ArtifactPayload { } } -function envelopeFromArxTuple(value: unknown, codec: Extract): PayloadEnvelope { +function envelopeFromArxTuple(value: unknown, codec: Extract): PayloadEnvelope { if (!Array.isArray(value)) { throw new Error("Invalid arx2 envelope tuple."); } @@ -1381,7 +1381,7 @@ export async function arxCompressBase64url(json: string): Promise { /** * The tuple-envelope pipeline's pre-entropy stage: envelope → compact tuple JSON → arx2 overlay * substitution → shared dictionary substitution. Every tuple codec entropy-codes exactly these - * bytes (Brotli for arx2/arx3, the context mixer for arx4), so they all share this one derivation. + * bytes (Brotli for arx2/arx3, the context mixer for arx4/arx5), so they all share this one derivation. */ export function substituteArxTupleText(envelope: PayloadEnvelope): string { // The arx2/arx3 overlay repurposes 0x7F (DEL) as a single-byte substitution @@ -1432,7 +1432,7 @@ export async function arxDecompress(encoded: string): Promise { */ export function envelopeFromSubstitutedArxTupleText( substituted: string, - codec: "arx2" | "arx3" | "arx4", + codec: "arx2" | "arx3" | "arx4" | "arx5", ): PayloadEnvelope { const v1Decoded = dictDecode(substituted); assertWithinExpansionBudget(v1Decoded); diff --git a/src/lib/payload/arx4-codec.ts b/src/lib/payload/arx4-codec.ts index 625cf88..8a35a07 100644 --- a/src/lib/payload/arx4-codec.ts +++ b/src/lib/payload/arx4-codec.ts @@ -2,8 +2,9 @@ * arx4 codec: the context-mixing entropy stage for the arx tuple pipeline. * * Pipeline: envelope → compact tuple JSON → overlay + shared dictionary substitution → - * context-mixing arithmetic coder → binary-to-text wire encoding. arx4 replaces only arx3's Brotli - * stage; the tuple and substitution stages are the same functions arx2/arx3 call. + * context-mixing arithmetic coder → binary-to-text wire encoding. arx4/arx5 replace only Brotli + * in the arx2 tuple pipeline; the tuple and substitution stages are the same functions arx2/arx3 call. + * arx5 is the emitted mixer codec (honest transport-length scoring). arx4 stays decodable. * * Fragment shape: ``. The prior id names the priming corpus the coder ran * before the payload, because a decoder has to reproduce the encoder's model state exactly. The @@ -1009,3 +1010,18 @@ export function arx4DecompressEnvelope(encoded: string): PayloadEnvelope { )); return envelopeFromSubstitutedArxTupleText(substituted, "arx4"); } + +/** + * Compresses a payload envelope with the arx5 pipeline. Same mixer bytes as {@link arx4CompressEnvelope}; + * the protocol difference is the compact `f` tag and honest transport-length wire selection. + */ +export function arx5CompressEnvelope(envelope: PayloadEnvelope, priorId?: Arx4PriorId): ArxWirePayloads { + return arx4CompressEnvelope(envelope, priorId); +} + +/** + * Decompresses an arx5 payload (prior id char + wire payload) and rebuilds the envelope stamped `arx5`. + */ +export function arx5DecompressEnvelope(encoded: string): PayloadEnvelope { + return { ...arx4DecompressEnvelope(encoded), codec: "arx5" }; +} diff --git a/src/lib/payload/examples.ts b/src/lib/payload/examples.ts index d64e028..47e9f40 100644 --- a/src/lib/payload/examples.ts +++ b/src/lib/payload/examples.ts @@ -85,7 +85,7 @@ export const sampleEnvelopes: PayloadEnvelope[] = [ title: "v3.0 release notes", filename: "RELEASE.md", content: - "# agent-render v3.0 — arx3 visible URL mode\n\n> **Everything you see here is encoded in the URL fragment.** No server ever receives this content.\n\n## What is arx3?\n\narx3 builds on arx2 instead of guessing a new dictionary. It keeps the measured tuple-envelope, overlay substitution, shared arx dictionary, and Brotli pipeline, then optimizes the final wire choice for decoded visible URL characters.\n\n1. **Tuple envelope transport** — artifact metadata is serialized without repeated JSON object keys\n2. **Overlay + shared dictionary substitution** — common tuple, code, markdown, and diff patterns are replaced with short byte sequences\n3. **Brotli compression** (quality 11) — the same compressed bytes as arx2\n4. **High-density binary-to-text encoding** — arx3 can choose baseBMP by visible character count:\n\n| Encoding | Character set | Bits per character | Typical use |\n| --- | --- | --- | --- |\n| base76 | ASCII printable | 6.27 | Maximum compatibility |\n| base64url | A-Z, a-z, 0-9, - and _ | 6.00 | Chat-safe ASCII |\n| base1k | U+00A1–U+07FF | 10.79 | Balanced size / compat |\n| baseBMP | U+00A1–U+FFEF | 15.92 | Smallest visible char count |\n\n## Compression benchmark gate\n\n| Codec | Corpus bytes | Visible chars | Result |\n| --- | ---: | ---: | --- |\n| arx | 5,544 | 7,585 | baseline |\n| arx2 | 5,410 | 7,416 | 2.42% fewer bytes |\n| arx3 | 5,410 | 2,931 | 60.48% fewer visible chars vs arx2 |\n\n## This bundle demonstrates\n\n- [x] Rich markdown with tables, task lists, code fences, and blockquotes\n- [x] Syntax-highlighted source code with line numbers\n- [x] Multi-file git diffs with split and unified views\n- [x] Tabular CSV data with sortable columns\n- [x] Structured JSON with collapsible tree navigation\n\n### How it works\n\n```\ntuple payload → overlay dictionary → arx dictionary → Brotli → baseBMP → visible URL fragment\n ↑\n servers never see this part\n```\n\nAll five artifact kinds, all encoded in a single URL. Zero server-side storage. Full client-side rendering.", + "# agent-render — arx5 honest transport\n\n> **Everything you see here is encoded in the URL fragment.** No server ever receives this content.\n\n## What is arx5?\n\narx5 is arx2's tuple pipeline with arx4's context mixer, scored by serialized transport length instead of visible Unicode count. Auto-emit uses it. arx3 and arx4 stay decodable and explicitly encodable, but they are not selected automatically because Discord and WhatsApp detonate those Unicode wires.\n\n1. **Tuple envelope transport** — artifact metadata is serialized without repeated JSON object keys\n2. **Overlay + shared dictionary substitution** — common tuple, code, markdown, and diff patterns are replaced with short byte sequences\n3. **Context mixer** — the deterministic entropy coder from arx4, primed on per-kind priors\n4. **Chat-safe binary-to-text encoding** — every wire is scored by percent-escaped transport length, so base64url usually wins:\n\n| Encoding | Character set | Bits per character | Typical use |\n| --- | --- | --- | --- |\n| base76 | ASCII printable | 6.27 | Maximum compatibility |\n| base64url | A-Z, a-z, 0-9, - and _ | 6.00 | Chat-safe ASCII |\n| base1k | U+00A1–U+07FF | 10.79 | Balanced size / compat |\n| baseBMP | U+00A1–U+FFEF | 15.92 | Smallest visible char count |\n\n## Compression benchmark gate\n\n| Codec | Corpus bytes | Visible chars | Result |\n| --- | ---: | ---: | --- |\n| arx | 5,544 | 7,585 | baseline |\n| arx2 | 5,410 | 7,416 | 2.42% fewer bytes |\n| arx3 | 5,410 | 2,931 | 60.48% fewer visible chars vs arx2 |\n\n## This bundle demonstrates\n\n- [x] Rich markdown with tables, task lists, code fences, and blockquotes\n- [x] Syntax-highlighted source code with line numbers\n- [x] Multi-file git diffs with split and unified views\n- [x] Tabular CSV data with sortable columns\n- [x] Structured JSON with collapsible tree navigation\n\n### How it works\n\n```\ntuple payload → overlay dictionary → arx dictionary → context mixer → transport-scored wire → chat-safe URL fragment\n ↑\n servers never see this part\n```\n\nAll five artifact kinds, all encoded in a single URL. Zero server-side storage. Full client-side rendering.", }, { id: "codec-src", @@ -94,7 +94,7 @@ export const sampleEnvelopes: PayloadEnvelope[] = [ filename: "arx-codec.ts", language: "typescript", content: - '/**\n * arx codec — Agent Render eXtreme compression\n *\n * Pipeline: tuple JSON → overlay/shared substitutions → Brotli → base76/base64url/base1k/baseBMP\n */\n\nimport type { ArxDictionary } from "./schema";\n\nconst HEADER_ARX76 = 0x01; // ASCII-safe base76\nconst HEADER_ARX1K = 0x02; // Unicode base1k (U+00A1–U+07FF)\nconst HEADER_ARXBMP = 0x03; // Full BMP base (U+00A1–U+FFEF)\n\nexport interface ArxCompressResult {\n encoded: string;\n encoding: "base76" | "base64url" | "base1k" | "baseBMP";\n compressedBytes: number;\n ratio: number;\n}\n\nexport async function arxCompress(\n json: string,\n dict: ArxDictionary,\n): Promise {\n // Stage 1: Dictionary substitution\n let substituted = json;\n for (const [pattern, replacement] of dict.entries) {\n substituted = substituted.replaceAll(pattern, replacement);\n }\n\n // Stage 2: Brotli compression (quality 11)\n const input = new TextEncoder().encode(substituted);\n const compressed = await brotliCompress(input, { quality: 11 });\n\n // Stage 3: Try every wire encoding, then pick by the target length metric.\n const candidates: ArxCompressResult[] = [\n { encoded: toBase76(compressed), encoding: "base76", compressedBytes: compressed.length, ratio: 0 },\n { encoded: toBase64url(compressed), encoding: "base64url", compressedBytes: compressed.length, ratio: 0 },\n { encoded: toBase1k(compressed), encoding: "base1k", compressedBytes: compressed.length, ratio: 0 },\n { encoded: toBaseBMP(compressed), encoding: "baseBMP", compressedBytes: compressed.length, ratio: 0 },\n ];\n\n candidates.sort((a, b) => visibleLength(a.encoded) - visibleLength(b.encoded));\n const best = candidates[0];\n best.ratio = json.length / best.encoded.length;\n return best;\n}\n\nexport async function arxDecompress(\n fragment: string,\n dict: ArxDictionary,\n): Promise {\n const header = fragment.charCodeAt(0);\n\n // Detect encoding from the wire marker.\n const compressed = header === HEADER_ARXBMP ? fromBaseBMP(fragment)\n : header === HEADER_ARX1K ? fromBase1k(fragment)\n : isBase64url(fragment) ? fromBase64url(fragment)\n : fromBase76(fragment);\n\n // Decompress Brotli\n const decompressed = await brotliDecompress(compressed);\n let json = new TextDecoder().decode(decompressed);\n\n // Reverse dictionary substitution\n for (const [pattern, replacement] of [...dict.entries].reverse()) {\n json = json.replaceAll(replacement, pattern);\n }\n\n return json;\n}', + '/**\n * arx codec — Agent Render eXtreme compression\n *\n * Pipeline: tuple JSON → overlay/shared substitutions → Brotli → base76/base64url/base1k/baseBMP\n */\n\nimport type { ArxDictionary } from "./schema";\n\nconst HEADER_ARX76 = 0x01; // ASCII-safe base76\nconst HEADER_ARX1K = 0x02; // Unicode base1k (U+00A1–U+07FF)\nconst HEADER_ARXBMP = 0x03; // Full BMP base (U+00A1–U+FFEF)\n\nexport interface ArxCompressResult {\n encoded: string;\n encoding: "base76" | "base64url" | "base1k" | "baseBMP";\n compressedBytes: number;\n ratio: number;\n}\n\nexport async function arxCompress(\n json: string,\n dict: ArxDictionary,\n): Promise {\n // Stage 1: Dictionary substitution\n let substituted = json;\n for (const [pattern, replacement] of dict.entries) {\n substituted = substituted.replaceAll(pattern, replacement);\n }\n\n // Stage 2: Brotli compression (quality 11)\n const input = new TextEncoder().encode(substituted);\n const compressed = await brotliCompress(input, { quality: 11 });\n\n // Stage 3: Try every wire encoding, then pick by the target length metric.\n const candidates: ArxCompressResult[] = [\n { encoded: toBase76(compressed), encoding: "base76", compressedBytes: compressed.length, ratio: 0 },\n { encoded: toBase64url(compressed), encoding: "base64url", compressedBytes: compressed.length, ratio: 0 },\n { encoded: toBase1k(compressed), encoding: "base1k", compressedBytes: compressed.length, ratio: 0 },\n { encoded: toBaseBMP(compressed), encoding: "baseBMP", compressedBytes: compressed.length, ratio: 0 },\n ];\n\n candidates.sort((a, b) => transportLength(a.encoded) - transportLength(b.encoded));\n const best = candidates[0];\n best.ratio = json.length / best.encoded.length;\n return best;\n}\n\nexport async function arxDecompress(\n fragment: string,\n dict: ArxDictionary,\n): Promise {\n const header = fragment.charCodeAt(0);\n\n // Detect encoding from the wire marker.\n const compressed = header === HEADER_ARXBMP ? fromBaseBMP(fragment)\n : header === HEADER_ARX1K ? fromBase1k(fragment)\n : isBase64url(fragment) ? fromBase64url(fragment)\n : fromBase76(fragment);\n\n // Decompress Brotli\n const decompressed = await brotliDecompress(compressed);\n let json = new TextDecoder().decode(decompressed);\n\n // Reverse dictionary substitution\n for (const [pattern, replacement] of [...dict.entries].reverse()) {\n json = json.replaceAll(replacement, pattern);\n }\n\n return json;\n}', }, { id: "migration-diff", @@ -102,7 +102,7 @@ export const sampleEnvelopes: PayloadEnvelope[] = [ title: "v1 to v3 migration", filename: "fragment.ts", patch: - "diff --git a/src/lib/payload/fragment.ts b/src/lib/payload/fragment.ts\nindex aaa1111..bbb2222 100644\n--- a/src/lib/payload/fragment.ts\n+++ b/src/lib/payload/fragment.ts\n@@ -1,18 +1,42 @@\n-import { deflateSync, inflateSync } from \"fflate\";\n-import { compressToEncodedURIComponent, decompressFromEncodedURIComponent } from \"lz-string\";\n+import { deflateSync, inflateSync } from \"fflate\";\n+import { compressToEncodedURIComponent, decompressFromEncodedURIComponent } from \"lz-string\";\n+import { arxCompress, arxDecompress } from \"./arx-codec\";\n+import { loadArxDictionary } from \"./arx-dictionary\";\n \n-const CODEC_PRIORITY = [\"deflate\", \"lz\", \"plain\"] as const;\n+const SYNC_CODECS = [\"deflate\", \"lz\", \"plain\"] as const;\n+const ASYNC_CODECS = [\"arx3\", \"arx2\", \"arx\", \"deflate\", \"lz\", \"plain\"] as const;\n \n export function encodeEnvelope(envelope: PayloadEnvelope): string {\n- const json = JSON.stringify(envelope);\n- const candidates = CODEC_PRIORITY.map((codec) => ({\n+ return encodeShortest(envelope, SYNC_CODECS);\n+}\n+\n+export async function encodeEnvelopeAsync(envelope: PayloadEnvelope): Promise {\n+ return encodeShortest(envelope, ASYNC_CODECS);\n+}\n+\n+function encodeShortest(envelope: PayloadEnvelope, codecs: readonly string[]): string {\n+ const json = JSON.stringify(envelope);\n+ const candidates = codecs.map((codec) => ({\n codec,\n- fragment: encodeWith(json, codec),\n+ fragment: encodeWith(json, codec),\n }));\n candidates.sort((a, b) => a.fragment.length - b.fragment.length);\n return candidates[0].fragment;\n }\n \n-export function decodeFragment(raw: string): PayloadEnvelope {\n+export async function decodeFragmentAsync(raw: string): Promise {\n+ const match = raw.match(/^agent-render=v1\\.(\\w+)\\.(.+)$/);\n+ if (!match) throw new Error(\"Invalid fragment format\");\n+ const [, codec, payload] = match;\n+\n+ if (codec.startsWith(\"arx\")) {\n+ const [dictVersion, ...rest] = payload.split(\".\");\n+ const dict = await loadArxDictionary(dictVersion);\n+ const json = await arxDecompress(rest.join(\".\"), dict);\n+ return JSON.parse(json);\n+ }\n+\n+ return decodeFragment(raw);\n+}\n+\n+export function decodeFragment(raw: string): PayloadEnvelope {\n const match = raw.match(/^agent-render=v1\\.(\\w+)\\.(.+)$/);\n if (!match) throw new Error(\"Invalid fragment format\");\n const [, codec, payload] = match;\n", + "diff --git a/src/lib/payload/fragment.ts b/src/lib/payload/fragment.ts\nindex aaa1111..bbb2222 100644\n--- a/src/lib/payload/fragment.ts\n+++ b/src/lib/payload/fragment.ts\n@@ -1,18 +1,42 @@\n-import { deflateSync, inflateSync } from \"fflate\";\n-import { compressToEncodedURIComponent, decompressFromEncodedURIComponent } from \"lz-string\";\n+import { deflateSync, inflateSync } from \"fflate\";\n+import { compressToEncodedURIComponent, decompressFromEncodedURIComponent } from \"lz-string\";\n+import { arxCompress, arxDecompress } from \"./arx-codec\";\n+import { loadArxDictionary } from \"./arx-dictionary\";\n \n-const CODEC_PRIORITY = [\"deflate\", \"lz\", \"plain\"] as const;\n+const SYNC_CODECS = [\"deflate\", \"lz\", \"plain\"] as const;\n+const ASYNC_CODECS = [\"arx5\", \"arx2\", \"arx\", \"deflate\", \"lz\", \"plain\"] as const;\n \n export function encodeEnvelope(envelope: PayloadEnvelope): string {\n- const json = JSON.stringify(envelope);\n- const candidates = CODEC_PRIORITY.map((codec) => ({\n+ return encodeShortest(envelope, SYNC_CODECS);\n+}\n+\n+export async function encodeEnvelopeAsync(envelope: PayloadEnvelope): Promise {\n+ return encodeShortest(envelope, ASYNC_CODECS);\n+}\n+\n+function encodeShortest(envelope: PayloadEnvelope, codecs: readonly string[]): string {\n+ const json = JSON.stringify(envelope);\n+ const candidates = codecs.map((codec) => ({\n codec,\n- fragment: encodeWith(json, codec),\n+ fragment: encodeWith(json, codec),\n }));\n candidates.sort((a, b) => a.fragment.length - b.fragment.length);\n return candidates[0].fragment;\n }\n \n-export function decodeFragment(raw: string): PayloadEnvelope {\n+export async function decodeFragmentAsync(raw: string): Promise {\n+ const match = raw.match(/^agent-render=v1\\.(\\w+)\\.(.+)$/);\n+ if (!match) throw new Error(\"Invalid fragment format\");\n+ const [, codec, payload] = match;\n+\n+ if (codec.startsWith(\"arx\")) {\n+ const [dictVersion, ...rest] = payload.split(\".\");\n+ const dict = await loadArxDictionary(dictVersion);\n+ const json = await arxDecompress(rest.join(\".\"), dict);\n+ return JSON.parse(json);\n+ }\n+\n+ return decodeFragment(raw);\n+}\n+\n+export function decodeFragment(raw: string): PayloadEnvelope {\n const match = raw.match(/^agent-render=v1\\.(\\w+)\\.(.+)$/);\n if (!match) throw new Error(\"Invalid fragment format\");\n const [, codec, payload] = match;\n", view: "unified", }, { @@ -119,7 +119,7 @@ export const sampleEnvelopes: PayloadEnvelope[] = [ title: "Artifact manifest", filename: "manifest.json", content: - '{\n "name": "agent-render",\n "version": "3.0.0",\n "description": "Fragment-powered artifact viewer with arx3 visible URL compression",\n "transport": {\n "method": "url-fragment",\n "prefix": "agent-render=v1",\n "codecs": ["plain", "lz", "deflate", "arx", "arx2", "arx3"],\n "preferred": "arx3",\n "maxVisibleFragmentChars": 8192,\n "maxDecodedChars": 200000\n },\n "arx3": {\n "pipeline": ["tuple-envelope", "overlay-substitution", "shared-dictionary-substitution", "brotli-q11", "visible-length-wire-selection"],\n "encodings": {\n "base76": { "charset": "ascii", "bitsPerChar": 6.27, "maxCompat": true },\n "base64url": { "charset": "ascii-url-safe", "bitsPerChar": 6.0 },\n "base1k": { "charset": "U+00A1-U+07FF", "bitsPerChar": 10.79 },\n "baseBMP": { "charset": "U+00A1-U+FFEF", "bitsPerChar": 15.92, "smallestVisible": true }\n },\n "dictionaries": {\n "shared": "/arx-dictionary.json",\n "overlay": "/arx2-dictionary.json"\n }\n },\n "artifacts": {\n "kinds": ["markdown", "code", "diff", "csv", "json"],\n "thisBundle": [\n { "id": "release-notes", "kind": "markdown", "title": "v3.0 release notes" },\n { "id": "codec-src", "kind": "code", "title": "arx-codec.ts (excerpt)" },\n { "id": "migration-diff", "kind": "diff", "title": "v1 to v3 migration" },\n { "id": "metrics", "kind": "csv", "title": "Bundle metrics" },\n { "id": "manifest", "kind": "json", "title": "Artifact manifest" }\n ]\n },\n "features": {\n "zeroRetention": true,\n "serverNeverSeesPayload": true,\n "clientSideOnly": true,\n "selfHostable": true,\n "openSource": true\n }\n}', + '{\n "name": "agent-render",\n "version": "3.0.0",\n "description": "Fragment-powered artifact viewer with arx5 honest transport compression",\n "transport": {\n "method": "url-fragment",\n "prefix": "agent-render=v1",\n "codecs": ["plain", "lz", "deflate", "arx", "arx2", "arx3", "arx4", "arx5"],\n "preferred": "arx5",\n "maxVisibleFragmentChars": 8192,\n "maxDecodedChars": 200000\n },\n "arx5": {\n "pipeline": ["tuple-envelope", "overlay-substitution", "shared-dictionary-substitution", "context-mixer", "transport-length-wire-selection"],\n "encodings": {\n "base76": { "charset": "ascii", "bitsPerChar": 6.27, "maxCompat": true },\n "base64url": { "charset": "ascii-url-safe", "bitsPerChar": 6.0 },\n "base1k": { "charset": "U+00A1-U+07FF", "bitsPerChar": 10.79 },\n "baseBMP": { "charset": "U+00A1-U+FFEF", "bitsPerChar": 15.92, "smallestVisible": true }\n },\n "dictionaries": {\n "shared": "/arx-dictionary.json",\n "overlay": "/arx2-dictionary.json"\n }\n },\n "artifacts": {\n "kinds": ["markdown", "code", "diff", "csv", "json"],\n "thisBundle": [\n { "id": "release-notes", "kind": "markdown", "title": "v3.0 release notes" },\n { "id": "codec-src", "kind": "code", "title": "arx-codec.ts (excerpt)" },\n { "id": "migration-diff", "kind": "diff", "title": "v1 to v3 migration" },\n { "id": "metrics", "kind": "csv", "title": "Bundle metrics" },\n { "id": "manifest", "kind": "json", "title": "Artifact manifest" }\n ]\n },\n "features": {\n "zeroRetention": true,\n "serverNeverSeesPayload": true,\n "clientSideOnly": true,\n "selfHostable": true,\n "openSource": true\n }\n}', }, ], }, @@ -142,7 +142,7 @@ export const sampleEnvelopes: PayloadEnvelope[] = [ const sampleDescriptions: Record = { "arx showcase": - "Tuple compression, Brotli, and visible-length Unicode encoding compress 5 rich artifacts into a single URL fragment.", + "Tuple compression and the context mixer, scored by honest transport length, compress 5 rich artifacts into a chat-safe URL fragment.", }; export const sampleLinks = sampleEnvelopes.map((envelope) => { diff --git a/src/lib/payload/fragment-arx.ts b/src/lib/payload/fragment-arx.ts index a2e8b1b..727cc1a 100644 --- a/src/lib/payload/fragment-arx.ts +++ b/src/lib/payload/fragment-arx.ts @@ -17,6 +17,8 @@ import { import { arx4CompressEnvelope, arx4DecompressEnvelope, + arx5CompressEnvelope, + arx5DecompressEnvelope, CURATED_PRIOR_IDS, EXPECTED_ARX4_PRIORS_VERSION, getActiveArx4PriorsVersion, @@ -25,6 +27,7 @@ import { import { packEnvelope } from "@/lib/payload/wire-format"; import { compactTagForCodec, + isArxMixerCodec, type ArxCodec, type PayloadCodec, type PayloadEnvelope, @@ -38,7 +41,7 @@ export type CandidateFragment = { value: string; codec: PayloadCodec; packed: boolean; - /** Default budget: percent-escaped transport length, except the arx3/arx4 baseBMP wire. */ + /** Default budget: percent-escaped transport length, except the deprecated arx3/arx4 baseBMP wire. */ transportLength: number; /** Budget for surfaces that URL-serialize the fragment: every wire percent-escaped. */ urlSerializedLength: number; @@ -54,9 +57,10 @@ const WIRE_ORDER = ["base76", "base1k", "baseBMP", "base64url"] as const satisfi * which previously each re-spelled the tag prefix + transport-length + four-candidate list. * * `bmpUsesVisibleLength` gives the dense baseBMP wire its DEFAULT budget in visible URL characters - * instead of percent-escaped transport length — see the POLICY note on buildArx3Candidates for why - * arx3 and arx4 do this. `urlSerializedLength` is unaffected, so the caller can still budget that - * same candidate by transport length for a URL-serializing surface. + * instead of percent-escaped transport length — the deprecated arx3/arx4 builders still do this so + * existing explicit encodes stay bit-identical. arx5 and the other live codecs leave it false. + * `urlSerializedLength` is unaffected, so the caller can still budget that same candidate by + * transport length for a URL-serializing surface. */ function wirePayloadsToCandidates( codec: ArxCodec, @@ -84,7 +88,7 @@ let arxDictionaryLoadPromise: Promise | null = null; let arx2OverlayDictionaryLoadPromise: Promise | null = null; let arx4PriorsLoadPromise: Promise | null = null; -// Compact ARX fragments (tags `a`/`b`/`c`/`e`) do NOT carry a dictionary version — the tag implies +// Compact ARX fragments (tags `a`/`b`/`c`/`e`/`f`) do NOT carry a dictionary version — the tag implies // the CURRENT dictionary, which keeps links short. The safety cost is that a build must not decode // with a dictionary NEWER than it was built for (a CDN/asset split serving a future dictionary, or a // version bump), because it would lack the new slots and could produce a structurally-valid-but- @@ -92,7 +96,7 @@ let arx4PriorsLoadPromise: Promise | null = null; // instead of mis-decoding. The built-in fallback dictionary (version 0) and the current external // dictionary (version 1) are both <= this and remain usable. Bumping a dictionary version is // therefore a wire change that also requires new compact tags and updating -// tests/arx-dictionary-pin.test.ts. arx4 depends on the same pin twice over, since its context-mixer +// tests/arx-dictionary-pin.test.ts. arx4/arx5 depend on the same pin twice over, since the context-mixer // prior is derived from the dictionary slot text as well as its substitution stage. const EXPECTED_ARX_DICTIONARY_VERSION = 1; const EXPECTED_ARX2_OVERLAY_VERSION = 1; @@ -101,11 +105,11 @@ const EXPECTED_ARX2_OVERLAY_VERSION = 1; // corpus this build's fragments were never coded against. That pin lives on the codec that codes with // it (EXPECTED_ARX4_PRIORS_VERSION in arx4-codec.ts); this module only drives the loader toward it. // -// arx4 holds its DICTIONARIES to that same exact standard, which is where it parts ways with +// arx4/arx5 hold their DICTIONARIES to that same exact standard, which is where they part ways with // arx/arx2/arx3. They tolerate the built-in fallback (version 0) because substitution alone degrades -// predictably; arx4 also primes its context mixer on the dictionary slot text, so a fragment coded -// against any other dictionary is one that healthy viewers cannot decode at all. Both sides therefore -// hold out for the pinned pair: encode leaves the candidate pool, decode refuses. +// predictably; the mixer also primes on the dictionary slot text, so a fragment coded against any +// other dictionary is one that healthy viewers cannot decode at all. Both sides therefore hold out +// for the pinned pair: encode leaves the candidate pool, decode refuses. /** * Thrown when an arx4 fragment reaches the decoder while the active dictionaries are not the exact @@ -122,7 +126,7 @@ export class Arx4DictionarySkewError extends Error { } } -/** True when both active dictionaries are exactly the versions arx4 codes against. */ +/** True when both active dictionaries are exactly the versions the mixer codecs code against. */ function arx4DictionariesMatchPins(): boolean { return ( getActiveDictVersion() === EXPECTED_ARX_DICTIONARY_VERSION && @@ -255,6 +259,12 @@ async function decodeArxAttempt( return await arx3DecompressEnvelope(encodedPayload); case "arx4": return arx4DecompressEnvelope(encodedPayload); + case "arx5": + return arx5DecompressEnvelope(encodedPayload); + default: { + const _exhaustive: never = codec; + throw new Error(`Unsupported arx codec: ${_exhaustive}`); + } } } @@ -300,30 +310,10 @@ export async function buildArx2Candidates( /** * Builds deferred `arx3` codec fragment candidates. - * ARX3 reuses the ARX2 tuple/overlay bytes; the only difference is how the dense baseBMP wire is - * budgeted. - * - * POLICY (deliberate, owned decision — not an incidental mechanism): the arx3 baseBMP candidate is - * budgeted by VISIBLE URL length (`value.length`), not by percent-escaped transport length, because - * the fragment surface preserves Unicode and the visible characters are what a human actually copies - * from the URL bar. Every other candidate in the shared pool — including arx2's byte-identical - * baseBMP payload — is measured with `computeTransportLength`, which inflates BMP characters ~9x for - * their UTF-8 percent-escaped size. - * - * CONSEQUENCE: because `selectCandidate` (fragment.ts) picks the global minimum transportLength, - * arx3 baseBMP is therefore selected ahead of arx2's escaped-byte measurement for the same payload. - * This is intended — it is how report-like artifacts stay human-copyable — and it means the arx3 - * baseBMP wire essentially always wins over arx2 by the metric, not by a real byte-size difference. - * - * CHANGING THIS REQUIRES A MAINTAINER DECISION: switching the arx3 baseBMP budget back to transport - * length would make arx2 and arx3 measure the same payload identically and would change which wire - * wins auto-selection. Do not flip the metric to "fix" the divergence without owning that trade-off. - * - * PER-SURFACE EXCEPTION: every candidate also carries `urlSerializedLength`, which measures the same - * baseBMP wire by transport length, for surfaces that URL-serialize the fragment (markdown links - * percent-encode baseBMP to ~9x). Selecting on that field is an additional surface-specific - * selection, not a reversal of the default policy above: the primary copy-paste URL keeps the - * visible-length budget. + * ARX3 reuses the ARX2 tuple/overlay bytes. Deprecated for auto-emit: it still budgets baseBMP by + * visible character count so explicit `{ codec: "arx3" }` stays bit-identical to already-shared `#c` + * links. Discord and WhatsApp percent-encode or mangle those Unicode wires. New auto links use + * {@link buildArx5Candidates}. */ export async function buildArx3Candidates( envelope: PayloadEnvelope, @@ -342,6 +332,10 @@ export async function buildArx3Candidates( * ARX4 reuses the ARX3 tuple/overlay stages and its baseBMP budgeting policy; it swaps Brotli for the * context mixer in arx4-codec.ts and puts a prior id char in front of the wire payload, so a * candidate reads ``. + * + * Deprecated for auto-emit: the visible-length baseBMP budget is the same Discord/Markdown + * detonation as arx3. Explicit `{ codec: "arx4" }` still produces those wires so existing tests and + * already-shared `#e` links stay reproducible. New auto links use {@link buildArx5Candidates}. */ export async function buildArx4Candidates( envelope: PayloadEnvelope, @@ -365,6 +359,28 @@ export async function buildArx4Candidates( return wirePayloadsToCandidates("arx4", false, payloads, computeTransportLength, true); } +/** + * Builds deferred `arx5` codec fragment candidates. + * + * ARX 4.5: ARX4's context mixer on ARX2's tuple/overlay pipeline, scored with ARX2's honest + * serialized transport length. Every wire — including baseBMP — is measured percent-escaped, so + * Unicode cannot win the pool and then explode in Discord markdown or WhatsApp. Existing `#e` + * (arx4) links remain decodable; arx2 stays in the auto pool for CSV regressions. + */ +export async function buildArx5Candidates( + envelope: PayloadEnvelope, + computeTransportLength: TransportLengthCalculator, +): Promise { + await ensureArx2DictionariesLoaded(); + if (!arx4DictionariesMatchPins()) return []; + + await loadArx4PriorsOnce(); + + const payloadEnvelope = { ...envelope, codec: "arx5" as PayloadCodec }; + const payloads = arx5CompressEnvelope(payloadEnvelope); + return wirePayloadsToCandidates("arx5", false, payloads, computeTransportLength); +} + /** * Decodes an ARX fragment remainder with the same versioned-payload fallback behavior as the main decoder. */ @@ -378,7 +394,7 @@ export async function decodeArxFragmentPayload( await ensureArx2DictionariesLoaded(); } - if (codec === "arx4" && !arx4DictionariesMatchPins()) { + if (isArxMixerCodec(codec) && !arx4DictionariesMatchPins()) { throw new Arx4DictionarySkewError(getActiveDictVersion(), getActiveArx2OverlayVersion()); } @@ -395,7 +411,7 @@ export async function decodeArxFragmentPayload( // re-encoding proxy or a handcrafted fragment can deliver `%6d` where the app writes `m`, and routing // on the raw char would leave that fragment asking for an asset nothing ever fetches. const priorIdChar = decodedPayload.charAt(0); - if (codec === "arx4" && CURATED_PRIOR_IDS.some((priorId) => priorId === priorIdChar)) { + if (isArxMixerCodec(codec) && CURATED_PRIOR_IDS.some((priorId) => priorId === priorIdChar)) { await loadArx4PriorsOnce(); } diff --git a/src/lib/payload/fragment.ts b/src/lib/payload/fragment.ts index d7c4f6e..0d21167 100644 --- a/src/lib/payload/fragment.ts +++ b/src/lib/payload/fragment.ts @@ -24,7 +24,7 @@ type EncodeOptions = { targetMaxFragmentLength?: number; codecPriority?: PayloadCodec[]; /** - * Budget every candidate (including arx3/arx4 baseBMP) by percent-escaped transport length. + * Budget every candidate (including deprecated arx3/arx4 baseBMP) by percent-escaped transport length. * For surfaces that URL-serialize the fragment, e.g. markdown link destinations. */ budgetByTransport?: boolean; @@ -35,7 +35,7 @@ type BudgetPolicy = "default" | "urlSerialized"; const BINARY_STRING_CHUNK_SIZE = 0x8000; const DEFAULT_SYNC_CODEC_PRIORITY: readonly PayloadCodec[] = ["deflate", "lz", "plain"]; -const DEFAULT_ASYNC_CODEC_PRIORITY: readonly PayloadCodec[] = ["arx4", "arx3", "arx2", "arx", "deflate", "lz", "plain"]; +const DEFAULT_ASYNC_CODEC_PRIORITY: readonly PayloadCodec[] = ["arx5", "arx2", "arx", "deflate", "lz", "plain"]; const PACKED_WIRE_MODES: readonly boolean[] = [true, false]; const UNPACKED_ONLY_WIRE_MODES: readonly boolean[] = [false]; const supportedCodecSet = new Set(codecs); @@ -93,8 +93,8 @@ export function getFragmentTransportLength(fragmentBody: string): number { /** * Returns the decoded visible length of a fragment body or hash. - * Browsers may expose Unicode fragments as percent-escaped text, while arx3/arx4 budget by the - * visible characters a user copies from the URL bar. + * Browsers may expose Unicode fragments as percent-escaped text. Deprecated arx3/arx4 still budget + * by the visible characters a user copies from the URL bar; live codecs use transport length. */ export function getVisibleFragmentLength(fragment: string): number { const fragmentBody = fragment.startsWith("#") ? fragment.slice(1) : fragment; @@ -146,7 +146,12 @@ function encodePayload(json: string, codec: PayloadCodec): string { case "arx2": case "arx3": case "arx4": + case "arx5": throw new Error("arx codec requires async encoding — use encodeEnvelopeAsync instead."); + default: { + const _exhaustive: never = codec; + throw new Error(`Unsupported codec: ${_exhaustive}`); + } } } @@ -182,7 +187,12 @@ function decodePayload(encoded: string, codec: PayloadCodec): string | null { case "arx2": case "arx3": case "arx4": + case "arx5": throw new Error("arx codec requires async decoding — use decodeFragmentAsync instead."); + default: { + const _exhaustive: never = codec; + throw new Error(`Unsupported codec: ${_exhaustive}`); + } } } @@ -274,12 +284,22 @@ async function buildArx4Candidates(envelope: PayloadEnvelope): Promise { + const { buildArx5Candidates: buildDeferredArx5Candidates } = await import("@/lib/payload/fragment-arx"); + return buildDeferredArx5Candidates(envelope, computeTransportLength); +} + async function buildCandidatesAsync(envelope: PayloadEnvelope, options: EncodeOptions): Promise { const codecsToTry = getAsyncCandidateCodecs(options); const wireModes = options.preferPacked === false ? UNPACKED_ONLY_WIRE_MODES : PACKED_WIRE_MODES; const candidates: CandidateFragment[] = []; for (const codec of codecsToTry) { + if (codec === "arx5") { + candidates.push(...await buildArx5Candidates(envelope)); + continue; + } + if (codec === "arx4") { candidates.push(...await buildArx4Candidates(envelope)); continue; @@ -366,7 +386,7 @@ export function encodeEnvelope(envelope: PayloadEnvelope, options: EncodeOptions * Async variant of {@link encodeEnvelope} that also supports ARX candidates. * * Returns the compact fragment body expected after `#`: a single codec tag char followed by the - * payload (e.g. `c` for arx3). The legacy `agent-render=v1..` form is still + * payload (e.g. `f` for arx5). The legacy `agent-render=v1..` form is still * accepted on decode for back-compat. * * Like the sync version, candidate encodings are generated across enabled codecs and packed @@ -390,10 +410,12 @@ export type EncodedEnvelopeSurfaces = { /** * Encodes an envelope once and returns both surface winners. * - * A link needs two selections over the same candidates: the copy-paste URL keeps the arx3/arx4 - * visible-length budget, while a markdown destination is measured percent-escaped. Running - * {@link encodeEnvelopeAsync} twice would recompress the payload, which for arx4 means running the - * context mixer a second time (~770 ms per 60 KB artifact) for two selections over identical bytes. + * A link needs two selections over the same candidates: the copy-paste URL uses each codec's default + * budget (honest transport length for live codecs; visible-length for deprecated arx3/arx4 baseBMP), + * while a markdown destination is always measured percent-escaped. Running + * {@link encodeEnvelopeAsync} twice would recompress the payload, which for the mixer codecs means + * running the context mixer a second time (~770 ms per 60 KB artifact) for two selections over + * identical bytes. */ export async function encodeEnvelopeSurfacesAsync( envelope: PayloadEnvelope, diff --git a/src/lib/payload/schema.ts b/src/lib/payload/schema.ts index b804489..46e02bf 100644 --- a/src/lib/payload/schema.ts +++ b/src/lib/payload/schema.ts @@ -3,20 +3,43 @@ export const MAX_DECODED_PAYLOAD_LENGTH = 200000; export const PAYLOAD_FRAGMENT_KEY = "agent-render"; export const artifactKinds = ["markdown", "code", "diff", "csv", "json"] as const; -export const codecs = ["plain", "lz", "deflate", "arx", "arx2", "arx3", "arx4"] as const; +export const codecs = ["plain", "lz", "deflate", "arx", "arx2", "arx3", "arx4", "arx5"] as const; export type ArtifactKind = (typeof artifactKinds)[number]; export type PayloadCodec = (typeof codecs)[number]; /** * The dictionary + entropy-coder + binary-to-text codecs, as opposed to plain/lz/deflate. - * arx/arx2/arx3 entropy-code with Brotli; arx4 uses the context mixer in arx4-codec.ts. + * arx/arx2/arx3 entropy-code with Brotli; arx4/arx5 use the context mixer in arx4-codec.ts. + * arx3 and arx4 remain decodable but are no longer auto-emitted: they score dense Unicode + * wires by visible character count, which Discord/WhatsApp then percent-encode or mangle. */ -export type ArxCodec = "arx" | "arx2" | "arx3" | "arx4"; +export type ArxCodec = "arx" | "arx2" | "arx3" | "arx4" | "arx5"; + +/** Tuple codecs that share the arx2 overlay + envelope reconstruction path. */ +export type ArxTupleCodec = "arx2" | "arx3" | "arx4" | "arx5"; + +/** Context-mixer codecs: same compressed bytes, different compact tags and selection policy. */ +export type ArxMixerCodec = "arx4" | "arx5"; /** True when `codec` is one of the arx-family codecs. */ export function isArxCodec(codec: PayloadCodec): codec is ArxCodec { - return codec === "arx" || codec === "arx2" || codec === "arx3" || codec === "arx4"; + return codec === "arx" || codec === "arx2" || codec === "arx3" || codec === "arx4" || codec === "arx5"; +} + +/** True when `codec` uses the context mixer and curated priors (arx4 / arx5). */ +export function isArxMixerCodec(codec: PayloadCodec): codec is ArxMixerCodec { + return codec === "arx4" || codec === "arx5"; +} + +/** True when a codec still decodes but must not be newly emitted. */ +export function isDeprecatedEmitCodec(codec: string): codec is "arx3" | "arx4" { + return codec === "arx3" || codec === "arx4"; +} + +/** Picker label for a codec option. Deprecated emit codecs are marked do-not-use. */ +export function codecPickerLabel(option: PayloadCodec | "auto"): string { + return isDeprecatedEmitCodec(option) ? `${option} (do not use)` : option; } // Compact fragment header: a single URL-unreserved tag char replaces the legacy @@ -26,7 +49,7 @@ export function isArxCodec(codec: PayloadCodec): codec is ArxCodec { // `B.` prefix, baseBMP U+FFF0 marker, base76/base1k length prefix), so the alphabet is not in the // header. Tags come from the RFC-3986 unreserved set so they never percent-escape, and none can // begin the legacy `agent-render=` literal, which keeps the two header forms unambiguous on decode. -// The arx family runs a, b, c, then e because d is taken by deflate. +// The arx family runs a, b, c, e, then f because d is taken by deflate. export const compactCodecTags = { plain: "p", lz: "l", @@ -35,6 +58,7 @@ export const compactCodecTags = { arx2: "b", arx3: "c", arx4: "e", + arx5: "f", } as const satisfies Record; const compactTagToCodec = new Map( diff --git a/tests/arx-codec.test.ts b/tests/arx-codec.test.ts index d3c7a0f..ffbf33b 100644 --- a/tests/arx-codec.test.ts +++ b/tests/arx-codec.test.ts @@ -379,6 +379,7 @@ describe("arx fragment round-trip", () => { compactTagForCodec("arx2"), compactTagForCodec("arx3"), compactTagForCodec("arx4"), + compactTagForCodec("arx5"), ]; expect(arxTags).toContain(autoHash.charAt(0)); }); diff --git a/tests/arx4-codec.test.ts b/tests/arx4-codec.test.ts index 81e85c4..c49fab6 100644 --- a/tests/arx4-codec.test.ts +++ b/tests/arx4-codec.test.ts @@ -198,14 +198,14 @@ describe("arx4 codec", () => { }); describe("selection", () => { - it("wins auto selection against arx3 on the report fixture", async () => { - const autoFragment = await encodeEnvelopeAsync(reportEnvelope); + it("still wins an explicit arx4 encode against arx3 on the report fixture", async () => { + const arx4Fragment = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx4" }); const arx3Fragment = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx3" }); - expect(autoFragment.startsWith(ARX4_TAG)).toBe(true); - expect(autoFragment.length).toBeLessThan(arx3Fragment.length); + expect(arx4Fragment.startsWith(ARX4_TAG)).toBe(true); + expect(arx4Fragment.length).toBeLessThan(arx3Fragment.length); - const parsed = await decodeFragmentAsync(`#${autoFragment}`); + const parsed = await decodeFragmentAsync(`#${arx4Fragment}`); expect(parsed.ok).toBe(true); if (!parsed.ok) return; expect(parsed.envelope).toEqual({ ...reportEnvelope, codec: "arx4" }); diff --git a/tests/arx4-dictionary-pin-guard.test.ts b/tests/arx4-dictionary-pin-guard.test.ts index 079e8ce..92abe82 100644 --- a/tests/arx4-dictionary-pin-guard.test.ts +++ b/tests/arx4-dictionary-pin-guard.test.ts @@ -14,11 +14,11 @@ import { decodeFragmentAsync, encodeEnvelopeAsync } from "@/lib/payload/fragment import { Arx4DictionarySkewError, decodeArxFragmentPayload } from "@/lib/payload/fragment-arx"; import { compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; -// arx4 codes with the dictionary text twice over (substitution stage plus the context-mixer prior) and -// the compact `e` tag carries no dictionary version, so a fragment coded against anything but the -// pinned dictionaries is a link no healthy viewer can decode. Unlike arx/arx2/arx3, which tolerate the -// built-in fallback dictionary, arx4 has to hold out for the exact pinned pair on both sides: encode -// drops out of the candidate pool, decode refuses. +// arx4/arx5 code with the dictionary text twice over (substitution stage plus the context-mixer prior) +// and the compact `e`/`f` tags carry no dictionary version, so a fragment coded against anything but +// the pinned dictionaries is a link no healthy viewer can decode. Unlike arx/arx2/arx3, which tolerate +// the built-in fallback dictionary, the mixer codecs hold out for the exact pinned pair on both sides: +// encode drops out of the candidate pool, decode refuses. const ARX4_TAG = compactTagForCodec("arx4"); const envelope: PayloadEnvelope = { v: 1, @@ -47,6 +47,7 @@ describe("arx4 dictionary pin guard", () => { expect(getActiveDictVersion()).toBe(0); expect(fragment.startsWith(ARX4_TAG)).toBe(false); + expect(fragment.startsWith(compactTagForCodec("arx5"))).toBe(false); // The rest of the pool still serves the link, and what it emits decodes. expect((await decodeFragmentAsync(`#${fragment}`, { skipFragmentBudget: true })).ok).toBe(true); diff --git a/tests/arx5-codec.test.ts b/tests/arx5-codec.test.ts new file mode 100644 index 0000000..23ca79f --- /dev/null +++ b/tests/arx5-codec.test.ts @@ -0,0 +1,90 @@ +import { readFileSync } from "node:fs"; +import { beforeAll, describe, expect, it } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { isBase64urlEncoded, isBaseBMPEncoded, loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { + arx4CompressEnvelope, + arx5CompressEnvelope, + arx5DecompressEnvelope, + loadArx4PriorsSync, +} from "@/lib/payload/arx4-codec"; +import { decodeFragmentAsync, encodeEnvelopeAsync, getFragmentTransportLength } from "@/lib/payload/fragment"; +import { compactTagForCodec, codecs, type PayloadEnvelope } from "@/lib/payload/schema"; + +const ARX5_TAG = compactTagForCodec("arx5"); + +const reportEnvelope: PayloadEnvelope = { + v: 1, + codec: "plain", + title: "Baanish Code Bench", + activeArtifactId: "baanish-code-bench", + artifacts: [ + { + id: "baanish-code-bench", + kind: "markdown", + title: "Baanish Code Bench", + filename: "results.md", + content: readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"), + }, + ], +}; + +describe("arx5 codec", () => { + beforeAll(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson); + }); + + it("registers arx5 as a supported codec with compact tag f", () => { + expect(codecs).toContain("arx5"); + expect(ARX5_TAG).toBe("f"); + }); + + it("reuses arx4 mixer bytes and stamps the rebuilt envelope arx5", () => { + const arx4Payloads = arx4CompressEnvelope({ ...reportEnvelope, codec: "arx4" }); + const arx5Payloads = arx5CompressEnvelope({ ...reportEnvelope, codec: "arx5" }); + + expect(arx5Payloads.base64url).toBe(arx4Payloads.base64url); + expect(arx5DecompressEnvelope(arx5Payloads.base64url)).toEqual({ + ...reportEnvelope, + codec: "arx5", + }); + }); + + it("wins auto selection with an ASCII wire, not visible-length Unicode", async () => { + const autoFragment = await encodeEnvelopeAsync(reportEnvelope); + const arx2Fragment = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx2" }); + const arx4Visible = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx4" }); + + expect(autoFragment.startsWith(ARX5_TAG)).toBe(true); + expect(autoFragment).toMatch(/^[\x21-\x7e]+$/); + expect(isBaseBMPEncoded(autoFragment.slice(2))).toBe(false); + expect(isBase64urlEncoded(autoFragment.slice(2)) || autoFragment.includes("B.")).toBe(true); + + expect(getFragmentTransportLength(autoFragment)).toBeLessThan(getFragmentTransportLength(arx2Fragment)); + expect(getFragmentTransportLength(autoFragment)).toBeLessThan(getFragmentTransportLength(arx4Visible)); + + const parsed = await decodeFragmentAsync(`#${autoFragment}`); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope).toEqual({ ...reportEnvelope, codec: "arx5" }); + }); + + it("does not emit deprecated arx3 or arx4 from the default auto pool", async () => { + const fragment = await encodeEnvelopeAsync(reportEnvelope); + expect(fragment.startsWith(compactTagForCodec("arx3"))).toBe(false); + expect(fragment.startsWith(compactTagForCodec("arx4"))).toBe(false); + expect(fragment.startsWith(ARX5_TAG)).toBe(true); + }); + + it("still decodes existing arx4 fragments", async () => { + const arx4Fragment = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx4" }); + const parsed = await decodeFragmentAsync(`#${arx4Fragment}`); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope).toEqual({ ...reportEnvelope, codec: "arx4" }); + }); +}); diff --git a/tests/arx5-markdown-link-fuzz.test.ts b/tests/arx5-markdown-link-fuzz.test.ts new file mode 100644 index 0000000..b421160 --- /dev/null +++ b/tests/arx5-markdown-link-fuzz.test.ts @@ -0,0 +1,208 @@ +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { fuzzBundles, fuzzDrafts } from "./fixtures/arx5-fuzz-drafts"; +import { DISCORD_MESSAGE_MAX_LENGTH } from "@/lib/markdown-link"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { loadArx4PriorsSync } from "@/lib/payload/arx4-codec"; +import { decodeFragmentAsync } from "@/lib/payload/fragment"; +import { + createGeneratedArtifactLinkAsync, + createGeneratedEnvelopeLinkAsync, + type GeneratedArtifactLink, + type LinkCreatorDraft, +} from "@/lib/payload/link-creator"; +import type { PayloadEnvelope } from "@/lib/payload/schema"; + +const BASE_URL = "https://agent-render.com/"; +const LIVE_AUTO_CODECS = new Set(["arx5", "arx2"]); + +type FuzzRow = { + name: string; + kind: string; + autoCodec: string; + autoMarkdown: number; + arx2Markdown: number; + arx4Markdown: number; + vsArx2: number; + vsArx4: number; + discordOk: boolean; + asciiMarkdown: boolean; +}; + +const rows: FuzzRow[] = []; + +function markdownFragment(link: GeneratedArtifactLink): string { + return link.markdownUrl.slice(link.markdownUrl.indexOf("#") + 1); +} + +function isChatSafeAsciiFragment(fragment: string): boolean { + return /^[A-Za-z0-9._~-]+$/.test(fragment); +} + +function percentEncodedPasteLength(link: GeneratedArtifactLink): number { + return new URL(link.url).toString().length; +} + +async function measureDraft(draft: LinkCreatorDraft): Promise { + const [auto, arx2, arx4] = await Promise.all([ + createGeneratedArtifactLinkAsync({ ...draft, codec: "auto" }, BASE_URL), + createGeneratedArtifactLinkAsync({ ...draft, codec: "arx2" }, BASE_URL), + createGeneratedArtifactLinkAsync({ ...draft, codec: "arx4" }, BASE_URL), + ]); + + const autoFragment = markdownFragment(auto); + const parsed = await decodeFragmentAsync(`#${autoFragment}`); + expect(parsed.ok).toBe(true); + if (parsed.ok) { + const expected = draft.kind === "diff" ? { patch: draft.content } : { content: draft.content }; + expect(parsed.envelope.artifacts[0]).toMatchObject(expected); + } + + expect(LIVE_AUTO_CODECS.has(auto.codec)).toBe(true); + expect(isChatSafeAsciiFragment(autoFragment)).toBe(true); + expect(auto.markdownLinkLength).toBe(auto.markdownLink.length); + expect(auto.markdownLinkLength).toBeLessThanOrEqual(arx4.markdownLinkLength); + + return { + name: draft.title, + kind: draft.kind, + autoCodec: auto.codec, + autoMarkdown: auto.markdownLinkLength, + arx2Markdown: arx2.markdownLinkLength, + arx4Markdown: arx4.markdownLinkLength, + vsArx2: Number((((arx2.markdownLinkLength - auto.markdownLinkLength) / arx2.markdownLinkLength) * 100).toFixed(2)), + vsArx4: Number((((arx4.markdownLinkLength - auto.markdownLinkLength) / arx4.markdownLinkLength) * 100).toFixed(2)), + discordOk: auto.markdownLinkLength <= DISCORD_MESSAGE_MAX_LENGTH, + asciiMarkdown: isChatSafeAsciiFragment(autoFragment), + }; +} + +async function measureBundle(envelope: PayloadEnvelope): Promise { + const [auto, arx2, arx4] = await Promise.all([ + createGeneratedEnvelopeLinkAsync(envelope, BASE_URL, "auto"), + createGeneratedEnvelopeLinkAsync(envelope, BASE_URL, "arx2"), + createGeneratedEnvelopeLinkAsync(envelope, BASE_URL, "arx4"), + ]); + + const autoFragment = markdownFragment(auto); + const parsed = await decodeFragmentAsync(`#${autoFragment}`); + expect(parsed.ok).toBe(true); + expect(LIVE_AUTO_CODECS.has(auto.codec)).toBe(true); + expect(isChatSafeAsciiFragment(autoFragment)).toBe(true); + expect(auto.markdownLinkLength).toBeLessThanOrEqual(arx4.markdownLinkLength); + + return { + name: envelope.title ?? envelope.activeArtifactId ?? "bundle", + kind: "bundle", + autoCodec: auto.codec, + autoMarkdown: auto.markdownLinkLength, + arx2Markdown: arx2.markdownLinkLength, + arx4Markdown: arx4.markdownLinkLength, + vsArx2: Number((((arx2.markdownLinkLength - auto.markdownLinkLength) / arx2.markdownLinkLength) * 100).toFixed(2)), + vsArx4: Number((((arx4.markdownLinkLength - auto.markdownLinkLength) / arx4.markdownLinkLength) * 100).toFixed(2)), + discordOk: auto.markdownLinkLength <= DISCORD_MESSAGE_MAX_LENGTH, + asciiMarkdown: isChatSafeAsciiFragment(autoFragment), + }; +} + +describe("arx5 markdown-link fuzz", () => { + beforeAll(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson); + }); + + it(`encodes ${fuzzDrafts.length} varied drafts and compares markdown link length`, async () => { + expect(fuzzDrafts.length).toBeGreaterThanOrEqual(40); + expect(fuzzDrafts.length).toBeLessThanOrEqual(50); + + for (const draft of fuzzDrafts) { + rows.push(await measureDraft(draft)); + } + }, 120_000); + + it(`encodes ${fuzzBundles.length} mixed bundles the same way`, async () => { + for (const envelope of fuzzBundles) { + rows.push(await measureBundle(envelope)); + } + }); + + it("never detonates an auto markdown destination the way a percent-encoded arx4 paste URL does", async () => { + const draft = fuzzDrafts.find((item) => item.title === "Bench excerpt"); + expect(draft).toBeDefined(); + if (!draft) return; + + const arx4 = await createGeneratedArtifactLinkAsync({ ...draft, codec: "arx4" }, BASE_URL); + const auto = await createGeneratedArtifactLinkAsync({ ...draft, codec: "auto" }, BASE_URL); + const detonatedPaste = percentEncodedPasteLength(arx4); + + expect(auto.markdownLinkLength).toBeLessThan(detonatedPaste); + expect(arx4.markdownLinkLength).toBeLessThan(detonatedPaste); + }); +}); + +afterAll(() => { + if (rows.length === 0) return; + + const beatsArx2 = rows.filter((row) => row.autoMarkdown < row.arx2Markdown).length; + const tiesArx2 = rows.filter((row) => row.autoMarkdown === row.arx2Markdown).length; + const losesArx2 = rows.filter((row) => row.autoMarkdown > row.arx2Markdown).length; + const beatsArx4 = rows.filter((row) => row.autoMarkdown < row.arx4Markdown).length; + const summary = { + cases: rows.length, + autoCodecs: Object.fromEntries( + [...new Set(rows.map((row) => row.autoCodec))].map((codec) => [ + codec, + rows.filter((row) => row.autoCodec === codec).length, + ]), + ), + asciiMarkdown: rows.every((row) => row.asciiMarkdown), + discordOk: rows.filter((row) => row.discordOk).length, + vsArx2: { beats: beatsArx2, ties: tiesArx2, loses: losesArx2 }, + vsArx4: { + beats: beatsArx4, + ties: rows.length - beatsArx4, + loses: rows.filter((row) => row.autoMarkdown > row.arx4Markdown).length, + }, + medianVsArx2: rows.map((row) => row.vsArx2).sort((left, right) => left - right)[Math.floor(rows.length / 2)], + rows, + }; + + const table = [ + "# arx5 markdown-link fuzz", + "", + `Cases: ${summary.cases}. Auto codecs: ${JSON.stringify(summary.autoCodecs)}.`, + `ASCII markdown destinations: ${summary.asciiMarkdown}. Discord-sized: ${summary.discordOk}/${summary.cases}.`, + `vs arx2 markdown length: ${beatsArx2} shorter, ${tiesArx2} tie, ${losesArx2} longer. Median gain ${summary.medianVsArx2}%.`, + `vs arx4 markdown length: ${beatsArx4} shorter, ${summary.vsArx4.ties} tie, ${summary.vsArx4.loses} longer.`, + "", + "| Artifact | Kind | Auto | auto md | arx2 md | arx4 md | vs arx2 | vs arx4 | Discord |", + "| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- |", + ...rows.map( + (row) => + `| ${row.name} | ${row.kind} | ${row.autoCodec} | ${row.autoMarkdown} | ${row.arx2Markdown} | ${row.arx4Markdown} | ${row.vsArx2}% | ${row.vsArx4}% | ${row.discordOk ? "ok" : "over"} |`, + ), + "", + ].join("\n"); + + const reportDirs = ["test-results"]; + if (process.env.CURSOR_ARTIFACTS_DIR) { + reportDirs.push(process.env.CURSOR_ARTIFACTS_DIR); + } else if (existsSync("/opt/cursor")) { + reportDirs.push("/opt/cursor/artifacts"); + } + + for (const dir of reportDirs) { + try { + mkdirSync(dir, { recursive: true }); + writeFileSync(path.join(dir, "arx5_markdown_link_fuzz.json"), `${JSON.stringify(summary, null, 2)}\n`); + writeFileSync(path.join(dir, "arx5_markdown_link_fuzz.md"), table); + } catch { + // Diagnostic output must not fail the suite. + } + } +}); diff --git a/tests/compact-header.test.ts b/tests/compact-header.test.ts index 31706cb..08657f7 100644 --- a/tests/compact-header.test.ts +++ b/tests/compact-header.test.ts @@ -35,6 +35,7 @@ describe("compact fragment header", () => { arx2: "b", arx3: "c", arx4: "e", + arx5: "f", }); }); @@ -51,7 +52,7 @@ describe("compact fragment header", () => { expect(parsed.ok).toBe(true); if (parsed.ok) expect(markdownContent(parsed)).toBe(envelope.artifacts[0].kind === "markdown" ? envelope.artifacts[0].content : ""); } - for (const codec of ["arx", "arx2", "arx3", "arx4"] as const) { + for (const codec of ["arx", "arx2", "arx3", "arx4", "arx5"] as const) { const parsed = await decodeFragmentAsync(`#${await encodeEnvelopeAsync(envelope, { codec })}`); expect(parsed.ok).toBe(true); if (parsed.ok) expect(markdownContent(parsed)).toBe(envelope.artifacts[0].kind === "markdown" ? envelope.artifacts[0].content : ""); diff --git a/tests/components/link-creator.test.tsx b/tests/components/link-creator.test.tsx index 3c741e4..c773746 100644 --- a/tests/components/link-creator.test.tsx +++ b/tests/components/link-creator.test.tsx @@ -63,9 +63,12 @@ describe("LinkCreator", () => { it("offers every registered codec in the compression selector", () => { render(); - for (const option of ["auto", "plain", "lz", "deflate", "arx", "arx2", "arx3", "arx4"]) { + for (const option of ["auto", "plain", "lz", "deflate", "arx", "arx2", "arx5"]) { expect(screen.getByRole("button", { name: option })).toBeInTheDocument(); } + + expect(screen.getByRole("button", { name: "arx3 (do not use)" })).toBeInTheDocument(); + expect(screen.getByRole("button", { name: "arx4 (do not use)" })).toBeInTheDocument(); }); it("keeps the newest generated link when async requests resolve out of order", async () => { diff --git a/tests/e2e/arx4-determinism.spec.ts b/tests/e2e/arx4-determinism.spec.ts index d838dcf..e063cd9 100644 --- a/tests/e2e/arx4-determinism.spec.ts +++ b/tests/e2e/arx4-determinism.spec.ts @@ -8,8 +8,9 @@ import arx4PriorsJson from "../../public/arx4-priors.json"; import arxDictionaryJson from "../../public/arx-dictionary.json"; import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; import { loadArx4PriorsSync, type Arx4PriorId } from "@/lib/payload/arx4-codec"; -import { createGeneratedArtifactLinkAsync, type LinkCreatorDraft } from "@/lib/payload/link-creator"; -import { compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; +import type { LinkCreatorDraft } from "@/lib/payload/link-creator"; +import { codecPickerLabel, compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; +import { createNodeGeneratedArtifactLink } from "./node-generated-link"; /** * Release gate for the arx4 wire format: a browser that codes even one bit differently from Node @@ -26,6 +27,7 @@ import { compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; */ const ARX4_TAG = compactTagForCodec("arx4"); +const ARX5_TAG = compactTagForCodec("arx5"); declare global { interface Window { @@ -99,7 +101,7 @@ async function fillCreatorDraft(page: Page, draft: LinkCreatorDraft) { await page.getByRole("textbox", { name: "Language", exact: true }).fill(draft.language); } await page.getByRole("textbox", { name: /^Content\b/ }).fill(draft.content); - await page.getByRole("button", { name: "arx4", exact: true }).click(); + await page.getByRole("button", { name: codecPickerLabel(draft.codec ?? "auto"), exact: true }).click(); } /** Reads the artifact body the viewer decoded, through the app's own copy action. */ @@ -202,7 +204,7 @@ test.describe("arx4 links from the shipped app bundle", () => { url.hash = ""; return url.toString(); }); - const nodeLink = await createGeneratedArtifactLinkAsync(draft, baseUrl); + const nodeLink = await createNodeGeneratedArtifactLink(draft, baseUrl); expect(nodeLink.codec).toBe("arx4"); expect(await generatedLink.inputValue()).toBe(nodeLink.url); @@ -215,3 +217,46 @@ test.describe("arx4 links from the shipped app bundle", () => { }); } }); + +const arx5CreatorDrafts: LinkCreatorDraft[] = [ + { + kind: "markdown", + title: "Release notes", + filename: "notes.md", + content: "# Release notes\n\n- Ship the arx5 codec\n- Score wires by transport length\n\n| Surface | State |\n| --- | --- |\n| viewer | ready |\n| creator | ready |\n", + language: "", + diffView: "unified", + codec: "arx5", + }, +]; + +test.describe("arx5 links from the shipped app bundle", () => { + for (const draft of arx5CreatorDrafts) { + test(`generates and previews the Node-identical ${draft.codec} link for a ${draft.title} draft`, async ({ page }) => { + await goToHash(page); + await waitForViewerState(page, "empty"); + await fillCreatorDraft(page, draft); + await page.getByRole("button", { name: "Generate link" }).click(); + + const generatedLink = page.getByLabel("Generated agent-render link"); + await expect(generatedLink).toBeVisible(); + const baseUrl = await page.evaluate(() => { + const url = new URL(window.location.href); + url.hash = ""; + return url.toString(); + }); + const nodeLink = await createNodeGeneratedArtifactLink(draft, baseUrl); + const generatedUrl = await generatedLink.inputValue(); + + expect(nodeLink.codec).toBe("arx5"); + expect(generatedUrl).toBe(nodeLink.url); + expect(new URL(generatedUrl).hash).toMatch(new RegExp(`^#${ARX5_TAG}[\\x21-\\x7e]+$`)); + + await page.getByRole("button", { name: "Preview here" }).click(); + await waitForViewerState(page, "artifact"); + await waitForRendererReady(page, draft.kind); + + expect(await copyArtifactBody(page)).toBe(draft.content); + }); + } +}); diff --git a/tests/e2e/node-generated-link.ts b/tests/e2e/node-generated-link.ts new file mode 100644 index 0000000..d9a4b8d --- /dev/null +++ b/tests/e2e/node-generated-link.ts @@ -0,0 +1,75 @@ +import { mkdirSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { build } from "esbuild"; +import type { GeneratedArtifactLink, LinkCreatorDraft } from "@/lib/payload/link-creator"; + +/** + * Playwright's Node runner compiles the importing spec, but a runtime `import()` of another + * TypeScript module (the deferred ARX stack in fragment.ts) is loaded as raw ESM and throws + * `Cannot use import statement outside a module`. Bundle the encoder once so Node encode in this + * spec matches the app without going through that loader gap. + */ +const NODE_ENCODER_ENTRY = ` +import arx2DictionaryJson from "./public/arx2-dictionary.json"; +import arx4PriorsJson from "./public/arx4-priors.json"; +import arxDictionaryJson from "./public/arx-dictionary.json"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "./src/lib/payload/arx-codec"; +import { loadArx4PriorsSync } from "./src/lib/payload/arx4-codec"; +import { createGeneratedArtifactLinkAsync } from "./src/lib/payload/link-creator"; + +loadArxDictionarySync(arxDictionaryJson); +loadArx2OverlayDictionarySync(arx2DictionaryJson); +loadArx4PriorsSync(arx4PriorsJson); + +export { createGeneratedArtifactLinkAsync }; +`; + +const repositoryRoot = path.resolve(__dirname, "../.."); + +type NodeLinkEncoder = { + createGeneratedArtifactLinkAsync: ( + draft: LinkCreatorDraft, + baseUrl?: string, + ) => Promise; +}; + +let encoderPromise: Promise | null = null; + +function loadNodeLinkEncoder(): Promise { + encoderPromise ??= build({ + stdin: { + contents: NODE_ENCODER_ENTRY, + resolveDir: repositoryRoot, + sourcefile: "node-generated-link-entry.ts", + loader: "ts", + }, + absWorkingDir: repositoryRoot, + tsconfig: "tsconfig.json", + bundle: true, + format: "esm", + platform: "node", + target: "es2022", + // Keep the encoder under the repo so leftover externals still resolve node_modules. + // brotli-wasm stays external: mixer codecs do not load it. + external: ["brotli-wasm"], + write: false, + }).then(async (result) => { + const outDir = path.join(repositoryRoot, "test-results"); + mkdirSync(outDir, { recursive: true }); + const outFile = path.join(outDir, `node-generated-link-encoder-${process.pid}.mjs`); + writeFileSync(outFile, result.outputFiles[0].text); + return import(pathToFileURL(outFile).href) as Promise; + }); + + return encoderPromise; +} + +/** Encodes a creator draft in Node through the same modules the app uses. */ +export async function createNodeGeneratedArtifactLink( + draft: LinkCreatorDraft, + baseUrl?: string, +): Promise { + const encoder = await loadNodeLinkEncoder(); + return encoder.createGeneratedArtifactLinkAsync(draft, baseUrl); +} diff --git a/tests/fixtures/arx5-fuzz-drafts.ts b/tests/fixtures/arx5-fuzz-drafts.ts new file mode 100644 index 0000000..a306d6a --- /dev/null +++ b/tests/fixtures/arx5-fuzz-drafts.ts @@ -0,0 +1,248 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import type { LinkCreatorDraft } from "@/lib/payload/link-creator"; +import type { PayloadEnvelope } from "@/lib/payload/schema"; + +const benchReport = readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"); + +function draft( + kind: LinkCreatorDraft["kind"], + title: string, + content: string, + extra: Partial = {}, +): LinkCreatorDraft { + const extension = + kind === "markdown" ? "md" : kind === "code" ? "txt" : kind === "diff" ? "patch" : kind === "csv" ? "csv" : "json"; + return { + kind, + title, + filename: `${title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "") || "artifact"}.${extension}`, + content, + language: "", + diffView: "unified", + ...extra, + }; +} + +function hexWall(bytes: number, seed: string): string { + let out = ""; + let index = 0; + while (out.length < bytes) { + out += createHash("sha256").update(`${seed}:${index}`).digest("hex"); + index += 1; + } + return out.slice(0, bytes); +} + +function patch(path: string, oldLines: string[], newLines: string[]): string { + const oldBody = oldLines.map((line) => `-${line}`).join("\n"); + const newBody = newLines.map((line) => `+${line}`).join("\n"); + return [ + `diff --git a/${path} b/${path}`, + "index 1111111..2222222 100644", + `--- a/${path}`, + `+++ b/${path}`, + `@@ -1,${oldLines.length} +1,${newLines.length} @@`, + oldBody, + newBody, + "", + ].join("\n"); +} + +/** 48 varied single-artifact drafts plus two multi-artifact envelopes. */ +export const fuzzDrafts: LinkCreatorDraft[] = [ + draft("markdown", "One liner", "# Hi\n"), + draft("markdown", "GFM table", "# Status\n\n| Surface | State |\n| --- | --- |\n| Discord | ok |\n| WhatsApp | ok |\n"), + draft("markdown", "Task list", "- [x] Ship arx5\n- [ ] Hide arx3\n- [ ] Hide arx4\n"), + draft( + "markdown", + "Mermaid flow", + "```mermaid\nflowchart LR\n A[Draft] --> B[arx5]\n B --> C[Discord]\n```\n", + ), + draft( + "markdown", + "Release notes", + "# 1.4.0\n\n## Added\n\n- arx5 honest transport scoring\n\n## Deprecated\n\n- arx3 and arx4 emit\n\n```ts\nexport const tag = \"f\";\n```\n", + ), + draft("markdown", "Accented prose", "Café résumé naïve Zürich. El niño comió piña.\n"), + draft("markdown", "CJK brief", "# 概要\n\nエージェント出力を静的なフラグメントで共有する。\n\n- 圧縮\n- 復号\n"), + draft("markdown", "Arabic note", "هذا رابط للمشاركة بدون خادم.\n"), + draft("markdown", "Emoji punch", "Ship it 🚀 then paste in Discord 📎 — no tofu.\n"), + draft("markdown", "Nested lists", "1. Protocol\n 1. fragment\n 2. envelope\n2. Renderers\n - markdown\n - code\n"), + draft("markdown", "Quoted spec", "> Artifact payloads live in the URL fragment.\n\nSee [docs](https://example.com/docs).\n"), + draft("markdown", "Bench excerpt", benchReport.slice(0, 2400)), + draft("markdown", "Repeated contract", `${"Keep the fragment client-side.\n".repeat(80)}`), + draft("markdown", "Incompressible wall", `# Digests\n\n${hexWall(1800, "md")}`), + draft("code", "Tiny TS", "export const ok = true;\n", { language: "ts", filename: "ok.ts" }), + draft( + "code", + "Wire picker", + "export function selectWire(candidates: { length: number }[]) {\n return candidates.reduce((best, next) => (next.length < best.length ? next : best));\n}\n", + { language: "ts", filename: "wire.ts" }, + ), + draft( + "code", + "Python parse", + "def parse_row(line: str) -> list[str]:\n return [cell.strip() for cell in line.split(\",\")]\n\nprint(parse_row(\"a,b,c\"))\n", + { language: "python", filename: "parse.py" }, + ), + draft( + "code", + "Rust match", + "fn codec_tag(name: &str) -> char {\n match name {\n \"arx5\" => 'f',\n \"arx2\" => 'b',\n _ => 'p',\n }\n}\n", + { language: "rust", filename: "tag.rs" }, + ), + draft( + "code", + "Go handler", + "package main\n\nimport \"fmt\"\n\nfunc main() {\n fmt.Println(\"fragment ready\")\n}\n", + { language: "go", filename: "main.go" }, + ), + draft( + "code", + "SQL rollup", + "SELECT kind, COUNT(*) AS n\nFROM artifacts\nWHERE codec IN ('arx2', 'arx5')\nGROUP BY kind\nORDER BY n DESC;\n", + { language: "sql", filename: "rollup.sql" }, + ), + draft( + "code", + "Regex heavy", + "const FRAGMENT = /^#([pldabcef])([A-Za-z0-9._~-]+)$/;\nexport const match = (hash: string) => FRAGMENT.exec(hash);\n", + { language: "ts", filename: "re.ts" }, + ), + draft( + "code", + "Comment novel", + `${"// Keep dictionary pins exact. A skewed overlay mints an undecodable mixer link.\n".repeat(40)}export const pin = 1;\n`, + { language: "ts", filename: "pins.ts" }, + ), + draft("code", "Shell install", "#!/bin/sh\nset -eu\nnpm ci\nnpm run check\n", { language: "bash", filename: "ci.sh" }), + draft("code", "Minified-ish", "export function x(a,b,c){return a ` item${index},`), + Array.from({ length: 24 }, (_, index) => ` item${index},`), + ), + ), + draft("csv", "Tiny table", "name,ok\nviewer,true\n"), + draft("csv", "Quoted commas", 'title,note\n"Big, link","uses, commas"\n"Second, row","still, quoted"\n'), + draft( + "csv", + "Wide metrics", + `kind,codec,visible,transport,ascii,discord\n${["markdown", "code", "diff", "csv", "json"].map((kind, index) => `${kind},arx5,${200 + index},${210 + index},true,ok`).join("\n")}\n`, + ), + draft("csv", "Sparse grid", "a,b,c,d,e\n1,,,,\n,,3,,\n,,,,5\n"), + draft("csv", "Numeric series", `n,value\n${Array.from({ length: 40 }, (_, index) => `${index},${(index * 1.7).toFixed(3)}`).join("\n")}\n`), + draft("csv", "Unicode headers", "名前,状態\nビューア,準備完了\n"), + draft( + "csv", + "Unique tokens", + `id,token\n${Array.from({ length: 30 }, (_, index) => `${index},${hexWall(16, `csv${index}`)}`).join("\n")}\n`, + ), + draft("csv", "Long row", `col,payload\n1,"${"cell ".repeat(80).trim()}"\n`), + draft("json", "Flat flags", '{\n "codec": "arx5",\n "ascii": true\n}\n'), + draft( + "json", + "Nested config", + JSON.stringify( + { + transport: { method: "fragment", tag: "f" }, + limits: { fragment: 8192, decoded: 200000, discord: 2000 }, + }, + null, + 2, + ), + ), + draft( + "json", + "Array of objects", + JSON.stringify( + [ + { id: "a", kind: "markdown" }, + { id: "b", kind: "code" }, + { id: "c", kind: "csv" }, + ], + null, + 2, + ), + ), + draft( + "json", + "Package slice", + JSON.stringify( + { + name: "agent-render", + scripts: { test: "vitest run", e2e: "playwright test" }, + dependencies: { next: "15.1.11", react: "19.1.0" }, + }, + null, + 2, + ), + ), + draft("json", "Unicode keys", JSON.stringify({ 概要: "静的", café: "ok" }, null, 2)), + draft("json", "Scalars", JSON.stringify({ n: 0, flag: false, empty: null, ratio: 1.25 }, null, 2)), + draft( + "json", + "Deep nest", + JSON.stringify({ a: { b: { c: { d: { e: { tag: "f", prior: "j" } } } } } }, null, 2), + ), + draft("json", "Compact blob", JSON.stringify({ tokens: Array.from({ length: 20 }, (_, index) => hexWall(8, `j${index}`) ) })), + draft("json", "Pretty repeated", `${JSON.stringify({ keep: "fragment client-side", codec: "arx5" }, null, 2)}\n`.repeat(12)), + draft("markdown", "Bracket title [beta]", "Notes with a title that markdown must escape.\n"), + draft("markdown", "Very long title for a markdown label that eats Discord budget", "Short body. The label is the variable.\n"), +]; + +export const fuzzBundles: PayloadEnvelope[] = [ + { + v: 1, + codec: "plain", + title: "Mixed bundle", + activeArtifactId: "notes", + artifacts: [ + { id: "notes", kind: "markdown", title: "Notes", filename: "notes.md", content: "# Bundle\n\nTwo artifacts.\n" }, + { id: "code", kind: "code", title: "code.ts", filename: "code.ts", language: "ts", content: "export const n = 2;\n" }, + ], + }, + { + v: 1, + codec: "plain", + title: "Triple bundle", + activeArtifactId: "table", + artifacts: [ + { id: "table", kind: "csv", title: "table.csv", filename: "table.csv", content: "k,v\narx5,1\n" }, + { id: "spec", kind: "json", title: "spec.json", filename: "spec.json", content: '{"k":"v"}\n' }, + { + id: "patch", + kind: "diff", + title: "change.patch", + filename: "change.patch", + patch: patch("a.txt", ["old"], ["new"]), + }, + ], + }, +]; diff --git a/tests/link-creator-encode-once.test.ts b/tests/link-creator-encode-once.test.ts index 3d4c2ab..d6b3d1d 100644 --- a/tests/link-creator-encode-once.test.ts +++ b/tests/link-creator-encode-once.test.ts @@ -8,21 +8,25 @@ import { decodeFragmentAsync } from "@/lib/payload/fragment"; import { createGeneratedArtifactLinkAsync, type LinkCreatorDraft } from "@/lib/payload/link-creator"; /** - * A generated link needs two selections over the same candidates: the copy-paste URL keeps the - * arx3/arx4 visible-length budget while the markdown destination is measured percent-escaped. Those - * are two reads of one pool, not two encodes: arx4's context mixer costs ~770 ms per 60 KB artifact, - * so a second pass would double every link creation's main-thread stall for identical bytes. + * A generated link needs two selections over the same candidates: the copy-paste URL uses each + * codec's default budget while the markdown destination is measured percent-escaped. Those are two + * reads of one pool, not two encodes: the context mixer costs ~770 ms per 60 KB artifact, so a + * second pass would double every link creation's main-thread stall for identical bytes. */ -const arx4Compressions = vi.fn(); +const mixerCompressions = vi.fn(); vi.mock("@/lib/payload/arx4-codec", async (importOriginal) => { const actual = await importOriginal(); return { ...actual, arx4CompressEnvelope: (...args: Parameters) => { - arx4Compressions(...args); + mixerCompressions(...args); return actual.arx4CompressEnvelope(...args); }, + arx5CompressEnvelope: (...args: Parameters) => { + mixerCompressions(...args); + return actual.arx5CompressEnvelope(...args); + }, }; }); @@ -49,13 +53,13 @@ describe("async link creation", () => { loadArxDictionarySync(arxDictionaryJson); loadArx2OverlayDictionarySync(arx2DictionaryJson); loadArx4PriorsSync(arx4PriorsJson as Arx4Priors); - arx4Compressions.mockClear(); + mixerCompressions.mockClear(); }); it("codes the payload once and picks both link surfaces from that pool", async () => { const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); - expect(arx4Compressions).toHaveBeenCalledTimes(1); + expect(mixerCompressions).toHaveBeenCalledTimes(1); // Both surfaces still come out of the pool decodable, and to the same envelope. const markdownFragment = generatedLink.markdownUrl.slice(generatedLink.markdownUrl.indexOf("#") + 1); @@ -68,9 +72,9 @@ describe("async link creation", () => { } }); - it("codes the payload once in auto mode too, where arx4 leads the codec priority", async () => { + it("codes the payload once in auto mode too, where arx5 leads the codec priority", async () => { await createGeneratedArtifactLinkAsync({ ...draft, codec: "auto" }, "https://agent-render.com/"); - expect(arx4Compressions).toHaveBeenCalledTimes(1); + expect(mixerCompressions).toHaveBeenCalledTimes(1); }); }); diff --git a/tests/link-creator.test.ts b/tests/link-creator.test.ts index 2ff3b7d..3cecd9c 100644 --- a/tests/link-creator.test.ts +++ b/tests/link-creator.test.ts @@ -119,9 +119,10 @@ describe("link creator payloads", () => { expect(parsed.ok).toBe(true); }); - it("lets the async auto encoder pick ARX4 when it wins the visible URL budget", async () => { + it("lets the async auto encoder pick ARX5 on honest transport length", async () => { loadArxDictionarySync(arxDictionaryJson); loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson); const draft: LinkCreatorDraft = { kind: "markdown", @@ -136,10 +137,11 @@ describe("link creator payloads", () => { const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); const parsed = await decodeFragmentAsync(generatedLink.hash); - expect(generatedLink.codec).toBe("arx4"); - expect(generatedLink.hash.startsWith(`#${compactTagForCodec("arx4")}`)).toBe(true); - expect(generatedLink.url).toContain(`#${compactTagForCodec("arx4")}`); - expect(generatedLink.fragmentLength).toBeLessThan(1900); + expect(generatedLink.codec).toBe("arx5"); + expect(generatedLink.hash.startsWith(`#${compactTagForCodec("arx5")}`)).toBe(true); + expect(generatedLink.url).toContain(`#${compactTagForCodec("arx5")}`); + expect(generatedLink.hash.slice(1)).toMatch(/^[A-Za-z0-9._~-]+$/); + expect(generatedLink.fragmentLength).toBeLessThan(MAX_FRAGMENT_LENGTH); expect(parsed.ok).toBe(true); }); diff --git a/tests/sample-link-data.test.ts b/tests/sample-link-data.test.ts index f9d4c89..dd9f9e0 100644 --- a/tests/sample-link-data.test.ts +++ b/tests/sample-link-data.test.ts @@ -1,10 +1,12 @@ import { describe, expect, it } from "vitest"; import { sampleLinkCards } from "@/components/home/sample-link-data"; import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { loadArx4PriorsSync } from "@/lib/payload/arx4-codec"; import { sampleEnvelopes, sampleLinks } from "@/lib/payload/examples"; import { decodeFragmentAsync } from "@/lib/payload/fragment"; import { compactTagForCodec } from "@/lib/payload/schema"; import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; import arxDictionaryJson from "../public/arx-dictionary.json"; describe("homepage sample link data", () => { @@ -36,24 +38,27 @@ describe("homepage sample link data", () => { ).toEqual(expectedCards); }); - it("uses a real ARX3 fragment for the homepage ARX showcase sample", async () => { + it("uses a real ARX5 fragment for the homepage ARX showcase sample", async () => { loadArxDictionarySync(arxDictionaryJson); loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson); const sample = sampleLinkCards.find((card) => card.title === "arx showcase"); - expect(sample?.hash?.startsWith(`#${compactTagForCodec("arx3")}`)).toBe(true); - expect(sample?.fragmentLength).toBeLessThan(1900); + expect(sample?.hash?.startsWith(`#${compactTagForCodec("arx5")}`)).toBe(true); + expect(sample?.hash?.slice(1)).toMatch(/^[A-Za-z0-9._~-]+$/); + expect(sample?.fragmentLength).toBeLessThan(4000); const parsed = await decodeFragmentAsync(sample?.hash ?? ""); + const source = sampleEnvelopes.find((envelope) => envelope.title === "arx showcase"); expect(parsed.ok).toBe(true); - if (!parsed.ok) { + expect(source).toBeDefined(); + if (!parsed.ok || !source) { return; } - expect(parsed.envelope.codec).toBe("arx3"); - expect(parsed.envelope.title).toBe("arx showcase"); + expect(parsed.envelope).toEqual({ ...source, codec: "arx5" }); expect(parsed.rawLength).toBe(sample?.fragmentLength); }); });