intake: one bounded read and one scan of every hook payload

[k4rthiknerella]

Stacked on #60 (store: group commit, one full sync per hook process); review and merge after it.

Every hook entry read its payload with io.ReadAll and decoded it into Go values: about eight copies of an attacker-influenced payload that can be 8 MiB, and a peak RSS of 56 MiB at the cap. This replaces that with one bounded read, one json.Valid pass and one walk of the top level that records where each field is rather than decoding it (RES-4, RES-5), reads large fields through fixed buffers (RES-6) and streams the Stop payload instead of holding it (RES-7). It also switches on the 1, 4 and 8 MiB rss cases, and records each case's measured figure beside a line that holds on the reference machine with no tolerance (RES-28).

The branch carries the contracts PR's commit (contracts: the types and stubs the 1.2.0 tracks build against) at its base because this change needs it. Review the commits above it: the intake change, the perf and H-17 follow-ups, and the schema commit. The contracts commit drops out when that PR merges.

What changed

• internal/hookio (new): Read takes a 4 KiB first chunk, then chunks that grow to an eighth of what has been read, capped at 64 KiB, then makes one exact-size copy. It reads at most MaxPayloadBytes+1 bytes, accepts exactly the cap and refuses one byte more with ErrTooLarge. From 1 MiB up it collects its chunks before returning, so the allocations after it reuse those pages (hook.1MiB: 10.09 MiB without, 9.59 with). Intake validates with json.Valid, then walks the top level once and records the spans of 22 keys, in a frozen order (TestPayloadKeyOrderIsFrozen; later PRs append and never reorder). Keys are matched as the old payload structs matched their fields: exactly or case-insensitively, the last match winning. Payload.Str, Bool, Int and Strs decode the small fields the way the struct decode did: null and absent read as empty or nil, and any other type fails the whole intake. Strs takes Into(key, &dst) pairs, so a destination cannot drift from its key; it returns on the first field of the wrong type, and every caller discards what it wrote on that error. A payload over the cap is refused with a *TooLargeError (errors.Is(err, ErrTooLarge) holds) that carries the session_id and cwd named in its first 4 KiB, which were read before the cap was reached, so the refusal is recorded in the run that sent it. IntakeStream validates and walks a payload in one pass and never holds it. It refuses a root that is not an object or null on its first byte, before anything is parsed, and returns ErrStopped when the callback ends the walk early. It has no cap of its own; its caller bounds the reader.
• internal/jsonscan (new, standard library only, nothing built at init): Top and Field walk a validated buffer and record byte ranges. The last duplicate key wins, as in encoding/json, and a top-level null leaves every key absent. Head, Tail and Lines unescape a string span through fixed buffers and never split a character. Stream and StreamOne validate as they go through a window that starts at 4 KiB and grows to 64 KiB. They copy out only the values at the wanted paths, as a head, a tail, both, a presence bit or lines. A string is passed unescaped, any other value as its JSON text, and null is not passed at all. Lookup and ValueSpan serve shape's single-field reads. NewKeySet matches keys exactly after unescaping, as a map decode does; NewFoldKeySet and Path.Fold match them as a struct field matches its name, under Unicode simple folding. Field, Lookup and ValueSpan document their precondition: a buffer json.Valid accepted. They do not validate.
• internal/hook: hook, post, probe and paused fill their fields from the spans Intake recorded (Payload.from, PostPayload.from, sessionPayload.from), and an over-cap payload's refusal goes to the session its head names (refusedSession), or to unattributed when it names none. tool_response is still never a value: backgroundedCall reads two presence bits off spans, in any case, as its old struct did. tool_input and tool_response are walked again for the few keys read from them; nothing decodes the payload. The failure message is read to 8 KiB, because the exit code is on its first line. closeReason maps a capture error to its coverage reason, payload_too_large among them. scratchpad_dir was declared and never read, and it is not among the scanned keys.
• internal/shape: command, run_in_background and the host fields are read as spans (inputField, stringValue). run_in_background was a struct field and is matched in any case; command and the host fields came from a map decode and stay exact. The package doc states the validated-input precondition its readers rely on. canonical keeps its decode into any for digest compatibility until RES-48, and that is the one decode a hook path may still make. It now streams json.Marshal's bytes into the HMAC through a 4 KiB buffer instead of building them (canonEncoder: sorted keys, HTML-safe escapes, ES6 number formatting, and U+2028/9 and invalid UTF-8 written as json.Marshal writes them). FuzzCanonicalMatchesMarshal holds the stream to json.Marshal byte for byte, and TestCanonicalDigestIsUnchanged pins a digest to the value the Marshal-based form gave.
• cmd/rashomon/intake.go: recap and digest --stdin stream the Stop payload through IntakeStream, reading at most MaxPayloadBytes+1 bytes through io.LimitReader, so a stdin that never ends is refused rather than read forever. The final message is captured as a 256 KiB head and a 64 KiB tail. The account reads 64 Ki runes, which UTF-8 holds in at most 256 KiB, so the head is exactly what the whole decode gave it. The tail is for the comparator (RES-17), and nothing on this base reads it. A payload over 8 MiB is still no payload, as before. Keys are matched in any case, session_id is read from its last copy, and a copy over 4 KiB names no session rather than failing the payload.
• internal/store: ReasonPayloadTooLarge, with its sentence in the report's reason text. docs/store-schema.json gains payload_too_large on coverage.reason in a separate commit, together with flush_failed on terminal.reason from the group-commit PR.
• Budgets: the 1, 4 and 8 MiB rss cases gate from this PR. Every case carries measured_mib, the median of nine runs on the reference machine when its line was set. LineHolds refuses a missing figure, and any figure over the line, with no tolerance (TestRES28h_EveryLineHoldsOnTheReferenceMachine, TestRES28h_LineHoldsTakesNoTolerance). Until RES-48 the hook cases carry 3.4 MiB per MiB of payload and the post cases 2.45 (RES-4's 2.0x read plus canonical's decoded copy of tool_input), with the basis on each row. RES28BudgetMiB takes the slope. New benchmark rows cover Top, Lines, Stream and Read. Each row this change moves is restated at what it now measures, with max equal to measured.
• Mutants re-anchored to the new code: H-14 (2), H-20 (2), TB (2), RES-7 (1, the stream's key match) and RES-4 (1, read's three results). Mutants re-anchored to the restated rows: DEL-7 (1), RES-36 (2).
• TestTopAllocatesNothingPerValue measures the walk through Field. Under -race, json.Valid's sync.Pool drops values at random by design, so Top as a whole is asserted allocation-free only without -race. Linux CI had failed it on that pool, not on the walk.
• RES-7's growth bound is 6 MiB under -race and stays 4 MiB without it. Linux CI on amd64 measured up to 4.32 MiB of race-mode growth; a recap that holds the 7 MB payload measured 86 MiB.
• hookio.Read is also tested under iotest.OneByteReader and a reader that errors partway through, so its io.ReadFull loop is checked against both a one-byte-at-a-time source and a mid-stream error. readFinalMessage's own cap (separate from Read's: recap streams through IntakeStream, which never calls Read) is tested at exactly MaxPayloadBytes and one byte over. Post.Capture and RunProbe get the same over-cap test Handler already had, since all three share hookio.Read's cap and closeReason's mapping to payload_too_large.
• The package comment on internal/hookio said large fields are read through Head, Tail and Lines; only Head has a caller yet, and the comment now says so.

Decisions

• Payload keys are matched as the old structs matched them, in any case, so {"SESSION_ID":"s"} still attributes the call to s. FuzzRES5_IntakeMatchesTheStructDecode, FuzzRES5_ToolInputFieldsMatchTheirOldDecode and FuzzRES7_FinalMessageMatchesTheStructDecode hold every field each entry reads to the struct or map it was decoded from.
• A repeated key is read from its last copy on the buffered path. The struct decode kept an earlier string under a later null, failed on any copy of the wrong type and merged repeated objects; Claude Code repeats no key, and the fuzz targets leave repeated keys out. The streamed path already matches the struct decode here, since it skips null and checks every copy's type.
• An over-long session_id (over 4 KiB) on the Stop payload names no session: the message is still read, and recap falls back to the newest run, its rule for a payload with no session_id. Technical spec 1.2.11 buckets an over-cap session_id under unattributed for records; recap writes none.
• The 64 KiB tail is still captured: Technical spec 8.2.2 has it captured beside the head until the comparator lands. Its ring grows by append to the value's own size, so a short message costs only its own bytes.
• Chunk growth is an eighth of what has been read, between 4 KiB and 64 KiB. Fixed 64 KiB chunks cost 2.7x at 40 KB, outside RES-4's 2.2x.
• Recap's head is 256 KiB, not 64 KiB, because a 64 KiB head would shrink the account for non-ASCII text.
• Path.String makes a non-string at a path stop the stream with ErrNotString. That keeps the old decode's type failure for session_id and last_assistant_message.
• RES-5's scan (TestRES5_PayloadScannedOnce, an AST walk) treats the whole of cmd/rashomon as hook path. Any json.Unmarshal or json.NewDecoder there fails it unless it is allowlisted by function name with its reason. A future non-hook command that decodes JSON does so in a package outside the list.
• BenchmarkTop measures the KeySet built per call plus the walk, after a warm-up call. Top alone allocates nothing per value, and the gate refuses a row of zero as a placeholder. Without the warm-up, json.Valid's pooled scanner refilled after a GC and moved the row between 0 and 6,652 B/op.

Measured

All figures are from this tree on the reference machine (Apple M3, go1.27.2, release flags for the RSS runs).

hookio.Read allocation per call (RES-4, TestRES4_ReadAllocBound):

Payload	Allocated	Ratio	Allocations
400 B	4,584 B	11.46x (the bound at 400 B is 8 KiB, not a ratio)	4
40 KB	83,456 B	2.09x	16
1 MiB	2,181,536 B	2.08x	48
4 MiB	8,459,136 B	2.02x	97
8 MiB	16,823,168 B	2.01x	161

Peak RSS, median of nine, against each case's line (RES-28, TestRES28_MaxRSSFollowsThePayloadBudget: pass):

Case	Measured MiB	Line MiB	Before this change
hook.400B	6.14	6.5008
hook.40KB	6.17	6.5839
hook.1MiB	9.52	9.90 (slope 3.4)	11.84
hook.4MiB	19.75	20.10 (slope 3.4)	27.59
hook.8MiB	31.81	33.70 (slope 3.4)	47.89
post.540B	5.98	6.5011
post.40KB	6.16	6.5839
post.1MiB	8.94	8.95 (slope 2.45)	13.28
post.4MiB	15.48	16.30 (slope 2.45)	32.97
post.8MiB	23.58	26.10 (slope 2.45)	55.80

Every case holds its line with no tolerance applied. The platform tolerance (15% on this runner until 2026-11-06, then 10%) covers other machines and run-to-run noise. The measured_mib on each row is this run's median.

Entry rows (TestRES28b_EveryEntryStaysWithinItsRow: pass):

Entry row	Measured MiB	Row p50
hook.bash_short	6.02	6.50
hook.edit_50k	6.22	6.61
post.bash_short	6.11	6.50
post.edit_314k	6.81	8.00
recap.stop	6.92	12.00
recap.pending	4.81	6.50
probe.start	6.00	8.00
probe.end	6.03	6.00

probe.end is 0.03 MiB over its row, which is inside the tolerance. That row is derived, not measured, and this PR moves probe onto Intake.

TestRES2_InitBudget: pass, with 42 packages, 32,360 bytes, 211 allocations and 0 GC lines.

Lines over a 4 MiB stdout of 145,398 lines, with the summary on the last line, allocates 24 bytes (TestRES6_LargeFieldScanIsConstantMemory). Recap's peak RSS at a 7 MB final message is measured by TestRES7_HugeFinalMessageIsConstant, which bounds it at 12 MiB and at no more than 4 MiB above a 2 KiB message. The 12 MiB bound is not applied under the race detector. The 4 MiB bound is.

Benchmark rows restated (best of three at 20 iterations; max equals measured)

Benchmark	Before (B/op / allocs/op)	Now
BenchmarkDerive/bash_short	5,386 / 77	4,624 / 71
BenchmarkDerive/bash_long_4k	428,860 / 2,333	430,688 / 2,333
BenchmarkDerive/edit_50k	116,032 / 38	63,464 / 28
BenchmarkLexTokens	128,544 / 750	128,544 / 750
BenchmarkRunHook/bash_short	49,394 / 352	53,828 / 361
BenchmarkRunHook/edit_50k	429,464 / 308	218,442 / 311
BenchmarkRunPost/bash_short	42,909 / 280	47,638 / 298
BenchmarkRunPost/edit_314k	1,172,882 / 284	893,014 / 298
BenchmarkRecap/10	238,319 / 534	262,910 / 553
BenchmarkRecap/100	1,047,751 / 3,822	1,072,590 / 3,843
BenchmarkRecap/1000	9,456,658 / 35,328	9,481,832 / 35,350
BenchmarkRecapPending/10	237,619 / 537	261,194 / 553
BenchmarkRecapPending/1000	9,457,111 / 35,339	9,480,470 / 35,351
BenchmarkTop (new)		616 / 6
BenchmarkLines/4MiB (new)		24 / 2
BenchmarkStream/64MiB_batch (new)		158,322 / 17
BenchmarkStream/187MB_line (new)		349,379 / 30
BenchmarkRead/400B (new)		4,584 / 4
BenchmarkRead/40KB (new)		83,456 / 16
BenchmarkRead/1MiB (new)		2,163,584 / 48
BenchmarkRead/4MiB (new)		8,459,136 / 97
BenchmarkRead/8MiB (new)		16,823,173 / 161

Where a row rose:

• The small-payload hook and post rows carry about 4 KB more. That is the 4 KiB first chunk, which replaces io.ReadAll's 512-byte start and doubling, plus the KeySet built per call.
• The recap rows carry about 24 KB more per call: the streamer's 4 KiB window, its nesting table, and the head and tail capture buffers, which grow by doubling to the message's size. This replaces the decode of the whole payload. For a 2 KiB Stop payload that costs more than the decode did. For a 7 MB one it is about 50 MiB less (RES-7).

The large-payload rows fall: edit_50k by half, edit_314k by a quarter.

Limits and what is still to run

Not in this PR

• The event entries (PostToolBatch, SubagentStop, PostCompact) do not exist on this base. Here, IntakeStream serves recap's Stop and catch-up. A later PR moves those entries onto it, with per-path caps.
• RES-12 (no whole-run read on a hook path).
• RES-48, the streaming canonicaliser. It retires canonical's decode and returns the hook and post rss lines to 2.2 MiB per MiB.
• The token, leading-cd and nesting bounds, runtime settings and the init tests.
• A reader for the 64 KiB tail of the final message. The tail is captured for the comparator, and nothing on this base reads it.
• An rss case for post with a large tool_input. The hook cases already gate canonical's decode of a tool_input that carries the whole payload (Edit payloads, slope 3.4); the post cases pad tool_response, so the same decode on the post path has no row of its own until RES-48 removes it.
• A production caller for jsonscan.Lines and jsonscan.Tail. Both are staged for later PRs in this stack; Head is the only one of the three a consumer calls today.

What could go wrong

• canonEncoder must stay byte-identical to json.Marshal, or every non-shell digest moves. The fuzz target and the pinned digest guard it, and RES-48 retires it.
• Read forces one GC from 1 MiB up, which holds hook.1MiB under its 9.90 MiB line (9.59 with it, 10.09 without). BenchmarkRead puts its cost at about 0.1 ms at 1 and 4 MiB, against the hook rows' 10 to 15 ms p50.
• A fold KeySet compares a key that misses exactly against every wanted key with EqualFold: 22 short comparisons at most, per top-level key.
• An over-cap payload is attributed from its first 4 KiB only: a session_id repeated past the head is not seen, and a head that is not a sound start of one object names nothing.
• Stream delivers CaptureLines pieces before it meets a syntax error. A caller discards them on error, as readFinalMessage does.

Still to run before review

• go test ./... -race -count=1: ran, and was red on internal/jsonscan's zero-allocation check for Top -- json.Valid's scanner comes from a sync.Pool the race detector drops, so the pool is never warm under -race. Fixed by measuring the walk through Field instead and checking Top as a whole only without -race, where the pool stays warm (jsonscan: measure the walk, not Valid's own pool, for zero allocation and jsonscan: keep Top's zero-allocation check where the pool is deterministic). After the commits that match keys in any case, attribute a refusal and bound recap's read, go test -race -count=1 reports ok for internal/jsonscan, internal/hookio, internal/hook, internal/shape, cmd/..., test/acceptance and test/mutation, and go test ./... -count=1 reports ok for every package.
• CI on hookio: say which fields Strs may have already written: Perf gates (linux) was red, the only red job there: about 9.5 MiB measured against roughly 6 MiB darwin rows. Mostly inherited from perf: budgets.json and the resource gates #59, which never recorded a linux baseline, so linux still gates against the M3 rows; this PR adds its own share by switching on three P-10 cases (hook.1MiB, post.1MiB, post.4MiB) that have the same gap.
• The eight entry rows are moved. test/perf/budgets.json now carries this tree's measured medians: hook.bash_short 5.95, hook.edit_50k 6.20, post.bash_short 5.94, post.edit_314k 6.73 (p95 moved to RES-28's line, 7.16, in place of 8 MiB), recap.stop 7.06, recap.pending 6.06, probe.start 5.97, probe.end 5.98, all MiB. recap.pending no longer reads the no-store early return (4.81 MiB); the seeded rig gives 6.06.
• The Linux recap memory measurement (TestRES7_HugeFinalMessageIsConstant, through the reaper): measured 2026-10-13, linux/arm64, go1.25.8 -race, an instrumented child's floor before recap does anything is 20.0 MiB at a 2 KiB message and 22.9 MiB at a 7 MB one (4.4 and 5.5 MiB without -race). Under the race detector the floor is already over the 12 MiB absolute bound, so that bound is skipped for an instrumented child and only the growth bound (no more than 4 MiB above the 2 KiB case) applies.
• Why the reaper and its child-running test helpers may import os/exec and syscall while hook-path code may not: TestH17_TestPackagesStayTestOnly now scopes the import ban to the packages a shipped binary can actually link — every package outside test/, plus any test/ package a non-test package imports. test/perf/peakrss and test/perf/peakrss/reaper are imported only from _test.go files, so they fall outside that scope and may use os/exec and syscall to run and reap the child processes RES-7 and the RSS cases spawn.
• Fuzz targets that exist on this tree: FuzzRES5_TopMatchesStdlib and FuzzRES7_StreamMatchesStdlib (internal/jsonscan), FuzzCanonicalMatchesMarshal and FuzzRES5_ToolInputFieldsMatchTheirOldDecode (internal/shape), FuzzRES5_IntakeMatchesTheStructDecode (internal/hook) and FuzzRES7_FinalMessageMatchesTheStructDecode (cmd/rashomon). The three struct-decode targets ran clean for 60 s, 45 s and 30 s (hook, recap, shape). No log on this tree shows a 120 s run of any target; that run is still open.
• The 12 mutants in RES-4_refusal-head.py, RES-5_fold-keys.py and RES-7_final-message-bound.py, and the two moved anchors in RES-4_read-bound.py and RES-7_stream.py, were each applied alone and turned their named tests red. The full mutation sweep over the RES-4, RES-5, RES-6, RES-7 and RES-28 mutant files has not been run on this tree. It no longer runs on every push or pull request — it is now a required, by-hand step (Actions > CI > Run workflow) before a release is tagged, or whenever a change adds or moves mutants, run locally with python3 test/mutation/sweep.py. Still open for this PR.
• The full benchmark gate: not run on this tree; still open.

Mutants and the test that must catch each (28)

Mutant	Test
RES-4 the payload is read with io.ReadAll's doubling	TestRES4_ReadAllocBound
RES-4 a payload of exactly the cap is refused	TestRES4_CapAtNAndNPlusOne
RES-4 the refusal is recorded as an internal error	TestRES4_CapAtNAndNPlusOne
RES-4 a refusal drops the head that names its session	TestRES4_TooLargeNamesTheSessionInItsHead, TestRES4_CapAtNAndNPlusOne
RES-4 a head cut short of its end still names a session	TestRES4_TooLargeNamesTheSessionInItsHead
RES-4 a refusal is not ErrTooLarge	TestRES4_CapAtNAndNPlusOne
RES-5 post decodes the payload again for the background keys	TestRES5_PayloadScannedOnce
RES-5 Top walks without json.Valid's verdict	TestRES5_TopDefersToJSONValid
RES-5 a fold KeySet matches only exactly	TestRES5_FoldKeySetMatchesStructFieldNames
RES-5 a streamed path ignores Fold	TestRES5_StreamFoldPathMatchesStructFieldNames
RES-5 intake matches payload keys exactly	TestRES5_IntakeMatchesStructFieldNames
RES-5 run_in_background is matched exactly	TestRES5_RunInBackgroundMatchesItsOldField
RES-5 the background keys are matched exactly	TestRES5_BackgroundKeysMatchTheirOldFields
RES-6 Lines unescapes the whole span into one slice	TestRES6_LargeFieldScanIsConstantMemory
RES-7 recap reads the Stop payload whole through Intake	TestRES7_HugeFinalMessageIsConstant
RES-7 Stream copies a value at a key nobody asked for (re-anchored)	TestRES7_StreamCopiesOnlyWantedPaths
RES-7 the final message is counted, not limited	TestRES7_FinalMessageReadStopsAtTheCap
RES-7 a top-level string passes the root check	TestRES7_IntakeStreamRejectsANonObjectRoot, TestRES7_FinalMessageRefusesATopLevelString
RES-7 an over-long session_id fails the payload	TestRES7_FinalMessageSessionIDLastCopyWins
RES-7 a stopped walk reads as a whole payload	TestRES7_IntakeStreamReportsAnEarlyStop
RES-28 a line is padded with the tolerance before the measurement is compared	TestRES28h_LineHoldsTakesNoTolerance
RES-28 a case with no measured figure beside its line passes	TestRES28h_LineHoldsTakesNoTolerance
H-14 shell digest covers the whole tool_input again (re-anchored)	TestH14_ShellDigestCoversTheCommandAlone
H-14 non-shell digest covers only the tool name (re-anchored)	TestH14_NonShellDigestCoversTheWholeInput
H-20 the post payload declares tool_response, and it reaches the debug log (re-anchored)	TestH20_ToolResponseNeverReachesDisk
H-20 the response is measured, and the measurement moves the record's width (re-anchored)	TestH20_ExecutionWidthIsIndependentOfTheResponse
TB the post path never reads the background keys (re-anchored)	TestTestBending_ARunMovedToTheBackground, TestH20_Backgrounded
TB a false backgroundedByUser sets the bit (re-anchored)	TestH20_BackgroundedIsReadForBashOnly

Rows closed

• RES-4: TestRES4_CapAtNAndNPlusOne, TestRES4_ReadAllocBound, TestRES4_TooLargeNamesTheSessionInItsHead
• RES-5: TestRES5_PayloadScannedOnce, TestRES5_TopDefersToJSONValid, FuzzRES5_TopMatchesStdlib, FuzzRES5_IntakeMatchesTheStructDecode
• RES-6: TestRES6_LargeFieldScanIsConstantMemory
• RES-7: TestRES7_HugeFinalMessageIsConstant, FuzzRES7_StreamMatchesStdlib, TestRES7_FinalMessageReadStopsAtTheCap, TestRES7_IntakeStreamRejectsANonObjectRoot
• RES-28: TestRES28h_EveryLineHoldsOnTheReferenceMachine, TestRES28h_LineHoldsTakesNoTolerance

[lutic1]

@k4rthiknerella, the intake logic is correct. The bound and its off-by-one are right (8 MiB accepted, 8 MiB+1 refused), the scanner matches the old decoder, and all three fuzz targets ran for 45 s clean. CI is red, though, and the description doesn't say so. Please fix:

Blocking

1. internal/jsonscan/jsonscan_test.go:90-99 TestTopAllocatesNothingPerValue fails under -race, which is how CI runs it ("Top allocated 1.0 times per run, want 0"). It failed 4 of 6 runs locally and failed "Test" on run 38610392847. json.Valid gets its scanner from a sync.Pool, and the race detector drops pooled items. Skip the test under race (cmd already has race_on_test.go/race_off_test.go), or measure only walkObject. Every PR stacked on this one (#62–#66) inherits the failure.

Should fix
2. The description says go test ./... -race -count=1 passed for every package; CI says otherwise. "Perf gates (linux)" is also red (about 9.5 MiB against roughly 6 MiB darwin rows). That's inherited from #59, but this PR tightens those rows. Please state both.
3. Missing tests:

• hookio.Read with short readers (iotest.OneByteReader) and a mid-stream error (ErrReader).
• readFinalMessage at N and N+1. Its cap at cmd/rashomon/intake.go:73 is now separate from Read's.
• payload_too_large for post and probe, not just Handler.

Nits
4. jsonscan.Lines/Tail have no production caller yet. Say they're staged for later PRs, or drop them.
5. internal/report/text.go:239: "refused it unread" is inaccurate, since the payload is read; parsing is what's skipped. Use "refused it without parsing".
6. hookio/payload.go:213-216: the Strs comment says it fails "before any is used", but it writes earlier fields into dst first.

---

Generated by Claude Code

[lutic1]

@k4rthiknerella, this is a follow-up to the earlier review: an adversarial pass over this PR. Please fix each item, or reply on the PR saying why it doesn't apply. The earlier -race blocker still stands.

Regressions from the old decoder

• cmd/rashomon/intake.go:56-76 readFinalMessage no longer limits how much it reads: it consumes the whole of stdin and only then checks cr.n > MaxPayloadBytes. The old code stopped at Max+1 through io.LimitReader. digest --stdin can now hang forever on an endless stream, and recap's goroutine keeps scanning after its 3 s give-up. Wrap the reader in io.LimitReader(in, MaxPayloadBytes+1).

• hookio/stream.go:18-34: the root-object check uses the first decoded byte, so a top-level string passes as an object. The reviewer confirmed that "", "{" and "{\"x\":1}" all return ok=true, where the old struct decode rejected them. Record the root value's kind, and add these inputs to the bad-input table.

• Key matching is now exact-case, but encoding/json matched struct fields case-insensitively, with the last key winning. Confirmed:

  • {"Last_Assistant_Message":"hi","SESSION_ID":"s"} now gives an empty message and an empty session.
  • With duplicate case-folded keys, a different value wins than before.
  • shape.backgrounded no longer matches Run_In_Background.

  The RES-5 fuzz oracle compares against map[string]any, so it can't see this. Implement struct-decode semantics, or document the change, and fuzz against the old struct types.

• intake.go:48,61-62,73: a truncated session_id (over 4 KiB) rejects the whole payload, and a repeated session_id keeps long set even when the last copy is short. The old decode accepted both. Make the last occurrence win, and decide and test what an over-long id should do.

Accuracy of the "one scan" claim

• Top runs a full json.Valid pass and then a walk; backgroundedCall walks tool_response again; and shape runs a separate Lookup, building a fresh KeySet, for each field. Build the key sets once at package level, scan tool_input once, and correct the "scanned once" wording.
• The non-shell path (Write/Edit) still decodes all of tool_input with json.Unmarshal into any (exempted in res5Allowed), so an 8 MiB Write is fully amplified in memory. Add an RSS gate for that path, and document the remaining amplification.

Robustness

• hookio/read.go:69-75 forces a synchronous runtime.GC() for every payload of 1 MiB or more, on the hook's critical path, justified by a single RSS measurement. Use GOMEMLIMIT/SetGCPercent instead, or gate it on the remaining time budget.
• Payload.Strs pairs []*string with variadic keys by position. A length mismatch panics, which means exit 2 and a blocked tool call, and a reordering silently swaps fields. Take pairs instead, or check the lengths.
• shape.Derive/LeadingDirectory now rely on input that hookio.Intake has already validated, because Lookup/Field never validate. Garbage after the object is accepted. Document the precondition, or require a validated type.
• IntakeStream: when fn stops early, the root check is skipped and the rest is never validated. Return a sentinel, or validate the rest.
• An over-cap payload's coverage record always goes to "unattributed", even though session_id is in the first chunk already read. Stream the heads of session_id and cwd so the record lands in its real run.
• The 64 KiB LastMessageTail ring is captured on every Stop, but nothing reads it. Drop it until RES-17/P-40 lands.
• report/text.go:239 "refused it unread" is inaccurate: the payload is read up to Max+1.

Missing tests

• Top-level string payloads, case-variant keys, an over-long or duplicated session_id.
• readFinalMessage on a reader producing 4×Max bytes, asserting it reads at most Max+1.
• A Post.Capture/Close N+1 case asserting payload_too_large.

---

Generated by Claude Code

[k4rthiknerella]

Thanks for the careful pass. Point by point:

Blocking 1. Fixed and verified: under -race the zero-allocation check measures the walk through Field, Top as a whole is checked only without -race where Valid's pool stays warm, and RES-7's race growth bound is its own 6 MiB, clear of CI's observed 4.32 MiB. and restack #62-#66 on top.

Should fix 2. Updated the description directly: it now says Test was red on the jsonscan allocation check specifically (fixed above), and Perf gates (linux) is red mostly because linux still gates against #59's darwin baseline, plus hook.1MiB/post.1MiB/post.4MiB, which this PR turns on. No code change, this is a description call.

Should fix 3. Added all three: Read under iotest.OneByteReader and a mid-stream iotest.ErrReader, readFinalMessage's own cap tested at exactly MaxPayloadBytes and one byte over, and payload_too_large through Post.Capture and RunProbe the same way Handler already had it.

Nit 4. Right, nothing calls Lines or Tail yet. The package comment now says only Head is used today and the other two are staged for later PRs.

Nit 5. Agreed, it reads up to 8 MiB plus one byte before refusing. Changed the text to 'refused it without parsing'.

Nit 6. Behavior is fine, since every caller discards dst on error, but the comment overstated it. Reworded to say earlier fields may already be written to dst and every caller drops it on error.

[lutic1]

@k4rthiknerella, I re-reviewed the PR at a3f9a9e. All six items from the first review are fixed and verified: go test -race -count=3 passes for jsonscan, hookio and cmd/rashomon, and the new short-reader and post/probe over-cap tests pass. The description is accurate now, apart from citing commits 2a7e2f9/c74f1c6, which aren't on the branch (the real ones are 82dce08/df17877).

The adversarial pass has had no reply, and the intake commit itself is unchanged since the review. These still reproduce on the head:

Blocking

1. readFinalMessage has no read limit (cmd/rashomon/intake.go:57,73). On a reader producing 4× the cap it consumed 33,554,432 bytes; it should stop at 8,388,609. digest --stdin can still hang on a stream that never ends. Wrap the reader in io.LimitReader(in, MaxPayloadBytes+1). The new N/N+1 test reads from a finite strings.Reader, so it can't catch this. Add a test with a 4×Max reader.
2. A top-level JSON string passes as an object (hookio/stream.go:19-34). "", "{" and "{\"x\":1}" all still return ok=true. Listing it under "What could go wrong" doesn't fix it. Check the root value's kind.
3. session_id handling. A 5,000-byte id still rejects the whole payload. A long id followed by a short one also returns ok=false, because intake.go:62 ORs long instead of letting the last copy win. Make the last copy win, and decide what an over-long id should do.

Should fix, or reply with reasoning

• Case-variant keys. "No case folding" is documented, but {"Last_Assistant_Message":"hi","SESSION_ID":"s"} now silently loses both the message and the session. Please reply explicitly, and fuzz against the old struct types.
• The "one scan" claim is still inaccurate. Validation and walking are separate passes, tool_response is walked again, and each Lookup builds its own KeySet.
• Write/Edit payloads still go through json.Unmarshal into any, and that path has no memory gate.
• read.go:75 forces a runtime.GC() on every payload of 1 MiB or more.
• Payload.Strs pairs values by position, so mismatched lengths panic.
• The precondition on shape Lookup/Field is undocumented.
• IntakeStream skips the root check when fn stops early.
• An over-cap record always lands in "unattributed".
• The unused 64 KiB LastMessageTail is still captured.

Missing tests: top-level strings, case-variant keys, over-long or duplicated session_id, and a 4×Max reader.

---

Generated by Claude Code

[lutic1]

@k4rthiknerella, since you don't have Claude Code, I ran the repo's live test test/live/l-items.sh against this PR's head (a3f9a9e). Each run used a real Claude Code 2.1.303 session with its own HOME and store, and main (7b325a0) is the baseline.

• L-1, a session using Bash, Write and an MCP tool: ✅ All 4 calls were recorded and all 4 transcript ids matched, with 0 missing. Coverage was verified, the same as main.
• L-2, another tool's PreToolUse hook beside ours: ✅ The other hook fired and we recorded that Bash call (1/1), the same as main.
• L-3, detach mid-session: coverage unverified, end_recorded false (run 1: run_not_closed; run 2: run_not_closed, transcript_mismatch, execution_mismatch). That's the same end state as main, but it isn't evidence either way.
  • On main, the reasons change from run to run (I ran it 4 times). Claude Code 2.1.303 runs sleep 25 in the background, so the second Bash call lands 0.08–1.1 s later and races detach.
  • l-items.sh needs a fix before L-3 can judge a PR. That's separate from this PR.

Recap line (#61): I ran it twice. Records landed both times (L-1 verified 4/4), but the end-of-turn ※ rashomon line never appeared. That's by design: it only prints when there is something to report. So this run doesn't show the recap path working for #61.

This adds to my review above; it doesn't replace it, and the open items there still stand.

---

Generated by Claude Code

[k4rthiknerella]

Thanks for the second pass. Point by point:

Description. You're right, those two SHAs were never on the branch. The description now names the commits by subject instead: "jsonscan: measure the walk, not Valid's own pool, for zero allocation" and "jsonscan: keep Top's zero-allocation check where the pool is deterministic".

Blocking 1. Fixed in "recap: read the Stop payload to the cap and no further": readFinalMessage now reads through io.LimitReader(in, MaxPayloadBytes+1). I reproduced your 33,554,432 bytes first; TestRES7_FinalMessageReadStopsAtTheCap now feeds it a 4x-cap reader and a reader that never ends, and asserts at most 8,388,609 bytes are taken from either.

Blocking 2. You're right, and listing it under "What could go wrong" was not a fix. In "hookio: match payload keys in any case, and attribute a refusal from its head", IntakeStream looks at the first byte past leading whitespace before anything is parsed, so "", "{" and "{\"x\":1}" are refused with ErrNotObject and the callback never runs (TestRES7_IntakeStreamRejectsANonObjectRoot, plus TestRES7_FinalMessageRefusesATopLevelString on the recap side).

Blocking 3. Fixed in "recap: read the Stop payload to the cap and no further": the last copy of session_id wins, and a copy over 4 KiB now means no session named, not no payload. The message is still read and recap falls back to the newest run, the rule it already applies to a payload without session_id. TestRES7_FinalMessageSessionIDLastCopyWins covers both of your shapes plus a null last copy, which keeps the earlier id, as the struct decode did.

Should fix: case-variant keys. You're right, and calling the old match an encoding/json accident was wrong: every payload struct got it, so I put it back. "jsonscan: match a key as a struct field matches its name, when asked" adds NewFoldKeySet and Path.Fold (exact match first, then Unicode simple folding, last match wins), and intake, backgroundedCall, run_in_background and recap use them, while command and the host fields stay exact because they came from a map decode. Three fuzz targets now hold every field to the old struct or map decode (FuzzRES5_IntakeMatchesTheStructDecode, FuzzRES5_ToolInputFieldsMatchTheirOldDecode, FuzzRES7_FinalMessageMatchesTheStructDecode), clean for 60 s, 30 s and 45 s; the buffered path still reads a repeated key from its last copy, so those targets leave repeated keys out, and the description says why.

Should fix: the "one scan" claim. Agreed, it overstated what happens. I reworded it everywhere it appears (hookio's package doc, Intake, hook's Payload, post, RES-5's test doc and mutant comment, and the description) to say what is true: one read, one json.Valid pass and one walk of the top level, then a walk of tool_input or tool_response for the few keys read from them. Intake's KeySet is now built once per process, and Lookup's one-key set never reaches the heap (go build -gcflags=-m reports []string{...} does not escape at jsonscan.go:371).

Should fix: Write/Edit payloads. Half right: the hook path is gated, since hook.1MiB, 4MiB and 8MiB are Edit payloads whose tool_input carries the whole payload (test/perf/payload/payload.go:88-101), gated at 3.4 MiB per MiB with canonical's decode named as the basis (test/perf/budgets.json:19-21). The post cases pad tool_response instead (payload.go:104-124), so post's own decode of a large tool_input has no row, and the description now lists that gap under "Not in this PR" until RES-48 removes the decode.

Should fix: read.go:75's runtime.GC(). Kept, and the comment now carries the measurement: it holds hook.1MiB at 9.59 MiB against its 9.90 line (10.09 without it), and BenchmarkRead puts its cost at about 0.1 ms at 1 and 4 MiB, against the hook rows' 10 to 15 ms p50 (read.go:82-89). GOMEMLIMIT or SetGCPercent would tune every later collection to save one, and with this line carrying no tolerance I'd rather keep the one collection I can measure.

Should fix: Payload.Strs pairing. Fixed in "hookio: match payload keys in any case, and attribute a refusal from its head": Strs takes Into(key, &dst) pairs, so a length mismatch can't happen and a reorder can't swap fields.

Should fix: the shape Lookup/Field precondition. Documented in "jsonscan: match a key as a struct field matches its name, when asked" and "hookio: match payload keys in any case, and attribute a refusal from its head": Field, Lookup and ValueSpan say they rely on a buffer json.Valid accepted and do not validate, and shape's package doc says Derive, LeadingDirectory, Hosts and Label require one value json.Valid accepts, as hookio.Intake's tool_input span is.

Should fix: IntakeStream when fn stops early. Fixed in "hookio: match payload keys in any case, and attribute a refusal from its head": the root's kind is now checked on its first byte, before fn can be called, and a walk that fn stops early returns ErrStopped, so a caller can't take a half-read payload for a validated one (TestRES7_IntakeStreamReportsAnEarlyStop).

Should fix: the over-cap record's attribution. Fixed in "hookio: match payload keys in any case, and attribute a refusal from its head": the first 4 KiB were already read, so Intake returns a TooLargeError carrying the session_id and cwd they name, and hook, post, probe and paused record the refusal in that run. A head that is cut inside the value, is not a sound start of an object, or names no session still goes to unattributed (TestRES4_TooLargeNamesTheSessionInItsHead, and the three RES-4 entry tests plus a new paused one).

Should fix: the unused 64 KiB LastMessageTail. Not changed: the tail is captured now on purpose, because the spec has it captured beside the head until the comparator lands (Technical spec 8.2.2), so that PR doesn't have to change recap's read. It costs only what it holds: the ring grows by append up to the value's own size (unquote.go:189-190), and fm.Tail is only filled when the head was cut (intake.go:76).

Missing tests. All four added: top-level strings (TestRES7_IntakeStreamRejectsANonObjectRoot, TestRES7_FinalMessageRefusesATopLevelString), case-variant keys (TestRES5_FoldKeySetMatchesStructFieldNames, TestRES5_IntakeMatchesStructFieldNames and the three struct-decode fuzz targets), over-long and repeated session_id (TestRES7_FinalMessageSessionIDLastCopyWins), and the 4x-cap reader (TestRES7_FinalMessageReadStopsAtTheCap). Each new decision has a mutant in RES-4_refusal-head.py, RES-5_fold-keys.py or RES-7_final-message-bound.py, and each one turns its named tests red.

[lutic1]

@k4rthiknerella, I re-checked #61 at af6a631 and I'm approving it.

Confirmed fixed:

• readFinalMessage is bounded. Our 4×cap reader consumed 33,554,432 bytes at a3f9a9e and now consumes 8,388,609. An endless reader returns too.
• A top-level string, a bare string with leading space, and an array are refused before the callback runs.
• session_id: the last copy wins, an over-long id means no session but keeps the message, and a null last copy keeps the earlier id.
• Case-insensitive keys are back (Last_Assistant_Message / SESSION_ID read correctly).
• The "one scan" wording, the Into pairs, the documented precondition, ErrStopped, and the over-cap attribution from the head.
• The answers on runtime.GC, LastMessageTail and the post-side Write/Edit gap are fine.

Build, vet, -race and sweep --check are clean, and Test is green. Perf gates (linux) is inherited from #59. Note that #59's report-only default currently leaves this PR's 1/4/8 MiB cases ungated on Linux; that's being raised on #59.

---

Generated by Claude Code