Bound the hook's worst-case inputs, and make the host a project remembers consistent

[k4rthiknerella]

Stacked on #61 (intake: one bounded read and one scan of every hook payload); review and merge after it.

What closes

The code-quality review's fix list named ten entries against the input side of the hook: a lexer with no token ceiling, a leading-cd fold that re-cleaned the whole path on every step, a nesting depth nobody tested, a baseline file two concurrent renders could still clobber, and a host that could be recorded under five different spellings of the same address. This branch closes entries 10, 14, 16, 17, 28, 29, 30, 53, and 66, plus the spec rows that scheduled the same work: RES-9, RES-10, RES-11, and RES-29.

Token, cd, and nesting bounds

The lexer stops at a token ceiling (RES-9). The ceiling is derived, not guessed: it is 16 MiB divided by the bytes a line costs per token across the whole derive step, not just the lexer's own allocation, because reading a line also re-reads it for the leading-cd fold and for the directory the hook reports. Measured on this branch after the fold was collapsed into a single scan, that cost is 268 bytes per token, which puts the bound at 62,601 tokens. An 8 MiB line of short words, which allocated over a gigabyte before the fix, now exits inside that bound.

The leading-cd DIR && fold had no step limit and re-cleaned the growing path on every step it added, so cost grew with the square of the step count: a 480 KB payload took 10.9 seconds against the hook's own 5-second timeout. The fold now stops at 64 steps (entry 14, RES-10) and walks the path once per step instead of re-cleaning it from scratch. Past 64 steps, the hook reports the starting directory as unknown rather than guessing wrong.

Nested command and arithmetic substitutions had no depth limit either, and the only thing stopping a stack overflow — a fatal error that exits 2, which blocks the tool call — was an unbounded recursive descent. maxNest now has a test on every construct that recurses: a quoted string inside a substitution costs two levels, so the flip between accepted and refused falls between 16 and 17 nested pairs; a substitution inside a substitution, and arithmetic inside arithmetic, flip at 33 and 34 (entry 16).

status_masked had its own square: it re-runs the whole parsed line once per distinctly-named runner it finds, so a line of thousands of them — one make target per occurrence — cost their product rather than their sum. 16,000 plain runners took 4.5 seconds; 4,000 piped to a later stage allocated 4.4 GB. Past 64 distinct runner groups the line now reads as unmasked (null) rather than re-running itself that many times.

Every input bound tested at its edge, not just described

Nine more bounds across the codebase had no test at the edge where they actually matter: a bound could move by one, or the code applying it could read one byte past it, and the full suite would still pass. Entry 17 adds a table test per bound, at the value and at one past it: the store's 16 MiB line length, recap's 64 tracked sessions, spend's 100,000,000-token sanity check (checked across every field a usage record carries), spend's 64 MiB transcript line, and report's 64 Ki-rune message cap and 64 MiB transcript line. The edge tests surfaced something the three line bounds had never stated plainly: each one counts the line's own newline, because a buffered scanner needs room for the terminator to find it, so the longest line actually read is one byte short of the stated bound. The comments on those constants now say so.

A host recorded once, and a baseline that survives two renders at once

Only the ssh destination path ever checked whether a token looked like a hostname. The URL path took whatever punctuation-delimited slice came back from parsing the authority, so curl https://api.example.com=ghp_[REDACTED]!x/ recorded a "host" of api.example.com=ghp_[REDACTED]!x, and nothing bounded its length — a 300,000-byte string was recorded whole. A single predicate, host.Shaped, now covers both paths: a bracketed IPv6 address, or a name of at most 253 bytes built from ASCII letters, digits, -, _, ., or non-ASCII letters, marks and digits, with no empty label (entry 10).

That authority parse had its own square, too: its terminator set skips @ on purpose (the whole authority has to reach the userinfo refusal below) and, past an unclosed [, skips ] the same way, so a line of nothing but one scheme repeated gave it no terminator before the line's own end. A command naming git@ or an unclosed https://[ enough times scanned the whole remaining line at every occurrence — 100 KB of git@ took over a second, and scales with the square of the line. The scan is now bounded to a host name's own length plus room for a port; past that window a candidate is refused outright rather than truncated, because a truncated authority can lose the very @ or ] the wider scan existed to catch.

Canonicalization lower-cases and strips ports, but a real client normalizes an address further before it connects — [2606:4700:0:0:0:0:0:1111] and 2606:4700::1111 are the same address to curl, and so are 167838211 and 10.1.2.3, an inet_aton-style decimal form. Before this fix, the project's novelty baseline treated two spellings of one address as two hosts, so the second spelling always read as new. IP literals now parse once through net/netip and are written in a single canonical form: IPv6 compressed, lower-case, and bracketed; every inet_aton form (decimal, octal, hex, or a short dotted form) rewritten as the equivalent dotted quad; an IPv4-mapped address written as the IPv4 address it maps. Numeric-looking names that are not actually addresses stay names. Five separate spellings of loopback are now recognized as loopback on the parsed address, plus localhost and names under .localhost (entry 30). forget --host canonicalizes its argument the same way records are canonicalized, so forgetting 167838211 now matches a record stored as 10.1.2.3; before this fix it matched nothing and printed "forgot 0 records" while leaving the host on disk (entries 10/30, canonical spellings including forget --host).

Canonicalizing new writes still leaves every baseline and run recorded before this change on disk under the old spelling. The baseline load now re-keys every host it reads through the same canonical form, merging whatever collapses together (the earlier session's ownership, the later of the two last-seen times) and dropping whatever no longer parses as a host; the report's own read of a run's declared hosts does the same before it ever reaches the baseline. Either side left alone would have made a host the project already knew read as new again, every time an old baseline or an old run crossed paths with a new one.

The project baseline that decides novelty had its own race: rendering a report did an unlocked load-modify-save, and two concurrent renders each loaded the file, folded in their own session, and renamed their copy over the other's — in a 200-round reproduction of the race, the second rename always discarded the first render's host. The decision is now a pure fold step wrapped in a locked load-save, using the same bounded file lock the store already uses elsewhere, so a lock that cannot be acquired reports novelty as unknown rather than guessing (entry 28). The host map inside the baseline is now capped at 4,096 hosts (RES-11); past the cap, the least recently seen hosts are evicted first, and a session is never asked to drop a host it just named itself — if one session alone names more than the cap, the render reports novelty as unknown and writes nothing rather than silently dropping that session's own hosts. A baseline file is also read through a size bound now; a file larger than the bound reads as corrupt (novelty unknown) instead of being read whole, and an unchanged render no longer rewrites the file at all (entry 53).

RES-29: bounded, and gated on the right thing

RES-29 is the table that pins every pathological-input case (an 8 MiB line of short words, a 1 MB cd a&& run, nesting one step past maxNest, among others) to exit 0 inside a deadline at a bounded peak RSS. Under the race detector its deadline failed on the 8 MiB cases, and TestRES10_LeadingCdFoldIsBounded failed its 50 ms bound, because both measured wall time. Gates here are allocated bytes and bounded work, never wall time: on the 1 MB cd a&& case the fold allocates 5,810,288 bytes with or without the detector, while its wall time goes from about 7 ms to about 93 ms. TestRES10_LeadingCdFoldIsBounded now asserts that allocation bound and only logs the time; a mutant that raises the leading-cd ceiling to 20,000 moves allocation to 613 MB against the test's 12 MiB budget. TestRES29_PathologicalInputsStayBounded enforces its time bound on uninstrumented builds and logs it under the race detector, as its memory bound already did: nothing in this tree kills a running hook at that deadline yet, so the test measures how fast the hook is, not whether a kill fired.

Two more tests in the same package had the same problem at a smaller scale. The nesting test's own cost is a few KiB regardless of how deep it goes, so it now gates on allocation directly. The here-document join search's cost is a scan over a slice already in memory either way — the search and the scan it replaced cost the same heap — so no allocation bound can tell them apart; it keeps a wall-clock bound, but only outside the race detector's own CPU inflation, the same way the hook-level deadline above already works.

[lutic1]

@k4rthiknerella, the token, leading-cd and nesting bounds work, and a cut never hides a failed test. But the PR says the hook's worst-case inputs are now bounded, and two shapes still blow past the 5 s hook timeout. Please fix:

Blocking

1. internal/shape/masked.go:121: runShells(body, g) runs over the whole line once per runner group, which is quadratic. Measured with Derive:

   • make t0;make t1;… with 16k runners (181 KB): 40.9 s.
   • make tN|tail; with 4k runners (63 KB): 26.6 s and 4.4 GB allocated.

   Cap the groups (e.g. return null when there are more than 64) or evaluate all groups in one pass. Add RES-29 rows for both shapes.

2. internal/shape/hosts.go:445: collect restarts authority() at every scheme match, and @ isn't a terminator. The https://[ branch at :481 has the same problem through strings.Index(s,"]"). Measured with shape.Hosts:

   • 100 KB of git@: 7.8 s.
   • 400 KB of git@: 2 min 4 s.
   • 900 KB of https://[: 5.8 s.

   Limit authority's scan to about maxName+8 bytes, and look for ] only inside that window.

Should fix
3. DeriveCut and HostsCut exist, but the hook calls Derive/Hosts (handle.go:107/115, post.go:181), so the cut flag is never recorded. ssh hosts past the token budget go missing, and the shorter list is presented as complete. Until schema 5 ships, record the ssh list as null when it was cut.
4. Stored hosts aren't migrated. Fold and novelHostCandidates compare keys exactly as stored, so after an upgrade a host stored as 167838211 and later named as 10.1.2.3 reads as novel. Run host.Canonical over keys in load, merging entries, and add a test against a pre-upgrade baseline.
5. TestRubricLLD6_JoinsAreSearchedNotScanned (rubric_lld6_test.go:29) uses a 2 s wall-clock limit and took 4.2 s under -race. That contradicts "gated on allocation, never wall time". Gate it on allocation.

Nit
6. Old records and baselines can hold host values like api.example.com=ghp_…, and nothing cleans them up. Worth a release note, or a cleanup pass.

---

Generated by Claude Code

[lutic1]

@k4rthiknerella, this is a follow-up to the earlier review: an adversarial pass over this PR. Please fix each item, or reply on the PR saying why it doesn't apply. The earlier quadratic-time blockers in masked.go and hosts.go still stand.

Bounds that don't hold

• hosts.go:185/201 sshCommandHosts: the shared budget only decides whether the next line gets read, and each line is still lexed up to the full maxTokens. Worst case is about 2× the bound, outside the 268 B/token budget. Pass the remaining budget into each line's lexer.
• tokenize.go:582 tokenEstimate counts blanks without regard to quotes, so one quoted 8 MiB string with spaces makes every lex call preallocate maxTokens+1 tokens, across 5 or more readings. Track quotes or cap the initial capacity, and add this shape to pathological.Lines.
• handle.go:202 startDirectory → LeadingDirectory decodes and lexes the whole line again, even though Derive already computed leadingRun, and Hosts lexes it once more. The budget assumes 3 readings; the hook does at least 5. Reuse the decoded string and Derive's result.
• res29_test.go: the peak is 59.5 MiB against a 64 MiB limit, measured only on darwin/arm64, with no wall-time check under -race. Add a portable allocation bound per reading, run the test on linux/amd64 in CI, and fail at about 80% of the limit.

Host canonicalisation

• host.go inetAton inside Canonical rewrites every all-numeric host: https://1234/ becomes 0.0.4.210, and 0 becomes 0.0.0.0, which Loopback then treats as this machine. Numeric ssh aliases would be rewritten too. Limit inet_aton folding to the inputs where clients actually do it, and test numeric aliases.
• host.go:119-129 vs :231: Canonical accepts zoned IPv6 ([fe80::1%eth0]) but Shaped rejects it. The link-local host is then dropped from declarations while still being compared on the sandbox side, which gives a false "reached but never named". Pick one rule, and test Shaped(Canonical(x)) for every accepted x.
• fold.go:59 / destinations.go:198: Fold keys on the exact stored string, so pre-upgrade spellings (167838211, uncompressed IPv6) read as novel and fail to match the sandbox trail. Re-key through host.Canonical in load(), merging entries. (This was also raised in the earlier review.)

Baseline behaviour

• destinations.go:200: ErrTooManyHosts, ErrLockTimeout and errTooLarge all render as "could not be read or written", so users may delete a healthy baseline. Give each its own reason.
• Result.Evicted is computed but never shown, so a host that returns after eviction at the 4096 cap reads as novel. Surface it in the report.
• fold.go:38 maxEntryBytes = 1 KiB doesn't cover the worst-case JSON escaping of a 256-byte session id (about 6×). Once the file exceeds maxBaselineBytes, every later save is refused and the project's baseline is stuck permanently. Derive the size from the worst case, or evict until it fits.
• clearBaselines (main.go:1824) takes its ReadDir before taking the per-file locks, so a baseline created by a concurrent render is never visited, and the forgotten host survives. Hold baseline/.lock across both.

Cut flag

• HostsCut has no callers, so an ssh list cut at the bound is presented as complete. Wire it in, or drop it until schema 5 (also raised earlier).
• hosts.go:185-193: cut is set when the rest is only blank lines, and is not set when a line stopped as uncertain or unterminated. Use TrimSpace(rest) != "", and decide whether an uncertain stop also counts as a cut.

---

Generated by Claude Code

[k4rthiknerella]

Thanks for the careful pass. Point by point:

Blocking 1. Reproduced: 16k make tN; runners took 4.5s and 4k make tN|tail; allocated 4.4GB here too. statusMasked now caps at 64 distinctly-worded runner groups and reads the line as unmasked (null) past that, before the loop that re-runs it per group.

Blocking 2. Confirmed: 100KB of git@ took over a second here, and an unclosed https://[ the same way. authority now bounds its scan to a host's own length plus room for a port, and refuses rather than truncates past it, so a cut never drops an @ or ] it should have seen.

Should fix 3. The cut is already stated in store-schema.json's ssh_hosts description and lands as shape.truncated at schema 5 (DeriveCut/HostsCut). I'm not nulling the list in the meantime: record.go:134-137 defines null as 'the call named none', so that would swap a short-but-true list for a false empty one. No code change.

Should fix 4. Confirmed: a baseline holding 167838211 makes 10.1.2.3 read as novel, and a pre-upgrade run re-rendered does the same one level up. load now re-keys every baseline host through host.Canonical/host.Shaped, merging what collapses together and dropping what no longer parses, and declaredHosts canonicalises an old run's own declared hosts the same way before novelty ever sees them.

Should fix 5. Right, that gate didn't belong there. The join search's cost is a scan over a slice already in memory either way, so allocation can't tell it apart from the regression -- it keeps a wall-clock bound, but only outside -race, the same way cmd/rashomon's own RES-29 deadline already works. The nesting test's cost is allocation-correlated, so it moved to that gate directly.

Nit 6. The baseline side clears itself out with the load-time re-key from the point above. For old run records I'm leaving the raw text alone -- no rewrite -- since records are moving to carrying full commands anyway, which will hold the same tokens regardless. No code change.

Also, found while rebasing. forget --host records the canonical spelling, but declaredHostsFiltered and sshDeclaredFiltered asked forgotten() about the stored one, so a declaration in a legacy spelling, or any ssh host, came back after a successful forget. Both canonicalise first now, with a test for each case.

[lutic1]

@k4rthiknerella, I re-reviewed the PR at 2719cac. Both blockers are fixed. Measured on linux/amd64:

Case	Before	Now
make tN; ×16k	40.9 s	137 ms
make tN|tail; ×4k	26.6 s	36 ms
400 KB git@	2 min 4 s	49 ms
900 KB https://[	5.8 s	18 ms

I also tried to get around the group cap, and the time stays linear. The stored-host migration in load is good work. The LLD6 change is fine.

Correction: our adversarial item about zoned IPv6 was wrong. plausible refuses %, so Canonical never produces a zoned host. Please ignore it.

Still blocking

1. inetAton rewrites hosts made only of digits (host.go). Canonical("1234") gives 0.0.4.210, and "0" gives 0.0.0.0, which then reads as loopback. Your migration now runs old baseline keys and old ssh declarations through this too, so numeric ssh aliases get rewritten. Limit the folding to inputs where clients actually do it, such as URL authorities, and test numeric aliases.
2. Fix the timings in the description and your reply. Both say 16k runners took "4.5 s", which matches neither the 40.9 s before nor the 137 ms now.

Adversarial pass: no reply yet. Please fix or answer each item:

• sshCommandHosts lexes each line to the full budget, so it can use about 2× the bound (hosts.go:183-193).
• tokenEstimate ignores quotes (tokenize.go:582). Add a quoted-blanks row to pathological.Lines.
• startDirectory re-lexes the line (handle.go:201-202).
• RES-29 is measured only on darwin, with no Linux gate and no margin.
• Baseline errors share one message (destinations.go:221).
• Result.Evicted is never shown.
• maxEntryBytes doesn't cover escaping, so the baseline can get stuck permanently.
• clearBaselines lists files before taking the lock (main.go:1826).
• cut is set on a blank remainder and not set on an uncertain stop (hosts.go:189).

HostsCut/DeriveCut still have no callers. I accept "null means named none", but say in the description that a cut ssh list is still stored as complete.

---

Generated by Claude Code