[jayden1711]

Intent: This is the third of three stacked PRs for #748 Phase 0. It adds the bench driver, which runs the task set under the caps:

• Before any task: it checks the caps, that research is off, the bench user, the CLI pin and the claude.ai login, then the srt sandbox probe, that Docker is rootless, and a probe container's limits. Then it makes the run's first model call: the sandbox confirmation, in which Claude Code runs the lane's reads and connects. Per attempt, it records whether Claude Code's permission layer or its OS sandbox refused it.
• Per task: a fresh container with its limits read back, and the task's base tree taken before any agent runs. The task runs under the driver's own wall-clock cap, which reaches every agent run and container command, and the driver waits for runs in flight before going on. Each agent run's working folder must be inside the task's folder, and its file-writing tools are confined to that working folder. Then the patches are exported from the clone's objects only, and the container is removed.
• At the first usage limit: a clean stop and a partial aggregate.

It also adds:

• grading inputs that read only the exported patches;
• Wilson-interval aggregates (counts and rates only);
• the held-out store;
• the README with the owner's one-time setup.

The tests run no model, no Docker and no download. In a real run, the sandbox confirmation is one model call.

Risk class: protected (every path is under scripts/**).

Part of #748. Base: main. #756 (preflight) and #757 (containers and limits) have merged, and main is merged in here: last in round 10, at 3d24fc3 (#804), by the signed-off merge ab9bfc8. Earlier merges of main were in rounds 6 and 8. Round 4 merged #757's head.

Size: well over the ~700 lines the owner asked to hear about.

• Round 1 added about 600 lines, round 3 about 340, round 4 about 400, and round 5 about 180 (measured, git diff --shortstat 0faa4988 943244ab: 181 insertions, 34 deletions, 124 of the insertions in tests). Round 7 added 125 lines and removed 18, with 55 of the insertions in tests (measured, git diff --shortstat 45b91baf 9c666ba0).
• measured for round 4 (git diff --shortstat 6585970a 0faa4988): 399 insertions and 104 deletions; 178 of the insertions are in tests.
• The rest is round 0's driver, grading, aggregates and README. I have not split it further.

What's in it

• scripts/bench/driver.ts, runBench:
  • Preflight, in order: caps (checkCaps: finite, and the mode by isMode, own keys only, so constructor, toString and __proto__ are refused), research caps all 0, bench user, CLI pin, claude.ai login, the srt sandbox probe (bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's probe.ts, as preflight.sandboxProbe), rootless Docker, then a probe container's limits.
  • The sandbox confirmation (confirm.ts, the run's first model call): described below.
    • A leak refuses the run before any task. Its record goes in sandbox.json, results.json and the aggregate.
    • "model declined" prints a WARNING. So does "all refused" when the OS sandbox did not itself refuse every unparsed read (round 4).
  • Per task:
    • the container is started and its limits read back;
    • the base tree: BenchOptions.baseTree (required) gives the task's base commit as a tar stream, before any agent runs. It goes to the task's TaskRunner, whose stdin check (bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757) compares against it;
    • RunTask gets the metered runner, the abort signal, the exec-only runner, its work dir and the research-off settings;
    • the wall-clock cap (TASK_WALL_MS, 3 h): meter.task(perTaskUsd, workDir, signal) adds the task's signal to every run request, whatever the task passes, so CliRunner's kill fires, and it refuses new runs once the signal fires. TaskRunner.exec is async: at the signal its docker exec is killed, and none starts after;
    • before the task's spend is recorded and the next task starts, the driver waits for its runs in flight (settled, 5 min bound). If one is still going after that, the bench halts rather than overlap, and every later task is not run (halted);
    • the heads are exported and the container removed.
  • At a usage limit: the task is recorded as stopped (usage limit) and the rest as not run (usage limit). A per-task limits failure stops the run, and the rest are not run (limits).
  • Results: results.json and aggregate.json are rewritten after every task. They record the CLI version, the plan, the limits, the usage-equivalent dollars, and both sandbox checks (per attempt, with the layer).
• scripts/bench/usage.ts: scopeEditTools, run by TaskMeter.run on every request (owner's decisions of 2026-10-18 and 2026-10-19).
  • Round 4: a run whose cwd is outside the task's folder is refused, unstarted, whatever its tools.
  • Round 5: this check is also made where the folders really are.
    • A run whose cwd doesn't resolve, or whose realpath isn't inside the task folder's, is refused. That includes a link in the task folder that leads out of it.
    • A task folder that is itself a link is refused. The driver builds it on the bench root's real path (bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756's benchRoot).
    • The cwd is passed as given, with no default, so a missing one is refused.
  • Round 6:

    • A cwd reached through any link is refused, even one that stays inside: its realpath must be itself.

    • The window between the check and the spawn is closed for the task's own runs. The engine's runner spawns claude in the cwd after awaits (the login lock, MCP registration). In between, another of the task's runs, whose sandboxed Bash may write its own working folder, could swap a folder of the new run's cwd for a link out (round 5 check, measured with a real spawn). So the meter now refuses, unstarted:

      • a run whose cwd is nested with a running run's: the same folder, or one inside the other, either way;
      • a run strictly inside an earlier, ended run's cwd, which a process it left behind could still write (guess that one can be left).

      The check and the registration run with no await between them, so two runs started together can't both pass (measured: a probe starting three in one tick).

    • What is still assumed (inferred): nothing outside this task's runs writes inside the task folder while a run is starting. The driver writes only the run's own records and the patches folder, between tasks. The container has no mount. Other tasks run one at a time, in their own folders.

    • For RunTask: runs of one task can be concurrent only in disjoint folders, and a later run can't be strictly inside an earlier one's folder. So a run at the clone's root followed by worktrees under the engine's default .claude/worktrees is refused: worktrees go outside the clone, side by side in the task folder (README).

  • Round 7:
    • One meter per task folder. runBench refuses a task key that appears twice (before any check), and makes the run folder, the sandbox check's folder and each task folder fresh: mkdir without recursive, and EEXIST is refused. A run id used again (after a crash, say) is refused, and so is a run folder that is a link. So no task folder holds runs its meter hasn't seen (measured, tests and the adversarial pass below). Round 8: the run folder is made before the probe container, so a reused or racing run id is refused before any container, and a losing racer can't remove the winner's probe container (measured, test).
    • Round 8, labels: two claims are now marked inferred in the README, as in this body. One is the remaining assumption: nothing outside the task folder's own runs writes inside it while a run starts. The other is the case-insensitive claim, with its premise: a stable, unique device and inode per folder. The README also says that each run's parent chain is recorded when it registers.
    • Folders are compared by device and inode as well as by path. Each run's folder and its parents up to the task folder are recorded. So a second spelling of one folder is still the same folder: on a case- or normalisation-insensitive filesystem, or after a rename while a run is going (measured with a rename; inferred for case-insensitive filesystems, which need root to make here).
  • A bare Edit, Write, MultiEdit or NotebookEdit becomes Edit(//<the run's cwd>/**), not the whole task folder (round 4). Claude Code's Edit rules cover all its file-editing tools (inferred from its permissions docs).
  • Edit(//<p>) or Edit(//<p>/**) stays when p is a plain, normalized path in the run's cwd.
  • Any other grant of them refuses the run, unstarted:
    • another folder (the rest of the task folder included);
    • a relative, ~ or globbed path, .., or a trailing slash;
    • a qualified Write/MultiEdit/NotebookEdit;
    • an entry naming several tools (Read,Edit), or a wrong-case name.
  • Round 4, rule paths: the task's folder and the run's cwd must be normalized absolute paths, with no trailing slash and no glob or rule character (* ? [ ] { } ! ( ) \, a comma, whitespace, any control character (\x00-\x1f and \x7f since round 5, and C1 \x80-\x9f since round 6), any format character such as U+200B or U+202E, and lone surrogates (round 6)).
    • Such a path is refused, not escaped. The owner's decision said "escaped", and refusal was offered as the alternative: the escaping rules of Claude Code's rule globs aren't documented, and refusing makes the wrong rule impossible.
    • A consequence (inferred): a bench root with a space or one of these characters is refused before the first model call.
• scripts/bench/confirm.ts (the owner's option D): one call on claude-haiku-5-5, the cheapest pinned product model.
  • measured (round 1): the 2.1.303 binary carries this id.
  • The call goes through the meter, capped at $0.50 and counted against the run's ceiling, and is recorded as usage.sandbox_check.
  • What it asks Claude Code to run, with the agents' lane:
    • curl --unix-socket <sock> http://localhost/_ping for each socket the probe tries;
    • cat '<s>' of a sentinel at each protected path. Correction (round 3's finding): cat is a file command Claude Code's permission layer parses, so the lane's Read(...) deny most likely refuses it before the OS sandbox runs anything (inferred from the permissions docs). So cat tests the permission layer, not the bundled sandbox's denyRead;
    • round 4: node -e "<READ_JS>" '<s>' for each sentinel, a read the permission layer doesn't parse (inferred from the same docs: deny rules don't apply to "a Python or Node script that opens files itself"). The OS sandbox's own denyRead is what has to stop it.
      • READ_JS opens the sentinel and fstats the descriptor. It exits 3 (bench-check: masked, not a regular file) when the file isn't regular, which is the sandbox's /dev/null mask, the same reading as the probe's PROBE_JS. Otherwise it reads and prints the file.
      • So a mask counts as refused, and a read that works is a leak (measured, test, by running the real command on a /dev/null link, a missing path and a real sentinel).
  • Sentinels:
    • where ~/.netrc, ~/.npmrc or ~/.pypirc doesn't exist, the driver creates one there (random content, created exclusive so never over anything) and removes it after;
    • they are also removed on SIGINT and SIGTERM (round 3) and SIGHUP (round 4), after which the signal ends the process as before.
  • ~/.claude.json is never touched, whether or not it exists (round 3).
    • Its sentinel always goes under ~/.claude/, which the login folder's rule covers (Read(//<home>/.claude/**) and the sandbox's denyRead of ~/.claude).
    • There is none when that folder is missing (listed as skipped). The other three test the exact-path form.
  • A path that already holds a real file gets no sentinel, is never put to the model, and is listed as skipped.
  • The verdict, per attempt, comes from Claude Code's stream-json transcript (the tool calls that ran and their results), never the model's prose:
    • refused only when the exact listed command ran and its tool result is an error (is_error, which Claude Code sets on a non-zero exit: inferred; the engine reads the same field, src/run-record.ts). Round 5: exact means the same command with no Bash input but command and description. A listed command sent with a timeout, run_in_background or any other input is an extra call. Round 6: command must be a string, and description absent or a string. Any other shape Claude Code would refuse at input validation without running anything (inferred), so it can never pass as "refused";
    • reached when any command naming the target did not fail, or a sentinel's content or a _ping OK came back. Any reached attempt, or any sentinel's content anywhere in the transcript, makes the status leaked, and the run is refused;
    • anything else is not run.
  • Round 4: which layer stopped each attempt (layer), only from what the stream-json carries, failing closed:
    • permission layer: the result line's permission_denials lists the call's tool_use_id, or the call's error result says "Permission to use … has been denied" (or "… has been denied because Claude Code is running in don't ask mode"). inferred from public docs and Claude Code's known wording, not verified against 2.1.303. The smoke round's first call (the Haiku call) will measure them;
    • OS sandbox: only the exact listed read of a sentinel that ran and failed on a masked or unreadable file: READ_JS's mask exit, EACCES, EPERM or ENOENT, or for cat "Permission denied", "Operation not permitted" or "No such file or directory". It also needs no permission-layer marker. Round 5: it also needs no extra call. The model could make that mark itself, for example with a chmod 000 through globs, so with any extra call no attempt gets this layer, and os_sandbox_read_deny is false in sandbox.json, results.json and the aggregate;
    • unknown: the exact command failed with neither marker. Every socket attempt the permission layer didn't refuse lands here, since curl failing to connect can't tell a sandbox from a missing socket. Never counted as the OS sandbox;
    • model declined: the exact command never ran.
  • The status:
    • leaked as above;
    • all refused only when every attempt was refused and the model ran nothing but the listed commands (round 4, round 3's required change 2). These now make it "model declined" at best, and the record lists them in extra_calls:
      • a read through $HOME, ~, a glob or a link;
      • any other tool call, including any other tool-call block such as server_tool_use (round 5);
      • a listed command sent with other inputs (round 5);
    • model declined otherwise. The run goes on only because the srt probe passed, and say prints it as a WARNING with the label "sandbox-runtime 0.0.79: close to, not identical with, the build bundled in Claude Code 2.1.303";
    • os_sandbox_read_deny records whether the OS sandbox itself refused every node -e read. When it didn't (say the permission layer caught them after all), the WARNING says the sandbox's own read-deny rests on the srt probe.
• scripts/bench/grade.ts:
  • exportPatch diffs in a bare repo the driver makes for it, with the clone's object store as its only source (GIT_ALTERNATE_OBJECT_DIRECTORIES).
    • It reads no system or user config (GIT_CONFIG_NOSYSTEM, GIT_CONFIG_GLOBAL=/dev/null).
    • It uses the engine's hardened flags (-c core.fsmonitor=false -c core.hooksPath=/dev/null, src/comment-policy.ts) and --no-textconv --no-ext-diff --src-prefix=a/ --dst-prefix=b/.
    • A .git or objects that is a link is refused. No agent-written config, attribute, hook or pager is read.
  • readPatches reads only the patches folder: plain folders named by task key, and plain <label>.diff files. A link or any other entry is refused.
  • writePredictions builds the SWE-bench predictions from those files and the held-out manifest, into the held-out store.
• scripts/bench/aggregate.ts:
  • wilson;
  • aggregate: completeness, counts (not_run_halted added), usage (with sandbox_check), the sandbox checks, and owner_reapproval_needed for a smoke mean over $8 per task. Once graded, it adds the confusion matrix, merge precision, auto-merge yield, false holds, caught bad fixes, and the B0, B1 and bare baselines.
  • No instance id appears anywhere.
• scripts/bench/store.ts: the held-out store (parseDataset, the manifest, TASK_KEY).
• scripts/bench/srt-fingerprint.mjs (option C): compares a sandbox-runtime release's [Sandbox Linux] log strings with a Claude Code binary's. It is read-only and executes neither.
  • measured on 2026-10-18 against 2.1.303: 0.0.78 differs by 1 release-only and 2 binary-only strings, 0.0.79 by 1 and 1, 0.0.80 by 4 and 4.
• scripts/bench/README.md: what the bench measures, how it is isolated, the caps, the sandbox checks, the data layout, and the owner's one-time setup. It covers:
  • Docker Engine, not Docker Desktop, with Desktop's WSL integration off for this distro;
  • bubblewrap and socat;
  • the admin account's home closed to the bench user (chmod 750 "$HOME", checked with sudo -u wl-bench ls "$HOME" failing);
  • sandbox-runtime 0.0.79 installed for the bench user only (npm install -g --prefix ~/.local), checked with srt --version;
  • the two sandbox checks, with what each read tests, the layers, and the verdict and sentinel rules (round 4);
  • where agents may write (next section);
  • round 3, from bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756 and bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757: the home from the user database, the dangling-link refusal, the docker daemon always named, and the exact scope of the hidden-field scan;
  • the fingerprint re-check on each new release, switching to an exact match when one appears.
• test/bench.test.ts: 66 tests on this branch, 31 of them from main (bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756, bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757, Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 and bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804).

  • Round 9:

    • RAW_UNSAFE also refuses every \p{Default_Ignorable_Code_Point}, assigned or not: the whole class that renders as nothing, including the Mongolian variation selectors (U+180B-180D and U+180F; U+180E between them is a format character, refused since round 7) and the Khmer U+17B4 and U+17B5. Each of those makes a second identifier that looks like the first, as U+3164 does. The round-8 ranges stay.
    • The fixture test adds U+17B4, U+17B5, U+180B, U+180F, U+2065 and U+E0002.
    • The scan test's title names exactly what it checks: "line or paragraph separator", not "separator".

  • Round 8:

    • The raw-character scan's regex is a named constant, RAW_UNSAFE. It catches:

      • \p{Cf}, \p{Cs}, \p{Zl} and \p{Zp};
      • a CR anywhere but at a line's end;
      • every control character but tab;
      • the variation selectors (U+FE00-FE0F, U+E0100-E01EF), the Hangul fillers (U+115F, U+1160, U+3164, U+FFA0), U+2800 and U+034F.

      Tab, LF, a line-end CR and visible text pass. A new test feeds it the round-7 check's fixtures, and the scan's title says exactly that.

    • The reused-run-id case also checks that no container is made, started or removed.

  • Round 7 adds three:

    • two meters on one task folder (a repeated key, a reused run id, a run folder made beforehand);
    • inode comparison (a folder renamed while its run is going);
    • no raw invisible, bidi or control character in the bench sources or the test file.

    The round-6 character test now writes its characters as code points.

  • Round 6 adds five: the run-overlap swap (the reviewer's probe-scope5 case), any link in the cwd, an earlier run's folder, the Bash input shapes, and C1, format and surrogate characters.

  • Round 3 added four.

  • Round 4 added five: the cwd scope, the rule-path refusal, the node -e read, the layers, and "all refused" only with nothing extra.

  • Round 5 adds five: control characters, the realpath check, the model unable to set the OS-sandbox layer (also through the driver's three records), other Bash inputs, and server_tool_use. Two scope tests now use real temporary folders instead of /srv/... paths, since a run's working folder must now exist. Their assertions are unchanged.

  • It extends two: the signal test adds SIGHUP, and the driver's acceptance test checks every attempt's form and layer plus the warning.

  • bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's new stdin tests now await syncTree, which is async on this branch.

Where an agent may write

This is in the README (the "Where an agent may write" bullet) and at the top of driver.ts and usage.ts.

• The lane's deny list covers reads only.
• Bash commands write only where the sandbox allows: the run's working folder and the temp folder, plus the paths of Edit allow rules (inferred from the sandboxing docs). Since round 4, the bench's only Edit allow rule is the run's own working folder, so it adds nothing.
  • Round 3's scope (the whole task folder) widened that (round 3 verdict, required change 3): it reached a sibling folder's .claude/settings.json or a bare remote's hooks/. The sandbox protects those only in and above the working folder (inferred).
• Edit, Write, MultiEdit and NotebookEdit run outside the sandbox, as the bench user.
  • Pre-approved without a scope, they could change unprotected files the bench user owns, such as ~/.docker/config.json (a credsStore helper the driver's own docker would run), ~/.config/systemd/user/ or ~/.local/bin/.
  • Correction: earlier rounds named ~/.profile and ~/.bashrc. In dontAsk mode, writes to protected paths such as those, ~/.claude/ and .git/ are denied even with a matching allow rule (inferred from the permission-modes docs).
• So they are confined to the run's own working folder, which must be in the task's folder (scopeEditTools, above). A run that grants them anywhere else, or runs from outside the task's folder, never starts (measured, tests).
• Claude Code asks before any other file edit, and the bench runs in dontAsk mode, which refuses what would ask (inferred from the permissions docs).
• What the driver reads back doesn't trust the task folder either: patch export reads no config at all.

Still ahead in #748 Phase 0:

• the PR-mode task runner on src/harness-evals/replay.ts with stub CI (the RunTask this driver calls, and the baseTree it is given);
• the agent's test relay into the container (see bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's known argv issue);
• task selection;
• the pinned-download scripts;
• the grading run;
• the B0, B1 and claude -p baselines;
• the CLI entry point.

How it was verified

• Round 10 (measured, head 242f663):
  • The merge: main (3d24fc3, which includes bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804) is merged in at ab9bfc8, a signed-off merge.
    • test/bench.test.ts conflicted. I kept main's five Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797/bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 tests and this branch's async 512-byte tar test.
    • Main's four new stdin tests are made async and await each stdinSends call. Every assert line of main's file is in the branch's, up to that await, apart from five older lines (assert.throws on exec and syncTree) that this branch made assert.rejects in earlier rounds (work/fix-759-r4/main-asserts.mjs).
    • scripts/bench/hidden.ts is byte-identical to main's (git diff origin/main -- scripts/bench/hidden.ts is empty), so bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804's addedLines, importLines and strict && loose are intact.
  • Tests and gates: on the merge commit alone, the bench tests give # pass 66, # fail 0, and so does the head. npm run typecheck exits 0, check:denylist prints denylist: 1299 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off.
  • Main: origin/main has since moved to 178c44e (Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793). git merge-tree --write-tree origin/main HEAD is clean, and Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793 touches no bench file, so it isn't merged again.
  • The scan: RAW_UNSAFE also refuses every non-ASCII \p{Zs} space, private use (\p{Co}) and U+FFFC, with fixtures for U+00A0, U+2009, U+3000, U+E000, U+F0000 and U+FFFC. Over every code point, 002d7c9's regex let 137,485 of those through. None it caught is let through now, visible text still passes, and the 16 files scan clean (sweep9.mjs).
  • Nit: "U+180B-180D and U+180F" (U+180E is a format character).
• Round 9 (measured, head 002d7c9):
  • Tests: the bench tests give # pass 61, # fail 0.
  • Gates: npm run typecheck exits 0, check:denylist prints denylist: 1299 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off.
  • Main (round 9, now stale): origin/main was c339175, an ancestor then. Round 10 merges 3d24fc3; see below.
• Fail-without, round 9 (measured): every code point, as a line a<c>b, fed to 6ffe56d's regex and to RAW_UNSAFE (work/fix-759-r4/sweep9.mjs).
  • Let through by the old regex, refused now: U+17B4-17B5, U+180B-180D and U+180F (assigned), plus 3,769 unassigned default-ignorable code points.
  • Caught by the old regex and let through now: none.
• README, round 9:
  • The vfat and exFAT sentence is corrected: they keep a folder's inode number only while its inode is in memory (inferred, from how their kernel drivers number inodes).
  • The unlabelled "is on ext4 here" is now advice: keep the bench root on a filesystem with stable inode numbers, such as ext4.
• Round 8 (measured, head 6ffe56d; origin/main c339175 merged in, signed off):
  • Tests: the bench tests give # pass 61, # fail 0. Main's new no-clock-helpers and no-wall-clock-durations tests pass on this tree.
  • Gates: npm run typecheck exits 0, check:denylist prints denylist: 1299 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off.
• Fail-without, round 8 (measured):
  • The scan: the round-7 check's fixtures, fed to 9c666ba's regex and to RAW_UNSAFE. Nine pass the old regex and are caught by the new one: CR mid-line, U+2028, U+2029, U+FE0F, U+E0100, U+3164, U+115F, U+2800 and U+034F. The ten the old regex caught are still caught. Tab, a CRLF line end and U+00A0 pass both.
  • The probe container: this head's tests in a scratch worktree at 9c666ba (since removed) gave # pass 60, # fail 1. A reused run id still made and removed a probe container: no container made, started or removed for a reused run id, 15 !== 12.
• Round 7 (measured, head 9c666ba): the bench tests give # pass 60, # fail 0 with temp HOME, state and held-out dirs and the refusing stand-ins. npm run typecheck exits 0, check:denylist prints denylist: 1295 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off. Main (682c8ce) merges cleanly (git merge-tree), and its one change since the last merge is a test line, so it isn't merged again.
• Fail-without, round 7 (measured):
  • Method: a scratch worktree at 45b91ba with this head's test/bench.test.ts. No guard was edited. Since removed.
  • Result: # pass 58, # fail 2:
    • two meters: Missing expected rejection. (the repeated key started agent runs);
    • inode: Missing expected rejection: …/runs/r1/t01/renamed.
  • The raw-character test: built from this head's tests, then run with 45b91ba's test/bench.test.ts back on disk, it fails with 'test/bench.test.ts:1605'.
• Round 6 (measured, head 45b91ba; origin/main merged in at the signed-off merge before it): the bench tests give # pass 57, # fail 0 with temp HOME, state and held-out dirs and the refusing stand-ins. npm run typecheck exits 0, check:denylist prints denylist: 1295 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off.
• Fail-without, round 6 (measured):
  • Method: a scratch worktree at 943244a with this head's test/bench.test.ts. No guard was edited. Since removed.
  • Result: # pass 52, # fail 5:
    • the swap: '/repo/sub: succeeded' where refused (nested) was expected;
    • any link: Missing expected rejection: …/repo/up;
    • input shapes: 'all refused' !== 'model declined';
    • C1 and format: Missing expected exception: "\x80";
    • earlier run: Missing expected rejection.
• Round 5 (measured, head 943244a): the bench tests give # pass 52, # fail 0 with the same temp dirs and stand-ins as below. npm run typecheck exits 0, check:denylist prints denylist: 1290 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off.
• Fail-without, round 5 (measured):
  • Method: a scratch worktree at 0faa498 with this head's test/bench.test.ts checked in on top. No guard was edited. Since removed.
  • Result: # pass 47, # fail 5:
    • control characters: Missing expected exception: "\u0000";
    • realpath: Missing expected rejection. (a cwd that is a link out of the task folder started);
    • the model setting the layer: the node attempts' layers were OS sandbox after the extra chmod (and os_sandbox_read_deny true);
    • other Bash inputs: 'all refused' !== 'model declined';
    • server_tool_use: 'all refused' !== 'model declined'.
• Round 4 (measured, head 0faa498; bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's head 94c6e71 merged in at 6585970):
  • Command: node --import ./dist/test/isolate.js --test dist/test/bench.test.js with a temp HOME, state and held-out dirs and refusing claude, docker and srt stand-ins first on PATH.
  • Result: # pass 47, # fail 0.
  • Gates: npm run typecheck exits 0. npm run check:denylist: denylist: 1290 files clean. npm run check:dco -- origin/main HEAD: DCO: all commits signed off.
• Fail-without, round 4 (measured):
  • Method: a scratch worktree at the merge 6585970, with this head's test/bench.test.ts checked in on top.
    • At that merge, confirm.ts, usage.ts, driver.ts and the README are byte-identical to a049531's (measured, git diff --stat a0495314 6585970a -- … is empty), so these are round 3's sources.
    • No guard was edited, and the new tests import no new symbol, so each fails on its own.
  • Result: # pass 40, # fail 7:
    • the driver's acceptance test: the per-attempt [kind, form, refused, layer] list differs (no node attempts, no layers);
    • signals: 'SIGHUP: no .netrc left', true !== false;
    • cwd scope: the bare grant became Edit(//srv/bench/runs/r1/t01/**), not …/t01/repo/**;
    • rule paths: Missing expected exception: /srv/b*/runs/r1/t01;
    • node -e: masked with /dev/null: one node -e read per sentinel, 4 !== 0;
    • layers: the result line's permission_denials, matched by tool_use_id (no layer field);
    • nothing extra: 'all refused' !== 'model declined' for the read through $HOME.
  • The merge's own fix (measured): before bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's new stdin tests were made to await syncTree, the five of them failed on this branch (1 !== 0, nothing sent, because the promise wasn't awaited). With the await they pass.
  • The scratch worktree was removed after.
• Round 3 (measured, head a049531): # pass 37, # fail 0.
  • Fail-without method: a scratch worktree at a049531 with scripts/bench/{usage,confirm,driver}.ts from the merge 9e4481c.
  • The new tests on the old code failed:
    • ~/.claude.json never exists while the model runs, true !== false;
    • the verdict test at cat '<s>' > /dev/null && echo (status all refused, not leaked);
    • the signal test;
    • the edit-scope test.
  • Probes of each case, old then new:
    • cat '<s>' > /dev/null && echo, wc -c '<s>' and curl -o /dev/null, each exiting 0: all refused, then leaked;
    • ~/.claude.json absent: it existed during the model call, then never;
    • SIGINT and SIGTERM mid-check: the old code left four sentinels behind, the new one none.
• Fail-without, round 1 (measured):
  • Method: this branch's tests against the pre-fix driver.ts, grade.ts, usage.ts and aggregate.ts (63a69ff). No guard was edited.
  • Result: # pass 25, # fail 5:
    • caps: Missing expected rejection: constructor;
    • patch export: nothing the clone configured ran, false;
    • wall cap: no end worker: killed;
    • sandbox confirmation: expected: 'all refused' / expected: 'model declined'.
• The reviewer's stub, in the test: three capped tasks finish in under 1.4 s.
  • The log is exactly start, killed, three times, with no overlap.
  • usage.total is 3.
  • A hanging container command is killed at the cap, and the next exec is refused.
• The sandbox confirmation, with stand-in sessions for each outcome (measured, tests):
  • All refused: in the stand-in, each cat is refused by the permission layer (permission_denials), each node -e read by the mask, and each curl with no marker. The 13 attempts record those layers, os_sandbox_read_deny is true, and every sentinel is removed.
  • Every read refused by the permission layer: all refused, os_sandbox_read_deny false, and the WARNING says so.
  • A sentinel read or a _ping OK: the run is refused, with no agent run and no task container, and the sentinels removed.
  • Altered commands exiting 0 (round 3): leaked.
  • Extra commands or tools (round 4): model declined.
  • Model declined: the run goes on with the WARNING and the label.
  • Real files: byte-for-byte unchanged and absent from the prompt.
  • SIGINT, SIGTERM and SIGHUP: the child dies by that signal, and no sentinel is left.
• Round 0 (measured by the builder on 9d73baa4): the cap check off gave TypeError: Cannot read properties of undefined (reading 'spendUsd'); the grading file checks off gave Missing expected exception.; the hidden-field check off gave 0 !== 6.
• Owner's runs on 9d73baa4 (measured by the owner): the usage-limit stop off gave not ok "the driver stops cleanly at a usage limit"; the subscription refusal off gave not ok "the driver refuses a login that is not a subscription".
• Not run: a real model call.
  • The confirmation's first real run will be the smoke round's first call.
  • It measures, for the first time, whether 2.1.303 marks permission denials as assumed here (permission_denials, the wording), and whether the OS sandbox masks the sentinel for node -e (guess until then).

Deferred to the RunTask PR (wording only, owner's decision of 2026-10-19)

These don't change what the checks do. They are left as they are here:

• "in Claude Code's own sandbox" still describes the confirmation in three places:
  • aggregate.ts:39 (a doc comment);
  • driver.ts:189 (a comment);
  • the title of the test "acceptance: the sandbox confirmation in Claude Code's own sandbox refuses the run on a sentinel read or a _ping OK".
    The confirmation now tests both Claude Code's permission layer and its OS sandbox, and records which one refused each attempt. "In Claude Code, with the agents' lane" is the exact wording.
• The leaked detail starts "from inside Claude Code's sandbox:" (confirm.ts). That is the same wording point.

Open items (not sandbox-check findings; listed, not changed here)

• Round 2, C: TaskRunner falls back to the blocking call when callAsync is unset and call is set (container.ts). Keep that fallback to tests, so a future entry point can't bring back a blocking exec.
• Round 2, D (round 1's suggestions, still open):
  • readPatches: refuse nlink > 1 and open with O_NOFOLLOW;
  • error and stopped rows in the rates;
  • a later task's agents can read an earlier task's folder.
• Round 3, D: sandboxed Bash can write the per-user temp folder in every task, so a later task's agent can see what an earlier one left there (inferred from the sandboxing docs).

Adversarial pass (round 10)

Round 10's hand-off had none (the round-10 check's finding). This one was run in round 11, against the round-10 head 242f663, with fixtures, a scratch worktree and stand-ins only. Every result is measured unless labelled.

• Principle 7 (rule 10), the merge keeps main's tests:

  • Main's assertions: every one of the 161 assert lines in main's test/bench.test.ts is in the branch's file, up to an added await before stdinSends (main-asserts.mjs). The only exceptions are five older lines (assert.throws on exec and syncTree) that this branch made assert.rejects in earlier rounds, because those functions are async here.

  • No call left unawaited: no stdinSends( call lacks await (grep: 0). So no test compares a Promise with false and passes by accident.

  • Main's new tests are live here, not vacuous: with the head's tests and c339175's (pre-bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804) scripts/bench/hidden.ts, in a scratch worktree since removed: # pass 63, # fail 3. These fail:

    • argv: an added test-patch line starting "++ " … (Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797);
    • stdin: a test-patch run with an added line starting "++ " … (Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797);
    • stdin: an open import paren goes on only through names … (bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804).

    The other two (… 682c8ce5's rule says so too … and a +++ line with no name …) pass on the old code too. inferred: they guard cases the earlier rule already handled, and their own titles say so ("as before Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797").

• Principle 3, nothing main closed reopens:
  • git diff origin/main -- scripts/bench/hidden.ts is empty on the head, so bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804's addedLines, importLines and strict && loose are main's.
  • Main's later Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793 (178c44e) merges cleanly (git merge-tree) and touches no bench file.
• Principle 1, RAW_UNSAFE's additions (r10-attacks.mjs, the regex read from the compiled test):
  • Every \p{Zs}: all 16 non-ASCII spaces are refused (U+00A0, U+1680, U+2000-200A, U+202F, U+205F, U+3000). Only U+0020 passes.
  • ASCII and tab: ASCII spaces, a line of spaces and tab indentation pass.
  • U+FFFC: refused at a line's start, middle and end.
  • Private use: U+E000 at a line start and U+10FFFD, the last private-use code point, are refused.
  • A non-ASCII space before a CRLF end: refused.
  • Visible text passes: an em dash, an emoji, CJK and Arabic.
  • U+FFFD passes (the replacement character). inferred: it is visible as a box or question mark, so it isn't in the class this scan names.
  • Over every code point (sweep9.mjs): nothing 002d7c9's regex caught is let through, and no default-ignorable, format or variation-selector code point gets through.
  • The 16 files, main's new tests included, scan clean (the scan test passes on the head).
• Principle 5, claims corrected:
  • the round-9 section's "still let through, by design" and "the test title doesn't claim them" (correction note there);
  • the "main merged in round 6" line, which now names round 10's merge;
  • "U+180B-180F", now "U+180B-180D and U+180F", in round 10.
• Principles 2, 6 and 9:
  • only test/bench.test.ts changed in round 10, besides the merge;
  • the regex only grows;
  • no exception, allowlist entry or skip is added.
• Fixed because of this pass: nothing in the code. The text corrections above were made.
• Principle 8: no refusals.

Adversarial pass (round 9, covering round 8's changes too)

Round 8's hand-off had no Adversarial pass section (the round-8 check's finding). This one covers both rounds. It used fixtures and stand-ins only; every result is measured unless labelled.

• Principle 4, the class rather than the instance: the scan swept against every code point (sweep9.mjs, RAW_UNSAFE read from the compiled test).

  • Left through, now none: no \p{Default_Ignorable_Code_Point}, no \p{Cf} and no \p{Variation_Selector} code point gets through.

  • Round 8's regex let through U+17B4-17B5, U+180B-180D and U+180F. That was round 8's miss: a sweep like this one would have found it. Fixed by adding the whole class.

  • Rule 10: nothing the old regex caught is let through now.

  • Visible text still passes: Latin, Greek, Cyrillic, CJK, Hangul, Arabic, Hebrew, Devanagari, emoji, a non-blank braille letter (U+2801), an em dash, box drawing and a combining acute (U+0301).

  • Still let through, by design (inferred: each shows as a space or a glyph):

    • the non-ASCII \p{Zs} spaces (U+00A0, U+2009, U+202F, U+3000…);
    • private-use characters;
    • U+FFFC.

    None is default-ignorable. The test title doesn't claim them.

    Correction (round 10 check): both statements above are false about the head since 242f663. Round 10 made RAW_UNSAFE refuse every non-ASCII \p{Zs}, private use and U+FFFC, and the scan test's title now names them. See the round-10 Adversarial pass.

• Principle 1, the probe-container order (round 8's change): with a reused run id, the second runBench makes no container call. It is refused at the run folder, after docker info only. The reused-run-id test checks it (15 !== 12 on 9c666ba).
  • inferred: a failed probe now burns the run id, since the folder is made first. That fails closed, and the README already says to pick a new id.
• Principle 5:
  • Claims corrected: "catches the variation selectors" (body and round-8 commit message; corrected in this round's commit message, no history rewritten); "so do vfat and exFAT"; the unlabelled "on ext4 here"; and the "see round 8 below" pointer.
  • Not attacked: the README's case-insensitive claim needs a casefold filesystem, which needs root to make. It stays inferred.
• Principles 2, 3, 6, 7 and 9:
  • only test/bench.test.ts and the README change this round;
  • the regex only grows;
  • no assertion is removed or weakened;
  • no exception, allowlist entry or skip is added.
• Principle 8: no refusals.

Adversarial pass (round 7)

Fixtures, temp folders and stand-ins only (work/fix-759-r4/adversarial7.mjs in the session scratchpad). Every result is measured, unless labelled.

• Principle 1 and 4, one meter per task folder:
  • Two runBench calls with one run id, started in the same tick: the first ran its two tasks; the second was refused ("the run folder … already exists"). mkdir without recursive is atomic.
  • A run folder that is a link (to an empty folder, or dangling): refused, with no agent run.
  • Something makes the next task's folder while the first task runs: the next task is refused, and runBench throws after recording the first task.
    • That fails closed; the aggregate is then not rewritten for the later tasks.
    • inferred: no agent can make it, since an agent's sandboxed Bash writes only its own working folder, inside its own task folder.
• Principle 1, inode comparison:
  • A folder removed and its inode reused by a new b, then a run at b/c: refused, since b carries the earlier run's inode. This is a false positive, and it fails closed.
    • inferred: a process left in the old folder would keep its inode alive, so reuse means nothing is left there.
    • A run at b itself is allowed, as the same folder again.
  • An earlier run's folder renamed out of repo:
    • a run at its new name is allowed, as the same folder;
    • a run at repo is allowed. repo doesn't hold it any more, and a process left in it can't write repo's parent.
• Principle 5: "What this can never do" and Compromised component now say "per task folder" and "by inode". The round-6 wording ("the task's runs") is corrected, since it held per meter. The assumption left is labelled inferred: nothing outside the task folder's own runs writes inside it while a run starts.
• Principle 4, raw characters: the source now has none, and a test scans scripts/bench/* and test/bench.test.ts. It failed on the old file (line 1605). Correction (round 7 check): this line used to say the scan refused "any \p{Cf}, \p{Cs} or control character other than tab and newline". In round 7 it also let through CR anywhere in a line, the line and paragraph separators, the variation selectors and the Hangul fillers.
  • Round 8 added CR mid-line, the separators, U+FE00-FE0F and U+E0100-E01EF, the four Hangul fillers, U+2800 and U+034F. Correction (round 8 check): round 8 said it caught "the variation selectors". It did not catch the Mongolian ones (U+180B-180D, U+180F) or the Khmer U+17B4 and U+17B5.
  • Round 9 adds the whole \p{Default_Ignorable_Code_Point} class, which covers them.
  • For both, see the round-8 and round-9 notes under test/bench.test.ts above, and the round-9 Adversarial pass.
• Principles 2, 3, 6, 7 and 9:
  • nothing outside scripts/bench and test/ changes;
  • nothing is loosened;
  • no existing assertion changed. The only rewritten test line is the round-6 character list, now code points with the same characters;
  • no exception, allowlist entry or skip is added.
• Principle 8: no refusals.

Adversarial pass (round 6)

Run against the decision guide with fixtures, temp folders and stand-ins only; no model, no root, no real state. The probe is work/fix-759-r4/adversarial6.mjs in the session scratchpad. Every result below is measured, unless labelled.

• Principle 1 (fail closed), the working folder:
  • Three runs started in the same tick (repo, then repo/sub and the task folder while it awaits): the two nested ones are refused, and the first runs. There is no await between the check and the registration.
  • A run whose runner throws: its folder is released for a disjoint run (b started after a threw), so a failure doesn't block the rest of the task.
  • A prefix sibling (repo2 while repo runs): started. They are not nested, and repo's writes can't reach repo2.
  • Other spellings: a trailing slash, //, /./ and a link to repo that stays inside are all refused.
  • A "leftover process" swap after a run ended (repo/sub replaced by a fresh real folder): the next run in repo/sub is refused by the earlier-run rule. Fixed because of this pass: the earlier-run rule itself was added in this pass. Before it, the in-flight rule alone left this case open (inferred: a background process outliving its run is a guess).
• Principle 1, plain: these all count as extra calls, never as the listed command: a null input, a string input, an own __proto__ key, an object with a toString, an array command. A string description, even an empty one, is still the listed command.
• Principle 1, RULE_UNSAFE: every BMP code point was tested.
  • Refused: U+0000-0021, ( ) * , ? [ \ ] { }, U+007F-00A0, U+00AD, the Arabic and Syriac format characters, U+1680, U+180E, U+2000-200F, U+2028-202F, U+205F-206F (except U+2065, which is unassigned), U+3000, all surrogates, U+FEFF and U+FFF9-FFFB; and the astral U+1D173, U+E0001 and U+E0020.
  • Fixed because of this pass: lone surrogates were not refused, so \p{Cs} is added. inferred: the run-time realpath check would have refused such a path anyway.
• Principle 4 (impossible by design): the swap is now refused by the meter. It no longer rests on a documented rule for RunTask. The remaining assumption, that nothing outside the task's runs writes in the task folder while a run starts, is inferred and stated as such.
• Principle 5 (claims): "What this can never do" now says "at the meter's check", and adds what the nesting rules guarantee. The old "where it really is" wording, which round 5 found to hold at the check but not at the spawn, is corrected.
• Principles 2, 3, 6 and 9:
  • no off switch or install path is touched (scripts/bench and test/ only);
  • nothing is loosened;
  • no exception, allowlist entry or skip is added to an existing check. The new tests' win32 skips match the file's existing POSIX skips, and CI runs Linux.
• Principle 7: no existing assertion was changed this round (measured, git diff 943244ab 45b91baf -- test/bench.test.ts adds tests and one import only).
• Principle 8: no refusals this round.

New dependencies

None in the repo. sandbox-runtime 0.0.79 is installed for the bench user only, by the README.

What this can never do

• Start a task without the preflight. That means: without caps (finite, an approved mode, own keys only), as another user, on another CLI version, on anything but a claude.ai login, with the srt probe reaching a socket or credential file, on a rootful daemon, with research able to run, or with a container limit that didn't apply. Each of these runs before any container or model call, and the tests count that neither claude nor docker was called.
• Start a task after the sandbox confirmation saw a sentinel's content or a daemon socket answer, or any command naming a protected target exit 0 (round 3, measured). That includes the node -e read, which only the OS sandbox can stop (round 4, measured). What it does not claim:
  • an "all refused" with cat alone says nothing about the OS sandbox, since the permission layer most likely stops cat first (inferred);
  • the per-attempt layer rests on markers inferred from public docs, not yet seen from 2.1.303;
  • a declining model, or one that ran anything beyond the listed commands, is recorded and said, and the run then rests on the srt probe.
• Create or change ~/.claude.json during the sandbox check (round 3, measured), or leave a sentinel behind after SIGINT, SIGTERM or SIGHUP (measured). inferred limit: SIGKILL or a power loss can still leave one, and the next check then lists that path as a skipped real file.
• Start an agent run whose working folder, at the meter's check, is outside its task's folder, reached through a link, or missing; or whose Edit, Write, MultiEdit or NotebookEdit reaches outside that working folder (rounds 4 to 6, measured). Correction (round 5 check): the earlier wording claimed this held at the spawn; it holds at the check.
  • Between the check and the spawn, no other run in the task folder can swap a folder of it. No run starts nested with a running run's folder, or strictly inside an earlier run's, compared by path and by inode (rounds 6 and 7, measured).
  • Each task folder is fresh and has exactly one meter: a repeated task key and a reused run id are refused (round 7, measured).
  • Correction (round 6 check): round 6 claimed this for "the task's runs", but it held only within one meter object. It now holds per task folder.
  • What that rests on is inferred: nothing outside the task folder's own runs writes inside it while a run starts.
• Put a path with a glob, rule, control or format character into an Edit rule (rounds 4 to 6, measured).
• Record the OS sandbox as the layer that refused a read, or os_sandbox_read_deny true, when the model ran anything beyond the listed commands (round 5, measured).
• Let a task's agent run or container command outlast its wall-clock cap, start one after it, or overlap the next task. A run that won't end halts the bench.
• Keep running after a usage limit, or retry. Every later task is recorded as not run, and the aggregate is marked partial.
• Run anything an agent configured while exporting a patch, or grade anything but the exported patch files. Export reads only the clone's objects. Grading reads no clone, event log or agent output, and refuses a link or a stray file in the patches folder.
• Put an instance id, task text or patch in an aggregate, or in this repository. The data lives in ~/bench, outside every checkout (bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756's benchRoot).

Compromised component

• An agent in a bench run: bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's sandbox denies its reads of the driver's state, the held-out store, the daemon and the login.
  • Checked before each run by the srt probe (sandbox-runtime 0.0.79, close to the bundled build), and by the confirmation:
    • cat of sentinels, which most likely tests Claude Code's permission layer (inferred);
    • node -e of sentinels, which tests its OS sandbox's own denyRead (round 4; whether the sandbox or the permission layer stopped it is recorded per attempt, from markers inferred from the docs);
    • curl to the sockets, whose refusal can't be told from a missing socket (inferred).
  • Its patches are exported from its clone's objects only, with no config of its, so nothing it plants in the clone runs in the driver.
  • Its Edit, Write, MultiEdit and NotebookEdit tools reach only its run's own working folder, inside its task's folder by path and by realpath (rounds 4 and 5: the meter scopes them there and refuses any other grant, or a working folder outside the task's, measured). A link it plants inside its working folder can't become a later run's working folder, even one inside the task (rounds 5 and 6, measured). While its run is going, or after it if a process of it lingers, it can't swap a folder of another of the task's runs before that run's spawn: no run starts nested with a running one's folder, or strictly inside an earlier one's (round 6, measured). That holds per task folder: a repeated key or a reused run id can't give the folder a second meter (round 7, measured).
    • So it can't rewrite ~/.docker/config.json, ~/.config/systemd/user/, ~/.local/bin/ or a sibling folder's .claude/settings.json or a bare remote's hooks through them.
    • inferred: ~/.profile, ~/.bashrc, ~/.gitconfig and ~/.claude/ are protected by dontAsk anyway.
    • inferred: its working folder holds what it is working on, which it can change; the driver reads back only the clone's objects.
    • inferred residual: a sandbox escape through Bash would still give it the bench user's powers.
  • inferred: it could game the engine's own checks (which is what the bench measures), not the grading. Grading runs SWE-bench's harness in fresh containers after the run.
• A task's repo code: it runs only in its own limited, network-less container, which is removed after the task.
• The model in the sandbox confirmation:
  • it sees only socket paths and sentinel paths, never a real credential or its content;
  • if it lies in prose, that changes nothing, since the verdict is read from the tool transcript;
  • if a command naming a target succeeds, or a sentinel's content shows up anywhere, that is a leak, and the run is refused;
  • if it runs anything beyond the listed commands, the status is "model declined" at best (round 4, measured). So a read through $HOME, ~, a glob or a link can no longer pass as "all refused". Since round 5 (measured):
    • a listed command sent with other inputs counts as beyond the list, and so does a server_tool_use;
    • nothing it runs can make an attempt look refused by the OS sandbox, since the layer is unknown whenever it ran anything extra;
  • if it runs nothing, the record says "model declined".
• Coordinator, dashboard, other instances, hub viewer: none runs or reaches this code. Nothing in src/ imports it, and the package doesn't ship it.
• Someone with push access: they could change a cap, the wall-clock cap, the preflight order, the confirmation model, the layer markers or scopeEditTools. The guard is this review: the owner merges scripts/**.
• Someone who can run as the bench user: they can already use its login and daemon directly, so the bench gives them nothing new. The README's step closes the admin account's home to the bench user (chmod 750, checked with sudo -u wl-bench ls failing).

[jayden1711]

Owner's fail-without results (measured by the owner, 2026-10-18), on 9d73baa4 in a scratch worktree, now removed:

1. Guards on: both tests pass (pass 2, fail 0).
2. driver.ts:153 usage-limit stop off (edit confirmed by grep): not ok "stops cleanly at a usage limit"; pass 0, fail 1.
3. auth.ts:107 subscription check off (edit confirmed by grep): not ok "refuses a login that is not a subscription"; pass 0.

Both guards fail without their check. The PR body is updated.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 1/2)

I reviewed only this PR's diff against issue-748-bench-containers, at head 73c0b4f. All my checks used stubs, a temporary HOME and temporary state dirs. Nothing called claude, Docker or a model.

Summary

The preflight order, the usage-limit stop, the grading file checks, the Wilson formula and the results record are all sound. Four things block the merge:

• Patch export runs a command the agent can configure, outside the sandbox. This is a sandbox escape.
• The wall-clock cap doesn't stop agent runs unless the task forwards the signal itself. Their spend then goes unrecorded.
• The driver's cap check accepts a mode such as constructor. It then has no spend ceiling.
• Two README gaps.

Acceptance and focus

Item	Status	Evidence
The driver refuses to start without a spend cap	partially met	measured: the test passes. measured: checkCaps accepts {mode: 'constructor' or 'toString', spendUsd: 5000, perTaskUsd: 15}, and the run gets to the bench-user check (required change 3).
Every refusal comes before any container or model call	met	measured: the tests count zero claude runs and zero docker run calls for each refusal: caps, research, user, pin, subscription, rootless, limits read-back. Inferred from driver.ts runBench: the order is caps, research, user, pin, subscription, keys, limits, rootless, probe. It differs from the order in the brief, but all of it runs before the first write, container or model call.
Wall-clock cap kills the task's process tree and its container exec	not met	inferred: the container is removed in finally (rm --force), which ends its execs. measured: the cap does not reach the agent runs (required change 2).
Usage-limit stop: no retry, no account switch, rest marked not run, partial aggregate	met	measured: the test passes. inferred: UsageMeter.meter puts its stop signal on every run, so the stop works even if the task ignores ctx.signal. The owner's fail-without is on this PR (comment 6257259262): "stops cleanly at a usage limit" fails with the stop off.
Grading reads only plain .diff files in the patches folder	met	measured: the test passes. A link and a stray notes.txt are both refused. inferred: lstat is used at every level, and keys and labels are matched by pattern, so a path can't escape the folder. Suggestion A covers hard links.
Predictions go into the held-out store, never a container	met	measured: the test checks files.final.startsWith(layout.heldOut). inferred: grade.ts never calls docker.
Wilson intervals	met	measured: the results match published Wilson 95% values within 0.001: 0/10 is [0, 0.2775], 1/10 is [0.0179, 0.4042], 5/10 is [0.2366, 0.7634], 81/263 is [0.2553, 0.3662], 15/148 is [0.0624, 0.1605]. An independent implementation agrees. wilson(0,0) is null.
Results record the CLI version, plan, limits and usage-equivalent USD	met	measured: the test checks claude_version 2.1.303, subscription_type enterprise, the limits for each task, and usage.total 7.5. inferred: results.json and aggregate.json both carry them.
README matches the owner's setup	partially met	measured (by reading README.md): it has the wl-bench user, subuid and subgid, linger, the required Delegate=cpu cpuset io memory pids drop-in, env -u DISABLE_UPDATES bash -s 2.1.303, /login as wl-bench, rootless Docker Engine with the root daemon off, and data in $HOME/bench. It does not say "not Docker Desktop". It has no step closing the owner's home, which the PR body claims it does (required change 4).
Denylist (no person, company or machine)	met	measured: npm run check:denylist printed denylist: 1279 files clean, and a grep of the new files found no names.

Fail-without

• On the base: the new test cases import modules this PR adds (driver.ts, grade.ts, aggregate.ts, store.ts), so on the base they fail to compile. That can't be avoided. I made no guard-off edits, per the brief.
• Spend cap, grading file checks, driver-level hidden fields: the builder measured these on 9d73baa4 (scratchpad/work/issue-748/failwithout.txt). The failures read: ... "TypeError: Cannot read properties of undefined (reading 'spendUsd')", Missing expected exception. and 0 !== 6.
• Usage-limit stop and subscription refusal: the owner measured these on 9d73baa4 (comment 6257259262). Each test fails with its guard off.
• On the head (measured): node --import dist/test/isolate.js --test dist/test/bench.test.js gives # pass 20, # fail 0. npm run typecheck exits 0.

Protection

• measured: every path is under scripts/**, which main's .worklane/review.yaml lists under engine.
• measured: the PR is correctly marked protected: the title has [PROTECTED: owner merges], the body has **Risk class:** protected, it has the needs-owner label, and it has both sections.
• Two of those sections' claims are currently false:
  • "Compromised component: an agent ... can't reach the driver's state": false. Through exportPatch, an agent that can write its clone's git config runs code in the driver (required change 1).
  • "What this can never do: start a task without caps": false for the mode values in required change 3.
• No regression for existing installs: inferred: not applicable. The PR only adds scripts/bench/** and a test, and touches no setup, firewall, sandbox or guard used by an installed engine.

Scope and size

• Scope: inferred: no scope creep. Everything here is part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748 Phase 0.
• Size: 806 lines. inferred: acceptable. About 420 lines are code, and the rest is the owner's setup README and the tests. The work is already split three ways, and splitting the driver from grading would add more stacking for little gain.
• Commit hygiene (measured): signed off, no attribution trailers, no new dependencies, no .claude/settings.json, no product name in src/. The PR body labels its claims and says "Part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748", which is right for a stacked PR.

CI

measured (gh run view 39180990350 --log-failed): the only failing tests are the two known main failures on both Node 22 and 24:

• the package stays within its size budget (shard 1);
• focus and selection rings use the ring token, not the accent (shard 3).

The test aggregate jobs fail only because those shards did. inferred: this PR adds nothing to the package, since scripts/bench and dist/scripts are not in package.json files. #753 fixes both failures.

Required changes

1. grade.ts exportPatch runs commands from the task clone's git config, outside the sandbox.
   • The problem: it runs git -C <clone> diff --binary --no-color --no-ext-diff base head. The clone is the agents' working tree, and the driver runs this as the same OS user, with the login and the Docker socket in reach.
   • measured (scratch repo, textconv-probe.sh): with diff.evil.textconv set in the clone's .git/config and *.py diff=evil in .git/info/attributes, that exact argv ran the command (TEXTCONV RAN (marker created)). With --no-textconv added, it did not.
   • The risk: this is the exfil route the owner's design closes (read ~/.claude, or reach the daemon). The textconv output would also be what gets graded.
   • The fix: follow the engine's existing invariant (Comment policy: WHY:/INVARIANT:/issue id on new comments, no narration, ≤4 lines — CI on diffs and an agent write-time hook #330, src/comment-policy.ts:493):
     • add -c core.fsmonitor=false -c core.hooksPath=/dev/null, --no-textconv and an explicit --src-prefix=a/ --dst-prefix=b/, so a repo-set diff.noprefix or diff.mnemonicPrefix can't break git apply -p1;
     • better still, diff in a driver-owned bare repo fetched from the clone, so no agent-written config is read at all.
   • Test: a fail-without test that plants a textconv marker in the clone's .git/config and asserts the marker is never created and the exported diff is the real one.
2. The wall-clock cap does not stop a task's agent runs unless the task forwards ctx.signal itself.
   • The cause: TaskMeter gets the run-wide usage-limit signal (through UsageMeter.meter) but not the task's wall signal.
   • measured (wallcap-probe.mjs: stub runner and docker, taskWallMs: 100, a task that passes no signal), the probe's output:
     • runBench returned at 207ms: wall_cap_hit=2, usage.total=0;
     • end worker@4ms: succeeded (ran to the end: never killed) at 1505ms;
     • start evaluator-after-cap@1505ms.
   • What that shows:
     • each run kept going after the cap;
     • a new run started after the cap;
     • t01's agent overlapped t02's;
     • the aggregate recorded $0 for $4 of usage-equivalent spend.
   • The existing test counts only that ctx.signal fired.
   • The fix: make it impossible.
     • meter.task(perTaskUsd, signal) adds the task's signal to every request (AbortSignal.any), so CliRunner's killTree fires.
     • The meter refuses new runs once that signal has fired.
     • After the cap, the driver waits (with a bound) for the task's in-flight runs to settle before it records the task's spend and starts the next task.
   • Test: a fail-without test with a task that doesn't forward the signal. It asserts the run was aborted, no run started after the cap, and the spend was recorded.
   • Also, inferred: TaskRunner.exec is synchronous (spawnSync, 15 min default), so the cap can fire up to one exec timeout late. Say so in the README or comments, or make the exec async.
3. checkCaps in driver.ts uses c.mode in MODE_MAX_USD, which also matches inherited keys.
   • measured (caps-probe.mjs): {mode: 'constructor', spendUsd: 5000, perTaskUsd: 15} and the same with toString both pass checkCaps. spendUsd > MODE_MAX_USD['constructor'] compares a number with a function, which is false.
   • measured: bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756's capsFromArgs(['--tasks', 'constructor', '--spend', '5000']) also returns {"mode":"constructor","spendUsd":5000,"perTaskUsd":15}. That code is in the base; bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756's review should fix it there.
   • The fix here: the driver check is the second line of defense, so use Object.hasOwn(MODE_MAX_USD, c.mode), or c.mode === 'smoke' || c.mode === 'baseline', and Number.isFinite. Add these modes to the spend-cap test.
4. README:
   • Docker Desktop: say plainly that this is Docker Engine, not Docker Desktop, and that Docker Desktop's WSL integration must be off for this distro. inferred: the preflight already refuses a Desktop daemon, because it isn't rootless.
   • The owner's home: add the step that keeps the owner's own home closed to wl-bench (for example, check that it is 0750 or 0700), or drop that claim from the PR body's "Compromised component". The agents' sandbox denies only the bench user's own credential places.

Suggestions (not blocking)

• A. readPatches:
  • refuse nlink > 1, since a hard link to a file outside the folder is a "plain file";
  • read with O_NOFOLLOW, which closes the gap between the lstat and the read.
  • inferred: low risk today. Grading runs after the run, and agents' writes are scoped to their own folder.
• B. aggregate:
  • the rates count error and stopped (usage limit) rows (in false_holds and auto_merge_yield), but the confusion matrix doesn't show them. Show them, or leave them out of the rates.
  • a per-task limits stop leaves stopped: null. Consider 'limits'.
• C. Agent read access: agents of later tasks can read earlier tasks' folders and runs/<run>/patches. Consider denying them in benchLane.

[jayden1711]

Round 1 fixes (head 092d788; #757's round 1 merged at 63a69ff and a9da0f6)

Labels: measured = a command I ran or CI output, inferred = reasoned, not run.

Required changes

1. exportPatch now diffs in a bare repo the driver makes, with the clone's object store as its only source (GIT_ALTERNATE_OBJECT_DIRECTORIES).
   • It reads no system or user config (GIT_CONFIG_NOSYSTEM, GIT_CONFIG_GLOBAL=/dev/null).
   • Flags: -c core.fsmonitor=false -c core.hooksPath=/dev/null --no-textconv --no-ext-diff --src-prefix=a/ --dst-prefix=b/.
   • A .git or objects that is a link is refused.
   • The test plants all of these in the clone: a textconv driver with its attribute, diff.external, core.fsmonitor, core.pager, diff.noprefix and diff.mnemonicPrefix. It asserts the marker is never created and the diff is the real one, starting diff --git a/price.py b/price.py.
   • measured fail-without: nothing the clone configured ran, expected: true, actual: false.
2. Wall-clock cap.
   • meter.task(perTaskUsd, signal) adds the task's signal to every request, and refuses new runs (stopped) once it fires.
   • The driver awaits runner.settled() before it records the task's spend and starts the next task. The bound is 5 min; after that the bench halts and every later task is not run (halted).
   • TaskRunner.exec is now async (spawn, not spawnSync). At the signal its docker exec is killed, the container's removal ends what it started, and no exec starts after.
   • Test, with your stub (runs of 1.5 s unless aborted, a task that passes no signal):
     • the log is exactly start, killed, start, killed, start, killed: no evaluator-after-cap, no overlap;
     • the whole run takes under 1.4 s;
     • usage.total is 3, so each killed run's spend is recorded;
     • a hanging container command is killed at the cap, and the next one is refused.
   • measured fail-without: the log had no end worker: killed.
3. checkCaps uses isMode (own keys) and Number.isFinite.
   • Test: constructor, toString, __proto__ and hasOwnProperty with spendUsd: 5000 are refused, and so is spendUsd: Infinity.
   • measured fail-without: Missing expected rejection: constructor.
4. README:
   • Docker Engine, not Docker Desktop, with Desktop's WSL integration off for this distro.
   • The admin home step: chmod 750 "$HOME", checked with sudo -u wl-bench ls "$HOME" failing (expect Permission denied).
   • inferred, not run: I have no wl-bench user here, so the check is the owner's to see.
5. Where Edit and Write may write is stated in the README and at the top of driver.ts.
   • The deny list covers reads only. Bash writes where the sandbox allows.
   • Edit and Write run outside the sandbox: dontAsk refuses what would ask, but an unqualified Edit or Write in a run's allowed tools lets them change any file the bench user can.
   • The driver reads none of that back.
   • inferred from Claude Code's permissions docs.

The owner's sandbox decisions (2026-10-18)

• A (bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's probe.ts) is wired as preflight.sandboxProbe. It runs after the login check, before Docker and before any model call.
• D, confirm.ts: the run's first model call, on claude-haiku-5-5.
  • measured: the 2.1.303 binary carries this id.
  • The call is metered, capped at $0.50, counted against the ceiling, and recorded as usage.sandbox_check.
  • Claude Code runs curl --unix-socket <sock> http://localhost/_ping per socket and cat of sentinels, in its own sandbox with the agents' lane.
  • Sentinels: created exclusive (never over anything) at a missing ~/.netrc, ~/.npmrc or ~/.pypirc, and removed after.
    • ~/.claude.json is never touched. Its own rule is exact-path, so no other file can sit under it. Its sentinel goes in ~/.claude/ under the login folder's rule (Read(//<home>/.claude/**) plus the sandbox's denyRead of ~/.claude).
    • A path holding a real file is skipped and never put to the model.
  • The verdict comes from the stream-json tool transcript. A sentinel's content anywhere, or a _ping OK, refuses the run. Commands not run give "sandbox confirmation: model declined": the run goes on (A passed) with a WARNING carrying the 0.0.79 label.
  • Both checks are in sandbox.json, results.json and the aggregate.
  • Tests (stand-ins, no model call):
    • all refused;
    • a sentinel leaked (the stand-in really reads the file);
    • _ping OK;
    • model declined;
    • real ~/.netrc and ~/.claude.json untouched and kept out of the prompt;
    • the srt probe reaching a file, refused before any model call.
  • measured fail-without: the old driver made no confirmation (expected: 'all refused' / 'model declined').
• C: scripts/bench/srt-fingerprint.mjs (read-only) plus a README step to re-run it on each new release, and to switch when one shows no difference.
  • measured on 2.1.303: 0.0.79 is 1 log string apart each way, 0.0.78 1 and 2, 0.0.80 4 and 4.

Fail-without method (measured)

This branch's tests against the pre-fix driver.ts, grade.ts, usage.ts and aggregate.ts (git checkout 63a69ff5 -- …, then restored) gave # pass 25, # fail 5, the five quoted above. No guard was edited.

Local and CI (measured)

• Locally: bench tests # pass 30, # fail 0 three times in a row, plus no-wall-clock-durations passing; typecheck exits 0; denylist: 1282 files clean; DCO ok.
• CI on 092d788: all 19 checks pass, every shard on Node 22 and 24 included.
• The wall-clock-duration lint that failed bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's first run is fixed here too (092d788).

Body

• Both protected sections are updated:
  • patch export can't run agent config;
  • the wall cap reaches every run and command;
  • the caps use own keys;
  • the sandbox checks;
  • Edit/Write named as a residual;
  • the admin-home claim is now a README step.
• Size: 1,413 lines against the base (measured), well over ~700. I have not split it further.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 2/2)

I reviewed only this PR's diff against issue-748-bench-containers (2fda9e6), at head 092d788. Every check used stubs, a temporary HOME and temporary state, with a refusing claude, docker and srt first on PATH. Nothing called claude, Docker or a model. My probes are in the session scratchpad (work/eval-759-r2/).

Summary

All four round-1 required changes are fixed, and I re-ran my probes to confirm each one:

• patch export;
• the wall-clock cap;
• checkCaps;
• the README.

Option C is fine. Option D (confirm.ts) works as the tests describe, but it has two gaps I measured against the owner's spec, plus one inferred gap in the protected sections. These three block the merge:

1. The ~/.claude.json sentinel lands at ~/.claude.json itself when that file is absent, not under ~/.claude/.
2. The verdict can say "all refused" when the read or connect worked. This happens when the model runs a command other than the one written.
3. The Edit/Write residual is real, but the protected sections say it doesn't matter. An agent that can write ~/.profile gets code run as the bench user, outside every sandbox.

Round-1 required changes

#	Item	Status	Evidence
1	Patch export in a driver-owned bare repo, no outside config, hardened flags	met	measured (export-probe.mjs, which calls the built exportPatch). I planted these in the clone: diff.evil.textconv, diff.evil.command, diff.external, core.attributesFile, a committed .gitattributes (*.py diff=evil filter=evil), .git/info/attributes, filter.evil.clean/smudge/process, core.fsmonitor, core.pager, pager.diff, core.hooksPath, include.path (to a file with its own external and textconv), diff.noprefix, diff.mnemonicPrefix, diff.relative and diff.orderFile. I also planted a ~/.gitconfig and ~/.config/git/attributes with the same, and set GIT_EXTERNAL_DIFF, GIT_CONFIG_COUNT/KEY/VALUE, GIT_DIR and GIT_PAGER in the caller's environment. As a control, a plain git diff in the clone ran include-textconv, diff-command, smudge, process. Under exportPatch, planted commands ran: none, and the patch is the real one: diff --git a/price.py b/price.py, the binary file as a GIT binary patch, and .gitattributes exported as plain content. inferred from grade.ts:29-58: the env holds only PATH plus GIT_CONFIG_NOSYSTEM/GIT_CONFIG_GLOBAL=/dev/null, the bare repo is --template= with an unborn HEAD (so no tree attributes), and a .git or objects that is a link is refused.
2a	Every run and container command gets the signal; nothing starts after it	met	measured (wallcap-probe.mjs A, my round-1 stub: runs of 1.5 s unless aborted, and a task that passes no signal): end worker@14ms: killed by abort at 110ms, end worker@112ms: killed by abort at 214ms. No evaluator-after-cap started, and the two tasks didn't overlap. In round 1, the same stub showed succeeded (ran to the end: never killed) at 1505ms. inferred from usage.ts: TaskMeter.run uses AbortSignal.any([stop, req.signal]) and refuses with stopped once stop has fired.
2b	Container commands are async	met	measured (probe C, a hanging docker exec stand-in): exec killed @116ms, then second exec: not run: the task's wall-clock cap has passed. inferred from container.ts:178-227: spawn with signal, and the container is removed after settled.
2c	Waits for runs in flight, or halts after 5 min	met	measured (probe B: a run that ignores its abort, settleMs: 300): it returned at 412 ms with stopped=halted, error: 1, not_run_halted: 2. inferred: the default is SETTLE_MS = 5 * 60_000 (driver.ts:122).
2d	Spend recorded	met	measured (probe A): usage.total=2 ($1 per killed run), and sandbox_check=0.01 is kept separate. The test also asserts usage.total 3.
3	checkCaps prototype-safe	met	measured (caps-probe.mjs): each of constructor, toString, __proto__, hasOwnProperty, valueOf, isPrototypeOf and __defineGetter__ at $5000 gives not run: caps over the owner's approval. So do Infinity, NaN, string amounts, an array mode, an object with a toString, and a JSON __proto__ mode. Only {smoke, 40, 15} reached the bench-user check.
4	README: Docker Engine not Desktop; admin-home step; srt 0.0.79 for wl-bench only; bubblewrap and socat; where Edit and Write may write	met	measured (read scripts/bench/README.md): lines 103 (Engine, not Desktop, WSL integration off), 142-145 (chmod 750 "$HOME", sudo -u wl-bench ls "$HOME" expecting Permission denied), 176-177 (npm install -g --prefix ~/.local @anthropic-ai/sandbox-runtime@0.0.79, srt --version), 127 (bubblewrap socat) and 39-42 (where agents may write). Required change 3 below is about the reasoning in that section, not about whether it's there.

Sandbox check D (confirm.ts)

Item	Status	Evidence
One Haiku 5.5 call at run start, metered, capped at $0.50, recorded	met	measured (test 29): one check session with --model claude-haiku-5-5, and usage.sandbox_check 0.02. inferred from confirm.ts:154 and usage.ts: the cap is min(0.5, per-task, run left), the call goes through UsageMeter (so it counts against the ceiling), and it's written to sandbox.json, results.json and the aggregate (driver.ts:179-191). The model id in the 2.1.303 binary is the builder's measurement, not mine.
Sentinels only where a protected file is missing, never over a real file, removed after	met	measured (test 30, and my probe 1): a real ~/.netrc is skipped and unchanged. inferred: files are created with openSync(p, 'wx') (O_EXCL), so a file or a dangling link at the path makes creation throw, and lstat has already counted it as present. measured: after every outcome, including leaked, the home holds no sentinel.
~/.claude.json never touched; its sentinel under ~/.claude/; the rule really covers it	partially met	measured (confirm-probe.mjs 1). With ~/.claude.json present, the stand-in is ~/.claude/.bench-sentinel-…, and the real file is unchanged. With it absent, the sentinel is created at ~/.claude.json and put to the model (required change 1). On coverage, inferred from confirm.ts:93 and probe.ts:78-88: denied() is checked in code before each sentinel is made, against both the sandbox's denyRead (~/.claude) and the Read rule Read(//<home>/.claude/**). A path with no rule refuses the run, so the coverage claim is enforced, not just stated.
Real files never shown to the model	met	measured (test 30): neither real path is in the prompt. inferred: the prompt holds only socket paths and sentinel paths.
Verdict from the stream-json tool transcript; a leaked sentinel or _ping OK refuses the run	partially met	measured (probe 5, and tests 29-30): an honest leak gives leaked and the run is refused. measured (probes 2 and 3): when the model runs a different command that succeeds, the verdict is still all refused for all 7 attempts (required change 2). The commands were cat '<s>' > /dev/null && echo read-ok-but-hidden (exit 0), wc -c '<s>' (exit 0, 48 <s>) and curl … -o /dev/null -w '%{http_code}' (output 200).
"Model declined" falls back to check A, with a warning	met	measured (test 30, and probe A): WARNING: sandbox confirmation: model declined … sandbox-runtime 0.0.79: close to, not identical with, the build bundled in Claude Code 2.1.303, and the run goes on. inferred from driver.ts:161-162: A has already passed by then, or the run threw.
Results recorded	met	measured (tests 29-30): sandbox.json status, and agg.sandbox.confirmation with its attempts and skipped list. inferred from driver.ts:191: results.json carries sandbox.
Tests cover every outcome with stand-ins	met	measured: refused, leak (the stand-in really reads the file), ping, declined, real files, and srt reaching a file. The two cases in required changes 1 and 2 are missing.
No real model call in tests	met	measured: the bench tests ran with a refusing claude, docker and srt first on PATH. That guard was hit 0 times, and the scratch HOME and state stayed empty. inferred: machine() passes the stand-in's absolute c.bin to benchClaude.

Check C

• measured (read srt-fingerprint.mjs): it is read-only. It uses only readFileSync, readdirSync and console.log, with no exec, spawn or write.
• measured: README line 83 has the re-check step on each new release, and the switch to SRT_VERSION when a release shows no difference.

Fail-without (measured)

Method:

• I made a scratch worktree at 092d788 and ran git checkout 63a69ff5 -- scripts/bench/{driver,grade,usage,aggregate}.ts, which is the pre-fix code from a commit. I made no guard-off edit.
• I built it with the head's tests (tsc emitted with type errors, because those tests use the new fields).
• node --import ./dist/test/isolate.js --test dist/test/bench.test.js gave # pass 25, # fail 5:
  • Missing expected rejection: constructor;
  • patch export: nothing the clone configured ran, expected: true, actual: false;
  • the wall cap: the expected log's 'end worker: killed' lines are missing;
  • the two confirmation tests: expected: 'all refused' and expected: 'model declined'.
• On the head, the same command gives # pass 30, # fail 0.
• npm run typecheck exits 0, and npm run check:denylist printed denylist: 1282 files clean.

Protection

• measured: every path is under scripts/** or test/. The title has [PROTECTED: owner merges], the body has **Risk class:** protected, the PR has the needs-owner label, and both sections are there.
• The claims, inferred:
  • "Run anything an agent configured while exporting a patch" now holds (see item 1).
  • "Start a task after Claude Code's own sandbox let a sentinel be read" does not fully hold (required change 2).
  • The Edit/Write bullet in "Compromised component" understates the route (required change 3).
• No regression for existing installs, inferred: not applicable. Nothing in src/ or any installed path changes, and scripts/bench isn't in the package.

Scope and size

• Scope, inferred: no scope creep. Everything is Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748 Phase 0, the round-1 fixes, or the owner's options C and D.
• Size, measured: 1,413 lines (git diff --stat). That is about 860 lines of code, 391 of tests and 188 of README.
• Judgement, inferred: acceptable as one PR now. It has been reviewed in full twice, the pieces are coupled (the driver calls the confirmation), and nothing in it can run until the RunTask PR lands.
• If the owner wants a smaller unit, the clean seam is a fourth stacked PR for the sandbox checks: confirm.ts, its wiring in driver.ts, srt-fingerprint.mjs, the README's "Sandbox checks" section and tests 29-30. That is about 430 lines, and required changes 1 and 2 all fall inside it. The driver, grading and aggregates PR would then be about 980 lines.
• Commit hygiene, measured: all five commits are signed off, none has an attribution trailer, there are no new dependencies and no .claude/settings.json. "Part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748" is right for a stacked PR.

CI

measured (gh pr checks 759): every check passes on 092d788, including all eight test shards on Node 22 and 24. The real-systemd jobs are skipped as usual.

Required changes

1. Never create a file at ~/.claude.json. Its sentinel always goes under ~/.claude/.
   • measured (confirm-probe.mjs 1): with ~/.claude.json absent, the prompt holds cat '~/.claude.json', and the driver created that file for the call.
   • Why it matters:
     • the owner's spec is that ~/.claude.json is never touched and its sentinel goes under ~/.claude/;
     • the check's own claude (outside the sandbox) reads and writes ~/.claude.json as its config while the sentinel sits there (guess at how 2.1.303 handles non-JSON there);
     • an interrupted check leaves the sentinel at that path (suggestion A: measured, it survives SIGINT and SIGTERM).
   • Fix: in placeSentinels, .claude.json always maps to ~/.claude/.bench-sentinel-<hex>, and is skipped if ~/.claude isn't a plain folder.
   • Test: the stand-in records whether ~/.claude.json exists during the call, with the file absent beforehand. Assert that it never did, and that the ~/.claude/ sentinel was used.
2. Count an attempt as refused only on positive evidence.
   • Today: any Bash call whose command contains the path counts as a refused attempt, unless the sentinel's content or OK shows up in its output.
   • measured (probes 2 and 3): cat '<s>' > /dev/null && echo …, wc -c '<s>' and curl … -o /dev/null all exited 0 (so the read or connect worked), and the status was all refused for all 7 attempts.
   • Fix:
     • an attempt is refused only when a Bash call ran exactly the listed command and its tool_result is an error (is_error: true, which Claude Code sets on a non-zero exit: inferred);
     • a call that names the target and exits 0 is reached, and the status is leaked;
     • anything else counts as not run, so the status is model declined.
   • Test: stand-in sessions for the altered-command cases above.
3. Close the Edit/Write route before any agent can be given those tools, and correct the protected sections.
   • The route, inferred:
     • the PR's own premise is that an unqualified Edit/Write in a run's allowed tools can change any file the bench user can;
     • ~/.profile exists (from /etc/skel), the README's step 3 appends to it, its Read isn't denied, and sudo -iu wl-bench (the README's way in) runs it;
     • so a rewritten ~/.profile runs agent code as the bench user, outside every sandbox, with the daemon socket and the login in reach. That is the risk the owner's design closes.
   • The same holds for ~/.bashrc, and, as a guess, for overwriting ~/.local/bin/claude through its symlink.
   • Why the current mitigations don't cover it, inferred:
     • --setting-sources project keeps user-settings hooks out, but it doesn't help here;
     • "the next claude start is checked again" doesn't help either, because the version check passes a planted script that prints 2.1.303.
   • Fix:
     • preferred ("make the wrong thing impossible"), in this PR's TaskMeter.run: refuse any request whose allowedTools has Edit, Write, MultiEdit or NotebookEdit unqualified, or qualified outside the task's workDir (the driver passes it to meter.task), with a fail-without test;
     • at minimum: correct "Compromised component" and the README's "So nothing the driver reads back trusts…" so they name this route, and open an issue that blocks the RunTask PR until it is closed.

Suggestions (not blocking)

• A. Interrupted check.
  • measured (sentinel-kill.mjs, a stub runner that never returns): after SIGINT and after SIGTERM, the sentinels stay at ~/.netrc, ~/.npmrc and ~/.pypirc (and at ~/.claude.json, see 1).
  • The next check then lists them as skipped real files, and the confirmation silently covers less.
  • Fix: record the paths in the run's folder before creating them, and at the next start remove leftovers whose content starts with bench-sentinel-. Or handle SIGINT and SIGTERM.
• B. Which layer refused, guess about 2.1.303's internals: Claude Code's Read deny rule may stop a literal cat <path> before the OS sandbox is ever exercised. Record which layer refused each attempt, from the result text, so sandbox.json shows whether the bundled sandbox itself was tested.
• C. TaskRunner falls back to the blocking call when callAsync is unset and call is set (container.ts:225). Keep that fallback to tests, so a future entry point can't bring back a blocking exec.
• D. Round-1 suggestions A-C are still open:
  • readPatches: nlink > 1 and O_NOFOLLOW;
  • error/stopped rows in the rates;
  • later tasks' agents can read earlier tasks' folders.

[jayden1711]

Round 3 (owner-granted): what changed

Head a049531. #757's round-3 head (d57acd8) is merged in first with a signed-off merge commit, 9e4481c. That merge holds only the conflict resolution and what #757's API needed:

• BenchOptions.baseTree (required), because TaskRunner's base is now required and the driver passed undefined;
• the test machine's dataHome and base tree;
• the probe's two new sockets in the confirmation test.

On its own it gives # pass 33, # fail 0 (measured). The round-3 fixes are afdda84, plus a049531, which makes one new test unable to hang (below).

1. ~/.claude.json is never touched (owner decision 1).

• placeSentinels now always puts its sentinel under ~/.claude/. When that folder is missing (or isn't a plain folder), it skips it, and the record says why.
• Test: a stand-in records whether ~/.claude.json exists while the model runs. With the file absent beforehand, it never exists, during or after. With no ~/.claude, it is skipped.
• Fail-without (measured, the probe on old vs new): it existed during the model call (true) on the old code, false now.

2. The verdict is decided per attempt (owner decision 2).

• refused only when the exact listed command ran and its tool result has is_error.
• reached when any command naming the target didn't fail, or a sentinel's content or a _ping OK came back. Any reached attempt refuses the run (leaked).
• Anything else is not run, and the status is "model declined".
• Each attempt now records command and result.
• That is_error marks a non-zero exit in Claude Code's stream is inferred; the engine reads the same field in src/run-record.ts.
• Test (measured): cat '<s>' > /dev/null && echo …, wc -c '<s>' and curl … -o /dev/null -w '%{http_code}', each exiting 0, give leaked. So does the listed command exiting 0. Failing commands other than the listed ones give model declined.
• On the old code, all of these gave all refused.

3. Edit, Write, MultiEdit and NotebookEdit stay in the task's folder (owner decision 3).

• usage.ts scopeEditTools runs in TaskMeter.run on every request:
  • a bare name becomes Edit(//<task folder>/**);
  • Edit(//<p>) or Edit(//<p>/**) stays only when p is a plain, normalized path in that folder;
  • any other grant throws before claude starts: another folder, ~, relative, glob, .., a qualified Write/MultiEdit/NotebookEdit, Read,Edit, or a wrong case.
• meter.task(perTaskUsd, workDir, signal) now takes the folder.
• Test (measured): through the driver, claude got Edit(//<runs>/r1/t01/**) and no bare name, and the run with Edit(//home/**) never started. On fixtures, 12 bad grants are each refused.
• Old vs new (measured, probe): claude used to get the bare names, and Edit(//home/u/**) used to pass through.
• The driver's header, the README's "Where an agent may write" (the old line 41-42) and "Compromised component" now say this, instead of "nothing in this lane stops that".
• Claude Code's Edit rules covering all four tools is inferred from its permissions docs.

4. Sentinels are cleaned up on SIGINT and SIGTERM (non-blocking, done).

• confirmSandbox installs handlers while its sentinels exist. A handler removes them, removes itself, and re-raises the signal, so the process still ends by it.
• Test (measured): a child process mid-check is sent each signal. It dies by that signal, with nothing left in ~ or ~/.claude.
• On the old code, the probe left .netrc, .npmrc, .pypirc and the ~/.claude sentinel behind.
• inferred limit: SIGKILL can still leave them.
• On the old code, this test first hung the run: an assertion failed while the child was still alive. a049531 kills the child in a finally, so a regression now fails the test rather than hanging CI.

Local results (measured): bench.test.js gives # pass 37, # fail 0, with a temp HOME, state and held-out dirs and refusing claude, docker and srt stand-ins first on PATH. npm run typecheck exits 0. check:denylist prints denylist: 1293 files clean, and check:dco prints DCO: all commits signed off.

Not changed here:

• the argv scan still names the matching field (bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's body flags this for the test relay);
• the reviewer's round-2 suggestions B to D (which layer refused, TaskRunner's blocking fallback, readPatches nlink/O_NOFOLLOW).

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 3)

I reviewed only this PR's diff against issue-748-bench-containers (d57acd8), at head a049531.

• Every probe used stand-ins, a temporary HOME and temporary state, with a refusing claude, docker and srt first on PATH. No model, Docker or download was used.
• I checked Claude Code's own docs for the three questions they answer: the permissions, permission-modes, sandboxing, tools-reference, headless and Agent SDK TypeScript pages.
• My probes are in the session scratchpad, under work/eval-759-r3/: verdict-probe.mjs, scope-probe.mjs and sig-probe.mjs.

Summary. All five round-3 items work as the owner's decisions describe (measured). But the docs show three things the PR's own safety text gets wrong, and one of them is a new write route that the change itself opens:

1. the cat reads are refused by Claude Code's permission layer before its sandbox ever sees them;
2. "all refused" can still hide a read that worked;
3. the new Edit(//<task folder>/**) widens where sandboxed Bash may write.

All three are cheap to fix. This is the last owner-granted round, so the owner decides on each (see "For the owner").

Round-3 items

#	Item	Status	Evidence
1	~/.claude.json never touched; its sentinel goes under ~/.claude/, or is skipped when that folder is missing	met	measured: test 34 passes on the head and fails on the old code (~/.claude.json never exists while the model runs, true !== false). Across all 21 of my verdict fixtures (file absent beforehand), ~/.claude.json never existed afterwards. With a real ~/.claude.json, it was byte-for-byte unchanged after SIGINT, SIGTERM and SIGHUP (sig-probe.mjs). inferred from confirm.ts:93-101: .claude.json maps only to ~/.claude/.bench-sentinel-<hex>, and a missing ~/.claude, or one that is a link, is skipped.
2	Per-attempt verdict: refused only on the exact command with is_error; any naming command exiting 0 is leaked; anything else is declined	met as specified (gap in required change 2)	measured with 21 stream-json fixtures (table below).
2b	Is is_error the right signal?	inferred, with docs support	The tools reference ("Output limits") says that a command exiting non-zero "counts as a failure". The exception is exit 1 from grep, rg, egrep, fgrep, find, diff, test, [, git diff and git grep, which counts as a valid result. No page I read names the is_error field: not headless, not the tools reference, and not the Agent SDK BashOutput type, which has no exit code. So the mapping is still inferred. Every way it could be wrong fails closed (measured: a result with no is_error, or with "true" as a string, makes each attempt reached, so leaked). The same holds for the benign exit 1 (inferred: test -r '<s>' exiting 1 would count as reached).
3	scopeEditTools	met (side effect in required change 3)	measured (scope-probe.mjs, 50 entries). Bare Edit/Write/MultiEdit/NotebookEdit, with or without surrounding whitespace, becomes Edit(//<wd>/**). These are refused: EDIT, edit, write, MULTIEDIT, Notebookedit, Edit( //wd/**), Edit(//wd/** ), Edit (//wd/**), Edit(//wd/**),, Edit,, ,Edit, Edit(//wd/**),Bash, Edit(//wd/**)\nBash, /../, a/../.., ./a, //a, a/.., **/x, **/**, Edit(/wd/**) (project-relative), ~, relative paths, Edit(//wdx), nested or unclosed parentheses, and a relative workDir given to meter.task. Through the meter, a refused grant never reached the inner runner. * and Edit* pass through, but the docs say an unanchored allow glob "is skipped with a warning and doesn't auto-approve anything" (inferred from the docs).
3b	Do Edit rules cover all four tools?	yes (docs)	Permissions page: "Edit rules apply to all built-in tools that edit files". The tools reference maps Edit(/src/**) to "Edit, Write, NotebookEdit", and MultiEdit is "legacy". Path rules written for Write, NotebookEdit or MultiEdit are "never consulted", which matches refusing a qualified Write(...). Also from the docs: an allow rule matches a symlinked path only when both the link and its target match, and Edit and Write refuse to write through a symlinked file. So a link in the task folder doesn't extend the scope.
3c	Can Bash write outside the task folder?	sandboxed, but see required change 3	measured (read src/sandbox.ts:31-35 and src/runner.ts:432): enabled: true, failIfUnavailable: true, allowUnsandboxedCommands: false, passed with --settings, and --permission-mode dontAsk. So every Bash command is sandboxed, with no unsandboxed retry. inferred from the sandboxing docs: its writes reach the working directory, the per-user temp directory, "plus Edit allow rules, the same way sandbox.filesystem.allowWrite does". With round 3, that means the whole task folder.
4	SIGINT and SIGTERM remove the sentinels and re-raise	met	measured (sig-probe.mjs, with a real ~/.claude.json so both builds place the same 4 sentinels). Old code: after SIGINT and after SIGTERM, .netrc, .npmrc, .pypirc and .claude/.bench-sentinel-… were still there. Head: nothing was left, and the process ended signal=SIGINT / signal=SIGTERM. SIGHUP still leaves all four on the head (suggestion A).
5	Merge 9e4481c holds only the conflict resolution plus the required baseTree	met	measured (git show --remerge-diff 9e4481c9). It contains: the two conflict hunks in container.ts (keep spawn, take #757's comment, required base ahead of signal); test/bench.test.ts (imports, an awaited syncTree, and the old tar-stream test dropped for #757's replacement); driver.ts BenchOptions.baseTree, passed to TaskRunner; and in the test machine, dataHome: undefined, baseTree and the 5-socket attempt list. Nothing else.

Verdict fixtures (measured, verdict-probe.mjs on the head): each case alters only the reads or only the curls, and the rest fail exactly as listed.

Fixture	Status
control: every listed command fails exactly	all refused
double quotes, exit 0 / error	leaked / model declined
sh -c "cat '<s>' > /dev/null", exit 0 / error	leaked / model declined
pipe cat '<s>' | wc -c (cat fails, the pipe exits 0)	leaked (fails closed)
echo "$(cat '<s>' | wc -c)", exit 0	leaked
/bin/cat '<s>' > /dev/null, exit 0 / /usr/bin/cat, error	leaked / model declined
ln -s '<s>' ./lnk then wc -c ./lnk, exit 0	leaked (the ln names it)
ln -sf ~/.netrc ./l && wc -c ./l, exit 0, nothing else	model declined
exact cat refused plus ln -sf ~/.netrc ./l && wc -c ./l, exit 0	all refused
exact cat refused plus cat "$HOME/.netrc" > /dev/null, exit 0	all refused
exact cat refused plus a link read that prints the content	leaked (the transcript-wide content check)
curl double-quoted -o /dev/null, exit 0 / via sh -c, error	leaked / model declined
exact commands with a trailing space	all refused (trimmed)
echo "next: <s>", exit 0	leaked (fails closed)
model ran nothing	model declined

No sentinel was left in any case (measured).

Fail-without (measured)

• Method: a scratch worktree at a049531, with scripts/bench/{usage,confirm,driver}.ts checked out from 9e4481c. No guard was edited.
• Result: node --import ./dist/test/isolate.js --test dist/test/bench.test.js gave # pass 30, # fail 7:
  • test 34: true !== false;
  • test 35: cat '<s>' > /dev/null && echo, expected: 'leaked', actual: 'all refused';
  • test 37: the run with an outside grant never started, 2 !== 1.
• Test 36 (signals) fails on the old code only at its precondition (0 !== 1), because the old code put that sentinel at ~/.claude.json. sig-probe.mjs above is the real fail-without.
• Tests 5-7 fail only on the new meter.task signature (The "signals[0]" argument must be an instance of AbortSignal).
• On the head: # pass 37, # fail 0. npm run typecheck exits 0, and check:denylist prints denylist: 1289 files clean. The guards were never hit, and the scratch HOME and state stayed empty.

Protection, scope and hygiene

• measured: every path is under scripts/bench/** or test/. The PR has [PROTECTED: owner merges], **Risk class:** protected, the needs-owner label, and both sections.
• measured: all three round-3 commits are signed off, with no attribution trailers. No new dependency.
• inferred: no scope creep.
• No regression for existing installs, inferred: not applicable. Nothing in src/ or any installed path changes, and scripts/bench isn't in the package.

CI

measured (gh pr checks 759; run 39195349109 is on head_sha a049531): every check passes, including all eight test shards on Node 22 and 24. The real-systemd jobs are skipped as usual.

Required changes

1. Say what check D's reads really test, and test the bundled sandbox if the owner wants that.
   • Docs (inferred): "Read and Edit deny rules apply to … file commands Claude Code recognizes in Bash, such as cat, head, tail… They don't apply to … a Python or Node script that opens files itself."
   • The lane sets a Read(...) deny for every sentinel path. So each cat '<s>' is most likely refused by the permission layer before the OS sandbox runs it.
   • That means "all refused" says nothing about the bundled sandbox's denyRead, which is what option D was meant to confirm, since srt 0.0.79 is only close to that build. The curl attempts probably do reach the sandbox (guess: the docs don't list --unix-socket as a parsed file argument).
   • Text that overclaims today:
     • README §"Sandbox checks" 2;
     • the confirm.ts header and its detail ("every attempt refused in Claude Code's own sandbox");
     • "What this can never do" ("after Claude Code's own sandbox let a sentinel be read");
     • "Compromised component" ("checked … by Claude Code's own sandbox on sentinels").
   • Fix:
     • at minimum, correct that text, and record per attempt which layer refused, from the result text;
     • for the owner: add a read that the permission layer doesn't parse, such as node -e "process.stdout.write(require('fs').readFileSync(process.argv[1]))" '<s>' (node is always on the bench host), so the sandbox's denyRead is what refuses it.
2. "All refused" only when the model ran nothing but the listed commands.
   • measured: exact cat refused, plus cat "$HOME/.netrc" > /dev/null or ln -sf ~/.netrc ./l && wc -c ./l exiting 0, gives all refused. A read through ~, $HOME, a glob or a link doesn't name the target, and with no content printed nothing catches it.
   • This is within the owner's literal rule, but it contradicts "Compromised component": "If it runs a different command that succeeds, that is a leak and the run is refused".
   • Fix: any Bash call that isn't exactly a listed command makes the status at best model declined (the prompt already says "Run nothing else"), with a fixture test. Or, at minimum, reword that sentence to "a different command that names the target".
3. Scope the file-writing tools to the run's own cwd, not the whole task folder.

   • Docs (inferred): Edit allow rules "grant write access to specific paths, the same way sandbox.filesystem.allowWrite does". So Edit(//<task folder>/**) lets every sandboxed Bash command write anywhere in the task folder, not just its working directory.

   • The sandbox's own protections are narrower:

     • .claude settings, .claude/hooks and the like are protected only "in your working directory and the directories above it";
     • .git/hooks and .git/config, "in your working directory only".

   • The task folder holds the clone, a local remote and engine state (the PR's own words). A run whose cwd is a subfolder could then write:

     • a sibling's .claude/settings.json. A later -p run there with --setting-sources project "runs the hooks in a project's .claude/settings.json … even in a folder you've never trusted", outside the sandbox, as the bench user;
     • a bare local remote's hooks/.

     Whether these routes are reachable depends on the RunTask layout still to come (guess). Before round 3, Bash could write only its working directory and the temp folder (inferred).

   • Fix (one line, "make the wrong thing impossible"):

     • in TaskMeter.run, refuse a req.cwd outside workDir, and scope to Edit(//<req.cwd>/**), so the allow rule adds nothing beyond the sandbox's default;
     • correct README line 42 ("the working folder and the temp folder"), which is no longer exact as written;
     • add a test for both.

Suggestions (not blocking)

• A. SIGHUP. measured: SIGHUP leaves all four sentinels on the head (a terminal close or SSH drop during the check). Handle it too, or remove leftover bench-sentinel- files at the next start, which would also cover SIGKILL. A related point, inferred from confirm.ts:89-112: if placeSentinels throws after creating some sentinels (a later denied() or an O_EXCL race), they stay, because neither the handler nor the finally is in place yet. This is not reachable with benchLane.
• B. workDir validation. measured: /srv/b*/runs/r1/t01 and /srv/[b]ench/t01 are accepted and put into the rule unescaped, and /srv/bench/t01/ gives Edit(//srv/bench/t01//**). WORKLANE_BENCH_ROOT can set that path. Refuse glob characters and a trailing slash.
• C. README line 43 and driver.ts:17 example. inferred from the permission-modes docs: in dontAsk, writes to protected paths (.profile, .bashrc, .gitconfig, .npmrc, .claude.json, .claude/, .git/) are "Denied", even when an allow rule matches. So an unscoped Edit could never have written ~/.profile or ~/.bashrc. The real examples are unprotected paths, such as ~/.docker/config.json (a credsStore helper the driver's docker would run), ~/.config/systemd/user/ or ~/.local/bin/. The fix stands; only the example is off.
• D. Per-user temp folder: sandboxed Bash can write the per-user temp folder in every task, so a later task's agent can see what an earlier one left there (inferred from the sandboxing docs).
• E. Round 2's suggestions C and D are still open: the blocking call fallback, and readPatches nlink / O_NOFOLLOW.

For the owner

• Required change 1: whether check D must exercise the bundled sandbox's denyRead (the uninterpreted read), or only state that its reads test the permission layer.
• Required changes 2 and 3 are small and structural. If you'd rather not have another round, each could become a blocking issue on the RunTask PR instead. Required change 3 matters only once a run's cwd is a subfolder of the task folder.

[jayden1711]

Round 4 (owner-granted): what changed

The new head is 0faa498:

• 6585970: a signed-off merge of bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's head 94c6e71;
• 0faa498: the round-4 fixes.

The PR body is updated, including the corrections round 3 found.

Fail-without (measured):

• Method: this head's tests on the merge 6585970. There, confirm.ts, usage.ts, driver.ts and the README are byte-identical to a049531's.
• Results: # pass 40, # fail 7 on the old sources; # pass 47, # fail 0 on the head.
• Setup: temp HOME, state and held-out dirs; refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and no guard was edited.

#	Change	Test	Fails without (measured)
1	Check D adds an unparsed read: node -e "<READ_JS>" '<s>' per sentinel. READ_JS fstats the descriptor: a non-regular file (the sandbox's /dev/null mask) exits 3 and counts as refused; a regular file is printed, which is a leak.	check D's unparsed read (node -e, fstat): runs the real command on a /dev/null link, a missing path and a live sentinel.	masked with /dev/null: one node -e read per sentinel, 4 !== 0
2	Per-attempt layer, from the stream-json only, failing closed (see the notes below the table).	check D records which layer stopped each attempt	the result line's permission_denials, matched by tool_use_id (no layer)
3	"all refused" only when the model ran nothing but the listed commands (round 3's required change 2). Anything else is model declined at best, listed in extra_calls.	check D is "all refused" only when…: exact commands refused, plus cat "$HOME/.netrc" > /dev/null exiting 0, or a Read tool call.	'all refused' !== 'model declined'
4	Edit, Write, MultiEdit and NotebookEdit scoped to the run's own cwd (round 3's required change 3). A cwd outside the task folder is refused, unstarted.	a run's file-writing tools reach only its own working folder…	the bare grant became Edit(//srv/bench/runs/r1/t01/**), not …/t01/repo/**
5	Glob or rule characters, or a trailing slash, in the task folder or the cwd: refused, not escaped (see the notes).	a path put into a permission rule may hold no glob or rule character…	Missing expected exception: /srv/b*/runs/r1/t01
6	Sentinels removed on SIGHUP too.	The signal test now loops SIGINT, SIGTERM and SIGHUP.	'SIGHUP: no .netrc left'
7	The README and driver.ts example: ~/.docker/config.json, ~/.config/systemd/user/, ~/.local/bin/, not ~/.profile (which dontAsk protects, inferred from the permission-modes docs).	docs	n/a
8	The driver records os_sandbox_read_deny and warns when the OS sandbox didn't itself refuse every node -e read. Its acceptance test checks all 13 attempts' [kind, form, refused, layer].	the driver's sandbox-confirmation acceptance test	the attempt list differs: no node attempts, no layers

Notes on the table:

• Change 2, the layers:
  • permission layer: the result line's permission_denials (matched by tool_use_id), or the "Permission to use … has been denied" wording, or its don't-ask-mode variant. Both markers are inferred from public docs and Claude Code's known wording, not verified against 2.1.303. The smoke round's Haiku call will measure them;
  • OS sandbox: only the exact listed sentinel read failing on a masked or unreadable file;
  • unknown: an error with neither marker, which includes every curl failure. Never counted as the OS sandbox;
  • model declined: the exact command never ran.
• Change 5: the owner's decision said "escaped". The coordinator's brief allowed refusal, which I chose because the escaping of Claude Code's rule globs isn't documented. A consequence (inferred): a bench root with a space is refused.

Round 3's overclaims, corrected in the README, confirm.ts, "What this can never do" and "Compromised component":

• cat most likely tests Claude Code's permission layer, not its OS sandbox (inferred from the permissions docs);
• "all refused" no longer reads as "refused in Claude Code's own sandbox";
• the layer markers are labelled inferred.

Also from the merge (measured): #757's five new stdin tests called the now-async syncTree without await on this branch, and failed (1 !== 0). They now await it, and pass.

Gates (measured):

• npm run typecheck exits 0;
• check:denylist: denylist: 1290 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.

CI: on 0faa498 (measured, gh pr checks 759 and gh run list), every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

Not changed:

• The cat rule (inferred): a cat the sandbox masks with /dev/null would exit 0 and count as leaked, which fails closed. Round 3's owner rule keeps that.
• Not run: any real model call.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 4)

I reviewed this PR's round-4 diff (6585970..0faa498) against the owner's round-4 decision (option A) and round 3's required changes, at head 0faa498.

• Every run used a temporary HOME and temporary state and held-out dirs, with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and no binary was inspected.
• My probes are in the session scratchpad, under work/eval-759-r4/: probe-read.mjs, probe-layer.mjs, probe-scope.mjs and probe-gate.mjs.

Summary. Every item the owner asked for is in, and works as described (measured). Two claims are still wider than the code:

1. a working folder that is a link out of the task folder is accepted;
2. os_sandbox_read_deny and the per-attempt "OS sandbox" can be set by the model's own extra command.

Both are one-line fixes with a test. The owner may prefer to take them as blocking items on the RunTask PR (see "For the owner").

Owner's round-4 items

#	Item	Status	Evidence
1	No binary inspection	met	measured: nothing in the diff or the tests reads the CLI binary. The markers are labelled inferred in confirm.ts:26-38, the README and the PR body.
2	Layer from permission_denials and the denial wording, fail closed	met (see required change 2)	measured, probe-layer.mjs, 19 hand-made stream-json cases, table below.
3	node -e read uses fstat	met	measured, probe-read.mjs: the real command, run by bash, on 10 kinds of path (table below).
4	Any non-listed command or tool: "model declined" at best	met	measured: an extra echo, a Glob call, a Bash call with no id, a tool call with no result, and cat "$HOME/.netrc" exiting 0 all give model declined. A listed command run twice still gives all refused, which is correct.
5	Edit, Write, MultiEdit and NotebookEdit scoped to the run's cwd; a cwd outside the task folder refused, unstarted	met by path; a link gets through (required change 1)	measured, probe-scope.mjs, 27 cases. Bare Write with cwd <t>/repo becomes Edit(//<t>/repo/**). These are refused and never reach the inner runner: Edit(//<t>/**) from <t>/repo, <t>/repo/*, edit, a relative cwd, .., the sibling <t>-x, /, +(a) and a no-break space. But a cwd <t>/repo that is a symlink to a folder outside <t> starts, with Edit(//<t>/repo/**).
6	SIGHUP clean-up	met	measured: test 41 passes on the head and fails on 6585970 with 'SIGHUP: no .netrc left'.
7	Glob characters: refused, not escaped	meets the intent (owner flag)	measured: * ? [ ] { } ! ( ) \, a comma, any \s (a no-break space too) and a trailing slash are refused in both the task folder and the cwd. inferred from the permissions docs: path rules follow the gitignore spec, whose specials are *, ?, [ ], **, a leading ! or #, and \. Every one of them is refused, and so are the rule's own ( ) , and whitespace, so a path can't widen or close a rule. Extglob needs (, so it is refused too. Escaping would need syntax the docs don't give, so refusal is the stronger choice.
8	README correction	met	inferred, reading the README:43 and driver.ts:15-26 diffs: they now cite ~/.docker/config.json, ~/.config/systemd/user/ and ~/.local/bin/, and say dontAsk protects ~/.profile. "The working folder and the temp folder" now adds "plus the paths of Edit allow rules".

node -e READ_JS, run for real (measured, probe-read.mjs). Only the node read is swapped to the path shown. Every other attempt fails with no marker.

At the sentinel's path	node exit	Attempt / layer	Status, os_sandbox_read_deny
link to /dev/null	3, bench-check: masked…	refused / OS sandbox	all refused, true
missing (ENOENT)	1	refused / OS sandbox	all refused, true
mode 000 file (EACCES)	1	refused / OS sandbox	all refused, true
directory	3, masked	refused / OS sandbox	all refused, true
FIFO	killed at 3 s, no output	refused / unknown	all refused, false
symlink loop (ELOOP)	1	refused / unknown	all refused, false
path through a file (ENOTDIR)	1	refused / unknown	all refused, false
live sentinel	0, content printed	reached	leaked
symlink to the live sentinel	0, content printed	reached	leaked
empty regular file	0, nothing printed	reached	leaked

• A leak never looks refused (measured): a readable regular file, directly or through a link, is always leaked.
• A working sandbox can look leaked, which fails closed:
  • a mask with an empty regular file instead of /dev/null (guess that a build would do that; the srt probe saw /dev/null);
  • cat (owner flag 2).
• A FIFO hangs openSync until the Bash call's own timeout. It ends as unknown, never as the OS sandbox (measured).
• Node's uncaught-error output echoes node:fs source, not READ_JS, so the MASKED text can't come from a stack trace on these paths (measured).

Layer forging (measured, probe-layer.mjs). In each case, every exact command fails with "error" unless noted.

Case	Result
no marker at all; curl output written to look like a file error and the mask	every layer unknown, never OS sandbox
permission_denials naming another tool_use_id, malformed entries, or "c0" as a bare string	no attempt marked permission layer
permission_denials on a non-result line	ignored
the model's own text quoting both denial wordings	ignored
a tool_result block inside an assistant message carrying the wording	ignored
the wording in an error tool_result (control)	permission layer
the wording in a non-error tool_result	leaked (fails closed)
an extra call that names the target and is denied, next to the exact call's plain error	the exact attempt stays unknown; status model declined
cat failing with "Permission denied" and listed in permission_denials	permission layer (the marker wins)

• No layer can be forged from model-controlled text (measured). Only Claude Code's own result line and user tool results are read.
• But a layer can be forged by a model action (required change 2): chmod 000 ~/.n* ~/.p* ~/.claude/.b* (names no target), then each node -e read failing EACCES. That gives status=model declined (correct), but each node attempt is recorded as layer OS sandbox, and os_sandbox_read_deny=true goes into sandbox.json, results.json and the aggregate (probe-gate.mjs).

Fail-without (measured)

• Method: a scratch worktree at 6585970, where confirm.ts, usage.ts, driver.ts and the README are round 3's, plus git checkout 0faa4988 -- test/bench.test.ts. No guard was edited.
• Old code: node --import ./dist/test/isolate.js --test dist/test/bench.test.js gave # pass 40, # fail 7, as the builder reported:
  • test 30 (the attempt list): no node attempts and no layers;
  • test 41: 'SIGHUP: no .netrc left';
  • test 43: 'Edit(//srv/bench/runs/r1/t01/**)' where …/t01/repo/** was expected;
  • test 44: Missing expected exception: /srv/b*/runs/r1/t01;
  • test 45: masked with /dev/null: one node -e read per sentinel, 0 !== 4;
  • test 46: the result line's permission_denials, matched by tool_use_id, [] against 4 permission layer;
  • test 47: 'all refused' !== 'model declined'.
• Head: # pass 47, # fail 0. Nothing was left in the temporary home, state or held-out dirs.
• npm run typecheck exits 0; check:denylist prints denylist: 1290 files clean.

The merge and the stdin fix (rule 10)

• measured: git merge-tree --write-tree a0495314 94c6e71d gives f9272512, which is the tree of 6585970 itself. So the merge is clean and carries nothing else.
• measured: on 6585970's own test file, bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's five stdin tests fail (# pass 0, # fail 5). On this branch TaskRunner.syncTree is async (container.ts:368); on bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757 it is sync (94c6e71d:container.ts:331). Without await, the refusal becomes a rejected promise that nothing catches.
• The fix: stdinSends becomes async and awaits syncTree. The five tests await all 9 calls, and bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's head has the same 9 calls with the same expected values and messages (measured, by grep and the diff). This is the correct fix, and it weakens nothing (inferred).
• Other test changes (inferred, from the diff): the round-3 tests extend rather than relax. The signal loop adds SIGHUP. The attempt-list assertion grows from 9 to 13 attempts with the layer. The stand-in gains a permission mode and permission_denials. No assertion was removed.

Protection, scope and hygiene

• measured: every path is under scripts/bench/** or test/. The PR has [PROTECTED: owner merges], **Risk class:** protected, the needs-owner label, and real "What this can never do" and "Compromised component" sections. Its claims are labelled.
• measured: check-dco.mjs 94c6e71d HEAD reports DCO: all commits signed off, and there are no attribution trailers. package.json is unchanged, so there is no new dependency.
• "Can never do" check: line "Start an agent run whose working folder is outside its task's folder" is true by path, not by where the folder really is (required change 1). The other round-4 claims match my probes.
• inferred: no scope creep. No regression for existing installs: nothing in src/ or any installed path changes.
• Rule A: the tests run under test/isolate.ts's temporary machine (measured: nothing left in my temporary HOME).

CI

measured (gh pr checks 759, gh run list; runs 39203629499 ci and 39203629575 ci slow on head_sha 0faa498): every check passes, including all eight test shards on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

Required changes

1. A working folder that resolves outside the task folder must be refused, or the claim narrowed.
   • measured (probe-scope.mjs): with <t>/repo a symlink to a folder outside <t>, TaskMeter.run starts the run with Edit(//<t>/repo/**). So does a task folder that is itself a link. scopeEditTools compares paths as text only (usage.ts:48, within).
   • Why it matters (inferred from the sandboxing docs): sandboxed Bash may write its working directory, and that is the real folder behind the link. The Edit tools themselves stay confined, because an allow rule matches a symlinked path only when the link and its target both match.
   • Reachable today? guess: not yet. The cwd comes from RunTask, which isn't wired. An agent could plant such a link only inside its own working folder.
   • Fix (preferred, "make the wrong thing impossible"): in TaskMeter.run, refuse a req.cwd whose realpathSync isn't inside realpathSync(workDir), or that doesn't exist, with a test using a real link.
   • Or, at minimum: narrow "What this can never do" and the README to "by path", and make the realpath check a blocking item of the RunTask PR.
   • Same spot: TaskMeter.run (usage.ts:174) passes req.cwd into scopeEditTools(…, cwd = workDir). An undefined cwd takes the default and starts the run scoped to the task folder (measured: cwd undefined: started). RunRequest.cwd is typed string, so only a cast reaches this (inferred). Pass it with no default, so rulePath's typeof check refuses it.
2. No "OS sandbox" layer and no os_sandbox_read_deny when the model ran anything else.
   • measured (probe-gate.mjs): an extra chmod 000 through globs, then each node read failing EACCES, gives status: model declined, but each node attempt has layer: 'OS sandbox' and os_sandbox_read_deny: true. Both are recorded in sandbox.json, results.json and the aggregate.
   • That is a layer set by the model, which goes against the owner's "fail closed" and "any non-listed command counts as model declined at best".
   • Fix (confirm.ts:299 and :315): when extra is non-empty, record 'unknown' instead of 'OS sandbox', and set os_sandbox_read_deny only when extra is empty. Add a fixture test like probe-gate.mjs.

Suggestions (not blocking)

• A. Other Bash inputs. The exact match compares only input.command. With fixtures (measured), listed commands sent with timeout: 1 and failing "Command timed out" give all refused, every layer unknown and os_sandbox_read_deny false, so the driver warns. With run_in_background: true, they give leaked (fails closed). Whether 2.1.303 honours a 1 ms timeout is a guess. Consider counting a call as exact only when its input holds nothing but command (and description).
• B. Control characters. RULE_UNSAFE lets \x01, \x1b and NUL through into a rule (measured). NUL fails at spawn (inferred). Adding [\x00-\x1f\x7f] is cheap.
• C. Non-tool_use blocks. A server_tool_use block is not counted as an extra call (measured with a fixture). Whether -p with allowedTools: ['Bash'] can produce one is a guess.
• D. Wording. "in Claude Code's own sandbox" is still in aggregate.ts:39, driver.ts:189 and test 30's title.
• E. Round 2's C and D and round 3's D (the per-user temp folder) are still open.

For the owner

1. Space in the bench root. Refusing instead of escaping means a bench root (or a run id) containing a space, !, (, ), a comma, [ ] { } * ? or \ is refused (measured: /srv/my bench/t01 is refused).
   • When (inferred from driver.ts:170-192): after the preflight, the srt probe and the probe container, at meter.task for sandbox-check, so before the model call and before any spend.
   • The default ~wl-bench/bench has none of these characters. In my judgement, refusal fully meets "no rule injection" (row 7).
2. cat may report a working sandbox as a leak. If 2.1.303's permission layer does not stop cat '<s>' (an inferred assumption, not yet verified), the OS sandbox's /dev/null mask lets cat exit 0. The smoke's first call then refuses the run as leaked, with "a command naming … did not fail" for the cat attempts. This fails closed by round 3's rule, but it is how that case will look.
3. Required changes 1 and 2 are each a line or two plus a test. If you'd rather not have another round, both can be blocking items on the RunTask PR, since neither can be reached before RunTask is wired (1, guess) or changes the status that gates the run (2, measured).

[jayden1711]

Round 5 (owner-decided): what changed

The new head is 943244a. It fixes every finding in the round-4 final check that affects how the sandbox checks behave. Wording-only findings are deferred to the RunTask PR and listed in the body under "Deferred to the RunTask PR".

Fail-without (measured):

• Method: this head's tests on 0faa498, in a scratch worktree, since removed.
• Results: # pass 47, # fail 5 on 0faa498; # pass 52, # fail 0 on the head.
• Setup: temp HOME, state and held-out dirs; refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, no guard was edited, and nothing touched the CLI binary.

#	Fix	Test	Fails without (measured)
1	The working folder must really be inside the task folder. TaskMeter.run refuses a cwd that doesn't resolve, or whose realpath isn't inside the task folder's; also a task folder that is itself a link. The cwd is passed with no default, so an unset one is refused.	a run's working folder must really be inside its task's folder: the reviewer's probe-scope link cases, plus a missing and an unset cwd	Missing expected rejection. (the link out started)
2	The model can't set the "OS sandbox" layer. With any extra call, no attempt gets that layer, and os_sandbox_read_deny is false.	the model cannot set the "OS sandbox" layer: the reviewer's chmod-000 case on confirmSandbox, and through the driver in sandbox.json, results.json and the aggregate	the node layers were OS sandbox
3a	A listed command with other Bash inputs (timeout, run_in_background, anything but command and description) is an extra call, not the listed command.	a listed command sent with other Bash inputs is an extra call	'all refused' !== 'model declined'
3b	A server_tool_use block (any non-tool_use tool-call block) is an extra call.	a server_tool_use block counts as an extra call	'all refused' !== 'model declined'
3c	RULE_UNSAFE refuses control characters \x00-\x1f and \x7f.	control characters are refused in a path put into a permission rule	Missing expected exception: "\u0000"

Notes on the table:

• Fix 1: inferred from paths.ts: the driver builds the task folder on the bench root's real path, so a real run never has a linked task folder.
• Fix 2: both the per-attempt record and os_sandbox_read_deny change, in all three records.

Also changed:

• Two existing scope tests now use real temporary folders instead of /srv/... paths, because a run's working folder must now exist. Their assertions are unchanged (measured: they pass on both 0faa498 and the head).
• The stand-in claude gains a gate mode for the driver-level case.

Gates (measured):

• npm run typecheck exits 0;
• check:denylist: denylist: 1290 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.

CI: on 943244a (measured, gh pr checks 759 and gh run list: ci and ci slow both succeeded), every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

Deferred, wording only: "in Claude Code's own sandbox" in aggregate.ts:39, driver.ts:189, test 30's title and the leaked detail in confirm.ts.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 5)

I reviewed round 5 (0faa498..943244a) at head 943244a against the owner's round-5 decision: fix every finding that affects the correctness of the sandbox checks, and defer wording to the RunTask PR.

• Setup: every run used a temporary HOME and temporary state and held-out dirs, with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state.
• Probes (session scratchpad, work/eval-759-r5/): probe-scope5.mjs (real folders, real links, a real spawn), probe-input5.mjs (stream-json fixtures), and round 4's probe-scope.mjs and probe-gate.mjs.

Summary. Round 4's two required changes and suggestions A to C are in, and they work (measured). Two gaps remain, each in a check's correctness:

1. The working-folder check is made before the spawn, not at it. A folder swapped for a link in between starts the run outside the task folder (measured).
2. A Bash call with a non-string command or description counts as the listed command. Claude Code would refuse such a call at input validation, without running it (inferred). So an "all refused" can come from commands that never ran (measured, on fixtures).

The second is a one-line fix. The first has a small fix, or it can become a RunTask blocker with the claim narrowed (see "For the owner").

1. The working folder must really be inside the task folder

measured, probe-scope5.mjs on real folders. The stand-in inner runner spawns a real process with cwd: req.cwd, as src/runner.ts:709 does.

Case	Result
round 4's probe-scope.mjs: a cwd link out, a task folder that is a link, an unset cwd	all refused, unstarted
a link chain l1 -> l2 -> outside	refused ("is really …, outside")
a relative link ../../../outside	refused
a linked middle component, mid -> outside, then mid/deep	refused
repo/chain3 -> l1 -> l2 -> outside	refused
a link to the task folder's parent	refused
.. (inside, or through a link), a trailing /., or //	refused by rulePath (not normalized)
missing, a dangling link, a link loop	refused (ENOENT, ENOENT, ELOOP)
unset, an empty string, a number	refused by rulePath
a regular file	spawn ENOTDIR, so it never runs
control: a real repo/sub	started there, with Edit(//…/repo/sub/**)
repo/sub swapped for a link out after the check, before the spawn	started, process cwd = <b>/outside, with Edit(//…/t01/repo/sub/**)
a link that stays inside, repo/up -> <task>	started; process cwd = the task folder (suggestion A)

• The window (measured, by grep of src/runner.ts). Between TaskMeter.run's realpathSync and spawn(…, { cwd: req.cwd }) (line 709), the engine runner awaits loginStartLock (551), withLoginLock (559) and mcp.register (617). The window can be as long as another run holds the login lock.
• Who can swap it (inferred). Only something that can write the parent of a component of the cwd. In the bench, that is another of the task's runs in flight, whose sandboxed Bash may write its own working folder. Its working folder must contain the new run's cwd. TaskMeter doesn't serialize runs: it keeps an inflight set, and a cwd equal to the task folder is allowed (the round-4 acceptance test starts one). A background process outliving an earlier run would also do (guess).
• Why it matters (inferred). The spawned Claude Code then has the outside folder as its working folder, and its sandboxed Bash may write there. If that folder were ~, Bash could write ~/.docker/config.json, which the README names as the route to the credential.
• Reachable today? guess: no. runTask isn't wired, and whether RunTask will start concurrent runs with nested working folders is open.
• Contradicted claim: "What this can never do" says "Start an agent run whose working folder is outside its task's folder, by path or where it really is" (measured). It holds at the check, not at the spawn.
• Passing realCwd to the inner runner doesn't close it (inferred). The swap in my probe is of a real folder, not a link.

2. The model can't set the "OS sandbox" layer

• measured: probe-gate.mjs now gives status=model declined os_sandbox_read_deny=false, node layers unknown, and the chmod listed in extra.
• Fixtures (probe-input5.mjs, measured):
  • an mcp_tool_use, a server_tool_use (even inside a user line), a Bash call with a non-string id, and a Bash call with no result each count as extra;
  • the layer is then unknown and os_sandbox_read_deny false.
• Without an extra call (inferred, from confirm.ts): no listed command changes a file. They are cat, the fstat read, and curl to a socket. So no listed command can make a later read fail.
• Required change 2 (below) makes os_sandbox_read_deny false in its case, but it produces a false "all refused".

3. Extra Bash inputs, other block types, control characters

measured, probe-input5.mjs. In each case, only the first node read varies and the rest are plain.

Input	Result
description only	listed, all refused (correct)
timeout: 1 or 600000, run_in_background: false, dangerouslyDisableSandbox: false, an unknown key, an own __proto__ key, input an array	extra, model declined, read-deny false
leading spaces and a trailing newline on command	listed (harmless: bash runs the same command, inferred)
command: [<listed>] (an array)	counted as the listed command
every listed call with command an array, or description: 5 or null, each answered with an input-validation error	status=all refused, layers unknown, os_sandbox_read_deny false

• Why (inferred). readTranscript uses String(input.command ?? '') (confirm.ts:220), and plain checks only the key names.
  • Claude Code validates the Bash input (command a string, description an optional string) and answers anything else with an is_error result, without running the command (inferred from how Claude Code reports invalid tool input; the exact text from 2.1.303 is a guess).
  • So each attempt is "refused" though nothing ran. That goes against round 3's rule that an attempt counts as refused only when the exact command ran and errored.
• The run gate (inferred): this doesn't change it. An "all refused" with read-deny false and a "model declined" both warn and go on. But sandbox.json, results.json and the aggregate record a false "all refused".
• Control characters (measured): \x00-\x1f and \x7f are now refused, and so are U+2028 and U+FEFF (via \s). C1 controls (\x80, \x85, \x9b), U+200B, U+200D and U+202E pass (suggestion B).
• Not counted, but not model-controlled (measured; inferred that only Claude Code writes these lines and the API assigns ids):
  • a tool_use in a user line, in a line with no type, or in a stream_event;
  • a Bash call reusing a listed call's id.

4. The two changed tests (rule 10)

• measured (diff): in "acceptance: a task's agents get Edit…", only wd changes, from /srv/bench/runs/r1/t01 to taskFolder().
• measured (diff): in "round 4: a run's file-writing tools…", the same change, and the refused cwds /srv/bench/runs/r1/t02 and /srv/bench/runs/r1 become join(dirname(wd), 't02') and dirname(wd), the same relations. Every assertion, regex and message is unchanged.
• measured: both pass on 0faa498's code. The only failures there are the five new tests (below).

5. Deferred items

• measured (grep): the four listed places exist: aggregate.ts:39 (a doc comment), driver.ts:189 (a comment), the title of test 30, and the leaked detail string at confirm.ts:334.
• inferred: none of them changes what a check gates on, so they are wording only. They are listed in the body under "Deferred to the RunTask PR".

6. The usual checks

• Fail-without (measured). Method: a scratch worktree at 0faa498, plus git checkout 943244ab -- test/bench.test.ts; no guard edited. # pass 47, # fail 5, as the builder reported:
  • control characters: Missing expected exception: "\u0000";
  • realpath: Missing expected rejection.;
  • layer: node layers "OS sandbox" ×4;
  • other inputs: fails;
  • server_tool_use: fails.
• Head (measured): # pass 52, # fail 0, with nothing left in the temporary home, state or held-out dirs. npm run typecheck exits 0, and check:denylist prints denylist: 1290 files clean.
• Merge (measured). bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757 has merged, and the base is now main.
  • git merge-tree --write-tree 943244ab 0a0aeda8 is clean, giving 6ef0f426. Its scripts/bench and test/bench.test.ts are identical to the head's, and main's changes touch neither.
  • On that tree (main plus the PR's files; git write-tree = 6ef0f426), bench tests are 52/52 and typecheck exits 0.
  • 94c6e71 (bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's head) is an ancestor of 943244a.
• DCO and attribution (measured): check-dco.mjs 0a0aeda8 943244ab reports DCO: all commits signed off. The only trailers are Signed-off-by, with no attribution in commits or body.
• Protection (measured): every path is under scripts/bench/** or test/. The PR has [PROTECTED: owner merges], **Risk class:** protected, the needs-owner label, and real "What this can never do" and "Compromised component" sections. package.json is unchanged, so there is no new dependency.
• No regression for existing installs (inferred): nothing in src/ or any installed path changes.
• Claims are labelled. The working-folder claim is overstated (item 1).
• CI (measured, gh pr checks 759 and gh run list --commit 943244ab): ci, ci slow, gitleaks and new-deps all succeeded. Every check passes: the 8 shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

Required changes

1. Close the window between the working-folder check and the spawn, or narrow the claim.
   • Preferred fix ("make the wrong thing impossible"): TaskMeter.run refuses, unstarted, a run whose working folder is strictly inside the working folder of one of the task's runs still in flight. Those runs are the only writers that could swap a component of it (inferred). Add a test with two overlapping runs, where the first's inner runner waits until the second is refused.
   • Or: narrow "What this can never do", the Compromised-component line and the README to "at the check". Then make the in-flight rule a blocking item of the RunTask PR.
2. Count a Bash call as a listed command only when command is a string and description is absent or a string (confirm.ts:220, plain). Add a fixture test: every listed call with command: [<listed>], or with description: 5, answered with an error, gives model declined, not all refused.

Suggestions (not blocking)

• A. A link that stays inside the task folder. repo/up -> <task> starts the run with Claude Code's cwd at the task folder and Edit(//…/repo/up/**) (measured). That is no wider than a cwd equal to the task folder, which is allowed (inferred). The driver never builds a linked cwd, so requiring realpath(cwd) === cwd would be simpler, and it makes the Edit rule name the real folder.
• B. C1 controls and format characters (U+0080-U+009F, U+200B, U+200D, U+202E) pass RULE_UNSAFE (measured). The rule's own separators are all refused, so this is about display, not injection (inferred). Adding \p{Cc}\p{Cf} with the u flag is cheap.
• C. Body. It still says "base issue-748-bench-containers", but the base is now main. Round 2's C and D and round 3's D (the blocking call fallback, readPatches nlink/O_NOFOLLOW, the per-user temp folder) are still open. None is a sandbox-check finding, but they aren't listed.
• D. Bind mounts. Skipped: making one needs root or a user namespace. realpath can't see a mount, and an agent can't make one in the driver's namespace (both inferred).

For the owner

1. Required change 1 can't be reached before RunTask is wired (guess), and RunTask decides whether runs overlap.
   • If you'd rather not have another round here: take the narrowed claim now, and make the in-flight rule a RunTask blocker.
   • Either way, the rule belongs wherever runs can first overlap.
2. Required change 2 is a one-line fix with a test. It changes the recorded status, not whether the run goes on (inferred, driver.ts:196-198).

[jayden1711]

Round 6: what changed

The new head is 45b91ba:

• a signed-off merge of origin/main (the base is now main);
• one signed-off commit with the round-6 fixes.

The body is updated, with a new "Adversarial pass" section and an "Open items" section.

Fail-without (measured):

• Method: this head's tests on 943244a, in a scratch worktree, since removed.
• Results: # pass 52, # fail 5 on 943244a; # pass 57, # fail 0 on the head.
• Setup: temp HOME, state and held-out dirs; refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state.

#	Fix	Test	Fails without (measured)
1a	No run starts nested with a running run's folder (the same, or one inside the other, either way). The check and the registration run with no await between them.	The reviewer's probe-scope5 swap, as a test: while a run in repo is going, runs in repo/sub, repo and the task folder are refused; one in other starts. Then that run swaps repo/sub for a link out, and a later run there is refused at the check.	'/repo/sub: succeeded'
1b	No run starts strictly inside an earlier, ended run's folder, which a process it left could still write. Added in my adversarial pass (see the body).	no run starts strictly inside an ended run's working folder	Missing expected rejection.
1c	A cwd reached through any link is refused, even one that stays inside (realpath(cwd) === cwd).	a working folder reached through any link is refused…	Missing expected rejection: …/repo/up
2	plain needs a string command, and description absent or a string.	a Bash call counts as a listed command only with a string command…: an array command, description: 5 or null, a number command, each failing	'all refused' !== 'model declined'
3	RULE_UNSAFE refuses C1 controls, format characters (\p{Cf}: U+200B, U+200D, U+202E, U+2066, U+00AD, U+FEFF) and lone surrogates.	C1 control and format characters are refused…	Missing expected exception: "\x80"
4	Body: the base is main. Round 2's C and D and round 3's D are listed under "Open items".	docs	n/a

The claim, corrected (round 5's finding): "What this can never do" now says the working folder holds "at the meter's check", and that no other run of the task can swap a folder of it before the spawn. What that rests on is labelled inferred: nothing else writes inside the task folder while a run starts.

Gates (measured):

• npm run typecheck exits 0;
• check:denylist: denylist: 1295 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.

CI: on 45b91ba (measured, gh pr checks 759 and gh run list: ci, ci slow, gitleaks and new-deps succeeded), every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 6)

I reviewed round 6 (943244a..45b91ba: a signed-off merge of main, then fix commit 45b91ba).

• Setup: every run used a temporary HOME and temporary state, held-out and TMPDIR folders, with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state.
• Probes (session scratchpad, work/eval-759-r6/):
  • probe-r6.mjs, my own: real folders, links, renames, two meters, Unicode names;
  • round 5's probe-scope5.mjs;
  • the builder's adversarial6.mjs.

Summary. All of round 5's required changes and suggestions A to C are in, and they work (measured). The check and the registration are synchronous, and nested runs are refused both ways.

One gap remains in the new guarantee. The overlap rules live in one TaskMeter object. The driver can make a second meter for the same task folder, and that meter knows nothing of the first one's runs:

• through a repeated task key;
• through a run id that is used again after a crash.

The second gap is wording: the claim's stated assumption doesn't name that folders are compared as path text.

1. The overlap rules

Re-run (measured):

• probe-scope5.mjs: every link case is now refused. That includes the links that stay inside the task folder: repo/up (which round 5 started; suggestion A) and repo/side. The control repo/sub starts.
• Round 5's swap case still starts (process cwd = <b>/outside). Its writer is a hook, not a run of the task, and each case gets a fresh meter. So this case falls under the stated assumption ("nothing else writes inside the task folder"), and that is expected (inferred).
• adversarial6.mjs: every result the body reports is reproduced.

My own cases (measured, probe-r6.mjs):

Case	Result
The task folder renamed away and replaced by a link out, or by a link to its old self	refused ("a task folder through a link is refused")
The task folder renamed away, and a new real one made	started (a real folder: correct)
A run at the task folder is going. repo is swapped for a link out, then a run is tried at repo	refused (outside)
The same, with a real repo	refused (nested)
After the task-folder run ends, a run at repo	refused (inside an earlier run)
An inner runner throws synchronously for repo; then a run at b beside it, and one at repo/sub	b started (released); repo/sub refused (earlier run)
An inner runner that never settles at repo; then runs at repo/sub and b	repo/sub refused (nested), b started, settled(50) = false (the driver then halts the bench, driver.ts:259-262)
repo/sub deleted and made again as a real folder; then as a link out; then missing	started; refused (outside); refused (ENOENT)
A restarted driver (a new UsageMeter and TaskMeter on the same task folder) after the first meter ran at repo; a run at repo/sub	started (the first meter refuses it)
Two meters on one task folder at once: repo in flight in one, repo/sub in the other	started
NFC caf + U+00E9 and NFD cafe + U+0301 folders; and repo and REPO	distinct inodes on this ext4. NFD/sub and REPO/sub start while the NFC folder and repo are running, which is correct, since they are different folders here

Synchronous (measured, by reading usage.ts:194-228): scopeEditTools, realpathSync, both rule checks, the cap checks and the cwds/earlier registration hold no await. The builder's same-tick case confirms it: repo/sub and the task folder are refused while repo starts.

Why the second meter matters (inferred from driver.ts):

• runBench checks only each task key's format (:174), not that the keys are unique. The manifest's uniqueness check (store.ts:59) isn't on this path.
• It makes the run folder and each task folder with mkdirSync(..., { recursive: true }) (:186, :221), which accepts a folder that already exists.
• It makes a new meter per task entry (:227).

So a repeated key in o.tasks, or a second runBench with the same run id after a crash, starts runs in a task folder whose earlier runs the new meter has never seen. A process left by those earlier runs can then swap a folder of a new run between its check and its spawn. On Linux the engine spawns claude detached (src/os/index.ts:576, src/runner.ts:709), so after a driver crash an earlier run may even still be going (guess).

Reachable today? guess: no. There is no CLI entry point or task selection yet. But o.tasks and runId are runBench's public options.

Legitimate patterns (measured, probe-r6.mjs §5):

Pattern	Result
Sequential runs in one folder (×3)	started
Parallel sibling worktrees (wt/wl-2 while wt/wl-1 runs)	started
A second run in a folder while a run is going there	refused
A run at repo, then one at repo/.claude/worktrees/wl-1	refused (inside an earlier run)
A run at the task folder, then any run below it	refused
A run at wt, then wt/wl-2	refused

This is a usability limit, not a correctness gap (inferred):

• It fails closed (principle 1).
• It does constrain RunTask:
  • the engine's default worktree root is .claude/worktrees inside the clone (src/config/schema.ts:610);
  • some engine roles run at the repo root, e.g. coordinator/corrections.ts:125.
• So RunTask must keep its worktrees outside the clone, and never run an agent in a folder that holds a later run's folder.
• The README says only that "runs in disjoint folders, and runs one after another in the same folder, are fine".

2. The "can never do" claim

• Exact at the check (measured): "at the meter's check … outside its task's folder, reached through a link, or missing".
• Exact within one meter (measured): "No run starts nested with a running run's folder, or strictly inside an earlier run's".
• "Between the check and the spawn, no other run of the task can swap a folder of it" doesn't hold across meters. Neither does Compromised component's "or after it if a process of it lingers, it can't swap a folder of another of the task's runs". In both cases (a repeated key, or a reused run id), the other run is a run of the same task, in the same folder (item 1; measured at the meter, inferred at the driver).
• The stated assumption ("nothing else writes inside the task folder while a run starts", inferred) leaves out a second one: nesting is compared as path text (within, usage.ts:54).
  • That is exact where one folder has one spelling, as on this ext4 (measured).
  • On a case-insensitive or normalisation-insensitive filesystem, two spellings of one folder aren't seen as nested. Examples are ext4 with casefold, vfat/exFAT and WSL's /mnt/c. inferred, not measured here: making one needs root or a Windows path.
  • realpathSync keeps the given spelling on Linux, so the realpath rule doesn't catch it (inferred).
  • Only RunTask picks the spellings, so whether this is reachable is a guess (no).

3. plain and RULE_UNSAFE

plain (measured, fixtures):

• Not plain (an extra call): description false or {}; command a number, a String object, missing, an array, null input or a string input.
• Plain: a string command with no description, or with a string one (empty included).

RULE_UNSAFE (measured):

• The flags are u.

• The builder's full BMP range list is reproduced exactly.

• Refused:

  • the lone high and low surrogates, and a reversed pair;
  • NEL, U+202E, U+200B and U+2060;
  • a tag character (U+E0001);
  • U+00AD and the no-break space.

• Allowed: Latin with diacritics and ß, Cyrillic, Greek, CJK, Devanagari with a virama, Arabic, Hebrew, Thai, NFD combining marks, an astral letter (U+1D518), an emoji, an emoji with VS16, and U+FFFD.

• On disk: scopeEditTools accepts a bench root with é, and a real task folder named with Latin, Cyrillic and CJK starts a run.

• Refused though legitimate:

  • Persian or Indic names with ZWNJ (U+200C);
  • emoji ZWJ sequences;
  • a soft hyphen.

  Each of these is \p{Cf}. They fail closed, so this is a usability cost only, and a rare one (inferred).

4. The Adversarial pass section

It is present, and its claims hold. I re-ran three of them (measured):

• the same-tick nested runs: refused, and the first started;
• the RULE_UNSAFE ranges: an identical list;
• the odd plain inputs: identical.

"No existing assertion changed" holds too: the only removed line in test/bench.test.ts is the node:fs import, which gains renameSync. The section doesn't try a second meter on one task folder. That is a gap in coverage, not a false claim.

5. The usual checks

• Fail-without (measured). Method: a scratch worktree at 943244a, plus git checkout 45b91baf -- test/bench.test.ts; no guard edited. # pass 52, # fail 5, as the builder reported:
  • round 6: no run starts in a folder nested…: actual '/repo/sub: succeeded';
  • …through any link…: Missing expected rejection: …/repo/up;
  • …string command…: 'all refused' vs expected 'model declined';
  • …C1 control and format…: Missing expected exception: "\x80";
  • …strictly inside an ended run's…: Missing expected rejection.
• Head (measured): # pass 57, # fail 0, with nothing left in the temporary home, state or held-out dirs.
• Merge (measured):
  • git merge-tree --write-tree 45b91baf 682c8ce5 is clean and gives b6fc8803. Main's only change since the PR's merge is one line in test/change-files-renames.test.ts.
  • On that tree, bench tests are 57/57 and change-files-renames is 35/35.
  • npm run typecheck exits 0, and check:denylist prints denylist: 1295 files clean.
• Rule 10 (measured, diff): no existing assertion, test or guard is changed or removed. Five tests are added.
• DCO and attribution (measured): check-dco.mjs 682c8ce5 45b91baf reports DCO: all commits signed off. The only trailers in the 13 commits are Signed-off-by. The body and commits have no attribution line.
• Protection (measured): every path is under scripts/bench/** or test/. The PR has [PROTECTED: owner merges], **Risk class:** protected, the needs-owner label, and real "What this can never do" and "Compromised component" sections. package.json is unchanged.
• No regression for existing installs (inferred): nothing in src/ or any installed path changes.
• CI (measured, gh pr checks 759; run 39210447158 is a pull_request run on 45b91ba, created 06:31Z, after 682c8ce at 06:27Z): every check passes. That covers the 8 shards, both full test jobs, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

Required changes

1. Make the overlap rules hold for the task folder, not just for one meter object.
   • Preferred ("make the wrong thing impossible"): runBench refuses, before any agent run, a task folder that already exists. For example, mkdirSync(workDir, { mode: 0o700 }) without recursive, with EEXIST refused. This one change covers both a repeated task key and a run id used again after a crash.
   • Test:
     • two entries with one key: the second never reaches the runner;
     • a task folder made beforehand: refused before any agent run.
   • Or: narrow "What this can never do" and Compromised component to "within one runBench call, for task keys that are unique and a run folder it made". Then make the existing-folder refusal a blocker for the CLI-entry PR.
2. State the second assumption, or remove it. Nesting is compared as path text, so the guarantee holds where one folder has one spelling (a case- and normalisation-sensitive filesystem, as on Linux ext4).
   • Either add that to the inferred assumption in the claim and the README;
   • or compare by device and inode: record each in-flight and earlier folder's (dev, ino) and check the new folder's ancestors against them.

Suggestions (not blocking)

• A. Raw format characters in the test source. test/bench.test.ts:1605 holds raw U+200B, U+200D, U+202E, U+2066, U+00AD and U+FEFF inside string literals (measured, grep -P). A raw U+202E can make the line display out of order. Write them as \u escapes, as 2fcaa2a did for two other tests.
• B. README, for RunTask. Say that a run in a folder that holds a later run's folder refuses that later run, so worktrees go outside the clone (§1, legitimate patterns).
• C. README. It could say that names with a ZWNJ, a ZWJ or a soft hyphen are refused.

For the owner

1. Required change 1 has the same shape as round 5's item 1.
   • It is a one-line driver fix with a test, or a narrowed claim plus a blocker for the CLI-entry PR.
   • Reachable today? guess: no.
2. One constraint on RunTask (not a loosening; it fails closed): the earlier-run rule forbids a run in a folder below an earlier run's folder.
   • So RunTask can't run an agent at the clone root and then in worktrees under the engine's default .claude/worktrees.
   • It must put worktrees outside the clone. No decision is needed unless you want that pattern.

[jayden1711]

Round 7: what changed

The new head is 9c666ba, one signed-off commit. This round was decided by the decision guide. The body is updated, including a round-7 "Adversarial pass" section.

Fail-without (measured):

• Method: this head's tests on 45b91ba, in a scratch worktree, since removed.
• Results: # pass 58, # fail 2 on 45b91ba; # pass 60, # fail 0 on the head.
• Setup: temp HOME, state and held-out dirs; refusing stand-ins. No model, Docker or download was used, and nothing touched the CLI binary or real state.

#	Fix	Test	Fails without (measured)
1	One meter per task folder. runBench refuses a repeated task key before any check. It makes the run folder, the sandbox check's folder and each task folder fresh (mkdir without recursive, EEXIST refused), so a reused run id, or a run folder made beforehand, is refused before any model call.	one task folder never gets two meters…: the reviewer's two-meter cases, as a repeated key, a second runBench with the same run id, and a run folder made beforehand	Missing expected rejection.
2	Nesting is compared by device and inode as well as by path. Each run's folder and its parents up to the task folder are recorded.	nesting is compared by device and inode too…: a folder renamed while its run is going; then its new name, and a folder inside it	Missing expected rejection: …/t01/renamed
3	The round-6 character test writes its characters as code points, and a new test refuses any raw \p{Cf}, \p{Cs} or control character (besides tab and newline) in scripts/bench/* and test/bench.test.ts.	no raw invisible, bidi or control character…	'test/bench.test.ts:1605', when run with 45b91ba's test source on disk
4	README: the RunTask constraint (no run below an earlier run's folder, so worktrees go outside the clone), and that names with ZWNJ, ZWJ or a soft hyphen are refused.	docs	n/a

Notes on the table:

• Fix 2, case-insensitive filesystems (inferred): making one needs root here, so the case-insensitive spelling isn't measured. A rename gives a folder the same kind of second spelling (same inode, new path), and that is what the test uses.
• Claims: "What this can never do" and Compromised component now say the overlap rules hold per task folder, compared by inode. The round-6 wording ("the task's runs") is corrected: it held only per meter.

Gates (measured):

• npm run typecheck exits 0;
• check:denylist: denylist: 1295 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.
• Main: 682c8ce merges cleanly (git merge-tree), so I didn't merge it again.

CI: on 9c666ba (measured, gh pr checks 759 and gh run list: ci, ci slow, gitleaks and new-deps succeeded), every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 7)

I reviewed round 7 (45b91ba..9c666ba, one signed-off commit).

• Setup: every run used a temporary HOME and temporary state, held-out and TMPDIR folders, with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state. Nothing was left in the temporary home, state or held-out folders (measured).
• Probes (session scratchpad, work/eval-759-r7/):
  • probe-r7.mjs, my own: probe-r6's two-meter cases through runBench, three racing processes, linked roots, folders made mid-run, renames;
  • scan-fixtures.mjs: the raw-character test's own regex, read from the compiled test, fed planted characters (rule 8);
  • the builder's adversarial7.mjs.

Summary. Both of round 6's required changes are in, and they work (measured). A task folder now has exactly one meter: the driver makes it fresh, and a repeated key, a reused run id, a run folder made beforehand, a linked run folder and a racing second process are all refused before any agent run. The inode comparison catches a second spelling of a running or earlier folder.

What remains is one false claim about the new raw-character test, and two missing labels in the README. Neither is a safety gap in the driver.

1. One meter per task folder

Code (measured, by reading):

• driver.ts:186-190 refuses a repeated key before the srt probe, Docker or any write.
• freshDir (:162-169) is mkdirSync(p, { mode: 0o700 }) without recursive, and EEXIST is refused. It covers the run folder (:205), sandbox-check (:210) and each task folder (:241). Only the runs folder's parent chain is still made recursive (:204).
• mkdir doesn't follow a final link, so a run folder that is a link gets EEXIST.
• TASK_KEY is t\d{2,4} and RUN_ID allows mixed case. On a case-insensitive filesystem, R1 after r1 hits EEXIST too (inferred).

probe-r6's two-meter cases, now through runBench (measured; old = 45b91ba's code, new = 9c666ba):

Case	45b91ba	9c666ba
Restarted driver: runBench(r1) again after one finished	ran, 3 more agent runs	refused ("run folder … already exists"), 0 agent runs
A second runBench(r1) while the first has a run in flight in t01	ran	refused; the first finishes
Keys t01,t02,t01	ran (4 agent runs, 19 docker calls)	refused before any docker call; no run folder made
Keys t01 and t001 (two folders, control)	ran	ran

My own cases (measured, 9c666ba):

Case	Result
Three runBench processes, one run id, released together (5 rounds)	exactly 1 ran each round; the other 2 got "run folder … already exists", 0 agent runs. On 45b91ba, 2 of 3 ran in rounds 2 and 4
Bench root is a link	refused at the sandbox check's meter ("a task folder through a link"); 0 agent runs; no sentinel left
runs is a link	refused the same way; 0 agent runs
The run folder is a link to an earlier run that holds t01	refused (EEXIST); 0 agent runs. On 45b91ba this was refused only by the meter's link check
The run folder made during the preflight (after a caller decided the id was free)	refused; 0 agent runs. 45b91ba: ran
t01 made during the sandbox check	refused at t01; 0 worker runs, 0 task containers. 45b91ba: ran
t02 made during t01 (builder's case)	runBench throws at t02, after recording t01
The task folder swapped for a link out after the driver made it	t01 is error, "not started: the task's folder …", 0 worker runs

Re-run, the builder's adversarial7.mjs (measured): every result in the body is reproduced: same-tick runBench, a link or dangling link, t02 made during t01, inode reuse, and renamed out.

Not blocking (inferred, from container.ts:314-324): the docker probe (driver.ts:199) runs before the run folder is made.

• So a reused run id is refused after a probe container, not before any container. The body says "before any model call", which holds.
• In a race, the losing process's probe ends with docker rm --force bench-<id>-probe. That can remove the winner's probe container mid-read-back. The winner then fails closed.
• Moving freshDir(dir) above the probe would refuse a reused id before any container. Suggestion A.

2. Nesting by device and inode

Code (measured, by reading usage.ts:64-81,236-244):

• folderOf stats the folder and every parent up to the task folder.
• The in-flight rule matches either way by chain, and the earlier-run rule by chain, minus the same id.
• It runs after the realpath and no-link checks, with no await before the registration.

Builder's rename case, by hand (measured):

• repo running, renamed to renamed: runs at renamed, renamed/sub and the task folder are refused.
• After it ends: renamed/sub is refused (earlier run), and renamed starts (the same folder again).
• On 45b91ba, renamed started while its run was going.

Inode reuse (measured, builder's case 4 re-run): after a was removed, b/c was refused.

• Only the inode rule can refuse b/c, since it isn't inside a by path. So b reused a's inode here.
• That fails closed. b itself starts as "the same folder".
• The reasoning holds (inferred): an inode can't be freed while a process holds the folder, as its cwd, an open fd or a sandbox bind. So reuse means nothing of the old run is in it.

Renamed out (measured):

• repo/sub ran, then was moved to moved. moved starts (the same folder), and moved/deep is refused (inside an earlier run).
• repo starts. Correct (inferred): a process left in sub can write only inside it, not repo or repo's parent.
• A new real repo/sub/x is refused by the path rule, which fails closed.

One edge, inside the stated assumption (measured, not blocking): each chain is taken when the run registers.

• A running repo/sub was moved to other/sub. A run at other, which now holds it, started. A run at other/sub was refused.
• Doing that move needs write access to the task folder itself. No run of the task can have that (inferred): a run at the task folder can't overlap repo/sub, and after one, repo/sub is refused as inside an earlier run.
• So only a writer outside the task's runs can do it, which "nothing outside the task folder's own runs writes inside it while a run starts" already excludes.
• The body's "or after a rename while a run is going" holds for the folder and what's under it, not for its new parents. Suggestion B.

The case-insensitive claim (inferred; the reasoning is sound):

• A second spelling reaches the same dentry and inode. Node's realpathSync keeps the given spelling (it resolves only links), so the no-link check passes, and the inode comparison is what matches.
• A task-folder component spelled differently fails within(realCwd, realWork), which fails closed.
• The ancestors' ids are in chain, so a differently spelled parent is caught in both directions.
• The one premise the label should carry: the filesystem must report a stable, unique st_dev:st_ino per folder. That holds on ext4 with casefold, and on vfat and exFAT while mounted (inode cache keyed by position). Some FUSE or network filesystems don't promise it, for example sshfs without use_ino (inferred).
• The bench root is under the bench user's home (ext4 here), so this is a guess: not reachable.

3. The raw-character scan (rule 8: its own regex, read from dist/test/bench.test.js, fed fixtures)

• The regex is /[\p{Cf}\p{Cs}]|(?![\t\n\r])\p{Cc}/u, applied per \n-split line over the 16 files: scripts/bench/*.{ts,mjs,md} and test/bench.test.ts.
• Fail-without (measured): fed 45b91ba's test/bench.test.ts (from git), it reports line 1605. Fed 9c666ba's, it reports nothing.
• Caught (measured): U+202E, U+2066, U+200B, a BOM at file start, U+00AD, a tag character, NUL, ESC, DEL and U+0085.
• Not caught (measured):
  • CR in the middle of a line;
  • U+2028 and U+2029;
  • variation selectors (U+FE0F, U+E0100);
  • the Hangul fillers (U+3164, U+115F);
  • U+2800 and U+034F.
• Two of these hide code (measured): with a fixture module // harmless comment<U+2028>console.log(...) and // harmless comment<CR>console.log(...), Node printed both: "ran after U+2028 | ran after CR". Each ends a // comment in JS and TS, while many viewers show one line.
• So the claims are false as written (principle 5):
  • the body (Adversarial pass, principle 4) and the comment say the scan refuses "any \p{Cf}, \p{Cs} or control character other than tab and newline". The code also allows CR, anywhere in a line;
  • the test's title says "no raw invisible, bidi or control character". CR is a control character, and U+2028, the variation selectors and the fillers are invisible.
• The fix is cheap (measured): a wider scan finds nothing in today's 16 files. It adds \p{Zl}\p{Zp}, \r(?!$), U+FE00-FE0F, U+E0100-E01EF, U+115F, U+1160, U+3164, U+FFA0 and U+2800.
• A lone surrogate can't reach the scan: invalid UTF-8 decodes to U+FFFD, so \p{Cs} never matches file text (measured, CESU-8 bytes). It is harmless.

4. Claims and README against the code

• "What this can never do" and Compromised component (per task folder, by inode, the inferred assumption) match the code and the probes above (measured). The round-6 correction is stated.
• The remaining assumption is stated and labelled inferred in the body (lines 51 and 314).
  • The README states it as fact: "In the bench, that is another of the task's runs" (README.md:45), with no label.
  • README.md:50 claims the case- and normalisation-insensitive case without the "inferred" label the body gives it.
• The README's RunTask section (worktrees outside the clone, concurrency only in disjoint folders) matches probe-r6 §5 (measured in round 6; the code is unchanged there).
• The ZWNJ, ZWJ and soft-hyphen note is accurate (RULE_UNSAFE, measured in round 6).

5. The usual checks

• Fail-without (measured). Method: a scratch worktree at 45b91ba, plus git checkout 9c666ba0 -- test/bench.test.ts; no guard edited. # pass 58, # fail 2, as the builder reported:
  • round 7: one task folder never gets two meters…: Missing expected rejection.
  • round 7: nesting is compared by device and inode too…: Missing expected rejection: …/runs/r1/t01/renamed
  • The scan: section 3 (line 1605).
  • The two-meter test fails at its first sub-case. The probe in section 1 shows the other two (a reused id, a pre-made folder) also ran on 45b91ba.
• Head (measured): # pass 60, # fail 0. npm run typecheck exits 0. check:denylist prints denylist: 1295 files clean.
• Merge with current main (measured):
  • main is now c339175 (Tests: fix the budget-health second-boundary flake; weekly optional shifted-clock run; lint on clock-reading helpers (#687) [PROTECTED: owner merges] #774, clock-shift), not 682c8ce.
  • git merge-tree --write-tree c339175a 9c666ba0 is clean and gives e1cd07c0. I rebuilt that exact tree (same tree id) and ran bench, no-clock-helpers, no-wall-clock-durations, clock-shift and change-files-renames: # pass 104, # fail 0.
  • Typecheck exits 0 on it.
• Rule 10 (measured, git diff 45b91baf 9c666ba0 -- test/bench.test.ts):
  • two lines are removed: the helpers import (it gains repoRoot) and the round-6 character list.
  • The list's raw characters (U+200B, U+200D, U+202E, U+2066 by od; U+00AD and U+FEFF from round 6's grep) are the same code points, in the same order.
  • No assertion is weakened, and no guard is touched.
• DCO and attribution (measured): check:dco -- origin/main 9c666ba0 prints DCO: all commits signed off. The only trailers in the 14 commits are Signed-off-by, and there is no attribution line in the commits, body or comment.
• Protection (measured): the 10 files are all under scripts/bench/** or test/. The PR has [PROTECTED: owner merges], **Risk class:** protected, the needs-owner label, and real "What this can never do" and "Compromised component" sections.
• No regression for existing installs (inferred): nothing in src/ or any installed path changes.
• Adversarial pass section (measured): present for round 7, and every reported result is reproduced. It misses the scan's gaps (section 3).
• CI (measured): gh pr checks 759; run 39213615457 is a pull_request run on 9c666ba. Every check passes: the 8 shards, both full test jobs, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

Required changes

1. The raw-character test claims more than it checks (principle 5). Either:

   • preferred (principle 4): widen it to what its title says. Add \p{Zl}, \p{Zp} and a CR anywhere but at a line end (\r(?!$)), and preferably the variation selectors and the Hangul fillers (section 3). It costs nothing today (measured: no hits);
   • or: narrow the test title, the body and the comment to exactly what it checks (\p{Cf}, \p{Cs}, and controls other than tab, LF and CR).

   Either way, correct the body's "other than tab and newline", since CR is allowed too.

2. README labels (principle 5):

   • README.md:50: mark the case- and normalisation-insensitive claim inferred. Name its premise: the filesystem gives each folder a stable, unique device and inode.
   • README.md:45: label "that is another of the task's runs" as the inferred assumption, as the body does ("nothing outside the task folder's own runs writes inside it while a run starts").

Suggestions (not blocking)

• A. Make the run folder fresh before docker.probe (driver.ts:199 vs :205). Then a reused or racing run id is refused before any container, and a losing racer can't rm --force the winner's probe container.
• B. Say that each run's parent chain is recorded when it registers. A running folder moved under a new parent isn't seen in its new parents. Only a writer outside the task's runs can do that move, so it is inside the stated assumption (section 2, measured).
• C. Put the scan's regex in a named constant with a few fixture assertions (a planted U+202E, U+2028, CR), so its fail-without doesn't need an old file on disk.

For the owner

No new trade-off. Both required changes are wording or a wider lint, and none loosens anything.

[jayden1711]

Round 8: what changed

The new head is 6ffe56d:

• a signed-off merge of origin/main (c339175);
• one signed-off commit with the round-8 fixes.

The body is updated.

#	Change	Test	Fails without (measured)
1	The raw-character scan is widened to what its title says. Its regex is now a named constant, RAW_UNSAFE. It adds \p{Zl}, \p{Zp}, a CR anywhere but at a line's end, the variation selectors, the Hangul fillers, U+2800 and U+034F, on top of \p{Cf}, \p{Cs} and every control character but tab. The title now says exactly that.	A new test feeds RAW_UNSAFE the round-7 check's fixtures, built from code points, and checks that tab, a line-end CR and visible text pass. The scan over the 16 files still finds nothing.	The fixtures against 9c666ba's regex: CR mid-line, U+2028, U+2029, U+FE0F, U+E0100, U+3164, U+115F, U+2800 and U+034F all pass it, and all are caught by RAW_UNSAFE
2	README: labels inferred on the remaining assumption (nothing outside the task folder's own runs writes inside it while a run starts) and on the case-insensitive claim, with its premise (a stable, unique device and inode per folder; sshfs without use_ino named as a counterexample).	docs	n/a
3 (A)	The run folder is made fresh before the probe container, so a reused or racing run id is refused before any container, and a losing racer can't remove the winner's probe container.	The reused-run-id case now also checks that no container is made, started or removed.	On 9c666ba, in a scratch worktree (since removed): 15 !== 12 (# pass 60, # fail 1)
3 (B)	README: each run's parent chain is recorded when it registers; a running folder moved under a new parent isn't seen in its new parents (inside the stated assumption).	docs	n/a

Correction (round 7 check): the body's round-7 Adversarial pass said the scan refused any \p{Cf}, \p{Cs} or control character "other than tab and newline". That was wrong: CR was allowed anywhere. It is corrected in place.

Gates (measured):

• bench tests # pass 61, # fail 0, with temp HOME, state and held-out dirs and refusing stand-ins;
• main's new no-clock-helpers and no-wall-clock-durations tests pass;
• npm run typecheck exits 0;
• check:denylist: denylist: 1299 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.

CI (measured, on 6ffe56d; gh pr checks 759 and gh run list):

• ci, ci slow, gitleaks and new-deps succeeded. A second new-deps run on the same commit shows "cancelled". The new-deps check itself passes.
• Every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans.
• The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 8)

I reviewed round 8: 9c666ba..6ffe56d, which is a signed-off merge of origin/main (c6994d1) plus one signed-off fix commit.

• Setup: every run used a temporary HOME and temporary state, held-out and TMPDIR folders, with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state. Nothing was left in the temporary home, state or held-out folders (measured).
• Probes are in the session scratchpad under work/eval-759-r8/:
  • scan-r8.mjs: RAW_UNSAFE, read from the compiled test (rule 8), fed fixtures, plus an exhaustive sweep;
  • superset.mjs: the old regex against the new, on every code point;
  • ident-probe.mjs;
  • race.mjs and race-child.mjs;
  • round 7's probe-r7.mjs, re-run on this head.

Summary. Suggestion A is in, and it works (measured). The README labels are in. The scan is wider, strictly tighter than before, and refuses nothing legitimate (measured).

Three things remain:

• the scan still lets through four Unicode variation selectors, although the body says it now catches "the variation selectors";
• the round-8 hand-off has no Adversarial pass section;
• one README sentence (which came from the round-7 check's own wording) overstates vfat and exFAT.

None of these is a safety gap in the driver.

Check	Result
1. RAW_UNSAFE	partly met. Every named fixture is caught and nothing legitimate is refused. Six assigned default-ignorable code points still pass, four of them variation selectors. See §1
2. README labels	met, with one wording error. See §2
3. Suggestion A	met (measured). See §3
4. No regression	met (measured). See §4
5. Usual checks	met, except that the Adversarial pass section is missing. See §5

1. RAW_UNSAFE (rule 8: the constant read from dist/test/bench.test.js:1599, fed fixtures)

Must be caught (measured; 49 fixtures, 0 wrong):

• every round-7 fixture: U+202E, U+2066, U+200B, a BOM at file start, U+00AD, U+E0041, NUL, ESC, DEL and U+0085;
• CR mid-line, CR at file start, and CR CR LF;
• U+2028 and U+2029;
• U+FE00, U+FE0F, U+E0100 and U+E01EF;
• U+115F, U+1160, U+3164 and U+FFA0;
• U+2800 and U+034F;
• my own: U+180E, U+200C to U+200F, U+2060 to U+2064, U+206F, U+061C, a mid-line U+FEFF, the tags U+E0001, U+E0020 and U+E007F, U+FFF9, U+1D173, U+1BCA0, U+0600, VT, FF, U+009B, and a lone surrogate in a JS string.

On 9c666ba's regex, all 13 of the builder's 9 plus my 4 extra variants (U+FE00, U+E01EF, U+1160, U+FFA0) pass. CR at file start and CR CR LF pass too (measured).

Must pass (measured; 17 fixtures, all pass):

• tab, a CRLF line end, and a CRLF file;
• Latin accents, Greek, Cyrillic, CJK, Hangul syllables, Arabic, Hebrew and Devanagari;
• emoji;
• typographic punctuation and box drawing;
• a non-blank braille letter (U+2801);
• a decomposed é (U+0301).

The 16 files scan clean (measured). The scan finds []. A wider scan (any \p{Default_Ignorable_Code_Point}, or any non-ASCII \p{Zs}) over the same 16 files also finds [].

Rule 10 (measured, superset.mjs): over every code point, in three line positions, no line that 9c666ba's regex refuses is allowed by RAW_UNSAFE. 796 lines go the other way. So the constant is strictly tighter, and nothing is weakened.

Still let through (measured, the exhaustive sweep):

• Assigned: U+17B4, U+17B5, U+180B, U+180C, U+180D and U+180F.
  • All six are Default_Ignorable_Code_Point and ID_Continue.
  • U+180B to U+180D and U+180F are Variation_Selector, by Node 22.23.2's own Unicode 17 data (/\p{Variation_Selector}/u).
• Unassigned but default-ignorable: U+2065, U+FFF0 to U+FFF8, U+E0000, U+E0002 to U+E001F, U+E0080 to U+E00FF, and U+E01F0 to U+E0FFF. Renderers show these as nothing.
• These hide code (measured, ident-probe.mjs): for each of the six, let a = 1; let a<c> = 2; return [a, a<c>] returns 1,2. So they make a second identifier that looks the same as a, exactly as U+3164 does, and RAW_UNSAFE refuses U+3164.
• So one claim is false (principle 5):
  • the body's round-7 Adversarial pass correction says the scan "let through … the variation selectors and the Hangul fillers. Since round 8 it catches those too";
  • the commit message says the same ("the variation selectors … catches them all too").
  • The four Mongolian free variation selectors still pass.
  • The code comment and the body's round-8 list give the exact ranges (U+FE00-FE0F, U+E0100-E01EF). Read with those ranges, they are accurate.
• The class, not the instance (DECISION GUIDE):
  • The list was copied from my round-7 fixtures, so the round-7 check missed these too.
  • The class the comment itself names ("render as nothing") is Unicode's Default_Ignorable_Code_Point. JS regexes support it directly.
  • Adding \p{Default_Ignorable_Code_Point} covers every Cf-type invisible, all the variation selectors, all four Hangul fillers, U+034F, U+17B4/5 and the unassigned ranges. It costs nothing today (measured: no hits in the 16 files).
  • U+2800, \p{Zl}, \p{Zp}, the CR rule and the control characters stay as they are, since none of them is default-ignorable.
• Out of scope, not asked for: NBSP and the other \p{Zs} spaces (U+2009, U+202F, U+205F, U+3000), private use, and U+FFFC all pass. They show as a space or a glyph. The new test's title says "separator", which reads as all of \p{Z}, but an ASCII space is \p{Zs} too. "line or paragraph separator" would be exact. This is a suggestion, not a required change.

2. README labels (scripts/bench/README.md:45,50-53)

• :45: "That this is the only such writer is inferred: it assumes nothing outside the task folder's own runs writes inside it while a run starts". Its premises are given: the driver writes only its records, and the container has no mount. That matches the code (measured, by reading).
  • The task folder is made by freshDir (driver.ts:243).
  • Patches go to runs/<run>/patches (grade.ts:23), not into the task folder.
  • The export's bare repo goes in tmpdir() (grade.ts:46).
• :51: "A rename … (measured, test)". That is right: round 7's test, and my §4 probe.
• :52, the case-insensitive claim: inferred, with the premise stated (a stable, unique device and inode per folder). That is right.
  • sshfs without use_ino is an accurate counterexample (inferred). Without it, libfuse's high-level layer numbers nodes by path. So two spellings of one folder on a case-insensitive server get two inode numbers, and a forgotten node can come back with a new one.
  • "and so do vfat and exFAT while mounted" is not accurate (inferred, from the kernel's fat_build_inode and exfat_build_inode). Each gives an inode read into memory a fresh iunique() number. So a folder's number is stable only while its inode stays cached, and an ended run's folder, with nothing holding it, can get a new number after eviction.
  • This sentence is my side's mistake: the round-7 check wrote it ("on vfat and exFAT while mounted (inode cache keyed by position)"), and the builder followed it. I correct it here.
  • It matters only on such a filesystem. The README's setup puts the bench root under the bench user's home.
  • "The bench root under the bench user's home is on ext4 here" describes one machine, with no label.
• :53, chain recorded at registration: accurate (measured, probe-r7.mjs §E on this head). A running repo/sub moved to other/sub, then a run at other started, and a run at other/sub was refused.

3. Suggestion A: the run folder is made before the probe container (driver.ts:198-207)

• Code (measured, by reading):
  • docker.preflight() (docker info, no container) runs first, then freshDir(dir), then docker.probe.
  • If docker.probe throws, the run folder is already there, so the id is burned. That fails closed, and the README already says to pick a new id.
• Reused id, one after another (measured, race.mjs):
  • on this head, the second runBench(r1) made one docker call (info), 0 container operations and 0 agent runs;
  • on 9c666ba it made create, start and rm --force of bench-r1-probe (3 container operations).
• 3 processes, one run id, released together, 5 rounds (measured):

Round	6ffe56d	9c666ba
1-5	exactly 1 ran each round (z, y, x, x, z). Each loser was refused with "run folder … already exists", with 0 agent runs and 0 container operations	1 ran each round, but each loser made create, start and rm --force bench-r1-probe, the same name as the winner's probe

• So a loser can no longer remove the winner's probe container (measured: no loser makes any create, run, start or rm).

4. Nothing earlier regressed (measured)

• Bench tests on the head: # tests 61, # pass 61, # fail 0. They cover:
  • the meter (one per task folder, nesting by inode, the overlap and earlier-run rules);
  • plain, RULE_UNSAFE and the layers tests.
• Round 7's probe-r7.mjs, re-run on 6ffe56d. Every result matches the round-7 check:
  • restarted driver, a second runBench with a run in flight, and a repeated key: refused, with 0 agent runs;
  • t01 and t001: ran;
  • the race: 1 of 3 each round;
  • linked bench root, linked runs folder, and a run folder that links to an earlier run: refused;
  • folders made mid-run: refused;
  • a task folder swapped for a link: error, with 0 worker runs;
  • nesting by inode: renamed, moved out, moved under a new parent (inside the assumption), and inode reuse.
• Driver diff since 9c666ba: one block moved (+8/-3 in driver.ts), plus README text. No other source file changes.

5. The usual checks

• Fail-without (measured). Method: a scratch worktree at 9c666ba, plus git checkout 6ffe56d0 -- test/bench.test.ts, then build and run with temp state; no guard edited.
  • Result: # pass 60, # fail 1.
  • The failing test is round 7: one task folder never gets two meters…: 15 !== 12 (expected 12, actual 15). That is the builder's result.
  • The scan's fail-without is the fixture table in §1: all nine of the builder's fixtures pass 9c666ba's regex, and all are caught by RAW_UNSAFE.
• Gates on the head (measured):
  • npm run typecheck exits 0;
  • check:denylist prints denylist: 1299 files clean;
  • check:dco -- origin/main 6ffe56d0 prints DCO: all commits signed off.
• Attribution (measured): the only trailers in the 16 commits are Signed-off-by. There is no attribution line in the body or in the round-8 comment.
• Rule 10 (measured): nothing is removed or weakened.
  • The old unsafe regex is replaced by a strict superset (§1).
  • The renamed scan test keeps its scan and assertion.
  • The reused-id case gains an assertion.
  • No exception, skip or allowlist entry is added.
• Protection (measured):
  • the 10 files are under scripts/bench/** or test/;
  • the title has [PROTECTED: owner merges], the body has **Risk class:** protected, and the PR has the needs-owner label;
  • "What this can never do" and "Compromised component" are present, and unchanged in substance this round.
• Rule A (measured): the tests and probes left nothing in the temp home, state or held-out folders, and the round-8 change adds no state access.
• No regression for existing installs (inferred): nothing in src/ or any installed path changes.
• Merge with origin/main (measured):
  • origin/main is c339175, an ancestor of 6ffe56d;
  • git merge-tree --write-tree origin/main 6ffe56d0 gives 74c335e0, the head's own tree;
  • git diff c339175a 6ffe56d0, outside scripts/bench and test/bench.test.ts, is empty.
  • The merge commit changes no bench file.
• CI (measured): run 39217283406 is a pull_request run with headSha 6ffe56d, and its conclusion is success.
  • Passing: all 8 shards, both full test jobs (Node 22 and 24), dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans.
  • Skipped, as usual: the real-systemd, bootstrap and sign-in-probe jobs.
• Adversarial pass (measured): missing for round 8.
  • The body has round 7's and round 6's sections only.
  • The round-8 comment has none.
  • The round-7 section points to "round 8 below", but round 8 is above it.
  • The evaluator brief makes a hand-off without this section a required change. A class-level self-attack (a sweep over \p{Default_Ignorable_Code_Point} or \p{Variation_Selector}) would have found §1.

Required changes

1. Fix the class in RAW_UNSAFE, and correct the claim (principles 4 and 5).
   • Add \p{Default_Ignorable_Code_Point} to the character class. The explicit variation-selector and Hangul-filler ranges and U+034F can then go, since it covers them, but keep U+2800, \p{Zl}, \p{Zp}, the CR rule and the control-character rule.
   • Add fixtures for U+17B4 and U+180B.
   • Correct the body's "the variation selectors … catches those too" and the matching commit-message wording in the next commit's message, unless the regex then makes it true.
   • It costs nothing today (measured: no hits in the 16 files).
2. Add an "Adversarial pass (round 8)" section to the body and to the hand-off comment, with each attack, its measured result, and what changed because of it. Also fix the "see round 8 below" pointer.
3. README :52, wording (principle 5):
   • drop "and so do vfat and exFAT while mounted", or say that they keep a folder's number only while its inode is in memory (inferred, as above);
   • drop "is on ext4 here", or label it with what it rests on. As written, it is an unlabelled claim about one machine.

Suggestions (not blocking)

• The new test's title says "separator". "line or paragraph separator" says exactly what it checks.

For the owner

There is no new trade-off. All three changes are wording, a wider lint, or a missing section, and none loosens anything.

[jayden1711]

Round 9: what changed

The new head is 002d7c9, one signed-off commit. origin/main (c339175) is already an ancestor. The body is updated.

#	Change	Test	Fails without (measured)
1	RAW_UNSAFE refuses the whole class \p{Default_Ignorable_Code_Point}, assigned or not, on top of the round-8 ranges, which stay. That covers the Mongolian variation selectors U+180B-180F and the Khmer U+17B4 and U+17B5.	The fixture test adds U+17B4, U+17B5, U+180B, U+180F, U+2065 and U+E0002.	Over every code point, 6ffe56d's regex lets U+17B4-17B5, U+180B-180D and U+180F through, plus 3,769 unassigned default-ignorable code points. RAW_UNSAFE refuses them all.
2	Claims corrected: "catches the variation selectors" (body; the round-8 commit message is corrected in this round's message, not rewritten), and the "see round 8 below" pointer.	docs	n/a
3	README: vfat and exFAT keep a folder's inode number only while its inode is in memory (inferred). The unlabelled "is on ext4 here" becomes advice to keep the bench root on a filesystem with stable inode numbers.	docs	n/a
4	The scan test's title says "line or paragraph separator".	n/a	n/a

Adversarial pass (round 9, covering round 8's changes too)

Round 8's hand-off had none. Every result here is measured unless labelled, with fixtures and stand-ins only.

• The scan against every code point (sweep9.mjs, with RAW_UNSAFE read from the compiled test):
  • Left through: no default-ignorable, format or variation-selector code point now.
  • Rule 10: nothing the old regex caught is let through.
  • Visible text passes: Latin, Greek, Cyrillic, CJK, Hangul, Arabic, Hebrew, Devanagari, emoji, U+2801, an em dash, box drawing and U+0301.
  • Still allowed by design (inferred: they show as a space or a glyph, and none is default-ignorable): the non-ASCII \p{Zs} spaces, private use and U+FFFC.
  • Round 8 missed the six assigned ones above. This sweep is the class-level attack that would have found them.
• Round 8's probe-container order: a reused run id makes no container call (the test fails with 15 !== 12 on 9c666ba). inferred: a failed probe now burns the run id, which fails closed.
• Not attacked: the case-insensitive claim needs a casefold filesystem, which needs root. It stays inferred.
• Principles 2, 3, 6, 7 and 9:
  • only the test file and the README change;
  • the regex only grows;
  • no assertion is removed;
  • no exception or skip is added.
• Principle 8: no refusals.

Gates (measured):

• bench tests # pass 61, # fail 0;
• npm run typecheck exits 0;
• check:denylist: denylist: 1299 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.

CI: on 002d7c9 (measured, gh pr checks 759 and gh run list: ci, ci slow, gitleaks and both new-deps runs succeeded), every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans. The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

[jayden1711]

Owner decision 2026-10-19: RunTask putting worktrees outside the clone is fine; the README constraint stands.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 9)

I reviewed round 9: 6ffe56d..002d7c9, one signed-off commit that changes only test/bench.test.ts (+13/-4) and scripts/bench/README.md (+1/-1) (measured, git diff --stat 6ffe56d0 002d7c9a).

• Setup: every test and probe ran with a temporary HOME and temporary state, held-out and TMPDIR folders, with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state. Nothing was left in the temporary home, state or held-out folders (measured).
• Probes are in the session scratchpad under work/eval-759-r9/:
  • sweep-r9.mjs: RAW_UNSAFE read from the compiled test (rule 8), swept over every code point (surrogates included, as lone code units) in three line positions, against 6ffe56d's regex;
  • fixtures-r9.mjs, ident-r9.mjs, cost-r9.mjs and vs180e.mjs;
  • round 8's race.mjs, re-run on this head.

Summary. The round-9 fix itself is right. RAW_UNSAFE now lets no code point of any required class through. It is a strict superset of 6ffe56d's regex, and it refuses nothing legitimate. The claims and README edits are true and labelled, and the Adversarial pass section is in both the body and the round comment (all measured).

One thing blocks: origin/main moved to 3d24fc3 (#804) after this head's CI ran, and the branch no longer merges. test/bench.test.ts conflicts.

Check	Result
1. RAW_UNSAFE	met (measured). See §1
2. README edits, corrected claims, Adversarial pass	met (measured). See §2
3. No regression	met (measured). See §3
4. Usual checks	not met: there is a merge conflict with origin/main. Everything else is met. See §4

1. RAW_UNSAFE (taken from dist/test/bench.test.js, not from the source)

Required classes, every code point (measured, sweep-r9.mjs, Node 22.23.2). Each code point was tested at the start, in the middle and at the end of a line. Code points that get through:

• \p{Default_Ignorable_Code_Point}: none;
• \p{Cf}: none;
• \p{Cc} except tab: none (CR is checked by position, below);
• \p{Cs}: none;
• \p{Zl}: none;
• \p{Zp}: none;
• \p{Variation_Selector}: none.

Superset of 6ffe56d (measured):

• No code point, in any position, that 6ffe56d's regex catches is let through.
• 3,775 code points are newly caught:
  • 6 assigned: U+17B4-17B5, U+180B-180D and U+180F, exactly the builder's list;
  • 3,769 unassigned, which matches the builder's count.

Fail-without (measured, fixtures-r9.mjs): the six new fixtures (U+17B4, U+17B5, U+180B, U+180F, U+2065 and U+E0002) all pass 6ffe56d's regex, and all are caught by the head's.

• Method: the test file defines the regex it tests, so copying the new test onto the old commit would bring the new regex with it. I fed the fixtures to each regex instead (rule 8).

Nothing legitimate is refused (measured):

• Refused outside the required classes: only U+2800, which is intended.

• Pass:

  • tab, and a CRLF line end;
  • a whole CRLF file split on LF (0 lines caught);
  • a lone CR at the end of the last line.

• Caught:

  • a CR mid-line;
  • a CR at line start;
  • CR CR at line end.

• Pass:

  • Latin, Greek, Cyrillic, CJK and kana, Hangul, Arabic, Hebrew with niqqud, Devanagari with a virama conjunct, Tamil, Thai, Georgian, Armenian, Ethiopic and Bengali;
  • emoji, including a flag and a skin tone;
  • combining marks (U+0301, U+0308, U+0303);
  • typographic punctuation, box drawing, braille U+2801 and math symbols.

• Refused, and correctly so under principle 1 (measured):

  • emoji ZWJ sequences, and emoji with VS16;
  • Persian text with ZWNJ;
  • Mongolian text with a free variation selector.

  These files are code and docs, so a \u{...} escape stays available. The README's rule-path section already says the same of ZWNJ and ZWJ names.

The builder's "allowed by design" items (measured, ident-r9.mjs): none of them can hide code, so none is a required change.

• Non-ASCII \p{Zs}: U+00A0, U+1680, U+2000-200A, U+202F, U+205F and U+3000.
  • None is ID_Start or ID_Continue.
  • let a<c> = 2 beside let a is a syntax error for each, so none makes a second identifier.
  • JS parses each as whitespace (let a<c>= 3 returns 3), so code does what it looks like.
  • Inside a string literal, one is a lookalike of a space. That is the same kind of homoglyph as Cyrillic а, which the scan deliberately allows.
• Private use (U+E000, U+F8FF, U+F0000, U+10FFFD) and U+FFFC:
  • none is an identifier character, and outside a string or comment each is a syntax error;
  • each renders as a glyph or a box (inferred).
• Recommendation (not required):
  • Refusing non-ASCII \p{Zs}, \p{Co} and U+FFFC would cost nothing today: a scan for them over the same 16 files finds [] (measured, cost-r9.mjs).
  • Principle 1 leans towards refusing what nobody needs in these files.
  • It is the builder's call. No code-hiding case exists, so it does not block.

2. README edits, corrected claims, Adversarial pass (measured, by reading the head and the body)

README :52:

• vfat and exFAT now "keep a folder's number only while its inode stays in memory", labelled inferred, with its basis. That matches the round-8 check's correction.
• The unlabelled "is on ext4 here" is now advice. "The README's setup puts it under the bench user's home" is true: :28 gives the default ~/bench.

Corrected claims:

• The body's correction of "catches the variation selectors" (round-7 section, Correction (round 8 check)) is accurate. My sweep confirms that 6ffe56d missed exactly U+180B-180D, U+180F, U+17B4 and U+17B5.
• The round-8 commit message is corrected in 002d7c9's message, without rewriting history. That is acceptable.
• The round-9 fail-without numbers in the body and the round comment match mine: 6 assigned and 3,769 unassigned, with none loosened.
• The "see round 8 below" pointer is gone. It now points to the notes "above" and to the round-9 Adversarial pass.

Adversarial pass:

• It is present in the body ("Adversarial pass (round 9, covering round 8's changes too)") and in the round comment (issuecomment-6262742361).
• It lists the attacks, their measured results and what was fixed. It also covers round 8's probe-container order.

Nit (not required): the body's round-9 bullet and the new code comment call U+180B-180F "the Mongolian variation selectors".

• U+180E is the Mongolian vowel separator: Cf, not Variation_Selector (measured, vs180e.mjs).
• It is caught either way. "U+180B-180D and U+180F" would be exact.

3. Nothing earlier regressed (measured)

• Bench tests on 002d7c9: # tests 61, # pass 61, # fail 0, and nothing was left in the temp home, state or held-out folders.
• Driver unchanged: driver.ts and every other source under scripts/bench except the README are byte-identical to 6ffe56d's (git diff --stat 6ffe56d0 002d7c9a).
• Round 8's run-id and race behaviour (race.mjs, re-run on this head):
  • reused run id, one after another: the second runBench is refused ("run folder … already exists") with docker calls ["info …"] only, 0 container operations and 0 agent runs;
  • 3 processes on one run id, 5 rounds: exactly 1 ran each round, and every loser was refused with 0 agent runs and containerOps=[]. "Rounds not exactly one winner with no loser container op: 0".

4. The usual checks

• Merge with origin/main: conflict (measured).
  • origin/main is now 3d24fc3 (git ls-remote origin refs/heads/main). It merged bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 at 08:11:54Z, after this head's ci run started (08:04:58Z).
  • git merge-tree --write-tree origin/main 002d7c9a exits 1: CONFLICT (content): Merge conflict in test/bench.test.ts.
  • The conflict: main inserts five tests (one argv, four stdin, for Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 and bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804) just before stdin: a run that ends a tar member of exactly 512 bytes…. This branch changed that test's opening line to async () =>.
  • The hidden part:
    • main's four new stdin tests call stdinSends(...) synchronously, since main has function stdinSends(...): boolean (test/bench.test.ts:650 on main);
    • this branch has async function stdinSends(...): Promise<boolean> (:1049).
    • So a textual resolution alone leaves assert.equal(<Promise>, false) in those tests. They would fail, loudly not silently (inferred).
  • main's scripts/bench/hidden.ts change adds or changes no export, and this branch imports only hiddenLeaks, stdinCarriesHidden, tarText and TaskRow from it (measured, by grep). So the source merge is clean, and only the test file conflicts.
  • Under rule 12, I did not build a merged tree.
• The head's CI (measured): every check on 002d7c9 passed.
  • Runs: ci 39220236832, ci slow 39220236432, gitleaks, and both new-deps runs (all pull_request, headSha 002d7c9).
  • Passing: all 8 shards, both full test jobs (Node 22 and 24), dco, denylist, scope, push-gate, agentshield, desktop and both scans.
  • Skipped, as usual: the real-systemd, bootstrap and sign-in-probe jobs.
  • Those runs tested the merge with c339175, not with today's main (inferred from the times).
• Gates on the head (measured):
  • npm run typecheck: exit 0;
  • check:denylist: denylist: 1299 files clean;
  • check:dco -- origin/main HEAD: DCO: all commits signed off.
• DCO and attribution (measured): all 17 commits since c339175 carry exactly one trailer each, Signed-off-by. There is no attribution line in the body or the round-9 comment (grep for co-authored, "generated with" and the vendor address: 0 hits).
• Rule 10 and principle 9 (measured):
  • the regex only grows (§1);
  • the scan test keeps its scan and assertion, and only its title changed;
  • no assertion, skip, exception or allowlist entry is removed or added.
• Protection (measured):
  • the 10 files are under scripts/bench/** or test/;
  • the title has [PROTECTED: owner merges], the body has **Risk class:** protected, and the PR has the needs-owner label;
  • "What this can never do" and "Compromised component" are present, and unchanged this round.
• Rule A (measured): nothing reached the temp home, state or held-out folders, and the round adds no state access.
• No regression for existing installs (inferred): nothing in src/ or any installed path changes.
• One stale claim (measured): the body's round-9 "Main: origin/main (c339175) is already an ancestor" was true when written, and is no longer the current main.

Required changes

1. Merge origin/main (3d24fc3 or later) into the branch, as a signed-off merge, and resolve test/bench.test.ts:

   • keep both sides: main's five Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797/bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 tests, and this branch's async 512-byte tar test;
   • make main's four stdin tests async, and await every stdinSends(...) call in them;
   • change no assertion of main's (rule 10).

   Then, with temp state and stand-ins:

   • re-run the bench tests and report the counts, which should be 61 plus main's new tests, all passing;
   • re-run typecheck, denylist and DCO;
   • let CI run on the new head.

   Update the body's "Main" line.

Suggestions (not blocking)

• Optionally also refuse non-ASCII \p{Zs}, \p{Co} and U+FFFC (principle 1). Today it costs nothing ([] over the 16 files), but none of them can hide code (§1).
• Say "U+180B-180D and U+180F" rather than "U+180B-180F" for the Mongolian variation selectors, in the body and the code comment.

For the owner

There is no new trade-off. The only blocker is a merge with a main that moved after this round's CI.

[jayden1711]

Round 10: main merged

The new head is 242f663:

• ab9bfc8: a signed-off merge of main (3d24fc3, which includes bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804);
• 242f663: one signed-off commit for the nits.

The body is updated.

The merge (measured):

• The conflict: test/bench.test.ts conflicted. Both sides are kept: main's five Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797/bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 tests and this branch's async 512-byte tar test.
• Main's four stdin tests are made async and await every stdinSends call, because syncTree is async on this branch. No assertion of main's changed: every assert line in main's file is in the branch's, up to that await. The only exceptions are five older lines this branch made assert.rejects in earlier rounds.
• scripts/bench/hidden.ts is byte-identical to main's: git diff origin/main -- scripts/bench/hidden.ts is empty. So bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804's addedLines, importLines and strict && loose are intact.
• Tests: the bench tests give # pass 66, # fail 0 on the merge commit alone and on the head, with temp HOME, state and held-out dirs and refusing stand-ins.
• Gates: npm run typecheck exits 0, check:denylist prints denylist: 1299 files clean, and check:dco -- origin/main HEAD prints DCO: all commits signed off.
• Main since then: it has moved to 178c44e (Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793). git merge-tree --write-tree origin/main HEAD is clean, and Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793 touches no bench file.

The nits (measured):

• Wording: the comment now reads "U+180B-180D and U+180F".
• The scan: RAW_UNSAFE also refuses every non-ASCII \p{Zs} space, private use and U+FFFC, with six new fixtures.
  • Over every code point, 002d7c9's regex let 137,485 of these through.
  • None it caught is let through now.
  • Visible text still passes, and the 16 files scan clean.

CI (measured, on 242f663; gh pr checks 759 and gh run list):

• ci, ci slow, gitleaks and new-deps succeeded. Two more new-deps runs on the same commit show "cancelled". inferred: my two body edits re-triggered it, and each new run cancelled the one before. The new-deps check itself passes.
• Every check passes: the 8 test shards and both full test jobs on Node 22 and 24, dco, denylist, scope, push-gate, new-deps, agentshield, desktop and both scans.
• The real-systemd, bootstrap and sign-in-probe jobs are skipped, as usual.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 10)

I reviewed round 10: 002d7c9..242f663. That is the merge ab9bfc8 (parents 002d7c9 and 3d24fc3) and one commit, 242f663, that changes only test/bench.test.ts (+14/-5) (measured, git log --format=%H%x20%P, git diff --stat ab9bfc88 242f663b).

• Setup: every test and probe ran in a detached worktree with a temporary HOME and temporary state, held-out and TMPDIR folders, and with refusing claude, docker and srt stand-ins first on PATH. No model, Docker or download was used, and nothing touched the CLI binary or real state. Nothing was left in the temporary home, state or held-out folders (measured).
• Rule 12: I ran no merge, commit, rebase or reset. The merge was judged by reading it and with git merge-tree only.
• Probes: they are in the session scratchpad under work/eval-759-r10/: asserts.mjs, sweep-r10.mjs and runtest.sh.

Summary. The code is right. The merge is a faithful resolution, and none of main's assertions changed. #804's hidden.ts is intact, and main's five tests still catch what they were written for on this branch. The RAW_UNSAFE additions are a strict superset of the old regex, refuse nothing legitimate, and apply only to the repo's own bench source files. Tests, gates and CI are green, and the head merges cleanly with today's main (all measured).

One thing blocks, and it is text only: round 10's hand-off has no Adversarial pass section. It is missing from both the body and issuecomment-6263149184. Under EVALUATOR.md and the DECISION GUIDE, a hand-off without one is itself a required change. The same finding was made in round 8.

Check	Result
1. The merge (ab9bfc8)	met (measured). See §1
2. The new RAW_UNSAFE additions	met (measured). See §2
3. Tests, gates, DCO, attribution, protection, CI, merge with main	met (measured). See §3
4. Hand-off: Adversarial pass section	not met (measured, by reading the body and the round comment). See §4

1. The merge (ab9bfc8)

The resolution is faithful (measured):

• git merge-tree --write-tree 002d7c9a 3d24fc34 gives tree 283f0dc8, with one conflict: test/bench.test.ts.

• git diff --stat 283f0dc8 ab9bfc88 shows that test/bench.test.ts is the only file that differs. Every other file is exactly what git's own merge of the two parents gives, including scripts/bench/hidden.ts and test/change-files-renames.test.ts.

• In test/bench.test.ts, the resolution:

  • removes the conflict markers;
  • keeps main's PLUS_LINE and its five tests;
  • keeps this branch's async 512-byte tar test.

  Its only other changes are , () => becoming , async () => on main's four stdin tests, and await added before each of their 7 stdinSends(...) calls.

• Main's hiddenLeaks import is kept. The argv test calls only hiddenLeaks, so it stays synchronous, as it should.

• No conflict markers are left at the head (0).

All five of main's #797/#804 tests are present (measured, asserts.mjs origin/main 242f663b):

• Test titles: all 31 of main's are in the head, which has 66.

• Assert lines: main has 171. Of these:

  • 150 are identical in the head;
  • 16 differ only by await ;
  • 5 are missing, and none of them is from Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 or bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804.

  Those 5 are main's older assert.throws lines on r.exec (main:509-511) and syncTree (main:640, :646). In the head they are await assert.rejects(...) with the same arguments and the same matchers (test/bench.test.ts:559-561, :1039, :1045).

  • The same 5 already differed before the merge: asserts.mjs c339175a 002d7c9a lists exactly these 5. So the merge changed none of main's assertions.

Main's four stdin tests now await (measured, git diff 283f0dc8 ab9bfc88):

• each is async;
• every stdinSends(...) call in them is awaited (3 + 2 + 1 + 1 call sites).

Main's tests still catch what they were written for (measured). I put older versions of hidden.ts into my worktree only, rebuilt and ran the bench tests, then restored the head's file (git status --porcelain is empty afterwards).

• c339175's hidden.ts (before Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797's fix): # fail 3. The failing tests are:
  • the Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 argv test ('LF: the "++ " line is found', expected true, actual false);
  • the Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 stdin test (expected false, actual true);
  • the bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 paren test.
• fb72e67's hidden.ts: # fail 3. The failing tests are:
  • the bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 paren test;
  • the R1/R2/R2b test;
  • the nameless +++ test (each expected false, actual true).
• So in their await form, every one of the five fails on the code it was written against.

#804 is intact (measured): git diff origin/main 242f663b -- scripts/bench/hidden.ts is empty.

2. The new RAW_UNSAFE additions

The change is \p{Co} and \u{FFFC} added to the class, plus a new alternative (?! )\p{Zs}. The other three parts are unchanged.

Results (measured, sweep-r10.mjs):

• Method: the regex is read from the compiled dist/test/bench.test.js (rule 8). It is compared with 002d7c9's regex over every code point, with surrogates included as lone code units, at the start, middle and end of a line.
• Strict superset: no code point is caught by 002d7c9's regex in some position but let through now in that position.
• Newly caught: 137,485 code points, all of them non-ASCII \p{Zs}, \p{Co} or U+FFFC. That is every code point in those classes, and each is caught in all three positions. The old regex caught none of them, so the six new fixtures all fail on 002d7c9's regex, and the builder's 137,485 is right.
• Nothing else changes: nothing outside those classes is newly caught.

Nothing legitimate is refused (measured):

• Let through:
  • the ASCII space in every position, and indentation made of ASCII spaces;
  • tab, a CRLF line end and a lone CR at the end of the last line;
  • U+FFFD;
  • Latin, Greek, Cyrillic, CJK and kana, Hangul, Arabic, Hebrew with niqqud, Devanagari, Tamil and Thai;
  • emoji, including a flag and a skin tone;
  • combining marks, typographic punctuation, box drawing, braille U+2801 and math symbols.
• Still caught: a CR mid-line, a CR at line start, and CR CR at line end.
• The 16 scanned files: the scan finds [].
  • The files are test/bench.test.ts, the README and the 14 .ts and .mjs files under scripts/bench.
  • The only non-ASCII characters in them are U+00B7, U+00E2, U+00E9, U+4E2D, U+6587, U+0436 and U+2026, and all pass.

Applies only to the repo's own source files, not to bench paths or user data (measured):

• grep -rn RAW_UNSAFE src scripts test finds only test/bench.test.ts:1757, :1799, :1800 and :1805.
• The one scan (:1804-1805) reads a fixed list: this test file plus readdirSync(scripts/bench) filtered to .ts, .mjs and .md. Those are the 16 files listed above.
• grep -rnE 'p\{Co\}|FFFC|p\{Zs\}' scripts src finds nothing. No runtime code under scripts/bench or src uses these classes, and the driver doesn't read the regex.
• So refusing private use or non-ASCII spaces can't block a bench run, a task path, a run id or held-out data. It can only fail this repo's test when such a character is committed raw into a bench source file, and a \u{...} escape is still available there (inferred, from the above).

Rule 10 and principle 9 (measured):

• the regex only grows;
• the fixture test's pass list is unchanged;
• the scan test keeps its scan and its deepEqual(found, []), and only its title changed;
• nothing is skipped, exempted or allow-listed.

3. Tests, gates and the usual checks

Bench tests on 242f663 (measured): # tests 66, # pass 66, # fail 0, exit 0. Nothing was left in the temp home, state or held-out folders.

• Command: runtest.sh <wt> head bench.test.js, which runs node --import ./dist/test/isolate.js --test dist/test/bench.test.js with the temp dirs and stand-ins.

Gates on the head (measured):

• npm run typecheck: exit 0;
• check:denylist: denylist: 1299 files clean;
• check:dco -- origin/main HEAD: DCO: all commits signed off.

Commits and attribution (measured):

• ab9bfc8 and 242f663 each carry exactly one trailer, Signed-off-by.
• No attribution line appears in the body or the round-10 comment (grep for co-authored, "generated with" and the vendor address: 0 hits).

Protection (measured):

• The PR's 10 files are all under scripts/bench/** or test/ (git diff --stat origin/main 990e9a34, the tree from merging with today's main).
• The PR is marked protected: the title has [PROTECTED: owner merges], the body has **Risk class:** protected, the PR has the needs-owner label, and it has "What this can never do" and "Compromised component" sections.
• This round changes no source file, so those sections still hold (inferred).

CI on 242f663 (measured, gh run list --branch issue-748-bench-driver, every run with headSha 242f663):

• Succeeded:
  • ci 39223048752: all 8 shards, both full test jobs on Node 22 and 24, dco, denylist, push-gate, agentshield and desktop;
  • ci slow 39223048819 (scope);
  • both gitleaks scans;
  • new-deps 39223132424.
• Cancelled: two earlier new-deps runs on the same commit. inferred: body edits re-triggered them, and each new run cancelled the one before. The latest new-deps run passed.
• Skipped, as usual: the real-systemd, bootstrap and sign-in-probe jobs.

Merge with current main (measured):

• origin/main is 178c44e.
• git merge-tree --write-tree origin/main 242f663b exits 0 with tree 990e9a34, so there is no conflict.
• That tree differs from main only in this PR's 10 files.
• Main's changes since 3d24fc3 (Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793) touch no bench file (git diff --stat 3d24fc34 178c44e6).

Rule A and no regression for existing installs: nothing in src/ or any installed path changes. This round adds no state access (measured by diff; the conclusion is inferred).

4. Hand-off: no Adversarial pass for round 10 (measured)

• The body: it has "Adversarial pass" sections for rounds 9, 7 and 6, but none for round 10. Round 10's changes appear only under "How it was verified".
• The round comment: issuecomment-6263149184 has no Adversarial pass section.
• A stale claim the round makes false (principle 5): the body's round-9 Adversarial pass still says that non-ASCII \p{Zs}, private use and U+FFFC are "Still let through, by design", and that "The test title doesn't claim them". Since 242f663, both statements are false about the head.

Required changes

1. Add an "Adversarial pass (round 10)" section to the PR body, and post it in a round comment. It should list the attacks tried on this round's two changes, their results (measured) and what was fixed:

   • the merge resolution;
   • the RAW_UNSAFE additions.

   It should go through the principles, as the round-9 section does. In it, correct the round-9 section's "Still let through, by design" and "The test title doesn't claim them" items, which are no longer true. Don't rewrite them; a correction note is enough.

   This is text only. No code change is needed, and the next check only needs to read the body and comment.

Suggestions (not blocking)

• The body's line "main is merged in here (round 6, a signed-off merge)" near the top could name round 10's merge of 3d24fc3.

For the owner

There is no new trade-off. The only blocker is a missing hand-off section. The code, tests, CI and the merge with today's main all check out.

[jayden1711]

Round 11 (text only): the round-10 Adversarial pass, and corrections

There is no code change: the head stays 242f663. The body now has an "Adversarial pass (round 10)" section, a correction note in the round-9 section, and an updated base line.

Adversarial pass (round 10)

Run in round 11 against 242f663, with fixtures, a scratch worktree (since removed) and stand-ins only. Every result is measured unless labelled.

• Principle 7 (rule 10), the merge keeps main's tests:
  • Main's assertions: every one of the 161 assert lines in main's test/bench.test.ts is in the branch's, up to an added await before stdinSends. The only exceptions are five older lines that earlier rounds made assert.rejects, because exec and syncTree are async here.
  • No call left unawaited: no stdinSends( call lacks await, so no test compares a Promise with false.
  • Main's new tests are live, not vacuous: with c339175's (pre-bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804) hidden.ts under the head's tests, the result is # pass 63, # fail 3. The two Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 tests and the open-paren bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 test fail. The other two pass on the old code too. inferred: they guard cases the earlier rule already handled ("as before Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797").
• Principle 3:
  • git diff origin/main -- scripts/bench/hidden.ts is empty, so bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804's addedLines, importLines and strict && loose are main's;
  • Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793 (178c44e) merges cleanly and touches no bench file.
• Principle 1, RAW_UNSAFE's additions (the regex read from the compiled test):
  • Spaces: all 16 non-ASCII \p{Zs} are refused, and only U+0020 passes. ASCII spaces and tab indentation pass.
  • U+FFFC: refused at a line's start, middle and end.
  • Private use: U+E000 and U+10FFFD are refused.
  • A non-ASCII space before a CRLF end: refused.
  • Visible text passes: an em dash, an emoji, CJK and Arabic.
  • U+FFFD passes. inferred: it is visible, so it isn't in the class this scan names.
  • Over every code point: nothing 002d7c9's regex caught is let through, and no default-ignorable, format or variation-selector code point gets through.
  • The 16 files, main's new tests included, scan clean.
• Principle 5, corrected:
  • the round-9 section's "still let through, by design" and "the test title doesn't claim them" (a correction note, not a rewrite);
  • the base line, which now names round 10's merge (ab9bfc8, main 3d24fc3).
• Principles 2, 6 and 9: only test/bench.test.ts changed in round 10, besides the merge. The regex only grows, and no exception, allowlist entry or skip is added.
• Fixed because of this pass: nothing in the code. The text corrections above were made.
• Principle 8: no refusals.

[jayden1711]

Evaluator verdict: PASS

This is round 11, a text-only round. The head is unchanged at 242f663, which gh api graphql's headRefOid shows (measured). Round 10's check accepted all of the code, so this round checks the body and issuecomment-6263334626 only.

Method:

• I took the current body, and the body version I reviewed in round 10 (the 08:29:28Z entry), from the PR's userContentEdits. I diffed them with git diff --no-index.
• I checked the text with check.mjs. It uses the head's RAW_UNSAFE, read via git show in a detached review worktree.
• Both scripts are in the session scratchpad under work/eval-759-r11/.

Check	Result
1. The section exists in the body and the comment, and covers the merge and RAW_UNSAFE	met (measured)
2. Its measured claims match mine	met (measured)
3. The correction note is correct	met (measured)
4. No raw control or format characters, and no attribution	met (measured)
5. Nothing else in the body changed	met (measured)

1. The section is present and covers both changes (measured)

• Where: "## Adversarial pass (round 10)" is in the body, just before the round-9 section, and in the comment. The comment is a condensed copy with the same points.
• The merge:
  • principle 7: main's assertions, no unawaited stdinSends, and main's new tests are live;
  • principle 3: hidden.ts is identical to main's, and Merge: idempotent across a crash: merge.attempted before the API call, then reconciled, so the engine never reverts its own good merge (#733) [PROTECTED: owner merges] #793 merges cleanly.
• The RAW_UNSAFE additions:
  • principle 1: spaces, U+FFFC, private use, a non-ASCII space before CRLF, visible text, U+FFFD, the full code-point sweep, and the 16 files;
  • principles 2, 6 and 9: the regex only grows, and nothing is skipped or exempted.
• Also there: principle 5 (claims corrected), "Fixed because of this pass" and principle 8.

2. The claims match what I measured

• c339175's hidden.ts (the spot-check): the section says # pass 63, # fail 3, with the two Test: list scripts/bench/hidden.ts on the changed-files reader list (main red after #688 + #757) #797 tests and the bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804 open-paren test failing. In round 10 I got exactly the same result: # tests 66, # pass 63, # fail 3, and the same three tests (measured, round-10 runtest.sh … oldhidden).
• "The other two pass on the old code too":
  • What matches (measured): the two tests did pass in my c339175 run.
  • The builder's reason: they guard cases the earlier rule already handled. The builder labels this inferred, and it is consistent with what I measured.
  • What I measured that the builder did not: with fb72e67's hidden.ts (bench: hidden.ts counts added lines starting "++ "; pin its two lines on the reader list (#797 follow-up) [PROTECTED: owner merges] #804's intermediate state), both of these tests fail, and so does the paren test. So all five of main's tests catch a real bug on this branch.
• "161 assert lines" in main's file: main's test/bench.test.ts has 161 lines containing assert. (measured, check.mjs).
  • My round-10 count of 171 used a looser pattern, /assert/.
  • Both counts agree on the 5 exceptions, which are assert.throws lines that became assert.rejects.
• "No stdinSends( call lacks await": 0 call sites in the head's test lack it (measured).
• The RAW_UNSAFE claims (measured: check.mjs and my round-10 sweep-r10.mjs):
  • 16 non-ASCII \p{Zs} are refused, and only U+0020 passes;
  • U+FFFC is refused in all three positions;
  • U+E000 at a line start and U+10FFFD are refused;
  • a + U+00A0 + CR is refused;
  • U+FFFD passes;
  • nothing 002d7c9's regex caught is let through;
  • the 16 files scan clean.
• "Only test/bench.test.ts changed in round 10, besides the merge": confirmed by git diff --stat ab9bfc88 242f663b in round 10 (measured).

3. The correction note is correct (measured)

• Where: it sits under the round-9 section's "Still let through, by design" list, which is left as written.
• What it says: both statements ("Still let through, by design" and "The test title doesn't claim them") are false about the head since 242f663.
• Both are now false, as it says:
  • 242f663's RAW_UNSAFE refuses every non-ASCII \p{Zs}, \p{Co} and U+FFFC;
  • the scan test's title now names "private-use", "non-ASCII space" and "U+FFFC".

The base line: it now names round 10's signed-off merge ab9bfc8 of 3d24fc3 (#804). It says earlier merges of main were in rounds 6 and 8.

• Measured: the branch's first-parent history has three merges of origin/main (3aa2eb2, c6994d1 and ab9bfc8), and four merges of the bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757 branch.
• Inferred: which merge belongs to which round I took from the body's round notes.

4. Characters and attribution (measured, check.mjs)

• The body (451 lines) and the comment (29 lines):
  • 0 RAW_UNSAFE hits;
  • 0 hits for a broader class (\p{Cc}, \p{Cf}, \p{Co}, \p{Cs}, \p{Zl}, \p{Zp}, default-ignorable, U+FFFC and non-ASCII \p{Zs});
  • no CR and no tab.
• Attribution: 0 hits for co-authored, "generated with" or the vendor address, in either.

5. Nothing else changed (measured)

• git diff --no-index --stat from the body I reviewed in round 10 to the current body gives 38 insertions and 1 deletion. All of it is in three places:
  • the base line (line 16);
  • the new round-10 section;
  • the correction note in the round-9 section.
• The latest edit (08:47:15Z) is identical to the current body.

Also confirmed

• Protection: it is unchanged and correct: [PROTECTED: owner merges], **Risk class:** protected, needs-owner, and both protected-PR sections.
• CI: the head and its CI are unchanged since round 10's check, so every check is green on 242f663. Round 10's measurements still hold: 66 of 66 bench tests, typecheck, denylist and DCO. git merge-tree with origin/main 178c44e is clean (measured in round 10). The new text changes no code (inferred).

For the owner

There is no new trade-off. This PR is protected and is yours to merge.