[jayden1711]

Intent: This is the second of three stacked PRs for #748 Phase 0. Each task's repo environment and tests run in a throwaway container in the bench user's rootless Docker; the agents run on the host in the engine's sandbox. Each container:

• has no network and no capabilities, and mounts nothing;
• has required memory (with an equal memory-swap), CPU and pids limits, read back from the daemon and from the container's own cgroup (memory.swap.max = 0 included) before anything runs in it;
• is reached only through an exec-only runner. Hidden fields never reach the agent or the container (taskView, the first wall). As a narrow second wall, the runner scans a command's argv and stdin for a driver bug that would pass one on (exactly what it checks is below).

The bench user's daemon could mount that user's claude login into a container, so no agent can reach it:

• the sandbox denies the socket, the runtime dir and the daemon's data root;
• the agent's environment carries no daemon variable;
• the driver's own docker always names its daemon, never a docker context's;
• a preflight probe checks this from inside sandbox-runtime 0.0.79's sandbox before every run.

Nothing here runs a model, Docker or a download.

Risk class: protected (every path is under scripts/**).

Part of #748. #756 (preflight) is merged; base main, with main 0b294b5 merged in. Next: #759 (driver, grading, aggregates, README).

Size: 1,282 lines against the base (measured, git diff --stat, round 4), over the ~700 the owner asked to hear about. 846 are in the five modules (code and comments) and 436 in the tests. I have not split it further.

What's in it

• scripts/bench/container.ts:

  • checkDockerArgs, which every docker command passes. Each option is parsed with its value. The required options are looked for among the parsed options only, so a value (--name --network) never stands in for one. Option names are own keys. It allows only:

    • info --format;
    • inspect --format <bench container>;
    • start and rm --force of a bench container;
    • create/run with exactly --name <bench container>, --network none, --cap-drop ALL, no-new-privileges, the four limits, --workdir /work and --entrypoint sleep;
    • exec with only a normalized --workdir under /work, --env NAME=value (the = is required, and no credential or daemon name is allowed) and --interactive.

    Any -v, --volume, --mount, --privileged, --device, --user or other network is refused.

  • dockerEnv: the docker CLI's environment carries no credential variable and always has DOCKER_HOST: the bench user's own, or else the rootless socket unix://$XDG_RUNTIME_DIR/docker.sock. So the CLI never takes its daemon from a docker context, which can name any socket. It refuses DOCKER_CONTEXT and a relative unix:// DOCKER_HOST.

  • TaskRunner, the task's commands: it allows only exec into its own container. syncTree sends a worktree as a tar stream on exec -i, so nothing is mounted. Its base (a tar of the task's base commit, made by the driver before any agent runs) is required. A task's stdin with no base tree is refused outright.

  • Limits:

    • limitsFrom: WORKLANE_BENCH_MEMORY/CPUS/PIDS, defaults 8g, 4, 4096. Refused: a zero or missing value, CPUs over the host's, memory over 75% of the host's, and memory that isn't a multiple of 4096 (the kernel keeps memory.max in pages, so it would never read back equal).
    • Docker.probe: create, then inspect (Memory, MemorySwap, NanoCpus, PidsLimit), then start, then read memory.max, memory.swap.max, cpu.max and pids.max inside, then rm.
    • Docker.start: the same read-back for each task's container.
    • checkLimits requires memory.swap.max to be 0. It raises LimitsNotApplied, naming the controller and the README's Delegate= drop-in.

  • Rootless check: docker info security options must include name=rootless.

• scripts/bench/hidden.ts:

  • taskView, the first wall: an allowlist of the base commit, the problem statement and the test runner. No hidden field (the gold patch, the test patch, FAIL_TO_PASS, PASS_TO_PASS, the hints, the difficulty, the instance id) is ever handed to an agent or a container.

  • The second wall is a UTF-8 substring scan, and it checks only the following.

    • Every docker argv (hiddenLeaks):

      • the added lines of the gold patch and the test patch that are at least 24 characters long;
      • FAIL_TO_PASS and PASS_TO_PASS test ids of at least 12 characters;
      • the hints, the instance id, and a difficulty key.

      A piece the problem statement holds doesn't count. A refusal names the field.

    • A command's stdin, the tar stream included (stdinCarriesHidden, owner's decision of 2026-10-18). Only what an agent can't plausibly write itself counts:

      • the instance id, or the hints, as a substring (at least 8 characters);
      • the test patch as a run of 3 or more consecutive added lines, matched as whole lines in a row:
        • applied, or still in diff form: each line is matched as it stands and again with one leading + dropped, so the test patch written into the tree as a .diff file (LF or CRLF) is refused (round 4);
        • blank lines are left out on both sides, a blank context line in the patch included (round 4), and the three lines must hold at least 24 non-space characters between them;
        • a window made only of import lines doesn't count (owner, round 4): Python import a / from a import b, the names inside an open from a import (, a \ continuation. Whether a line is an import line is read from the new side of the patch's hunk. A run of imports and code still counts;
        • a NUL (tar padding) ends a line, as a newline does, and the stream gets a NUL before each tar header block (tarText, round 4), so a file of exactly 512·n bytes with no trailing newline doesn't run its last line into the next file's name.

      Gold-patch lines, test names and test ids are not checked on stdin: an agent's correct fix may equal the gold line, and its own regression test may share the upstream test's name. A piece the base tree already holds doesn't count (for a test-patch run, in either form). The id and the hints also don't count when the problem statement holds them.

    • Every stdin refusal says the same words, whatever matched, so a refusal relayed to an agent never tells it which of its lines match.

  • What the scan can't see:

    • compressed or binary content (a gzip member, a .git object pack);
    • other encodings (UTF-16);
    • shorter lines, such as a changed expected value in an existing test;
    • on stdin: single test-patch lines, runs of fewer than 3, a run of import lines only, a run split across files or reordered, gold lines (the gold patch as a .diff file included: the owner's round-2 decision) and test names.

    Those rely on the first wall.

• scripts/bench/agent.ts:
  • benchLane: built with the engine's withDeniedReads, so the Read tool gets both rule forms for each path, Read(//p/**) and the exact Read(//p). A file such as ~/.claude.json, ~/.netrc, ~/.npmrc or ~/.pypirc matches only the exact form. Commands reach no host. The lane denies:
    • $XDG_RUNTIME_DIR (its docker.sock, bus and the rootless containerd socket included) and its docker.sock by name;
    • /run/docker.sock and /var/run/docker.sock;
    • a unix DOCKER_HOST's socket (a relative one is refused);
    • the rootless daemon's data root (~/.local/share/docker, and $XDG_DATA_HOME/docker when set);
    • ~/.claude, ~/.claude.json, ~/.docker (contexts included) and the other credential stores;
    • the bench's state and held-out store.
  • LaneInput's dockerHost and dataHome are required keys (string | undefined), so a caller can't silently leave them out.
  • daemonHost: the daemon the driver's docker talks to (see dockerEnv).
  • benchClaude: the engine's CliRunner with retries off (NO_RETRY, budgetMs: 0). A transient failure, a 429 included, is one session, reported at once. bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756's meter then stops the bench on it.
• scripts/bench/probe.ts (owner's requirement and choice, option A, 2026-10-18): the sandbox probe, before every run.
  • Version check: srt --version must be exactly 0.0.79, or the run is refused.
  • The probe: under srt --settings <file> -- node -e <probe>, with the file holding the lane's own denyRead and allowed hosts as sandbox-runtime configuration, it tries:
    • to connect to every daemon socket the lane denies, plus $XDG_RUNTIME_DIR/bus (through it, systemd-run --user runs a command outside the sandbox) and the rootless containerd socket $XDG_RUNTIME_DIR/docker/containerd/containerd.sock;
    • to open ~/.claude.json, ~/.netrc, ~/.npmrc and ~/.pypirc. It reports only what each one is (fstat) and never reads a byte.
  • The settings file and its temp folder are removed once srt returns, whatever happened.
  • Refused means: the lane has the deny rule for the path (for a read, the Read rule too), and inside the sandbox the attempt failed or opened something other than a regular file (a masked file is /dev/null).
    • A missing file counts as refused only when its deny rule is in place. That is how the cases are told apart: the deny rule is checked in the generated settings, and on_host (an lstat outside the sandbox) is recorded alongside.
  • The run refuses unless every attempt is refused. The per-attempt report goes on the result, and on the error when it refuses. bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 records it in the summary and aggregate.
  • Label: "sandbox-runtime 0.0.79: close to, not identical with, the build bundled in Claude Code 2.1.303".
    • Claude Code 2.1.303 has no model-free command that runs a command in its own sandbox (measured: its --help commands list, and the CLI reference).
    • Fingerprint (measured, static, nothing executed): I compared the ASCII [Sandbox Linux] … log strings of each published sandbox-runtime release with the 2.1.303 binary, using scripts/bench/srt-fingerprint.mjs (added in bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759, read-only). The bundled build lacks 0.0.79's "Re-covering a mount point an earlier sandbox left behind". It has a "No placeholder for …: neither bwrap nor the command can create …" line that no release has. 0.0.80 differs in about 4 more lines. So it sits between 0.0.79 and 0.0.80, and 0.0.79 is closest.
    • The install step and the re-check on new releases go in bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759's README.
• scripts/bench/auth.ts: agentBase drops DOCKER_*, XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS and CONTAINER_HOST. It applies to the runner's base, to each request's own env and to the bench's probes.
• test/bench.test.ts: 26 tests in all on this branch (10 from bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756).

Known issue (for the planned test relay)

hiddenLeaks (the argv scan) refuses an exec whose argv names a FAIL_TO_PASS or PASS_TO_PASS test by its pytest id (measured by #757's reviewer, and again in round 3). The agent's test relay (after #759) will hit this when an agent asks to run such a test, and the refusal names the field. That relay must not echo argv refusals to the agent as they are (inferred: the same oracle as on stdin), and should exempt ids already in the base tree.

Still ahead in #748 Phase 0: the PR-mode task runner on src/harness-evals/replay.ts with stub CI, the agent's test relay into the container (the agent can't reach the daemon, so its test runs need a driver-side relay through TaskRunner), task selection, the pinned-download scripts, the grading run, the B0, B1 and claude -p baselines, and the CLI entry point. The driver and grading are in #759.

How it was verified

• Round 4 (measured, head 94c6e71: 90b1bab with main 0b294b5 merged in):
  • With a temp HOME, state, held-out and XDG dirs and refusing claude, docker and srt stand-ins first on PATH (their hit log was never created): bench.test.js gives # pass 26, # fail 0. npm run typecheck exits 0. npm run check:denylist: denylist: 1283 files clean. npm run check:dco -- origin/main HEAD: DCO: all commits signed off.
  • git diff --stat origin/main...HEAD: only this PR's 6 files, 1,282 insertions and 6 deletions, the same as against the old base.
  • Fail-without: a scratch worktree at d57acd8 (round 3's code, no guard edited) with only the new test/bench.test.ts checked out: # pass 21, # fail 5. Each new test fails there:
    • the test patch as a .diff file (LF): true !== false (sent);
    • the same with CRLF: true !== false (sent);
    • an agent's own test file with the new test file's imports: the agent's own test file, with the same imports: sent, false !== true (refused);
    • a blank context line: split by a blank context line " ": refused, true !== false;
    • a run ending a 512-byte tar member: a run ending a 512-byte file, then a header: refused, true !== false.
  • The reviewer's round-3 probe on the new build: the .diff test patch, with and without the instance id elsewhere and with CRLF, is now refused. The two 2-line blocks split by a blank context line are refused. The run ending at exactly 512 bytes is refused. The agent's test file with the same 3 imports is sent. The gold patch as a .diff file is still sent (owner's decision). Every "must pass" and "must refuse" case keeps its round-3 result. Split-across-files and reordered runs are still sent.
• measured (round 3, head d57acd8, bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756 merged in): node --import ./dist/test/isolate.js --test dist/test/bench.test.js with a temp HOME, state and held-out dirs and refusing claude, docker and srt stand-ins first on PATH: # pass 21, # fail 0. npm run typecheck exits 0. npm run check:denylist: denylist: 1282 files clean. npm run check:dco -- origin/main HEAD: DCO: all commits signed off.
• Fail-without, round 3 (measured):
  • Method: a scratch worktree at d57acd8 with scripts/bench/{hidden,container,agent,probe}.ts checked out from its parent. No guard was edited.
  • The new tests on the old code: # pass 18, # fail 3:
    • the tar-stream test: refused: this docker command's input would carry the task's hidden patch (not in its base tree) on an agent fix identical to the gold line;
    • the daemon test: Missing expected exception: the lane: unix://docker.sock;
    • the probe test: the settings file is removed, true !== false.
  • A probe of each case against both builds (stand-in docker). Old code, then new:
    • an agent fix identical to the gold line: refused hidden patch, then sent;
    • an agent test with the upstream test's name: refused hidden test_patch, FAIL_TO_PASS, then sent;
    • an applied 3-line test-patch hunk, the hints, the instance id: refused naming the field, then refused with the same words each time (… input carries hidden task content that its base tree lacks);
    • an argv naming a FAIL_TO_PASS id: refused on both (unchanged);
    • probe sockets: 3, then 5 (with bus and the containerd socket);
    • no DOCKER_HOST: the CLI got none, then got unix:///run/user/1001/docker.sock;
    • DOCKER_CONTEXT=elsewhere, and a relative unix://docker.sock: sent, then refused before anything was sent. The lane with that relative socket: accepted with no rule for it, then refused.
  • inferred: base and the LaneInput keys being required is enforced by tsc. The runtime refusal of a task's stdin with no base is in the tar-stream test.
• The real srt 0.0.79 (round 2, measured). I ran the npm tarball's own CLI, with this branch's generated lane, against a temp home holding a sentinel ~/.claude.json and a listening unix socket as the daemon's. There was no model, no Docker and no real home. socat and bubblewrap 0.9.0 are on this machine.
  • The bench lane passed: all 7 attempts were refused:
    • each socket connect: EPERM;
    • ~/.claude.json: EACCES (it exists on the host);
    • ~/.netrc, ~/.npmrc, ~/.pypirc: ENOENT, missing on the host, deny rule present.
  • A lane missing the ~/.claude.json and runtime-dir rules was refused: read <tmp>/u/.claude.json: no deny rule, where the sandbox saw opened file.
  • inferred: the socket EPERM comes from sandbox-runtime's seccomp filter, which blocks every unix socket, a socket not denied included. So for sockets the probe also requires the deny rule (checked in the settings). The two sockets added in round 3 weren't run under the real srt (inferred: both sit under the runtime dir the lane denies).
  • measured: srt leaves an existing denied socket file in place on the host (lstat before and after: socket -> socket).
• Fail-without, round 2 (measured), against this PR's own pre-fix modules (git checkout a52dafd7 -- scripts/bench/{agent,container,hidden}.ts, then restored; no guard edited):
  • With the old container.ts/hidden.ts: # pass 15, # fail 3:
    • Missing expected exception: exec --env HOME bench-r1-t01 ls;
    • Missing expected exception: cgroup-swap;
    • Missing expected exception: an added line of the test patch (the tar stream; that check is replaced in round 3).
  • With the old agent.ts too (the test file no longer links, so I probed the build directly), the old code:
    • allowed run --name --network, --name --cap-drop and --name --security-opt (the real flag removed), run --name victim, exec --workdir=/work/../etc and memory 100000000;
    • had Read(//home/u/.claude.json) in the deny list: false; ~/.local/share/docker and the DOCKER_HOST socket denied: false;
    • benchClaude on a 429: transient waits started=1.
  • The fixed build refuses each case, has each rule, and gives transient waits started=0, reason=failed.
  • The probe is a new module, so it can't run on the base. Its test covers both outcomes with stand-in srts: one that honours the settings (passes), and one with no sandbox and a live socket (refused, naming connected and opened file). It also covers a wrong version and a lane without the exact Read rules.
• Round 0 (measured on 9d73baa4 by the builder): limits off gave Missing expected rejection: inspect-memory; agentBase off gave run: XDG_RUNTIME_DIR; the verb allowlist off gave Missing expected exception: cp x bench-r1-t01:/x; the hidden-field check off gave 0 !== 6.
• Unix-socket connect to a read-denied path: settled by the bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757 reviewer (measured with bwrap's own mounts: tmpfs over the folder gives ENOENT, /dev/null over the socket gives ECONNREFUSED), and by the srt run above.

New dependencies

None. sandbox-runtime 0.0.79 is installed for the bench user only (README step in #759). The repo's own pin (0.0.78, used by the engine's MCP broker) is unchanged.

What this can never do

• Mount anything into a task container, or give it a network, a capability or a device: checkDockerArgs refuses every such option on parsed options (a value can't impersonate one), and makeArgs builds none.

• Run a task container without its memory, CPU and pids limits, with swap beyond its memory, or with a limit that silently didn't apply. Both the daemon's record and the container's own cgroup are read back, and a mismatch refuses.

• Let an agent reach the bench user's daemon or user bus through the environment or the sandbox. Every run is refused unless sandbox-runtime 0.0.79's sandbox with the lane's settings refuses each socket (the user bus and the containerd socket included) and credential file. That is close to, not identical with, the build Claude Code 2.1.303 bundles. bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 adds a confirmation in Claude Code's own sandbox.

• Talk to a daemon other than the one whose socket the lane denies through a docker context or a relative socket path: the driver's docker always gets an absolute DOCKER_HOST, and DOCKER_CONTEXT is refused (measured, test and probe). inferred from the docker CLI's documented precedence: an explicit DOCKER_HOST overrides the configured context.

• Hand a hidden field to an agent or a container: taskView is an allowlist (the first wall). The second wall is narrower, and it can miss:

  • a hidden piece in compressed or binary content, in another encoding, or in a line under 24 characters;
  • on stdin: a test-patch run under 3 lines, a run of import lines only, a run split across files or reordered, a gold line (the gold patch as a .diff file included) or a test name.

  What it refuses is listed in the hidden.ts entry above.

Compromised component

• An agent in a bench run:
  • Its commands are sandboxed with no network. They are denied the runtime dir, the sockets, the daemon's data root, the login and the credential files (the Read tool gets the exact rule for each file), and the bench's state. Its environment has no daemon variable.
  • It runs as the bench user, so the OS doesn't stop it reading that user's files; the sandbox does, for commands and the Read tool.
  • inferred: Edit and Write run outside the sandbox, and this deny list covers reads only. bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 confines them to the task's own working folder.
  • A tree it writes can't learn which hidden field it matches from a stdin refusal: the words are the same for every match (measured). Its own correct fix, same-named test, or test file that starts with the same imports as a new upstream test file goes through (measured).
  • inferred: the residual is a sandbox escape. That would give the bench user's own powers: its login and its daemon. Which is why the owner runs the bench under its own user, apart from their own account.
• A task's own repo code (tests run in the container): no network, no mounts, capped resources, rootless (container root is the bench user's unprivileged uid). It can damage only its own container.
• The srt the bench user installed: it runs only the probe, with the lane's settings, as the bench user. A tampered srt could report false refusals. The version is checked but not the bytes, so a reviewed hash pin would be the next step (inferred).
• Coordinator, dashboard, other instances, hub viewer: none runs or reaches this code. Nothing in src/ imports it.
• Someone with push access: they could widen RUN_OPTIONS or EXEC_OPTIONS, loosen a limit, change SRT_VERSION or STDIN_RUN, or widen what counts as an import line. The guard is this review: the owner merges scripts/**.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 1/2)

Reviewed: the diff of issue-748-bench-containers (a52dafd) against its base issue-748-bench-preflight (0a45e71) only. Labels: measured = a command I ran (or CI output I cite), inferred = from reading code or docs, not run.

Acceptance and focus checks

Check	Result	Evidence
Docker argv: no mount, --privileged, --cap-add, --device, host network/pid/userns, other --security-opt, cp	Met for every path the driver has	measured (probe script against the compiled checkDockerArgs): refused --network=host, --net host, --cap-add=ALL, --security-opt=seccomp=unconfined, --pid=host, --userns=host, --device=/dev/fuse, --volumes-from, --tmpfs, -v, exec --privileged=true, exec --user=0, exec -i, exec --interactive=true, exec -- …, a target starting with -, another container, an empty target, cp, volume, and a global --host before the verb. inferred: send is module-private, and create/run argv come only from makeArgs, so no path builds anything else.
The checker on its own	Partly: see required change 4	measured: run --name --network … with the real --network none removed is ALLOWED, and so are --name --cap-drop and --name --security-opt with the real flag removed; run --name victim is ALLOWED. The required-flag check scans raw argv strings, so a value can stand in for a flag, and --name is never checked against NAME. inferred: docker rejects a container named --network, so this isn't reachable today.
Env values with newlines	Met	measured: a newline in a name is refused; one in a value is allowed. That's harmless: it's argv, not a shell.
Limits read back: inspect Memory, MemorySwap, NanoCpus, PidsLimit	Met	container.ts:202-204. measured: refused on MemorySwap: -1 and on NanoCpus given as a string.
Limits read back: cgroup memory.max, memory.swap.max, cpu.max, pids.max	Not met: memory.swap.max isn't read	CGROUP_READ (container.ts:215) reads only memory.max, cpu.max and pids.max. See required change 1.
Refuses on any mismatch; units	Met	measured: refused on max in each of the three files, on missing lines and on 405000 100000 against 4 CPUs. It passes on bytes, 400000 100000 and 4096. The units are right: bytes, quota/period, and max always refused.
Sandbox deny list: $XDG_RUNTIME_DIR, rootless and rootful docker.sock, ~/.claude, ~/.claude.json, held-out store	Met for commands. Partly met for the Read tool	agent.ts:21; the test asserts each entry. measured (probe on benchLane): the file entries get only Read(//<home>/.claude.json/**), and likewise .netrc, .npmrc and .pypirc. See required change 3.
Env strips DOCKER_*, XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS	Met	measured with a probe of cliEnv and BenchAgentRunner. On the base (0a45e71) both keep XDG_RUNTIME_DIR, DOCKER_HOST, DBUS_SESSION_BUS_ADDRESS and CONTAINER_HOST. On the head they keep none.
Unix socket connect to a read-denied path	Settled: blocked	See the next section.
Hidden fields: every docker argv	Met	send (container.ts:165-170); the test passes.
Hidden fields: the tar stream	Not met	measured: syncTree sent a tar holding a test-patch line, the instance id and the hint. hiddenLeaks on the same text returns ["test_patch","hints_text","instance_id"], but send never checks input. hidden.ts:4 says "Everything sent to a container is also checked", which overclaims. See required change 2.

Unix-socket connect (the builder's open inference)

• inferred from sandbox-runtime 0.0.78, the engine's pinned copy (dist/sandbox/linux-sandbox-utils.js ~1395 and ~2138): on Linux a denyRead entry becomes

  • --tmpfs over a directory;
  • --ro-bind /dev/null over anything else, a socket included;
  • nothing at all when the path is absent.

• measured: socat isn't installed here, so srt itself won't start. I applied those same bwrap mounts directly, with a test unix-socket server in a scratch dir:

  • bwrap with no deny: CONNECTED;
  • tmpfs over the socket's directory: ENOENT;
  • /dev/null over the socket file: ECONNREFUSED.

  So the read deny alone blocks a connect by path.

• inferred (sandbox-runtime README, "Unix Socket Restrictions"): a second layer exists. seccomp refuses socket(AF_UNIX) inside the sandbox, but only on x64/arm64. Where seccomp isn't available the runtime just warns, so the deny is the layer that holds.

• inferred: the sandbox unshares the network namespace, which also cuts off abstract sockets.

• Not verified: that Claude Code 2.1.303's bundled runtime emits the same mounts as 0.0.78.

• No required change. Please replace the PR body's "inferred, not run" with this result.

Fail-without

• measured: the PR's test/bench.test.ts on the base (in a worktree at 0a45e71) doesn't compile: TS2307: Cannot find module '../scripts/bench/container.js', and the same for agent.js and hidden.js; TS2305: … no exported member 'agentBase'.
  • This is unavoidable: every guard here is a new module.
• measured: the one guard that changes existing code, agentBase in cliEnv and BenchAgentRunner, fails without the PR. A probe of the base's compiled auth.js passes XDG_RUNTIME_DIR, DOCKER_HOST, DBUS_SESSION_BUS_ADDRESS and CONTAINER_HOST through.
• measured: on the head, node --import ./dist/test/isolate.js --test dist/test/bench.test.js gives # pass 12, # fail 0. npm run typecheck exits 0. npm run check:denylist gives denylist: 1274 files clean.
• The builder's guard-off output is kept privately (combined branch 9d73baa4). I didn't repeat it, per the brief: no guard-off edits.

Protection and housekeeping

• measured: every path is under scripts/** or test/**. The PR is marked protected:
  • the title has [PROTECTED: owner merges];
  • the body has **Risk class:** protected;
  • it carries the needs-owner label;
  • it has real "What this can never do" and "Compromised component" sections.
• "Can never do":
  • "send a hidden field … in any docker argv" is true as worded. The module comment isn't (change 2).
  • "Let an agent reach the daemon … through the environment or the sandbox" holds (measured, above).
• "Compromised component, agent" says the sandbox stops reads "for commands and the Read tool". For credential files that is only partly true (change 3).
  • inferred: it doesn't mention Edit/Write, which run outside the sandbox. See recommendation C.
• No regression for existing installs: measured, the diff touches nothing in src/. inferred: no install route changes.
• measured: one commit, signed off, no Co-Authored-By or other attribution. No new dependencies. No .claude/settings.json.
• The body says "Part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748" and has no "Closes". That fits a middle PR in the stack (inferred).
• Real state: inferred from the code. The new tests use a fake claude, temp dirs, and HOME, which isolate.js moves on Linux. None reads or writes real state.

Scope

• inferred: there is no scope creep. Everything serves Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748 Phase 0, part 2.

CI

• measured (gh pr checks 757; gh run view 39180734140 --log-failed): the only failing tests are in shard 4/4 on both Node 22 and 24. They are "the package stays within its size budget" and "focus and selection rings use the ring token, not the accent", both known main failures (Main green: package budget for the merged growth since v0.5.2, and drop the stale RING_OWED entry [PROTECTED: owner merges] #753).
• The two test (ubuntu-latest, *) jobs are the aggregate over the shards (inferred from their 2-4 s duration).
• The release npx-init job didn't run in this workflow.
• Nothing fails because of this PR.

Required changes

1. Read back memory.swap.max. Add /sys/fs/cgroup/memory.swap.max to CGROUP_READ and require 0. On cgroup v2, --memory-swap equal to --memory gives memory.swap.max = 0. Refuse on max or any other value, naming the memory controller. Add a fake-docker case (cgroup-swap → max) to the "a limit that did not apply refuses" test. Today the kernel side of the swap limit is never checked: only the daemon's record is.

2. The tar stream and hidden fields. Do one of these and update hidden.ts:4 to match:

   • (a) check send's input too. A plain hiddenLeaks over the whole tar would refuse on common test-patch lines that already exist in the base tree (inferred), so scan only what differs from the base commit. At least cover the test-patch added lines, the hints, the instance id, and FAIL_TO_PASS names that aren't in the base tree.
   • (b) if the owner accepts the argv-only wall, say plainly in the module comment and in "What this can never do" that the tar stream isn't checked, and that the first wall (taskView plus the held-out deny) is all that covers it.

3. Read-tool rules for credential files. Build the lane's deny list with the engine's withDeniedReads (src/sandbox.ts:60). It adds both Read(/p/**) and Read(/p). sandboxSettings alone emits only the /** form (measured). Under the gitignore semantics Claude Code documents for Read rules, p/** matches what's inside p, not the file p (inferred; the engine's own withDeniedReads adds the exact form for files for this reason). Then ~/.claude.json, ~/.netrc, ~/.npmrc and ~/.pypirc are likely readable by the agent's Read tool. Extend the agent test to assert Read(//<home>/.claude.json) is in permissions.deny.

4. Make checkDockerArgs self-sufficient.

   • Check --name against NAME.
   • Run the required-flag check on the flags the loop parsed, not on any argv string. Today a value (--name --network) satisfies it (measured).
   • Add the --name --network case to the test.

   No path reaches this today (inferred). But the "can never do" claim rests on this function, and the fix is a few lines.

Recommendations (not blocking)

• A. hiddenLeaks refuses an exec that names an existing PASS_TO_PASS test by its pytest id (measured). Those tests exist in the base tree, and running them leaks nothing. The agent's test relay (bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 and later) will hit this. Consider dropping PASS_TO_PASS from the argv check, or excluding names already in the base tree.
• B. Small checker tightenings:
  • exec --env NAME with no = is allowed (measured). Docker then copies NAME's value from the driver's environment, which dockerEnv filters only by SECRET_NAME. Require =.
  • exec --workdir=/work/../etc passes the checker (measured). TaskRunner normalizes, but the checker could too.
  • A memory limit that isn't a multiple of 4096 (e.g. 100000000) is accepted (measured). The kernel rounds memory.max down, so the read-back would then refuse every run. Round it or refuse it in limitsFrom.
• C. Also deny the rootless daemon's data-root (~/.local/share/docker) to commands and Read (inferred). Container layers are owned by the bench user there, so any grading container that exists while agents run would expose its hidden tests to cat.
  • Also derive a deny entry from a unix:// DOCKER_HOST.
  • And have bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 state where Edit/Write may write. Today the deny list covers reads only, and the tools run outside the sandbox. A write to ~/.docker/config.json (credsStore) or a shell rc file would reach the driver.

[jayden1711]

Round 1 fixes (head 2fda9e6; #756's round 1 merged at 2c8a6b8)

Labels: measured = a command I ran or CI output, inferred = reasoned, not run.

Required changes

1. memory.swap.max read back.
   • CGROUP_READ now reads memory.swap.max, and checkLimits requires 0.
   • Anything else refuses with memory (memory.swap.max) limit did not apply, naming the memory controller.
   • The fake docker gained cgroup-swap (→ max), and it is in the "a limit that did not apply refuses" test.
   • measured fail-without: Missing expected exception: cgroup-swap.
2. The tar stream is scanned (option a).
   • send checks the stdin of a command too.
   • TaskRunner takes the task's base tree (a tar of the base commit), and a hidden piece counts only if the base tree lacks it. The pieces are the added test-patch and gold lines, hints, the instance id, FAIL_TO_PASS ids and test names. Context lines and PASS_TO_PASS tests already in the base are not counted.
   • With no base tree given, every piece counts.
   • The hidden.ts header now says exactly that.
   • measured fail-without: Missing expected exception: an added line of the test patch.
3. Read rules come from withDeniedReads.
   • benchLane now gets both Read(//p/**) and Read(//p).
   • The test asserts the exact rule for .claude.json, .netrc, .npmrc, .pypirc and .docker/config.json.
   • measured fail-without (a probe of the pre-fix build): Read(//home/u/.claude.json) in permissions.deny: false.
4. checkDockerArgs parses options with their values.
   • The required options are looked for among the parsed options only.
   • --name must match a bench container, and option names are own keys.
   • The test has your cases: run --name --network, --name --cap-drop and --name --security-opt (each with the real flag removed), plus run --name victim and a run with no --name.
   • measured fail-without (a probe of the pre-fix build): all four were ALLOWED. The fixed build refuses each.

Also done

• --env requires NAME=value. exec --env HOME and --env=PATH are refused (measured fail-without: Missing expected exception: exec --env HOME bench-r1-t01 ls).
• --workdir must already be normalized. /work/../etc, /work/t/../../etc and /work//t are refused (pre-fix: measured ALLOWED).
• Memory that isn't a multiple of 4096 is refused, with a suggested value (pre-fix: measured ALLOWED for 100000000).
• Denied to commands and Read: ~/.local/share/docker, $XDG_DATA_HOME/docker and a unix DOCKER_HOST's socket. A tcp DOCKER_HOST adds nothing (pre-fix: measured not denied).
• benchClaude has retries off (NO_RETRY, budgetMs: 0), as the owner approved for bench: preflight for the SWE-bench run (subscription-only login, CLI pin, caps, usage meter) [PROTECTED: owner merges] #756.
  • Test: the 429 stand-in through benchClaude gives one session, failed + transient, reported at once, with no transient wait started.
  • measured fail-without: transient waits started=1.
• The sandbox probe, the owner's option A. probe.ts, checked with srt --version = 0.0.79.
  • Under srt --settings <lane as sandbox-runtime config> -- node -e <probe>, it tries every daemon socket and opens ~/.claude.json, ~/.netrc, ~/.npmrc and ~/.pypirc, reporting only what each one is.
  • The run is refused unless every attempt is refused with its deny rule in place. A missing file counts only with its rule there; on_host is recorded beside each attempt.
  • Label: "sandbox-runtime 0.0.79: close to, not identical with, the build bundled in Claude Code 2.1.303".
  • Fingerprint (measured): scripts/bench/srt-fingerprint.mjs (in bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759). 0.0.79 differs from the bundled build by 1 log string each way, 0.0.78 by 1 and 2, 0.0.80 by 4 and 4.
  • Real srt 0.0.79 (measured): I ran the npm tarball's CLI with this branch's generated lane, on a temp home only (a sentinel ~/.claude.json and a listening socket).
    • The bench lane passed: sockets EPERM, ~/.claude.json EACCES, and the three missing files ENOENT with their rules present.
    • A lane missing the ~/.claude.json and runtime-dir rules was refused, and the sandbox saw opened file.
    • srt left an existing denied socket file in place.
  • Its test uses stand-in srts for both outcomes, a wrong version and a lane without the exact Read rules. It is a new module, so on the base it doesn't compile.
• Known issue for the planned test relay, now in the body: an argv naming a PASS_TO_PASS test by its pytest id is refused.

Body

• "What this can never do" and "Compromised component" are updated: argv and stdin, swap, the exact Read rules, the probe, Edit and Write running outside the sandbox (with a pointer to bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759), and a tampered srt (version checked, not hash).
• The unix-socket section now cites your bwrap result and the srt run.
• Size: 969 lines against the base (measured), over ~700. I have not split it.

Fail-without method (measured)

• This branch's tests against the PR's own pre-fix modules (git checkout a52dafd7 -- scripts/bench/{container,hidden}.ts) gave # pass 15, # fail 3.
• With agent.ts reverted too, the test file no longer links (the new benchDenied export is missing), so those guards were measured with a probe of the pre-fix build.
• Both were then restored. No guard was edited.

Local and CI (measured)

• Locally: bench tests # pass 18, # fail 0; typecheck exits 0; denylist: 1275 files clean; DCO ok.
• First CI run (0950e0e): red on shard 4/4, no test file measures a new duration on the wall clock: test/bench.test.ts: 1, allowed 0. That was mine: the no-retry test timed itself with Date.now(). Fixed in 2fda9e6 with performance.now().
• CI on 2fda9e6: all 19 checks pass, every shard on Node 22 and 24 included.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 2/2)

Reviewed: the diff of issue-748-bench-containers at 2fda9e6 against its base issue-748-bench-preflight (f122bfd, an ancestor of the head) only. Labels: measured = a command I ran (or CI output I cite), inferred = from reading code or source, not run.

All four round-1 required changes are done, and so are the extras and the owner's check A. Two things are left, and both come from the new tar-stream scan:

• it claims more than it covers;
• it refuses an agent's own correct work.

The scan was my round-1 option (a). I asked for test-patch lines and FAIL_TO_PASS names to be counted, and I missed that the tree being scanned is the agent's own.

Round-1 required changes

#	Change	Result	Evidence
1	memory.swap.max read back, must be 0	Met	CGROUP_READ (container.ts:240) reads it, and checkLimits (container.ts:232) requires '0'. measured (probe on the head build): 0 passes. max, 4096, an empty line and a missing line are each refused with memory (memory.swap.max) limit did not apply. The test has the cgroup-swap case.
2	The tar stream is checked against the base tree	Met as built, with gaps: see required changes 1 and 2	send (container.ts:186-195) runs hiddenLeaks on input, and a piece found anywhere in base doesn't count. Details below.
3	Read rules via withDeniedReads, with exact-file rules	Met	benchLane (agent.ts:58-60). measured: Read(//home/u/.claude.json), .netrc, .npmrc, .pypirc and .docker/config.json are each true. Every denied path has both Read(//p/**) and Read(//p).
4	checkDockerArgs parses flags with their values	Met	measured: I re-ran my round-1 injection probe (43 cases) on the head build. Each round-1 hole is now refused: run --name --network / --cap-drop / --security-opt with the real flag removed, --name=x, and --name victim. Every case refused in round 1 is still refused. Option names are own keys: --__proto__, --toString and --hasOwnProperty are refused.

Extras

Item	Result	Evidence (measured unless noted)
--env needs =	Met	exec --env HOME, --env PATH and --env XDG_RUNTIME_DIR are refused (only NAME=value). So are --env =x, GH_TOKEN=x and DOCKER_CONFIG=x. A=b and --env=A= are allowed.
--workdir normalised	Met	Refused: /work/../etc, /work/t/../../etc, /work//t, /work/./t, /workx and work/t. Allowed: /work and /work/.
Memory a multiple of 4096	Met (refused, not rounded)	100000000 is refused with a suggested 95m. 8g, 512m, 65536k and 67108864 pass.
Data root and DOCKER_HOST denials	Met	Commands and Read are both denied ~/.local/share/docker, $XDG_DATA_HOME/docker and unix:///srv/alt.sock's socket. A tcp DOCKER_HOST adds nothing. See recommendation C.
benchClaude uses budgetMs: 0	Met	NO_RETRY = {"budgetMs":0}. inferred from src/runner.ts:534-561: with a budget of 0, now() + wait - first > budget holds on the first transient, so the run returns with no wait. The test "the bench never retries a run" passes on the head.

Owner's sandbox check A (probe.ts)

I read srt 0.0.79's dist/cli.js and dist/sandbox/linux-sandbox-utils.js from the builder's unpacked tarball. Nothing downloaded was run.

Item	Result	Evidence
srt version pin	Met	SRT_VERSION = '0.0.79'. requireSrt refuses unless the trimmed stdout of srt --version is exactly that. inferred from srt's cli.js: program.version(pkg.version), and its package.json has "version": "0.0.79". measured (test): a missing srt and a 0.0.78 srt are both refused before any attempt. The install step is in #759's README (npm install -g --prefix ~/.local @anthropic-ai/sandbox-runtime@0.0.79).
Attempts cover every Docker socket path and the four files	Met for the owner's list	measured probeAttempts: connects to $XDG_RUNTIME_DIR/docker.sock, /run/docker.sock, /var/run/docker.sock and the DOCKER_HOST socket (every socket benchDenied lists), and reads of ~/.claude.json, .netrc, .npmrc and .pypirc. See recommendation B for two more sockets under the runtime dir.
Run refuses unless every attempt is refused	Met	sandboxProbe (probe.ts:136-148) refuses in any of these cases: a non-zero exit, unreadable output, a count mismatch, or any attempt not refused. measured (test): the no-sandbox stand-in is refused, naming connected and opened file, and the sentinel isn't in the message.
Missing file counts only with its deny rule	Met	verdict = rule && …. measured: verdict(false,'read','ENOENT') is false and verdict(true,'read','ENOENT') is true. The lane without exact Read rules (sandboxSettings alone) is refused with no deny rule. on_host is recorded beside each attempt.
"close to, not identical" label	Met	measured: sandbox-runtime 0.0.79: close to, not identical with, the build bundled in Claude Code 2.1.303. The test pins this exact string.
Result recorded	Met for this PR's part	The report (with runner) is returned, and it's attached to the error on refusal. The summary write is #759's: driver.ts:161, const probe = o.preflight.sandboxProbe() (measured by grep on issue-748-bench-driver; not reviewed here).
Stand-in tests for pass and refuse	Met	fakeSrt('honors') decides from the denyRead in the generated srt settings file and passes. fakeSrt('none') refuses, and the no-exact-rule lane refuses. The none stand-in sends every attempt outside its temp dir to a missing path, so the test never touches the machine's real sockets (inferred from the code).
srt semantics	Consistent	inferred from srt's source: a read-denied file gets --ro-bind /dev/null (linux-sandbox-utils.js:2375), which the probe reads as opened char-device = refused. socket(AF_UNIX) is blocked by seccomp unless allowAllUnixSockets is set (:2631). The lane never sets it (src/sandbox.ts:28-44). If the apply-seccomp binary is missing, srt only warns (:2633). The probe still requires the deny rule for sockets, so that case doesn't pass silently on the rule check.

The tar scan: can hidden content get in?

All measured by probing TaskRunner.syncTree on the head build, with a base tree given.

• A file the base already has, changed by the test patch: Yes, when every added line is under 24 characters. A test patch that changes assert f(-1) == -5 to assert f(-1) == 0 inside the existing FAIL_TO_PASS test test_negatives was SENT. The test's name is in the base, so it doesn't count, and the changed line is too short to be a piece. This is the common SWE-bench shape: an existing test's expected value changes. A longer added line in the same file was refused.

• A binary or encoded file: Yes. The new FAIL_TO_PASS test was SENT in three forms:

  • as a gzip member inside the tar;
  • as a whole gzipped tar stream;
  • as a UTF-16LE file.

  The same text as plain UTF-8 was refused. The scan is a UTF-8 substring search, so it can't see inside compressed members (.git/objects included, if a tree ever carries .git) or other encodings.

• A tar header field: No gap found.

  • The instance id as a file name in a header was refused, and so was the hints text in a pax record.
  • inferred: a piece can split across ustar's prefix and name fields only at a /, so instance ids, test names and hints, which have no /, can't be split.
  • inferred: numeric header fields (mode, mtime) could carry data only by deliberate encoding, which isn't this wall's threat (a driver bug).

• The other direction (it refuses legitimate work):

  • An agent's own correct fix, identical to the gold line, was refused (hidden patch). The line was return sum(c * q for c, q in items if q > 0) with no marker.
  • An agent's own regression test that happens to share the upstream test's name was refused (hidden test_patch, FAIL_TO_PASS).
  • The PR's own test asserts the first case: "the gold patch" in bench.test.ts is the fix line plus a marker comment.
  • inferred: the refusal names the matching field. Once the planned relay passes errors back to the agent, that makes an oracle: the agent can learn which of its lines or test names match the hidden patch.

• No base given: the unchanged base tree itself was refused (hidden FAIL_TO_PASS, PASS_TO_PASS), because send passes '' and so every test name counts. This fails closed. But bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 builds new TaskRunner(o.docker, c, t.row, undefined, signal) (measured by grep), so any real tree synced there would be refused. See recommendation A.

Fail-without

• measured: this PR's tests were run against the round-1 head's own modules, not a guard-off edit. I ran git checkout a52dafd7 -- scripts/bench/container.ts scripts/bench/hidden.ts in my worktree, then built and ran the tests: # pass 15, # fail 3:

  • Missing expected exception: exec --env HOME bench-r1-t01 ls;
  • Missing expected exception: cgroup-swap;
  • Missing expected exception: an added line of the test patch.

  This matches the builder's numbers. tsc reported one type error, TaskRunner called with 4 arguments where the old code takes 3, and still emitted.

• measured, with round-1 agent.ts as well: the test file no longer links (benchDenied is missing), so I probed the build instead.

  • Read(//home/u/.claude.json): false.
  • Data root and DOCKER_HOST socket denied: false.
  • These were ALLOWED: run --name --network, --name --cap-drop and --name --security-opt, each with the real flag removed; exec --workdir=/work/../etc; exec --env HOME; and memory 100000000.
  • inferred: the old benchClaude built CliRunner with no retry options, so it used the default budget.

• measured: after restoring the head modules, the bench tests give # pass 18, # fail 0. npm run typecheck exits 0. npm run check:denylist gives denylist: 1275 files clean. All runs used a temp HOME, state dir and held-out dir, with XDG_RUNTIME_DIR, DOCKER_HOST and the bus address unset.

• probe.ts is new, so there is no old code to run it against. Its stand-in tests cover both outcomes (above).

Size

• measured: 969 lines (git diff --stat). That's about 421 non-comment code lines in the four new modules, plus 15 in auth.ts, and 309 of tests.
• Acceptable (inferred): it's one cohesive unit (containers, limits, the hidden wall, the agent lane), and the owner added option A mid-review.
  • probe.ts and its test (about 220 lines) were the one clean split. Splitting now would cost another review round for little gain.

Protection and housekeeping

• measured: every path is under scripts/** or test/**. Protection is marked: the title has [PROTECTED: owner merges], the body has **Risk class:** protected, the PR has the needs-owner label, and both sections are present.
• "What this can never do":
  • The mount and network bullet holds (measured, above).
  • The limits bullet holds (measured).
  • The daemon bullet: the probe's refusal is wired in bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759, not here (inferred).
  • The hidden-field bullet ("or in a tar stream except where the task's base tree already holds it") is not true as worded (measured, above). See required change 1.
• No regression for existing installs: measured, the diff touches nothing in src/. inferred: no install route, permission or exposure changes.
• measured: 4 commits, all signed off (DCO CI green). No Co-Authored-By or other attribution. No new dependencies. No .claude/settings.json. No hardcoded product name added.
• Real state (rule A), inferred from the code:
  • The tests use a fake claude, stand-in srts, temp dirs and the moved HOME.
  • The probe test only lstats the host's /run/docker.sock and /var/run/docker.sock (metadata) and never connects to them.
  • Nothing reads or writes the owner's state.

Scope

• inferred: there is no scope creep. Option A and NO_RETRY are the owner's decisions of 2026-10-18.

CI

• measured (gh pr view 757 --json statusCheckRollup, head 2fda9e6): every check is SUCCESS, and the only others are the ci slow jobs, which are SKIPPED. That covers all 8 test shards on Node 22 and 24, dco, denylist, new-deps, gitleaks and agentshield.
• The first CI run on 0950e0e failed on the wall-clock-durations check. That was the builder's own test, and 2fda9e6 fixes it.

Required changes

1. Make the tar-scan claims match what it covers. The places to change:

   • hidden.ts:4-8 ("every byte on a command's stdin");
   • container.ts:15-16;
   • the PR body's "What this can never do" hidden-field bullet.

   Each should say the scan is a UTF-8 substring check of:

   • test-patch and gold added lines of at least 24 characters;
   • test ids and names of at least 12 characters;
   • the hints and the instance id.

   And say what it can't see:

   • compressed or binary members;
   • other encodings;
   • shorter lines, such as a changed expected value in an existing test.

   The first wall (taskView plus the held-out deny) is what covers those. measured: the four "SENT" cases above.

2. Stop the stdin scan from refusing an agent's own work. measured: an agent fix identical to the gold line is refused, and so is an agent test that shares the upstream test's name. Concretely:

   • On stdin, count only pieces an agent can't plausibly write itself: the instance id, the hints, and the test patch as a run of at least 3 consecutive added lines (what a driver bug that applied the patch would carry), not single lines.
   • Drop gold-patch lines and test names from the stdin check.
   • Add a test: a tree whose fix equals the gold line, and one with a same-named test, both go through, while an applied test-patch hunk is still refused.

   If the owner prefers to keep single-line matching, the alternative is to record a match on the task's result for the owner instead of refusing. Either way, the relay must not echo which hidden field matched (inferred oracle, above).

Recommendations (not blocking)

• A. Make TaskRunner's base required.
  • With no base, every real tree is refused (measured), and bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 passes undefined today.
  • In the same way, make LaneInput.dockerHost and dataHome required keys (string | undefined), so a caller can't silently forget them. No caller passes them yet (measured, grep of bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759).
• B. Also probe $XDG_RUNTIME_DIR/bus and the rootless containerd socket ($XDG_RUNTIME_DIR/docker/containerd/containerd.sock; path inferred from rootless dockerd's exec-root).
  • Both are covered only by the runtime-dir deny today. Either one gives the same reach as the Docker socket: systemd-run --user through the bus runs commands outside the sandbox.
• C. dockerHostSocket ignores a relative unix://name.sock (measured: no deny is added). Docker contexts (~/.docker/contexts) can also name a socket. Consider refusing to start when the driver's daemon socket isn't under a denied path.
• D. sandboxProbe leaves its settings file in os.tmpdir(). Remove it after the run.

[jayden1711]

Round 3 (owner-granted): what changed

Head d57acd8. #756's round-3 head (571f3c0) is merged in first with a signed-off merge commit, b79b30b. The only conflict was test/bench.test.ts, where both sides added a test at the end; both are kept (measured).

1. The tar-scan claims are now exact (owner decision 1).

• hidden.ts's header, container.ts's header and the PR body now say the scan is a UTF-8 substring check, and list what it covers:
  • argv: added lines of at least 24 characters, test ids of at least 12, the hints, the instance id;
  • stdin: the decision-2 set below.
• They also say what it can't see: compressed or binary content, other encodings, shorter lines (and on stdin, single test-patch lines, gold lines and test names).
• They say the first wall is taskView: hidden fields never reach the agent or the container.
• "What this can never do" now says the second wall can miss these.

2. Stdin is refused only for what an agent can't write (owner decision 2).

• A new stdinCarriesHidden counts only:

  • the instance id or the hints (as substrings);
  • the test patch as a run of 3 or more consecutive added lines, matched as whole lines in a row.

• Gold-patch lines and test names (and test ids) are dropped from the stdin check. A piece the base tree holds still doesn't count.

• Every stdin refusal is one fixed message: refused: this docker command's input carries hidden task content that its base tree lacks.

• Two choices of mine, inferred as needed for the rule to work on real trees:

  • blank lines are left out of a run. Otherwise two blank lines plus a same-named def would be a "run", and PEP 8's two blank lines are everywhere;
  • a run's three lines must hold at least 24 non-space characters between them, so three closing brackets are no patch.

  A NUL ends a line, as a newline does, so a hunk at the very start of a file in the tar is still caught (measured, test).

• The test: an honest change, a gold-identical fix, a same-named test, two test-patch lines followed by the agent's own line, a FAIL_TO_PASS id in a comment, and a 3-line run the base holds all go through. An applied hunk (in place, as a new file's first lines, and with CRLF ends), the hints and the instance id are refused, with nothing sent. All five refusals give one message, which names no field.

• Fail-without (measured): the new test fails on the old code at the gold-identical fix (… hidden patch (not in its base tree)). In a probe against both builds, the same-named test went from refused (hidden test_patch, FAIL_TO_PASS) to sent, and the applied hunk, hints and instance id went from field-naming refusals to the one generic message.

• The argv scan is unchanged. Its refusal still names the field. The PR body's "Known issue" now says the planned test relay must not echo argv refusals to an agent (inferred).

3. Non-blocking items, all done.

• TaskRunner's base is required, and so are LaneInput.dockerHost and dataHome (string | undefined). At runtime, a task's stdin with no base is refused (measured, test).
• The probe also tries $XDG_RUNTIME_DIR/bus and $XDG_RUNTIME_DIR/docker/containerd/containerd.sock. measured in the probe: 3 sockets before, 5 after. Both sit under the runtime dir the lane already denies.
• A relative unix:// DOCKER_HOST is refused, by the lane and by the driver's docker. DOCKER_CONTEXT is refused. The driver's docker CLI now always gets DOCKER_HOST: the bench user's own, else the rootless unix://$XDG_RUNTIME_DIR/docker.sock. So a docker context in ~/.docker never picks the daemon. measured on old vs new: no DOCKER_HOST sent before, the rootless socket now; DOCKER_CONTEXT and a relative socket were sent before, and are refused now with nothing sent. That an explicit DOCKER_HOST beats a configured context is inferred from the docker CLI's documented precedence.
• The probe's settings file and its temp folder are removed once srt returns, in a finally. measured: the test records the path srt was given and asserts that the file and folder are gone. On the old code it fails with the settings file is removed, true !== false.

Local results (measured): bench.test.js gives # pass 21, # fail 0, run with a temp HOME, state and held-out dirs and refusing claude, docker and srt stand-ins first on PATH. npm run typecheck exits 0. check:denylist prints denylist: 1282 files clean, and check:dco prints DCO: all commits signed off. On the old modules, the new tests give # pass 18, # fail 3.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 3)

Reviewed: the diff of issue-748-bench-containers at d57acd8 against its base issue-748-bench-preflight (571f3c0) only. Labels: measured = a command I ran (or CI output I cite), inferred = from reading code, not run, guess = judgement.

Round 3 does what the owner asked, with one exception. The new stdin rule doesn't catch the test patch when it is written into the tree as a .diff file, and the "can't see" lists don't mention that. Round 2's code refused it, so this is a regression in the second wall, against the driver-bug threat it exists for. The fix is small (required change 1). Everything else checks out.

All runs used a temp HOME, state, held-out and XDG dirs, with refusing claude/docker/srt stand-ins first on PATH. None of them was called: their hit log was never created (measured). No model, Docker or download was used.

Round-3 items

#	Item	Result	Evidence
1	Claims in hidden.ts, container.ts and the body match the scan	Met for what's listed; one omission	inferred, from reading hidden.ts:1-18, 62-175: - argv: added lines of at least 24 trimmed characters, test ids of at least 12, the whole hints and instance id (at least 8), and the difficulty key regex (the difficulty value gives no piece). That is exactly what pieces and hiddenLeaks do. - stdin: the id or hints as substrings of at least 8 characters; 3-line windows with blank lines dropped, at least 24 non-space characters, NUL as a line end. That is exactly what stdinCarriesHidden, textLines and windows do. Omission (measured, below): the "can't see" lists don't say that a test patch in diff form (+-prefixed lines) goes through on stdin. Required change 1.
2	stdinCarriesHidden refuses only the id, the hints, or a run of 3+ test-patch lines, with one fixed message	Met	measured (probe of TaskRunner.syncTree on the head build, with a real ustar stream I built). Each of these was refused with exactly refused: this docker command's input carries hidden task content that its base tree lacks: - the applied hunk in place; - the hunk as the first lines of a new file; - the same with no trailing newline; - the hints; - the instance id in a file, and as a tar member's name; - the row dumped as JSON (caught by the id). STDIN_REFUSED contains none of patch, FAIL_TO_PASS, PASS_TO_PASS, hints, instance or test. The test asserts this too. The only other stdin message, "…checked against its base tree, and there is none", doesn't depend on content (inferred).
3	A gold-identical fix, a same-named test and a test id pass	Met on stdin	measured, head: SENT for each. Also SENT: two of the hunk's lines followed by the agent's own line. Round-2 code refused all four (measured, the same probe on the parent modules: hidden patch, hidden test_patch, FAIL_TO_PASS, hidden FAIL_TO_PASS, hidden test_patch, FAIL_TO_PASS). On argv, a FAIL_TO_PASS or PASS_TO_PASS id is still refused, naming the field (measured). That is the open item the body records; it isn't required here.
4	The builder's two choices	Both sound	See "The two choices", below.
5a	base required	Met	TaskRunner's base: Buffer and LaneInput's dockerHost/dataHome: string | undefined are required by the types (inferred; npm run typecheck exits 0, measured). At runtime, undefined as Buffer is refused with checked against its base tree, and there is none (measured, test).
5b	The probe tries $XDG_RUNTIME_DIR/bus and the containerd socket	Met	measured (test): the report lists ~/run-user/bus and ~/run-user/docker/containerd/containerd.sock, each refused: true. There are 5 connects, before the 4 reads.
5c	A relative DOCKER_HOST and DOCKER_CONTEXT are refused	Met	measured (test): unix://docker.sock and unix://./run/alt.sock are refused by the lane and by the driver's docker. DOCKER_CONTEXT=elsewhere is refused with nothing sent. inferred: an empty unix:// is refused too (isAbsolute('') is false).
5d	The driver's docker always gets an absolute DOCKER_HOST	Met	measured (test): with no DOCKER_HOST, the CLI gets unix:///run/user/1001/docker.sock. DOCKER_CONFIG and GH_TOKEN are dropped. A set unix:///srv/docker/alt.sock is passed on unchanged, and the lane denies it. inferred from the docker CLI's context resolution: a set DOCKER_HOST selects the default context ahead of currentContext in the config.
5e	The temp settings file is removed	Met	rmSync(dir, {recursive, force}) runs in a finally (probe.ts:138-147). measured (test): the file and its folder are both gone.

The two choices, and the tar edge cases

All measured with my probe on the head build, using a real ustar stream (512-byte headers, NUL padding).

• Blank lines don't count toward a run: sound.

  • A hunk with blank lines between its lines in the tree is refused.
  • A patch hunk whose blank lines are added lines is refused, whether the tree keeps those blanks or not.
  • One inconsistency: a blank context line in the patch still ends a run, but the tree side drops it. So two 2-line added blocks separated by a blank context line are SENT once applied. That's rare (guess). See recommendation A.

• 24 non-space characters per run: sound.

  • Real short hunks are caught: def test_neg(): plus two asserts (41 characters), and @pytest.mark.xfail / def test_x(self): / pass (38).
  • Only trivial ones go through, such as x = f(1) / y = g(2) / assert x == y (22). That costs little.

• A hunk at the very start of a file in the tar: refused, with and without a trailing newline. The header's NUL padding ends the line before it.

• Split across files, or reordered: SENT in each case:

  • 2 lines in file A and 1 in file B;
  • the lines in the order 2, 1, 3;
  • a line inserted between them.

  inferred: none of these is a driver-bug shape. A driver that applies or copies the patch writes a hunk whole and in order. An agent can't arrange it on purpose without the hidden text, which the first wall withholds. Acceptable.

• A run at the end of a file of exactly 512·n bytes with no trailing newline: SENT. The next header's name joins the last line, so it no longer matches. Narrow (guess). See recommendation B.

• The test patch written as a .diff file: SENT on the head, REFUSED on round 2's code. The same happens with CRLF line ends. Its lines start with +, so they never equal a whole patch line. See required change 1.

• A gold .diff file: SENT. This follows from the owner's decision to drop gold lines from stdin. I note it so the owner sees it.

• A false positive (owner-visible): an agent's own new test file that starts with the same 3 import lines as a new upstream test file (import pytest / from price import total, subtotal / from price.models import Item) is refused. This follows from the 3-line rule, and the 24-character floor doesn't help. How often a whole 3-line import block matches and the base lacks it: guess, occasional. See recommendation C.

Fail-without

measured: I made a scratch worktree at d57acd8 and ran git checkout b79b30ba -- scripts/bench/{hidden,container,agent,probe}.ts (the parent's modules; no guard edited). tsc built it with no errors. bench.test.js gave # pass 18, # fail 3:

• the driver's docker always names its daemon…: Missing expected exception: the lane: unix://docker.sock;
• the sandbox probe test: expected: false, actual: true (the settings file is still there);
• the tar-stream test: refused: this docker command's input would carry the task's hidden patch (not in its base tree).

These match the builder's numbers. On the head: # pass 21, # fail 0. npm run typecheck exits 0. check:denylist prints denylist: 1282 files clean.

Protection, size, housekeeping

• measured:
  • Every path is under scripts/bench/** or test/**. The title has [PROTECTED: owner merges], the body has **Risk class:** protected, and the PR has the needs-owner label. "What this can never do" and "Compromised component" are both present.
  • Nothing in src/ is touched. inferred: no install route, permission or exposure changes, so there is no regression for existing installs.
• "What this can never do":
  • The daemon and context bullet holds (measured, 5c and 5d).
  • The hidden-field bullet's list of what the second wall can miss lacks the diff-form test patch (required change 1).
• Rule A, inferred from the code: the new tests use an in-process docker, a stand-in srt, temp dirs and a moved HOME. No test reads or writes real state.
• measured: 6 commits, all signed off. No Co-Authored-By and no attribution line in the commits or the body. No new dependencies. No .claude/settings.json.
• measured: 1,142 lines against the base (git diff --stat), matching the body.
• Scope: nothing beyond the owner's round-3 decisions (inferred).

CI

measured: gh pr checks 757, run 39193365737, headSha d57acd8. Everything passed: all 8 test shards (Node 22 and 24), dco, denylist, new-deps, scope, scan, agentshield, desktop and push-gate. Only the slow real-systemd jobs were skipped.

Required changes

1. Catch the test patch in diff form on stdin, or list it as a gap.

   • measured: a tree holding the task's test patch as a .diff file (LF or CRLF) is sent on the head and was refused by round 2's code. A driver bug that copies the held-out patch file into a worktree is the plainest shape of the leak this wall exists for. The owner's rule says "a run of 3 or more test-patch lines", and those lines are in it.

   • Preferred fix (inferred, a few lines):

     • in stdinCarriesHidden, also match each text line with one leading + removed (a second pass over textLines(text).map(l => l.startsWith('+') ? l.slice(1) : l), with the same base exemption);
     • add a test case: the test patch as a .diff file is refused with STDIN_REFUSED.

     inferred: this adds no false-positive risk beyond the applied form.

   • If the owner prefers not to change the code: add "the test patch as a diff (+-prefixed lines)" to the "can't see" lists in hidden.ts:16-18, the body's hidden.ts entry, and "What this can never do".

Recommendations (not blocking)

• A. In testPatchRuns, let a blank context line leave a run open, as a blank added line does. The tree side already drops every blank line, so the two sides would agree. (measured miss above.)
• B. A run that ends a file of exactly 512·n bytes with no trailing newline is missed (measured). Either list it, or end lines at 512-byte block boundaries too.
• C. For the owner: the 3-line rule refuses an agent's own test file whose first 3 import lines match a new upstream test file (measured). If that matters, the alternative from round 2 still applies: record a stdin match on the task's result instead of refusing, or skip windows made only of import/from … import lines.
• D. Body nit: on stdin, the id and hints are also exempt when the problem statement holds them (hidden.ts:160). The body mentions only the base tree.

Open item (noted, not required here)

The argv scan still names the matching field (measured: hidden FAIL_TO_PASS and hidden PASS_TO_PASS for pytest ids, hidden patch for the gold line). The body's "Known issue" says the future test relay must not echo argv refusals to an agent, and should exempt ids the base already holds. That needs to be a requirement on the relay PR.

[jayden1711]

Round 4 (owner-granted): what changed

New commit 90b1bab on top of d57acd8 (no force-push). Labels: measured = a command I ran, inferred = from reading code, guess = judgement.

Owner decision 1: the test patch as a .diff file is refused again

• stdinCarriesHidden matches each stdin line as it stands and again with one leading + dropped. The base exemption applies to both forms: a run the base tree already holds, applied or as a diff, doesn't count.
• measured: two new tests, LF and CRLF, each run on its own. Both fail on d57acd8 (true !== false: the stream was sent) and pass on the head (the stream is refused with STDIN_REFUSED, and nothing is sent).
• measured: the reviewer's probe (eval-757-r3/probe.mjs) on the head refuses all three .diff cases: with the instance id elsewhere, without it, and with CRLF.
• The gold patch as a .diff file is still sent. That is unchanged, per the owner's round-2 decision.

Owner decision 2: an import-only window is no longer a run

• A 3-line window made only of import lines is skipped. Import lines are Python import a / from a import b, the names inside an open from a import ( up to its ), and a \ continuation. Whether a line is an import line is read from the new side of the test patch's hunk, context lines included.
• measured:
  • a new test shows an agent's own test file that starts with the same 6 import lines as a new upstream test file (a parenthesized import among them) is sent. It fails on d57acd8: the agent's own test file, with the same imports: sent, false !== true.
  • the same test shows two cases are still refused: the applied upstream file (imports, then its test), and a real 3-line hunk of from price import subtotal plus a def and an assert.
  • the probe's 3-import false-positive case is now SENT.
• inferred: the rule also matches ES-module import … lines, since the pattern is import\s+\S. SWE-bench Verified's tasks are all Python.

Non-blocking items (all done)

• A. Blank context line: in the test patch, a blank context line (a lone space, or an empty line) no longer ends a run, which matches the tree side. measured: the new test fails on d57acd8 (split by a blank context line " ": refused, true !== false), and the probe's split-by-blank case is now refused.
• B. 512-byte boundary: tarText puts a NUL before each 512-byte block whose ustar magic sits at offset 257. Both the stream and the base go through it. measured:
  • the new test (a real ustar stream, with a 512-byte member that ends in the run and has no newline) fails on d57acd8 (a run ending a 512-byte file, then a header: refused, true !== false) and passes on the head;
  • the probe's case is refused through syncTree.
  • inferred: a non-header block whose content happens to hold ustar at offset 257 gets a spurious line break there. At worst, that hides a run crossing that exact point.
• D. Body nit: the body and hidden.ts now say that on stdin, the id and the hints are also exempt when the problem statement holds them.

"Can't see" lists

hidden.ts, the body's hidden.ts entry and "What this can never do" now list these on stdin:

• runs under 3 lines;
• a run of import lines only;
• a run split across files or reordered;
• gold lines (the gold .diff included);
• test names.

The diff-form test patch was taken off them, because it is now caught.

Local results (measured)

All runs used a temp HOME, state, held-out and XDG dirs, with refusing claude/docker/srt stand-ins first on PATH. Their hit log was never created.

• bench.test.js: # pass 26, # fail 0.
• Fail-without at d57acd8 with only the new test file: # pass 21, # fail 5.
• npm run typecheck: exit 0.
• check:denylist: denylist: 1282 files clean.
• check:dco -- origin/issue-748-bench-preflight HEAD: DCO: all commits signed off.
• Size: 1,282 lines against the base. The body says so.

#759 was not touched.

[jayden1711]

Evaluator verdict: PASS

Round 4 (owner-granted), final check of head 94c6e71 (fix 90b1bab plus a signed-off merge of main 0b294b5), against base main. Labels: measured = a command I ran or CI output I cite; inferred = from reading the code, not run; guess = judgement.

All my runs used a throwaway HOME, state, held-out and XDG dirs, with refusing claude/docker/srt stand-ins first on PATH. Their hit log was never created (measured). No model, Docker or download was used.

1. The two owner fixes hold

Probe: a Django-shaped test patch with two hunks in an existing file (a parenthesised import block, then a new test whose 2nd line is from django.db.models.functions import Coalesce) and a new test file opening with import x, import os, \ + sys, from a import b, from a import ( ... ). Each case goes through TaskRunner with an in-process stand-in docker, on the head and on round 3's d57acd8 (measured).

Case	Want	Head 94c6e71	d57acd8
test patch on stdin of exec -i ... git apply -, LF	refuse	refused	sent
same, CRLF	refuse	refused	sent
test patch as a .diff file in the tar, LF / CRLF	refuse	refused / refused	sent / sent
the new test file applied (tar)	refuse	refused	refused
hunk 2 applied: def, from ... import, code (tar)	refuse	refused	refused
exactly 3 test-patch lines, one an import, applied (tar)	refuse	refused	refused
the same 3 lines as a diff on stdin, LF / CRLF	refuse	refused / refused	sent / sent
agent's own test file with the same import block (import, \ continuation, parens)	send	sent	refused
the import block alone, plain and +-prefixed on stdin	send	sent / sent	refused / sent
hunk 1 applied (paren import block, import unittest)	send	sent	refused
the base tree itself	send	sent	sent

Every stdin refusal is the single STDIN_REFUSED text, and nothing reached the stand-in (measured, the probe checks both).

Evasions (measured on the head, all sent):

• two leading + (a diff of a tree that holds the .diff), on stdin and in the tar;
• a space or a tab before the +;
• the lookalikes U+FF0B, U+2795 and U+FE62 in place of +;
• CR-only line ends;
• > -quoted lines.

Trailing spaces after each line, and a -U0 diff, are refused.

Which matter:

• Only ++ is a shape a tool produces: git diff of a tree that already holds the test-patch .diff (inferred). That tree itself is refused at syncTree, in single-+ form, before any diff of it exists, if the driver syncs it. It matters only if a later driver path (e.g. the test relay) sends an agent-phase diff on stdin without syncing the tree first (inferred; nothing in bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 does so today: its only TaskRunner user hands the runner to the stubbed runTask, scripts/bench/driver.ts:235).
• The others aren't produced by any driver step. They would need a deliberate transformation by something that already holds the test patch, and the first wall (taskView) keeps it from the agent (inferred). Not required.

2. The non-blocking extras don't weaken anything that matters

• Blank context lines: testPatchRuns now skips them instead of flushing (inferred from the code). This only adds windows that span a blank context line; every window of the old code is still produced, except the import-only ones the owner exempted. The probe shows no case refused on d57acd8 and sent on the head apart from the intended import-only ones (measured, table above).
• NUL before ustar headers: at a real header it only breaks the stream where one member's content already ends. A crafted plain stdin holding ustar at byte 512+257 splits the line crossing byte 512 and hides a 3-line run: sent on the head, refused on d57acd8 (measured, probe-nul.mjs). The builder already disclosed this case as inferred. It needs ustar at an exact offset, which is not a driver-bug shape (inferred). A header-checksum check in tarText would close it (non-blocking).

3. Fail-without (measured)

Scratch worktree at d57acd8. Only test/bench.test.ts was taken from 94c6e71, and no guard was edited. I built with tsc -p tsconfig.json, then ran node --import ./dist/test/isolate.js --test dist/test/bench.test.js: # pass 21, # fail 5.

• .diff LF and CRLF: expected: false, actual: true.
• the import-only test: expected: true, actual: false.
• the blank context line: expected: false, actual: true.
• the 512-byte member: expected: false, actual: true.

On the head: # pass 26, # fail 0. These match the builder's numbers and messages.

4. The rest

• Merge commit: the tree of 94c6e71 (bdf53849) equals git merge-tree --write-tree 90b1bab1 0b294b53 (bdf53849) (measured).

• DCO: npm run check:dco -- 0b294b53 94c6e71d: DCO: all commits signed off. All 8 commits carry Signed-off-by (measured).

• Attribution:

  • no Co-Authored-By or tool line in any commit (measured, git log grep);
  • none in the PR body (measured);
  • the body has no raw control or format characters, only U+00B7 (middle dot) and U+2026 (ellipsis) (measured).

• Typecheck and denylist: npm run typecheck exits 0, and check:denylist gives denylist: 1283 files clean (measured).

• Scope: 6 files, all scripts/bench/* and test/bench.test.ts; git diff --stat origin/main...94c6e71d gives 1,282 insertions and 6 deletions (measured). No dependency, src/ or .claude/ change.

• Protected marking: all present (measured, gh pr view):

  • the title has [PROTECTED: owner merges];
  • the body has **Risk class:** protected;
  • the needs-owner label is set;
  • the body has real "What this can never do" and "Compromised component" sections.

  "Compromised component" now names widening the import-line rule as a push-access risk (measured, body).

• No regression for existing installs: the only change to merged code is auth.ts, which only removes more variables from agent and probe environments (measured, diff). No route reopens.

• Claims:

  • every claim in the body and the round-4 comment is labelled;
  • the ones I re-ran hold (measured): 26/26; 21/5 with those messages; denylist 1283; DCO; the .diff, CRLF, import, blank-context and 512-byte results; and "a run of imports and code still counts".
  • Nit: the body's split of 846 lines in modules and 436 in tests counts the 3 deleted test lines. The insertions are 849 and 433 (measured, --stat). The total of 1,282 is right.

• CI on 94c6e71: finished and green (measured, gh pr checks 757, run 39199014136 headSha 94c6e71, conclusion success). All test shards on 22 and 24, dco, denylist, scope, scan, new-deps, agentshield, push-gate and desktop pass. The real-systemd jobs are skipped, as on any scripts-only PR.

Required changes

None.

Non-blocking (for the owner's merge note)

1. For exact claims, add to the stdin "can't see" lists (body, hidden.ts header and "What this can never do"):

   • a diff of the diff (++ lines);
   • CR-only line ends;
   • a stray ustar at offset 257 of a content block.

   All three are measured above.

2. If a later driver path sends agent-phase diffs on stdin, either sync the tree first or match with up to two leading + dropped (inferred).