[jayden1711]

Intent: pin exactly what the runners start before the runner seam (#701) and the executor seam (#702 step 5) move how they start it. For fixed requests, a new test records every process CliRunner and CliAgentRunner start: claude, sudo, codex and gemini. For each one it compares four things with values recorded on main: the argv (--settings and --json-schema parsed), the whole environment, the working directory and the first line on stdin. A refactor that changes any of them fails here. This is slice C3 of the #701 plan, and it adds only a test file.

Risk class: logic. It changes test/ only. It touches no src/**, scripts/**, .github/**, .worklane/** or guard file (measured: git diff --stat origin/main...HEAD lists only test/runner-golden.test.ts). It isn't locked yet. The #701 plan's L0 slice (Lane A, after #839) adds test/runner*.test.ts to the self-modification locks, so a later tests-only PR can't weaken this file at L0.

Part of #701

What it pins (six cases)

Case	Path
claude, this user, cli runtime: instance guardrails, a lane sandbox, and every request field the command line or environment reads (effort, tools, system prompt file, JSON schema, resume, task file, state dir, allow-once host, project variable)	spawn as this user
claude, this user, api_key runtime with a key file: the key reaches claude only as apiKeyHelper	spawn as this user
claude on a hosted provider: ANTHROPIC_BASE_URL, its key helper, an empty config dir, no --effort, no auth probe	spawn as this user
claude as the agent OS user: the auth probe and the run	sudo -n -u … -- /usr/bin/env -i …; the env -i list is pinned in order; sudo itself gets {PATH} only
codex as the agent OS user	the same sudo path
gemini as the agent OS user	the same sudo path

The engine environment the cases start from holds a GitHub token, an API key, an auth token, a proxy, the SSH agent, an AWS secret and an engine state-dir variable, all fake values. The golden values show none of them reaching any process (measured: the recorded environments).

Order is pinned for argv and for sudo's env -i list. Object key order is not pinned: --settings JSON and a spawn's environment are both read by name (inferred).

How it was verified

• On main (3e193c7, then merged with 2ce7726, which changed only CHANGELOG.md): node --import ./dist/test/isolate.js --test dist/test/runner-golden.test.js gives # pass 6, # fail 0. It passed on each of three runs: two before a fixture rename for the denylist, and one after (measured).
• Fail-without. A golden test of current behaviour passes on main by design. So the check is that it catches a change. I made two scratch copies of the built dist/ outside the repo (in the session scratchpad), edited one line of dist/src/runner.js in each, and ran the same test from each copy (measured):
  • GITHUB_TOKEN added to agentEnv's PASS_THROUGH: # pass 0, # fail 6. Every case fails, showing the token in the environment or in sudo's env -i list.
  • --strict-mcp-config dropped from cliArgs: # pass 2, # fail 4. The four claude cases fail; codex and gemini don't use that flag.
• The two open PRs that touch the runner keep it green (measured). I put this file into worktrees at each PR's head, ran npm ci, built, and ran it: Engine: security reads of the environment see only variables that are set, never Object.prototype's (#786) [PROTECTED: owner merges] #844 at 045d62a gives # pass 6, and Wizard: the Claude step matches the login to agent_runtime and offers an API key (#738) [PROTECTED: owner merges] #819 at a39945b gives # pass 6. So both keep their "no output change" claims for these cases.
• npm run typecheck: clean (measured).
• npm run check:denylist: "1320 files clean" (measured, after renaming two fixtures the first run flagged).
• npm run check:dco -- origin/main HEAD: "all commits signed off" (measured).
• test/machine-path-writes.test.ts (Tests: stateDir() refuses the real state dir in a test run, like slotsDir() (#806 follow-up) #815's lint over test files): # pass 9 with this file in the tree (measured).
• test/changelog.test.ts: # pass 10 (measured, before the main merge).
• Full suite: the local broad run (fullsuite.sh, which excludes mcp-* and plugin-fetch) was still running at hand-back. CI runs the whole suite.

How it stays off the host

• The four stand-ins (claude, sudo, codex, gemini) are one node script with an absolute process.execPath shebang. They record and answer, and exec nothing. sudo's stand-in reads which program it would start and answers as that program.
• Their directory is the test process's whole PATH, set at the top of the file, and every run's.
  • killTreeAs calls sudo with the process's own PATH, so it also reaches only the stand-in (inferred: from src/os/index.ts).
  • No run reaches /usr/bin or /bin by PATH.
• Known gap: a program started by absolute path would not be caught. None is in these paths today, apart from /usr/bin/env and /bin/kill, which appear only as arguments to sudo (inferred: from reading asUser and killTreeAs).
• Everything else runs under test/isolate.ts: a temp HOME, state dir and system root.

Adversarial pass

1. Does a one-line env change slip through? No: 6 of 6 cases fail (measured, above).
2. Does a one-line argv change slip through? No: all 4 claude cases fail (measured, above).
3. Could the normalisation hide a real change? It replaces only these: the case dir, the stand-in dir, the test's temp dir, the test process's cwd, a 6-character mkdtemp suffix after <tmp>/name-, the fresh settings check value, and the product name and env prefix.
   • A file moved to another directory still shows (inferred).
   • The auth probe's working directory is pinned as <test-cwd>, so a refactor that started the probe in the worktree would fail (inferred).
   • Hardcoding the product name instead of BRAND would not show here. test/brand.test.ts covers that class.
4. Could a second process slip in? One started by name on PATH would add a record and fail the comparison (inferred). One started by absolute path would not (the known gap above).
5. Flakiness: the first case took 77 ms (measured: duration_ms in the TAP output). No timer matters: the stall and timeout caps are 60 s. It passed on all three runs (measured).
6. Real state: nothing reads or writes outside the test machine. The provider-hosts file comes from allowProviderHosts, which writes into the test's temp dir (measured: test/helpers.ts:80-86).

New dependencies

None.

Package size

Delta against main: 0 bytes. npm pack --dry-run --json lists neither test/runner-golden.test.ts nor CHANGELOG.md (measured: a grep of its output found neither). The package's files are dist/src, dist/web, templates, examples and notices (package.json:17-30). Per file: test/runner-golden.test.ts is +361 lines, not packaged.

CHANGELOG

No line. Main's v0.5.3 section went back to unreleased in #850 and now holds only hand-written upgrade and guardrail notes. The release generates the PR lists, so this PR is listed from its title. A test-only change has no upgrade or guardrail note to add. My first commit had a bullet; the second removes it, so the branch's net diff touches test/ only.

[jayden1711]

Evaluator verdict: PASS

Final review of head 189c96431072c1e238d843f3456f1a0fdbe0fac5 (test/runner-golden.test.ts only; part of #701, slice C3).

Host tools (checked before running)

• The test sets process.env.PATH = bin (line 70) and PATH: bin in every run's base env (line 86). bin is a fresh mkdtemp that holds only the four stand-ins: claude, sudo, codex and gemini. It contains no /usr/bin or /bin (measured: from reading the file).
• Each stand-in is one node script with an absolute process.execPath shebang. It parses argv, appends to a log, prints canned JSON and exits. It has no spawn, exec or require('child_process'). The sudo stand-in only works out the program name. It never starts it (measured: from reading lines 29-64).
• No runner dependency spawns anything by absolute path. Only src/os/index.ts spawns, and it does so by name: sudo, perl, getent and id. With this PATH those names reach a stand-in or nothing (inferred: from a grep of spawn and exec across runner.ts, cli-agents.ts and their 16 imported modules). /usr/bin/env and /bin/kill appear only as arguments to sudo. That is the body's "known gap", and I agree no absolute-path spawn is on these paths today (inferred).

Is the golden data pinned (not vacuous)?

• assert.deepEqual (strict) runs on the whole record array. That pins the number of processes, argv order, the whole environment (not a subset), cwd and the stdin line (measured: lines 176-181).
• The normaliser replaces only these: the case dir, the bin dir, tmpdir, the 6-character mkdtemp suffix after <tmp>/name-, the settings-check token, the test cwd, and the brand prefix and CLI name. It never touches a secret value (measured: lines 154-169).
• Mutation 4 below shows that stdin is compared.

Mutations (on scratch copies of dist/ in the session scratchpad; no repo file edited)

Mutation	Result (measured)
Control, unmutated copy	6 pass / 0 fail
GITHUB_TOKEN added to PASS_THROUGH	0 pass / 6 fail (matches the builder's claim)
--strict-mcp-config dropped from claude argv	2 pass / 4 fail (matches; codex and gemini pass)
Mine: SSH_AUTH_SOCK added in runAsEnv (sudo path only)	3 pass / 3 fail (exactly the 3 sudo cases)
Mine: prompt written to stdin changed by one trailing space	2 pass / 4 fail (claude cases; the diff shows content":"do the task ")

Other checks

• Local run on the head: node --import ./dist/test/isolate.js --test dist/test/runner-golden.test.js gives # pass 6, # fail 0 (measured). The head's src is identical to origin/main (git diff --stat origin/main HEAD lists only the test file), so this is the "passes on main by design" result (measured).
• CI: all checks green. The golden test ran on Node 22 (shard 4/4, ok 239-ok 244) and on Node 24 (shard 4/4, all three sudo cases shown passing). That covers both Node versions and CI's shard environment (measured: CI logs of run 39353640186).
• Brittleness: the test passes from a different cwd (a scratch copy outside the repo). It reads nothing from the test process's environment: the base env is explicit and sudo gets {PATH} only. Node's default per-file process isolation keeps PATH = bin from leaking into other files (measured: control run; inferred: from the code and ci.yml line 106).
• Real state: every write goes under tmpdir(), which is the isolated test machine. The base env's HOME is the case dir and <PREFIX>_STATE_DIR is a fake path that is filtered out. req.stateDir is a temp dir (inferred: from the code). test/machine-path-writes.test.ts gives # pass 9 with this file in the tree (measured).
• Durability: no change to src/. The test adds no parsing of Claude Code output to src/ (measured: diff).
• npm run typecheck: clean. npm run check:denylist: "1320 files clean" (measured).
• Size delta: 0. Only test/ changed, and the package's files is dist/src, dist/web and the like, not dist/test (inferred: from the files field I read).
• Protection: no protected path touched. Risk class "logic" is correct (measured: diff).
• Commits: all 3 are signed off and carry no attribution trailers (measured). The PR body labels its claims and has an Adversarial pass and a Package size section.

Notes (not blocking)

1. Stdin is pinned only up to its first line (STDIN_LINE). A refactor that adds extra stdin lines after the prompt would not show (inferred). This is acceptable: claude's stdin stays open by design for console messages.
2. A new spawn by absolute path would not be recorded. The body already names this known gap.
3. The file isn't locked yet. The Runner: agent backends declare the guard guarantees they provide, and the coordinator refuses a backend that lacks one a route needs (Claude CLI first) #701 L0 slice adds test/runner*.test.ts to the self-modification locks, as the body says.