[jayden1711]

Intent: This is the first of three stacked PRs for #748 Phase 0, the gated-merge benchmark on SWE-bench Verified. It adds the bench's preflight under scripts/bench/:

• Bench user: the bench runs only as its own OS user (default wl-bench), with its home taken from the user database (the passwd entry), never from $HOME.
• Login: a claude.ai login only (any plan, which is recorded), on Anthropic's own API, in a config dir inside the bench user's home. There is no API key and nothing reads the login's files.
• CLI: pinned (2.1.303), with DISABLE_UPDATES=1 on every claude the bench starts.
• Caps: the owner's approval in usage-equivalent dollars: --spend required, $40 per smoke round, $300 baseline, $15 per task, and re-approval over $8 per task.
• Usage meter: records the CLI's own total_cost_usd and tokens, and stops cleanly at the first usage or rate limit of any kind, with no retry.

Nothing here runs a model, Docker or a download.

Risk class: protected (every path is under scripts/**, the L3 review point the issue names).

Part of #748. Stacked: this PR (preflight) → #757 (containers and limits) → #759 (driver, grading, aggregates, README).

What's in it

• scripts/bench/paths.ts:
  • benchHome: the bench user's home from os.userInfo().homedir (the passwd entry). When $HOME is set and isn't that home (compared on real paths), it refuses; it doesn't silently ignore $HOME, because claude itself follows $HOME to its login.
  • The bench root defaults to ~/bench in that home; WORKLANE_BENCH_ROOT overrides it. It is resolved with realpath (the longest existing prefix), and so is each place it is checked against.
  • A root with a path component that exists but doesn't resolve (a dangling link, or a link loop) is refused. The places it is checked against follow such a link through readlink, so a root where a dangling ~/.ssh points is refused too.
  • It is refused inside a git checkout, a credential store (~/.claude, ~/.ssh, ...), the real engine state dir, or a held-out store the environment names. A ~/bench that is a symlink into one of those is refused too, whether or not the link resolves.
• scripts/bench/auth.ts:
  • requireBenchUser;
  • requireSubscription:
    • accepts only authMethod claude.ai with apiProvider firstParty, and a configDirectory inside the bench user's home (recorded as config_directory, with the plan). That home is benchHome's, never $HOME;
    • refuses oauth_token, api_key, api_key_helper, third_party, signed-out, and a status that isn't a JSON object;
    • refuses, before claude is even asked, any variable matching LOGIN_SWITCH_VAR (a prefix rule, not a list: next section), and a $HOME that isn't the bench user's home;
  • requireClaudePin;
  • BenchAgentRunner, which adds DISABLE_UPDATES=1 to every run.
• scripts/bench/caps.ts: capsFromArgs, built on the existing spend gate (src/spend-gate.ts). The mode is checked with Object.hasOwn (isMode), so constructor, toString and __proto__ are refused.
• scripts/bench/usage.ts: UsageMeter and TaskMeter.
  • The meter stops the bench on any non-success result the engine's classifyTransient calls a rate limit. That covers rate_limited and a failed run on a 429 with no result line.
  • It also hooks onTransientRetry. If a runner starts to wait out a transient error, the meter aborts the wait: a rate limit stops the bench, and anything else ends that run.
  • The bench's own runner is built with retries off in bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757 (benchClaude, budgetMs: 0).
• tsconfig.json: compiles scripts/bench/**/*.ts. The output is dist/scripts/bench/, which the package's files list doesn't include (below).
• test/bench.test.ts: ten tests. The stand-in claude answers with the real CLI's status shapes (next section), the usage-limit result (rate_limit_event rejected, then an is_error result), and an exit 1 on API Error: 429 with no result line.

The login variables (checked against code.claude.com/docs/en/env-vars, 2026-10-18)

LOGIN_SWITCH_VAR refuses these:

• every ANTHROPIC_*. That covers keys, tokens, base URLs, federation (ANTHROPIC_FEDERATION_RULE_ID + ANTHROPIC_ORGANIZATION_ID), ANTHROPIC_PROFILE, ANTHROPIC_CUSTOM_HEADERS (an Authorization header) and the model overrides. None is exempt: the bench needs none, and a model override would change what is benchmarked (inferred);
• CLAUDE_CODE_USE_*, CLAUDE_CODE_OAUTH_*, CLAUDE_CODE_SKIP_*_AUTH, CLAUDE_CODE_API_KEY*, CLAUDE_CODE_PROVIDER_* (PROVIDER_MANAGED_BY_HOST) and CLAUDE_CODE_CLIENT_* (a gateway's mTLS identity);
• any *_FILE_DESCRIPTOR (CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR and CLAUDE_CODE_API_KEY_FILE_DESCRIPTOR, which the reviewer found in the binary; they aren't on the docs page) and any *_BASE_URL;
• AWS_*, AZURE_*, GOOGLE_APPLICATION_CREDENTIALS, CLOUD_ML_*, and any name containing BEDROCK, VERTEX, FOUNDRY or MANTLE.

measured: every variable the reviewer listed matches the rule (the test asserts each one, plus a made-up ANTHROPIC_A_VARIABLE_A_LATER_CLI_ADDS and an empty ANTHROPIC_BASE_URL). Unrelated variables such as DISABLE_TELEMETRY and CLAUDE_CODE_MAX_OUTPUT_TOKENS still pass.

The real status shapes

measured: the real 2.1.303 claude auth status --json, run with a throwaway HOME and CLAUDE_CONFIG_DIR, no login and no model call:

Variable set	exit	loggedIn	authMethod	apiProvider	other fields
none	1	false	none	firstParty	analyticsDisabled, projectsDirectory, configDirectory
ANTHROPIC_FEDERATION_RULE_ID + ANTHROPIC_ORGANIZATION_ID	0	true	oauth_token	firstParty	the same
CLAUDE_CODE_OAUTH_TOKEN	0	true	oauth_token	firstParty	the same
ANTHROPIC_API_KEY	0	true	api_key	firstParty	plus apiKeySource
CLAUDE_CODE_USE_BEDROCK	0	true	third_party	bedrock	the same
ANTHROPIC_BASE_URL or AWS_BEARER_TOKEN_BEDROCK alone	1	false	none	firstParty	the same

• The tests use these shapes.
• The claude.ai shape adds email, orgId, orgName and subscriptionType to the same fields. That is inferred from the CLI's status code (the reviewer's static read), not run, since no real login is used.
• The old synthetic claude.ai + bedrock shape is replaced. A claude.ai + vertex case remains for the provider check, labelled synthetic in the test.

Still ahead in #748 Phase 0, after this stack: the PR-mode task runner on src/harness-evals/replay.ts with stub CI, the agent's test relay into the container, task selection, the pinned-download scripts, the grading run, the B0, B1 and claude -p baselines, and the CLI entry point.

How it was verified

• measured (round 3, head 571f3c0, main merged in at 3519839): node --import ./dist/test/isolate.js --test dist/test/bench.test.js with a temp HOME, state and held-out dirs and a refusing claude, docker and srt first on PATH: # pass 10, # fail 0.
• measured: npm run typecheck (both tsconfigs) exits 0. npm run check:denylist: denylist: 1278 files clean. npm run check:dco -- origin/main HEAD: DCO: all commits signed off.
• measured (round 0): npm pack --dry-run --ignore-scripts lists no file under scripts/bench or dist/scripts.
• Fail-without, round 3 (measured):
  • Method: a scratch worktree at 571f3c0 with scripts/bench/auth.ts and paths.ts checked out from its parent (the code before this round). No guard was edited.
  • The new tests on the old code: # pass 8, # fail 2: not ok 3 (the $HOME test: the first assertion got the old config-dir message, not a HOME refusal) and not ok 10, Missing expected exception: ~/bench -> ~/.claude/not-yet (dangling).
  • A probe of each case against both builds (a stand-in claude that reports a claude.ai login in $HOME/.claude). Old code, then new:
    • requireSubscription with no home given and HOME at a folder holding a claude.ai login that the user database doesn't name: ACCEPTED, then REFUSED (HOME is …, not this user's home … from the user database);
    • benchRoot with HOME at another folder, home given, and with no home given: both ACCEPTED, then both REFUSED;
    • benchRoot with ~/bench → ~/.claude/not-yet (dangling), a root under a dangling link, and a root under a link loop: all ACCEPTED, then all REFUSED (… is a link that doesn't resolve);
    • a root where a dangling ~/.ssh points: ACCEPTED, then REFUSED (overlaps);
    • controls (HOME equal to the bench home, a plain root): ACCEPTED on both.
• Fail-without, round 1 (measured):
  • Method: commit f122bfd's tests were run against the pre-fix modules (git checkout 0a45e717 -- scripts/bench/{auth,caps,paths,usage}.ts, the PR's own commit before these guards existed; then restored with git checkout HEAD --). No guard was edited.
  • Result: # pass 4, # fail 4.
  • Caps: Missing expected exception: __proto__.
  • Login: Missing expected exception: an oauth_token (CLAUDE_CODE_OAUTH_TOKEN or federation: the real shape).
  • 429: retries on: exactly one claude session, expected: 1, actual: 8.
  • Symlinked root: Missing expected exception: ~/bench is a link to ~/.claude.
  • measured by the round-2 reviewer: the "retries off" half of the 429 test also fails without the meter change.
• Fail-without, round 0:
  • Spend cap (measured by the builder): on the combined branch (commit 9d73baa4, all three PRs' code), with the driver's cap check switched off, acceptance: the driver refuses to start without a spend cap failed: The input did not match the regular expression /spend cap|over the owner's approval/ ... "TypeError: Cannot read properties of undefined (reading 'spendUsd')".
  • Owner's runs (measured by the owner, 2026-10-18): on 9d73baa4 in a scratch worktree with a temp HOME and state dirs. The usage-limit stop (driver.ts:153 off) gave not ok "the driver stops cleanly at a usage limit". The subscription-only login (auth.ts:107 off) gave not ok "the driver refuses a login that is not a subscription".

New dependencies

None.

What this can never do

• Start a bench run on anything but a claude.ai login. The check reads claude auth status --json and refuses:

  • any authMethod other than claude.ai (an oauth_token, which also covers gateway bearers, federation and Console profiles, included);
  • any apiProvider other than firstParty;
  • a config dir outside the bench user's home, where that home is the passwd entry's (measured, round-3 probe);
  • a $HOME that isn't that home, before claude is asked (measured);
  • any variable matching the prefix rule above, before claude is asked.

  inferred limit: a user-level settings env block is not read by this check. For one that sets a token or key, the CLI's own status shows the changed authMethod, and that is refused. A settings-level ANTHROPIC_BASE_URL beside a claude.ai login is not detected here.

• Put the bench root in a credential store, the engine's state, a held-out store or a checkout, including through a symlink, a chain of symlinks, a dangling link or a link loop (measured, tests 9 and 10). inferred limit: only the root is checked; a link planted inside an existing root later (~/bench/state → somewhere) is not, so the sandbox in bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757/bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 must keep agents from writing to ~/bench.

• Read, copy or relay the login. The check asks the CLI and never opens a file in ~/.claude.

• Retry after a usage or rate limit. The meter stops the whole bench on any result the engine classes as a rate limit, a failed 429 included. It aborts any transient wait a runner starts, so a second session never starts. Every later run is refused unstarted.

• Spend past the owner's caps. --spend is required. The per-mode maxima and the $15 per task are constants a reviewed change must raise, and only smoke and baseline (own keys) are modes.

Compromised component

• Agent user, coordinator, dashboard, other instances, hub viewer: none of them runs or reaches this. It is a script the bench user runs by hand, and nothing in src/ imports it.
• The bench user's own shell (anyone who can run as wl-bench): they could run claude directly anyway, so this code gives them nothing new. The caps bound only what the bench itself starts. They can also set WORKLANE_BENCH_CLAUDE_VERSION and WORKLANE_BENCH_USER, which change the pin and the expected user without a reviewed change (inferred; kept for the owner's later pin bumps). Moving $HOME is refused, not honoured (measured).
• An agent during a bench run (the next PRs): the meter is in the driver's process, not the agent's. An agent can't raise its cap or clear the stop.
• Someone with push access: they could raise a cap constant, widen SUBSCRIPTION_AUTH or narrow LOGIN_SWITCH_VAR. The guard is this review: the owner merges scripts/**.

[jayden1711]

Owner's fail-without results (measured by the owner, 2026-10-18), on 9d73baa4 in a scratch worktree, now removed:

1. Guards on: both tests pass (pass 2, fail 0).
2. driver.ts:153 usage-limit stop off (edit confirmed by grep): not ok "stops cleanly at a usage limit"; pass 0, fail 1.
3. auth.ts:107 subscription check off (edit confirmed by grep): not ok "refuses a login that is not a subscription"; pass 0.

Both guards fail without their check. The PR body is updated.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 1/2)

Reviewed head 0a45e717 against base main (99c7d5b3, the merge base). Only this PR's diff is in scope: scripts/bench/{auth,caps,paths,usage}.ts, test/bench.test.ts and tsconfig.json.

Summary

The structure is sound, and most of it does what the owner asked. Three guards can be bypassed (all measured below):

• the spend cap, with --tasks constructor;
• the subscription-only check, with federation credentials;
• the no-retry stop, when a 429 is one the engine classes as transient.

The PR body's "What this can never do" section claims all three, so they are required changes.

Acceptance criteria

Requirement (issue #748 / the owner's 2026-10-18 decisions, as far as #756 owns it)	Status	Evidence
No run without a spend cap. --spend is required. $40 smoke, $300 baseline, $15 per task, re-approval over $8 mean	Partly met	The numbers match the owner's (caps.ts:11-15), and test 1 covers each refusal (measured: # pass 6). But capsFromArgs(['--tasks','toString','--spend','1000000']) returns {"mode":"toString","spendUsd":1000000,"perTaskUsd":15}. The same happens with constructor and __proto__ (measured, probe below). Cause: caps.ts:36 mode in MODE_MAX_USD matches Object.prototype keys, and line 43 then compares against a function (NaN), so it never throws (inferred from the code; the probe confirms the result). #759's reviewer found the same bug independently.
Subscription login only (claude.ai or oauth_token, any plan, plan recorded). API-key, helper, Bedrock, Vertex and Foundry logins refused, and so are the variables that switch the login	Partly met	api_key, api_key_helper, signed-out, {unexpected:true} and a non-first-party apiProvider are refused, and the enterprise shape is accepted. Each of these is tested and passes (measured). But federation credentials are accepted: see required change 2 (measured against the real 2.1.303 CLI).
CLI pin 2.1.303, claude --version checked, DISABLE_UPDATES=1 on every claude call	Met	Test 3 asserts the pin and DISABLE_UPDATES=1 on each of version, the bench's auth check, the runner's own auth check, and the run (measured: passes). DISABLE_UPDATES is a real CLI variable (measured, code.claude.com/docs/en/env-vars: "Set to 1 to block all updates including manual claude update and claude install. Stricter than DISABLE_AUTOUPDATER").
Usage from the CLI's own total_cost_usd	Met	src/runner.ts:847 reads r?.total_cost_usd, and tests 4 and 5 assert 0.42 and 1.25 recorded with tokens (measured: pass).
Usage or rate limit: clean stop, no retry, later runs not started	Partly met	A rate_limited result stops the meter and refuses later runs (test 4, measured). But a 429 the engine classes as transient is retried inside the meter: see required change 3 (measured).
Bench root outside the repo, real state and credential stores	Met, with notes	paths.ts refuses: an overlap with .claude, .claude.json, .config, .ssh, .aws, .gnupg, .docker or .netrc; the engine's state dir; the held-out env store; any ancestor holding .git; and a relative path (inferred from the code). Only .claude, relative and git are tested (measured). Notes below.
npm run typecheck	Met	Exit 0 on the head (measured).
test/bench.test.ts	Met	# tests 6 # pass 6 # fail 0 on the head (measured).

Auth field names (as asked)

• authMethod, measured (docs): code.claude.com/docs/en/cli-reference, claude auth status: "The JSON's authMethod field is one of none, claude.ai, oauth_token, api_key, api_key_helper, or third_party." The docs also name configDirectory. They don't document apiProvider or subscriptionType.
• apiProvider and subscriptionType, measured (static read of the installed 2.1.303 binary, no execution): the status object is built as {loggedIn, authMethod, apiProvider, analyticsDisabled, projectsDirectory, configDirectory}. email, orgId, orgName and subscriptionType are added only when authMethod is claude.ai. So for an oauth_token login the recorded plan is always null (inferred from that code).
• What the real CLI reports, measured. I ran claude auth status --json (2.1.303) under a throwaway HOME and CLAUDE_CONFIG_DIR, with no login, no model call, and updates and nonessential traffic off. One variable was set at a time:

Variable set	loggedIn	authMethod	apiProvider
(none)	false	none	firstParty
ANTHROPIC_AUTH_TOKEN	true	oauth_token	firstParty
ANTHROPIC_FEDERATION_RULE_ID + ANTHROPIC_ORGANIZATION_ID	true	oauth_token	firstParty
CLAUDE_CODE_OAUTH_TOKEN	true	oauth_token	firstParty
ANTHROPIC_API_KEY	true	api_key (plus apiKeySource)	firstParty
CLAUDE_CODE_USE_BEDROCK	true	third_party	bedrock
CLAUDE_CODE_USE_ANTHROPIC_AWS	true	third_party	anthropicAws
CLAUDE_CODE_USE_MANTLE	true	third_party	mantle

So oauth_token is not only a subscription. Per the binary's token-source code (measured static read, mapping inferred), it also covers:

• an ANTHROPIC_AUTH_TOKEN gateway bearer;
• Workload Identity Federation and ant profiles (ANTHROPIC_PROFILE), which are Console (API-billed) credentials;
• CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR.

The real Bedrock shape is third_party/bedrock. The test's {authMethod:'claude.ai', apiProvider:'bedrock'} is synthetic: it exercises the provider check, but the CLI never emits it (measured above).

Fail-without

• On main: copying test/bench.test.ts onto main and building gives error TS2307: Cannot find module '../scripts/bench/auth.js' (likewise caps, paths, usage) (measured). That's unavoidable: every module is new.
• Spend cap (guard off on the combined branch): measured by the builder, quoted in the PR body. I didn't redo it.
• Usage-limit stop and subscription check: measured by the owner on 9d73baa4 (comment 6257258797). The subscription guard switched off there is auth.ts:107. That's the same check as this PR's auth.ts:70, if (!SUBSCRIPTION_AUTH.includes(method)) (measured with git show 9d73baa4:scripts/bench/auth.ts). I didn't redo it.
• I made no guard-off edit.

Protection

• Protected: every new file is under scripts/**, which is engine in main's .worklane/review.yaml, so this is L3 (measured). tsconfig.json and test/bench.test.ts aren't in any protected list (measured).
• Marking: the PR is correctly marked: the [PROTECTED: owner merges] title, **Risk class:** protected, the needs-owner label, and real "What this can never do" and "Compromised component" sections (measured).
• "Can never do" claims checked against the code:
  • "Read, copy or relay the login": true. The code only spawns claude auth status --json and claude --version (inferred).
  • "Start a bench run with an API key, an auth helper or a cloud provider ... refuses ... any of the variables that would switch claude off the login": false for federation credentials (measured, required change 2).
  • "Retry after a usage or rate limit": false for a 429 the engine classes as transient (measured, required change 3).
  • "Spend past the owner's caps ... constants a reviewed change must raise": false via --tasks constructor/toString (measured, required change 1).
• Regressions: nothing in src/ or package.json references scripts/bench (measured by grep). No existing install's route reopens; the change touches no setup, firewall, sandbox, lock or guard (inferred).
• Owner rule A: the tests don't touch real state. They run under test/isolate.ts; every claude call uses the stand-in's absolute path; benchRoot gets a temp home. I ran the suite with a temp HOME and state dirs and a refusing claude first on PATH, and it passed (measured).

Scope

• tsconfig.json now includes scripts/bench/**. It's needed to compile the modules.
• The package is unaffected: this PR's CI reports an unpacked size of 5818647 bytes, the same as main's run at the base (39179509850) (measured).
• No other scope creep (inferred).

Hygiene

• One commit, Signed-off-by present, no Co-Authored-By or other attribution trailer (measured, gh pr view --json commits).
• No new dependencies (measured: the new-deps check passes and package.json is untouched).
• No .claude/settings.json; nothing in src/ changed.
• Denylist: denylist: 1271 files clean (measured).
• Claims in the body are labelled.
• The body says "Part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748" and has no Closes, which is right for the first of a stack: bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 should close it (inferred).

CI

Red, and not because of this PR (measured from gh run view 39180502595 --log-failed):

• Shard 1/4 (Node 22 and 24): the package stays within its size budget: 5818647 bytes, over its budget of 5810000. That's identical to main's run 39179509850 at the base.
• Shard 3/4 (Node 22 and 24): focus and selection rings use the ring token: ring-blue is no longer there: drop it from RING_OWED. Same as main.
• The two test (ubuntu-latest, …) aggregates: they only roll up the shards.
• These are known main failures that Main green: package budget for the merged growth since v0.5.2, and drop the stale RING_OWED entry [PROTECTED: owner merges] #753 fixes. No bench test failed.

Required changes

1. Spend cap bypass. In caps.ts:36, replace mode in MODE_MAX_USD with Object.hasOwn(MODE_MAX_USD, mode) or an explicit ['smoke','baseline'] check. Add a fail-without test that --tasks constructor, --tasks toString and --tasks __proto__ with --spend 1000000 all throw. Today all three return a $1,000,000 ceiling (measured). bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759's checkCaps, if it reuses the same check, needs the same fix there.

2. Login-switching variables. Federation credentials (ANTHROPIC_FEDERATION_RULE_ID + ANTHROPIC_ORGANIZATION_ID) pass requireSubscription. Measured against the real 2.1.303 CLI under a throwaway home, it returns {"method":"oauth_token","subscription_type":null}, though that is a Console org credential, not a subscription. OTHER_AUTH_VAR (auth.ts:26) also misses these documented variables:

   • ANTHROPIC_PROFILE, CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR, CLAUDE_CODE_API_KEY_FILE_DESCRIPTOR and CLAUDE_CODE_OAUTH_REFRESH_TOKEN;
   • ANTHROPIC_FOUNDRY_API_KEY / _AUTH_TOKEN, ANTHROPIC_AWS_API_KEY, and the ANTHROPIC_{BEDROCK,BEDROCK_MANTLE,VERTEX,FOUNDRY,AWS}_BASE_URL variables;
   • CLAUDE_CODE_USE_ANTHROPIC_AWS, CLAUDE_CODE_USE_MANTLE, CLAUDE_CODE_SKIP_*_AUTH and CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST.

   The USE_* ones are already refused through third_party (measured), but not by the variable check the body claims.

   Prefer a rule the CLI can't outgrow over a list: refuse every ANTHROPIC_*, every CLAUDE_CODE_USE_*, CLAUDE_CODE_SKIP_*_AUTH, CLAUDE_CODE_OAUTH_*, *_FILE_DESCRIPTOR and AWS_BEARER_TOKEN_BEDROCK.

   Add tests for:

   • the federation pair;
   • each listed variable that is currently untested (ANTHROPIC_BASE_URL, CLAUDE_CODE_OAUTH_TOKEN, AWS_BEARER_TOKEN_BEDROCK);
   • an unreadable (non-JSON) status.

3. No-retry stop. A rate limit the engine classes as transient is retried inside the meter. A stand-in claude exits 1 with no result line and API Error: 429 {"type":"error","error":{"type":"rate_limit_error",...}} on stderr. Through UsageMeter(new BenchAgentRunner(new CliRunner('cli', …))), that gave reason=failed transient=true, 8 claude sessions started, meter.limited=null (measured, with fake time). The cause: CliRunner.run retries failed results whose detail matches TRANSIENT_ERROR (rate.?limit|\b429\b) within its 30-minute budget, and only rate_limited is exempt (src/runner.ts:562-566). bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757's benchClaude builds the CliRunner with default retry options (measured at 9d73baa4), so the combined code has the gap too.

   Make retrying impossible for the bench: build the bench's CliRunner with retries off (RetryOptions.budgetMs: 0), or have the meter stop on any result classifyTransient calls rate_limit. Add a fail-without test with this stand-in.

Not required (for the owner)

• oauth_token (guess on intent; facts measured): this login type also covers gateway bearers, federation and ant profiles. The CLI reports no plan for it, so "the plan is recorded" can't hold for an oauth_token login. The owner may want claude.ai only. A user-level settings env block can also set ANTHROPIC_AUTH_TOKEN and similar where the process-env check can't see them (inferred). Accepting only claude.ai closes that too.
• Login check (suggestion): record configDirectory from the status (documented since 2.1.268), and refuse it unless it is the bench user's own ~/.claude. That catches a CLAUDE_CONFIG_DIR switch to another login (inferred).
• Environment overrides (inferred): WORKLANE_BENCH_CLAUDE_VERSION and WORKLANE_BENCH_USER let the environment change the pin and the bench user without a reviewed change. Unlike the caps, these aren't constants. That may be intended.
• paths.ts (inferred):
  • resolve() doesn't follow symlinks, so a ~/bench symlinked into ~/.claude or a checkout would pass; realpath on the existing prefix closes that.
  • The real-state and held-out overlaps are untested.

[jayden1711]

Round 1 fixes (head f122bfd; main merged at 858f627)

Each point from the verdict and the owner's decisions of 2026-10-18. Labels: measured = a command I ran, inferred = reasoned, not run.

1. Spend cap: prototype keys.
   • capsFromArgs checks the mode with Object.hasOwn (isMode in caps.ts).
   • The test refuses __proto__, constructor, toString, hasOwnProperty and valueOf with --spend 1000000.
   • measured fail-without: Missing expected exception: __proto__.
2. Login: claude.ai only.
   • Only authMethod: "claude.ai" with apiProvider: "firstParty" is accepted. oauth_token is refused, and so are api_key, api_key_helper, third_party, signed-out and a non-JSON status.
   • configDirectory is recorded (config_directory) and must be inside the bench user's home after realpath.
   • Login-switching variables are refused by a prefix rule (LOGIN_SWITCH_VAR), before claude is asked:
     • every ANTHROPIC_*, none exempt (justification in the body);
     • CLAUDE_CODE_USE_*, CLAUDE_CODE_OAUTH_*, CLAUDE_CODE_SKIP_*_AUTH, CLAUDE_CODE_API_KEY*, CLAUDE_CODE_PROVIDER_*, CLAUDE_CODE_CLIENT_*;
     • *_FILE_DESCRIPTOR, *_BASE_URL, AWS_*, AZURE_*, GOOGLE_APPLICATION_CREDENTIALS, CLOUD_ML_*, and any name containing BEDROCK, VERTEX, FOUNDRY or MANTLE.
   • measured: every variable the verdict listed is checked against the env-vars docs page and asserted in the test. That includes the federation pair, ANTHROPIC_BASE_URL, CLAUDE_CODE_OAUTH_TOKEN, AWS_BEARER_TOKEN_BEDROCK, an empty ANTHROPIC_BASE_URL and a made-up future ANTHROPIC_*.
   • Real shapes (measured): the real 2.1.303 claude auth status --json, with a throwaway HOME and CLAUDE_CONFIG_DIR, no login and no model call, for none, federation, CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY, CLAUDE_CODE_USE_BEDROCK, ANTHROPIC_BASE_URL and AWS_BEARER_TOKEN_BEDROCK. The full table is in the body, and the test fixtures use these shapes.
   • The claude.ai shape adds email, orgId, orgName and subscriptionType. That is inferred from your static read; I used no real login.
   • The synthetic claude.ai + bedrock shape is gone. One labelled synthetic claude.ai + vertex case remains.
   • measured fail-without: Missing expected exception: an oauth_token (CLAUDE_CODE_OAUTH_TOKEN or federation: the real shape).
3. No retry; stop on any rate limit.
   • The meter stops the bench on any non-success result that classifyTransient calls rate_limit, which includes a failed 429 with no result line.
   • It also sets onTransientRetry on every request. If a runner starts to wait out a transient error, the meter aborts that wait: a rate limit stops the bench, and any other transient error ends that run.
   • The bench's own CliRunner is built in bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757 (benchClaude). It gets budgetMs: 0 there, in that PR's round.
   • Test: your stand-in (exit 1, API Error: 429 {...rate_limit_error...} on stderr, no result line), run twice: once with the engine's default retry on a test clock, and once with budgetMs: 0. Each time it asserts exactly 1 claude session, meter.limited set (detail matching 429), meter.signal aborted, and three later runs (same task, two new tasks) refused as not started.
   • measured fail-without: retries on: exactly one claude session, expected: 1, actual: 8.
   • inferred: the budgetMs: 0 half fails too without the meter change, because the old meter stopped only on rate_limited.
4. Bench root (owner's addition):
   • resolved with realpath on the longest existing prefix, on both sides of every overlap check;
   • a ~/bench linked to ~/.claude is refused, and so is a root under a link into a checkout;
   • a link to a folder of its own is resolved and used;
   • the real-state and held-out overlaps are now tested;
   • measured fail-without: Missing expected exception: ~/bench is a link to ~/.claude.

How the fail-without was run (measured): f122bfd's test file against the PR's own pre-fix modules (git checkout 0a45e717 -- scripts/bench/{auth,caps,paths,usage}.ts, then git checkout HEAD -- to restore). Result: # pass 4, # fail 4, the four failures quoted above. No guard was edited.

Local, on f122bfd (measured): run with a temp HOME and state dirs, and a refusing claude first on PATH:

• node --import ./dist/test/isolate.js --test dist/test/bench.test.js: # pass 8, # fail 0;
• npm run typecheck: exit 0;
• npm run check:denylist: denylist: 1271 files clean;
• npm run check:dco -- origin/main HEAD: DCO: all commits signed off.

Body: "What this can never do" and "Compromised component" now match the code. That includes a new inferred limit: a settings-file ANTHROPIC_BASE_URL beside a claude.ai login is not detected by this check.

CI: running at the time of this comment (dco, denylist, scan, scope, new-deps and shard 2/4 on Node 22 passed; the rest pending). Main includes #753, so any red is this PR's to explain.

[jayden1711]

Evaluator verdict: CHANGES NEEDED (round 2/2)

I reviewed head f122bfd9 against base main. The merge base is b6a95a5c. Main has moved on to b0791732 since, which doesn't affect this review. Only this PR's diff is in scope: scripts/bench/{auth,caps,paths,usage}.ts, test/bench.test.ts and tsconfig.json.

Summary

Every round-1 required change and every owner decision for #756 is met. I checked each one with my own probes against the built head (measured, details below).

Two narrow claims in "What this can never do" are still not true in edge cases (measured):

• the config-dir check follows $HOME;
• a dangling ~/bench link into ~/.claude is accepted.

Neither leads to a non-claude.ai login, and neither lets the bench write into ~/.claude. Each fix is about one line plus a test. The owner may instead take them as corrections to the PR body. They're listed below.

Round-1 required changes and owner decisions

Item	Status	Evidence
Caps: a prototype-safe lookup, with tests for __proto__, constructor and toString	Met	isMode uses Object.hasOwn (caps.ts). My probe ran capsFromArgs(['--tasks', m, '--spend', '1000000']). Every one of these was refused with --tasks takes smoke or baseline, not <m> (measured): __proto__, constructor, toString, hasOwnProperty, valueOf, __defineGetter__, __lookupGetter__, isPrototypeOf, propertyIsEnumerable, toLocaleString and "". --spend values Infinity, NaN, -5 and 0 are refused, and 1e9 is over the $40 cap (measured). The test covers 5 of these names.
claude.ai only (oauth_token refused)	Met	SUBSCRIPTION_AUTH = ['claude.ai'], and apiProvider must be firstParty. These were all refused (measured): oauth_token, a non-JSON status, an empty status, JSON null, a JSON string, a JSON array, loggedIn: "true" (a string), and a {"__proto__":{...claude.ai...}} smuggle. The claude.ai/firstParty shape is accepted, with subscription_type recorded (measured).
Login variables refused by prefix, before claude is asked	Met	The stand-in logs every call it gets. Each of these was refused and the stand-in was never called (measured: claude asked: false): ANTHROPIC_FEDERATION_RULE_ID + ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_BASE_URL (also when set to ""), CLAUDE_CODE_OAUTH_TOKEN, AWS_BEARER_TOKEN_BEDROCK, ANTHROPIC_PROFILE, CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR, CLAUDE_CODE_API_KEY_FILE_DESCRIPTOR, a made-up SOMETHING_ELSE_FILE_DESCRIPTOR, CLAUDE_CODE_USE_FOUNDRY, and a lower-case anthropic_api_key. I also ran all 417 variable names on the env-vars docs page (round 1's copy) through LOGIN_SWITCH_VAR. None of the names it doesn't match switches the login. The nearest are CLAUDE_CONFIG_DIR, which configDirectory covers, and proxy/CA/gateway-hint variables, which are no login (measured scan; the reading of each description is inferred).
The meter: a failed 429 stops the bench with exactly 1 session	Met	The stand-in exits 1 with API Error: 429 {...rate_limit_error...} and no result line. Run through UsageMeter(BenchAgentRunner(CliRunner)) (measured): - engine default retry (test clock): sessions=1, limited set, signal.aborted=true. The next task gets rate_limited / not started, and still sessions=1; - budgetMs: 0: the same. The default sleepFor returns false on an already-aborted signal (src/runner.ts:404-411), so a real-clock wait ends the same way (inferred).
configDirectory is recorded and checked	Met, with one gap	It's recorded as config_directory (measured). These are refused (measured): a relative path, /srv/x/.claude, home/../other/.claude, a path in the home that is a symlink out of it, and a missing value. Gap: "the bench user's home" is os.homedir(), which is $HOME. See required change 1.
Bench root: resolved with realpath, symlinks into protected places refused	Met, with one gap	Each of these is refused (measured): ~/bench → ~/.ssh; ~/bench → a → ~/.claude (a chain); a root under a link into a checkout; a root in the real ~/.claude when the home is given through a link. A link to a folder of its own is resolved and used, by design (test line 266). That is resolution, not a blanket refusal of links. Gap: a dangling link is neither resolved nor refused. See required change 2.

Fail-without (measured)

• Method: a scratch worktree at 858f6272, the PR's commit just before the round-1 fixes. I took test/bench.test.ts from f122bfd9. No guard was edited.
• Build: tsc reports TS2554: Expected 1-2 arguments, but got 3 because requireSubscription gained its home argument. It still emits, and the extra argument is ignored at runtime.
• Result: node --import ./dist/test/isolate.js --test dist/test/bench.test.js gave # pass 4, # fail 4:
  • Missing expected exception: __proto__;
  • Missing expected exception: an oauth_token (CLAUDE_CODE_OAUTH_TOKEN or federation: the real shape);
  • 429: expected: 1, actual: 8;
  • Missing expected exception: ~/bench is a link to ~/.claude.
• The builder's inferred claim, now measured: my meter probe on the pre-fix modules gave limited=null and aborted=false for the budgetMs: 0 half too. The next task then started a second session. So both halves of the 429 test fail without the meter change.
• On the head: # pass 8, # fail 0, with a temp HOME and state dirs and a refusing claude first on PATH.
• npm run typecheck exits 0, and npm run check:denylist gives denylist: 1271 files clean.

"What this can never do" and "Compromised component": are they true now?

• "Start a bench run on anything but a claude.ai login": true for authMethod, apiProvider and every variable family I tried (measured, above). Two of its sub-claims aren't fully true yet:
  • "a config dir outside the bench user's home" is refused: not when $HOME is changed. With HOME=<another folder> holding a claude.ai login there, requireSubscription returned ACCEPTED (measured). os.userInfo().homedir and os.homedir() differed in that probe. The check refuses CLAUDE_CONFIG_DIR but not HOME, which picks another login just as well. The login is still claude.ai/firstParty, so the owner's core rule holds (measured). The accidental su wl-bench (no -) case keeps the admin's HOME and fails closed, because wl-bench can't traverse the admin's 750 home (inferred from the brief's measured mode).
  • "A ~/bench that is a symlink into one of those is refused too": not for a dangling link. ~/bench → ~/.claude/bench-data, where the target doesn't exist yet, gave benchRoot ACCEPTED with root=~/bench (measured). realpathSync fails on the dangling link, so realPath falls back to the unresolved path. Nothing can be written through it: mkdirSync(root|runs|heldOut, {recursive:true}) and writeFileSync(root/x) all gave ENOENT, and ~/.claude was unchanged (measured). So it fails closed today, but by accident.
• "Read, copy or relay the login": true. scripts/bench/*.ts has no readFile, openSync, createReadStream, copyFile or readdir (measured, grep). It only spawns claude auth status --json and claude --version.
• "Retry after a usage or rate limit": true (measured, the meter row above).
• "Spend past the owner's caps": true (measured, the caps row).
• Compromised component: accurate.
  • Nothing in src/ or package.json references scripts/bench, and files doesn't include dist/scripts (measured).
  • The bench-user-shell entry discloses WORKLANE_BENCH_CLAUDE_VERSION and WORKLANE_BENCH_USER (inferred). It should also name HOME unless required change 1 lands.

Protection

• Protected: every new file is under scripts/**, the engine path in .worklane/review.yaml (measured in round 1; unchanged).
• Marking: the PR is correctly marked: the [PROTECTED: owner merges] title, **Risk class:** protected, the needs-owner label, and both required sections (measured).
• Regressions: no setup, firewall, sandbox, lock or guard is touched, so no existing install's route reopens (inferred).
• Owner rule A: the tests use temp homes and the stand-in's absolute path, under test/isolate.ts. I ran everything under env -i with a temp HOME, state dir, held-out dir and XDG_STATE_HOME (measured: passes).

Scope and hygiene

• Scope: the same 6 files as round 1. tsconfig.json only adds scripts/bench/**/*.ts (measured, git diff). No scope creep (inferred).
• Commits: all three are signed off. There's no Co-Authored-By or other attribution trailer (measured).
• Dependencies: none new; new-deps passes (measured).
• Product name: not hardcoded in scripts/bench (measured, grep).
• Body: claims are labelled, and "Part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748" is right for the first of a stack (inferred).

CI

Green (measured, gh pr checks 756):

• every check passes: all 8 test shards, both aggregates, dco, denylist, scan, scope, new-deps, agentshield, push-gate and desktop;
• the ci slow real-systemd jobs are skipped;
• an earlier new-deps run shows CANCELLED, superseded by a passing run (39183717286).

Required changes

1. The config-dir check follows $HOME (auth.ts requireSubscription, default home = homedir()). Take the bench user's home from the passwd entry (os.userInfo().homedir) instead, or refuse when $HOME differs from it. Do the same for benchRoot's default home, whose credential places are also derived from it (inferred: the same root cause). Add a test where HOME points at another folder holding a claude.ai login there, and expect a refusal. Today it is accepted (measured). Or, if the owner prefers, narrow the body's claim to "outside $HOME" and add HOME to the "bench user's own shell" entry.
2. Dangling ~/bench link (paths.ts realPath). When a path component exists as a symlink (lstatSync) but realpathSync fails, refuse the root, or resolve the link with readlinkSync before checking. Add a test where ~/bench → ~/.claude/not-yet and expect a refusal. Today it is accepted (measured), and nothing can be written through it (measured ENOENT). Or narrow the body's claim to links that resolve.

Not required (notes for the owner)

• A link inside an existing root: ~/bench/state → ~/.claude is accepted, and layout.state then points into ~/.claude (measured). Only the root itself is checked. Planting that link takes write access to ~/bench, so it's bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757/bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759's sandbox that must keep agents from writing there (inferred). An lstat check on the layout's directories in the driver would close it.
• Repeated --tasks: --tasks smoke --tasks baseline --spend 40 gives mode: smoke, because the first flag wins (measured). bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 must take the mode from Caps, never re-read argv (inferred).
• Accepted config dirs: configDirectory equal to the home itself is accepted, being "inside" it (inferred from inside()). Round 1 suggested the bench user's own ~/.claude exactly. That's harmless either way (guess).

[jayden1711]

Round 3 (owner-granted): what changed

Head 571f3c0. origin/main (3519839, with #751 and #761) is merged in first with a signed-off merge commit (measured: the merge was clean, with no conflicts).

1. The home comes from the user database, never $HOME (owner decision 1).

• paths.ts has a new benchHome(env). It takes the home from os.userInfo().homedir (the passwd entry). It refuses when $HOME is set and differs from that home (compared on real paths); it doesn't ignore it. The reason: claude itself follows $HOME to its login, so ignoring it in our checks would leave claude on a login the checks never looked at (inferred).
• requireSubscription (the config-dir check) and benchRoot (and the credential places derived from the home) now default to benchHome. When a home is passed in, $HOME is still checked against it (measured, tests and probe).
• Test: "the bench user's home is its passwd entry's, never $HOME". HOME points at another folder that holds a claude.ai login; claude isn't asked. It also covers the default path, where this process's HOME is test/isolate's temp home, which the user database doesn't name.
• Fail-without (measured, a probe against old and new builds): with no home given and HOME at such a folder, requireSubscription was ACCEPTED on the old code and is REFUSED now (HOME is …, not this user's home … from the user database). The same goes for benchRoot, with and without a home given. A HOME equal to the bench home is still accepted.

2. A ~/bench link that doesn't resolve is refused (owner decision 2).

• realPath now refuses a path when a component exists (lstat) but doesn't resolve: a dangling link, or a link loop.
• The places the root is checked against follow such a link through readlink, so a root where a dangling ~/.ssh points is refused too.
• Test: "a bench root through a link that doesn't resolve is refused". It covers ~/bench → ~/.claude/not-yet, a dangling link further up, a loop, and a dangling ~/.ssh.
• Fail-without (measured): the new tests on the old auth.ts and paths.ts give # pass 8, # fail 2, including Missing expected exception: ~/bench -> ~/.claude/not-yet (dangling). In the probe, all four cases were ACCEPTED on the old code and are REFUSED now. A plain root is still accepted.

Local results (measured): bench.test.js gives # pass 10, # fail 0, run with a temp HOME, state and held-out dirs and refusing claude, docker and srt stand-ins first on PATH. The stand-ins were never called. npm run typecheck exits 0. check:denylist prints denylist: 1278 files clean, and check:dco prints DCO: all commits signed off.

Body: the paths and auth bullets, the test count, verification, "What this can never do" (a new bench-root bullet with its one inferred limit) and the bench-user-shell entry are updated to match.

Owner rule A (inferred from the code): the default-path test resolves the real passwd home's path with realpath, only to compare it. It doesn't read or write anything in it, and nothing under ~/.claude* or ~/.local/state.

[jayden1711]

Evaluator verdict: PASS

I reviewed head 571f3c07 against base main. The PR's own diff (merge base 3519839b) is the same 6 files as before: scripts/bench/{auth,caps,paths,usage}.ts, test/bench.test.ts and tsconfig.json (measured, git diff origin/main...HEAD --stat). Round 3 changed only auth.ts, paths.ts and the test (measured, git diff f122bfd9 HEAD). Main has since moved to 32c9143f, and the head still merges into it with no conflicts (measured, git merge-tree).

Both round-2 required changes are fixed, following the owner's round-3 decision. Every earlier pass still holds. I ran every probe and test under env -i with a temp HOME, state dir, held-out dir and XDG dirs, with refusing claude, docker and srt stand-ins first on PATH. None of the stand-ins was ever called.

Round-3 changes

Item	Status	Evidence
benchHome refuses a $HOME that differs from the passwd home (compared on real paths)	Met	My probe used an injected passwd home in a temp dir (measured). Accepted: HOME unset; HOME equal to the home; the home with a trailing /; home/.claude/..; a symlink to the home; a two-link chain to the home; and a passwd home that is itself a link, with HOME at its target. Refused: HOME at another folder, "", /, a path that doesn't exist, and a dangling link. A passwd home that is empty or relative is refused too. With the real passwd default: HOME at another folder is refused, and so is the sandbox's own HOME. HOME unset, or a symlink to the real home, is accepted.
Every default uses it	Met	requireSubscription(..., home = benchHome(env)) and benchRoot(env, home = benchHome(env)). No homedir() or process.env.HOME is left in scripts/bench (measured, grep). Results (measured): - requireSubscription, no home given, HOME at a folder holding a claude.ai login: REFUSED, claude asked: false. - requireSubscription, home given, HOME elsewhere: REFUSED, claude asked: false. - requireSubscription, home given, HOME a symlink to it: ACCEPTED. - benchRoot, HOME elsewhere, with or without a home given: REFUSED. - benchRoot, HOME a symlink to the given home: ACCEPTED, with the root at <home>/bench.
realPath refuses a dangling component, a loop, and a root where a dangling ~/.ssh points	Met	Refused (measured): a dangling middle component, a dangling leaf, a self-loop, and a two-link loop (in both refuse and follow mode). Under benchRoot, these were all refused as overlaps: a root at, under or above where a dangling ~/.ssh points; a relative dangling ~/.aws (../elsewhere/aws); the end of a two-hop dangling chain from ~/.gnupg; and a dir holding where a dangling ~/.claude.json points. A ~/.netrc that is a self-loop fails closed: the root is refused. A sibling root is accepted (the control). Nothing was created at any link target (elsewhere contents after: []).

Earlier passes, re-run on this head (round 2's probe, unchanged)

• Caps are prototype-safe (measured):
  • every one of these is refused with --tasks takes smoke or baseline: __proto__, constructor, toString, hasOwnProperty, valueOf, __defineGetter__, __lookupGetter__, isPrototypeOf, propertyIsEnumerable, toLocaleString and "";
  • --spend Infinity, NaN, -5 and 0 are refused, and 1e9 is over the $40 cap.
• claude.ai only, and the prefix variable rule (measured):
  • each of these is refused with claude asked: false: federation, ANTHROPIC_BASE_URL (also when empty), CLAUDE_CODE_OAUTH_TOKEN, AWS_BEARER_TOKEN_BEDROCK, ANTHROPIC_PROFILE, both *_FILE_DESCRIPTOR variables plus a made-up one, CLAUDE_CODE_USE_FOUNDRY, and a lower-case anthropic_api_key;
  • these statuses are refused: oauth_token, non-JSON, empty, null, a string, an array, a __proto__ smuggle, and loggedIn: "true";
  • these config dirs are refused: a relative one, one outside the home, home/../other, and a symlink out of the home;
  • the good shape is accepted.
• The meter stops on 429 (measured): the stand-in exits 1 with API Error: 429 and no result line.
  • Engine default retry (fake clock): sessions=1, limited set. The next task gives rate_limited / not started, still with sessions=1.
  • budgetMs: 0: the same.

The default-path test touches nothing under the real home or ~/.claude*

• Method: I ran bench.test.js under strace -f -e trace=%file,%desc.
• Result (measured): under /home/<user>/, the trace touched only these paths:
  • statx of the home directory itself (5 calls, AT_SYMLINK_NOFOLLOW): the realpath that benchHome compares;
  • the node binary;
  • the worktree.
• What it did not touch (measured): nothing under ~/.claude* or ~/.local/state, and no held-out store. The only .claude paths are inside the temp sandbox, plus the test's synthetic /srv/other-login/.claude, which got ENOENT.
• My own probes, under the same trace (measured): the same result.
• Why the guarded branch runs (inferred): on Linux, test/isolate moves HOME, so the test's early return (when HOME is the passwd home) is not taken and the default-path assertions run.

Fail-without (measured)

• Method: a scratch worktree at c2ab1a73 (the PR's parent commit: main already merged in, before round 3), with only test/bench.test.ts taken from 571f3c07. No guard was edited, and it compiles cleanly.
• The new tests on the old code: node --import ./dist/test/isolate.js --test dist/test/bench.test.js gave # pass 8, # fail 2:
  • not ok 3, on its first assertion (home given, HOME at another login). The old code refused that case too, but through the config-dir check: claude reports its login in …/home/.claude, not inside the bench user's home. So test 3 fails on its message regex, before it reaches the default-home case;
  • not ok 10: Missing expected exception: ~/bench -> ~/.claude/not-yet (dangling).
• The bug itself, probed on the old build and then on the new (measured):
  • requireSubscription with no home given, and HOME at a folder holding a claude.ai login that the user database doesn't name: ACCEPTED (claude asked: true), then REFUSED (claude asked: false);
  • benchRoot with no home given, and HOME at that folder: ACCEPTED, then REFUSED;
  • benchRoot with a home given and HOME elsewhere: ACCEPTED, then REFUSED;
  • ~/bench → ~/.claude/not-yet, a root under a dangling link, and a root under a loop: all ACCEPTED, then all REFUSED;
  • a root where a dangling ~/.ssh points: ACCEPTED, then REFUSED;
  • the control, a plain root: ACCEPTED on both.
• On the head: # pass 10, # fail 0. npm run typecheck exits 0, and npm run check:denylist prints denylist: 1278 files clean (measured).

Protection

• Protected: every file is under scripts/**, the engine path in .worklane/review.yaml (measured in round 1; the file set is unchanged).
• Marking: the PR is still correctly marked: the [PROTECTED: owner merges] title, the needs-owner label, **Risk class:** protected, and both required sections (measured, gh pr view).
• "What this can never do": the bench-root bullet now covers dangling links and loops, and the config-dir bullet names the passwd home. Both are true per the probes above (measured). The disclosed limit (a link planted inside an existing root) is still accepted (measured: ~/bench/state → ~/.claude gives ACCEPTED), as the body says.
• Compromised component: accurate. The bench-user-shell entry now says that moving $HOME is refused (measured).
• Regressions: no setup, firewall, sandbox, lock or guard file is touched, so no existing install's route reopens (inferred).
• Owner rule A: met. The tests use temp homes and stand-ins, and the strace above shows nothing under the real home or ~/.claude* is touched (measured).

Scope and hygiene

• Scope: only the two round-3 fixes and their tests. No scope creep (measured, the diff).
• Commits: all five are signed off. There is no Co-Authored-By or other attribution trailer (measured, git log).
• Dependencies: none new; new-deps passes (measured).
• Product name: not hardcoded in scripts/bench (measured, grep).
• Body: claims are labelled, and "Part of Bench: gated-merge benchmark on SWE-bench Verified (smoke, baseline, per-release rerun) in scripts/bench/ #748" is right for the first PR of a stack (inferred).

CI

Green and complete (measured, gh pr checks 756). Everything passes: all 8 test shards, both test aggregates, dco, denylist, both scans, scope, new-deps, agentshield, push-gate and desktop. The ci slow real-systemd jobs are skipped.

Notes for the owner (not required)

1. A home passed in skips the user-database lookup. requireSubscription(env, bin, home) and benchRoot(env, home) then compare $HOME only with the home given (measured: with a home given and HOME unset, benchRoot accepts it). Nothing in this stack passes a home: the only call sites are tests, and bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759's driver takes the check injected (measured, grep of the local bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757/bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759 branches). The CLI entry point that comes later must call both with no home argument (inferred).
2. Test 3's fail-without shows a wrong message, not an acceptance. The default-home acceptance on the old code is shown by the probe above, not by the test's failing line (measured). Putting the default-home assertions first would make the test's own failure show the bug. That's optional.
3. A link inside an existing root (~/bench/state → somewhere) is still not checked, as disclosed. bench: task containers in rootless Docker with required limits, exec-only, no daemon for agents [PROTECTED: owner merges] #757/bench: run driver, patch export, grading inputs, aggregates and setup README [PROTECTED: owner merges] #759's sandbox must keep agents from writing to ~/bench (inferred).
4. HOME unset is accepted. claude then falls back to the passwd home, the same one the bench checks (guess: the CLI's own homedir fallback, not run).