Skip to content

test: prove the extra shots' budget on a mock clock, not the runner's wall time (#720) - #835

Merged
jayden1711 merged 3 commits into
mainfrom
issue-720
Oct 20, 2026
Merged

jayden1711 merged 3 commits into
mainfrom
issue-720

Conversation

@jayden1711

@jayden1711 jayden1711 commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Intent: the browser-run budget tests never fail because a CI runner is slow, and they still prove that the dark and phone shots share one budget and that every extra command is cut off at what remains. They now run on node:test's mock clock instead of the wall clock. The production cap in src/browser/run.ts is unchanged, and no src file changes.

Risk class: logic (test-only). The only changed path is test/browser-run.test.ts. It is not in .worklane/review.yaml's engine / engine-guards / release-signing lists and is touched by none of the open PRs checked (#769, #770, #788, #789, #803, #816, #818, #819, #820, #825), measured with gh pr diff --name-only.

Closes #720

What changed

  • A mockClock(t) helper.
    • It enables t.mock.timers for setTimeout and Date (from 0), and mocks performance.now to read the pinned Date. performance.now is captureShots' monotonic clock.
    • It returns run(work), which moves the clock 1 ms at a time once everything due by then has run (setImmediate stays real). It returns the result and the mock time taken.
    • The elapsed time the tests check is therefore what the timers say (inferred from the code; measured below).
  • A stall() in the stand-ins. Every stand-in command blocks the event loop for 40 ms of real time (Atomics.wait). This is a slow runner built into the test, as the issue's acceptance asks. The mock clock never sees it.
  • The three budget checks move to the mock clock:
    • "share a short budget" (50 ms budget);
    • "never take longer than their budget" (1100 ms budget);
    • its "late" part (600 ms budget).

Every changed assertion, and why it is no weaker

test before after why not weaker
never longer than budget (1100 ms) took < 1100 + 250 (wall clock) took <= 1100 (mock clock) It is the same quantity: elapsed time on captureShots' own clock, from the desktop shot's end to the return. The 250 ms of slack is gone.
late part (600 ms) lateTook < 600 + 120 (wall clock) lateTook <= 600 (mock clock) Same reasoning; the 120 ms of slack is gone.
share a short budget (50 ms) (no time check) new ms <= 50 Added; nothing was removed.

Every other assertion in these tests is unchanged, including views, messages, files, the verdict, the sent commands and their per-command timeout. git diff shows no other assert line touched (measured).

Scope: what these tests prove, and what they don't

  • They prove captureShots' budget logic, on its own clock (performance.now). The dark and phone views share one budget. Each extra command is raced against what remains and refused once nothing remains. The pause after the frame wait is cut to what remains. On that clock, the total is at most the budget. This is measured with the mutants below.
  • They no longer prove anything about wall time. Real-time cost with no timer, such as a synchronous stall inside captureShots, is invisible to the mock clock.
  • Neither the old tests nor the new ones prove that captureShots reads a monotonic clock (inferred). Under the mock, Date and performance.now read the same clock.

Round 2: a real-time bound on the budget tests

  • What changed:
    • Both budget tests now carry node:test's per-test { timeout: 60_000 } (BUDGET_TEST_TIMEOUT_MS). It fires on the real clock even while t.mock.timers holds setTimeout.
    • The clock-moving loop also stops once the test's t.signal is aborted, so a timed-out test stops moving the mock clock (it exits rather than spinning on).
  • Why a bound is needed: without it, a regression that never settles is stopped only by the 60,000-turn guard. A starved runner could stretch that to hours (inferred in the round-1 review), and the required shards have no other bound.
  • Why 60 s:
    • On a normal runner each budget test takes under 0.4 s (measured).
    • On a runner whose event loop is blocked 200 ms in every 250 ms, both together took 2 s (measured), so 60 s is about 30× that.
    • 60 s is also what this file already gives its real-Chrome tests.
    • It only adds a failure path, so it fails closed and loosens nothing.
  • Accepted cost: at the extreme 300 ms-in-every-100 ms block, the 1100 ms test took 512 s in round 1, so it now times out at 60 s (measured by the round-2 review: cancelled at 60.4 s). That is a runner whose own event loop is starved 75% of the time. I take that as acceptable, since the alternative is no bound at all.
  • Fail-without, measured with a never-settling run.js mutant (extra commands never answer and the budget's cut never fires), on that starved runner, with an outer timeout 150 / timeout 400:
tree result
f121552 (no per-test timeout) No test had finished when the outer timeout cut the run at 150 s (exit 124). It would have run on until the 60,000-turn guard.
this head Both tests stopped with test timed out after 60000ms (60.4 s and 60.4 s; cancelled 2). The run exits 1 after 121 s on its own.

On a normal runner the same mutant fails at the turn guard in 0.66 s for the whole run ("the extra shots still ran after a minute of mock time"), as before (measured).

  • Re-checked on this head (measured):
    • Budget tests on a normal runner: 2/2 pass.
    • Mutants: no-race 2 fail, per-view-only 2 fail, full-pause 1 fail (840 ms > 600).
    • Slow runner at 200 ms every 250 ms: 2/2 pass, 1.6 s.
    • Typecheck: exit 0.
    • Every test file CI selects for this change, across all 4 shards (affected.mjs --event pull_request, then shard.mjs s/4 for s = 1–4): 76 files. I ran 75 of them; mcp-broker-unit was excluded because this machine must never run the escape fixture, and CI runs it. Result: 999 tests, 985 pass, 0 fail, 0 cancelled, 14 skipped (the real-Chrome and similar tests, since this machine has no Chrome).

Correction (round 1)

This body first said every check was "measured", but the first head (e7b2de0) failed a required check that I had not run.

  • What failed: shard 4/4 on Node 22 and 24 (run 39328722273), in test/no-wall-clock-durations.test.ts (Clocks: dashboard listing caches use a monotonic injectable clock, so a wall-clock step cannot keep or cut a cache window; lint new Date.now() durations in tests (v0.5.3) #582), with test/browser-run.test.ts: 2, allowed 1.
  • Cause: the mockClock helper measured time as Date.now() - t0, which that lint counts.
  • Fix (f121552): the helper now reads performance.now(), the clock captureShots reads, which the helper already points at the mock clock. The allowance is not raised, and every bound is unchanged (<= 1100, <= 600, <= 50).
  • Verified locally on f121552 (measured):
    • no-wall-clock-durations and no-clock-helpers: 4/4 pass.
    • CI's own shard 4/4 file list for this branch (scripts/ci/affected.mjs --event pull_request then scripts/ci/shard.mjs 4/4, 19 files): 217 tests, 217 pass, 0 fail. No mcp-* or plugin-fetch file was in the list.
    • The budget tests on the normal runner: 2/2 pass. Against the run.js mutants: no-race 2 fail, per-view-only 2 fail, full-pause 1 fail.
    • Typecheck: exit 0.
  • What I missed before: I ran browser-run and no-clock-helpers, but not the shard lists CI selects for this change.

How it was verified

Scratch tools live only in the scratchpad; no repo file was edited to make a mutant.

  • Slow runner: an --import preload that blocks the event loop for SLOW_BLOCK_MS every SLOW_EVERY_MS of real time.
  • Budget-cap mutants: a module load hook that rewrites dist/src/browser/run.js as it is loaded:
    • no-race: each extra command waits for its full answer.
    • per-view-only: extra commands are neither refused nor cut, which is the bug the test's comment describes.
    • full-pause: the pause after the frame wait is not cut to what is left.

Run: node --import ./dist/test/isolate.js [--import <preload>] --test --test-name-pattern budget dist/test/browser-run.test.js, with HOME and the state dirs pointed at a scratch dir.

Old tests (origin/main 6e0d7d8), measured:

runner result
normal 2/2 pass
slow, block 300 ms every 100 ms 1 fail: "never longer than budget", views came out dark: skipped… / phone: not tried…
slow, block 200 ms every 250 ms 1 fail, same test (deep-equal on views)
slow, block 140 ms every 300 ms 2/2 pass. The flake depends on load, matching CI's intermittent 768 ms vs 600 ms.
mutant no-race 2 fail
mutant per-view-only 2 fail
mutant full-pause 1 fail (the extra shots took 843 ms, past their 600 ms budget)

New tests (this head), measured:

runner result
normal (stand-ins stall 40 ms per command) 2/2 pass, about 0.2–0.35 s each
slow, block 300 ms every 100 ms On f121552: 2/2 pass, but it took 512 s of wall time, because the event loop is starved and the clock moves once per loop turn. On c461a29: 1 pass, 1 fail. "never take longer…" is cancelled by its own 60 s timeout at 60.4 s (measured by the round-2 review). This is the accepted cost described in "Round 2".
mutant no-race 2 fail
mutant per-view-only 2 fail
mutant full-pause 1 fail: the extra shots took 840 ms, past their 600 ms budget
  • Whole file dist/test/browser-run.test.js: 17 tests, 12 pass, 0 fail, 5 skipped. The 5 skips are the real-Chrome tests, because this machine has no Chrome (measured).
  • test/no-clock-helpers (the Tests: no result depends on today's date; five intake tests go red on main ~2026-11-02 (fix before 2026-11-01) #687 lint on clock-reading helpers): 2/2 pass (measured).
  • npm run typecheck: exit 0 (measured).
  • npm run check:denylist: denylist: 1309 files clean (measured).
  • Package: no packaged files change (measured). npm pack --dry-run --ignore-scripts lists no test/ file except examples/basic/test/price.test.js, and the package files field ships only dist/src, dist/web, templates, examples/basic and the notices.

Adversarial pass

  • Does a mock clock still test the real cap? captureShots reads performance.now() and arms setTimeout, and both are what the mock drives, so the production code path runs unmodified (inferred). All three mutants of that code fail the new tests (measured).
  • Could the test pass with the cap removed because time no longer moves? No. The stand-ins' answers are mock-clock timers (390, 590, 300, 80 ms), so a mutant that waits for them accrues mock time and fails <= budget (measured: the mutants fail).
  • Slow runner: the mock clock ignores real delays. Measured with the in-test 40 ms stalls, and with a preload that blocks the loop 75% of the time. Under that extreme, f121552 was slow (512 s) but correct. c461a29 fails closed instead: "never take longer…" is cancelled by its 60 s timeout (measured by the round-2 review), which is the accepted cost in "Round 2". On CI, contention slows each turn without starving the loop (inferred).
    • Follow-up idea (the round-2 reviewer's), not done here: advance the mock clock straight to the next due timer instead of 1 ms per turn. The tests would then need a handful of turns rather than about 1,700, which would make the 60 s bound almost immune to load (inferred).
    • Corrected in round 2: round 1 said here that "the file's tests are far under --test-timeout". That was false. The required shards set no --test-timeout and no job timeout-minutes (measured by the round-1 review); only the weekly clock-shift job sets one.
    • What bounds the budget tests now is their own real-time { timeout: 60_000 }, added in round 2 (measured, see "Round 2").
  • Does mocking leak into other tests? t.mock is restored when each test ends (node:test documents this). The rest of the file passes in the same process (measured: 12/12 non-Chrome tests).
  • A dangling stand-in timer from the first run (the phone command the budget cut off) fires during the "late" run. It only resolves a promise nobody awaits any more (inferred), and the late run's assertions pass (measured).
  • Rule 10: the bounds are tighter (<= budget, no slack); no assertion was removed; the production cap is untouched.
  • Durability rule: the protection here is the runtime cap in src/browser/run.ts, which is by design and unchanged. This PR only makes its test deterministic; it adds no lint.

New dependencies

None.

… wall time (#720)

The browser-run budget tests timed captureShots with performance.now() against real timers, so a loaded runner
overran them (CI: 768 ms for a 600 ms budget, on a change that touched no browser file). They now run on
node:test's mock clock: setTimeout and Date pinned, performance.now (captureShots' monotonic clock) reading the
pinned Date, and the clock moved 1 ms at a time once everything due has run. The stand-ins also block the event
loop for 40 ms of real time per command, a slow runner the mock clock never sees.

The checks are tighter, not looser: the extra shots must take at most their budget (was budget + 250 ms and
budget + 120 ms of slack), and the 50 ms-budget test now checks its time too. No src change.

Test (fails without): on a simulated slow runner (an --import that blocks the event loop) the old tests fail and
the new ones pass; the new ones fail against run.js mutants with the budget's race removed, with the budget checked
per view only, and with the pause after the frame wait not cut to what is left.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
…te.now minus a start (#720)

The new mockClock helper measured the extra shots' time as Date.now() - t0. That is the shape
test/no-wall-clock-durations.test.ts (#582) counts, so browser-run.test.ts went to 2 against its allowance of 1, and
shard 4/4 failed on Node 22 and 24 (run 39328722273). It now reads performance.now(), the clock captureShots itself
reads, which the helper already points at the mock clock. The value is the same: every bound stays as tight as it was
(<= 1100, <= 600, <= 50), and the allowance is not raised.

Test (fails without): no-wall-clock-durations reports "test/browser-run.test.ts: 2, allowed 1" on the previous head
and passes on this one.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 1/2)

Reviewed head f121552 against #720. I also checked e7b2de0, the first head. Base: 6e0d7d8, which is the merge base and current main (measured). Every claim below is labelled measured, inferred or guess. All local runs used a scratch HOME and state dirs, plus --import ./dist/test/isolate.js. Mutants and the slow runner come from --import preloads kept in the scratchpad; no repo file was edited.

Acceptance criteria (#720)

criterion result evidence
The test passes under a simulated slow runner met See "Fail-without" below (measured).
It still fails if the budget cap is removed from src/browser/run.ts met All 3 cap mutants fail on f121552 (measured, below).
It keeps the guarantee: the budget is shared, and each command is capped at what remains met no-race and per-view-only fail both tests, and full-pause fails the late part (measured).
It uses an injected clock and timers met t.mock.timers covers setTimeout and Date, and performance.now is mocked to read the mock Date (inferred from the diff).
The production cap is not weakened met No src/ file changed (measured: git diff --stat shows only test/browser-run.test.ts).
npm run typecheck met exit 0 on f121552 (measured).

Rule 10: are the bounds weaker?

  • No.
    • took < 1100 + 250 became took <= 1100.
    • lateTook < 600 + 120 became lateTook <= 600.
    • ms <= 50 is new.
  • No other assert line changed (measured from the diff).
  • f121552 changed only how mockClock takes t0 and ms: performance.now() replaced Date.now(). Both read the same mocked clock. The bounds are unchanged, and ALLOWED in test/no-wall-clock-durations.test.ts is unchanged, since that file is not in the diff (measured: git diff e7b2de00 f1215522).

Fail-without

Slow runner: a preload that blocks the event loop for B ms every E ms, using real setInterval, which the test doesn't mock. Budget tests only (--test-name-pattern budget). All results measured:

tree runner result
base 6e0d7d8 (old tests) block 300 / every 100, 2 runs 1 fail each run: "never take longer…"; views came out dark: skipped… / phone: not tried…
base 6e0d7d8 (old tests) block 200 / every 250, 2 runs 1 fail each run, the same views
f121552 normal 2/2 pass. The tests take 0.13 s and 0.30–0.34 s; the whole budget-only run takes 0.8 s wall
f121552 block 200 / every 250 2/2 pass, 2 s wall
f121552 block 300 / every 100 2/2 pass. The tests take 15.5 s and 512 s; the run takes 529 s wall

Run.js mutants, applied by a load hook (each one was confirmed applied). All results measured:

mutant old tests (base) new tests (f121552)
no-race: commands wait for their full answer 2 fail 2 fail
per-view-only: no refusal or cut per command 2 fail 2 fail
full-pause: the pause is not cut to what is left 1 fail (850 ms > 600) 1 fail (took 840 ms, past their 600 ms budget)
sync-stall: a synchronous 300 ms block per extra view 1 fail (views) 2 pass

What the new tests prove, and what they no longer prove

  • They prove (measured via the mutants): captureShots' budget logic, on captureShots' own clock (performance.now). The two views share one budget. Each command is raced against what remains, and the race is refused once nothing remains. The pause is cut to what remains. The total, on that clock, is at most the budget.
  • They no longer prove anything about wall time. The sync-stall mutant passes the new tests but failed the old ones (measured).
  • Not proven by either version (inferred): that captureShots reads a monotonic clock. Under the mock, Date and performance.now are the same clock.

Can it hang in CI?

  • The run loop's only guard is 60,000 loop turns, one mock ms each.
    • On a normal runner, a stand-in that never settles trips the guard in 0.26–0.42 s (measured with a never-settles mutant: "the extra shots still ran after a minute of mock time").
    • Each turn costs real time under load, though. At block 300 / every 100, the passing tests needed about 1,700 turns for 512 s, so a never-settling regression under that load would hold the test for about 60,000 / 1,700 × 512 s, roughly 5 h (inferred from the measured rate).
  • There is no real-time bound.
    • The required shards run node --import ./dist/test/isolate.js --test "${files[@]}", with no --test-timeout and no job timeout-minutes (measured: grep of .github/workflows/ci.yml). The GitHub default job timeout is 360 min (guess, from GitHub's documented default).
    • Only clock-shift.yml passes --test-timeout=600000.
    • The two budget tests have no { timeout } option (measured).
  • node:test's per-test timeout still fires in real time while t.mock.timers mocks setTimeout and Date. I measured this with a scratch probe test that never settles under mock timers: "test timed out after 2000ms" on Node 22.23.

Protection check

  • test/browser-run.test.ts is in none of .worklane/review.yaml's lists, and in no SELF_MODIFICATION_PARTS or default locked glob, only tests (measured with the head's selfModificationPart and globToRegExp, against a control path that did match).
  • Not protected.

Scope, hygiene, CI

  • Scope: only test/browser-run.test.ts changed (measured).
  • Both commits are signed off and have no attribution trailer (measured). No new dependencies, and new-deps passes (measured).
  • npm run check:denylist: denylist: 1309 files clean (measured).
  • Package: npm pack --dry-run --ignore-scripts lists 473 files on both base and head. No test/ file is in it except examples/basic/test/price.test.js (measured).
  • Locally on f121552 (all measured):
    • browser-run, no-clock-helpers and no-wall-clock-durations together: tests 21, pass 16, fail 0, skipped 5. The 5 skips are the real-Chrome tests ("no Chrome or Chromium on this machine").
    • CI's own shard 4/4 selection: affected.mjs --event pull_request --base 6e0d7d81 selected 76 files, and shard.mjs 4/4 gave 19 of them. Result: tests 217, pass 217, fail 0.
  • CI:
    • On e7b2de0 (run 39328722273), shard 4/4 failed on Node 22 and 24 with test/browser-run.test.ts: 2, allowed 1 from no-wall-clock-durations. This PR caused it, and the body now owns it in "Correction (round 1)" (measured).
    • On f121552 (run 39331671690, headSha f121552): every check passes, including shard 4/4 on 22 and 24. The budget tests ran in shard 2/4 and passed on both Node versions (measured from the job logs).

Required changes

  1. Correct a false claim (principle 5).
    • The Adversarial pass says "the file's tests are far under --test-timeout". The required shards set no --test-timeout (measured, see above); only the weekly clock-shift.yml does.
    • Replace it with what actually bounds the test.
  2. Bound the budget tests in real time (principle 1, fail closed).
    • Today a regression that never settles is stopped only by the 60,000-turn guard. That takes about 0.4 s on a normal runner, but on a starved one it could hold a shard for hours (inferred, above).
    • Give the two budget tests a real-time { timeout } that is generous enough not to bring back load-dependence. For example, the file already uses { timeout: 60_000 }, or use something above the 512 s extreme you measured, and say why you chose it.
    • Show it firing with a never-settling stand-in (measured).
    • This adds a bound and loosens nothing.

Not required (named for the record)

…per test) (#720)

On the mock clock the budget tests advance one mock ms per event-loop turn, so a regression that never settles was
stopped only by the 60 000-turn guard, which a starved runner could stretch to hours: the required shards set no
--test-timeout. Both tests now carry node:test's per-test { timeout: 60_000 }, which fires on the real clock while
t.mock.timers holds setTimeout, and the loop stops once the test's signal is aborted. 60 s is about 30 times the 2 s
both took on a runner blocked 200 ms in every 250 ms, and what this file gives its real-Chrome tests. It only adds a
failure path; no bound is loosened.

Test (fails without): with a run.js stand-in whose extra commands never settle, on a runner blocked 300 ms in every
100 ms, the previous head had finished no test after 150 s; this one stops both with "test timed out after 60000ms".

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Round 2 is pushed: head c461a29. Each required change is addressed:

  1. The false claim is corrected. The Adversarial pass no longer says the tests are "far under --test-timeout". It now states that the required shards set no --test-timeout or timeout-minutes, and names what bounds the budget tests instead.
  2. Both budget tests now have a real-time bound.
    • Each has { timeout: 60_000 }, and the clock-moving loop stops once t.signal is aborted.
    • The reason for 60 s is in the code and the body: about 30× the 2 s measured on a runner blocked 200 ms in every 250 ms, and the same bound this file gives its real-Chrome tests.
    • Fail-without (measured): a never-settling run.js mutant, on a runner blocked 300 ms in every 100 ms.
      • f121552 had finished no test when an outer 150 s limit cut the run (exit 124).
      • c461a29 stopped both tests with test timed out after 60000ms (cancelled 2), and the run exited 1 on its own after 121 s.
  3. The scope is named in the body. These tests prove logic on captureShots' own clock, not wall time; your sync-stall mutant passes them, and Flaky: browser-run budget test measures wall time on CI (768 ms vs 600 ms on Node 24); prove the cap with an injected clock #720 accepts that trade.

Re-checked on this head (measured):

  • Normal runner: 2/2 pass.
  • Mutants: no-race 2 fail, per-view-only 2 fail, full-pause 1 fail.
  • Runner blocked 200 ms in every 250 ms: 2/2 pass.
  • Typecheck: exit 0.
  • Every CI-selected file across all 4 shards (75 of 76; mcp-broker-unit excluded because this machine must not run it): 999 tests, 0 fail.

Filed from your notes:

@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 2/2), PR body only

Reviewed head c461a29 (round 2) against #720 and my round-1 verdict. Every claim below is labelled measured, inferred or guess. All runs were local, with a scratch HOME and state dirs and --import ./dist/test/isolate.js. The mutants and the slow runner come from --import preloads kept in the scratchpad; no repo file was edited.

The code is done. The one required change is in the PR body: two lines still describe f121552's behaviour at the extreme load as this head's. Once the body is corrected, this is PASS with no further code review needed (inferred: the delta would be text only).

Round-1 required changes

# change result evidence
1 Correct the --test-timeout claim met The Adversarial pass now says the required shards set no --test-timeout or timeout-minutes, and names the per-test timeout (measured: the PR body).
2 A real-time bound on the budget tests met Both have { timeout: BUDGET_TEST_TIMEOUT_MS } = 60 000, and the loop stops on t.signal.aborted (measured: git diff f1215522 c461a297).

The timeout, reproduced with my own stand-in

  • I used my own never-settles mutant, which makes captureShots await a promise that never resolves before the views loop. It was applied through a load hook on dist/src/browser/run.js, and the hook confirmed it applied once.
  • The runner was blocked 300 ms in every 100 ms, with an outer timeout 200.
  • Result on c461a29 (measured): both tests failed with 'test timed out after 60000ms' (cancelled 2, pass 0). The run exited 1 on its own after 125 s of wall time; the outer limit's code would have been 124.
  • The 60 s is real time.
    • The turn guard would have needed 60,000 loop turns. On this runner the passing tests needed about 1,700 turns for 512 s in round 1, so the guard alone would take hours (inferred from measured rates). The tests stopped at 60 s instead.
    • My round-1 probe test, under t.mock.timers for setTimeout and Date, also timed out at 2,002 ms of real time (measured, Node 22.23).
  • Normal runner: the same mutant still fails at the turn guard in under a second (round 1 measured 0.26–0.42 s per test; the code is unchanged there).

Rule 10: bounds and allowance

  • The assertions are unchanged from f121552: ms <= 50, took <= 1100 and lateTook <= 600. The round-2 diff touches no assert line (measured).
  • ALLOWED in test/no-wall-clock-durations.test.ts is unchanged. The PR's whole diff against base 6e0d7d8 is test/browser-run.test.ts only (measured with git diff --stat).
  • no-clock-helpers plus no-wall-clock-durations: 4/4 pass on c461a29 (measured).

Re-checks on c461a29 (all measured)

  • Normal runner: 2/2 pass, in 167 ms and 331 ms.

  • At block 200 / every 250: 2/2 pass, in 349 ms and 1,355 ms.

  • Mutants:

    mutant result
    no-race 2 fail
    per-view-only 2 fail
    full-pause 1 fail (took 840 ms, past their 600 ms budget)
    sync-stall 2 pass, as named in the body's new Scope section
  • At block 300 / every 100, unmutated: "share a short budget" passes (15.5 s). "never take longer…" now fails: it is cancelled by the 60 s timeout at 60.4 s.

    • That is the "Accepted cost" the body predicts, now measured.
  • Typecheck exit 0. Denylist: 1309 files clean.

  • git merge-tree --write-tree origin/main c461a297 merges cleanly. Main (now 71316eb) has not changed browser-run, the two clock lints or src/browser since 6e0d7d8.

Commit and body

  • c461a29 is signed off and has no attribution trailer. Its message has a fail-without (measured).
  • The body has a Scope section, a "Round 2" section and the corrected --test-timeout text (measured). It has no attribution line.

CI on c461a29

  • Run 39337267830 ran on headSha c461a29 and concluded success (measured).
  • Every check passes, including test shards 1–4 on Node 22 and on Node 24, dco, denylist and push-gate. The real-systemd jobs show as skipping (measured: gh pr checks 835).

Required change

  1. Correct two stale claims in the PR body (principle 5). They describe f121552, not this head.
    • The "New tests (this head), measured" table's row "slow, block 300 ms every 100 ms: 2/2 pass. It took 512 s…".
    • The Adversarial pass line "under that extreme the run is slow (512 s) but correct".
    • On c461a29 that runner gives 1 pass, 1 fail: "never take longer…" is cancelled at 60.4 s by its timeout (measured above). The body's own "Accepted cost" says so, so the body currently contradicts itself.
    • Relabel that row as f121552's, or replace it with this head's measured result.

For the owner (a trade-off, not a required change)

  • The real-time bound brings back a load threshold at the extreme.
    • At 75% event-loop starvation, the 1100 ms test now fails at 60 s instead of passing in 512 s (measured).
    • At 200 ms in every 250 ms it passes with 30x headroom (measured).
    • I asked for the bound. The builder chose 60 s, the bound the file already gives its real-Chrome tests, and said why.
  • A cheaper design could remove most of that threshold. It would tick the mock clock to the next due timer instead of 1 ms per loop turn, which cuts the turns per run from about 1,700 to about 20 (inferred). That would make a 60 s bound almost load-proof. It is a suggestion, not required.

Host tools in the builder's 75-file run (static read, not rerun)

The files selected for this PR are affected.mjs --event pull_request --base 6e0d7d81, 76 files (measured). add-project and setup-scripts are not among them (measured). From reading the code (inferred unless marked):

Files that set their own PATH reaching /usr/bin or /bin for a spawned process:

  • instance-records: runs instance-rename.sh and instance-remove.sh with PATH=/usr/bin:/bin and no stand-ins.
    • Every argument list is invalid, and each run must exit 2. The scripts source lib.sh and then parse arguments, exiting 2 before acting (instance-remove.sh:34-44, instance-rename.sh:50-62).
    • So no host tool should run, beyond bash and sourcing lib.sh. The top level of lib.sh only runs set -euo pipefail and defines functions and arrays (measured by grep for top-level commands).
  • owner-terminal-removed:
    • Lines 243 and 296 use PATH=<stub-bin>:/usr/bin:/bin, with stand-ins for sudo, bash and id, and a printing sudo at 287.
    • Its header says systemd, sudo and setfacl are stand-ins for the dashboards.sh rerun.
    • Line 290 runs only bash -c printf.
  • ci-preview-done: PATH=<bin>:$PATH. The stand-ins are sudo (it execs its arguments without privilege), journalctl and getent.
  • engine-update:569: sudo -u <user> env PATH=…:/usr/bin:/bin runs only under systemCi. That is GITHUB_ACTIONS === 'true' && sudo -n systemctl --version, which short-circuits off GitHub, so locally no sudo runs.
  • testrun-gpu and testrun-userns-unsupported: PATH=<fake bin> only. Privileged tools are stubbed. Only coreutils, plus install behind a wrapper that drops -o/-g, are linked from /usr/bin or /bin.

Files where the system PATH is only a string or a parsed value, and no spawn uses it:

  • bench:436 (handed to a fake docker), isolate:289, machine-helper:522, perf-dashboard-ids:94, and update-canary, update-migrate and update-sandbox (systemctl show text fed to a parser).

Files that use the inherited PATH for host tools:

  • machine-helper:
    • visudo -cf on temp files.
    • Helper copies, with /etc, the log and updates.sh rewritten to temp paths. They run as non-root, so needsUnit never reaches systemd-run (worklane-machine.cjs:665).
    • The claude-account and tailscale argument lists are all refused with exit 2.
  • updater-setup, updater-self and release-channel: git and ssh-keygen in temp dirs, and helper copies as non-root.
  • discord-setup: node only.

Net (inferred):

  • No file in the set should have reached a privileged host tool.
  • The two files without stand-ins that put /usr/bin:/bin first are instance-records (its argument-refusal tests) and owner-terminal-removed:290 (printf only).
  • I did not rerun any of them.

Issues #848 and #849

@jayden1711
jayden1711 merged commit f21583d into main Oct 20, 2026
27 of 28 checks passed
@jayden1711
jayden1711 deleted the issue-720 branch October 20, 2026 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: browser-run budget test measures wall time on CI (768 ms vs 600 ms on Node 24); prove the cap with an injected clock

1 participant