Skip to content

Tests: fix the budget-health second-boundary flake; weekly optional shifted-clock run; lint on clock-reading helpers (#687) [PROTECTED: owner merges] - #774

Merged
jayden1711 merged 5 commits into
mainfrom
issue-687-time-bombs
Oct 19, 2026
Merged

jayden1711 merged 5 commits into
mainfrom
issue-687-time-bombs

Conversation

@jayden1711

@jayden1711 jayden1711 commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Intent: build #687 as the owner restated it on 2026-10-18. Three things: (1) fix the one real flaky test the investigation found (GitHub budget health "per credential", off by one second); (2) add a small clock-shift helper that one optional weekly CI job uses to run the unit suite 45 and 400 days ahead; (3) add a lint, narrow on purpose, against test helpers that read the wall clock on every call. There is no deadline.

Risk class: protected. It adds .github/workflows/clock-shift.yml. .github/** is engine in .worklane/review.yaml, and a CI workflow is also a self-modification path: "the CI workflows", src/review.ts:40. Everything else is under test/, and none of it is in engine-guards or the self-modification parts. test/isolate.ts is unchanged.

Closes #687

The issue's premise was wrong

The issue said five intake tests would start failing on main around 2026-11-02 and had to be fixed before 2026-11-01. They are not date bombs.

  • Measured (previous builder, recorded in the owner's restatement on Tests: no result depends on today's date; five intake tests go red on main ~2026-11-02 (fix before 2026-11-01) #687): the five fail under any clock shift of more than 14 days, but only when Date moves and file timestamps don't. The intake store deletes raw reports whose files look older than 14 days (src/intake/store.ts), so a file written a second earlier looked 45 days old. With the file times shifted as well, all 40 tests in those three files pass.
  • Measured (this PR, with the helper as the weekly job loads it: TEST_CLOCK_SHIFT plus NODE_OPTIONS=--import=…/dist/test/clock-shift.js): intake-coordinator, intake-github and intake-replier gave tests 40 / pass 40 / fail 0 at both +45d and +400d.
  • So the issue's five tests have no deadline, and neither does this PR. The full suite under the shift does still have red tests: see "Findings from the full shifted run" below (measured).

What changed

  1. The flaky test (test/github-budget-health.test.ts). The test set a credential's reset time with at(600) and later expected at(600) again. at reads Date.now() in whole seconds each time it's called, so a second ticking between the two calls broke the comparison. The fix reads the reset time once and reuses it. The test now also runs on a fixed clock (node:test's mock.timers, Date only) set 1 ms before a second boundary. The setup moves that clock 2 ms, across the boundary, so the case that used to flake now runs on every run.
  2. The clock-shift helper (test/clock-shift.ts, 90 lines). It accepts only relative shifts (+45d, -2h, +1500ms; the instant form is gone). It moves Date's "now" and the file times fs reports and takes (stat/lstat/fstat and utimes/lutimes/futimes, in sync, callback and promise forms). It captures the real Date.now once, at load, and makes Date.now an ordinary property holding the shifted clock. A test that mocks Date.now therefore sees its own mock, and a mock that calls the clock it found there can't recurse (round 1: test/dashboard-auth.test.ts's wallReal = Date.now.bind(Date) mock recursed without end and hung the first run). Only the new workflow loads it for the suite, through NODE_OPTIONS. Because of that, the Node children a test starts run at the same shift. Those children were what made github-budget and real-chrome-lock hang when only the parent moved. test/isolate.ts is not edited, and the required shards never load the helper.
  3. The weekly job (.github/workflows/clock-shift.yml, workflow "clock shift", job "unit tests on a shifted clock"). It runs every Monday at 05:17 UTC, can be started by hand, and also runs on a PR that changes test/clock-shift.ts or the workflow itself, which is why it runs on this PR. It runs the whole unit suite at +45d and then at +400d. Each run uses --test-timeout=600000 --test-force-exit, and each step has a 75-minute limit, so a hang is named with its file and the next step still runs. Nothing requires it: not the ruleset, not .worklane/config.yaml, not the updater's checks. No ci.yml job or check name changes. I made it a separate workflow rather than adding a schedule: trigger to ci.yml, because that trigger would rerun every ci.yml job under the required names. --test-force-exit also hides leaked handles in this job only; the required shards run without it, so a leak still hangs them there (measured by the round-2 review).
  4. The lint (test/no-clock-helpers.test.ts). It counts helpers declared on one line (const f = (…) => … or function f(…) { … }) whose body reads Date.now() or new Date(). That is the shape of the helper that flaked. The 13 helpers already in test/ are allowed per file, each with an exact count that can only go down, the same ratchet test/no-wall-clock-durations.test.ts uses (Clocks: dashboard listing caches use a monotonic injectable clock, so a wall-clock step cannot keep or cut a cache window; lint new Date.now() durations in tests (v0.5.3) #582). A file with none can never gain one.

Why the lint is this narrow (the full version was measured and rejected)

  • Measured (scratch script clock-stats.mjs over test/): the full version, "a test reads the wall clock without an injectable clock", would flag 372 reads in 109 of 429 test files. A narrower "mixes a fixed ISO date with a wall-clock read and doesn't pin Date" rule still flags 130 reads in 29 files.
  • Inferred: many of those reads are correct. For example, a stamp "a minute ago" compared with a file's real mtime is the right use of the wall clock, and pinning Date there would be wrong, because a pinned Date can't move file times. A full lint would mostly be noise, and it would push people toward the wrong fix.
  • Inferred: the "fixed date mixed with the wall clock" rule would not have caught the one real failure (the budget health test has no fixed date). It also targets the date-bomb class, which the weekly job now catches by running the suite late.
  • The rule here is the narrowest one that targets the failure that actually happened. Its known gaps: it misses a helper whose body spans several lines, and a clock read repeated inline instead of through a helper (inferred).

How it was verified

  • Fail-without, the flaky test (measured): I put the second at(600) read back into the new test and kept the fixed clock. It failed: tests 10, pass 9, fail 1, with actual resetAt: '2026-10-19T03:06:22.000Z' and expected resetAt: '2026-10-19T03:06:23.000Z'. With the fix: tests 10, pass 10, fail 0. Command: node --import ./dist/test/isolate.js --test dist/test/github-budget-health.test.js, with a temporary HOME, state and held-out dirs and a fake claude first on PATH.
  • Fail-without, the shifted clock (measured, fixture): test/clock-shift.test.ts writes a fixture test in a temporary directory that has a 30-day window. It passes on the real clock and fails at +45d with "past the date". The other three tests in that file pass: Date and the file times move together, a Node child inherits the shift, and nothing moves without the preload or with an empty shift.
  • Fail-without, the lint (measured, fixtures only): source text is fed to the lint's own function, and no repo file is planted. The at helper that flaked and six similar helpers are each found. A clock read once and reused, a fixed date, the clock passed by reference, RealDate.now() and comments are not.
  • The two files that hung under the previous builder's shift (measured): with the job's NODE_OPTIONS, at +45d, github-budget.test.js gave 26/26 together with github-budget-health.test.js and clock-shift.test.js, and real-chrome-lock.test.js gave 7/7.
  • Fail-without, round 1, the Date.now mock (measured): I ran the new clock-shift.test.ts case on the old helper. It failed: a fixed mock read 3888012345 instead of 12345, and the wallReal pattern failed with Maximum call stack size exceeded. With the fix, the file passes 5/5, and dashboard-auth.test.js at +45d with the job's env passes 22/22 and exits.
  • Fail-without, round 1, the job's flags (measured, scratch fixture): I used a fixture test that never finishes and keeps an interval open. Plain node --test never ends: timeout 60 exits 124. With --test-timeout=5000 --test-force-exit, it exits 1 after about 5 s and names hangs.test.mjs "test timed out after 5000ms".
  • Related tests (measured): 132 tests across clock-shift, no-clock-helpers, no-wall-clock-durations, isolate (the "every way the suite runs preloads the isolation" check reads the new workflow line), and the workflow-reading tests ci-slow-fixture-sha, ci-preview-done, ci-slow-trigger, ci-batch, update-checks, web-ui-required, testrun-tmpdir, self-hosted-guard, ci-new-deps, push-gate, release-workflow and starter-ci. That first run had one failure: the lint counted RealDate.now() inside the helper itself. I narrowed the lint's pattern so a name that merely ends in Date doesn't match, added a fixture for that case, and the rerun of the lint tests passed 8/8.
  • npm run typecheck: clean (measured). npm run check:denylist: 1277 files clean (measured). npm run check:dco -- origin/main HEAD: all commits signed off (measured). npm run check:push: 3 commit(s) and 1 tip(s) clean (measured).
  • The full suite under the shift (measured, CI): this PR's pull_request trigger ran the workflow (run 39204725103, head d781239c). workflow_dispatch can't start it yet, because GitHub only dispatches a workflow file that exists on the default branch. The job finished in 26m29s, and both steps ran to the end:
    • +45d: tests 3543, pass 3539, fail 2, cancelled 0;
    • +400d: tests 3543, pass 3537, fail 4, cancelled 0.
    • On this head, every check except this optional job passes, the required ones included: 27 pass (measured, gh pr checks 774). This job is red because of the findings below.

Findings from the full shifted run (for the owner; no file is excluded or skipped)

File +45d +400d Likely cause
test/pr-hygiene-coordinator.test.ts "a stale PR of the harness gets main merged in…" (line 20) and "an abandoned PR … is closed…" (line 60) fail fail The PR's age comes from git commit times, and git isn't a Node process, so it stamps commits with the real clock. The shifted coordinator then sees a fresh PR as 45 or more days stale: it merges main in on the "fresh" pass, and it closes, where the test expects a skip (inferred from the test and the failure counts). This is not a date bomb on a real machine.
test/claude-account-script.test.ts "switch pauses every instance, installs the token…" (a year, line 183) pass fail expiresAt is written by a Node child of the script (scripts/setup/lib.sh:625), which runs on the real clock because the test starts the script with env: { PATH, HOME } only (test/claude-account-script.test.ts:136), dropping NODE_OPTIONS and TEST_CLOCK_SHIFT; the test compares it with Node's shifted Date.now() + 300 days (measured by the round-2 review: passing both variables through gives 22/22 at +400d). Faking time for the shell alone would not fix it.
test/dashboard-switch-speed.test.ts "gated, 120 ms, 10 Mbit/s: … under 300 ms (median of 5)" (line 328) pass fail A timing test: card count: 0 on switch 5 of 5. It passed at +45d in the same job, so it looks like runner load rather than the date (guess).

All three come from the helper's known limit, which is that only Node processes move, plus one timing test. None of them is a test whose result depends on today's date on a real machine (inferred). Options, for the owner to choose: give those tests' git and shell steps a fake time (for example libfaketime in the weekly job only), mark the stale-PR tests as "needs the whole machine's clock", or keep the job red as a report. This PR picks none of them (rule 10: nothing is skipped to get green).

New dependencies

None.

What this can never do

  • Shift the clock in a required check. Only clock-shift.yml sets TEST_CLOCK_SHIFT and NODE_OPTIONS for the suite. In the required shards, test/clock-shift.test.ts sets them only for the probe and fixture child processes it starts, never for itself or another test file. The helper does nothing when TEST_CLOCK_SHIFT is unset or empty, and test/isolate.ts doesn't import it (inferred from the diff; the "not shifted" test measures that a preloaded process with TEST_CLOCK_SHIFT set but without the job's NODE_OPTIONS reads the real clock).
  • Change engine behaviour: no src/ or scripts/ file changes, and the helper compiles to dist/test/, which package.json files doesn't publish (dist/src, dist/web, …) (measured: read package.json files).
  • Rename or remove a required check: no ci.yml job changes (inferred from the diff).

Compromised component

  • Agent user / coordinator / dashboard / other instances / hub viewer: none of them runs the new workflow or loads the helper. It is CI-only test code.
  • Someone with push access to a PR branch: they could already edit any workflow. This workflow has permissions: contents: read, uses no secrets, and its PR trigger is limited to two paths. A PR that changes the helper runs the suite with it loaded, as ci.yml already runs any PR's test code.
  • The weekly run on main: it runs main's own code with read-only permissions. A red run reports and blocks nothing, because nothing requires it.
  • A test that sets NODE_OPTIONS itself (for example, push-gate-hook's frozen-time import): in the weekly job, that test replaces the job's value for its own child, so that child runs on the real clock while the parent is shifted. That could produce a false red in the optional job only, never in a required check (inferred). On this PR's full run, push-gate-hook passed at both shifts (measured: it isn't among the failures).
  • The workflow as a self-modification path: a later change to clock-shift.yml changes what this CI runs, so it stays engine and self-modification, and the owner merges it. It holds no secrets and can't push.

…time once, on a fixed clock

The test set a credential's window to reset at at(600) (now + 600 s, in whole seconds) and later expected
at(600) again. When a second ticked between the two reads (the budget's takes and the view run in between), the
expected reset was one second later than the stored one and the test failed (#687).

The reset time is now read once and reused, and the test runs on a fixed clock (node:test's mock Date) set 1 ms
before a second boundary, which the setup crosses, so the case that flaked runs every time.

Test (fails without): test/github-budget-health.test.ts "per credential" with the second at(600) read put back
fails on the fixed clock every run: resetAt '...:22.000Z' stored, '...:23.000Z' expected.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
…days ahead

A test whose result depends on today's date passes the day it is written and fails on main weeks later (#687).
The new clock-shift workflow runs the whole unit suite with TEST_CLOCK_SHIFT=+45d, then +400d, once a week (and
on a PR that changes the shift or the workflow, and by hand). It is required by nothing, and no ci.yml check
changes.

test/clock-shift.ts moves Date's "now" and the file times fs reports and takes by the same amount. The file times
move too because the engine compares the clock with mtimes: with only Date moved, the intake purged a file written
a second earlier as 14 days old, which is what failed the five intake tests #687 first named. The job loads it
through NODE_OPTIONS, so the Node processes a test starts run at the same shift: test/github-budget.test.ts and
test/real-chrome-lock.test.ts start Node children without the test preload, and hung when only the parent moved.
test/isolate.ts is unchanged: the required shards never load the shift.

Test (fails without): test/clock-shift.test.ts "a test that only passes before a date fails on the shifted
clock": a fixture test with a 30-day window passes on the real clock and fails at +45d. The other tests check that
Date and the file times move together, that a Node child inherits the shift, and that nothing moves without it.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
The flake #687 found came from a helper, `const at = (s) => Math.floor(Date.now() / 1000) + s`: two calls read two
clocks, and a test that set a value with one and expected it with the other failed whenever a second ticked in
between. test/no-clock-helpers.test.ts counts such helpers (declared on one line, `const f = (...) => ...` or
`function f(...) { ... }`, whose body reads Date.now() or new Date()) in every file under test/. The 13 already there
are allowed by file, each count exact and only ever lowered, as test/no-wall-clock-durations.test.ts does (#582);
a file with none may never get one.

Narrow on purpose: every wall-clock read in test/ is some 370 reads in over 100 files, and many are right (a stamp
some while ago compared with a file's real mtime, which a pinned Date cannot move).

Test (fails without): the lint's fixtures, source text fed to its function (no repo file planted): the helper
that flaked and six of its kin are each found; a clock read once and reused, a fixed date, the clock passed by
reference, another Date's now() and comments are not.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: CHANGES NEEDED (round 1/2)

Reviewed head 5001a0bd against #687 as the owner restated it on 2026-10-18. The flake fix and the lint hold up. The weekly job doesn't work yet: on this PR it hung at +45d until the 2 h timeout, and the +400d step never ran. The cause is a bug in the helper, which I reproduced locally.

Acceptance criteria (restated #687)

# Criterion Status Evidence
1 No deadline met The PR body drops the deadline. The intake tests passed under the helper: measured, see below.
2 Fix the budget-health "per credential" flake met The test reads at(600) once (test/github-budget-health.test.ts:57,73) on a fixed clock that crosses a second boundary. Fail-without below (measured).
3 A smaller clock-shift helper (Date plus file times, inherited by Node children), used by one optional weekly job only partly met Not required: it is absent from the ruleset's required checks (gh api repos/jayden1711/worklane/rules/branches/main) and from .worklane/config.yaml required_checks (measured). Not on any shard: test/isolate.ts is unchanged and no other workflow loads it (measured, grep). But the job doesn't finish: see required change 1.
4 A lint, or the narrowest version that holds met test/no-clock-helpers.test.ts sets a per-file ratchet. It runs on any PR, because scripts/ci/affected.mjs selects it for a change to test/web-render.test.ts with "always: reads the whole repo" (measured). The narrowing is argued with counts (measured by the builder; I didn't rerun clock-stats.mjs).

Fail-without (measured)

  • Flake: I put the second at(600) read back into the PR's test (expected resetAt: new Date(at(600) * 1000)) and ran node --import ./dist/test/isolate.js --test dist/test/github-budget-health.test.js with a temporary HOME, temporary state and held-out dirs, and a fake claude. It failed two runs out of two: tests 10 / pass 9 / fail 1, actual resetAt: '2026-10-19T04:40:59.000Z', expected '...04:41:00.000Z'. The same file as committed passes: tests 10 / pass 10.
  • Main plus the PR's test files: github-budget-health, clock-shift and no-clock-helpers give 16/16 on main. That's expected: the fix lives in the test itself, so the reverted read above is the real fail-without. clock-shift.test.ts needs the new test/clock-shift.ts, so on main it can only be compiled together with it. Its fail-without is internal: a fixture passes on the real clock and fails at +45d.
  • Lint: fixture strings are fed to clockHelpers(), and no repo file is planted (measured: read the test). 8/8 pass.

Head checks (measured, in my own worktree, temporary HOME and state)

  • npm run typecheck: clean. npm run check:denylist: 1277 files clean.
  • github-budget-health, clock-shift, no-clock-helpers, no-wall-clock-durations and isolate.test: 42/42.
  • With the job's TEST_CLOCK_SHIFT + NODE_OPTIONS at +45d, the PR's tests plus github-budget and intake-coordinator/-github/-replier: 68/68. At +400d, the PR's tests plus the three intake files: 54/54. At +45d, network-tls, github-hosts, push-gate-hook and real-chrome-lock: 44/44.

The weekly job hangs (measured)

  • CI: unit tests on a shifted clock failed after 2h0m27s. The log has no output from 03:16:08 to 05:09:54 ##[error]The operation was canceled., and the step "the tests, 400 days later" never ran.
  • I matched every test file's names against that log. 309 of 403 files never reported, and the first one is dashboard-auth.test.ts (measured). I can't tell from the log whether the files after it pass under the shift (inferred: the hang blocked their output).
  • Locally at +45d with the job's env, dist/test/dashboard-auth.test.js never exits: timeout 240 returned 124. On the real clock it passes 22/22 in 6.1 s and exits 0. The #622: on the hub, a wall-clock step back/forward tests fail with Maximum call stack size exceeded. The stack alternates between Proxy.now (dist/test/clock-shift.js:77) and dashboard-auth.test.js:348. That test does const wallReal = Date.now.bind(Date); t.mock.method(Date, 'now', () => wallReal() + offset.wall).
  • Cause (inferred from that stack): the Proxy's now calls RealDate.now(), read on every call. mock.method(Date, 'now') goes through the Proxy to the real Date and replaces RealDate.now with the mock. The mock calls wallReal, which is the Proxy's now, which calls the mock again, and so on without end. A small probe confirms the related part: under the shift, mock.method(Date, 'now', () => fixed) leaves Date.now() returning fixed + 45d, not fixed. On the real clock it returns fixed (measured).

Protection

  • .github/workflows/clock-shift.yml is engine in main's .worklane/review.yaml. It is also self-modification: "the CI workflows", src/review.ts:40 (measured: read both). The PR is correctly marked protected: the title, **Risk class:** protected, the needs-owner label, and both sections are present (measured). The other four files are under test/, and none is in engine-guards or SELF_MODIFICATION_PARTS (measured).
  • "Can never do" claims: the claim that it shifts no required check holds. Only the new workflow loads the helper for the suite, and clock-shift.test.ts shifts only its own probe children (measured: grep and read). "Only clock-shift.yml sets TEST_CLOCK_SHIFT and NODE_OPTIONS" is too broad as written, because clock-shift.test.ts sets both for its children in the required shards, harmlessly (inferred). The claim that it changes no engine behaviour holds: no src/ or scripts/ files change (measured: diff).
  • Compromised component: the workflow has permissions: contents: read and no secrets, and the action SHAs and the gitleaks checksum match ci.yml (measured). Nothing found missing.
  • No regression for existing installs: not applicable, since the PR touches no setup, firewall, sandbox, locks or visibility (inferred from the file list).
  • Owner's real state (rule A): the tests write only under tmpdir() and run under the isolation preload, and the workflow runs on CI only (inferred from reading them). I ran everything here with a temporary HOME and state.

Scope

Hygiene (measured)

  • All 3 commits have Signed-off-by. There are no Co-Authored-By or vendor lines in the commits or the body, and no new dependencies. Nothing changes .claude/ or src/. Closes #687 is present, and the body's claims are labelled.

CI (measured, gh pr checks 774)

  • Every required check passes: test (ubuntu-latest, 22/24), desktop, agentshield, push-gate, denylist, dco, scan, new-deps. The ci-slow jobs pass too.
  • unit tests on a shifted clock fails (timeout). The cause is this PR's helper, as shown above.

Required changes

  1. Make the helper work with tests that mock Date.now. Capture the real now once (for example const realNow = RealDate.now.bind(RealDate)), and don't call RealDate.now() from inside the Proxy. Make sure a test's t.mock.method(Date, 'now', …) both takes effect and can't recurse, including the Date.now.bind(Date) pattern at test/dashboard-auth.test.ts:348. Add a case to test/clock-shift.test.ts that fails without the fix: a shifted child that mocks Date.now with that pattern finishes and sees the mocked value.
  2. Show the weekly job finishing on this PR at both shifts. Either both steps go green, or any red is named per file and either fixed or explicitly reported with its cause. Consider a per-file --test-timeout in the job, so a future hang names its file instead of timing out silently after 2 h.
  3. Update the PR body with the job's measured result. The current claims "nothing on main goes red on a date" and "no file needed to be excluded" came from a subset of the suite, and the full run hung. Also narrow the "only clock-shift.yml sets TEST_CLOCK_SHIFT and NODE_OPTIONS" line (see Protection).

…t's Date.now mock works and can't recurse

The weekly clock-shift job hung for its whole 2 h on #774 at +45d. test/dashboard-auth.test.ts mocks the clock with
`const wallReal = Date.now.bind(Date); t.mock.method(Date, 'now', () => wallReal() + offset)`. The Proxy's now
called RealDate.now() on every read, and the mock had replaced RealDate.now, so the mock called the shifted clock,
which called the mock, until "Maximum call stack size exceeded". A plain mock didn't take effect either:
Date.now() returned the mocked value plus the shift.

The real clock is now captured once at load, and Date.now becomes an ordinary property holding the shifted clock.
A test can replace it, the replacement takes effect, and anything it calls reaches the captured real clock.

Test (fails without): test/clock-shift.test.ts "shifted: a test that mocks Date.now sees its mock, and a mock
calling the clock it found there finishes": on the old helper, a fixed mock read 3888012345 instead of 12345, and
the wallReal pattern failed with "Maximum call stack size exceeded". dashboard-auth.test.js at +45d with the job's
env now gives 22/22.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
…h shifts

On #774 one hung file held the +45d step until the job's 2 h timeout. The log was silent, and the +400d step never
ran. Each step now runs with --test-timeout=600000, so a test that hangs is cancelled after 10 minutes and named
with its file in the summary, and with --test-force-exit, so a file whose tests are done exits even with a handle
left open. Each step also has its own 75-minute limit, so +400d runs whatever +45d does, and the job's limit is
170 minutes.

Test (fails without): a scratch fixture test that never finishes and keeps an interval open. Plain
`node --test` never ends: `timeout 60` exits 124. With the job's two flags (timeout 5 s), it exits 1 in about 5 s
and names `hangs.test.mjs` "test timed out after 5000ms".

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Round 1 fixes (head d781239c)

  1. Helper fixed (925c9d30). test/clock-shift.ts now captures RealDate.now once, at load. Date.now is an ordinary property holding the shifted clock, and the Proxy has no get trap any more. A test's t.mock.method(Date, 'now', …) takes effect, and a mock that calls Date.now.bind(Date) reaches the captured real clock, so it can't recurse.
    • Fail-without (measured): I ran the new test/clock-shift.test.ts case "a test that mocks Date.now sees its mock, and a mock calling the clock it found there finishes" on the old helper. The fixed mock read 3888012345 instead of 12345, and the wallReal pattern failed with Maximum call stack size exceeded. With the fix: 5/5.
    • dashboard-auth.test.js at +45d with the job's env: 22/22, and it exits (measured).
  2. The job finishes and names files (d781239c). Each step runs --test-timeout=600000 --test-force-exit and has timeout-minutes: 75. The job's limit is 170 minutes.
    • Fail-without (measured, scratch fixture: a test that never finishes and keeps an interval open): plain node --test exits 124 under timeout 60. With the flags (timeout 5 s), it exits 1 and names hangs.test.mjs "test timed out after 5000ms".
    • How it ran on this PR: through the workflow's pull_request path trigger, run 39204725103. workflow_dispatch can't start it, because GitHub only dispatches a workflow file that exists on the default branch. The job finished in 26m29s, and both steps ran (measured):
      • +45d: tests 3543, pass 3539, fail 2: pr-hygiene-coordinator.test.ts lines 20 and 60.
      • +400d: tests 3543, pass 3537, fail 4: the same two, plus claude-account-script.test.ts line 138/183 ("a year") and dashboard-switch-speed.test.ts line 328.
    • No file is excluded or skipped. The PR body lists all of these as findings for the owner, with their likely causes: git and the shell script stamp the real clock while Node's is shifted (inferred), and the speed test looks like runner load (guess).
  3. Body updated:
    • I replaced "nothing on main goes red on a date" and "no file needed to be excluded" with the measured full-run result.
    • I narrowed the TEST_CLOCK_SHIFT/NODE_OPTIONS claim: clock-shift.test.ts sets both for its own children.
    • I named the workflow as a self-modification path (src/review.ts:40).

Checks (measured): npm run typecheck clean; check:denylist 1277 files clean; check:dco -- origin/main HEAD all signed off; check:push 5 commit(s) and 1 tip(s) clean. Related tests (clock-shift, no-clock-helpers, no-wall-clock-durations, isolate, self-hosted-guard, update-checks, ci-batch, testrun-tmpdir): 68/68. On CI, every check except the optional clock-shift job passes.

@jayden1711

Copy link
Copy Markdown
Owner Author

Evaluator verdict: PASS

Round 2 of 2, head d781239c. Both round-1 code changes are done and hold up: the helper no longer recurses, and the weekly job now finishes and names each failing file. Required CI is green. The shifted job is red with 2 findings at +45d and 4 at +400d. As briefed, that is a report and doesn't block. One claim in the body's findings table names the wrong cause. The correct cause is below (measured), and the body should be fixed before merge (see "Before merge"). It's text only, so it doesn't block.

All local runs used my own worktrees with a temporary HOME, temporary state and held-out dirs, and a fake claude first on PATH. Local Node is v22.23.2; the job uses 24.

Round-1 required changes

# Change Status Evidence
1 The helper works with tests that mock Date.now and can't recurse met test/clock-shift.ts:77-86 captures realNow once and makes Date.now a plain property; the Proxy has no get trap. Measured: see "Fail-without" and "The clock".
2 The weekly job finishes on this PR at both shifts, with each red named met Run 39204725103: +45d gives tests 3543 / pass 3539 / fail 2 / cancelled 0, and +400d gives tests 3543 / pass 3537 / fail 4 / cancelled 0, in 26m29s (measured: the job log).
3 The body gives the measured full-run result and narrows the NODE_OPTIONS claim met, one wrong cause The full-run counts and the narrowed claim are in the body (measured: read it). The claude-account-script row gives the wrong cause (see the findings below).

Acceptance criteria (#687 as restated by the owner)

Criterion Status Evidence
No deadline met The intake files are not date bombs. The body says so, and round 1 measured it.
The budget-health "per credential" flake is fixed met The test reads at(600) once, on a fixed clock that crosses a second boundary. Round 1 measured the fail-without. I measured 10/10 now on the real clock, at +45d and at +400d.
A small clock-shift helper, used only by one optional weekly job met test/isolate.ts is unchanged, and no other workflow loads the helper (measured: grep). It isn't a required check: the ruleset requires agentshield, dco, denylist, new-deps, push-gate, scan, test (ubuntu-latest, 22/24) (measured: gh api .../rules/branches/main). It isn't in .worklane/, src/ or scripts/ either (measured: grep).
A lint, narrow on purpose met Unchanged since round 1. 8/8 pass (measured).

1. The Date.now capture (measured)

  • Fail-without. In a scratch worktree at round 1's head (5001a0bd, the old helper), I checked out the new test/clock-shift.test.ts and ran it with node --import ./dist/test/isolate.js --test dist/test/clock-shift.test.js. Result: tests 5 / pass 4 / fail 1. The failing case is "a test that mocks Date.now sees its mock", with actual: 3888012345 against expected: 12345 (that's 12345 plus 45 days) and error: 'Maximum call stack size exceeded'. At the head, the same file passes 5/5.
  • dashboard-auth at +45d, with the job's env and flags: tests 22 / pass 22 / fail 0, exit 0, in 5.1 s. That includes all four #622 wall-clock step tests, which use the wallReal = Date.now.bind(Date) mock. Without --test-force-exit it also exits by itself: 22/22, exit 0. So the fix doesn't depend on force-exit. At +400d, dashboard-auth, clock-shift and github-budget-health together give 37/37.
  • The clock. I wrote a probe (scratch probe.mjs) and ran it under TEST_CLOCK_SHIFT=+45d with the job's NODE_OPTIONS, then without the shift:
    • new Date(), Date() and Date.now() agree: 0 ms apart, and Date() matches to the second.
    • A fixed mock reads 12345, and the clock is shifted again after restore.
    • The wallReal pattern gives -600 s and +600 s steps with no recursion.
    • A direct Date.now = … assignment and its restore work, mock.timers with Date reads its own clock and restores, and class X extends Date gets the shifted now.
    • All of this matches the unshifted run exactly. As in plain Node, new Date() doesn't follow a Date.now mock in either run.
  • performance.now and hrtime are untouched. Over a 50 ms spin they read 49.47 ms and 49.43 ms. performance.timeOrigin + performance.now() - Date.now() is -3887999999 ms, which is exactly 45 days, so perf time stays on the real clock as the header says. One side effect follows (inferred): code that builds an epoch from performance.timeOrigin disagrees with Date by the shift. That's a harness limit, already listed under "What doesn't move".

2. --test-timeout=600000 --test-force-exit (measured, scratch fixtures)

  • I used a fixture whose test passes but leaves an interval open. Plain node --test never ends: timeout 30 exits 124. With the job's flags it exits 0 and passes. So yes, --test-force-exit hides a leaked handle that a normal run would surface as a hang.
  • A test that never finishes is caught. Plain node --test exits 124. With --test-timeout=5000 --test-force-exit it exits 1 with not ok 1 - hang.test.mjs … 'test timed out after 5000ms'. The timeout lands on the file-level test, so a whole file over 10 minutes would fail too. No file came close in this run (measured: no timed out failures in the job log).
  • Only the weekly job is affected. The flags appear only in .github/workflows/clock-shift.yml. ci.yml:97 and package.json test run node --import ./dist/test/isolate.js --test without them (measured: grep). The PR doesn't change ci.yml. The required shards therefore still turn a leaked handle into a hang, and the weekly job's blind spot is covered by them.
  • Inferred: the one thing lost is a leaked handle that appears only on a shifted clock while every assertion passes, for example a timer scheduled from a far-off date. That's narrow, and the workflow's comment discloses the trade ("exits even with a handle left open"). I accept it for an optional report job. The 75-minute steps and the 170-minute job fit: 2 × 75 plus setup.

3. The shifted-clock findings: my judgement per test

Test Kind Cause (from the code) What I recommend to the owner
pr-hygiene-coordinator :20 "a stale PR…" and :60 "an abandoned PR…" (both shifts) Test-harness artefact. It isn't a product date bug and it isn't flaky. It fails every time when the shift is larger than the thresholds. The builder's cause is right. src/coordinator/pr-hygiene.ts:72 and :87 read the commit time with git log --format=%ct, and the test harness's commits come from git (test/repro-state-graph/harness.ts:18), which stamps them with the real clock. The shifted hygieneNow therefore sees a base that is already 45 days old: merged_in 1 !== 0 on the "fresh" pass, and closed: 1 where skipped: 1 is expected. Measured: in a scratch worktree I gave the harness's git helper GIT_COMMITTER_DATE/GIT_AUTHOR_DATE set from Node's Date.now(). The file then passed 6/6 at +45d, at +400d and on the real clock. Don't mark it. The stale and abandon logic is exactly the date-dependent code this job exists to watch. Best fix: a follow-up issue to date the harness's commits from Node's clock (test-only, about 2 lines, measured above). libfaketime for git in the weekly job would also work (inferred), but it adds a system dependency to the job. Until one of those lands, keep the job red as a report.
claude-account-script :138/:171 "a year" (+400d only) Test-harness artefact. It isn't a product date bug and it isn't flaky. It fails every time once the shift is over 65 days (365 < shift + 300). The builder's cause is wrong. expiresAt isn't written by the shell. It's written by a Node child, node --input-type=module -e at scripts/setup/lib.sh:625 (Date.now() + 365 days). That child runs on the real clock because the test starts the script with env: { PATH, HOME } only (test/claude-account-script.test.ts:136), which drops NODE_OPTIONS and TEST_CLOCK_SHIFT. Measured: in a scratch worktree, passing those two variables through that env gave tests 22 / pass 22 at +400d. Without them it reproduces locally: not ok 1 … 'wl-alpha-agent: a year'. Don't mark it, and libfaketime won't fix it. An LD_PRELOAD/FAKETIME set in the environment would be stripped by the same env: {…} (inferred). Best fix: a follow-up issue to pass TEST_CLOCK_SHIFT and NODE_OPTIONS through in that test's spawn env when they're set (test-only, measured above). Until then, keep it red as a report.
dashboard-switch-speed :328 "gated, 120 ms, 10 Mbit/s…" (+400d only) Flaky, already known. It isn't the date. The same assertion, busy's Board after its fresh answer, is open issue #639. That issue records failures on unshifted ci.yml runs (2 in about 42 after 2026-10-18). This run's new diagnostic line shows board-instance-error: "busy: can't load its board: instance busy's dashboard isn't answering on 127.0.0.1:44639 (read ECONNRESET)" (measured: the job log). That fits the mechanism #639 infers: one failed fresh read drops the board already shown. It passed at +45d in the same job. The builder's "runner load" label (guess) is consistent. Inferred: not date-related, since the browser isn't shifted and the failure is a dropped loopback connection. Keep it red as a report, with nothing marked for the clock-shift job. Attach this log line to #639 as evidence: it's the first failure there that says which check failed and why.

None of these is skipped or excluded in the PR (measured: the diff; no skip or todo was added. Each step's skipped 2 are the two existing real-agent-user tests that need WORKLANE_TEST_AGENT_USER). That complies with rule 10.

4. PR body

  • Every claim is labelled, and the full-run numbers are cited from run 39204725103 (measured: read the body).
  • The workflow is named as self-modification ("the CI workflows", src/review.ts:40) and as engine (measured: both files read).
  • Two inaccuracies:
    • (a) The claude-account-script cause is wrong, as above. It matters, because it points the owner at libfaketime, which wouldn't fix that test.
    • (b) "(test/clock-shift.ts, 82 lines)": the file has 90 (measured: wc -l). That's trivial.
  • --test-force-exit hiding leaked handles is disclosed in the workflow comment but not in the body. I'd add one line saying so.

Protection (measured)

  • .github/workflows/clock-shift.yml is engine (.worklane/review.yaml:7) and self-modification (src/review.ts, "the CI workflows").
  • The PR is marked protected: [PROTECTED: owner merges] in the title, **Risk class:** protected, the needs-owner label, and real ## What this can never do and ## Compromised component sections.
  • None of the four test/ files is in engine-guards or SELF_MODIFICATION_PARTS.
  • "Can never do" holds:
    • No required check loads the helper.
    • No src/ or scripts/ file changes.
    • No ci.yml job changes.
  • The workflow has permissions: contents: read and no secrets.
  • No regression for existing installs: not applicable, since nothing touches setup, firewall, sandbox, locks or visibility (inferred from the file list).
  • Owner's real state (rule A): the new tests write only under tmpdir() and run under the isolation preload (inferred from reading them, and all my runs used temporary dirs).

5. Overlap with #790 and merging with main (measured)

  • git merge-tree of d781239c with 5b8618e0 is clean, and so is the merge with current origin/main ce8a5c50 (egress-firewall: drop each covered user's subordinate uids too, not only its own uid (#740) [PROTECTED: owner merges] #778 landed since).
  • Test: budget health reads the limited credential's reset time once #790 (e599df00) makes the same resetLimited change to test/github-budget-health.test.ts, and its hunks are identical to this PR's. Merging the two heads is clean, and the merged file equals this PR's version, so whichever lands second is a no-op on that file.
  • On ce8a5c50 merged with this head (built in a scratch worktree, not committed), the real clock gives 92/92 for clock-shift, no-clock-helpers, github-budget-health, isolate.test (its import changed on main), no-wall-clock-durations, dashboard-auth (changed on main, and wallReal is still at :344), update-checks, self-hosted-guard and ci-batch. At +45d, clock-shift, github-budget-health and dashboard-auth give 36/36.

6. Usual checks (measured)

  • Fail-without: see section 1. The flake fail-without was measured in round 1, and the hang fixture in section 2.
  • At the head: npm run typecheck is clean, and npm run check:denylist reports 1277 files clean. clock-shift, no-clock-helpers, github-budget-health, isolate.test, no-wall-clock-durations and dashboard-auth pass 65/65.
  • All 5 commits have Signed-off-by: Jayden Vasquez. There are no Co-Authored-By or vendor lines in the commits or the body, no new dependencies, and nothing under .claude/ or src/. Closes #687 is present.
  • CI (gh pr checks 774): every required check passes: agentshield, dco, denylist, new-deps, push-gate, scan, test (ubuntu-latest, 22) and test (ubuntu-latest, 24). All shards and every ci-slow job pass too. Only the optional unit tests on a shifted clock fails, for the findings above. That's this PR's job reporting, not a defect in the PR.

Scope

Nothing outside #687. The pull_request path trigger is how the job gets checked, which is reasonable (inferred).

Before merge (body only, not blocking)

  1. Fix the claude-account-script row. The cause is a Node child (scripts/setup/lib.sh:625) on the real clock, because the test's spawn env drops NODE_OPTIONS and TEST_CLOCK_SHIFT. libfaketime wouldn't reach it.
  2. Change "82 lines" to 90, and add one line saying --test-force-exit hides leaked handles in this job only, with the required shards still catching them.

Proposed follow-ups (owner's choice; none is in this PR)

@jayden1711

Copy link
Copy Markdown
Owner Author

Round-2 verdict (6260924401) body fixes, made by the orchestrator: claude-account-script's cause corrected (a Node child on the real clock because the test passes only PATH and HOME), clock-shift.ts is 90 lines, and a line that --test-force-exit hides leaked handles in the weekly job only. Body only; no code change.

@jayden1711
jayden1711 merged commit c339175 into main Oct 19, 2026
29 of 30 checks passed
@jayden1711
jayden1711 deleted the issue-687-time-bombs branch October 19, 2026 07:11
jayden1711 added a commit that referenced this pull request Oct 25, 2026
)

Round 1 dropped --test-force-exit and relied on --test-timeout to end a file whose process doesn't exit. That holds
on Node 22 only: on Node 24, which the job uses, a file with a handle left open is never ended, so the step would
wait for its 75-minute limit with no summary (#774 again; the new open-handle test failed on the Node 24 shard).

scripts/ci/clock-shift-run.mjs runs every test file in a `node --test` of its own, without force-exit, as many at
once as node --test does. A file passes only when it exits 0 with a summary; one still running after --file-timeout
(25 minutes in the job) is killed with its process group and named, and the run fails. The run ends with totals and
the failed files.

The tests read both steps from the workflow and require their exact command lines (no `|| true`, no
continue-on-error on any step or the job), then run each step on fixtures: a test registered after an await runs; a
file with an open handle, a failing test and a hanging test each fail the step, named; no test file fails it.

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests: no result depends on today's date; five intake tests go red on main ~2026-11-02 (fix before 2026-11-01)

1 participant