test(daemon): de-flake rebuild/watchdog/sched timing tests for slow -race Windows CI#5898
Merged
Conversation
… on slow -race runners Harden the cmd/grafel rebuild test cluster that reddens on windows-latest under `go test -race` (2-10x slower, contended). No product changes — these were measurement-timing / tight-threshold test flakes, verified by the product parallelising correctly on macos+ubuntu. - TestRebuildSemaphoreCapRespected: replace the fixed 30ms sleep window (peak concurrency observed via scheduler luck → flaked at peak=1) with a deterministic 2-party barrier. The first two workers park until both have arrived, so they are provably in-flight simultaneously; peak>=2 is guaranteed by construction. Cap<=2 is still enforced by the semaphore/gate. A generous barrier timeout turns a hypothetical single-slot regression into a peak assertion failure rather than a hang. - TestRebuildWatchdogFailure_PersistedAndSurfacedInStatus: the "stuck" repo must breach the watchdog while "fast" must not, in one shared call — an intrinsically wall-clock assertion. Inject RepoTimeout=2s (was 300ms) giving the instant "fast" repo ample headroom over its per-repo bookkeeping cost. - TestRebuildPerRepoTimeoutSurfacesStalledRepo: bump the per-repo watchdog 1.5s -> 3s for the same fast-repo-headroom reason; still well under the 10s ceiling the test asserts. Stability: GOMAXPROCS=2 go test -race -count=50/-count=20 green locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
Two same-class timing-fragile daemon tests that pass in isolation but flake under full-suite parallel load on slow -race runners. Product logic unchanged. - TestBoot_WatcherSubscriptionDoesNotBlockBind: replace the 5s time.Sleep stall (raced a 4s serve deadline) with an INDEFINITE channel block the test releases after confirming the daemon served. Because the stall never clears on its own, the only way an RPC is answered within the deadline is if watcher subscription is off the boot critical path — a synchronous regression misses by an infinity, not a slim margin. Serve deadline widened 4s -> 10s (pure headroom now that the wall-clock race is gone). - TestGroupAlgoReArmsOnNewLinkCompletion: widen GroupAlgoDebounce 200ms -> 2s so the re-arm race is decided structurally: /a's group-algo timer cannot fire for ~2s, but /b is enqueued at ~100ms and re-arms it far inside that window even on a heavily contended runner. The old 200ms could let /a's original timer fire before /b's re-arm landed. Stability: GOMAXPROCS=2 go test -race -count=30 green locally for both; full `go test -race ./cmd/grafel/ ./internal/daemon/...` = 1070 passed / 20 packages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the cluster of pre-existing timing-fragile daemon tests that flake on the slow, contended
-racewindows-latest CI runner (unrelated to the v0.1.9 memory epic — these are hardcoded-real-time-threshold tests). Surfaced across v0.1.9 acceptance CI runs. Refs #5850.All 6 audited tests are test-fragility, not product bugs — product parallelism + watchdog behavior confirmed correct; full
-racesuite green throughout.TestRebuildSemaphoreCapRespected: replaced scheduler-luck overlap (Sleep(30ms)) with a deterministic 2-party barrier sopeak≥2holds by construction; real semaphore still enforcescap≤2.TestRebuildSuccess_ClearsPriorFailureMarker(the run-PORT-2: port YAML rule engine + framework rules #3 failure): success path now disables the watchdog (RepoTimeout:"0"), removing wall-clock dependence; the must-time-out path usesfailFnthat blocks forever (deterministic).TestRebuildWatchdogFailure_.../TestRebuildPerRepoTimeoutSurfacesStalledRepo: intrinsic-timeout thresholds inflated with generous headroom over per-repo bookkeeping cost (300ms→2s, 1.5s→3s), still under the asserted ceilings.TestGroupAlgoReArmsOnNewLinkCompletion(sched): debounce 200ms→2s so the re-arm is decided structurally, not by timer race.TestBoot_WatcherSubscriptionDoesNotBlockBind: replaced a5ssleep-stall with an indefinite channel block released only after the daemon serves — a synchronous regression now misses by infinity, not a slim margin.Timing-dependence removed entirely where possible (barrier / disabled-watchdog / infinite-block); constants inflated only where the assertion is intrinsically wall-clock. Stability:
-race -countup to 50 underGOMAXPROCS=2(2-core-runner sim), all green.go test -race ./cmd/grafel/ ./internal/daemon/...= 1070 pass.🤖 Generated with Claude Code
https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o