test(engine,resolve): min-based perf-scaling assertions for noisy Windows CI#5899
Merged
Conversation
…I flake TestResponseShapePython_NearLinearScaling failed on Windows CI (run #4, pre-existing flake, not caused by the epic under test): median-of-5 timing at N=1000 got dragged to 130ms by runner contention, pushing the ratio to 32.28x over the 25x bound with no real algorithmic regression. Wall-clock noise (GC, scheduler contention, co-scheduled load) only ever ADDS time to a sample, never subtracts it, so a noisy median has no ceiling on how far it can be pulled upward. Switch both scaling-ratio tests in the repo (the only two computing a timed-corpus ratio: response_shape_python and resolve/scale_bench) from median-of-samples to min-of-samples: the minimum is the least-contended observation and the best available estimate of true algorithmic cost, while a genuine O(n^2)/O(n^2) regression is still ~100x slower on every run including the cleanest one, so detection power is unchanged. - response_shape_python_perf_test.go: median->min, samples 5->8, bound 25->30 (observed min-based ratio locally: 9.77-10.31x vs 10x expected linear, comfortably under 30 and far under the 100x quadratic signal). - resolve/scale_bench_test.go: same median->min swap, samples 5->8, bound kept at 40 (observed min-based ratio locally: 11.6-12.7x vs ~15x expected n-log-n, comfortably under 40). Repo-wide audit (grep for ratio :=/ratio = timing assertions in *_test.go) confirms these are the only two tests using this pattern; no other siblings needed hardening.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the sole windows-latest failure in v0.1.9 acceptance CI #4:
TestResponseShapePython_NearLinearScaling→ratio=32.28 (want <25x). Pre-existing flake, not a regression (passed the 3 prior CI runs that already contained every epic change); failed in the non--racepass = a noisy median-of-5 perf measurement inflated by Windows-runner contention. Refs #5850.Fix:
medianCorpus→minCorpus(min of per-N samples). Wall-clock noise only ever adds time, so the minimum is the least-contended observation and the best estimate of true algorithmic cost, while a noisy median has no upward ceiling. A real O(n²) path is ~100x at N=1000/N=100 on every run including the cleanest, so min-based aggregation loses no regression-detection power.samples5→8; threshold 25→30 (3x over linear≈10x, 3.3x under quadratic≈100x — wide separation).Sibling audit: hardened the one other same-pattern test identically —
internal/resolve/scale_bench_test.go(TestBuildIndexFromModules_SubQuadratic, min-based, samples 5→8). Other grep hits were false positives (unrelated stats helpers / "autoscaling" substrings).Gates:
NearLinearScaling -count=5→ 5/5 pass (ratios 9.77–10.31 vs bound 30);SubQuadratic -count=5→ 5/5 pass. Structurally noise-robust, not a re-tuned constant.🤖 Generated with Claude Code
https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o