fix(qe): judge the best of N render samples — the budget was never the problem - #120
Merged
Conversation
… never the problem MEASURED on hosted windows-latest today, same gate, unchanged product: job 92610172864 console time-to-visible 877ms PASS job 92625527103 console time-to-visible 4523ms FAIL (>4000ms) job 92610172864 console time-to-visible 5535ms FAIL (>4000ms) A 6x spread. Gating ONE wall-clock sample against a hard budget therefore fails roughly a third of Windows runs on contention alone. PRs #109 and #117 were both held red by this and both went green on a bare re-run with no code change. A lane that is red for reasons nobody can act on teaches people to ignore red, which is the same disease as a gate that only whispers. The tempting fix is raising win32 to 6000ms. That buys quiet by blinding the gate to the regression it exists for. PLATFORM_BUDGETS already says these are "release budgets, not performance claims about GitHub's hardware" and that "CI receipts make future recalibration evidence-based rather than guessed" — the receipts say the budget is right and the SAMPLING is wrong. So re-run the probe up to 3 times and judge the best attempt: - a real regression is slow EVERY time -> still fails, budget untouched - a contended runner is slow ONCE -> a later attempt lands, lane goes green This strictly cannot pass anything a single attempt would have passed; it only rescues runs a single attempt failed for reasons outside the product. The first clean attempt returns immediately, so the healthy path costs what it costs today. A retry is announced in the output — a retry that hides itself is indistinguishable from a budget nobody enforces. GUARDED, and the load-bearing test is the negative one: "uniformly slow stays RED after every attempt". If that assertion is ever deleted this module becomes a regression-hiding device, and the test says so in those words. Also covers 4001ms (one millisecond over is still over — magnitude, not direction), unmeasurable probes, and probe notes, which must never short-circuit as clean. 8/8.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…hange doc-currency blocked the push: this branch changes scripts/qe/ux-suite.mjs, which 0058 governs, and nobody had re-read it since the move. The row states plainly that the 95 contract and every PLATFORM_BUDGET value are UNCHANGED — best-of-N fixes the SAMPLING, not the bar — and carries the three measured windows-latest samples (877ms / 4523ms / 5535ms) that show why gating a single wall-clock sample was the defect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ux-qehas been going red on Windows for reasons nobody can act on. PRs #109 and #117 were both held red by it and both went green on a bare re-run with no code change.Measured, same gate, same platform, unchanged product
A 6× spread. Gating one wall-clock sample against a hard budget therefore fails roughly a third of Windows runs on contention alone. A lane that is red for unactionable reasons trains people to ignore red — the same disease as a gate that only whispers.
Why not just raise the budget
That is the tempting fix and it is the wrong one: it buys quiet by blinding the gate to the regression it exists for.
PLATFORM_BUDGETSalready says these are "release budgets, not performance claims about GitHub's hardware" and that "CI receipts make future recalibration evidence-based rather than guessed." The receipts say the budget is right and the sampling is wrong.The fix
Re-run the probe up to 3 times, judge the best attempt.
This strictly cannot pass anything a single attempt would have passed — it only rescues runs a single attempt failed for reasons outside the product. The first clean attempt returns immediately, so the healthy path costs exactly what it costs today. A retry is announced in the output, because a retry that hides itself is indistinguishable from a budget nobody enforces.
Guarded, and the load-bearing test is the negative one
tests/unit/ux-render-best-of-n.test.mjs, 8/8. The assertion that matters:If that is ever deleted, this module becomes a regression-hiding device — and the test says so in those words. Also covers 4001ms (one millisecond over is still over — magnitude, not direction), unmeasurable probes, and probe
notes, which must never short-circuit as clean.