perf(pr): median-of-N interleaved A/B gate (validation prototype) - #14
Merged
Conversation
mparrett
force-pushed
the
perf/pr-median-gate
branch
from
July 12, 2026 07:13
632b3d2 to
a36a8c4
Compare
Ports the let-go median-of-N gate to paserati. Single-shot perf-pr A/B is ungateable here too (nooga#21): the register-only anchor misses memory-bandwidth contention, so memory-bound families (GetOwn deep-object, PrototypeMethodAccess chain) blow out 14-27% on no-op PRs while the anchor stays flat. Median-of-N across interleaved base/head cycles suppresses it. Composes on top of the min-of-count reducer (nooga#22): min within a capture, median across cycles. perf-pr.yml + driver only, no cmd/bench-ratchet change. Cycles are counterbalanced (ABBA): odd cycles bench base-first, even head-first, so first-vs-second-position drift cannot masquerade as a head regression the median can't remove. N=7 (not 5): a 4-sample no-op validation at N=5/budget 10% held 0 false positives but only marginally — the worst family median reached 7.63% (GetOwn/n=16/round-robin), a 2.37% margin, with several memory-bound families each drawing a bad run at random. Bumping to 7 tightens the median tail rather than widening the budget (which would blind the gate to real ~10% regressions). Matches the let-go settle point (N=7/8%).
mparrett
force-pushed
the
perf/pr-median-gate
branch
from
July 12, 2026 13:59
a36a8c4 to
bc26e4e
Compare
…0 FP) Real-data evidence for the gate (parallels the min-reducer's verdict test on nooga#22). Feeds captured no-op A/B samples through the gate's real decision (_summarize, extracted here from main so it's callable) and asserts: single-shot (one cycle) trips the budget on 3-4 families per run (up to +30%), median-of-N gates 0. Fixtures testdata/perf_pr_{n5,n7}_run.json are same-code fork runs, so every gate is a false positive by construction. Deterministic; run with python3 scripts/test_ab_repeat.py or pytest.
N=7 tolerates floor(6/2)=3 contaminated cycles; observed CI contamination reached 2/5 and N=7 held 0 FP/10. N=9 tolerates 4 for extra margin, nearer benchstat's >=10-sample guidance without going even — an even-N median averages the two middle cycles, reintroducing the tail sensitivity we want to escape. Cost: 18 vs 14 suite runs per gated PR; the gate is opt-in via the perf-repeat label. The principled alternative (retain >=10 raw samples + a benchstat distribution comparison, per the nooga#22 review) is a larger follow-up, backlogged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mparrett
added
perf-repeat
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork validation PR for the median-of-N perf-pr gate (nooga#21). Ports the let-go prototype to paserati, composing on top of the min-of-count reducer (nooga#22): min within a capture, median across cycles.
Perf-neutral by construction — this PR only adds
scripts/ab_repeat.py+.github/workflows/perf-pr-repeat.yml(nopkg/**change), so base and head are perf-identical. Labeling itperf-repeatruns the gate against itself: a same-code A/B where every gate is a false positive, which must be 0 if the mechanism works.Validates two things on paserati's own suite: (1) does median-of-N drive false-positives to 0 where single-shot blows out 14-27%, and (2) what N/budget. Not for merge — this is the proving run before proposing upstream.