Skip to content

Pin the divergent-default fallback SLO in the tests that assert its result - #398

Merged
zzet merged 1 commit into
mainfrom
fix/pin-divergent-default-fallback-slo
Jul 28, 2026
Merged

Pin the divergent-default fallback SLO in the tests that assert its result#398
zzet merged 1 commit into
mainfrom
fix/pin-divergent-default-fallback-slo

Conversation

@zzet

@zzet zzet commented Jul 28, 2026

Copy link
Copy Markdown
Owner

The failure

test (macos-latest) on main fails intermittently:

--- FAIL: TestHandleExplorePromotesDivergentDefaultOwnerFromSQLitePHPIndex
    explore_divergent_default_owner_test.go:429:
    Error: "1" is not greater than or equal to "2"

The explore envelope carries only StreamHandler.write, without the promoted RotatingFileHandler constructor and type.

Cause

exploreDefaultOwnerFallbackSLO — the 25ms slice bounding the ranked-callable fallback in explore_divergent_default_owner.go.

That fallback is the only route to the child pair in this test: promotion enters with a single ranked target, so no base constructor/type pair is pre-ranked and exploreDivergentDefaultBases returns nothing. The slice is a silent bound — each expiry check returns "no bases", the caller promotes nothing, and the response is a smaller symbol set with no error and no log.

Idle, the batched inbound lookup plus node hydration cost ~150µs against a 25ms budget. Under load those SQLite reads get descheduled past it and the guards at :534 / :557 trip. Instrumented on unmodified main under CPU saturation: 4 failures, 4 guard trips, one-to-one (hydTook=26.87ms, fnTook=46.21ms).

Fix

Production is not wrong — a fail-closed SLO on a best-effort recovery path is deliberate, and widening it was already tried and rejected (d86ef8c0, which deleted the budget outright, was archived unmerged). What is wrong is a test asserting which owner gets promoted while racing that SLO against machine load.

So the slice becomes overridable and the assertions pin it — exactly as pinExploreSourceLiteralRecallBudget already does for the 75ms source-literal recall, the same failure class in the same pipeline.

Two seams, because two call paths reach the fallback: an optional trailing argument for the direct calls, and Server.divergentDefaultFallbackSLOOverride for the handleExplore path. Both default to the production constant, so nothing ships differently. The argument is variadic so the fifteen call sites that never reach the fallback stay untouched.

TestDivergentDefaultOwnerCallableFallbackFailsClosed deliberately passes no override: its nodeBatchDelay is exploreDefaultOwnerFallbackSLO + 1ms, so it must keep racing the production budget for its deadline contract to mean anything.

Verification

Reproduced the failure and confirmed the fix under CPU saturation (20 spinners on 10 cores, -race):

tree result
372167a2 (before the repo-prefix refactor) 2/10 failed — line 429, byte-identical to CI
main, unfixed 3/12 failed — lines 416 and 429
main + this fix 0/14 failed

Also golangci-lint clean and the full internal/mcp package green.

Not caused by #396

This flake predates the repo-prefix refactor. The same test failed the same way on macOS in run 29881065044 (2026-07-22), on an unrelated branch, six days before #396 existed — at the assertion that calls promoteExploreDivergentDefaultOwner directly, bypassing everything #396 touched. The 2/10 pre-refactor reproduction above confirms it independently.

Worth noting separately: the macOS internal/mcp job is chronically timing-flaky — 10 macOS test-job failures across 188 CI runs in the 07-19..07-29 window, 6 distinct tests, 4 of them explore wall-clock-budget tests, none repeating on the same SHA. This PR fixes one of those four; the pattern suggests the others deserve the same treatment.

…esult

TestHandleExplorePromotesDivergentDefaultOwnerFromSQLitePHPIndex fails
intermittently on the macOS CI runner with "1 is not greater than or equal to
2" — the explore envelope carries only StreamHandler.write, without the
promoted RotatingFileHandler constructor and type.

The cause is exploreDefaultOwnerFallbackSLO, the 25ms slice bounding the
ranked-callable fallback. That fallback is the only route to the child pair
here: promotion enters with a single ranked target, so no base
constructor/type pair is pre-ranked and exploreDivergentDefaultBases returns
nothing. The slice is a SILENT bound — each expiry check returns "no bases",
so the caller promotes nothing and the response is a smaller symbol set with
no error and no log. Idle, the batched inbound lookup plus node hydration cost
~150us; under load those SQLite reads are descheduled past 25ms and the guards
at explore_divergent_default_owner.go:534 and :557 trip.

Production is not wrong. A fail-closed SLO on a best-effort recovery path is
deliberate, and widening it was already tried and rejected (d86ef8c0, which
deleted the budget outright, was archived unmerged). What is wrong is a test
asserting WHICH owner gets promoted while racing that SLO against machine
load. So the slice becomes overridable and the assertions pin it, exactly as
pinExploreSourceLiteralRecallBudget already does for the 75ms source-literal
recall — the same failure class in the same pipeline.

Two seams, because two call paths reach the fallback: an optional trailing
argument for the direct calls, and Server.divergentDefaultFallbackSLOOverride
for the handleExplore path. Both default to the production constant, so
nothing ships differently. The argument is variadic so the fifteen call sites
that never reach the fallback stay untouched.

TestDivergentDefaultOwnerCallableFallbackFailsClosed deliberately passes no
override: its nodeBatchDelay is exploreDefaultOwnerFallbackSLO+1ms, so it must
keep racing the production budget for its deadline contract to mean anything.

Verified by reproducing the failure and confirming the fix under CPU
saturation (20 spinners on 10 cores, -race):

  before   2/10 and 3/12 failed, at the same two assertions as CI
  after   0/14 failed

This flake predates the repo-prefix refactor: the same test failed the same
way on macOS in run 29881065044 (2026-07-22), on an unrelated branch.
@zzet
zzet merged commit abb4108 into main Jul 28, 2026
11 checks passed
@zzet
zzet deleted the fix/pin-divergent-default-fallback-slo branch July 28, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant