perf(ci): split the expand/contract migration rehearsal off the critical path - #817
Merged
Conversation
jason931225
enabled auto-merge
August 19, 2026 03:04
…cal path Measured on merge-group run 32206464663: `backend` was 1176s and the whole run was 1284s, i.e. wall clock was almost exactly `preflight (113s) + backend`. One step inside it -- the PR 473 gate -- was 445s, the largest single step anywhere in CI. Moving it to its own job takes `backend` to ~731s, which puts it below the PostgreSQL shards (~810s after #815's rebalance); the shards become the critical path instead. Deliberately ONE new job, not three. Splitting `backend` by category (lint / gates / buck) gives a worse maximum -- 809s, because the rehearsal would be grouped with the other topology-dependent steps -- and splitting further than this is wasted until the shards get faster. RENAMED, because "PR 473" names a pull request rather than a behaviour and is unreadable to anyone who was not in it. The job is now `migration-expand-contract`, displayed as "Migration expand/contract rehearsal -- 0165 ontology key-revision, 0166 leave", and the step says what it rehearses. What it actually guards: migrations 0165 and 0166 are mid-expand (`docs/release/PR-473-EXPAND-CONTRACT.gate.json` records release_phase=expand, deployment_authorized=false, all production_authority flags false, and a rollback floor), and the gate runs 11 named regressions through 4 Buck disposable-postgres targets. NOT path-gated, on purpose. An expand/contract gate exists to catch ANY change breaking a half-migrated schema during the window; "affected files" analysis for a cross-cutting schema invariant is how a false green gets built. A parallel job buys the same latency at zero coverage cost, so narrowing coverage to buy it would be the wrong trade. Adding a job to `exactCiJobIds` does NOT force the mirror to protect it: this job was required, and green, with no run contracts, no action contracts, no envelope digest and no env allowlist. All are now pinned, and coverage RISES rather than falls -- run steps 121 -> 125, setup actions 35 -> 38, bypass matrices 363 -> 375 and 70 -> 76. Five registries name CI jobs or steps and all five needed updating: check-ci-preflight.mjs, its test, scripts/verify.mjs (JOBS + PLAN), scripts/check-production-hardening.mjs (which located the wrapper by the `backend` job id) and its fixtures. Historical ledger entries keep the old name: they record what was measured then. Mutation-proven: soft-failing the rehearsal -> red; neutering its command to `true` -> red; restored -> green. Verified: check-ci-preflight -> passed, 61 pass 0 fail; verify.test -> 15 pass; check-production-hardening -> passed (239 checks), 91 pass 0 fail; npm run check:ci-preflight -> exit 0; check:doc-citations -> exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
force-pushed
the
ci/split-backend-pr473
branch
from
August 19, 2026 03:16
28dac5c to
038f484
Compare
jason931225
added a commit
that referenced
this pull request
Aug 21, 2026
Measured on merge_group run 32402124838: CI wall clock was ~19m and this one job was the 17m critical path. After #853 moved every PostgreSQL shard to nextest (now 4-8m each), Backend is the only thing left holding the number up. Its 17m is not a hot spot but an accumulation -- three steps are 757s of ~1,070s of work (dev-auth PG suites 355s, clippy 217s, console-app unit 185s) and the other 25 steps are all under 82s. So the lever is parallelism, not speed. The job becomes a three-leg strategy.matrix: cargo rustfmt, clippy, the eleven cargo gates, mutation suites, PR 473 contract tests, boot smoke (16 steps) buck-app platform-authz unit, console-app unit, OpenAPI drift, console-app inline PG (4 steps) buck-dev-auth the dev-auth feature PostgreSQL suites (1 step) Setup, the topology reconcile and collect-failures run on every leg (7 steps). Every one of the 28 steps lands on EXACTLY one leg or on all three -- verified by parsing the workflow, not by reading it. fail-fast is off so one red leg never hides the others, the same fail-slow rule the step layer already follows. WHY A MATRIX AND NOT SEPARATE JOBS. Splitting into new jobs was designed and refuted: it touches every pinned contract structure and creates two false-green surfaces -- a new job not wired into required-ci's `needs` runs and gates nothing, and a `steps.topology.outcome` condition carried into a job with no topology step skips forever while the job stays green. A matrix keeps the job id, its membership in required-ci, and every step's place in the bypass-mutation matrix exactly where they were. Ratchets do not move: run-step count and bypass-mutation count are unchanged (130 / 390 after #853), because no step was added, removed, or renamed. THE CACHE, and a pre-existing hole this closes. rust-cache has ONE writer by contract. It had two: migration-expand-contract carried `save-if: main` under a comment reading "The ONLY writer" -- copied from `backend` when that job was split out in #817 and false in place ever since -- and was absent from `cargoRustCacheJobs`, so the one-writer check never saw it. It is now restore-only and listed. With a matrix, "one writer" also stops being enough: three legs saving the same key race on every main push, and the textual `save-if: false` test cannot tell one leg saving from three. The writer is now exactly the cargo leg, and a new assertion requires the writer's save-if to name exactly one leg. TWO NEW ASSERTIONS, both mutation-proven, both from the refutation: a stale leg literal (rename a leg in the matrix, leave a step's `if`) -> "names matrix leg 'buck-app', which is not in strategy.matrix.leg -- that step would run on ZERO legs" the writer's save-if no longer leg-scoped -> "backend's rust-cache save-if must name exactly ONE matrix leg" restored -> preflight exit 0 Without the first, a coordinated matrix rename plus hash recompute could leave a proof step running on no leg at all while every leg, required-ci, and check-executed-tests stayed green. Preflight: exit 0. Contract suite: 61/61. Gate sweep 13/13. check-workflow-hardening and check-executed-tests exit 0. clippy and the first gate remain adjacent on the cargo leg, so the gate-order lock holds. WHAT IS PREDICTED, NOT MEASURED. Per-leg setup is ~55s. The design puts the legs at cargo ~6.7m, buck-dev-auth ~7.0m, buck-app ~8-10m -- the last because app-unit's 185s was measured WARM after dev-auth had built the platform crates in the same daemon, and buck-out does not cross runners. If that holds, Backend drops from 17m to ~8-10m and the shards (4-8m) and Backend become comparable, putting wall clock near 10m. That is a prediction until the first merge_group run measures it. 5m is not reachable by this change alone: the platform shard carries one 263s test, and the buck-app leg pays a cold platform build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 21, 2026
Measured on merge_group run 32402124838: CI wall clock ~19m, and this one job was the **17m critical path**. After #853 moved every PostgreSQL shard to nextest (now 4–8m each), Backend is the only thing holding the number up. Its 17m isn't a hot spot — it's an accumulation. Three steps are **757s of ~1,070s** (dev-auth PG suites 355s, clippy 217s, console-app unit 185s); the other 25 steps are all under 82s. So the lever is parallelism, not speed. ## Three legs | leg | steps | |---|---| | `cargo` | rustfmt, clippy, the 11 cargo gates, mutation suites, PR 473 tests, boot smoke (16) | | `buck-app` | platform-authz unit, console-app unit, OpenAPI drift, console-app inline PG (4) | | `buck-dev-auth` | the dev-auth feature PostgreSQL suites (1) | Setup, topology reconcile and collect-failures run on every leg (7). **Every one of the 28 steps lands on exactly one leg or on all three** — verified by parsing the workflow, not by reading it. `fail-fast: false`, so one red leg never hides the others. ## Why a matrix and not separate jobs Separate jobs were designed and **refuted**: they touch every pinned contract structure and create two false-green surfaces — a new job not wired into `required-ci`'s `needs` runs and gates nothing; a `steps.topology.outcome` condition carried into a job with no topology step skips forever while the job stays green. A matrix keeps the job id, its `required-ci` membership, and every step's place in the bypass-mutation matrix exactly where they were. **Ratchets do not move** (130 / 390 after #853): no step was added, removed, or renamed. ## The cache — and a pre-existing hole this closes rust-cache has *one writer* by contract. **It had two.** `migration-expand-contract` carried `save-if: main` under a comment reading *"The ONLY writer"* — copied from `backend` when that job was split out in #817, and false in place ever since — and was absent from `cargoRustCacheJobs`, so the one-writer check never saw it. Now restore-only and listed. With a matrix, "one writer" also stops being enough: three legs saving the same key race on every main push, and the textual `save-if: false` test can't tell one leg from three. The writer is now exactly the `cargo` leg, enforced by a new assertion. ## Two new assertions, both mutation-proven, both from the refutation | mutation | result | |---|---| | rename a leg in the matrix, leave a step's `if:` stale | *"names matrix leg 'buck-app', which is not in strategy.matrix.leg — that step would run on ZERO legs"* | | writer's `save-if` no longer leg-scoped | *"save-if must name exactly ONE matrix leg"* | | restored | preflight exit 0 | Without the first, a coordinated matrix rename plus hash recompute could leave a proof step running on **no leg at all** while every leg, `required-ci`, and `check-executed-tests` stayed green. Preflight exit 0 · contract suite **61/61** · gate sweep 13/13 · workflow-hardening and executed-tests exit 0 · clippy→first-gate adjacency holds on the cargo leg. ## Predicted, not measured Per-leg setup ~55s. Design estimate: cargo ~6.7m, buck-dev-auth ~7.0m, buck-app ~8–10m — the last because app-unit's 185s was measured **warm** after dev-auth had built the platform crates in the same daemon, and buck-out doesn't cross runners. If that holds, Backend drops 17m → ~8–10m and wall clock lands near **10m**. That is a prediction until this PR's own CI run measures it. **5m is not reachable by this change alone**: the platform shard carries one 263s test, and the buck-app leg pays a cold platform build. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Measured on merge-group run
32206464663:backendwas 1176s and the whole run 1284s — wall clock is almost exactlypreflight (113s) + backend. One step inside it was 445s, the largest single step anywhere in CI.Moving it out takes
backendto ~731s, below the PostgreSQL shards (~810s after #815's rebalance). The shards become the critical path instead.One new job, not three. Splitting
backendby category (lint / gates / buck) gives a worse maximum — 809s, because the rehearsal would be grouped with the other topology-dependent steps. Splitting further is wasted until the shards get faster.Renamed, because "PR 473" names a pull request
The old name is unreadable to anyone who wasn't in that PR. Now
migration-expand-contract, displayed as "Migration expand/contract rehearsal — 0165 ontology key-revision, 0166 leave".What it actually guards: migrations 0165 and 0166 are mid-expand.
docs/release/PR-473-EXPAND-CONTRACT.gate.jsonrecordsrelease_phase: expand,deployment_authorized: false, allproduction_authorityflags false, and a rollback floor. The gate runs 11 named regressions through 4 Buck disposable-postgres targets.Not path-gated, on purpose
An expand/contract gate exists to catch any change breaking a half-migrated schema during the window. "Affected files" analysis for a cross-cutting schema invariant is how a false green gets built — and a parallel job buys the same latency at zero coverage cost, so narrowing coverage to buy it would be the wrong trade.
A gap this exposed
Adding a job to
exactCiJobIdsdoes not force the mirror to protect it. This job was required, and green, with no run contracts, no action contracts, no envelope digest and no env allowlist. All are now pinned — and coverage rises:Five registries
All needed updating:
check-ci-preflight.mjs, its test,scripts/verify.mjs(bothJOBSandPLAN),scripts/check-production-hardening.mjs(which located the wrapper by thebackendjob id) and its fixtures. Historical ledger entries keep the old name — they record what was measured then.Mutation proof
continue-on-error: true)trueVerification
check-ci-preflight→ passed, 61 pass / 0 fail ·verify.test→ 15 pass ·check-production-hardening→ passed (239 checks), 91 pass / 0 fail ·npm run check:ci-preflight→ exit 0 ·check:doc-citations→ exit 0Follow-up worth a bead
The migration has been parked in
expandwithdeployment_authorized: falsesince 2026-07-21 — four weeks — with no exit condition recorded. The real fix is finishing the contract phase and retiring this gate entirely; this PR only stops it dominating every run in the meantime.🤖 Generated with Claude Code