ci: wire the duration-weighted partitioner and retire the family scheme - #815
Merged
Conversation
jason931225
enabled auto-merge
August 19, 2026 02:36
This was referenced Aug 19, 2026
jason931225
force-pushed
the
ci/wire-shard-partition
branch
from
August 19, 2026 03:07
dba037b to
a216df5
Compare
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
…cal path Measured on merge-group run 32206464663: `backend` was 1176s and the whole run was 1284s, i.e. wall clock was almost exactly `preflight (113s) + backend`. One step inside it -- the PR 473 gate -- was 445s, the largest single step anywhere in CI. Moving it to its own job takes `backend` to ~731s, which puts it below the PostgreSQL shards (~810s after #815's rebalance); the shards become the critical path instead. Deliberately ONE new job, not three. Splitting `backend` by category (lint / gates / buck) gives a worse maximum -- 809s, because the rehearsal would be grouped with the other topology-dependent steps -- and splitting further than this is wasted until the shards get faster. RENAMED, because "PR 473" names a pull request rather than a behaviour and is unreadable to anyone who was not in it. The job is now `migration-expand-contract`, displayed as "Migration expand/contract rehearsal -- 0165 ontology key-revision, 0166 leave", and the step says what it rehearses. What it actually guards: migrations 0165 and 0166 are mid-expand (`docs/release/PR-473-EXPAND-CONTRACT.gate.json` records release_phase=expand, deployment_authorized=false, all production_authority flags false, and a rollback floor), and the gate runs 11 named regressions through 4 Buck disposable-postgres targets. NOT path-gated, on purpose. An expand/contract gate exists to catch ANY change breaking a half-migrated schema during the window; "affected files" analysis for a cross-cutting schema invariant is how a false green gets built. A parallel job buys the same latency at zero coverage cost, so narrowing coverage to buy it would be the wrong trade. Adding a job to `exactCiJobIds` does NOT force the mirror to protect it: this job was required, and green, with no run contracts, no action contracts, no envelope digest and no env allowlist. All are now pinned, and coverage RISES rather than falls -- run steps 121 -> 125, setup actions 35 -> 38, bypass matrices 363 -> 375 and 70 -> 76. Five registries name CI jobs or steps and all five needed updating: check-ci-preflight.mjs, its test, scripts/verify.mjs (JOBS + PLAN), scripts/check-production-hardening.mjs (which located the wrapper by the `backend` job id) and its fixtures. Historical ledger entries keep the old name: they record what was measured then. Mutation-proven: soft-failing the rehearsal -> red; neutering its command to `true` -> red; restored -> green. Verified: check-ci-preflight -> passed, 61 pass 0 fail; verify.test -> 15 pass; check-production-hardening -> passed (239 checks), 91 pass 0 fail; npm run check:ci-preflight -> exit 0; check:doc-citations -> exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
… asserts #815 replaced the harness's inline Python selection with this partitioner, which also replaced the code path that emitted "no map entries selected" on an empty selection. backend/ci/gates/writer-ownership's `cargo_needs_postgres_harness_executes_the_enforcement` runs the harness with a deliberately-absent --only name and asserts that phrase appears -- that is how it proves canonical enforcement runs BEFORE target selection, not after. CI caught it: shard `platform`, `FAIL writer-ownership-canonical-census-pg`, 27 passed 1 failed. Both properties now hold and are distinct: empty selection -> "no map entries selected: --only matched none of N ..." partial match -> "--only selected 1 of 2 requested targets ..." both exit 1. The stricter partial-match check added in #815 is additive to the existing contract rather than a replacement for it; a misspelled --only that matches a SUBSET still fails closed rather than silently running less. Pinned with a test that shells out to the CLI, so the phrase cannot be dropped again by someone who does not know a Rust gate three directories away depends on it. Verified: node --test tools/ci/postgres-partition.test.mjs -> 20 pass, 0 fail; npm run check:ci-preflight -> exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
makes it the live shard assignment and deletes the scheme it replaces, so the repo describes one partition rather than two. Shard assignment had TWO implementations: tools/ci/postgres-shard.mjs, and a hand-maintained Python copy of the same family logic inline in cargo_needs_postgres.sh. Two implementations of the rule that decides which tests run is a false green waiting to happen -- the harness could select a set the checker never validated. Both are gone; the harness now calls postgres-partition.mjs, which is also what check-postgres-cargo-map.mjs validates. Effect on the real map (209 workflow targets, complete and disjoint under both): family scheme app=56 platform=39 ontology=26 domain-a=44 domain-b=44 duration packed app=68 platform=28 ontology=37 domain-a=33 domain-b=43 Entry counts are deliberately uneven now: the packer balances SECONDS, not targets. Measured on run 32115833327 that takes the slowest shard from 877.1s to 660.4s, spread 1.80x -> 1.00x. The five shard ids are kept as HISTORICAL LABELS and documented as such. They used to mean "the app package", "the ontology packages"; under duration packing they mean nothing but bin 0..4. They survive only because they are ci.yml job ids that the preflight mirror, scripts/verify.mjs and the doc-citation gate all name -- renaming them is a separate change across those four registries, and leaving a stale name is the lesser evil against a five-way rename landing in the same commit as a behaviour change. One bug caught by testing rather than by CI: the harness always passes --only, empty when the caller gave none, and the first version treated an empty value as "select nothing" instead of "no filter" -- every shard would have run zero tests and reported success. Empty now means no filter, a misspelled --only exits non-zero rather than running a silently smaller set, and both are tested. Verified: node --test tools/ci/postgres-partition.test.mjs -> 19 pass, 0 fail (including completeness/disjointness over the real map); check-postgres-cargo-map -> OK, facets 68/28/37/33/43; check-ci-preflight -> passed, 61 pass 0 fail; dark-suites --strict -> exit 0; npm run check:ci-preflight -> exit 0; bash -n on the harness. Stacked on #811 because it edits the same shard jobs' surroundings; retarget to main once that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… asserts #815 replaced the harness's inline Python selection with this partitioner, which also replaced the code path that emitted "no map entries selected" on an empty selection. backend/ci/gates/writer-ownership's `cargo_needs_postgres_harness_executes_the_enforcement` runs the harness with a deliberately-absent --only name and asserts that phrase appears -- that is how it proves canonical enforcement runs BEFORE target selection, not after. CI caught it: shard `platform`, `FAIL writer-ownership-canonical-census-pg`, 27 passed 1 failed. Both properties now hold and are distinct: empty selection -> "no map entries selected: --only matched none of N ..." partial match -> "--only selected 1 of 2 requested targets ..." both exit 1. The stricter partial-match check added in #815 is additive to the existing contract rather than a replacement for it; a misspelled --only that matches a SUBSET still fails closed rather than silently running less. Pinned with a test that shells out to the CLI, so the phrase cannot be dropped again by someone who does not know a Rust gate three directories away depends on it. Verified: node --test tools/ci/postgres-partition.test.mjs -> 20 pass, 0 fail; npm run check:ci-preflight -> exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
force-pushed
the
ci/wire-shard-partition
branch
from
August 19, 2026 03:58
facc8fd to
d745393
Compare
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
`check:executed-tests` refused the branch:
1 test binary(ies) execute nowhere and are not named in
docs/program/executed-tests-baseline.json:
backend/crates/ontology/canonical-adapter-postgres/tests/employment_reassign_as_runtime_role.rs
Wire each into a workflow step, or the repository has 1 more test(s) that cannot fail.
The generated Buck face was regenerated earlier in this branch, so the TARGET
existed -- but no shard selected it, because tools/ci/postgres-cargo-map.json is
what decides which targets a PostgreSQL shard runs, and the new binary was absent
from it. A test that builds and never executes is the false green this ratchet
exists to catch.
Wired rather than baselined. Baselining would have satisfied the gate by
declaring the test permanently dark, which is the opposite of what a new
fail-closed refusal test is for.
The entry mirrors its sibling employment_port_as_runtime_role in the same crate.
`measured_seconds` is deliberately ABSENT: the test has never run in CI, so there
is no measurement, and inventing one would have the duration partitioner
(#815/#810) balance shards on a number nobody took.
Workflow targets 209 -> 210; facets app=56 platform=39 ontology=27 domain-a=44
domain-b=44.
Verified: node tools/ci/check-postgres-cargo-map.mjs -> OK (210 entries);
npm run check:executed-tests -> exit 0, and employment_reassign no longer listed
(the residual "executed nowhere: 1" is seaweedfs_worm.rs, already baselined on
main); npm run check:ci-preflight -> exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
Rebasing onto main surfaced a latent red. #815 retired the semantic-family partitioner and deleted tools/ci/postgres-shard.{mjs,test.mjs}, removing the suite from check:ci-preflight's `&&` chain at the same time. This branch had already replaced that chain with the manifest, so the deletion could not conflict textually — the manifest kept naming a file that no longer exists. Merged as-is, `node --test` would have been handed a missing path and the ci-tools-suites gate would have failed on main for a reason unrelated to anything either PR was about. The conflict git DID raise was package.json, and only because both sides edited the same line. Resolution keeps this branch's `node tools/ci/gate-sweep.mjs` and carries main's suite-list change into the manifest, which is where the list now lives. Verified: 12 gates swept, 0 failed; dark-suites --strict, both reachability scanners, executed-tests-baseline and package-lock all exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
`check:executed-tests` refused the branch:
1 test binary(ies) execute nowhere and are not named in
docs/program/executed-tests-baseline.json:
backend/crates/ontology/canonical-adapter-postgres/tests/employment_reassign_as_runtime_role.rs
Wire each into a workflow step, or the repository has 1 more test(s) that cannot fail.
The generated Buck face was regenerated earlier in this branch, so the TARGET
existed -- but no shard selected it, because tools/ci/postgres-cargo-map.json is
what decides which targets a PostgreSQL shard runs, and the new binary was absent
from it. A test that builds and never executes is the false green this ratchet
exists to catch.
Wired rather than baselined. Baselining would have satisfied the gate by
declaring the test permanently dark, which is the opposite of what a new
fail-closed refusal test is for.
The entry mirrors its sibling employment_port_as_runtime_role in the same crate.
`measured_seconds` is deliberately ABSENT: the test has never run in CI, so there
is no measurement, and inventing one would have the duration partitioner
(#815/#810) balance shards on a number nobody took.
Workflow targets 209 -> 210; facets app=56 platform=39 ontology=27 domain-a=44
domain-b=44.
Verified: node tools/ci/check-postgres-cargo-map.mjs -> OK (210 entries);
npm run check:executed-tests -> exit 0, and employment_reassign no longer listed
(the residual "executed nowhere: 1" is seaweedfs_worm.rs, already baselined on
main); npm run check:ci-preflight -> exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
`check:executed-tests` refused the branch:
1 test binary(ies) execute nowhere and are not named in
docs/program/executed-tests-baseline.json:
backend/crates/ontology/canonical-adapter-postgres/tests/employment_reassign_as_runtime_role.rs
Wire each into a workflow step, or the repository has 1 more test(s) that cannot fail.
The generated Buck face was regenerated earlier in this branch, so the TARGET
existed -- but no shard selected it, because tools/ci/postgres-cargo-map.json is
what decides which targets a PostgreSQL shard runs, and the new binary was absent
from it. A test that builds and never executes is the false green this ratchet
exists to catch.
Wired rather than baselined. Baselining would have satisfied the gate by
declaring the test permanently dark, which is the opposite of what a new
fail-closed refusal test is for.
The entry mirrors its sibling employment_port_as_runtime_role in the same crate.
`measured_seconds` is deliberately ABSENT: the test has never run in CI, so there
is no measurement, and inventing one would have the duration partitioner
(#815/#810) balance shards on a number nobody took.
Workflow targets 209 -> 210; facets app=56 platform=39 ontology=27 domain-a=44
domain-b=44.
Verified: node tools/ci/check-postgres-cargo-map.mjs -> OK (210 entries);
npm run check:executed-tests -> exit 0, and employment_reassign no longer listed
(the residual "executed nowhere: 1" is seaweedfs_worm.rs, already baselined on
main); npm run check:ci-preflight -> exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #811. Review the top commit only. Retarget to main once #811 lands.
Why
#810 landed the partitioner and its measured weights, but nothing called it. This makes it the live shard assignment and deletes the scheme it replaces, so the repo describes one partition instead of two.
The duplication this removes
Shard assignment had two implementations:
tools/ci/postgres-shard.mjs, and a hand-maintained Python copy of the same family logic inline incargo_needs_postgres.sh. Two implementations of the rule that decides which tests run is a false green waiting to happen — the harness could select a set the checker never validated.Both are gone. The harness now calls
postgres-partition.mjs, which is also whatcheck-postgres-cargo-map.mjsvalidates.Effect on the real map
209 workflow targets, complete and disjoint under both schemes:
Entry counts are deliberately uneven — the packer balances seconds, not targets. Measured on run
32115833327: slowest shard 877.1s → 660.4s, spread 1.80× → 1.00×.The shard ids are now historical labels
app,platform,ontology,domain-a,domain-bused to describe contents. Under duration packing they mean nothing but bin 0..4, and the code says so.They survive because they're
ci.ymljob ids named by the preflight mirror,scripts/verify.mjs, and the doc-citation gate. Renaming is a separate change across those four registries — landing a five-way rename in the same commit as a behaviour change is the worse option.A bug caught by testing, not CI
The harness always passes
--only, empty when the caller gave none. The first version treated an empty value as "select nothing" rather than "no filter" — every shard would have run zero tests and reported success.Empty now means no filter, a misspelled
--onlyexits non-zero rather than silently running a smaller set, and both are tested.Verification
postgres-partition.test.mjscheck-postgres-cargo-map.mjscheck-ci-preflight.mjs/ its suitecheck-mjs-dark-suites --strictnpm run check:ci-preflight🤖 Generated with Claude Code