ci: stop freeing runner disk that was never scarce - #811
Merged
Conversation
The `Free runner disk` step ran in seven ci.yml jobs. Measured on run 32111278481 (postgres shard) and 32108350947 (backend), the runner state immediately BEFORE it ran: /dev/root 145G total, 58G used, 87G available (40%) and after: 110G available. It spends 58s (shards) to 82s (backend) reclaiming 23G on a machine that already had 87G free, and grep finds zero ENOSPC or "no space" in any job log, ever. Chesterton's Fence: it arrived in PR #155 WITH the dev-up orchestrator, which brings up a full Docker compose project -- that genuinely can exhaust disk. It was then applied to jobs that start one postgres container. dev-up-smoke has since moved to nightly.yml, so the justification and the cost no longer live in the same workflow. nightly.yml KEEPS the step for exactly that reason; only ci.yml drops it. Removal is self-verifying: if disk ever were scarce the job fails loudly with ENOSPC. This cannot degrade into a false green, which is why it is safe to remove on evidence rather than after another instrumented run. Worth ~58s on the critical-path shard, ~82s on backend (which becomes the critical path once the shards are rebalanced), and ~370s of compute per run. The mirror pins setup actions by ABSOLUTE step index, so removing a step shifts every index after it; those are decremented per job rather than re-derived. Ratchets step down by exactly what was removed and no more: setup-action coverage 42 -> 35 (7 steps), bypass matrix 84 -> 70 (7 x 2 mutations), identity/interleaving matrix 277 -> 249, and per-job inventories by one each. One trap worth recording: "company-conformance" appears in BOTH requiredRunStepCounts and requiredActionStepCounts, so a first-match replace silently edited the run-step map instead of the action-step map and moved a count this change does not touch. Both are now set explicitly. Verified: node scripts/check-ci-preflight.mjs -> passed; node --test scripts/check-ci-preflight.test.mjs -> 60 pass, 0 fail; node --test scripts/verify.test.mjs -> 15 pass; check-nightly-workflow -> 14 pass; npm run check:ci-preflight -> exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
force-pushed
the
ci/drop-free-runner-disk
branch
from
August 19, 2026 01:28
a191a67 to
e3ca1b8
Compare
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
#810 landed the partitioner and its measured weights but nothing called it. This makes it the live shard assignment and deletes the scheme it replaces, so the repo describes one partition rather than two. Shard assignment had TWO implementations: tools/ci/postgres-shard.mjs, and a hand-maintained Python copy of the same family logic inline in cargo_needs_postgres.sh. Two implementations of the rule that decides which tests run is a false green waiting to happen -- the harness could select a set the checker never validated. Both are gone; the harness now calls postgres-partition.mjs, which is also what check-postgres-cargo-map.mjs validates. Effect on the real map (209 workflow targets, complete and disjoint under both): family scheme app=56 platform=39 ontology=26 domain-a=44 domain-b=44 duration packed app=68 platform=28 ontology=37 domain-a=33 domain-b=43 Entry counts are deliberately uneven now: the packer balances SECONDS, not targets. Measured on run 32115833327 that takes the slowest shard from 877.1s to 660.4s, spread 1.80x -> 1.00x. The five shard ids are kept as HISTORICAL LABELS and documented as such. They used to mean "the app package", "the ontology packages"; under duration packing they mean nothing but bin 0..4. They survive only because they are ci.yml job ids that the preflight mirror, scripts/verify.mjs and the doc-citation gate all name -- renaming them is a separate change across those four registries, and leaving a stale name is the lesser evil against a five-way rename landing in the same commit as a behaviour change. One bug caught by testing rather than by CI: the harness always passes --only, empty when the caller gave none, and the first version treated an empty value as "select nothing" instead of "no filter" -- every shard would have run zero tests and reported success. Empty now means no filter, a misspelled --only exits non-zero rather than running a silently smaller set, and both are tested. Verified: node --test tools/ci/postgres-partition.test.mjs -> 19 pass, 0 fail (including completeness/disjointness over the real map); check-postgres-cargo-map -> OK, facets 68/28/37/33/43; check-ci-preflight -> passed, 61 pass 0 fail; dark-suites --strict -> exit 0; npm run check:ci-preflight -> exit 0; bash -n on the harness. Stacked on #811 because it edits the same shard jobs' surroundings; retarget to main once that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
makes it the live shard assignment and deletes the scheme it replaces, so the repo describes one partition rather than two. Shard assignment had TWO implementations: tools/ci/postgres-shard.mjs, and a hand-maintained Python copy of the same family logic inline in cargo_needs_postgres.sh. Two implementations of the rule that decides which tests run is a false green waiting to happen -- the harness could select a set the checker never validated. Both are gone; the harness now calls postgres-partition.mjs, which is also what check-postgres-cargo-map.mjs validates. Effect on the real map (209 workflow targets, complete and disjoint under both): family scheme app=56 platform=39 ontology=26 domain-a=44 domain-b=44 duration packed app=68 platform=28 ontology=37 domain-a=33 domain-b=43 Entry counts are deliberately uneven now: the packer balances SECONDS, not targets. Measured on run 32115833327 that takes the slowest shard from 877.1s to 660.4s, spread 1.80x -> 1.00x. The five shard ids are kept as HISTORICAL LABELS and documented as such. They used to mean "the app package", "the ontology packages"; under duration packing they mean nothing but bin 0..4. They survive only because they are ci.yml job ids that the preflight mirror, scripts/verify.mjs and the doc-citation gate all name -- renaming them is a separate change across those four registries, and leaving a stale name is the lesser evil against a five-way rename landing in the same commit as a behaviour change. One bug caught by testing rather than by CI: the harness always passes --only, empty when the caller gave none, and the first version treated an empty value as "select nothing" instead of "no filter" -- every shard would have run zero tests and reported success. Empty now means no filter, a misspelled --only exits non-zero rather than running a silently smaller set, and both are tested. Verified: node --test tools/ci/postgres-partition.test.mjs -> 19 pass, 0 fail (including completeness/disjointness over the real map); check-postgres-cargo-map -> OK, facets 68/28/37/33/43; check-ci-preflight -> passed, 61 pass 0 fail; dark-suites --strict -> exit 0; npm run check:ci-preflight -> exit 0; bash -n on the harness. Stacked on #811 because it edits the same shard jobs' surroundings; retarget to main once that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jason931225
added a commit
that referenced
this pull request
Aug 19, 2026
makes it the live shard assignment and deletes the scheme it replaces, so the repo describes one partition rather than two. Shard assignment had TWO implementations: tools/ci/postgres-shard.mjs, and a hand-maintained Python copy of the same family logic inline in cargo_needs_postgres.sh. Two implementations of the rule that decides which tests run is a false green waiting to happen -- the harness could select a set the checker never validated. Both are gone; the harness now calls postgres-partition.mjs, which is also what check-postgres-cargo-map.mjs validates. Effect on the real map (209 workflow targets, complete and disjoint under both): family scheme app=56 platform=39 ontology=26 domain-a=44 domain-b=44 duration packed app=68 platform=28 ontology=37 domain-a=33 domain-b=43 Entry counts are deliberately uneven now: the packer balances SECONDS, not targets. Measured on run 32115833327 that takes the slowest shard from 877.1s to 660.4s, spread 1.80x -> 1.00x. The five shard ids are kept as HISTORICAL LABELS and documented as such. They used to mean "the app package", "the ontology packages"; under duration packing they mean nothing but bin 0..4. They survive only because they are ci.yml job ids that the preflight mirror, scripts/verify.mjs and the doc-citation gate all name -- renaming them is a separate change across those four registries, and leaving a stale name is the lesser evil against a five-way rename landing in the same commit as a behaviour change. One bug caught by testing rather than by CI: the harness always passes --only, empty when the caller gave none, and the first version treated an empty value as "select nothing" instead of "no filter" -- every shard would have run zero tests and reported success. Empty now means no filter, a misspelled --only exits non-zero rather than running a silently smaller set, and both are tested. Verified: node --test tools/ci/postgres-partition.test.mjs -> 19 pass, 0 fail (including completeness/disjointness over the real map); check-postgres-cargo-map -> OK, facets 68/28/37/33/43; check-ci-preflight -> passed, 61 pass 0 fail; dark-suites --strict -> exit 0; npm run check:ci-preflight -> exit 0; bash -n on the harness. Stacked on #811 because it edits the same shard jobs' surroundings; retarget to main once that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #810. Review the top commit only.
The measurement
Free runner diskran in sevenci.ymljobs. Runner state immediately before it ran (runs32111278481shard,32108350947backend):After: 110G available. It spends 58s (shards) to 82s (backend) reclaiming 23G on a machine that already had 87G free — and
grepfinds zeroENOSPCor "no space" in any job log, ever.Chesterton's Fence
It arrived in PR #155 with the dev-up orchestrator, which brings up a full Docker compose project — that genuinely can exhaust disk. It was then applied to jobs that start one postgres container.
dev-up-smokehas since moved tonightly.yml, so the justification and the cost no longer live in the same workflow.nightly.ymlkeeps the step for exactly that reason; onlyci.ymldrops it.Why this is safe to remove on evidence
It's self-verifying: if disk were ever scarce the job fails loudly with
ENOSPC. This cannot degrade into a false green, which is what distinguishes it from removing a correctness gate.Value
backend— which becomes the critical path once ci: duration-weighted shard partitioning (module + measured weights, not wired) #810's rebalance landsTwo traps worth recording
The mirror pins setup actions by ABSOLUTE step index, so removing a step shifts every index after it. Decremented per job rather than re-derived.
company-conformanceappears in BOTHrequiredRunStepCountsandrequiredActionStepCounts— a first-match replace silently edited the run-step map instead, moving a count this change doesn't touch. The suite caught it ascompany-conformance setup-action inventory drifted. Both are now set explicitly.Ratchets
Step down by exactly what was removed and no more:
Verification
check-ci-preflight.mjs→ passed ·check-ci-preflight.test.mjs→ 60 pass, 0 fail ·verify.test.mjs→ 15 pass ·check-nightly-workflow→ 14 pass · full chain → exit 0🤖 Generated with Claude Code