ci: run every PostgreSQL shard on cargo-nextest, not just the pilot

[jason931225]

#840 put one shard on nextest to measure it. Measured on merge_group run 32402124838, the pilot against its nearest neighbour:

shard	targets	runner	wall
domain adapters B	45	nextest	6m
domain adapters A	40	cargo	15m

More tests, less than half the wall clock — the 2.35× benchmark reproduced on real CI against a bigger shard. The pilot has done its job, so the remaining four shards move to the same hash-pinned installer and runner.

What this does not do — and the measurement matters more than the change

The critical path of a merge_group run is not the shards:

Backend — fmt / clippy / test / gates   17m   <- critical path
Test PostgreSQL — platform              15m
Test PostgreSQL — domain adapters A     15m
Test PostgreSQL — app                   15m

Taking the shards to ~6–7m leaves wall clock pinned near 17m by Backend, whose own 17m is an accumulation rather than a hot spot: 5m dev-auth PG suites, 3m clippy, 3m console-app unit, ~4m setup, and 24 further steps at or under a minute. Splitting it is the next lever and a much larger ci.yml change.

A second measurement, taken locally while checking isolation: the platform shard runs 154 tests in 263s, and 263s of that is one test — console-gate-writer-ownership::census_executes_against_postgres. No runner change takes that shard below ~4.5m. Shards are not uniformly runner-bound, and the remaining gains are not evenly distributed.

Isolation — the actual risk, and why it was piloted

nextest runs test binaries in parallel where cargo ran them serially, so a suite sharing state would newly fail. Verified locally before pushing: the platform shard passes 154/154 under nextest, with the cluster-global serial group in .config/nextest.toml doing its job.

Ratchets moved up: run-step coverage 126 → 130, bypass matrix 378 → 390, because each new install step goes through the same three mutations as every other. 61 preflight contract tests pass.

🤖 Generated with Claude Code

[jason931225]

Executive Overview & Adversarial Quality Assessment

PR #853 rolls out cargo-nextest across the remaining four PostgreSQL test shards (app, platform, ontology, domain-a), generalizing the pilot implemented in PR #840 (domain-b). The migration replaces serial test binary execution with parallel scheduling while preserving database isolation via cluster-global serial groups in .config/nextest.toml.

The PR is exceptionally well-structured and disciplined: workflow steps use hash-pinned binary installations via tools/ci/install_nextest.sh, preflight step contracts and SHA-256 proof digests are strictly maintained, and preflight bypass ratchets are incremented monotonically (run-step count 126 → 130; mutation count 378 → 390).

---

16-Lens Canonical Framework Analysis

Lens	Evaluation	Status
1. Cartesian Doubt	Solves the shard duration bottleneck proven during pilot (15m → 6m). Accurately identifies that critical path remains bound by monolithic Backend (17m).	Pass
2. Essentialism / YAGNI	Minimal and necessary; uniformly applies the proven pattern across all 4 remaining reachability facets.	Pass
3. Chesterton's Fence	Respects prior CI design and verification architecture by upholding all step contract proofs and anti-bypass constraints.	Pass
4. Contrarian / 10x	Validates the nextest optimization across shards while properly deferring the larger Backend pipeline decomposition to subsequent PRs.	Pass
5. Socratic	Contracts are tight. Identified a minor schema inconsistency in postgresReachabilityFacetSetupCommands where domain-b uses an Array while other facets use a joined String.	Minor Nit
6. Pragmatism	Balances immediate ~36 compute-minute CI reduction with zero runtime production risk.	Pass
7. Red Team	Pinned binary download is verified against NEXTEST_PINNED_SHA256 and executed within isolated runner temp paths.	Pass
8. Systems Thinking	Relieves GitHub Actions runner pool contention across concurrent PR merge queues.	Pass
9. Operability / Day-2	Nextest provides superior test failure granularity, execution timing, and targeted rerun ergonomics.	Pass
10. Opportunity Cost	Standardizes all database test shards onto a single runner implementation, reducing cognitive overhead.	Pass
11. Blast-radius	Confined strictly to CI test execution; preflight contract tests guarantee harness integrity before merge.	Pass
12. Constant-work	Parallelized test binary dispatch bounds execution time across multi-core CI runners.	Pass
13. Shared-nothing	Independent disposable Postgres instances prevent cross-shard state pollution.	Pass
14. FinOps / Unit-cost	High-impact FinOps win: saves ~36 runner-minutes per full CI run across 4 shards.	Pass
15. Telemetry-first	Shard timing benchmarks and nextest execution metrics are fully visible.	Pass
16. Zero-trust / Defense-in-depth	Preflight ratchet validation increased from 378 to 390 bypass mutations to prevent unverified skips.	Pass

---

Key Findings & Polish

1. Data Structure Consistency (check-ci-preflight.mjs): postgresReachabilityFacetSetupCommands["postgres-reachability-domain-b"] remains an array literal [...] while the other four facets map to the string nextestInstallSetupCommands. Normalizing domain-b to nextestInstallSetupCommands improves type uniformity.
2. Comment Typo (check-ci-preflight.test.mjs): Duplicate sentence fragment ("Coverage GREW; ... Coverage GREW: the new ...") in the ratchet history doc comment.

Findings not addressable in the diff

• scripts/check-ci-preflight.mjs:582 [RIGHT] -- diff unavailable or unparseable: Minor consistency nit: postgres-reachability-domain-b is still assigned an array literal [...], whereas app, platform, ontology, and domain-a use nextestInstallSetupCommands (which is .join("\n")). Consider setting "postgres-reachability-domain-b": nextestInstallSetupCommands to maintain uniform type semantics across all shard keys.
• scripts/check-ci-preflight.test.mjs:703 [RIGHT] -- diff unavailable or unparseable: Minor documentation polish: Stuttered copy-paste in comment (Coverage GREW; each new install step goes through the same three bypass mutations as every other run step. Coverage GREW: the new step goes through the same three bypass mutations as every other one.). Consider deduplicating the repeated phrase.

---

🤖 [Reviewed] by Oyatie Anvil

[jason931225]

❌ Blocked — 12 finding(s) across 72 gates; 9 gate(s) produced no measurement.

• doc-parity — errored: doc parity probe supervision failed: Execution failed: doc parity probe exited with status exit status: 1: Error: Eligibility check failed: UNAVAILABLE (code 503): The service is currently unavailable.
  • fix: update the affected docs, or add an ADR under docs/decisions/
  • note: this gate is partial fidelity and does not fully measure what its name implies
• clean-arch — not measured: no core/ports/adapters/facade layering found: 0 of 1 file(s) examined belong to a recognised layer
• slo — not measured: no Prometheus or OpenTelemetry endpoint is configured, so error budget consumption over any window was never queried
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• cluster-audit — not measured: no Kubernetes API or ArgoCD cluster access is configured, so no live state was read back and no comparison against Git was performed
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• ci-wallclock — not measured: no GitHub Actions workflow-run timing API access is configured, so neither this PR's CI duration nor its billable compute was read
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• remote-cache — not measured: no sccache or Buck2 CAS statistics endpoint is configured, so no cache hit rate was read and no lockfile was hashed
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• shadow-traffic — not measured: no traffic mirror and no replay target are configured, so no production requests were sampled and no responses were compared
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• brand-absence — failed: 12 name(s) or PR-visible string(s) stamp an aspiration instead of naming what the code verifies
• shape — warning: no shape spec adopted (.anvil/shape.json absent); see anvil shape validate-spec
  • fix: run anvil shape plan --repo-dir <clone> for the move plan; a regression on a blocking rule needs an entry in .anvil/baselines/shape.signoff.json
  • note: this gate is partial fidelity and does not fully measure what its name implies
• automated-canary — not measured: no canary deployment and no Prometheus or OpenTelemetry metrics endpoint are configured, so no baseline or canary latency samples were ever read
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• stacked-diffs — not measured: no pull request DAG was read from the forge, so this PR's parent branch and any children stacked on it are unknown and no stack was evaluated
  • note: this gate is aspirational fidelity and does not fully measure what its name implies
• microbench — not measured: no criterion benchmark harness or published baseline exists for this repository, so neither the base nor the head ns/op figure was ever measured
  • note: this gate is aspirational fidelity and does not fully measure what its name implies

---

🤖 [Blocked] by Oyatie Anvil

[jason931225]

Executive Overview

This incremental update to PR #853 introduces a targeted [[profile.ci.overrides]] rule in .config/nextest.toml for the census_executes_against_postgres test binary, raising its slow-timeout ceiling from 360s (120s × 3) to 960s (120s × 8).

During the rollout of cargo-nextest across all PostgreSQL test shards (expanding beyond the domain adapters B pilot), the platform shard encountered timeout terminations on hosted CI runners. While the heavy census test completes in ~263s in optimized local environments, hosted virtual runners experience higher I/O and Docker virtualization overhead across the test's 8 sequential migration and enforcement cycles. This override permits the long-running serial binary to complete reliably without loosening the 360s termination ceiling across the other 153 tests in the shard.

---

16-Lens Canonical Adversarial Review Matrix

#	Lens	Evaluation & Adversarial Findings
1	Cartesian Doubt	Validated. The override directly targets the empirical failure mode (hosted virtualization overhead on sequential DB migrations) rather than arbitrarily bumping CI timeouts globally.
2	Essentialism / YAGNI	Minimal. The diff adds only 13 lines of configuration and documentation, applying the override strictly to the single offending binary target.
3	Chesterton's Fence	Preserved. The global 360s timeout ceiling (period = "120s", terminate-after = 3) was established to detect genuine deadlocks in PostgreSQL suites; it remains intact for all other test targets.
4	Contrarian / 10x	Opportunity. The 8 sequential migration+enforce passes represent a legacy monolithic test pattern. Isolating them into ephemeral schemas or database instances would allow nextest to run all 8 passes concurrently.
5	Socratic Contract	Sound. Filter target binary(census_executes_against_postgres) cleanly captures the integration test harness binary, accommodating the internal census_binds_to_an_executed_database test routine.
6	Pragmatism	Approved. Unblocks the remaining 4 shards from adopting nextest (halving shard runtimes from ~15m to ~6m) without requiring a high-friction refactor of the census suite.
7	Red Team / Security	Low Risk. The 960s ceiling prevents indefinite runner hangs or denial-of-service in CI workflows, operating well within standard CI job timeout limits.
8	Systems Thinking	Cohesive. Works in tandem with the .config/nextest.toml cluster-global serial group to prevent Postgres connection and state collisions during parallel test execution.
9	Operability / Day-2	High Visibility. period = "120s" ensures nextest emits periodic slow-test heartbeat warnings to CI logs every 2 minutes, preventing silent execution blackholes.
10	Opportunity Cost	Favorable. Enables immediate compute savings across all PostgreSQL shards with negligible configuration maintenance overhead.
11	Blast-Radius / Cell-based	Contained. Blast radius is strictly constrained to binary(census_executes_against_postgres); no other platform, adapter, or app tests inherit the extended timeout.
12	Constant-Work / Anti-fragility	Robust. The 960s threshold absorbs standard variance in hosted CI hypervisor and disk I/O performance under multi-tenant load.
13	Shared-Nothing	Verified. Serial execution of the 8 cycles within the single binary guarantees migration script idempotency without interfering with concurrent shard execution.
14	FinOps / Unit-Cost	Positive. Unlocks a ~50% reduction in runner-minutes across 4 shards, significantly reducing total CI billing and queue contention.
15	Telemetry-First	Enabled. Nextest outputs structured JUnit timing telemetry per binary, allowing automated tracking of timing drift in the census harness.
16	Zero-Trust / Defense	Compliant. Follows the principle of least privilege in CI runner configuration by isolating the timeout extension to an explicit filter.

---

Critical Risks & Recommendations

1. Ceiling Proximity to Critical Path: 960s (16m) is close to the 17m Backend critical path. While this binary currently finishes in ~5–6m on hosted runners, any substantial test additions to census_executes_against_postgres could make this binary the new critical path bottleneck.
2. Decomposition Roadmap: File a tracking issue to decompose the 8 sequential migration/enforcement cycles into discrete integration tests backed by isolated database schemas/templates.

Findings not addressable in the diff

• .config/nextest.toml:42 [RIGHT] -- diff unavailable or unparseable: filter = 'binary(census_executes_against_postgres)' correctly scopes the extended slow-timeout to the integration test binary target. Note that while the preceding docstring mentions census_binds_to_an_executed_database (the test case function), the nextest filter properly matches the compiled binary harness name census_executes_against_postgres.
• .config/nextest.toml:43 [RIGHT] -- diff unavailable or unparseable: slow-timeout = { period = "120s", terminate-after = 8 } raises the termination ceiling to 960s (16 minutes) with 120s heartbeat logging. This accommodates the 8 sequential migration/enforce cycles on resource-constrained hosted CI runners without weakening global timeout protections for the remaining 153 tests. As a follow-up item, consider splitting the 8 migration cycles across ephemeral databases/schemas to re-enable parallel execution under nextest.

---

🤖 [Reviewed] by Oyatie Anvil

[jason931225]

Measured outcome. This PR predicted the shards would reach ~6–7m and — explicitly — that wall clock would not move until Backend was split. Both held.

Shards across merge_group runs since: app 7–8m, platform 8–15m (noisy), ontology 5m, domain-a 4–6m, domain-b 5–6m — from 13–15m before.

Wall clock stayed ~19m until #855 split Backend, exactly as stated. It is now 14m after #855 and #857.

One correction to the reasoning here, for anyone reading this as precedent: the 2.35× runner ratio is real for these PostgreSQL suites, but it does not generalise to unit-test jobs. I measured a 27× cargo-vs-nextest ratio on the domain-unit lib sweep locally and nearly shipped a PR on it — that turned out to be a macOS process-spawn artifact applied to ~1s of real execution on the runner, where that job's 714s was 685s of compilation. See #857: the fix there was a rust-cache key, not a runner.