Repository navigation
Measure and reduce Rust/Nix CI compilation latency safely #149
Description
Problem
Oxid has no reproducible cold/warm CI performance baseline, while required checks repeatedly realize a broad Nix shell and compile overlapping Rust graphs with different profiles. CI caches registries and a large commit-keyed target/ tree, but lacks a measured compiler-cache strategy; Cargo, coverage, docs, and Nix packaging cannot safely reuse all work. The broad composition dependency cone also makes targeted UI/headless checks compile adapters they do not require.
Optimizing by uploading raw Rust artifacts or introducing crates without measurement risks stale-feature reuse, cache poisoning, larger graphs, and slower full-workspace builds. CI duration must be improved as a quality attribute while preserving exact test/security evidence.
Acceptance criteria
- Cold and warm medians and p90 are recorded for devshell realization, check/clippy, unit tests, UI feature matrix, headless tests, coverage, docs, Nix packages, and Compact artifacts.
- Measurements include compiler timings, target/cache size, Nix closure size, runner characteristics, toolchain, lockfile, target, features, and profile.
- A minimal CI devshell is evaluated without Dioxus/agent/Compact tooling that required PR jobs do not need.
- Pinned
sccacheor an equivalent content-addressed compiler cache is evaluated with keys that preserve compiler/target/profile/feature/source correctness. - Coverage-instrumented objects remain isolated from ordinary builds.
- Fork/untrusted caches cannot influence privileged release/signing jobs and no credentials enter caches.
-
cargo nextest, job parallelism, path scoping, and Nix dependency-layer approaches are accepted only with measured equivalent confidence. - New compile-unit crates require an architectural capability and measured targeted-build benefit; full-workspace time may not regress more than 5%.
Definition of done
- Required PR CI remains at least as strict and has a documented <=10-minute target.
- Warm median improves by an agreed threshold (initial hypothesis: >=20%) or the experiment is rejected with evidence.
- Cache invalidation and rollback procedures are tested.
- CI-time regression measurement is part of the quality strategy.
Non-goals
- Weakening required checks or hiding flaky tests with retries.
- Reusing raw
rlibbundles across incompatible builds. - Adding micro-crates solely for cache keys.
- Running private/device/credential evidence in public CI.
Activity
yshyn-iohk commentedon Aug 31, 2026
Measured evidence from the current stabilization run:
UI and application profiles;Optimized UI release artifacttook 16m31s, while the relevant headless lane took 6m09s.run.shis intentionally classified as a global build input, so this one-time governance change selected every hosted lane.headless-linuxbecausescripts/architecture/capability-facades.jsoncurrently falls through to the genericcorepath rule. The actual affected runtime target is UI/desktop.Recommended bounded follow-up under this issue:
scripts/architecture/**and the capability-facade checker as repository-governance inputs handled by the basic architecture gate; combine that area with actual changed component areas instead of promoting the change to shared core;integrationpush backstop, but define a feature-PR desktop/UI lane with a <=10-minute target;This preserves fail-closed integration coverage while removing known unrelated work from feature PRs.
yshyn-iohk commentedon Sep 1, 2026
Incremental exact-head rerun evidence from #237
A required signed base sync caused a second exact-head run of the same macro change after #236 advanced
integration. This gives a useful cache comparison against the immediately preceding run:Interpretation: registry/object caches materially help some debug/test lanes, but they do not solve the 16–18 minute release/UI critical path. Path-aware fanout and avoiding avoidable stale-base reruns remain the larger levers. The factory should also avoid stacking independent changes behind a PR that is about to merge unless the dependency is real; otherwise strict current-base freshness pays the full matrix twice.
yshyn-iohk commentedon Sep 5, 2026
Additional exact-head evidence from PR #269 at f989967: the UI/application profile lane completed all builds/tests and printed final sccache statistics, then GitHub cancelled it during post-job cleanup at the hard 20-minute job limit. A failed-lane-only rerun completed successfully in 17m07s. This confirms the current 20-minute limit has insufficient cleanup/runner-variance margin even when product checks pass; it can turn successful work into a red repository aggregate. No product-code change was required in #269.
yshyn-iohk commentedon Sep 6, 2026
Worktree triage on 2026-09-06 preserved the historical six-commit experiment under an exact private archive ref and removed its stale checkout. The branch assumptions predate the current split ci-rust/ci-ui/ci-coverage/ci-quality shells and sccache design, and its own evidence did not establish hosted cold/warm latency or the proposed improvement threshold. Do not replay it mechanically; restart the remaining measurements from current develop. No acceptance criterion is being marked complete.
yshyn-iohk commentedon Sep 6, 2026
Fresh current-design evidence from PR #297 (run 34043049850, job 101513144700): the
unit-linuxjob passed in 630 s wall time; itsCompile and run workspace unit testsstep took 613 s (15:40:54Z–15:51:07Z). That is 13 s above the documented 600 s command SLO, though still inside the 12-minute job cap. Basic passed in 103 s and headless in 367 s.This is now a measured current
ci-rust/sccache case, not evidence for reviving the archived pre-split implementation. The next iteration should inspect the emitted sccache hit/miss statistics and unit ownership, then make one isolated change from currentdevelop; do not weaken or skip the unit gate to hide the SLO miss.yshyn-iohk commentedon Sep 6, 2026
Additional cache evidence identifies the immediate bottleneck:
developpush job 101265915223 was configuredREAD_WRITE, but also reported 0 hits and 1,037 write errors. It therefore seeded no objects for later PRs.The next slice should first expose the actual GHAC write error class on the trusted writer, then test one cache change. Check current sccache GHA key configuration (
SCCACHE_GHA_CACHE_TO/FROM) and the upstream v0.17.0 retry/rate-limit gap before changing runner topology. Upstream evidence: mozilla/sccache#2821 and current configuration: https://github.com/mozilla/sccache/blob/main/docs/Configuration.md.Acceptance should require: one trusted develop/milestone writer with zero unexplained write errors, then a fresh PR run with a nonzero Rust hit rate and unit command <=600 s. Do not make the gate optional; fix the cache evidence.
yshyn-iohk commentedon Sep 8, 2026
CI evidence from PR #353 (exact head
44727f08c849458303ce1ec27620101f5a8b30ae, 2026-09-08): Basic 1m36s; headless 5m41s; unit 10m21s; UI/application profiles 20m02s. The UI-only product slice added one capability-facade inventory entry, which conservatively selected every Rust lane. The unit lane reported 1,228 compile requests, 0 cache hits, 1,037 misses, and 1,037 read-only sccache write errors. This is a useful cold-cache baseline for the target-planner/cache work; no gate was weakened in #353.yshyn-iohk commentedon Sep 12, 2026
2026-09-12 cache audit confirms that capacity, not missing cache syntax, is the immediate blocker:
nix7-build-Linux-*entries alone consume about 8.6 GB, including duplicate current identities scoped to bothdevelopandmilestone-0.2.0;Resource not accessible by integration;contents: read, and grantingactions: writeto that PR-capable job would expose destructive cache authority to candidate code.Bounded remediation is tracked in #429: keep
developas the single trusted Nix archive writer, remove ineffective purge from the PR-capable job, and precisely reclaim superseded Nix entries while preserving the newest default-branch archive. After capacity is restored, the next trusted writer and Rust PR must record the actual sccache backend result. If writes still fail, expose the HTTP error class before changing topology, consistent with the existing upstream v0.17.0 retry/rate-limit concern.No gate should be weakened, and #166 remains the longer-term signed Nix binary-cache design.
yshyn-iohk commentedon Sep 12, 2026
Post-#430 canary evidence rules out repository cache-capacity pressure as the current sccache writer failure.
developpush run 34684273888, exact head1f2f2d54588b5da4e00bf3d14d24c76899c9f4be, completed all full-profile lanes green without supersession.ghacwithsccache-v0.17.0-oxid-v1.The repository already uses one trusted writer, so the upstream multi-writer HTTP-429 diagnosis in mozilla/sccache#2821 does not explain the all-object failure by itself. The next bounded slice should expose a redacted backend error class from the trusted writer (HTTP status/category, no token or URL credentials) before changing cache topology. Do not raise test timeouts or weaken gates to hide this.
6 remaining items
yshyn-iohk commentedon Sep 13, 2026
PR #500 provides another exact-head compilation-latency sample from CI run 34773643414 (2026-09-13):
basic: 1m43sheadless-linux: 6m27sunit-linux: 10m57s (57s beyond the 10-minute target)ui-linux: 21m14s (critical-path outlier)The ownership-map update conservatively classified the PR as
ci, core, docs, harness, ui, so all four lanes ran. No gate was weakened and all active replacement lanes were green. Full work-item metrics: #500 (comment)yshyn-iohk commentedon Sep 13, 2026
Fresh measurement from PR #506 (
a41f170): unit-linux passed in 10m58s (target 10m, +58s); headless-linux passed in 4m34s. The prior head on the same one-file test PR recorded unit 10m34s and headless 5m03s. This repeat suggests unit compilation latency rather than the new replay assertion; no gate budget was widened.yshyn-iohk commentedon Sep 13, 2026
PR #508 adds another exact-head CI measurement. A test-only change under
apps/oxid-headless/tests/portal_profile_flow.rsselected every Linux product lane, not only Basic + unit/headless: UI 13m49s, optimized UI release 16m36s, coverage 14m14s, Nix package 17m19s, and Compact 3m36s also ran. The relevant lanes were Basic 1m44s, headless 4m26s, and unit 11m05s (1m05s over the ten-minute target). Runner admission added roughly 12–16 minutes of queue time to the heavy lanes; the complete hosted window was 36m48s. This is strong evidence to split test-file ownership from theapps/oxid-headlesspackage-level all-target classification while retaining explicit on-demand/full-branch backstops. The metrics receipt is #508 (comment).yshyn-iohk commentedon Sep 13, 2026
PR #510 exposed a second CI-cost source beyond compilation: lifecycle ordering. The first bounded Pi draft completed in about six minutes, but the supervisor marked it ready before hosted exact-head review. That transition selected every Linux lane because the slice added a root repository-gate registration plus a new unclassified script. Two valid parser findings then required new heads, canceling and restarting the heavy matrix each time.
Recommended flow change: keep implementation PRs draft while requesting hosted Codex review; resolve exact-head review findings and run focused/local gates first; only then mark ready once to admit hosted product lanes. Preserve the conservative planner for root gate changes, but avoid multiplying that cost by review repair pushes. Separately classify narrow repository tooling where ownership is unambiguous. Final timings/metrics will be posted from #510 after merge.
yshyn-iohk commentedon Sep 14, 2026
Measured on #510: the Pi implementation completed in 6m21s, but the delivery took 3h06m45s with 14 repair pushes, 17 canceled workflow runs, and two complete all-target matrices. The final exact-head matrix took 26m53s; UI profiles were the 20m27s product-lane tail.
Draft deferral worked only partially: we marked ready after requesting review, not after observing the hosted reviewer terminal response. A late valid finding therefore invalidated a fully green matrix. Recommended harness rule: draft admission requires an explicit exact-head hosted-review terminal signal (or a recorded bounded-review waiver), not merely a request; then mark ready once. Cap additional review iterations by configured blocking severity, and turn non-blocking findings into linked follow-up issues, as #512 demonstrates.
yshyn-iohk commentedon Sep 14, 2026
2026-09-14 observation: ready transition widens an unchanged milestone PR
PR #513 exposed a reproducible CI admission-cost issue on exact head
09b420464a1f1289605c40a2809202a66d171c57:This is non-blocking for #513, but it spends parallel compute and increases ready-to-merge latency without adding change-specific evidence. A bounded follow-up should preserve/reuse the exact-head plan and successful evidence across the draft-to-ready transition, or rerun only lanes selected by that plan. Milestone acceptance should widen the matrix only when policy explicitly requests it, not merely because draft state changed.
Measured result
The first ready-state matrix completed in 21m26s (UI/application profiles: 21m08s). Two legitimate late review repairs then changed the head and caused the same full matrix to run again for 16m13s, despite the delta being receipt provenance checks; cumulative final-head CI wall time was 46m25s. The repository size adapter also false-blocked the final 43 weighted-LOC shell/test slice because
.shremained unclassified and crossed the 50% ambiguity threshold, although the immediately preceding same-slice result passed at 41 LOC. Evidence reuse should cover review/size classification as well as CI, and the Oxid classifier should explicitly handle shell scripts.yshyn-iohk commentedon Sep 17, 2026
Fresh CI timing evidence from PR #580 exact head
b03630e: unit-linux took 11m35s (695s), exceeding the 10-minute target while basic passed in 1m48s and headless in 6m10s. The final CI critical path was 11m55s. This was a focused wallet-application policy change with sccache enabled; do not widen the gate budget—investigate cache/compiler reuse. Metrics: #580 (comment)yshyn-iohk commentedon Sep 17, 2026
PR #582 exact-head CI adds another reproducible unit-lane over-budget sample:
Unit tests (Linux host)ran 12m18s (07:12:31Z–07:24:49Z) while basic completed in 1m45s and headless in 6m15s. The exact metrics receipt is #582 (comment). This reinforces thatunit-linux, not headless, is the critical-path target for the application-layer slice.yshyn-iohk commentedon Sep 17, 2026
2026-09-17 observation — PR #594 superseded exact-head run
Run 35236300645 restored the cargo-registry cache and the locked Nix-store cache, but object-level compiler caching was ineffective in every measured Rust lane because
SCCACHE_GHA_RW_MODE=READ_ONLYhad no populated objects for the currentoxid-v1key:The run was intentionally cancelled after a valid review finding required a new exact head, so these are diagnostic observations, not final PR metrics. The next optimization experiment should establish one trusted bounded cache-writer path (or remove the nonfunctional read-only object cache) and compare exact-key hit rate plus lane wall time. It should not weaken the required gates.
yshyn-iohk commentedon Sep 29, 2026
Fresh warm-cache evidence from merged PR #838 (exact head
087c9095): basic 1m51s, unit-linux 13m13s, ui-linux 23m05s, total CI wall 23m29s. Queue/admission delay was only about 10–11s, so the regression is execution/compilation rather than runner scarcity. The UI lane nearly consumed its 25-minute cold-fallback timeout despite a read-only compiler cache. Private/public factory metrics are attached to #838. This should be included in the cold/warm baseline and used to prioritize cache-hit diagnostics and UI feature-matrix decomposition; do not weaken the gate.yshyn-iohk commentedon Oct 3, 2026
Milestone 0.2.0 completion audit (2026-10-04): moved to 1.0.0 without closing. This is refactoring, performance, factory/develop-targeted, or reusable-library work rather than a remaining product acceptance criterion for the 0.2.0 milestone. Existing PRs and work remain valid.