Skip to content

fix(ci): size Windows sccache and reject near-zero warm reuse - #1598

Merged
drsnuggles8 merged 8 commits into
masterfrom
feature/windows-sccache-warm-misses-1584
Oct 12, 2026
Merged

drsnuggles8 merged 8 commits into
masterfrom
feature/windows-sccache-warm-misses-1584

Conversation

@drsnuggles8

@drsnuggles8 drsnuggles8 commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

The hosted MSVC Release cache restored successfully but returned 0 hits because its 2G cap evicted matching objects before their compile requests. Size it from a complete current-object measurement: 2070 objects / 2688.31 MiB, with a 3200M cap and 511.69 MiB (19.03%) growth reserve. The existing shared-cache ceiling remains unchanged.

Add a native-JSON health guard that fails visibly for near-zero warm reuse, unavailable/malformed statistics or incomplete builds. Report the matched restore key, counts, rate and exact bytes as an artifact. Stamp normal hosted lineages with actual compiler bytes/path, MSVC/SDK/image and sccache identity, so image rolls start an explicit cold lineage. Only successful non-PR builds can bank compiler snapshots. The opt-in 8G residency probe cannot save its compiler cache.

Refs #1584. The issue remains open: default-branch producer and following-PR recovery proof requires this fix to land after explicit merge authorization.

Diagnosis and history

Earlier fixes repaired missing entries/budget refusal (#1073/#1075), producer cadence (#1009/#1084), ASan setup (#1082), and compiler/SDK identity (#1118/#1170). They did not size today's MSVC object set. The new guide records their evidence limits.

  • Matched 402f1ce55 producer/nightly: 0/2060 hits at 2G; image 20260925.250.1, MSVC 19.51.36260.0, zero cache errors. Same-commit ASan control: 2015/2017 hits (99.90%).
  • Fresh master 38166502504 and 8G no-save probe 38168947158 restored the same snapshot and compiled the same C++ tree. At 2G: 0/2070 hits, 135m33s. At 8G: 586/2070 hits, exactly all 586 requested keys resident at restore; 110m44s. New objects alone exceeded 2G. Both builds succeeded, without cache errors.
  • Exact sizing probe 38177455638 restored fresh master: all 1208 restored keys hit, 862 misses; complete 2070-key / 2,818,896,940-byte current set, no missing keys, 61m57s. windows-cache-probe and windows-cache-health artifacts contain the derived evidence. This is sizing evidence, not the target recovery rate.
  • Local unchanged-input MSVC/sccache control restarted its daemon between each pass: 0/5 warm hits at 1MiB, 5/5 at 8MiB, full set 1,318,365 bytes. Every compile used the shared build lock.

Live storage limit: 25 GB. The dated listing contains 10,596,655,761 bytes; non-Release entries cost 8,450,054,529 bytes. Budget arithmetic records selected overlaps: three full new Release snapshots, the legacy snapshot and measured dependency/ASan rotations. Simultaneous replacement of the entire fleet can still exceed the limit; confirmed-save pruning and the janitor remain necessary. No cache purge, quota increase, ASan cap change or self-hosted enablement.

Verification matrix

Cell Evidence / state
Hosted master baseline Run 38166502504: successful build/save, 0/2070, 2G, 135m33s
Matched hosted capacity/sizing controls Runs 38168947158 and 38177455638 fully green; complete probe/native JSON artifacts, all initially matching objects reused
New cold default-branch producer Not yet run: new fingerprinted workflow is not on master; run after authorized merge
New warm master producer Not yet run: needs the preceding default-branch producer; target 94.8-98.5% with source drift accounted for
Following PR consumer Not yet run: needs shared-master cache; full build/tests and matched toolchain evidence required after merge
Warm failure / cold / malformed / partial controls 12 passing helper tests, including CLI error annotation, artifact and summary checks; native v0.15 JSON parser verified
Hosted guard integration Sizing run 38177455638: healthy-warm, 1208/2070. PR run 38182446948, build 114597678279: correct cold disposition, no restore, 0/2070 hits, zero errors, 2,818,897,842B under 3,355,443,200B cap; full build 218m29s passed. all three Windows test shards and coverage passed
ASan Same-source control 38103213360: 2015/2017 (99.90%). Current PR run 38182446933, build 114597891966: 2025/2027 (99.90%), zero cache errors, build and all four test shards/coverage passed. ASan workflow/flags/cap unchanged
Fork and self-hosted routing Existing fork boundary preserved; probe forced hosted/no-save. Self-hosted switch remains off and no Windows runner registered
Local checks 12 helper tests, actionlint, docs consistency and all-files pre-commit pass
Runtime/render paths Workflow/helper changes only; no C++/CMake, binding or asset behavior changed. Hosted app smoke/tests remain in CI

Found and fixed

  • 77e42efe5 fix(ci): shard XML upload referenced undocumented runner.workspace; use github.workspace. Found while inspecting artifacts; actionlint and hosted probe shard uploads verify the context.
  • cad761119 fix(ci): helper CLI fixtures inherited real Actions summary/output destinations; isolate them and reject invalid unsigned counters. CLI regression tests verify the behavior.
  • a53f3f1a8 ci(cache): native ANSI log prefixes contaminated object names; strip escapes and test real-formatted records. Also correct cache-prune's misleading storage-limit diagnostic.
  • f7e97b2ce docs(cache): shared save-action diagnostics conflated the historical 10 GB/billing refusal with today's storage limit; clarify the error without changing save/prune logic.

Filed, not fixed

None.

Review guide

Where I'd look hardest

  1. Windows.yml hosted identity/restore/guard/save sequence: prefix matches must count as warm, mismatched toolchains must stay cold, partial builds must never bank snapshots.
  2. check_sccache_health.py: native counters must be consistent and not double-count advanced language breakdowns; empty or malformed evidence must fail.
  3. Probe/current-set and shared-budget arithmetic: obsolete objects must not inflate the measured working set; the diagnostic 8G cap must never be saved.

What I verified, and how - Paired hosted controls, exact current-set artifact, restarted-daemon eviction reproduction, twelve helper tests, actionlint/docs/pre-commit, and independent standards/spec reviews. Independent current-head reviews and CodeRabbit are clean. All six PR workflows passed at 95500a304: 31 successful checks and two inapplicable hosted ASan+LSan shard jobs skipped because the self-hosted arm ran its full tests inline. Zero unresolved review threads; PR mergeable.

Least confident about - How quickly future graph/source growth consumes the measured reserve. The artifact and visible guard make that measurable; remote default-branch/following-PR acceptance is explicitly still pending.

@drsnuggles8

Copy link
Copy Markdown
Owner Author

Self-review @ 95500a304

Reviewed the current diff through independent standards and specification passes.

  • Findings fixed: helper fixture Actions destinations, invalid counters, native ANSI prefixes, unsupported XML upload context, historical cache diagnostics, and a budget overclaim.
  • The final budget table explicitly covers selected overlaps and quantifies the over-limit simultaneous whole-fleet replacement case; it does not promise a universal concurrency bound.
  • No unresolved self-review findings. Twelve helper tests, actionlint, documentation consistency and all-files pre-commit passed before this push.
  • Root cause is supported by the paired hosted capacity contrast and complete current-set artifact. Master/following-PR recovery acceptance remains pending authorized merge.

PR CI and CodeRabbit are now running; pending checks are not success.

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository YAML (base), Repository UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 87ca2528-c07b-4161-b6d4-b0118361b233





📥 Commits

Reviewing files that changed from the base of the PR and between ada4198 and 95500a3.






📒 Files selected for processing (11)
  • .github/actions/save-cache-pruned/action.yml
  • .github/workflows/Windows.yml
  • .github/workflows/cache-prune.yml
  • docs/agent-rules/README.md
  • docs/agent-rules/actions-cache-budget.md
  • docs/agent-rules/sccache-cap-vs-object-set.md
  • docs/agent-rules/windows-sccache-warm-reuse.md
  • scripts/check_sccache_health.py
  • scripts/test_sccache_health.py
  • scripts/test_sccache_probe.py
  • scripts/windows_cache_probe.py





Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.







📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough

Walkthrough

The Windows workflow now uses a larger sccache cap and fingerprinted cache lineage. It adds an opt-in cache probe, production cache-health checks, and supporting scripts, tests, and documentation for cache capacity and reuse diagnosis.

Changes

Windows sccache cache flow

Layer / File(s) Summary
Capacity evidence and budget
.github/actions/save-cache-pruned/action.yml, .github/workflows/Windows.yml, .github/workflows/cache-prune.yml, docs/agent-rules/actions-cache-budget.md, docs/agent-rules/sccache-cap-vs-object-set.md, docs/agent-rules/windows-sccache-warm-reuse.md, docs/agent-rules/README.md
The configured cache cap changes from 2G to 3200M. The documentation records object-set measurements, storage overlap estimates, and guidance for comparing cache residency and computed keys. Cache-limit descriptions are updated.
Cache lineage and probe
.github/workflows/Windows.yml, scripts/windows_cache_probe.py, scripts/test_sccache_probe.py, docs/agent-rules/windows-sccache-warm-reuse.md
Manual dispatch can enable a hosted cache probe with an 8G cap and no cache save. Normal restores and saves use a fingerprint based on compiler, MSVC, Windows SDK, image, and sccache identity. Probe reporting compares cache snapshots with compiler logs and statistics.
Build health checks and validation
.github/workflows/Windows.yml, scripts/check_sccache_health.py, scripts/test_sccache_health.py, docs/agent-rules/README.md, docs/agent-rules/windows-sccache-warm-reuse.md
Hosted builds collect sccache statistics and classify cache health using the restore key and build outcome. The workflow uploads health artifacts, and tests cover health classifications, CLI results, and probe reports. The test-results artifact path now uses github.workspace.

Sequence Diagram(s)

sequenceDiagram
  participant Windows workflow
  participant sccache
  participant windows_cache_probe.py
  participant Probe artifact
  Windows workflow->>sccache: Restore cache and run build
  Windows workflow->>windows_cache_probe.py: Provide cache snapshots, log, and statistics
  windows_cache_probe.py->>Probe artifact: Write residency and lookup report
Loading











Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 95500

No actionable merge-blocking issue was established. The planned master and following-PR runs remain necessary to confirm warm-cache recovery after merge.

Pre-merge checks | Passed 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main changes: sizing Windows sccache and rejecting near-zero warm cache reuse.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@drsnuggles8

Copy link
Copy Markdown
Owner Author

Final verification @ 95500a304

All six PR workflows passed: Windows 38182446948, Sanitizers 38182446933, Vulkan-off 38182446926, Steam stub 38182446924, shader floor 38182446961, and pre-commit 38182446932. All 31 applicable checks succeeded; two hosted ASan+LSan shard jobs are intentionally skipped because the self-hosted arm ran its full suite inline (that test step passed).

Windows build 114597678279 passed in 218m29s. Artifact windows-cache-health 11724055428 confirms the new lineage is correctly cold, with no restored key, 2070 requests/misses, zero cache errors, 2,818,897,842 bytes of objects under the 3,355,443,200-byte cap. Cache save correctly skipped on the PR. All three Windows test shards and coverage passed. Windows ASan build 114597891966 preserved 2025/2027 hits (99.90%) and all four test shards/coverage passed.

Independent standards/specification self-reviews cover this exact head; CodeRabbit covered the same head with no actionable comments. No unresolved review threads. PR is mergeable and the worktree is clean.

Merge authorization gate: CLAUDE.md explicitly gates PR merging. The new default-branch producer, warm matched master producer and following-PR recovery proof cannot run until this fix lands. #1584 stays open; no final 94.8-98.5% recovery claim yet.

@drsnuggles8
drsnuggles8 merged commit a60137f into master Oct 12, 2026
33 checks passed
@drsnuggles8
drsnuggles8 deleted the feature/windows-sccache-warm-misses-1584 branch October 12, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant