You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(ci): size Windows sccache and reject near-zero warm reuse - #1598
The hosted MSVC Release cache restored successfully but returned 0 hits because its 2G cap evicted matching objects before their compile requests. Size it from a complete current-object measurement: 2070 objects / 2688.31 MiB, with a 3200M cap and 511.69 MiB (19.03%) growth reserve. The existing shared-cache ceiling remains unchanged.
Add a native-JSON health guard that fails visibly for near-zero warm reuse, unavailable/malformed statistics or incomplete builds. Report the matched restore key, counts, rate and exact bytes as an artifact. Stamp normal hosted lineages with actual compiler bytes/path, MSVC/SDK/image and sccache identity, so image rolls start an explicit cold lineage. Only successful non-PR builds can bank compiler snapshots. The opt-in 8G residency probe cannot save its compiler cache.
Refs #1584. The issue remains open: default-branch producer and following-PR recovery proof requires this fix to land after explicit merge authorization.
Diagnosis and history
Earlier fixes repaired missing entries/budget refusal (#1073/#1075), producer cadence (#1009/#1084), ASan setup (#1082), and compiler/SDK identity (#1118/#1170). They did not size today's MSVC object set. The new guide records their evidence limits.
Matched 402f1ce55 producer/nightly: 0/2060 hits at 2G; image 20260925.250.1, MSVC 19.51.36260.0, zero cache errors. Same-commit ASan control: 2015/2017 hits (99.90%).
Fresh master 38166502504 and 8G no-save probe 38168947158 restored the same snapshot and compiled the same C++ tree. At 2G: 0/2070 hits, 135m33s. At 8G: 586/2070 hits, exactly all 586 requested keys resident at restore; 110m44s. New objects alone exceeded 2G. Both builds succeeded, without cache errors.
Exact sizing probe 38177455638 restored fresh master: all 1208 restored keys hit, 862 misses; complete 2070-key / 2,818,896,940-byte current set, no missing keys, 61m57s. windows-cache-probe and windows-cache-health artifacts contain the derived evidence. This is sizing evidence, not the target recovery rate.
Local unchanged-input MSVC/sccache control restarted its daemon between each pass: 0/5 warm hits at 1MiB, 5/5 at 8MiB, full set 1,318,365 bytes. Every compile used the shared build lock.
Live storage limit: 25 GB. The dated listing contains 10,596,655,761 bytes; non-Release entries cost 8,450,054,529 bytes. Budget arithmetic records selected overlaps: three full new Release snapshots, the legacy snapshot and measured dependency/ASan rotations. Simultaneous replacement of the entire fleet can still exceed the limit; confirmed-save pruning and the janitor remain necessary. No cache purge, quota increase, ASan cap change or self-hosted enablement.
Verification matrix
Cell
Evidence / state
Hosted master baseline
Run 38166502504: successful build/save, 0/2070, 2G, 135m33s
Matched hosted capacity/sizing controls
Runs 38168947158 and 38177455638 fully green; complete probe/native JSON artifacts, all initially matching objects reused
New cold default-branch producer
Not yet run: new fingerprinted workflow is not on master; run after authorized merge
New warm master producer
Not yet run: needs the preceding default-branch producer; target 94.8-98.5% with source drift accounted for
Following PR consumer
Not yet run: needs shared-master cache; full build/tests and matched toolchain evidence required after merge
12 passing helper tests, including CLI error annotation, artifact and summary checks; native v0.15 JSON parser verified
Hosted guard integration
Sizing run 38177455638: healthy-warm, 1208/2070. PR run 38182446948, build 114597678279: correct cold disposition, no restore, 0/2070 hits, zero errors, 2,818,897,842B under 3,355,443,200B cap; full build 218m29s passed. all three Windows test shards and coverage passed
ASan
Same-source control 38103213360: 2015/2017 (99.90%). Current PR run 38182446933, build 114597891966: 2025/2027 (99.90%), zero cache errors, build and all four test shards/coverage passed. ASan workflow/flags/cap unchanged
Fork and self-hosted routing
Existing fork boundary preserved; probe forced hosted/no-save. Self-hosted switch remains off and no Windows runner registered
Local checks
12 helper tests, actionlint, docs consistency and all-files pre-commit pass
Runtime/render paths
Workflow/helper changes only; no C++/CMake, binding or asset behavior changed. Hosted app smoke/tests remain in CI
Found and fixed
77e42efe5 fix(ci): shard XML upload referenced undocumented runner.workspace; use github.workspace. Found while inspecting artifacts; actionlint and hosted probe shard uploads verify the context.
cad761119 fix(ci): helper CLI fixtures inherited real Actions summary/output destinations; isolate them and reject invalid unsigned counters. CLI regression tests verify the behavior.
a53f3f1a8 ci(cache): native ANSI log prefixes contaminated object names; strip escapes and test real-formatted records. Also correct cache-prune's misleading storage-limit diagnostic.
f7e97b2ce docs(cache): shared save-action diagnostics conflated the historical 10 GB/billing refusal with today's storage limit; clarify the error without changing save/prune logic.
Filed, not fixed
None.
Review guide
Where I'd look hardest
Windows.yml hosted identity/restore/guard/save sequence: prefix matches must count as warm, mismatched toolchains must stay cold, partial builds must never bank snapshots.
check_sccache_health.py: native counters must be consistent and not double-count advanced language breakdowns; empty or malformed evidence must fail.
Probe/current-set and shared-budget arithmetic: obsolete objects must not inflate the measured working set; the diagnostic 8G cap must never be saved.
What I verified, and how - Paired hosted controls, exact current-set artifact, restarted-daemon eviction reproduction, twelve helper tests, actionlint/docs/pre-commit, and independent standards/spec reviews. Independent current-head reviews and CodeRabbit are clean. All six PR workflows passed at 95500a304: 31 successful checks and two inapplicable hosted ASan+LSan shard jobs skipped because the self-hosted arm ran its full tests inline. Zero unresolved review threads; PR mergeable.
Least confident about - How quickly future graph/source growth consumes the measured reserve. The artifact and visible guard make that measurable; remote default-branch/following-PR acceptance is explicitly still pending.
Reviewed the current diff through independent standards and specification passes.
Findings fixed: helper fixture Actions destinations, invalid counters, native ANSI prefixes, unsupported XML upload context, historical cache diagnostics, and a budget overclaim.
The final budget table explicitly covers selected overlaps and quantifies the over-limit simultaneous whole-fleet replacement case; it does not promise a universal concurrency bound.
No unresolved self-review findings. Twelve helper tests, actionlint, documentation consistency and all-files pre-commit passed before this push.
Root cause is supported by the paired hosted capacity contrast and complete current-set artifact. Master/following-PR recovery acceptance remains pending authorized merge.
PR CI and CodeRabbit are now running; pending checks are not success.
Reviewing files that changed from the base of the PR and between ada4198 and 95500a3.
📒 Files selected for processing (11)
.github/actions/save-cache-pruned/action.yml
.github/workflows/Windows.yml
.github/workflows/cache-prune.yml
docs/agent-rules/README.md
docs/agent-rules/actions-cache-budget.md
docs/agent-rules/sccache-cap-vs-object-set.md
docs/agent-rules/windows-sccache-warm-reuse.md
scripts/check_sccache_health.py
scripts/test_sccache_health.py
scripts/test_sccache_probe.py
scripts/windows_cache_probe.py
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
Walkthrough
The Windows workflow now uses a larger sccache cap and fingerprinted cache lineage. It adds an opt-in cache probe, production cache-health checks, and supporting scripts, tests, and documentation for cache capacity and reuse diagnosis.
The configured cache cap changes from 2G to 3200M. The documentation records object-set measurements, storage overlap estimates, and guidance for comparing cache residency and computed keys. Cache-limit descriptions are updated.
Cache lineage and probe .github/workflows/Windows.yml, scripts/windows_cache_probe.py, scripts/test_sccache_probe.py, docs/agent-rules/windows-sccache-warm-reuse.md
Manual dispatch can enable a hosted cache probe with an 8G cap and no cache save. Normal restores and saves use a fingerprint based on compiler, MSVC, Windows SDK, image, and sccache identity. Probe reporting compares cache snapshots with compiler logs and statistics.
Build health checks and validation .github/workflows/Windows.yml, scripts/check_sccache_health.py, scripts/test_sccache_health.py, docs/agent-rules/README.md, docs/agent-rules/windows-sccache-warm-reuse.md
Hosted builds collect sccache statistics and classify cache health using the restore key and build outcome. The workflow uploads health artifacts, and tests cover health classifications, CLI results, and probe reports. The test-results artifact path now uses github.workspace.
Sequence Diagram(s)
sequenceDiagram
participant Windows workflow
participant sccache
participant windows_cache_probe.py
participant Probe artifact
Windows workflow->>sccache: Restore cache and run build
Windows workflow->>windows_cache_probe.py: Provide cache snapshots, log, and statistics
windows_cache_probe.py->>Probe artifact: Write residency and lookup report
Loading
Priority: ➖ Normal
Merge Risk:⚪ Minimal · up to 95500
No actionable merge-blocking issue was established. The planned master and following-PR runs remain necessary to confirm warm-cache recovery after merge.
Pre-merge checks | 4
✅ Passed checks (4 passed)
Check name
Status
Explanation
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title clearly and concisely describes the main changes: sizing Windows sccache and rejecting near-zero warm cache reuse.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
All six PR workflows passed: Windows 38182446948, Sanitizers 38182446933, Vulkan-off 38182446926, Steam stub 38182446924, shader floor 38182446961, and pre-commit 38182446932. All 31 applicable checks succeeded; two hosted ASan+LSan shard jobs are intentionally skipped because the self-hosted arm ran its full suite inline (that test step passed).
Windows build 114597678279 passed in 218m29s. Artifact windows-cache-health 11724055428 confirms the new lineage is correctly cold, with no restored key, 2070 requests/misses, zero cache errors, 2,818,897,842 bytes of objects under the 3,355,443,200-byte cap. Cache save correctly skipped on the PR. All three Windows test shards and coverage passed. Windows ASan build 114597891966 preserved 2025/2027 hits (99.90%) and all four test shards/coverage passed.
Independent standards/specification self-reviews cover this exact head; CodeRabbit covered the same head with no actionable comments. No unresolved review threads. PR is mergeable and the worktree is clean.
Merge authorization gate:CLAUDE.md explicitly gates PR merging. The new default-branch producer, warm matched master producer and following-PR recovery proof cannot run until this fix lands. #1584 stays open; no final 94.8-98.5% recovery claim yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The hosted MSVC Release cache restored successfully but returned 0 hits because its 2G cap evicted matching objects before their compile requests. Size it from a complete current-object measurement: 2070 objects / 2688.31 MiB, with a 3200M cap and 511.69 MiB (19.03%) growth reserve. The existing shared-cache ceiling remains unchanged.
Add a native-JSON health guard that fails visibly for near-zero warm reuse, unavailable/malformed statistics or incomplete builds. Report the matched restore key, counts, rate and exact bytes as an artifact. Stamp normal hosted lineages with actual compiler bytes/path, MSVC/SDK/image and sccache identity, so image rolls start an explicit cold lineage. Only successful non-PR builds can bank compiler snapshots. The opt-in 8G residency probe cannot save its compiler cache.
Refs #1584. The issue remains open: default-branch producer and following-PR recovery proof requires this fix to land after explicit merge authorization.
Diagnosis and history
Earlier fixes repaired missing entries/budget refusal (#1073/#1075), producer cadence (#1009/#1084), ASan setup (#1082), and compiler/SDK identity (#1118/#1170). They did not size today's MSVC object set. The new guide records their evidence limits.
402f1ce55producer/nightly: 0/2060 hits at 2G; image20260925.250.1, MSVC19.51.36260.0, zero cache errors. Same-commit ASan control: 2015/2017 hits (99.90%).windows-cache-probeandwindows-cache-healthartifacts contain the derived evidence. This is sizing evidence, not the target recovery rate.Live storage limit: 25 GB. The dated listing contains 10,596,655,761 bytes; non-Release entries cost 8,450,054,529 bytes. Budget arithmetic records selected overlaps: three full new Release snapshots, the legacy snapshot and measured dependency/ASan rotations. Simultaneous replacement of the entire fleet can still exceed the limit; confirmed-save pruning and the janitor remain necessary. No cache purge, quota increase, ASan cap change or self-hosted enablement.
Verification matrix
healthy-warm, 1208/2070. PR run 38182446948, build 114597678279: correctcolddisposition, no restore, 0/2070 hits, zero errors, 2,818,897,842B under 3,355,443,200B cap; full build 218m29s passed. all three Windows test shards and coverage passedactionlint, docs consistency and all-files pre-commit passFound and fixed
77e42efe5fix(ci): shard XML upload referenced undocumentedrunner.workspace; usegithub.workspace. Found while inspecting artifacts;actionlintand hosted probe shard uploads verify the context.cad761119fix(ci): helper CLI fixtures inherited real Actions summary/output destinations; isolate them and reject invalid unsigned counters. CLI regression tests verify the behavior.a53f3f1a8ci(cache): native ANSI log prefixes contaminated object names; strip escapes and test real-formatted records. Also correct cache-prune's misleading storage-limit diagnostic.f7e97b2cedocs(cache): shared save-action diagnostics conflated the historical 10 GB/billing refusal with today's storage limit; clarify the error without changing save/prune logic.Filed, not fixed
None.
Review guide
Where I'd look hardest
Windows.ymlhosted identity/restore/guard/save sequence: prefix matches must count as warm, mismatched toolchains must stay cold, partial builds must never bank snapshots.check_sccache_health.py: native counters must be consistent and not double-count advanced language breakdowns; empty or malformed evidence must fail.What I verified, and how - Paired hosted controls, exact current-set artifact, restarted-daemon eviction reproduction, twelve helper tests, actionlint/docs/pre-commit, and independent standards/spec reviews. Independent current-head reviews and CodeRabbit are clean. All six PR workflows passed at
95500a304: 31 successful checks and two inapplicable hosted ASan+LSan shard jobs skipped because the self-hosted arm ran its full tests inline. Zero unresolved review threads; PR mergeable.Least confident about - How quickly future graph/source growth consumes the measured reserve. The artifact and visible guard make that measurable; remote default-branch/following-PR acceptance is explicitly still pending.