fix(ci): size Windows sccache and reject near-zero warm reuse

.github/actions/save-cache-pruned/action.yml

@@ -10,13 +10,14 @@ description: >
   VERIFIED present, so each lineage holds one entry per ref without waiting for the
   `cache-prune` sweep -- which two crons a day could not keep up with (#1073).
 
-  The repo's Actions cache store is capped at 10 GB, and past the cap GitHub does not
-  evict -- it turns the WHOLE STORE READ-ONLY ("Cache reservation failed: You have
-  reached your configured budget, your cache is now read only"). The steady set of live
+  In #1073, the configured Actions cache budget made the WHOLE STORE READ-ONLY
+  ("Cache reservation failed: You have reached your configured budget, your cache is
+  now read only"). The repository cap was then 10 GB; it is now 25 GB, as documented in
+  docs/agent-rules/actions-cache-budget.md. The steady set of live
   entries measured ~5 GB before this job's own snapshot is counted, and the three Linux
-  sanitizer entries are ~1.25 GB apiece -- so the moment two or three heavy jobs each hold
-  an old AND a new snapshot at once, the cap is breached and every save in the repo is
-  refused until the next sweep. A twice-daily sweep cannot fix that: the breach happens
+  sanitizer entries were ~1.25 GB apiece. When two or three heavy jobs each held an old
+  AND a new snapshot at once, the configured budget refused every save in the repository
+  until the next sweep. A twice-daily sweep could not prevent that failure: it happened
   between two saves minutes apart, not between two days.
 
   It is not a hypothetical. It is the measured state that made
@@ -360,6 +361,6 @@ runs:
           exit 0
         fi
         gh api "repos/${GH_REPO}/actions/cache/usage" \
-          --jq '"cache store: \(.active_caches_count) entries, \(.active_caches_size_in_bytes/1048576|floor) MiB against a 10 GB cap"' || true
+          --jq '"cache store (usage API may lag): \(.active_caches_count) entries, \(.active_caches_size_in_bytes/1048576|floor) MiB"' || true
         echo "::error::cache entry '${KEY}' was NOT written even after pruning this ref. The next run WILL start cold."
         echo "::error::This is the repo Actions cache budget (#1073): look for 'Cache reservation failed ... your cache is now read only' in the save steps above, then see docs/agent-rules/actions-cache-budget.md."

.github/workflows/cache-prune.yml

@@ -387,7 +387,7 @@ jobs:
           used_mib=$((kept_bytes / 1048576))
           if [ "${used_mib}" -gt "${headroom_floor_mib}" ]; then
             echo "::error::Actions cache store is ${used_mib} MiB after pruning, over the ${headroom_floor_mib} MiB working ceiling."
-            echo "::error::Past the cap (25 GB, ~23841 MiB) GitHub does not evict -- it makes the whole store READ-ONLY and refuses every save in the repo, silently (actions/cache/save warns and exits 0). Nothing left here is superseded, so this needs a human: docs/agent-rules/actions-cache-budget.md."
+            echo "::error::The repository storage limit (25 GB, ~23841 MiB) can evict useful snapshots; a configured billing budget can refuse new writes earlier. Nothing left here is superseded; review docs/agent-rules/actions-cache-budget.md."
             exit 1
           fi
           echo "headroom: $((headroom_floor_mib - used_mib)) MiB below the working ceiling."

docs/agent-rules/README.md

@@ -110,6 +110,7 @@ Each entry is one sentence stating the rule. The story that taught it is inside
   one that works.
 - [actions-cache-budget.md](actions-cache-budget.md): the Actions cache store is a fixed budget every key must fit in; past the cap it goes read-only and refuses every save in the repo, silently.
 - [sccache-cap-vs-object-set.md](sccache-cap-vs-object-set.md): set a compiler cache's size cap above the object set it has to hold, measured with `du`, then let the store's worst case overrule the number; and put whatever the key is hashed over into the key.
+- [windows-sccache-warm-reuse.md](windows-sccache-warm-reuse.md): require native compiler statistics after a complete warm restore, and compare source, toolchain and resident objects before repeating an earlier cache fix.
 - [self-hosted-ccache-slot-multiplier.md](self-hosted-ccache-slot-multiplier.md): size a self-hosted compiler cache by configurations x the runner slots that can build them, make the slots share rather than paying for both, and let the filesystem overrule the number.
 - [cache-entry-version-is-the-path-string.md](cache-entry-version-is-the-path-string.md): a cache's restore and its save must be handed the same path string, character for character; two spellings of one directory are two caches.
 - [compiler-cache-uncacheable-compiles.md](compiler-cache-uncacheable-compiles.md): print the cache statistics after every CI build and read the uncacheable line; a PCH without `CCACHE_SLOPPINESS` and a per-commit macro each made every engine object a miss.
@@ -575,6 +576,7 @@ The check passes for a correct implementation and for a broken one.
 | [ci-cache-that-looks-alive.md](ci-cache-that-looks-alive.md) | A cache has no wrong-looking failure state: it fails by being slow. Every save in the repo had been refused for six days and every run stayed green. |
 | [actions-cache-budget.md](actions-cache-budget.md) | The Windows sccache entry had not existed for weeks. Restore said "Cache not found", save said "your cache is now read only" and exited 0, and the only visible symptom was a build that took four hours. |
 | [sccache-cap-vs-object-set.md](sccache-cap-vs-object-set.md) | A healthy restore of a current entry that still hits 32 %, and `--show-stats` rounds away the number that would explain it — `Cache size 1 GiB / Max cache size 1 GiB` printed both for a cache 20 % over its cap and for one with 259 MiB spare. Two different causes, one symptom: too small a cap, or a key that does not cover the SDK that changed. |
+| [windows-sccache-warm-reuse.md](windows-sccache-warm-reuse.md) | The nightly restored the preceding master's same-commit, same-toolchain snapshot and still missed on all 2060 requests while ASan hit 99.90%. |
 | [self-hosted-ccache-slot-multiplier.md](self-hosted-ccache-slot-multiplier.md) | A cumulative `76.40 %` hit rate over a cache pinned at `30.0 / 30.0` with 5,884 cleanups, while every job recompiled half the tree. The rate is a lifetime counter shared by every job on the box, so it averages the warm era with the thrash; and half the misses were structural, two runner slots each caching the same object under its own `_work` path. |
 | [cache-entry-version-is-the-path-string.md](cache-entry-version-is-the-path-string.md) | The vcpkg key matched exactly and the entry was still invisible: restore and save spelled the same directory two ways, so every entry ever written had `last_accessed_at == created_at`. |
 | [compiler-cache-uncacheable-compiles.md](compiler-cache-uncacheable-compiles.md) | The build step takes compile time on a shader-only change while the cache directory is warm. A PCH-using compile is uncacheable without `CCACHE_SLOPPINESS`; a per-commit macro on every TU misses every run. |

docs/agent-rules/actions-cache-budget.md

@@ -438,6 +438,51 @@ Do not assume one cause explains every cold cache in a repo; check each prefix's
 `created_at` (is anything written?) and `created_at == last_accessed_at` (is anything
 read?) separately.
 
+## 2026-10-11 MSVC Release capacity
+
+Size Release from unique objects requested by a complete build, then account for
+concurrent saves and key rotations before choosing its cap. #1584's no-save hosted
+probe [38177455638](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38177455638)
+measured **2070 current objects / 2,818,896,940 bytes (2688.31 MiB)**, all resident at
+completion. The previous 2G cap caused LRU eviction before reuse; the older 4.55 GB
+union included obsolete source keys and overstated current needs. Release now uses
+**3200M**, leaving **511.69 MiB (19.03%)** for growth. See the
+[matched controls](windows-sccache-warm-reuse.md) for the root-cause evidence.
+
+The live REST storage-limit endpoint confirmed **25 GB**. The cache listing contained
+12 entries / **10,596,655,761 bytes**: the legacy Release snapshot cost **2,146,601,232
+bytes**, and everything else cost **8,450,054,529 bytes**. Budget the new snapshot at
+its full cap, not an assumed archive-compression discount:
+
+| Selected overlap scenario | Conservative bytes | Limit remaining |
+|---|---:|---:|
+| Other entries + one full new Release snapshot | 11,805,497,729 | 13,194,502,271 |
+| Other entries + three full new snapshots + legacy 2G snapshot | 20,662,985,361 | 4,337,014,639 |
+| Above + Windows vcpkg, Vulkan SDK, Windows ASan and Linux vcpkg rotations | 24,326,580,474 | 673,419,526 |
+
+Three snapshots allow the old entry plus simultaneous push/nightly saves before
+confirmed-save pruning. The additional legacy identity can survive until the janitor
+groups it with the fingerprinted lineage. Rotation allowances use the current entry
+sizes: Windows vcpkg **1,482,075,738**, Windows Vulkan **268,769,601**, Windows ASan
+**1,568,935,714**, Linux vcpkg **343,814,060** bytes. This stress case retains **642.22
+MiB** under the limit. This is not a bound for simultaneous replacement of the whole
+fleet: also overlapping all three Linux sanitizer snapshots adds **3,863,327,875
+bytes**, making this scenario **28,189,908,349 bytes**, over the limit. Do not schedule
+extra branch producers or deliberately rotate all lineages together on the strength
+of this table. The diagnostic 8G probe cannot save, and PRs still cannot save compiler
+snapshots. Confirmed-save pruning and the janitor remain necessary; remeasure peak
+occupancy if producer concurrency or save cadence changes.
+
+Keep the **20000 MiB** working ceiling: it leaves **3841.86 MiB** below 25 GB, exceeding
+the new largest single capped entry (3200 MiB). No quota increase, cache purge or ASan
+cap change is part of this fix. These are dated measurements, not a permanent budget: remeasure
+the complete object set and listing after graph, image or dependency changes. The
+current MSVC cache has no separate preprocessor entries in the native control; if
+compiler/direct-mode behavior changes, budget the full archived directory as well
+as the requested object set. Actual
+matched master/following-PR recovery remains required; capacity arithmetic is not that
+acceptance evidence.
+
 ## See also
 
 - [sccache-cap-vs-object-set.md](sccache-cap-vs-object-set.md) — how to pick a

docs/agent-rules/sccache-cap-vs-object-set.md

@@ -72,8 +72,12 @@ key is hashed over.** The Windows ASan key now carries `-vk<version>` in both `k
 predecessor rather than stranding ~1230 MiB for thirty days.
 
 **Two failure modes, one symptom.** A cache that misses on everything looks the same
-whether the cap is too small or the key is too coarse. `du -sm` against the cap separates
-them: at or above the cap it is eviction, comfortably below it is invalidation.
+whether the cap is too small or the key is too coarse. Exact directory bytes show whether
+the cap has headroom; a full directory alone does not prove eviction. A rolling compiler
+can also fill it with incompatible objects. Compare matched source/toolchain runs and
+computed keys against the restored resident set, then change only the capacity in a
+controlled experiment. [windows-sccache-warm-reuse.md](windows-sccache-warm-reuse.md)
+records that distinction for the later MSVC Release regression.
 
 ## See also
 

docs/agent-rules/windows-sccache-warm-reuse.md

@@ -0,0 +1,127 @@
+# Measure compiler reuse after a warm restore
+
+Require a complete build and parse native sccache JSON after a hosted Windows restore;
+a successful cache download does not prove that any compiler request reused an object.
+Issue [#1584](https://github.com/drsnuggles8/OloEngineBase/issues/1584) follows several
+earlier fixes with the same zero-hit symptom and different causes.
+
+## What the earlier fixes established
+
+| Work | Finding and change | Limit of that evidence |
+|---|---|---|
+| #1009 | Split restore/save, preserve default-branch producers and stop cancelling them. A six-day-old snapshot was not a warm control. | Source drift and producer cadence remain independent of cache capacity. |
+| #1073 / #1075 | The Windows compiler cache entry did not exist: the store refused saves because its budget was exhausted. Added confirmed saves and pruning; removed PR-private large saves. | A snapshot that exists and restores can still contain no usable objects. |
+| #1082 | Windows ASan cache support and server idle-timeout handling were repaired. | clang-cl ASan is a control, not proof that MSVC Release reuses objects. |
+| #1118 | Linux sanitizer object sets exceeded the 900M cap; measured 1300M caps. Stamped Windows ASan caches with the Vulkan SDK. | These measurements did not size the MSVC Release object set. |
+| #1084 / #1170 | Warmed master per merge and stamped Release caches with the Vulkan SDK. Measured warm Release hit rates of 94.8-98.5%. | `windows-2025` is a runner label, not an immutable compiler/image version. The Release cache was already at its 2G cap. |
+
+Do not retry a previous fix based only on the symptom. Check restore lineage, source and
+toolchain identity, exact resident bytes, and actual compiler reuse.
+
+## Matched controls collected on 2026-10-11
+
+Release producer [38086179377](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38086179377)
+and following nightly [38104242016](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38104242016)
+both built commit `402f1ce55`, image `20260925.250.1`, MSVC `19.51.36260.0`.
+The nightly restored the producer's snapshot. Both returned **0 hits / 2060 misses**, with
+zero read/write errors and zero uncacheable calls, at the 2G cap.
+
+The same-commit Windows ASan nightly
+[38103213360](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38103213360)
+used the same image, clang-cl `23.1.0`, and its previous producer's cache. It returned
+**2015 hits / 2 misses (99.90%)**. The cache service and producer cadence were working.
+
+The September transition also occurred within one image/compiler generation:
+Release run [34797475126](https://github.com/drsnuggles8/OloEngineBase/actions/runs/34797475126)
+returned 1438/1772 hits; producer
+[34811731975](https://github.com/drsnuggles8/OloEngineBase/actions/runs/34811731975)
+returned 448/1777; its successor
+[34812956055](https://github.com/drsnuggles8/OloEngineBase/actions/runs/34812956055)
+returned 1/1783. All used MSVC `19.51.36256.0`. These different source revisions locate
+the degradation; they do not isolate its cause.
+
+## Hosted capacity contrast identifies eviction
+
+Fresh master run [38166502504](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38166502504)
+restored the same `38104242016` snapshot and returned **0/2070 hits** at 2G.
+The successful build took 135m33s and confirmed its cache save. The no-save 8G probe
+[38168947158](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38168947158)
+compiled the same C++ tree (`ada41984c`; only the localized BuildInfo commit input
+differs) from that same restored snapshot. Its complete capture recorded **586 hits /
+1484 misses**: all **586 keys resident at restore** were reused. There were no cache
+read/write errors or failed/non-cacheable compilations.
+
+The original snapshot held 1235 objects / 2,146,612,421 bytes. The probe finished with
+2719 objects / 4,548,515,240 bytes. Its 1484 new objects added **2,401,902,819 bytes**,
+already exceeding the 2,147,483,648-byte production cap without counting the 586
+reused objects. At 2G, early misses evict matching objects before their turn; the
+enlarged arm preserves them. This establishes capacity/LRU eviction, rather than a
+globally changing hash input, as the cause of the zero-hit regression.
+
+The final snapshot also retains obsolete source keys. It is an upper bound on the
+current object set, not its exact size. The refined probe measures unique requested
+keys separately and requires every requested key to remain resident before claiming
+a complete object-set measurement.
+
+Following no-save probe
+[38177455638](https://github.com/drsnuggles8/OloEngineBase/actions/runs/38177455638)
+restored fresh master `38166502504` at the same image/compiler. It reused all **1208
+keys resident at restore**, with 862 misses and no errors/non-cacheable calls. Its
+complete successful build took **61m57s**. All **2070 unique requested objects** remained
+resident, totaling **2,818,896,940 bytes (2688.31 MiB)**, with no obsolete objects left
+in this snapshot. The native JSON and complete probe are in `windows-cache-health`
+and `windows-cache-probe` artifacts. This is sizing evidence; 58.36% reuse from the
+truncated 2G snapshot is not the recovery criterion.
+
+Production uses **3200M (3,355,443,200 bytes)**, reserving **511.69 MiB / 19.03%** above
+the measured current set. Keep measuring as the compiled graph grows. The shared-store
+arithmetic, including overlapping snapshots and dependency rotations, is in
+[actions-cache-budget.md](actions-cache-budget.md#2026-10-11-msvc-release-capacity).
+
+## Controlled eviction reproduction
+
+A local control used sccache 0.15.0 and five deterministic MSVC translation units,
+each exporting 65,536 random unsigned integers, with `/O2 /Z7 /MD`. Inputs, paths and
+compile order stayed identical; the daemon restarted between every pass. Builds used
+the build lock and a separate cache/server port. Native JSON recorded:
+
+| Pass | Cap | Hits / requests | Resident bytes |
+|---|---|---|---|
+| Cold | 1MiB | 0/5 | 791,015 |
+| Warm, same order | 1MiB | 0/5 | 791,015 |
+| First enlarged pass | 8MiB | 3/5 | 1,318,365 |
+| Warm, same order | 8MiB | 5/5 | 1,318,365 |
+
+The smaller cache retained three objects, but earlier misses evicted them before their
+turn. Enlarging it preserved all three restored objects and then the entire set. This
+isolates the zero-hit eviction mechanism; it does not measure the engine's object set
+or establish the hosted regression's cause.
+
+## Probe and health contract
+
+`Windows.yml`'s opt-in `cache_probe` dispatch uses an 8G temporary cap and **does not save
+that compiler cache**. It explicitly restores the original pre-fingerprint lineage for
+diagnosis; normal production restores have no such fallback. Independent probes do not
+share a concurrency lane with each other's test shards. It snapshots object keys before configure, captures generated keys,
+and reports which were resident at restore. Raw daemon logs stay on the disposable runner;
+only derived records and native statistics are uploaded. A missing log or incomplete capture
+cannot be reported as a complete probe. `windows_cache_probe.py` scans object entries, not
+the separate preprocessor cache. Object basenames can collide, so lookup outcomes are
+aggregated rather than attributed to individual basenames.
+
+The production guard reads `cache-matched-key`, which is populated by a prefix restore even
+though `cache-hit` is false. It records counts, hit rate, restore key, build outcome and exact
+cache/cap bytes. A successful build with at least 500 cacheable requests is required. Missing,
+malformed or inconsistent statistics fail; skipped/failed builds and smaller samples fail.
+A complete cold run passes without requiring hits. A warm result below 5% fails with an
+Actions annotation and a machine-readable artifact. The threshold detects this regression;
+passing it alone is not the 94.8-98.5% recovery criterion.
+
+Hosted cache prefixes include a fingerprint of compiler bytes/path, MSVC and Windows SDK
+versions, the actual image revision and sccache version. A toolchain roll starts a cold lineage;
+there is no fallback to another identity. The fingerprint follows the Vulkan stamp so the
+existing janitor strips run/attempt, then fingerprint, then SDK, grouping superseded lineages.
+Cache saves require a successful build and remain disabled on PRs and oversized probes.
+
+Recovery still requires named hosted master producer and following PR consumer runs at a
+matched toolchain. Branch dispatches and local helper tests cannot replace that evidence.