Stabilize Shared Compiler Caches and Move Image Scans Off the Main Build

[chet]

TLDR

The shared Rust compiler cache could become permanently read-only after one transient startup error, while pull requests saved large cache copies that later runs could not use and advisory image scans extended the main Core build. PR #4693 fixes those paths and moves the reusable Grype scan into a separate post-Core workflow.

Details

The sccache backend performs one write check when it starts. Before this change, any error -- including a temporary rate limit -- wrapped remote storage as read-only for the rest of the process. At the same time, restore-prefix matches caused pull requests to save separate 5 GB compiler-test archives and 1.3 GB mkosi package caches, consuming the quota intended for reusable main entries.

PR #4693 adds bounded recovery for the compiler-cache write check, restricts those large saves to main, and moves advisory Grype work into a workflow_run that scans the exact pushed image after Core completes. The reusable scan produces the required reports without extending the main Core critical path.

Expected impact

• Metric affected: sccache write errors and hits, large cache copies and quota use, Grype duration, and image-build cleanup.
• Expected green-run effect: No combined wall-time estimate is credited yet. The PR observed Grype taking roughly 24 minutes on affected main runs and moves that advisory work off the main build, while compiler savings still depend on one trusted write followed by a later real hit.
• What it really buys us: A transient cache-service response no longer makes the entire build silently read-only, pull requests stop banking large one-off copies, and advisory image scanning no longer extends the required Core build.
• Confidence: High for the corrected write and save decisions; low to medium for recurring time savings until hosted cache hits and post-change scan durations are recorded.
• PRs affected: Core main and pull request runs using the shared compiler/package caches, plus image jobs using the reusable Grype workflow.
• How we will measure it: Record sccache write errors, successful trusted writes, later hits, cache bytes by key, and one-scan Grype duration across post-merge runs.

What this involves

• Adopt merged PR ci: Fix sccache discarding writes, gate large cache saves to main, fix grype #4693 as the implementation for this issue.
• Keep retry behavior bounded and clear the stale write-error flag only after recovery.
• Keep the large compiler-test and mkosi saves restricted to trusted main runs while pull requests can restore the intended entries.
• Keep the reusable image-scan workflow on one underlying scan per image and preserve its required reports.
• Remove loaded images after the job completes so persistent runners do not accumulate them.
• Record actual cache hits and hosted scan times before updating the epic's performance baseline.

Related prior work: #4574 and #4605.

Part of #4572.