Repository navigation
Conversation
--------- Co-authored-by: Sylvestre Ledru <sledru@mozilla.com>
* Bump MSRV to 1.91.0 In preparation of the opendal bump to version 0.58.1. opendal 0.58.1 has an MSRV of 1.91.0, so sccache needs to match this in order to bump the opendal dependency. * Make cargo clippy/check pass This fixes Clippy findings after bumping the MSRV
…ackend (mozilla#2802) sccache's Azure backend only supported connection-string (shared-key) auth, which fails against storage accounts that disable shared-key access. Add a passwordless path that authenticates with Microsoft Entra ID (OAuth bearer tokens) instead. * When no connection string is set, the Azblob operator is built with no account key or SAS token, so OpenDAL/reqsign resolve credentials from the ambient environment (service principal, workload identity, or a managed identity via IMDS) and sign each request with a bearer token. * The blob endpoint comes from SCCACHE_AZURE_ENDPOINT (validated as an https URI; http is allowed only for a loopback host) or is synthesized from SCCACHE_AZURE_STORAGE_ACCOUNT as https://{account}.blob.core.windows.net. * The connection-string path is unchanged, and the two auth modes are mutually exclusive, enforced on both the env and file config surfaces. Blank values from a file config are treated as absent. * An explicit [cache.azure] block with no auth source errors clearly at startup, whereas an ambient container env var with no auth source only warns and disables the backend so it cannot take down an unrelated cache. Includes unit tests for endpoint resolution, auth-mode selection, mutual exclusivity, endpoint/account validation, and config parsing (env + TOML), plus documentation in docs/Azure.md and docs/Configuration.md. Signed-off-by: babrekel <babrekel@microsoft.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Matthew Marshall <matthew.marshall090396@gmail.com>
…ing backslashes (mozilla#2811) * nvcc (Windows): protect escaped quotes in dryrun lines before flattening backslashes nvcc --dryrun prints string-valued defines with escaped quotes (-D "FILE_NAME=\"x.dll\""). The Windows line mangling replaces every backslash with a forward slash BEFORE tokenization, turning \" into /" and collapsing the line's quote structure. shlex then groups everything after the first such define into one giant token (measured: a 493-char token containing ~30 -D pairs), so the host preprocess runs without those defines and #ifdef-guarded explicit instantiations silently vanish from the final object - a dropped-instantiation miscompile that surfaces as lld-link undefined symbols (reproduced with ONNX Runtime v1.28.0 bias_softmax_impl.cu: bare 3189 defined symbols vs wrapped 2628). Protect the \" escapes with a sentinel before the backslash flatten and restore them afterwards so shlex sees the original quoting. * Update nvcc.rs removing long comments on added escaped quotes for windows and its tests
…ozilla#2715) * Bump opendal to 0.58.1 - Required bump of reqwest to 0.13.0 - Rename and remove features deprecated in that release - Remove direct `reqsign` dependency that was not used in the code - Add new (optional) opendal-* dependencies that were split out of opendal * Fix fallout in code - `HttpClientLayer` was removed and now has to be built from the `OperationContext` - `LoggingLayer` was moved to its own crate - `allow_anonymous` was deprecated in favor of `skip_signature` - Use `sha256` instead of `sha1`, as that is quasi-deprecated. I'm not sure if this is actually necessary. I think it fixed some issue in the tests? In any case, I think this is a good change - Check if `native_certs` are available when building a reqwest client, and if not set the TLS certs to an empty vector. With reqwest 0.13, the client would fail otherwise. * Update Cargo.lock * Formatting (toml/rust) * cargo update aws-lc-rs dependency This removes the bindgen dep that made the cross-compilation stages error out
…illa#2816) nvcc's diagnostic-control options take an error-number list and accept the SEPARATED form (--diag-suppress 1394,1388). None of the family was in the argument table, so the separated value parsed as a bare token, was taken for a second input file, and the whole compile was rejected as CannotCache(multiple input files). Measured on OpenCV 5.0.0 with CUDA: every one of its 155 .cu compiles carries exactly this flag pair (via -Xcudafe --display_error_number --diag-suppress 1394,1388) and was forwarded uncached; with the entries added the same command is cached. Adds double- and single-dash forms, CanBeSeparated('='), PassThrough, plus a regression test for the separated shape.
* msvc: support arm64EC flag The commit is inspired by rapidsai@ac61177 * msvc: support fastfail flag This commit is inspired by rapidsai@33b6bce
The tests were AI-generated.
GCC always assembles by spawning `as`, and clang does too when its integrated assembler is turned off, so the object file depends on a program that nothing in the cache key accounted for: two invocations with different binutils could hand each other the wrong object. Compiler detection now asks the compiler which assembler it would run and identifies it the way the compiler itself is identified, by the contents of the binary plus the version it reports. It only reaches the key when a compilation really runs an external assembler, so clang keys don't move unless -fno-integrated-as is passed, and neither do MSVC's. Keys where no assembler could be identified don't move either, which keeps existing cache entries usable and is why CACHE_VERSION stays put. A hit on a preprocessor cache entry hands back the object cache key stored in it rather than computing one, so the assembler has to be part of that key as well. Full background in https://bugzilla.mozilla.org/show_bug.cgi?id=2070810.
Our current approach is to try and close the jobserver file descriptors inherited from make, using heuristics based on parsing the MAKEFLAGS searching for explicit FD numbers. However, in some circumstances, make will dup() those file descriptors internally, and we'll inherit and leak those duplicated FDs, which can lead to build script deadlocking. This happens to me frequently when working on Firefox on my macOS machine: when I run `./mach lint --outgoing` with substantial Rust work, once the clippy run ends, `make` idles waiting for job tokens to be handed back through those open file descriptors, until the daemonized server finally finishes through its idle timeout, closing their end of the pipes. So, instead of trying to pinpoint what must absolutely be closed, we take the opposite approach, and assume that most file descriptors don't need to be inherited when spawning a daemon.
Recognize standard #line directives when collecting included files for preprocessor cache manifests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MultiLevelStorage::check() started from ReadWrite and demoted the composite to ReadOnly as soon as any level reported ReadOnly. The server then wraps the whole storage in ReadOnlyStorage, so every cache write is refused before any write-error policy is consulted, including writes to writable levels and preprocessor cache updates. That inverts the documented semantics: MultiLevel.md promises that read-only levels are "automatically skipped during writes, regardless of write policy", and put() already implements exactly that per-level skipping. The practical fallout is that the documented read-only fallback topology (fast writable local disk in front of a read-only shared remote, e.g. SCCACHE_S3_RW_MODE=READ_ONLY or an azure SAS token without write permission) silently stops populating the local disk cache entirely. Aggregate in the other direction: the composite is writable when any level is writable, and read-only only when every level is read-only. The existing single-level read-only test still passes; new tests cover mixed chains in both orders and the all-read-only chain. Fixes mozilla#2773 Signed-off-by: Steven Noonan <steven@edera.dev>
The minio/minio and minio/mc images are no longer available. Switch the S3 test backend to Silo (pgsty/silo and pgsty/mc), a maintained MinIO fork, renaming the compose services and endpoint host accordingly. Silo still reads the MINIO_* environment variables and serves the /minio/health/live endpoint, so the credentials and the healthcheck are left unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SCCACHE_BASEDIRS strips the base directories from the preprocessed source, but the compiler arguments are hashed verbatim. That leaves out the one thing that most reliably ties a cache entry to a single checkout: a flag that names the tree it is building. -ffile-prefix-map=/home/user/project=. The flag exists precisely so the object file does not depend on where the tree lives - every path it records is rewritten to `.` - so two checkouts of the same commit produce identical objects. They just never agree on the hash, because the flag itself spells the path out, and so they never share an entry. Which is the case basedirs is for. Strip basedirs from each argument before hashing it, at the places where an argument is expected to spell a pathname, and only there: the whole argument, as in a source file or the path of a separated option; the value of an option written with an `=`, which covers either half of a prefix map; and the value glued to a short option such as -I. A basedir appearing anywhere else is left alone, so -DROOT="/home/user/project", which the compiler bakes into the output verbatim, still counts towards the hash. Basedirs carry a trailing slash so that they only match whole path components; in an argument a component can also end at the `=` of a prefix map, or at the end of the argument, so accept those two as well. A sibling /home/user/project-docs is still not a match. The longest basedir wins. Bump CACHE_VERSION and the preprocessor cache's FORMAT_VERSION, since the arguments now reach the hash as bytes rather than through OsString's Hash impl. An integration test is added to tests/integration.
mozilla#1159 only guards the staticlib case; a cdylib sibling is .so/.dylib/.dll/.wasm, so the rlib is trimmed and the job fails on the build server.
`AsyncCommand::spawn` calls `jobserver::Client::configure` on every child it
spawns. On Unix that registers a `pre_exec` closure to clear `CLOEXEC` on the
jobserver's two file descriptors -- and the presence of *any* `pre_exec` makes
`std` abandon its `posix_spawn` fast path and fall back to `fork` + `exec`.
That is a bad trade for a server process. `fork` duplicates the parent's page
tables, and sccache's whole job is to be a long-lived process holding a large
cache; the child then throws the copy away microseconds later in `exec`. The
cost scales with how much memory the server has touched, so it grows over the
life of a build.
An empty closure is enough to trigger it:
let mut c = Command::new("/bin/true");
if pre_exec { unsafe { c.pre_exec(|| Ok(())); } }
c.spawn()
without: clone3({flags=CLONE_VM|CLONE_VFORK|CLONE_CLEAR_SIGHAND, ...})
with: clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|...|SIGCHLD)
`CLONE_VM|CLONE_VFORK` shares the address space and copies nothing. The second
form is a real fork, and it is what shows up in a profile as `dup_mmap` ->
`copy_page_range` -> `copy_pte_range`.
Almost nothing sccache spawns can use a jobserver. It reaches a child through
`CARGO_MAKEFLAGS`, and only `rustc` reads it: preprocessors, version probes and
C/C++ compilers all ignore it. So stop sharing by default and let the callers
that spawn `rustc` ask, via `RunCommand::share_jobserver`.
For the local compile, which is one code path shared by every frontend, the
opt-in is a field on `SingleCompileCommand` rather than a builder call. That
makes the compiler ask every frontend the question, so a new one cannot get it
wrong by omission -- and getting it wrong in this direction is what matters,
since a `rustc` without a jobserver spawns as many codegen threads as there
are CPUs, per concurrent `rustc`, which is the oversubscription the jobserver
exists to prevent.
Measured on a 2456-file LLVM build (X86 only, Release, clang, `-j16`, 16
cores). Two scenarios, interleaved rounds:
All compiles are cache hits, with `SCCACHE_DIRECT=false` so the preprocessor
still runs -- this isolates the spawn cost, since it is nearly all the server
does (4 rounds):
| arm | wall | server CPU | of which system |
|--------|------------------|------------------|-----------------|
| before | 40.7 s (+-0.7) | 28.2 s (+-0.4) | 22.8 s |
| after | 33.5 s (+-0.2) | 10.6 s (+-0.1) | 5.8 s |
All compiles are cache misses, so each one both preprocesses and compiles
(2 rounds):
| arm | wall | server CPU | of which system |
|--------|------------------|------------------|-----------------|
| before | 336.5 s | 96.3 s | 45.8 s |
| after | 333.1 s | 70.1 s | 21.2 s |
The saving is almost entirely system time, which is what a page-table copy
costs. Wall clock barely moves on the miss build because it is dominated by the
compiler itself; the win there is 26 s of a core given back, not a faster
build.
`strace` on a 1300-compile miss build confirms the mechanism, and shows why
both spawn sites had to be covered:
| build | fork | posix_spawn |
|------------------|------|-------------|
| before | 2617 | 0 |
| preprocess only | 1319 | 1300 |
| this change | 0 | 2619 |
Under `perf record -p <server>` on the cache-hit build, the address-space
symbols (`copy_pte_range`, `copy_present_ptes`, `zap_pte_range`,
`smp_call_function_many_cond` and friends) go from 4.20% of the server's
samples to 0.05%.
One behaviour change worth naming: the client-side fallback in `commands.rs`,
which runs the compiler itself when the server declines the job, no longer
shares its jobserver either. That jobserver is private to a single short-lived
client process with exactly one child, so it never limited anything across
compiles; the server's jobserver is the one that does real work.
SCCACHE_BASEDIRS now normalizes cwd, CARGO_MANIFEST_DIR, CARGO_WORKSPACE_DIR, CARGO_TARGET_TMPDIR, CARGO_MANIFEST_PATH, CARGO_BIN_EXE_*, dep-info env var values, and the concatenated argument string in the Rust compiler's hash key computation. This enables cache hits when the same crate is compiled from different absolute paths on different machines (e.g., CI runners with different checkout roots). strip_basedir_prefix now also matches when the value equals the basedir minus its trailing '/', so `cwd == basedir` strips to the empty string rather than passing through. Without this, two machines with different checkout paths produced different hashes even with matching basedirs -- the feature's central claim.
…w (sccache#2595) PR mozilla#2678 wired basedirs into the Rust hash key, but only from the daemon's startup config (storage.basedirs()). A long-lived server cannot know about git worktrees created after it started, so cross-checkout Rust hits required restarting the server whenever the worktree set changed — the exact pain sccache#2595 describes. Read SCCACHE_BASEDIRS from THIS compile's environment (each worktree sets its own root via cargo `[env]`) when present, falling back to the server's config basedirs. Each build now strips its own root per-request: no server-startup staleness, no restarts, parallel-safe across worktrees. Works in both the default and client-side compile paths (the daemon hashes either way and reads the per-request env). - config.rs: extract split_basedirs + normalize_basedirs (shared, so config parsing and the hasher produce byte-identical basedirs). - compiler/rust.rs: generate_hash_key prefers per-compile env basedirs. - tests/sccache_cargo.rs: add test_rust_cargo_basedirs_per_compile_env_cross_dir_cache_hit — the sccache server is started with NO basedirs; the per-compile env alone drives the cross-directory cache hit (fails before this change).
…dentical (sccache#2595) Per-compile basedirs (previous commit) align the cache KEY across checkouts, but the shared dependency graph still misses: rustc bakes absolute paths into the rlib — notably the build-script OUT_DIR that deps like serde `include!` — so the same dependency compiled in two git worktrees emits byte-different rlibs. sccache folds rlib content into the key (hash of the externs), so that divergence cascades misses up the whole graph. Inject `--remap-path-prefix=<basedir>/=` (one per SCCACHE_BASEDIRS entry from the compile's own env) into the local rustc command in generate_compile_commands, so the embedded paths become checkout-relative and the rlib is byte-identical across checkouts. This is the OUTPUT half of basedirs support, complementing the KEY half: the key is computed separately from the original args (basedirs already stripped), so the injection never re-keys; cargo computes `-C metadata` without it, so symbol names stay checkout-stable. Mirrors the existing dist-path `--remap-path-prefix` injection a few lines below. Effect: a dependency compiled in one worktree is reused by every other worktree (and the main checkout) instead of recompiling — no RUSTFLAGS/config plumbing, automatic wherever sccache is the rustc wrapper.
PR Summary by QodoRebase sccache fork onto upstream v0.18.0
AI Description
Diagram
High-Level Assessment
Files changed (61)
|
|
|
@greptile review |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit e9e8951 |
|
@greptile review |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit e9e8951 |
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit e9e8951 |
This fork of sccache has been
v0.17.0plus three commits. Upstream releasedv0.18.0on 2026-09-14, and it carries mozilla#2840, which strips basedirs from C/C++ compiler arguments. That is the C/C++ half of the problem this fork solves for Rust: a worktree-heavy workflow where the same source is compiled from many absolute checkout roots, and every root used to key its own cache entries. This PR moves the fork ontov0.18.0, so the fork gets that half from upstream and stays close to it.What changed
The three fork commits are re-applied onto
v0.18.0(01c35e6) with their messages intact:bb1a338strip basedirs from the Rust hash key (mozilla#2678)ce840e1src/compiler/compiler.rs:2642) still calledMockStorage::newwith two arguments; this commit adds thebasedirsargument there.ad72a7bper-compileSCCACHE_BASEDIRS05de0efee4f1e6--remap-path-prefixinjection1f53695git range-diff v0.17.0..ee4f1e6 v0.18.0..1f53695shows exactly those two hunks.On top sits
e9e8951, agit merge -s oursof the oldanneal-main. Its tree is identical to1f53695. It exists so that this branch fast-forwardsanneal-main, and so thatee4f1e6(the old tip, taggedv0.17.0-anneal.1) stays reachable fromanneal-main. Without it the PR shows a conflict (only the rustfmt reflow above), and a squash or rebase merge would orphan the old tip.Merge with "Create a merge commit" or a fast-forward, not squash or rebase. Consumers of this fork pin commits by SHA, and the tags
v0.18.0-anneal.1(e9e8951) andv0.17.0-anneal.1(ee4f1e6) must stay reachable fromanneal-main.The GitHub diff for this PR is large because it includes all 30 upstream commits between v0.17.0 and v0.18.0. The fork's own delta is
git diff v0.18.0 1f53695.mozilla#2840 and the fork's Rust hasher do not overlap
Upstream mozilla#2840 adds
util::strip_basedirs_from_argand calls it fromc.rshash_arguments, which feeds the C/C++ key and the preprocessor-cache key. The fork's Rust path has its ownstrip_basedirs_in_arginrust.rs, called once per argument inRustHasher::generate_hash_key. Neither path calls the other's helper, so no argument is stripped twice.They are deliberately not identical. The C/C++ matcher strips only where a pathname is expected (the whole argument, after
=, or glued to a short option), and it leaves-Wl,-rpath,/pathalone because the linker bakes that path into the output. The Rust matcher also strips after,, because-Clink-arg=-Wl,-rpath,/absshows up in Rust keys, and the fork's--remap-path-prefixinjection makes those paths checkout-relative in the output anyway. Unifying the two is a candidate for an upstream PR, not for this rebase.One difference in where basedirs come from matters. The fork's Rust hasher reads
SCCACHE_BASEDIRSfrom each compile's own environment (for example, cargo[env]set per checkout), and falls back to the server config. The C/C++ path, including client-side mode, still uses the server's configured basedirs (IpcStoragetakes them from the daemon handshake). So C/C++ objects are shared only across checkouts listed in the server's config when the server starts. A checkout created after the server started gets no C/C++ sharing until the next restart.What v0.18.0 brings
4a78001): C/C++ arguments such as-I<checkout>/target/.../outand-ffile-prefix-map=<checkout>=.no longer tie an entry to one checkout. It bumps the C/C++CACHE_VERSIONfrom 12 to 13 and the preprocessor cacheFORMAT_VERSIONfrom 0 to 1, so existing C/C++ entries go cold once.4ffa89b: gcc and clang make the assembler part of the key (more correct C/C++ keys).be297ee: the daemon inherits only an allow-list of file descriptors, and01c35e6stops handing the jobserver to children that cannot use it. Both reduce odd interactions between a long-lived server and cargo's jobserver.17a6736,6089d04: multilevel cache fixes.46e96ab: MSRV 1.91.SCCACHE_BASEDIRS+ Preporcessor cache mode hands out another checkout's object file mozilla/sccache#2863 (open) reports that basedirs together with preprocessor cache mode can hand one checkout another checkout's C/C++ object when their headers differ. The proposed patch for it, fix preprocessor cache mode vs basedirs mozilla/sccache#2868, is still open. The local disk cache enables preprocessor cache mode by default. Anyone using this release with basedirs should either carry fix preprocessor cache mode vs basedirs mozilla/sccache#2868 or turn preprocessor cache mode off until it lands.rust.rsCACHE_VERSIONis still6, and the rebased fork code is byte-identical apart from formatting. So Rust entries written by 0.17.0-anneal.1 stay valid under 0.18.0-anneal.1 (measured below).Is the fork's work upstreamable?
Yes, in pieces, and upstream is converging on the first two. The third is the fork's alone, and it is the piece that makes the worktree workflow actually hit.
ce840e1Rust key honors basedirs (from mozilla#2678, by mmastrac)05de0efper-compileSCCACHE_BASEDIRSgenerate_hash_keyinstead of a largeStoragewrapper.generate_hash_keyand falls back tostorage.basedirs(), with no wrapper. That is a small, concrete contribution to offer on mozilla#2814, with the tests fromtests/sccache_cargo.rs.1f53695inject--remap-path-prefix=<basedir>/=into the Rust compileserde_core/serdeinclude!theirOUT_DIR, their rlibs embed the checkout path, and every crate behind them misses in each new worktree.What upstream will likely push back on, to fix before proposing: the Rust and C/C++ argument matchers should share one helper (with the comma case as an explicit Rust-only extension); the per-compile env override should also apply to the C/C++ path in client-side mode (today it does not, see above); and the injection should sit behind a config switch.
Known follow-ups for worktree churn
--remap-path-prefix, and on macOS each dylib embeds its own absolute output path as its install name. The sameserde_derivebuilt in two checkouts is byte-different for that reason alone (same size, one embedded path). The fork hashes every--externfile's contents, so every dependent of a proc-macro re-keys per checkout. Built with a fixed-Wl,-install_name,@rpath/libserde_derive.dylib, the two dylibs are byte-identical. Giving proc-macro links a checkout-independent install name, or hashing proc-macro externs without it, should turn most remaining cross-checkout misses into hits.generate_hash_key-level per-compile basedirs on server: scope SCCACHE_BASEDIRS to compiler requests mozilla/sccache#2814, and drop fork commits as rust: normalize cache keys with SCCACHE_BASEDIRS mozilla/sccache#2813 and server: scope SCCACHE_BASEDIRS to compiler requests mozilla/sccache#2814 land..d) verbatim, so it names the absolute paths of the checkout that wrote the entry. Cargo then judges the consumer's freshness by the donor's files: when the donor rebuilds or itstarget/is removed, no-op builds in the consumer re-run many units (all cache hits, but not free). This is the same on 0.17 and 0.18.Review findings on upstream code
Because this PR carries the whole v0.17.0..v0.18.0 range, the review bots also reviewed upstream code that the fork takes verbatim. Their findings there are real but not the fork's: unquoted
-Dpaths lose their checkout root in C/C++ keys (mozilla#2840,src/util.rs), the new assembler digest can go stale (4ffa89b,src/compiler/compiler.rs),cicc.rsinput selection, and mutable integration-test images. None of those lines appear ingit diff v0.18.0 1f53695, and each thread is answered with the reason. They are candidates for upstream reports, not fork changes in a rebase.Verification
Cross-checkout cache behaviour. Measured on a large Rust workspace with C and assembly build scripts, in isolated runs: each run used its own
SCCACHE_SERVER_PORT,SCCACHE_DIRand config, and several copies of the same commit under different absolute roots.-ffile-prefix-map,-I<checkout>/...).Fork test suite on
1f53695(macOS arm64, rustc 1.97.1, a non-defaultSCCACHE_SERVER_PORT, ownSCCACHE_DIR, emptySCCACHE_CONF):cargo test --locked --no-fail-fast: 561 passed, 2 failed, 15 ignored. The two failures aretests/system.rstest_sccache_command::{with,without}_preprocessor_cache, stat-count assertions in a C/C++ test the fork does not touch. They are intermittent: a rerun of--test systemfailed a different assertion once and passed 8/8 the next time, on a heavily loaded machine.test_s3_invalid_argstimed out waiting for server startup in one run (it talks to S3 with invalid credentials); it passed in the--no-fail-fastrun.sccache_cargo.rsbasedirs cases, includingtest_rust_cargo_basedirs_per_compile_env_cross_dir_cache_hit: pass.cargo clippy --locked --all-targets -- -D warnings -A unknown-lints -A clippy::type_complexity -A clippy::new-without-default(the upstream CI's flags): clean.cargo fmt --check: clean.The system tests are not isolatable by environment.
tests/harness/mod.rsprune_command()strips everySCCACHE_*variable, includingSCCACHE_SERVER_PORT, from the commandstests/system.rsruns. So running the suite on a machine with a live sccache server stops that server on the default port. #3 makes the tests refuse to run next to a live server; run them in a sandbox.This fork runs no CI on pull requests yet.
Stack: #1 (this) → #3 (open, based on this branch). Merge this first; then retarget #3 to
anneal-main.