Skip to content

Build the released binaries with profile-guided optimization - #2943

Open
sylvestre wants to merge 3 commits into
rust-direct-modefrom
pgo-build
Open

sylvestre wants to merge 3 commits into
rust-direct-modefrom
pgo-build

Conversation

@sylvestre

@sylvestre sylvestre commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Add a script to build sccache with profile-guided optimization

scripts/build-pgo.sh builds an instrumented sccache with the same LTO and
codegen-unit settings as the release build, trains it on the integration
tests (system, sccache_cargo, sccache_rustc, cache_hit_rate, sccache_args)
and a generated C/C++ corpus of cache misses and hits, merges the profile
and builds the optimized binary. --train-only stops after the profile, for
CI to pass -Cprofile-use itself; --bench also builds a plain binary and
compares the two with scripts/bench-pgo.sh, which uses hyperfine.

The test executables are built with cargo test --no-run, but the sccache
they drive is then rebuilt with cargo build: under cargo test,
dev-dependencies unify extra features into shared crates, which renames
their symbols and sccache's, and the profile would not match the release
build. The test harnesses are instrumented too, so only the profile written
by sccache itself is kept.

On Windows, std::process::exit skips the atexit handler the profiling
runtime writes the counters from, so the instrumented build
(--cfg pgo_training) writes them before exiting.

On Linux x86_64, PGO cuts the server's instructions per direct-mode cache
hit by 3-5%: most of its time is in blake3, libc and the kernel, which the
profile does not change. It is not enabled in CI.


ci: build the released sccache binaries with PGO

On tags and on main, the x86_64 Linux (musl), macOS (arm64 and x86_64) and
x86_64 Windows sccache builds train a profile with scripts/build-pgo.sh,
then build the uploaded binary with -Cprofile-use, after checking that it
made it into RUSTFLAGS. Those are the released targets whose runner can
execute the binary it builds, which training needs. Pull requests keep the
plain build, as do the cross-compiled targets and sccache-dist.

The musl container gets gcc and g++ so that the training covers C/C++
compiles too, and the PGO jobs get a longer timeout.

build-pgo.sh now builds the integration test executables without LTO: they
are not trained on, and the sccache they drive is rebuilt with the release
settings anyway.


build-pgo.sh: check that llvm-profdata matches the LLVM of rustc

The profile format changes between LLVM major versions, and an llvm-profdata
from another one either fails to merge what the instrumented binary wrote,
or writes a profile rustc cannot use. Either way that only showed up after
the instrumented build and the training.

Compare the major LLVM versions reported by rustc -vV and
llvm-profdata --version as soon as llvm-profdata is found, and stop with
the version to use if they differ. That matters mostly without rustup, when
llvm-profdata comes from the distribution through --llvm-profdata.

@sylvestre
sylvestre added this pull request to stack #2944 October 24, 2026 13:02
@sylvestre sylvestre changed the title pgo build Build the released binaries with profile-guided optimization Oct 24, 2026
@codecov-commenter

codecov-commenter commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.53%. Comparing base (4e4f932) to head (29769ed).

Additional details and impacted files
@@                Coverage Diff                @@
##           rust-direct-mode    #2943   +/-   ##
=================================================
  Coverage             75.53%   75.53%           
=================================================
  Files                    74       74           
  Lines                 44177    44177           
=================================================
+ Hits                  33369    33370    +1     
+ Misses                10808    10807    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sylvestre
sylvestre force-pushed the pgo-build branch 3 times, most recently from 3758950 to 619521b Compare October 24, 2026 18:00
@sylvestre
sylvestre marked this pull request as ready for review October 25, 2026 08:56
@sylvestre
sylvestre removed this pull request from stack #2944 October 25, 2026 13:14
@sylvestre
sylvestre added this pull request to stack #2952 October 25, 2026 13:14
@sylvestre
sylvestre force-pushed the pgo-build branch 2 times, most recently from 684abd0 to 95379fd Compare October 25, 2026 22:08
scripts/build-pgo.sh builds an instrumented sccache with the same LTO and
codegen-unit settings as the release build, trains it on the integration
tests (system, sccache_cargo, sccache_rustc, cache_hit_rate, sccache_args)
and a generated C/C++ corpus of cache misses and hits, merges the profile
and builds the optimized binary. --train-only stops after the profile, for
CI to pass -Cprofile-use itself; --bench also builds a plain binary and
compares the two with scripts/bench-pgo.sh, which uses hyperfine.

The test executables are built with `cargo test --no-run`, but the sccache
they drive is then rebuilt with `cargo build`: under `cargo test`,
dev-dependencies unify extra features into shared crates, which renames
their symbols and sccache's, and the profile would not match the release
build. The test harnesses are instrumented too, so only the profile written
by sccache itself is kept.

On Windows, std::process::exit skips the atexit handler the profiling
runtime writes the counters from, so the instrumented build
(--cfg pgo_training) writes them before exiting.

On Linux x86_64, PGO cuts the server's instructions per direct-mode cache
hit by 3-5%: most of its time is in blake3, libc and the kernel, which the
profile does not change. It is not enabled in CI.
On tags and on main, the x86_64 Linux (musl), macOS (arm64 and x86_64) and
x86_64 Windows sccache builds train a profile with scripts/build-pgo.sh,
then build the uploaded binary with -Cprofile-use, after checking that it
made it into RUSTFLAGS. Those are the released targets whose runner can
execute the binary it builds, which training needs. Pull requests keep the
plain build, as do the cross-compiled targets and sccache-dist.

The musl container gets gcc and g++ so that the training covers C/C++
compiles too, and the PGO jobs get a longer timeout.

build-pgo.sh now builds the integration test executables without LTO: they
are not trained on, and the sccache they drive is rebuilt with the release
settings anyway.
The profile format changes between LLVM major versions, and an llvm-profdata
from another one either fails to merge what the instrumented binary wrote,
or writes a profile rustc cannot use. Either way that only showed up after
the instrumented build and the training.

Compare the major LLVM versions reported by `rustc -vV` and
`llvm-profdata --version` as soon as llvm-profdata is found, and stop with
the version to use if they differ. That matters mostly without rustup, when
llvm-profdata comes from the distribution through --llvm-profdata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants