Build: one ld.lld link is 11.17 GiB — the Linux memory ceiling is the link, and nothing bounds it there #1313

Is your feature request related to a problem? Please describe.

#1305's measurement landed, and the headline is not the one anyone expected: the single
largest memory consumer in a Linux build is one ld.lld link, not any translation unit —
and nothing bounds it there.

Measured by .github/workflows/build-memory.yml on olo-ci-1 (PR #1308), 1761 compiles per
cell at 99.9% coverage, compiler cache off, PCH and unity off:

11.17 GiB is 80% of the actions-runner-*.service MemoryMax=14G. The figure it
replaces was 8.7 GB, so the real number is 38% higher — and it is 10.03 GiB with no
sanitizer instrumentation at all, so this is not an ASan artifact.

Why nothing stops it

OLO_LINK_JOBS (the olo_link pool) and olo_heavy are Ninja job pools, and the root
CMakeLists.txt guards both with if(CMAKE_GENERATOR MATCHES "Ninja"). Every Linux job in
this repo configures with no -G and therefore gets Unix Makefiles, where both pools
silently do not exist — see
build-trees-and-windows-asan.md §5e.
cmake/LinkSemaphore.cmake does not cover it either, and this is confirmed in the run's own
log rather than inferred — the box has no PowerShell, so the wrapper is never wired and the
module fails open by design:

Link semaphore: neither 'pwsh' nor 'powershell' found on PATH — linking
WITHOUT cross-tree throttling.

So under make -j2 nothing prevents that link running concurrently with a compile:

• link + heaviest compile = 19.26 GiB — past the 14 GiB unit cap, at the 19 GiB
  user-1004.slice cap
• link + two p95 compiles = 12.9 GiB — inside 14 GiB, but with 1.1 GiB to spare

In practice the OloEngine-Tests link lands at the very end of the build when little else
is left, which is almost certainly why this has not been failing constantly. That is luck of
scheduling, not a guarantee.

Why this is worth its own issue

It reframes two long-standing conclusions without contradicting either:

• Build: instrument the build (CMake Instrumentation API), then revisit the -j6 parallelism cap with data #759 concluded OLO_LINK_JOBS=2 is not the serialisation point, compilation is. That
  was a wall-clock finding and is still correct — links are off the critical path. For
  memory the opposite holds, and the two were never distinguished.
• The cgroup caps were sized against per-TU compile memory. At 8.09 GiB the "~9.0 GB per
  clang++ TU" folklore turns out roughly right, so the caps are not wrong for compiles —
  they are simply not sized against the thing that actually peaks. This is also a better
  candidate explanation for historical OOM kills than compile memory ever was, though note
  the 2026-09-17 kills were separately diagnosed as page-cache reclaim against MemoryHigh
  and are not evidence for this.

Describe the solution you'd like

1. Bound link concurrency under Makefiles. Options, cheapest first:
   • CMAKE_JOB_POOLS is Ninja-only, so the pool approach cannot be ported. The Makefile
     generator has no equivalent.
   • A CMAKE_CXX_LINKER_LAUNCHER wrapper that takes a real semaphore — this is what
     LinkSemaphore.cmake already is, and making it work on Linux (a flock-based shell
     wrapper instead of pwsh) would cover both the intra-tree and cross-slot cases. The two
     olo-ci slots are in ONE 19 GiB slice, so two concurrent links across slots is
     22.3 GiB — already over, and equally unguarded today.
   • Or accept it and size the caps for it: a 14 GiB unit cap has 2.8 GiB of headroom
     over one link, which is the number to defend rather than --parallel.
2. Ask whether the link needs 11 GiB at all. OloEngine-Tests is a single enormous
   executable. -gsplit-dwarf + --gdb-index is the obvious lever (debug info stops being
   loaded by the linker) and Build: per-TU peak RSS is unmeasured — --parallel 2 and the 14 GiB cap rest on five numbers that disagree #1305 verified ccache caches and restores .dwo correctly.
   Unverified before adopting: this repo maps DW_AT_comp_dir to /olo via
   -ffile-prefix-map, and that is what a sanitizer symbolizer would have to resolve the
   .dwo through — confirm a stack still symbolises first.
3. Do not raise --parallel as a substitute. The compile side has room (non-sanitizer
   Debug fits 4 lanes), but the link is what is tight and more lanes only increases the
   chance something is running beside it.

Describe alternatives you've considered

• Lower the caps' pressure by splitting OloEngine-Tests. Real, and much larger work; the
  test binary's size is also a runtime problem already noted in Build: per-TU peak RSS is unmeasured — --parallel 2 and the 14 GiB cap rest on five numbers that disagree #1305 (gtest_discover_tests
  gives one process per case, at 7-14.5 GB each). Different problem, do not merge them.
• Leave it. Defensible while the link stays at the end of the build — but it is unguarded,
  and the margin is 2.8 GiB on a box that has livelocked twice.

Additional context

• Measurement, recipe and traps: docs/agent-rules/build-memory-per-tu.md (PR build: measure per-TU and per-link peak RSS, and publish the ranking (#1305) #1308).
• Artifacts: build-memory-debug / build-memory-debug-asan, published weekly.
• Related: Build: per-TU peak RSS is unmeasured — --parallel 2 and the 14 GiB cap rest on five numbers that disagree #1305 (the measurement), Build: the olo_heavy file set is mispopulated in both directions — re-derive it from #1305's ranking #1307 (olo_heavy membership — compile side only),
  CI (umbrella): cut the per-PR critical path — measured breakdown of the ~4 h wait #1084 (CI critical path), CI: the Video (FFmpeg backend) Linux job has failed every run for 10+ weeks (bare --parallel OOM, then an unpinned compiler) #796 (the Makefiles/--parallel generator trap).

Score

capability: 4
craft: 1
stability: 3
decay: 2
effort: 3
confidence: 0.6
learning: 3
fun: 1
kano: performance
blocked_by: []
blocks: []

Scoring notes: capability 4 — no engine capability, but it is the memory ceiling every
self-hosted Linux job runs against, and it is currently unguarded on the generator CI
actually uses; both cgroup caps are sized against the wrong quantity. craft 1 —
invisible. stability 3 — an unbounded 11 GiB step under a 14 GiB cap produces real
job-killing failures, and two concurrent links across the slice's two slots exceed it
outright; it bites occasionally rather than corrupting anything. decay 2 — the link grows
with the test binary, so the margin shrinks quietly, but slowly. effort 3, confidence
0.6 — a flock link wrapper is small, but making it compose with the existing pwsh
semaphore without a fail-closed deadlock needs care, and the -gsplit-dwarf arm depends on
the symbolizer question, which is genuinely open. performance — makes existing work
survivable rather than adding capability.