Skip to content

Raspberry Pi: Add --os to build with the target OS's glibc - #24345

Open
BicMak wants to merge 3 commits into
pytorch:mainfrom
BicMak:rpi-os-glibc-sysroot
Open

BicMak wants to merge 3 commits into
pytorch:mainfrom
BicMak:rpi-os-glibc-sysroot

Conversation

@BicMak

@BicMak BicMak commented Oct 30, 2026 •

Copy link
Copy Markdown

Problem

setup.sh builds against the glibc 2.40 sysroot bundled with the Arm toolchain (14.3.rel1). As a result,
llama_main and libllama_runner.so require GLIBC_2.38, GLIBCXX_3.4.32 and CXXABI_1.3.15, so on
Bookworm (glibc 2.36) the loader fails with version 'GLIBC_2.38' not found.

Changes

  • Add --os <bookworm|trixie>: build against the target Debian release's glibc. An arm64 sysroot is
    downloaded with mmdebstrap, and libstdc++/libgcc are linked statically.
  • Add glibc_compat.c: the toolchain's static libstdc++ references __isoc23_strtoul, which glibc < 2.38
    does not provide, so linking fails. glibc 2.38 added this symbol for calls to strtoul compiled in
    C23 mode (it also accepts a 0b prefix). The behavior is the same for the inputs libstdc++ passes, so
    the wrapper simply forwards to strtoul. It is compiled and linked only when the sysroot's glibc is
    older than 2.38. Aliasing it with -Wl,--defsym does not work because strtoul is a shared-library
    symbol (verified).
  • After the build, check that the GLIBC version the binaries require is not newer than what the target OS provides.
  • Without --os the behavior is unchanged (the toolchain's bundled glibc), and a warning points to --os.

Scope and limits

  • Bullseye (glibc 2.31) is not supported and exits with an error. The libraries bundled with the Arm
    toolchain (e.g. libgcc's unwinder) require glibc >= 2.35.
  • Changes are limited to examples/raspberry_pi.
  • Tested only on Raspberry Pi 5. The pi4 target is not tested.

Test plan

Build on the host:

./examples/raspberry_pi/setup.sh pi5 --os bookworm   # or --os trixie

Run on a Raspberry Pi 5 (put llama_main and libllama_runner.so in the same folder):

LD_BIND_NOW=1 ./llama_main --help        # loads the binary and resolves every symbol
./llama_main --model_path llama3_2.pte --tokenizer_path tokenizer.model \
  --prompt "The capital of France is" --temperature 0 --seq_len 32
Build Pi OS Result
--os bookworm Bookworm (glibc 2.36) pass
--os bookworm Trixie (glibc 2.41) pass
--os trixie Trixie (glibc 2.41) pass

Note: the same Raspberry Pi 5 was used for every run. Bookworm and Trixie were installed on two separate
microSD cards, and the card was swapped to boot each OS.

"Pass" means --help prints its usage, and inference exits 0 and generates 26 tokens that contain "Paris"

Sample output (Bookworm build, Bookworm OS)
The capital of France is Paris. The capital of Germany is Berlin. ...
PyTorchObserver {"prefill_token_per_sec":64.9351,"decode_token_per_sec":17.1165,"prompt_tokens":5,"generated_tokens":26, ...}

🤖 Generated with [Claude Code](https://claude.com/claude-code)

cc @nil-is-all @psiddh @AdrianLundell @digantdesai @rascani

@pytorch-bot

pytorch-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24345

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 24 Awaiting Approval

As of commit 7422c28 with merge base 5bfbb59 (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@meta-cla

meta-cla Bot commented Oct 30, 2026

Copy link
Copy Markdown

Hi @BicMak!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@executorch-triage executorch-triage Bot added the community: contribution PRs coming from community (excluding hardware partners) label Oct 31, 2026
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 1, 2026
@psiddh
psiddh self-requested a review November 1, 2026 07:18
@rascani

rascani commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@claude review this PR

@claude

claude Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Claude finished @rascani's task in 4m 0s —— View job


This PR works on a clean first build, but a few things need fixing first: switching to --os or changing its value reuses build state from the earlier sysroot, the script deletes a header from the downloaded toolchain, and the tutorial still points users to the old manual GLIBC workarounds.

Detailed review

PR Review: #24345

Summary

This PR adds --os <bookworm|trixie> to examples/raspberry_pi/setup.sh. With it, the script builds against an arm64 Debian sysroot made with mmdebstrap, links libstdc++/libgcc statically, adds an __isoc23_strtoul shim, and checks the required glibc version after the build.

A clean first build looks right. The main problem is re-running the script: switching to --os, or changing its value, silently reuses build state from the earlier sysroot. That breaks the exact recovery path this PR is meant to provide. The user docs also still recommend the old manual workarounds.

Code Quality

  • Re-running with a different --os reuses stale build state. Major; all three reviewers found it.
    • Core libs (setup.sh:568-574): build_executorch_core returns early if libextension_module.a exists. Nothing records which sysroot built it.
      • The case this PR targets: a Bookworm user runs setup.sh pi5, gets GLIBC_2.38 not found, follows the new warning, and reruns with --os bookworm.
      • The core .a files are still compiled against glibc 2.40 headers, which redirect strtol/sscanf and similar to __isoc23_*.
      • The result is either undefined __isoc23_* link errors (the shim only covers strtoul) or the new check failing with "Binaries require glibc 2.38 …". Neither message mentions --force-rebuild.
    • Cached CMake flags (setup.sh:509): if(NOT CMAKE_EXE_LINKER_FLAGS MATCHES "glibc_compat") checks the cached value, and the flags are written with CACHE … FORCE.
      • Going from --os bookworm to --os trixie changes CMAKE_SYSROOT but keeps the bookworm -isystem/-B/-L paths and bookworm glibc_compat.o.
      • Going from --os bookworm to no --os keeps all of them on top of the bundled sysroot.
    • Suggested fix: write the target OS (or sysroot path) to a stamp file in cmake-out and force a clean reconfigure/rebuild when it changes. At minimum, mention --force-rebuild in the warning and in the error at setup.sh:695.
  • setup.sh:457 deletes a file from the shared downloaded toolchain. It runs rm -f …/include-fixed/pthread.h, and nothing puts it back without --force-download. After one --os run, later builds without --os use a modified toolchain, so "behavior unchanged without --os" is no longer true. Consider a per-sysroot fix (for example, include-path ordering) instead of changing the toolchain. The comment also says "glibc 2.38", while the PR description says the bundled glibc is 2.40. glibc_compat.c:10-11 has the same mismatch.
  • Static libstdc++/libgcc is linked into both libllama_runner.so and llama_main (setup.sh:505,512-514).
    • llama_runner is built SHARED on Linux (examples/models/llama/runner/CMakeLists.txt:40) and linked PUBLIC into llama_main.
    • So the process has two copies of libstdc++ state (EH emergency pool, iostream/locale init, new/delete). The .so also exports them, because there is no -Wl,--exclude-libs,ALL.
    • Testing covered --help and one short inference, which exercises little across that boundary.
    • Consider building llama_runner static on the --os path, or adding --exclude-libs,ALL to the shared library link. The same applies to __isoc23_strtoul, which is exported from the .so with default visibility; marking it hidden is cheap.
  • setup.sh:84-87: a trailing --os with no value exits silently. setup.sh pi5 --os makes shift 2 fail with $# == 1, and set -e exits with status 1 and no message. Separately, setup.sh --os pi5 takes pi5 as the OS. Check $# -ge 2 and print an error.
  • setup.sh:681 and :692: the version check is fragile.
    • sort -t. -k2 -n compares only the minor number. sort -V is simpler and correct.
    • Under pipefail, if grep -o matches nothing (objdump fails, and its stderr goes to /dev/null), the required=$(…) assignment kills the script with no message.
    • If objdump is missing, the check is skipped silently (:676-678).
    • Only GLIBC_* is checked. The no---os path also needs GLIBCXX_3.4.32/CXXABI_1.3.15 (from the PR description), but the warning only mentions GLIBC. On the --os path, nothing confirms that the GLIBCXX dependencies are actually gone.
  • setup.sh:23: TARGET_OS="${TARGET_OS:-}" reads a very generic environment variable. Any environment that already exports TARGET_OS (for example TARGET_OS=linux) makes every run fail with "Unsupported --os". Initialize it to empty and only set it from the CLI.
  • setup.sh:419 checks for mmdebstrap late. The check runs after setup_environment (which may run a full install_executorch.sh) and after the toolchain download. Move it to check_prerequisites, next to the --os validation, so missing tools fail fast.
  • setup.sh:415 treats the sysroot as complete if features.h exists. An interrupted mmdebstrap run that already extracted libc6-dev but not linux-libc-dev is reused forever. Extract to a temp dir and mv it into place, or write a stamp file when extraction finishes.
  • setup.sh:19-20: unrelated change. WORKSPACE_DIR now comes from the script location plus a cd, which the PR description doesn't mention. It also makes the "run from the ExecuTorch root directory" check (around :239) dead code. The display_next_steps relabeling is unrelated churn too.
  • The PR description says the shim is "compiled and linked only when the sysroot's glibc is older than 2.38". In fact, glibc_compat.o is always compiled and linked on the --os path; only the function body is #if-guarded. Fix the description, or skip the object when it isn't needed.

Infrastructure

  • The tutorial is not updated. docs/source/raspberry_pi_llama_tutorial.md:192-325 ("Issue 1: GLIBC Version Mismatch") still recommends upgrading glibc on the Pi (Solution A, marked "Recommended") or hand-building a sysroot and editing the generated toolchain file (Solution B). setup.sh overwrites that toolchain file on every run, and line 321 calls a nonexistent rpi_setup.sh. The build step at :92 doesn't mention --os or the mmdebstrap/keyring prerequisites. That troubleshooting section should now just point users to --os.
  • No CI runs examples/raspberry_pi/setup.sh. The only Raspberry Pi workflow covers pico2/**, so this path and its regressions go untested. Low confidence that it's worth adding, given the network and sysroot download involved.

Security

  • Sysroot downloads are not pinned and may fail on older hosts. They use apt's default mirror and rely on the host's debian-archive-keyring. On older hosts such as Ubuntu 22.04, the keyring may not include the trixie archive keys, and the install hint wouldn't fix that. Nothing is pinned (no snapshot), so builds aren't reproducible. Low confidence: suspected, not verified.

Recommendation

Request Changes

The stale-state problem when adding or switching --os needs fixing, or at least a clear --force-rebuild instruction, because it breaks the recovery path this PR is meant to provide. The toolchain should also not be modified in place, and the tutorial's GLIBC troubleshooting section should point to --os.

setup.sh builds with the Arm toolchain's bundled glibc (2.40), so the
binaries require GLIBC_2.38, GLIBCXX_3.4.32 and CXXABI_1.3.15. Raspberry Pi
OS Bookworm (glibc 2.36) fails to load them with "GLIBC_2.38 not found".

--os <bookworm|trixie> builds against that Debian release's glibc, using an
arm64 sysroot from mmdebstrap, and links libstdc++/libgcc statically. The
toolchain's static libstdc++ references __isoc23_strtoul, which glibc < 2.38
lacks, so glibc_compat.c provides it only for older sysroots (--defsym cannot
alias it because strtoul lives in a shared library). After the build, the
required GLIBC version is checked against what the target OS ships.

Without --os the behavior is unchanged and a warning points to the new option.
Bullseye (glibc 2.31) is rejected: the libraries bundled with the Arm toolchain
require glibc >= 2.35.

Authored with Claude Code.
@BicMak
BicMak force-pushed the rpi-os-glibc-sysroot branch from 38ea848 to b97f0bf Compare November 8, 2026 13:05
@BicMak
BicMak requested a review from mergennachin as a code owner November 8, 2026 13:05
@nil-is-all nil-is-all added the module: microcontrollers For embedded MCUs like Cortex-M, or RTOS like Zephyr, does not track NPU backend like Arm Ethos. label Nov 9, 2026
@nil-is-all
nil-is-all requested a review from rascani November 9, 2026 22:10
@BicMak

BicMak commented Nov 12, 2026 •

Copy link
Copy Markdown
Author

Thank you for the feedback on using --exclude-libs,ALL in the previous review. I did not include this change in the latest commit, and I thought it would be helpful to explain why. I've summarized my findings below and would appreciate your thoughts.

Summary

After applying and testing --exclude-libs, keeping the current code turned out to be the better option.

  • The premise does not match the measurement. In the current build, the process has a single copy of libstdc++. Only 2 libstdc++ objects end up in the exe (eh_ptr.o, condition_variable.o), and neither exists in the .so, so 0 objects overlap. For everything else, the exe uses the .so's copy (Section 3).
  • Only libgcc overlaps, and that is safe. The unwinder (libgcc_eh) and the libgcc helpers exist twice, but they are designed not to need any state shared between copies (Section 6).
  • --exclude-libs does not fix the root problem; it creates it. ALL duplicates the backend registry and crashes (Section 4). libstdc++.a runs, but turns the single libstdc++ into two copies (Section 5).

So the "root problem of libstdc++ being instantiated in both main and runner" does not occur in the current code, and the second submission left the link options unchanged.

Details

1. Background: why libstdc++ is statically linked

  • An --os build compiles with the toolchain's recent GCC and takes glibc from the sysroot (Bookworm, glibc 2.36).
  • Because the compiler is recent, the binaries require GLIBCXX_3.4.32 and CXXABI_1.3.15. Bookworm's system libstdc++ (GCC 12) does not provide them, so a dynamically linked build fails to load on the Pi.
  • Therefore the C++ runtime is linked into the binaries with -static-libstdc++ -static-libgcc.
  • These flags apply to both llama_main (exe) and libllama_runner.so, which is why the reviewer asked whether two copies of libstdc++ would conflict.

2. Criterion: when are two copies a problem?

Having two copies of the code is not a problem by itself. It breaks when both copies must see the same global state but each sees its own.

Component State that must be shared With two copies
ExecuTorch backend registry Table of registered backends Registration and lookup use different tables → immediate crash
libgcc_eh (unwinder) None. Walks all loaded modules via dl_iterate_phdr each time Safe (the intended behavior of -static-libgcc)
libstdc++ Mostly none, but some global state exists: in-flight exception state (__cxa_eh_globals), std::locale, iostream Usually invisible, but not guaranteed

3. Measurement: the current build shares a single libstdc++

The link commands for cmake-out (Bookworm) were re-run unchanged from link.txt to inspect the linker maps and symbols.

  • libstdc++ objects: 111 in libllama_runner.so, only 2 in llama_main (eh_ptr.o, condition_variable.o)
  • The exe does not contain __cxa_throw, __gxx_personality_v0, operator new, etc.; it uses the ones exported by the .so (exe U / .so T)
  • Cause: in the link order, the .so comes before libstdc++.a (examples/models/llama/CMakeLists.txt:250), and the .so exports these symbols, so the linker resolves them from the .so first
  • Result: the libstdc++ actually used is the single copy inside the .so
Verification script, result tables, logs
Verification script (libstdcxx_sharing_check.sh <executorch root> <out dir>)
#!/usr/bin/env bash
# Measures how libstdc++/libgcc_eh are split between llama_main and libllama_runner.so
# in a flagless --os build (no --exclude-libs).
#
# The deployed llama_main is stripped (-Wl,-s), so both binaries are relinked from
# CMake's own link.txt with -Wl,-s removed and -Wl,-Map added. Outputs go to OUT_DIR;
# the build directory is not modified.
#
# Usage: libstdcxx_sharing_check.sh <executorch root> <out dir>
set -uo pipefail

ET_ROOT="$(cd "${1:?executorch root}" && pwd)"
OUT_DIR="${2:?out dir}"
mkdir -p "$OUT_DIR"
OUT_DIR="$(cd "$OUT_DIR" && pwd)"
LLAMA_DIR="$ET_ROOT/cmake-out/examples/models/llama"
TC_BIN="$(ls -d "$ET_ROOT"/examples/raspberry_pi/gen/arm-toolchain/*/bin | head -1)"
NM="$TC_BIN/aarch64-none-linux-gnu-nm"
STRIP="$TC_BIN/aarch64-none-linux-gnu-strip"
export LC_ALL=C

# Name prefixes of C++ runtime symbols (libstdc++/libsupc++/libgcc_eh). This is a name
# match, so it also counts std:: template instantiations generated from headers.
RT='^(_ZNSt|_ZNKSt|_ZSt|_ZTISt|_ZTVSt|_ZTSSt|_ZNSo|_ZNSi|_ZNKSo|_ZTVN10__cxxabiv|_ZTIN10__cxxabiv|__cxa_|__gxx_personality|_Unwind_|__gcc_|_ZdlPv|_Znwm|_ZdaPv|_Znam|_ZN9__gnu_cxx|_ZNSs|_ZNSbIw|_ZN10__cxxabiv)'

step() { printf '\n==== %s ====\n' "$1"; }
run() { printf '$ %s\n' "$*"; eval "$@"; }

step "0. Inputs"
run "date -Iseconds"
run "cd '$LLAMA_DIR'"
run "sha256sum llama_main runner/libllama_runner.so"
run "stat -c '%n %s' llama_main runner/libllama_runner.so"
run "grep -oE 'CMAKE_SYSROOT \"[^\"]*\"|STATIC_RUNTIME_FLAGS \"[^\"]*\"' '$ET_ROOT'/arm-toolchain-pi*.cmake"
run "'$TC_BIN/aarch64-none-linux-gnu-g++' --version | head -1"

step "1. Relink with linker maps (same link.txt, -Wl,-s removed)"
so_cmd=$(sed -e "s# -o libllama_runner.so # -o $OUT_DIR/libllama_runner.so -Wl,-Map=$OUT_DIR/libllama_runner.map #" runner/CMakeFiles/llama_runner.dir/link.txt)
exe_cmd=$(sed -e 's/ -Wl,-s / /' -e "s# -o llama_main # -o $OUT_DIR/llama_main -Wl,-Map=$OUT_DIR/llama_main.map #" CMakeFiles/llama_main.dir/link.txt)
echo "\$ (cd runner && <link.txt with -o $OUT_DIR/libllama_runner.so -Wl,-Map=...>)"
(cd runner && eval "$so_cmd") || { echo "relink .so failed"; exit 1; }
echo "\$ <link.txt without -Wl,-s, with -o $OUT_DIR/llama_main -Wl,-Map=...>  (links against runner/libllama_runner.so)"
eval "$exe_cmd" || { echo "relink exe failed"; exit 1; }
echo "Relink sanity check: stripped relink vs deployed binary"
run "'$STRIP' -o '$OUT_DIR/llama_main.stripped' '$OUT_DIR/llama_main' && stat -c '%n %s' '$OUT_DIR/llama_main.stripped' llama_main"
run "stat -c '%n %s' '$OUT_DIR/libllama_runner.so' runner/libllama_runner.so"

step "2. Link input order (llama_main.map LOAD lines)"
run "grep -E '^LOAD ' '$OUT_DIR/llama_main.map' | grep -E 'llama_runner|libstdc|libgcc|libc\\.so|libm\\.so'"

members() {  # archive members of libstdc++/libgcc_eh/libgcc pulled into a binary
  sed -n '1,/^Discarded input sections/p' "$1" | grep -E '^/' |
    grep -oE '(libstdc\+\+|libgcc_eh|libgcc)\.a\([^)]*\)' | sort -u
}

step "3. Runtime archive members pulled into each binary (from linker maps)"
for b in libllama_runner llama_main; do
  members "$OUT_DIR/$b.map" > "$OUT_DIR/$b.members"
  echo "-- $b: libstdc++.a $(grep -c '^libstdc' "$OUT_DIR/$b.members")," \
       "libgcc_eh.a $(grep -c '^libgcc_eh' "$OUT_DIR/$b.members")," \
       "libgcc.a $(grep -c '^libgcc\.a' "$OUT_DIR/$b.members")"
done
echo "-- llama_main members (all):"; cat "$OUT_DIR/llama_main.members"
echo "-- libllama_runner libgcc_eh members:"; grep '^libgcc_eh' "$OUT_DIR/libllama_runner.members"
echo "-- llama_main runtime members that also exist in libllama_runner.so:"
comm -12 "$OUT_DIR/llama_main.members" "$OUT_DIR/libllama_runner.members"

step "4. Why each libstdc++/libgcc_eh member was pulled into llama_main (map: member, then 'referencing file (symbol)')"
awk '/^Archive member included/{f=1} /^Discarded input sections/{f=0} f' "$OUT_DIR/llama_main.map" |
  grep -A1 -E '^/.*(libstdc\+\+|libgcc_eh)\.a\(' | grep -v '^--$' |
  sed -E "s#$ET_ROOT/##g; s#examples/raspberry_pi/gen/arm-toolchain/[^[:space:]]*/(libstdc\+\+|libgcc_eh|libgcc)\.a#\1.a#g" | c++filt

step "5. Dynamic symbols"
run "'$NM' -D --defined-only runner/libllama_runner.so | wc -l"
echo "\$ nm -D --defined-only runner/libllama_runner.so | awk '{print \$3}' | grep -E \"\$RT\" | sort -u | wc -l"
"$NM" -D --defined-only runner/libllama_runner.so | awk '{print $3}' | grep -E "$RT" | sort -u | wc -l
run "'$NM' -D --undefined-only llama_main | wc -l"
"$NM" -D --undefined-only llama_main | awk '{print $2}' | grep -v '@GLIBC' > "$OUT_DIR/llama_main.undef_nonlibc"
echo "llama_main UNDEF not versioned to glibc (resolved from libllama_runner.so): $(wc -l < "$OUT_DIR/llama_main.undef_nonlibc")"
echo "  of which C++ runtime names (RT pattern): $(grep -cE "$RT" "$OUT_DIR/llama_main.undef_nonlibc")"
echo "-- the C++ runtime names llama_main takes from libllama_runner.so:"
grep -E "$RT" "$OUT_DIR/llama_main.undef_nonlibc" | c++filt

step "6. Key symbols: where each is defined (T/t = defined, U = imported, - = absent; relinked unstripped binaries)"
printf '%-55s %-6s %-6s %s\n' symbol exe so so-exported
for s in __cxa_throw __cxa_begin_catch __cxa_get_globals __gxx_personality_v0 _Znwm _ZdlPv \
         _ZNSt6locale7classicEv _ZSt17rethrow_exceptionNSt15__exception_ptr13exception_ptrE \
         _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE \
         _Unwind_RaiseException _Unwind_Resume _Unwind_SetGR dwarf_reg_size_table; do
  e=$("$NM" "$OUT_DIR/llama_main" | awk -v s="$s" '$NF==s{print $(NF-1); exit}')
  o=$("$NM" "$OUT_DIR/libllama_runner.so" | awk -v s="$s" '$NF==s{print $(NF-1); exit}')
  d=$("$NM" -D --defined-only "$OUT_DIR/libllama_runner.so" | awk -v s="$s" '$NF==s{print "yes"; exit}')
  printf '%-55s %-6s %-6s %s\n' "$(echo "$s" | c++filt)" "${e:--}" "${o:--}" "${d:-no}"
done
echo
echo "done. maps and member lists: $OUT_DIR"

What each step of the script checks:

Step Input Output
0 Deployed binaries sha256 and size (records which build was measured)
1 Both link.txt → relink Linker maps and unstripped binaries. The stripped relink's size is compared with the deployed binary to confirm they match
2 LOAD lines in llama_main.map Link order
3 Archive member lists in both maps Number of libstdc++ objects in each binary
4 "Archive member included" in llama_main.map Why the exe pulled each object
5, 6 nm Symbols the exe imports from the .so, and where each key symbol is defined

Result: libstdc++ objects pulled in

Binary libstdc++.a libgcc_eh.a libgcc.a
libllama_runner.so 111 5 16
llama_main 2 (eh_ptr.o, condition_variable.o) 5 12

Where key symbols are defined (T = defined and exported, t = defined but hidden, U = imported from elsewhere):

Symbol exe .so Exported by .so?
__cxa_throw U T yes
__cxa_begin_catch U T yes
__cxa_get_globals U T yes
__gxx_personality_v0 U T yes
operator new(unsigned long) U T yes
std::rethrow_exception(...) T - no
std::condition_variable::wait(...) T - no
_Unwind_RaiseException T t no
_Unwind_Resume T t no
Script log (excerpt)
==== 3. Runtime archive members pulled into each binary (from linker maps) ====
-- libllama_runner: libstdc++.a 111, libgcc_eh.a 5, libgcc.a 16
-- llama_main: libstdc++.a 2, libgcc_eh.a 5, libgcc.a 12
-- llama_main members (all):
libgcc.a(_divdc3.o)
libgcc.a(_divsc3.o)
libgcc.a(_muldc3.o)
libgcc.a(_mulsc3.o)
libgcc.a(cas_8_4.o)
libgcc.a(divtf3.o)
libgcc.a(floatunditf.o)
libgcc.a(ldadd_4_4.o)
libgcc.a(lse-init.o)
libgcc.a(sfp-exceptions.o)
libgcc.a(swp_8_4.o)
libgcc.a(trunctfdf2.o)
libgcc_eh.a(__aarch64_have_sme.o)
libgcc_eh.a(__arm_tpidr2_save.o)
libgcc_eh.a(__arm_za_disable.o)
libgcc_eh.a(unwind-dw2-fde-dip.o)
libgcc_eh.a(unwind-dw2.o)
libstdc++.a(condition_variable.o)
libstdc++.a(eh_ptr.o)
-- libllama_runner libgcc_eh members:
libgcc_eh.a(__aarch64_have_sme.o)
libgcc_eh.a(__arm_tpidr2_save.o)
libgcc_eh.a(__arm_za_disable.o)
libgcc_eh.a(unwind-dw2-fde-dip.o)
libgcc_eh.a(unwind-dw2.o)
-- llama_main runtime members that also exist in libllama_runner.so:
libgcc.a(cas_8_4.o)
libgcc.a(floatunditf.o)
libgcc.a(ldadd_4_4.o)
libgcc.a(lse-init.o)
libgcc.a(sfp-exceptions.o)
libgcc.a(swp_8_4.o)
libgcc_eh.a(__aarch64_have_sme.o)
libgcc_eh.a(__arm_tpidr2_save.o)
libgcc_eh.a(__arm_za_disable.o)
libgcc_eh.a(unwind-dw2-fde-dip.o)
libgcc_eh.a(unwind-dw2.o)

==== 6. Key symbols: where each is defined (T/t = defined, U = imported, - = absent; relinked unstripped binaries) ====
symbol                                                  exe    so     so-exported
__cxa_throw                                             U      T      yes
__cxa_begin_catch                                       U      T      yes
__cxa_get_globals                                       U      T      yes
__gxx_personality_v0                                    U      T      yes
operator new(unsigned long)                             U      T      yes
operator delete(void*)                                  -      T      yes
std::locale::classic()                                  -      T      yes
std::rethrow_exception(std::__exception_ptr::exception_ptr) T      -      no
std::condition_variable::wait(std::unique_lock<std::mutex>&) T      -      no
_Unwind_RaiseException                                  T      t      no
_Unwind_Resume                                          T      t      no
_Unwind_SetGR                                           T      t      no
dwarf_reg_size_table                                    b      b      no

Why: link order + symbols exported by the .so

examples/models/llama/CMakeLists.txt:250:

target_link_libraries(llama_main PUBLIC llama_runner ${link_libraries})

The actual llama_main link command (CMakeFiles/llama_main.dir/link.txt):

$ tr ' ' '\n' < examples/models/llama/CMakeFiles/llama_main.dir/link.txt | grep -nE 'llama_runner|\.a$|static-lib'
16: -static-libstdc++
38: .../libxnnpack_backend.a   # whole-archive: backend registration code goes into the exe
45: runner/libllama_runner.so  # the .so comes first
46: .../libexecutorch.a ...    # the .a files come after
  • The linker resolves symbols in order, and libllama_runner.so comes before libstdc++.a.
  • The .so has no symbol-hiding option, so the libstdc++ symbols statically linked into it are exported as-is.
  • So the C++ runtime the exe needs is resolved from the .so first, and only the 2 objects missing from the .so are pulled from libstdc++.a.
  • As a result, the libstdc++ actually used is the single copy inside the .so.

4. --exclude-libs,ALL: crash

  • Cause: all symbols of the .so are hidden, so the exe gets its own copy of the backend registry. Backends register into the exe's registry, while lookups go to the .so's registry.
  • Result: on Pi 5 (Bookworm), the process exits with backend not registered.
gdb log
executorch::runtime::Program const*, executorch::runtime::BackendInitContext&, executorch::runtime::BackendDelegate*) () from /home/jiwon/poc-exp/flag/libllama_runner.so
CHECK returned x0=0x0  (NULL: backend not registered)
[Inferior 1 (process 2015) exited with code 01]

5. --exclude-libs,libstdc++.a: runs, but creates two copies

  • Confirmed that it runs on Pi 5.
  • However, isolating libstdc++ leaves one copy in the exe and one in the .so: two copies (U __cxa_throw disappears from nm -D llama_main).
  • In other words, it does not fix the state the reviewer was concerned about; it creates it.

6. Scope and limitations

  • This PR only changes setup.sh and the generated toolchain file.
  • Limitation 1: it depends on link order. Sharing a single libstdc++ comes from the .so being linked before the .a files. This order is not introduced by this PR; it is the existing upstream structure (examples/models/llama/CMakeLists.txt:250).
  • Limitation 2: the libgcc_eh unwinder already exists twice. libgcc_eh is built with hidden symbols in the archive itself, so the .so does not export it (exe T, .so t in the symbol table in Section 3 details). However, the unwinder is designed to walk all loaded modules via dl_iterate_phdr (unwind-dw2-fde-dip.o) each time, with no shared state, so it can unwind across the exe and the .so even with two copies. This is the intended behavior of -static-libgcc.

7. Conclusion

Configuration libstdc++ Registry Verdict
Current One copy (shared) One copy (shared) Keep
--exclude-libs,ALL Two copies Two copies Crashes; not usable
--exclude-libs,libstdc++.a Two copies One copy Runs, but creates the two-copy state the reviewer was concerned about

The reviewer's concern ("two copies of the same library will conflict") is valid. However, measurement shows the current structure already uses a single libstdc++, and --exclude-libs creates that concern rather than resolving it. Therefore the current code is left unchanged.

cc @nil-is-all @psiddh @AdrianLundell @digantdesai @rascani

rascani
rascani previously approved these changes Nov 19, 2026
@rascani

rascani commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Sorry for the slow review! We appreciate the contribution.

@rascani rascani closed this Nov 19, 2026
@rascani rascani reopened this Nov 19, 2026
@pytorch-bot
pytorch-bot Bot dismissed rascani’s stale review November 19, 2026 01:09

This PR was reopened (likely due to being reverted), so your approval was removed. Please request another review.

@rascani

rascani commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

@BicMak - Would you mind merging main into the branch? There are a couple required checks that seem to have an issue running.

@BicMak

BicMak commented Nov 21, 2026

Copy link
Copy Markdown
Author

@rascani Thank you for approving my PR. I've merged the latest main into the branch as requested. Could you please check it again?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. community: contribution PRs coming from community (excluding hardware partners) module: microcontrollers For embedded MCUs like Cortex-M, or RTOS like Zephyr, does not track NPU backend like Arm Ethos.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants