LLVM compiler discovery can register external `llvm+clang` even when `libclang` is not build-usable

[bernhardkaindl]

Bug: LLVM compiler discovery can register external llvm+clang even when libclang is not build-usable

• Fixed by llvm: Add libclang variant with external detection #5561
• Slack discussion: https://spackpm.slack.com/archives/D0BG6A6H4D7/p1783765910667869 (#packaging)

Summary

spack compiler find can auto-register a system LLVM installation as an external llvm+clang package based on the presence of a detected clang compiler, even when the host installation cannot actually compile/link dependent packages against libclang.

This can cause packages with depends_on("llvm+clang") to select the auto-detected external LLVM and fail later during CMake configuration or linking.

Description

When Spack discovers compilers, it may also register external packages associated with those compilers. For clang, Spack can add an external llvm package with compiler metadata under extra_attributes.compilers.

The detected external LLVM spec may also include variants such as +clang.

However, +clang is used by packages that need the clang/libclang development interface, not just a working clang executable. On Linux distributions, it is common for the compiler executable and some LLVM runtime libraries to be installed without all development files or transitive development dependencies required to build new software against libclang.

For example, a system may have:

• a working clang-18 executable
• LLVM and clang development packages installed
• libclang libraries present
• missing transitive development dependencies needed by LLVM's exported CMake targets, such as libzstd-dev

In that situation, Spack may still register the external LLVM as llvm@18+clang, causing dependent packages to select it. The failure is then deferred until the dependent package tries to use LLVM/Clang through CMake.

Expected behavior

Spack should only auto-register an external LLVM as +clang when the detected installation is usable by dependent packages that require the clang/libclang development interface.

If libclang cannot be compiled, linked, and used successfully, the external LLVM should either not be marked +clang, or should otherwise not satisfy depends_on("llvm+clang").

Actual behavior

Spack registers the system LLVM as external llvm@18+clang after detecting the system clang compiler, even though packages that use LLVM/Clang through CMake fail because a transitive dependency target is missing.

Reproducer

Precondition: no existing Spack-built llvm@18+clang should satisfy the dependency, so the auto-detected external LLVM is selected.

$ spack uninstall -y -a --dependents llvm+clang@18
$ grep -A5 llvm ~/.spack/packages.yaml
# no llvm external present

Install a system clang compiler and clang development package, but remove the zstd development package:

$ sudo apt install -y clang-18
$ sudo apt remove -y libzstd-dev
$ sudo apt install -y libclang-18-dev

Run compiler discovery:

$ spack compiler find
==> Added 1 new compiler to /home/bkaindl/.spack/packages.yaml
    llvm@18.1.3
==> Compilers are defined in the following files:
    $HOME/.spack/packages.yaml

Spack now shows the external LLVM as satisfying llvm+clang@18:

$ spack find -p llvm+clang@18
-- linux-ubuntu24.04-x86_64 / no compilers ----------------------
llvm@18.1.3  /usr
==> 1 installed package

The generated external package entry includes +clang:

llvm:
  externals:
  - spec: llvm@18.1.3+clang~flang~lld~lldb
    prefix: /usr
    extra_attributes:
      compilers:
        c: /usr/bin/clang
        cxx: /usr/bin/clang++

Now install a package that depends on llvm+clang:

$ spack install -v proteus %llvm@18

The build fails during CMake configuration:

-- Found LLVM libraries: LLVM
-- Found Clang libraries: clang-cpp
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done

CMake Error at /usr/lib/llvm-18/lib/cmake/llvm/LLVMExports.cmake:73 (set_target_properties):
  The link interface of target "LLVMSupport" contains:

    zstd::libzstd_shared

  but the target was not found.

Relevant backtrace:

/usr/lib/llvm-18/cmake/LLVMConfig.cmake:375 (include)
cmake/SetupLLVM.cmake:3 (find_package)
CMakeLists.txt:106 (include)

Why this matters

The external package was added automatically by compiler discovery, without the user explicitly declaring that /usr provides a usable llvm+clang development installation.

Because the external is marked +clang, Spack can choose it for packages that require llvm+clang. The failure then appears later in an unrelated package build, even though the root cause is that the auto-detected external LLVM does not actually provide a complete build-usable clang/libclang development interface.

Possible fixes

The "clang" variant likely should not be changed, at least it is not easy: llvm.spec providing "c" and "cxx" compilers depends on the "+clang" variant, and this is apparently not easy to change, and +clang/~clang would change for existing hosts if we change the detection.

• I tried this in llvm: require usable libclang before detecting external +clang #5554 and changes for the spack solver, which failed so far.

Instead, llvm could add a new +libclang variant which determines if llvm provides libclang development functionality and for external packages, before setting it to +libclang, verify that the detected installation can compile, link, and run a minimal program using libclang, using include and library paths from the paired llvm-config

• Implemented by llvm: Add libclang variant with external detection #5561