ENH: minimally activated compilers

[h-vetinari]

One unavoidable consequence of making compilers easily installable (whether as gcc_linux-64 or through the compilers metapackage), is that people and projects started using them, a lot.

This leads to friction because although we never promised to support use outside conda-forge, it is a de facto reality nowadays.

On top of that there are efforts to better handle build profiles (e.g. release/debug/etc.), c.f. conda-forge/ctng-compiler-activation-feedstock#157.

What I'm thinking about is that we should introduce a third level in the hierarchy between (currently):

• ${<lang>_compiler}_<platform> (fully activated compiler, geared towards conda-forge)
• ${<lang>_compiler}_impl_<platform> (complete dependencies, no activation)

into something like

• ${<lang>_compiler}-cf_<platform> (fully activated compiler, geared towards conda-forge)
• ${<lang>_compiler}_<platform> (minimally activated compiler, e.g. pointing to $PREFIX, but no extra flags)
• ${<lang>_compiler}_impl_<platform> (complete dependencies, no activation)

That would allow external users of our compiler stack to avoid having to play around with stripping out flags, while conda-forge could keep all its default config (e.g. -O2 -DCMAKE_BUILD_TYPE=Release etc.) in the conda-forge-specific compiler flavours.

FYI @wolfv @ruben-arts @jaimergp @mgorny @rgommers
CC @conda-forge/clang-compiler-activation @conda-forge/core

• Move clang++-21 to clang-21 package clangdev-feedstock#387
• Add ld64, llvm-openmp, compiler-rt to dependencies of clang clangdev-feedstock#390
• Move cfg files from clang_impl to clang clangdev-feedstock#417
• Add cc, c++, cpp to match gcc clangdev-feedstock#419
• move clang_impl for native compilers here clangdev-feedstock#405
• openmp v21.1.2 openmp-feedstock#189
• Stop copying omp.h for clang>=21 openmp-feedstock#193
• Remove dependence on clang cctools-and-ld64-feedstock#91
• Add back clang contraint as run_constrains cctools-and-ld64-feedstock#92
• Add llvm version to build string cctools-and-ld64-feedstock#93
• fix build string cctools-and-ld64-feedstock#100
• Remove clang run dependency compiler-rt-feedstock#159
• use sdkroot_env_* clang-compiler-activation-feedstock#176
• use clang_impl_ from clangdev for native clang-compiler-activation-feedstock#182
• Add new sdkroot_env_* output osx-sysroot-feedstock#7
• Make sure sdkroot_env gets activated before compilers osx-sysroot-feedstock#11

[isuruf]

You can already use gcc, gxx, gfortran packages.

[beckermr]

Unfortunately the language platform format is baked into conda-build, so inserting cf_ into it will be non-trivial.

[h-vetinari]

You can already use gcc, gxx, gfortran packages.

I'm fine with making that the pattern, but it's not compatible with the way that clang / clangxx are currently set up on osx. I opened the issue here, because GCC is the most complicated compiler feedstock, but I'm talking about something across our entire compiler stack.

Unfortunately the language platform format is baked into conda-build, so inserting cf_ into it will be non-trivial.

We fully override this in the global pinning, so we could easily change the names of the compiler packages (assuming the respective outputs are installable). We could also introduce a separate name for the "minimal" compilers, but I liked the aspect that the <lang>-compilers metapackages would do the "more obvious thing", while we confine the cf-specific stuff into a package that the vast majority of people won't install by accident.

[rgommers]

This issue largely overlaps with #1998 and #2002, right?

[h-vetinari]

I already referenced the second issue (had forgotten about the first). To me at least, these two were more for discussing the issue(s), whereas this issue is a concrete proposal for how to solve this.

[beckermr]

To override in the pinnings we'd have to have compiler('c_cf') etc in recipes. That's not so much fun to me.

[h-vetinari]

To override in the pinnings we'd have to have compiler('c_cf') etc in recipes.

Not at all. It would suffice to do

 c_compiler:
-  - gcc                        # [linux]
-  - clang                      # [osx]
-  - vs2022                     # [win]
+  - gcc-cf                     # [linux]
+  - clang-cf                   # [osx]
+  - vs2022-cf                  # [win]

in the pinning, and so on for cxx_compiler and fortran_compiler, with no changes to individual recipes.

[isuruf]

Since there's already gcc, gxx, gfortran in this feedstock that does the minimal thing, I don't see the need to add another output here. So, I suggest you raise this in the clang-compiler-activation-feedstock

[isuruf]

Or in the conda-forge.github.io repo where you can talk about the entire compiler stack.

[h-vetinari]

Or in the conda-forge.github.io repo where you can talk about the entire compiler stack.

Transferred

[isuruf]

IMO, relying on activation scripts is the wrong solution for letting our compilers be used by users outside of conda-build.
This is because we want the compilers to work in all sorts of build-systems and make it robust enough that users can't tell the difference between our compilers and the ones they usually see on their system. A solution should take into account that some build systems override CFLAGS, some don't really use CFLAGS or LDFLAGS and that g++ hello.cpp should work out of the box, etc.

In Linux with our gcc package the way things work is the following

1. gcc_impl_linux-64 has all executables prefixed with x86_64-conda-linux-gnu- and no activation flags
   The compiler's default spec file has been customized to use a conda.specs file if it exists, but it doesn't exist in this package.
2. gcc_linux-64 depends on gcc_impl_linux-64 only and adds CC, CFLAGS, LDFLAGS env variables.
3. conda-gcc-specs - this package adds a conda.specs file that adds -isystem $CONDA_PREFIX/include to compile flags and -L $$CONDA_PREFIX/lib -Wl,-rpath,$CONDA_PREFIX/lib when linking.
4. gcc depends on conda-gcc-specs (there's a variant that doesn't, but it's not default and not relevant to this conversation) and also has cc, gcc, cpp symlinks into the prefixed executables.

What this means is that if a user does g++ hello.cpp from an activated environment or /path/to/g++ hello.cpp from a not activated environment or an environment where the user has some values for CFLAGS, LDFLAGS, all of them will do the same thing.
Also the user will not have to do g++ $CXXFLAGS hello.cpp $LDFLAGS and can just do g++ hello.cpp. Technically even without conda-gcc-specs the user can do g++ hello.cpp, but running the executable will sometimes fail because the dynamic loader will be able to load the system libstdc++.so.6, but that maybe old.

TLDR; in Linux g++ hello.cpp with just conda install gxx works perfectly without needing to activate the environment or use environment variables. On macOS, this is not the case since clang doesn't have a concept of spec files. (clang configuration files are not useful for this scenario). We can make macOS work like Linux too with a little bit of work.

[h-vetinari]

IMO, relying on activation scripts is the wrong solution for letting our compilers be used by users outside of conda-build.

I think the "unactivated" use of our compilers is a nice-to-have, but less of a priority to support (than our so-far-explicit lack thereof for any of our compilers) from my POV. Anyone using compilers is already happily using the compilers in activated environments.

But since GCC offers a mechanism to do this without activation scripts, that's great to use it IMO! However, we should also switch the compilers meta-package to depend on gcc / gxx / gfortran IMO.

We can make macOS work like Linux too with a little bit of work.

How do you envision that? I don't mind following this path, but we'd have to come up with some alternate output naming (or rename the existing clang / clangxx, which are just the binaries, not the symlinks with your mechanism).

Also, I'm guessing we won't be able to avoid activation on windows anyway...?

[mgorny]

I think this is a nice idea, and the proposed implementation sounds like a good compromise. And it will definitely make it easier for us to adjust the defaults for conda-forge builds in the future without affecting third parties.

[isuruf]

I think this is a nice idea, and the proposed implementation sounds like a good compromise.

@mgorny thank you for your input. Which idea in particular are you talking about here?

or rename the existing clang / clangxx, which are just the binaries, not the symlinks with your mechanism

clang, clangxx packages are already symlinks to clang-<major> executable in clang-<major> package.

How do you envision that? I don't mind following this path, but we'd have to come up with some alternate output naming (or rename the existing clang / clangxx, which are just the binaries, not the symlinks with your mechanism).

We can add support to clang to load a configuration file conda-clang.cfg that appends options in addition to the configuration file that prepends. Also add a conda-clang-cfg similar to conda-gcc-specs

Then we can downprioritize the clang * default_* variant and add a new clang * user_* variant that depends on conda-clang-cfg, clang-<major>, compiler-rt and other helper packages and also adds the symlinks.