Flag 'coordination_agent_recoverable' was defined more than once but with differing types.

[timpalpant]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

When the CUDA build is installed via conda, it fails with the error:

ERROR: Flag 'coordination_agent_recoverable' was defined more than once but with differing types. Defined in files 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc' and 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc'.

This error does not occur with the cpu build variant.

To repro: On Ubuntu 22.04:

1. Installed JAX as described here:

conda create -n test-jax --override-channels -c conda-forge jax 'jaxlib=*=*cuda*'

2. Try using it:

python -c 'import jax.numpy; jax.numpy.array([1,2,3])'

ERROR: Flag 'coordination_agent_recoverable' was defined more than once but with differing types. Defined in files 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc' and 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc'.

Installed packages

# packages in environment at /home/tim/conda/envs/test-jax:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.5               hb9d3cd8_0    conda-forge
ca-certificates           2025.8.3             hbd8a1cb_0    conda-forge
cuda-cccl_linux-64        12.9.27              ha770c72_0    conda-forge
cuda-crt-dev_linux-64     12.9.86              ha770c72_2    conda-forge
cuda-crt-tools            12.9.86              ha770c72_2    conda-forge
cuda-cudart               12.9.79              h5888daf_0    conda-forge
cuda-cudart-dev_linux-64  12.9.79              h3f2d84a_0    conda-forge
cuda-cudart-static_linux-64 12.9.79              h3f2d84a_0    conda-forge
cuda-cudart_linux-64      12.9.79              h3f2d84a_0    conda-forge
cuda-cupti                12.9.79              h9ab20c4_0    conda-forge
cuda-cupti-dev            12.9.79              h9ab20c4_0    conda-forge
cuda-nvcc-tools           12.9.86              he02047a_2    conda-forge
cuda-nvrtc                12.9.86              h5888daf_0    conda-forge
cuda-nvtx                 12.9.79              h5888daf_0    conda-forge
cuda-nvvm-tools           12.9.86              h4bc722e_2    conda-forge
cuda-version              12.9                 h4f385c5_3    conda-forge
cudnn                     9.11.0.98            hbcb9cd8_0    conda-forge
importlib-metadata        8.7.0              pyhe01879c_1    conda-forge
jax                       0.7.0              pyhd8ed1ab_0    conda-forge
jax-cuda12-pjrt           0.7.0                    pypi_0    pypi
jax-cuda12-plugin         0.7.0                    pypi_0    pypi
jaxlib                    0.7.0           cuda129py313h2550629_200    conda-forge
ld_impl_linux-64          2.44                 h1423503_1    conda-forge
libabseil                 20250127.1      cxx17_hbbce691_0    conda-forge
libblas                   3.9.0           34_h59b9bed_openblas    conda-forge
libcblas                  3.9.0           34_he106b2a_openblas    conda-forge
libcublas                 12.9.1.4             h9ab20c4_0    conda-forge
libcublas-dev             12.9.1.4             h9ab20c4_0    conda-forge
libcudnn                  9.11.0.98            hf7e9902_0    conda-forge
libcudnn-dev              9.11.0.98            h58dd1b1_0    conda-forge
libcufft                  11.4.1.4             h5888daf_0    conda-forge
libcufft-dev              11.4.1.4             h5888daf_0    conda-forge
libcurand                 10.3.10.19           h9ab20c4_0    conda-forge
libcurand-dev             10.3.10.19           h9ab20c4_0    conda-forge
libcusolver               11.7.5.82            h9ab20c4_1    conda-forge
libcusolver-dev           11.7.5.82            h9ab20c4_1    conda-forge
libcusparse               12.5.10.65           h5888daf_1    conda-forge
libcusparse-dev           12.5.10.65           h5888daf_1    conda-forge
libexpat                  2.7.1                hecca717_0    conda-forge
libffi                    3.4.6                h2dba641_1    conda-forge
libgcc                    15.1.0               h767d61c_4    conda-forge
libgcc-ng                 15.1.0               h69a702a_4    conda-forge
libgfortran               15.1.0               h69a702a_4    conda-forge
libgfortran5              15.1.0               hcea5267_4    conda-forge
libgomp                   15.1.0               h767d61c_4    conda-forge
libgrpc                   1.71.0               h8e591d7_1    conda-forge
liblapack                 3.9.0           34_h7ac8fdf_openblas    conda-forge
liblzma                   5.8.1                hb9d3cd8_2    conda-forge
libmpdec                  4.0.0                hb9d3cd8_0    conda-forge
libnvjitlink              12.9.86              h5888daf_1    conda-forge
libopenblas               0.3.30          pthreads_h94d23a6_2    conda-forge
libprotobuf               5.29.3               h7460b1f_2    conda-forge
libre2-11                 2025.06.26           hba17884_0    conda-forge
libsqlite                 3.50.4               h0c1763c_0    conda-forge
libstdcxx                 15.1.0               h8f9b012_4    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
ml_dtypes                 0.5.1           py313ha87cce1_0    conda-forge
nccl                      2.27.7.1             h49b9d9a_0    conda-forge
ncurses                   6.5                  h2d0b736_3    conda-forge
numpy                     2.3.2           py313hf6604e3_0    conda-forge
openssl                   3.5.2                h26f9b46_0    conda-forge
opt_einsum                3.4.0              pyhd8ed1ab_1    conda-forge
pip                       25.2               pyh145f28c_0    conda-forge
python                    3.13.5          hec9711d_102_cp313    conda-forge
python_abi                3.13                    8_cp313    conda-forge
re2                       2025.06.26           h9925aae_0    conda-forge
readline                  8.2                  h8c095d6_2    conda-forge
scipy                     1.16.1          py313h3a520b0_0    conda-forge
tk                        8.6.13          noxft_hd72426e_102    conda-forge
tzdata                    2025b                h78e105d_0    conda-forge
zipp                      3.23.0             pyhd8ed1ab_0    conda-forge

Environment info

active environment : test-jax
    active env location : /home/tim/conda/envs/test-jax
            shell level : 2
       user config file : /home/tim/.condarc
 populated config files : /home/bizon/anaconda3/.condarc
                          /home/tim/.condarc
          conda version : 24.7.1
    conda-build version : 24.11.2
         python version : 3.10.12.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen3
                          __conda=24.7.1=0
                          __cuda=12.7=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /home/bizon/anaconda3  (writable)
      conda av data dir : /home/bizon/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/tim/conda/pkgs
                          /home/bizon/anaconda3/pkgs
       envs directories : /home/tim/conda/envs
                          /home/bizon/anaconda3/envs
                          /home/tim/.conda/envs
               platform : linux-64
             user-agent : conda/24.7.1 requests/2.32.3 CPython/3.10.12 Linux/6.5.0-41-generic ubuntu/22.04.5 glibc/2.35 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.8
                UID:GID : 1008:1001
             netrc file : None
           offline mode : False

[h-vetinari]

Can you reproduce the problem without the plugins?

jax                       0.7.0              pyhd8ed1ab_0    conda-forge
jax-cuda12-pjrt           0.7.0                    pypi_0    pypi
jax-cuda12-plugin         0.7.0                    pypi_0    pypi
jaxlib                    0.7.0           cuda129py313h2550629_200    conda-forge

[timpalpant]

If I manually pip uninstall them then the issue doesn't occur, but this removes CUDA support:

$ python -c 'import jax.numpy; jax.numpy.array([1,2,3])'
WARNING:2025-08-18 18:05:24,516:jax._src.xla_bridge:872: An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu.

I believe these are installed by the jaxlib conda package:

jaxlib-feedstock/recipe/build.sh

Lines 143 to 144 in 273ed0d

	unzip $SRC_DIR/dist/jax_cuda*_plugin*.whl
	unzip $SRC_DIR/dist/jax_cuda*_pjrt*.whl

[h-vetinari]

I believe these are installed by the jaxlib conda package:

Even if we unpack them during the build here, they would not show up as separate packages in conda list. Please recreate a fresh environment with only jax and see if the error still occurs.

[traversaro]

I believe these are installed by the jaxlib conda package:

Even if we unpack them during the build here, they would not show up as separate packages in conda list. Please recreate a fresh environment with only jax and see if the error still occurs.

Unfortunately they are indeed listed by conda list, see #293 .

[rHermes]

I also got hit with this exact same error, is this universal for all cuda users, or do we know what triggers this?

[traversaro]

Thanks for opening the issue. I quickly checked with:

[workspace]
name = "jaxlib-cuda-repro"
channels = ["conda-forge"]
platforms = ["linux-64"] # CUDA jaxlib wheels on conda-forge are linux-64

[feature.jax060.system-requirements]
cuda = "12"

[feature.jax060.dependencies]
python = "3.12.*"                # avoid ambiguity warning
jax = "==0.6.0"
jaxlib = { version = "==0.6.0", build = "*cuda*" }

[feature.jax062-200.system-requirements]
cuda = "12"

[feature.jax062-200.dependencies]
python = "3.12.*"                # avoid ambiguity warning
jax = "==0.6.2"
jaxlib = { version = "==0.6.2", build = "*cuda*_200" }


[feature.jax062-201.system-requirements]
cuda = "12"

[feature.jax062-201.dependencies]
python = "3.12.*"                # avoid ambiguity warning
jax = "==0.6.2"
jaxlib = { version = "==0.6.2", build = "*cuda*_201" }

[feature.jax070.system-requirements]
cuda = "12"

[feature.jax070.dependencies]
python = "3.12.*"
jax = "==0.7.0"
jaxlib = { version = "==0.7.0", build = "*cuda*" }

[environments]
jax060 = ["jax060"]
jax062-200 = ["jax062-200"]
jax062-201 = ["jax062-201"]
jax070 = ["jax070"]

[tasks]
repro = "python -c 'import jax.numpy; jax.numpy.array([1,2,3])'"

and the last working version is 0.6.0, while the first failing version is 0.6.2 with build number 200 (that is the first cuda build) so this is a 0.6.2 regression, either in some dependency, in packaging (#315) or in jax itself (jax-ml/jax@jax-v0.6.0...jax-v0.6.2).

[traversaro]

xref conda-forge/pymbar-feedstock#48 (comment) fyi @mikemhenry

[traversaro]

Interestingly, there are really few occurences of coordination_agent_recoverable in general. I have no hits on google, and also on GitHub the only occurences are either XLA source code or issue in downstream repositories that are actually due to this issue: https://github.com/search?q=coordination_agent_recoverable .

[traversaro]

Ok, I think I fully understand the problem now (all scripts related: https://github.com/traversaro/jaxlib-feedstock-320-debug, related ChatGPT discussion: https://chatgpt.com/share/68b6f69b-dd68-8006-80c4-b8f72b8e284b).

In a nutshell, the problem can be reproduced also by loading directly with dlopen the $CONDA_PREFIX/lib/python3.12/site-packages/jaxlib/libjax_common.so and $CONDA_PREFIX/lib/python3.12/site-packages/jax_plugins/xla_cuda12/xla_cuda_plugin.so see:

       traversaro@IITBMP014LW012:~/pixiws/jaxlib320$ pixi run -e jax070 simple_repro
✨ Pixi task (simple_repro in jax070): python hardcoded_dlopen.py                                                       dlopen -> /home/traversaro/pixiws/jaxlib320/.pixi/envs/jax070/lib/python3.12/site-packages/jaxlib/libjax_common.so
  success
dlopen -> /home/traversaro/pixiws/jaxlib320/.pixi/envs/jax070/lib/python3.12/site-packages/jax_plugins/xla_cuda12/xla_cuda_plugin.so
ERROR: Flag 'coordination_agent_recoverable' was defined more than once but with differing types. Defined in files 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc' and 'external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc'.

Why this happens? It happens as there is a duplicate symbol between libjax_common.so and xla_cuda_plugin.so, since 0.6.2 :

traversaro@IITBMP014LW012:~/pixiws/jaxlib320$ pixi run -e jax060 print_symbols
✨ Pixi task (print_symbols_jax_common in jax060): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jaxlib/libjax_common.so | grep coordination_agent_recoverable                                                                               000000000d01e100 d FLAGS_coordination_agent_recoverable
000000000d0f4660 b FLAGS_nocoordination_agent_recoverable
                                                                                                                        ✨ Pixi task (print_symbols_xla_cuda_plugin in jax060): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jax_plugins/xla_cuda12/xla_cuda_plugin.so | grep coordination_agent_recoverable                                                        traversaro@IITBMP014LW012:~/pixiws/jaxlib320$ pixi run -e jax070 print_symbols
✨ Pixi task (print_symbols_jax_common in jax070): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jaxlib/libjax_common.so | grep coordination_agent_recoverable                                                                               000000000eabbc60 d FLAGS_coordination_agent_recoverable
000000000eb78bd0 b FLAGS_nocoordination_agent_recoverable
                                                                                                                        ✨ Pixi task (print_symbols_xla_cuda_plugin in jax070): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jax_plugins/xla_cuda12/xla_cuda_plugin.so | grep coordination_agent_recoverable                                                        000000001036f720 d FLAGS_coordination_agent_recoverable
00000000103c0fc0 b FLAGS_nocoordination_agent_recoverable                                     

Interestingly, the duplicate symbols since 0.6.1 are also on PyPI:

traversaro@IITBMP014LW012:~/pixiws/jaxlib320$ pixi run -e jax061pypi print_symbols
✨ Pixi task (print_symbols_jax_common in jax061pypi): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jaxlib/libjax_common.so | grep coordination_agent_recoverable                                                                           000000000ced0f88 d FLAGS_coordination_agent_recoverable
                                                                                                                        ✨ Pixi task (print_symbols_xla_cuda_plugin in jax061pypi): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jax_plugins/xla_cuda12/xla_cuda_plugin.so | grep coordination_agent_recoverable                                                    traversaro@IITBMP014LW012:~/pixiws/jaxlib320$ pixi run -e jax062pypi print_symbols
✨ Pixi task (print_symbols_jax_common in jax062pypi): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jaxlib/libjax_common.so | grep coordination_agent_recoverable                                                                           000000000d1d0a40 d FLAGS_coordination_agent_recoverable
                                                                                                                        ✨ Pixi task (print_symbols_xla_cuda_plugin in jax062pypi): nm -s $CONDA_PREFIX/lib/python3.12/site-packages/jax_plugins/xla_cuda12/xla_cuda_plugin.so | grep coordination_agent_recoverable                                                    000000000f11c370 d FLAGS_coordination_agent_recoverable

So, I had two questions:

• Q1 : What changed in 0.6.2 so that FLAGS_coordination_agent_recoverable started being included also in xla_cuda_plugin.so ?
• Q2: Why the duplicate symbol results in error in conda-forge, while no error in PyPI?

The answers are:

• A1: The FLAGS_coordination_agent_recoverable symbol is defined in https://github.com/openxla/xla/blob/ef1eec1f7cb52e7a01f10bc64feb261672bc6f6e/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc#L60, that is part of the coordination_service_agent bazel target. That target was not included in xla_cuda_plugin.so, but was then included (transitively) by the XLA change Abort NCCL collectives on machine failure. openxla/xla#26427, that was then included in jax 0.6.2 .
• A2: In PyPI, libjax_common.so and xla_cuda_plugin.so have each one an internal statically linked libaseil, so the each one have its own registry flag, so the duplicate symbol does not result in any problem (see https://github.com/abseil/abseil-cpp/blob/d5ccaa212c8257b066c9910b13b879754df9dd9e/absl/flags/reflection.cc#L133-L169). In conda-forge, as abseil is shared, the duplicate flag trigger this problem.

[traversaro]

So, I think the possible solutions are:

• S1: Just remove the coordination_agent_recoverable flag from XLA. I did not find any documentation or reference for it online, so I think we should be safe to just remove it.
• S2: Swich to static abseil, as we switched to static protobuf in Statically link libprotobuf #90 after jaxlib 0.3.0 conda-forge package is broken #89 .

[traversaro]

fyi @conda-forge/jax @conda-forge/jaxlib

[basnijholt]

I should have searched for existing issues but I just spent hours debugging this and it seems like @traversaro pretty much figured it out! 😄

I can confirm this issue affects the latest versions on high-end hardware:

Environment:

• System: Ubuntu 24.04 (Linux 6.8.0)
• GPUs: 8x NVIDIA A100-SXM4-80GB
• Driver: 570.148.08
• Python: 3.12.11
• Package manager: micromamba 2.0.7

Tested versions:

• ✅ JAX 0.6.0 (jaxlib=0.6.0=cuda126py312hbc630d6_200) - Works correctly
• ❌ JAX 0.6.2 - Fails with both:
  • jaxlib=0.6.2=cuda126py312h59a9b8a_200 (CUDA 12.6)
  • jaxlib=0.6.2=cuda129py312h050a67a_201 (CUDA 12.9)
• ❌ JAX 0.7.0 (jaxlib=0.7.0=cuda129py312h050a67a_200) - Same error

Minimal reproduction:

# Create fresh environment
micromamba create -n jax-test python=3.12 -c conda-forge -y
# Install JAX 0.7.0 (latest)
micromamba install -n jax-test jax=0.7.0 jaxlib=0.7.0=cuda129py312* -c conda-forge -y
# Test import
micromamba run -n jax-test python -c "import jax; print(jax.devices())"
# ERROR: Flag 'coordination_agent_recoverable' was defined more than once...

This confirms the issue persists in the latest JAX 0.7.0 release and affects both CUDA 12.6 and 12.9 builds. Currently using JAX 0.6.0 as a workaround.