[BUG] Evo2 doesnt work in Bionemo V3.0.0

[TaoDFang]

BioNeMo Recipes Version

v3.0.0 (6c10c66)

Bug Description

After i build bionemo images and cant run evo2 due to 'causal_conv1d_cuda' and 'Qwen3NextForCausalLM' issue

Steps to Reproduce

~/my_code/bionemo-recipes-3.0.0/recipes/evo2_megatron> podman build -t evo2_megatron_recipe-$(git rev-parse --short HEAD) .
~/my_code/bionemo-recipes-3.0.0> podman run -it --rm --device=nvidia.com/gpu=all --network host --ipc=host -v ${PWD}:/workspace/bio evo2_megatron_recipe-$(git rev-parse --short HEAD) /bin/bash
/workspace/bionemo# evo2_convert_nemo2_to_mbridge -h

Error Messages and Logs

root@daint-ln004:/workspace/bionemo# evo2_convert_nemo2_to_mbridge -h
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
/workspace/.venv/lib/python3.12/site-packages/modelopt/torch/__init__.py:53: UserWarning: transformers>=5.0 support is experimental. Unified Hugging Face checkpoint export for quantized checkpoints may not work for some models yet.
  _warnings.warn(
/workspace/.venv/lib/python3.12/site-packages/megatron/core/models/mimo/config/base_configs.py:12: UserWarning: MimoModelConfig is experimental and still under active development. The API may change without notice in future releases.
  class MimoModelConfig:
Traceback (most recent call last):
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2096, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2290, in _get_module
    raise e
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2288, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/models/qwen3_next/modeling_qwen3_next.py", line 53, in <module>
    from causal_conv1d import causal_conv1d_fn, causal_conv1d_update
  File "/workspace/.venv/lib/python3.12/site-packages/causal_conv1d/__init__.py", line 3, in <module>
    from causal_conv1d.causal_conv1d_interface import causal_conv1d_fn, causal_conv1d_update
  File "/workspace/.venv/lib/python3.12/site-packages/causal_conv1d/causal_conv1d_interface.py", line 6, in <module>
    from causal_conv1d.cpp_functions import causal_conv1d_fwd_function, causal_conv1d_bwd_function, causal_conv1d_update_function
  File "/workspace/.venv/lib/python3.12/site-packages/causal_conv1d/cpp_functions.py", line 5, in <module>
    import causal_conv1d_cuda
ModuleNotFoundError: No module named 'causal_conv1d_cuda'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/.venv/bin/evo2_convert_nemo2_to_mbridge", line 4, in <module>
    from bionemo.evo2.utils.checkpoint.nemo2_to_mbridge import main
  File "/workspace/bionemo/src/bionemo/evo2/utils/checkpoint/nemo2_to_mbridge.py", line 24, in <module>
    from megatron.bridge.training.checkpointing import save_tokenizer_assets
  File "/workspace/.venv/lib/python3.12/site-packages/megatron/bridge/__init__.py", line 16, in <module>
    from megatron.bridge.models.conversion.auto_bridge import AutoBridge
  File "/workspace/.venv/lib/python3.12/site-packages/megatron/bridge/models/__init__.py", line 80, in <module>
    from megatron.bridge.models.mimo.mimo_bridge import MimoBridge
  File "/workspace/.venv/lib/python3.12/site-packages/megatron/bridge/models/mimo/mimo_bridge.py", line 27, in <module>
    from megatron.bridge.models.qwen.qwen2_bridge import Qwen2Bridge
  File "/workspace/.venv/lib/python3.12/site-packages/megatron/bridge/models/qwen/__init__.py", line 18, in <module>
    from megatron.bridge.models.qwen.qwen3_next_bridge import Qwen3NextBridge  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.12/site-packages/megatron/bridge/models/qwen/qwen3_next_bridge.py", line 20, in <module>
    from transformers import Qwen3NextForCausalLM
  File "/workspace/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2184, in __getattr__
    raise ModuleNotFoundError(
ModuleNotFoundError: Could not import module 'Qwen3NextForCausalLM'. Are this object's requirements defined correctly?

Docker Image

No response

System Information

Environment Details:

• OS: [e.g., Ubuntu 20.04]
• CPU: [e.g., Intel i9-12900K]
• RAM: [e.g., 64GB]

GPU Details:
GPU Model(s): NVIDIA GH200 120GB ×4 (reported by nvidia-smi)
GPU Memory: nvidia-smi reports 97871 MiB (~95.6 GiB) per GPU; device name string shows 120GB
CUDA Version: 13.1 (from nvidia-smi)
CUDA Driver: Driver Version 590.48.01 (from nvidia-smi)
nvcc (CUDA toolkit): nvcc: not found (CUDA toolkit not in PATH)
cuDNN: Not detected (no /usr/include/cudnn_version.h and no libcudnn found)

Additional Context

No response