[BUG] DataLoader worker exited unexpectedly after fresh installation and default example

[Bloeci]

Describe the bug
I downloaded OpenFold, installed it using UV Sync, and downloaded the models. When I run the example, I get an error saying that the data loader failed unexpectedly. I want to use it on an HPC cluster; I'm currently using an NVIDIA A30.

Apart from the cluster, the same happend for me using just a local installation on my machine

To Reproduce
Run command on cluster

srun --partition=paula --time=01:00:00 --gres=gpu:1 --mem=20G \
    run_openfold predict \
        --query-json examples/example_inference_inputs/query_ubiquitin.json \
        --inference-ckpt-path weights/of3-p2-155k.pt

Run command locally

run_openfold predict \
    --query_json=examples/example_inference_inputs/query_ubiquitin.json \
    --inference-ckpt-path weights/of3-p2-155k.pt

Expected behavior
The default output structure for the example.

Stack trace
Stack trace cluster

(openfold3) (base) ↑1 (login02.sc.uni-leipzig.de) mb97hape-software/openfold-3 git:(main) ▶ srun --partition=paula --time=01:00:00 --gres=gpu:1 --mem=20G run_openfold predict --query-json examples/example_inference_inputs/query_ubiquitin.json --inference-ckpt-path cache/weights/of3-p2-155k.pt
srun: job 21044665 queued and waiting for resources
srun: job 21044665 has been allocated resources
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:10 remaining: 00:00]
/work2/mb97hape-software/openfold-3/openfold3/core/data/tools/colabfold_msa_server.py:335: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  tar_gz.extractall(path)
No complexes found for paired MSA generation. Skipping...
/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=48523) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|██████████| 1/1 [00:18<00:00, 18.13s/it]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
SLURM auto-requeueing enabled. Setting signal handlers.
/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:424: UserWarning: This DataLoader will create 10 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  self.check_worker_number_rationality()
/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1310, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
           ^^^^^^^^^^^^^^^^^^^
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 1136, in wait
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 49238) is killed by signal: Segmentation fault. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/work2/mb97hape-software/openfold-3/.venv/bin/run_openfold", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/openfold3/run_openfold.py", line 195, in predict
    expt_runner.run(query_set)
  File "/work2/mb97hape-software/openfold-3/openfold3/entry_points/experiment_runner.py", line 696, in run
    self.trainer.predict(
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 941, in predict
    return call._call_and_handle_interrupt(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/call.py", line 49, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path, weights_only=weights_only)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 1079, in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py", line 1118, in _run_stage
    return self.predict_loop.run()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/loops/utilities.py", line 179, in _decorator
    return loop_run(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/loops/prediction_loop.py", line 122, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
                                       ^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fetchers.py", line 134, in __next__
    batch = super().__next__()
            ^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fetchers.py", line 61, in __next__
    batch = next(self.iterator)
            ^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
          ^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/combined_loader.py", line 142, in __next__
    out = next(self.iterators[0])
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 741, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1524, in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1483, in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1323, in _try_get_data
    raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 49238) exited unexpectedly
Predicting: |          | 0/? [00:07<?, ?it/s]srun: error: paula08: task 0: Exited with exit code 1

Stack trace local

(openfold3) (base) Ξ software/openfold-3 git:(main) ▶ run_openfold predict --query_json=examples/example_inference_inputs/query_ubiquitin.json --inference-ckpt-path weights/of3-p2-155k.pt 
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Mapping file /tmp/of3_colabfold_msas/mappings/seq_to_rep_id.json already exists. Appending new sequences.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Mapping file /tmp/of3_colabfold_msas/mappings/rep_id_to_seq.json already exists. Appending new sequences.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
No complexes found for paired MSA generation. Skipping...
/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=2591200) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 181.53it/s]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting: |                                                                                                             | 0/? [00:00<?, ?it/s]ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1275, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/queues.py", line 111, in get
    if not self._poll(timeout):
           ~~~~~~~~~~^^^^^^^^^
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 1148, in wait
    ready = selector.select(timeout)
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/selectors.py", line 398, in select
    fd_event_list = self._selector.poll(timeout)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: DataLoader worker (pid 2591499) is killed by signal: Segmentation fault. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/data/software/openfold-3/.venv/bin/run_openfold", line 10, in <module>
    sys.exit(cli())
             ~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
  File "/media/data/software/openfold-3/openfold3/run_openfold.py", line 195, in predict
    expt_runner.run(query_set)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/media/data/software/openfold-3/openfold3/entry_points/experiment_runner.py", line 696, in run
    self.trainer.predict(
    ~~~~~~~~~~~~~~~~~~~~^
        model=self.lightning_module,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        datamodule=self.lightning_data_module,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_predictions=False,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 941, in predict
    return call._call_and_handle_interrupt(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self,
        ^^^^^
    ...<6 lines>...
        weights_only,
        ^^^^^^^^^^^^^
    )
    ^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/call.py", line 49, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path, weights_only=weights_only)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 1079, in _run
    results = self._run_stage()
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 1118, in _run_stage
    return self.predict_loop.run()
           ~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/utilities.py", line 179, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/prediction_loop.py", line 122, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
                                       ~~~~^^^^^^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/fetchers.py", line 134, in __next__
    batch = super().__next__()
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/fetchers.py", line 61, in __next__
    batch = next(self.iterator)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/utilities/combined_loader.py", line 142, in __next__
    out = next(self.iterators[0])
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 732, in __next__
    data = self._next_data()
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1482, in _next_data
    idx, data = self._get_data()
                ~~~~~~~~~~~~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1444, in _get_data
    success, data = self._try_get_data()
                    ~~~~~~~~~~~~~~~~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1288, in _try_get_data
    raise RuntimeError(
        f"DataLoader worker (pid(s) {pids_str}) exited unexpectedly"
    ) from e
RuntimeError: DataLoader worker (pid(s) 25914

Configuration (please complete the following information):
Installation

uv sync

Set up env variables and load modules

module load CUDA/12.8.0
export TRITON_CACHE_DIR=/work2/mb97hape-software/openfold-3/cache/triton
export OPENFOLD_CACHE=/work2/mb97hape-software/openfold-3/cache

OS

(openfold3) (base) Ξ (login02.sc.uni-leipzig.de) mb97hape-software/openfold-3 git:(main) ▶ head /etc/os-release 
NAME="Rocky Linux"
VERSION="9.7 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.7"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.7 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"

[jandom]

Hi there @Bloeci – thanks for the bug report, this is certainly not the expected behavior on UBQ ;-) Kudos for the bug report, it's certainly one of the best written bug reports.

Two quick questions – did you install the pip package, or are you running this from a specific commit on the main branch?

[jandom]

Good news/bad news, this works for me – we need more details to reproduce

Details

run_openfold predict \
    --query_json=examples/example_inference_inputs/query_ubiquitin.json \
    --inference-ckpt-path ~/.openfold3/of3-p2-155k.pt 
/home/jandom/workspace/openfold-3/.pixi/envs/openfold3-cuda13-pypi/lib/python3.13/site-packages/torch/cuda/__init__.py:435: UserWarning: 
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)
    
  queued_call()
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
COMPLETE: 100%|██████████████████████████████████████████████████| 150/150 [elapsed: 00:03 remaining: 00:00]
/home/jandom/workspace/openfold-3/openfold3/core/data/tools/colabfold_msa_server.py:335: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  tar_gz.extractall(path)
No complexes found for paired MSA generation. Skipping...
/home/jandom/workspace/openfold-3/.pixi/envs/openfold3-cuda13-pypi/lib/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=50442) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|████████████████████████████████████████████████| 1/1 [00:13<00:00, 13.29s/it]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/home/jandom/workspace/openfold-3/.pixi/envs/openfold3-cuda13-pypi/lib/python3.13/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
/home/jandom/workspace/openfold-3/.pixi/envs/openfold3-cuda13-pypi/lib/python3.13/site-packages/torch/cuda/__init__.py:435: UserWarning: 
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)
    
  queued_call()
Predicting DataLoader 0:   0%|                                                        | 0/1 [00:00<?, ?it/s]Seed set to 42
/home/jandom/workspace/openfold-3/.pixi/envs/openfold3-cuda13-pypi/lib/python3.13/site-packages/biotite/structure/io/pdbx/convert.py:912: DeprecationWarning: `include_bonds` parameter is deprecated, intra-residue are always written, if available
  warnings.warn(
Predicting DataLoader 0: 100%|████████████████████████████████████████████████| 1/1 [00:08<00:00,  0.12it/s]
==================================================
    PREDICTION SUMMARY (COMPLETE)    
==================================================
Total Queries Processed: 1
  - Successful Queries:  1
  - Failed Queries:      0
==================================================

Predicting DataLoader 0: 100%|████████████████████████████████████████████████| 1/1 [00:08<00:00,  0.12it/s]
Removing empty log directory...
Cleaning up MSA directories...

Shot-in-the-dark: can you try the following runner YAML

# runner.yaml
model_update:
  presets: 
    - predict
    - low_mem

data_module_args: {num_workers: 1} 

You can call it like so

run_openfold predict     --query_json=examples/example_inference_inputs/query_ubiquitin.json     --inference-ckpt-path ~/.openfold3/of3-p2-155k.pt --runner_yaml runner.yaml

[Bloeci]

Thank you very much for the quick response!

Regarding your question: I did not install the pip package, I installed it from source. I also don't change the branch. The only commands I executed (for completeness) were (in order, apart from the predict command)

uv sync
source .venv/bin/activate
setup_openfold
uv pip install -e .

The environment variables were then set as mentioned in the first post above.

---

For the second part, thanks for pointing out the num_worker section. I've seen in the cluster stack trace the line

/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:424: \
UserWarning: This DataLoader will create 10 worker processes in total. \ 
Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to \
create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, \
lower the worker number to avoid potential slowness/freeze if necessary.

but (unfortunately) I haven’t done much research to look for the settings in the openfold package. So thank you for pointing out the data_module_args section. I was confused because this line does not appear in the local run trace. Unfortunately, this also did not work on my system - I just tested it locally for now.

Run settings

yaml file

(openfold3) (base) ↑1 software/openfold-3 git:(main) ▶ cat examples/example_runner_yamls/low_mem.yml 
# Model changes for low memory 
model_update:
  presets: 
    - predict
    - low_mem
    
data_module_args: {num_workers: 1}

Command

run_openfold predict \
    --query_json=examples/example_inference_inputs/query_ubiquitin.json \
    --inference-ckpt-path weights/of3-p2-155k.pt \
    --runner_yaml examples/example_runner_yamls/low_mem.yml 

Stack trace

(openfold3) (base) ↑1 software/openfold-3 git:(main) ▶ run_openfold predict --query_json=examples/example_inference_inputs/query_ubiquitin.json --inference-ckpt-path weights/of3-p2-155k.pt --runner_yaml examples/example_runner_yamls/low_mem.yml 
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Mapping file /tmp/of3_colabfold_msas/mappings/seq_to_rep_id.json already exists. Appending new sequences.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Mapping file /tmp/of3_colabfold_msas/mappings/rep_id_to_seq.json already exists. Appending new sequences.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
No complexes found for paired MSA generation. Skipping...
/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=2652176) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 218.11it/s]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'predict_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=23` in the `DataLoader` to improve performance.
Predicting: |                                                                                                                                                                          | 0/? [00:00<?, ?it/s]ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1275, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/queues.py", line 111, in get
    if not self._poll(timeout):
           ~~~~~~~~~~^^^^^^^^^
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 1148, in wait
    ready = selector.select(timeout)
  File "/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/selectors.py", line 398, in select
    fd_event_list = self._selector.poll(timeout)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: DataLoader worker (pid 2652326) is killed by signal: Segmentation fault. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/data/software/openfold-3/.venv/bin/run_openfold", line 10, in <module>
    sys.exit(cli())
             ~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
  File "/media/data/software/openfold-3/openfold3/run_openfold.py", line 195, in predict
    expt_runner.run(query_set)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/media/data/software/openfold-3/openfold3/entry_points/experiment_runner.py", line 696, in run
    self.trainer.predict(
    ~~~~~~~~~~~~~~~~~~~~^
        model=self.lightning_module,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        datamodule=self.lightning_data_module,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_predictions=False,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 941, in predict
    return call._call_and_handle_interrupt(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self,
        ^^^^^
    ...<6 lines>...
        weights_only,
        ^^^^^^^^^^^^^
    )
    ^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/call.py", line 49, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path, weights_only=weights_only)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 1079, in _run
    results = self._run_stage()
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/trainer.py", line 1118, in _run_stage
    return self.predict_loop.run()
           ~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/utilities.py", line 179, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/prediction_loop.py", line 122, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
                                       ~~~~^^^^^^^^^^^^^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/fetchers.py", line 134, in __next__
    batch = super().__next__()
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/loops/fetchers.py", line 61, in __next__
    batch = next(self.iterator)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/utilities/combined_loader.py", line 142, in __next__
    out = next(self.iterators[0])
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 732, in __next__
    data = self._next_data()
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1482, in _next_data
    idx, data = self._get_data()
                ~~~~~~~~~~~~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1444, in _get_data
    success, data = self._try_get_data()
                    ~~~~~~~~~~~~~~~~~~^^
  File "/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1288, in _try_get_data
    raise RuntimeError(
        f"DataLoader worker (pid(s) {pids_str}) exited unexpectedly"
    ) from e
RuntimeError: DataLoader worker (pid(s) 2652326) exited unexpectedly
Predicting: |          | 0/? [00:02<?, ?it/s]                                                                                                                                                                

System settings
Memory

(base) Ξ software/openfold-3 git:(main) ▶ free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        11Gi       8.6Gi       909Mi        42Gi        51Gi
Swap:          2.0Gi       1.6Gi       386Mi

CPU info

(openfold3) (base) Ξ software/openfold-3 git:(main) ▶ cat /proc/cpuinfo                                               
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 25
model		: 33
model name	: AMD Ryzen 9 5900X 12-Core Processor
stepping	: 0
microcode	: 0xa201016
cpu MHz		: 2872.696
cache size	: 512 KB
physical id	: 0
siblings	: 24
core id		: 0
cpu cores	: 12
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 16
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap ibpb_exit_to_user
bugs		: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso ibpb_no_ret tsa vmscape
bogomips	: 7386.03
TLB size	: 2560 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

GPU

(base) Ξ software/openfold-3 git:(main) ▶ nvidia-smi
Fri Mar 27 13:16:58 2026       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.288.01             Driver Version: 535.288.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3070        Off | 00000000:08:00.0  On |                  N/A |
| 39%   48C    P3              35W / 270W |   1237MiB /  8192MiB |     17%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2041      G   /usr/lib/xorg/Xorg                          598MiB |
|    0   N/A  N/A      2225      G   /usr/bin/gnome-shell                         95MiB |
|    0   N/A  N/A      5460      G   /usr/lib/thunderbird/thunderbird            164MiB |
|    0   N/A  N/A   2358304      G   /opt/zoom/zoom                               77MiB |
|    0   N/A  N/A   2358318      G   ...495da0-2169-484c-90a8-12199f6d30ce}        3MiB |
|    0   N/A  N/A   2446294      G   ...irefox/7967/usr/lib/firefox/firefox      215MiB |
|    0   N/A  N/A   2642255      G   /home/iwe34/software/pymol/bin/python        21MiB |
+---------------------------------------------------------------------------------------+

OS

(base) Ξ software/openfold-3 git:(main) ▶ head /etc/os-release                                                                     
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

[jandom]

I'll try with uv next, thanks – my only worry is that some of the packages will not be present for the platform/arch of my test box.

Update yeah this won't fly

uv sync
Using CPython 3.11.15 interpreter at: /home/jandom/micromamba/envs/openfold-research/bin/python3
Creating virtual environment at: .venv
Resolved 165 packages in 2.09s
error: Distribution `mkl==2025.3.1 @ registry+https://pypi.org/simple` can't be installed because it doesn't have a source distribution or wheel for the current platform

hint: You're on Linux (`manylinux_2_39_aarch64`), but `mkl` (v2025.3.1) only has wheels for the following platforms: `manylinux_2_28_x86_64`, `win_amd64`; consider adding "sys_platform == 'linux' and platform_machine == 'aarch64'" to `tool.uv.required-environments` to ensure uv resolves to a version with compatible wheels

Let me think what'd be best next, i could try a cloud instance (?). Your setup is completely sane/standard, there is nothing that should cause what you're seeing.

I hate to ask – could you try either the A) conda build B) the docker image and see if these are any different? It'd potentially point to some silently updated dep that's crashing us.

[Bloeci]

Yeah, sure, I will try the Docker image first, because I’ve been trying for quite some time to do everything using either UV or Docker, so I don’t have to use Conda (anymore). I’ll post another update or drop you a line in this thread.

Let’s see what happens :)

[jandom]

Thanks and sorry about this – we're spending a lot of time on the build in Q1/Q2, so things will get better

[Bloeci]

No, thank you, first of all, for making this software and tool public so that the community can use it extensively. We’ve probably all been working in the field of protein design for a while now (with varying degrees of success, haha), and it’s always nice to see a community release a new tool that’s completely open source and ready to use. I’m looking forward to every new update and can’t wait to see what’s coming next.

---

However, short info, for other users may be reading this. There is another Docker.md in the docker folder :D (deleted my wrong suggestiongs for updating docs). Also you need a HuggigFace token as mentioned in the docs. The default command https://github.com/aqlaboratory/openfold-3/blob/main/docker/DOCKER.md#development-images therefore (because it uses the affinity image) also needs a HF token..

(base) Ξ software/openfold-3 git:(main) ▶ docker build -t openfold3:v1 -f docker/Dockerfile .
[...]
1.539 Still waiting to acquire lock on /opt/aqaffinity/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)                                                                                       
1.540 Still waiting to acquire lock on /opt/aqaffinity/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)                                                                                       
Fetching 48 files:   0%|          | 0/48 [00:00<?, ?it/s]
1.952 Error: Access denied. This repository requires approval

Stack trace 2 (docker build from main)

(base) Ξ software/openfold-3 git:(main) ▶ docker build -t openfold3:v1 -f docker/Dockerfile .
[+] Building 507.2s (28/29)                                                                                                                                                             docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                              0.1s
 => => transferring dockerfile: 5.71kB                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04                                                                                                            0.4s
 => [internal] load .dockerignore                                                                                                                                                                 0.1s
 => => transferring context: 273B                                                                                                                                                                 0.0s
 => [builder  1/13] FROM docker.io/nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04@sha256:21196d81f56b48dbee70494d5f10322e1a77cc47ffe202a3bf68eab81533c20f                                           73.8s
 => => resolve docker.io/nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04@sha256:21196d81f56b48dbee70494d5f10322e1a77cc47ffe202a3bf68eab81533c20f                                                      0.0s
 => => sha256:21196d81f56b48dbee70494d5f10322e1a77cc47ffe202a3bf68eab81533c20f 743B / 743B                                                                                                        0.0s
 => => sha256:cc55d151af1e8e083f3210af753a5cfbcbc5455421531eb0459887026bb4699f 2.84kB / 2.84kB                                                                                                    0.0s
 => => sha256:d3262c91d05f9c01fe802bf2c67321ef4c8281f25d97c8a5c875f7071e0e9916 18.70kB / 18.70kB                                                                                                  0.0s
 => => sha256:4bc954eb910af405a9ee95a0d504f14370aa97e028f19f50779604c01b4ea00b 1.47GB / 1.47GB                                                                                                   65.4s
 => => extracting sha256:4bc954eb910af405a9ee95a0d504f14370aa97e028f19f50779604c01b4ea00b                                                                                                         8.1s
 => [internal] load build context                                                                                                                                                                 0.3s
 => => transferring context: 58.42MB                                                                                                                                                              0.3s
 => [builder  2/13] RUN apt-get update && apt-get install -y     wget     libopenmpi-dev     libaio-dev     git     python3-dbg     build-essential     ninja-build     && rm -rf /var/lib/apt/  85.3s
 => [devel 2/8] RUN apt-get update && apt-get install -y     libopenmpi3     libaio1     libaio-dev     && rm -rf /var/lib/apt/lists/*     && apt-get clean                                      25.8s
 => [devel 3/8] RUN rm -rf /usr/local/cuda/doc     && rm -rf /usr/local/cuda/extras     && rm -rf /usr/local/cuda/samples     && rm -rf /usr/local/cuda/src     && rm -rf /usr/local/cuda/nsight  1.0s
 => [builder  3/13] RUN wget -P /tmp     "https://github.com/conda-forge/miniforge/releases/download/25.11.0-1/Miniforge3-Linux-x86_64.sh"     && bash /tmp/Miniforge3-Linux-x86_64.sh -b -p /op  5.8s 
 => [builder  4/13] COPY environments/production-linux-64.lock /opt/openfold3/production-linux-64.lock                                                                                            0.2s 
 => [builder  5/13] COPY environments/production-linux-64.yml /opt/openfold3/production-linux-64.yml                                                                                              0.1s 
 => [builder  6/13] RUN mamba install -n base -c conda-forge conda-lock --yes     && if [ "yaml" = "lock" ]; then          conda-lock install --name openfold3 /opt/openfold3/production-linux  280.4s 
 => [builder  7/13] COPY pyproject.toml /opt/openfold3/                                                                                                                                           0.2s 
 => [builder  8/13] COPY openfold3/__init__.py /opt/openfold3/openfold3/                                                                                                                          0.1s 
 => [builder  9/13] COPY scripts/ /opt/openfold3/scripts/                                                                                                                                         0.2s 
 => [builder 10/13] WORKDIR /opt/                                                                                                                                                                 0.1s 
 => [builder 11/13] RUN /opt/openfold3/scripts/install_third_party_dependencies.sh                                                                                                                3.9s 
 => [builder 12/13] WORKDIR /opt/openfold3                                                                                                                                                        0.1s 
 => [builder 13/13] RUN if [ "false" = "true" ]; then         uv pip install --no-build-isolation --editable ".[cuequivariance]";     else         uv pip install --no-deps --editable ".";       1.4s 
 => [devel 4/8] COPY --from=builder /opt/conda /opt/conda                                                                                                                                        25.9s 
 => [devel 5/8] COPY --from=builder /opt/cutlass /opt/cutlass                                                                                                                                     1.2s 
 => [devel 6/8] RUN /opt/conda/bin/conda init bash     && echo "conda activate openfold3" >> /root/.bashrc                                                                                        0.5s 
 => [devel 7/8] COPY . /opt/openfold3                                                                                                                                                             0.3s 
 => [devel 8/8] WORKDIR /opt/openfold3                                                                                                                                                            0.1s 
 => [affinity 1/5] WORKDIR /opt/openfold3                                                                                                                                                         0.1s 
 => [affinity 2/5] RUN uv pip install --no-deps .                                                                                                                                                 2.6s 
 => [affinity 3/5] RUN uv pip install huggingface_hub                                                                                                                                             1.1s 
 => ERROR [affinity 4/5] RUN --mount=type=secret,id=hf_token     HF_TOKEN=$(cat /run/secrets/hf_token)     hf download SandboxAQ/aqaffinity     --local-dir /opt/aqaffinity                       2.1s 
------                                                                                                                                                                                                 
 > [affinity 4/5] RUN --mount=type=secret,id=hf_token     HF_TOKEN=$(cat /run/secrets/hf_token)     hf download SandboxAQ/aqaffinity     --local-dir /opt/aqaffinity:                                  
0.287 cat: /run/secrets/hf_token: No such file or directory                                                                                                                                            
Fetching 48 files:   0%|          | 0/48 [00:00<?, ?it/s]Still waiting to acquire lock on /opt/aqaffinity/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)                                    
1.539 Still waiting to acquire lock on /opt/aqaffinity/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)                                                                                       
1.540 Still waiting to acquire lock on /opt/aqaffinity/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)                                                                                       
Fetching 48 files:   0%|          | 0/48 [00:00<?, ?it/s]
1.952 Error: Access denied. This repository requires approval.
1.952 Set HF_DEBUG=1 as environment variable for full traceback.
------
Dockerfile:151
--------------------
 150 |     
 151 | >>> RUN --mount=type=secret,id=hf_token \
 152 | >>>     HF_TOKEN=$(cat /run/secrets/hf_token) \
 153 | >>>     hf download SandboxAQ/aqaffinity \
 154 | >>>     --local-dir /opt/aqaffinity
 155 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c HF_TOKEN=$(cat /run/secrets/hf_token)     hf download SandboxAQ/aqaffinity     --local-dir /opt/aqaffinity" did not complete successfully: exit code: 1

[jnwei]

Thank you for the kind words @Bloeci :)

Regarding docker installation: we have a separate set of instructions for pulling just the OpenFold3 docker image here. In short, could you try docker pull openfoldconsortium/openfold3:stable

To help further debug the issue:

• When you ran setup_openfold did you happen to run the integration tests there? It would be the last prompt from the menu. If you did run the ingeration tests, did those pass?
• Can you attempt to run with data_workers=0, to remove multiprocessing entirely? E.g.

model_update:
  presets: 
    - predict
    - low_mem
    
data_module_args: {num_workers: 0}

[Bloeci]

Hello hello @jnwei ,

Okay, I'll break down the answer into several sections based on the different approaches.

Docker / Singularity run

TLDR

This approach works for the ubiquitin example, but only (for Docker, e.g.) when I set the shared memory size to a decent amount (here, 8GB). My only concern (or perhaps I'm reading too much into the results) is that the confidence in the ubiquitin output structure is much lower than I would have expected, but that's an issue for another time—I just wanted to mention it here.

File: ubiquitin_seed_42_sample_1_confidences_aggregated.json

{
    "avg_plddt": 80.06190490722656,
    "gpde": 0.7111918330192566,
    "iptm": 0.0,
    "ptm": 0.6701362133026123,
    "disorder": 0.0,
    "has_clash": 0.0,
    "sample_ranking_score": 0.13402724266052246,
    "chain_ptm": {
        "A": 0.6701362133026123
    },
    "chain_pair_iptm": {},
    "bespoke_iptm": {}
}

To reproduce

Docker command

docker pull openfoldconsortium/openfold3:stable
mkdir TEST_OUTPUT
docker run \
    -v /home/iwe34/software/openfold-3/weights:/root/.openfold3/ \
    -v /home/iwe34/software/openfold-3/TEST_OUTPUT:/opt/openfold3/output \
    --shm-size=8gb --gpus all --rm \
    openfoldconsortium/openfold3:stable \
    run_openfold predict \
    --query-json examples/example_inference_inputs/query_ubiquitin.json --output-dir output

Singularity

singularity pull docker://openfoldconsortium/openfold3:stable
srun --partition=paula --time=01:00:00 --gres=gpu:1 --mem=32G --ntasks=1 --cpus-per-task=8 \
    singularity run \
    --env TORCH_EXTENSIONS_DIR="/work2/mb97hape-software/openfold-3/cache" \
    --env TRITON_CACHE_DIR="/work2/mb97hape-software/openfold-3/cache/triton" \
    --nv \
    /work2/mb97hape-software/sif_containers/openfold3_stable.sif \
    run_openfold predict \
    --query-json examples/example_inference_inputs/query_ubiquitin.json \
    --output-dir output \
    --inference-ckpt-path /work2/mb97hape-software/openfold-3/cache/weights/of3-p2-155k.pt

Stack trace

Docker output

(base) Ξ software/openfold-3 git:(main) ▶ docker run -v /home/iwe34/software/openfold-3/weights:/root/.openfold3/ -v /home/iwe34/software/openfold-3/TEST_OUTPUT:/opt/openfold3/output --shm-size=8gb --gpus all --rm openfoldconsortium/openfold3:stable run_openfold predict --query-json examples/example_inference_inputs/query_ubiquitin.json --output-dir output

==========
== CUDA ==
==========

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

df: /root/.triton/autotune: No such file or directory
/opt/openfold3/openfold3/core/utils/checkpoint_loading_utils.py:49: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(ckpt_path)
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:03 remaining: 00:00]
/opt/openfold3/openfold3/core/data/tools/colabfold_msa_server.py:331: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  tar_gz.extractall(path)
No complexes found for paired MSA generation. Skipping...
/opt/conda/envs/openfold3/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=1) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|██████████| 1/1 [00:17<00:00, 17.83s/it]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Seed set to 42
/opt/conda/envs/openfold3/lib/python3.12/site-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/opt/conda/envs/openfold3/lib/python3.12/site-packages/biotite/structure/io/pdbx/convert.py:912: DeprecationWarning: `include_bonds` parameter is deprecated, intra-residue are always written, if available
  warnings.warn(

==================================================
    PREDICTION SUMMARY (COMPLETE)    
==================================================
Total Queries Processed: 1
  - Successful Queries:  1
  - Failed Queries:      0
==================================================

Predicting ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 0:03:36 • 0:00:00 0.00it/s 
Removing empty log directory...
Cleaning up MSA directories...
/opt/conda/envs/openfold3/lib/python3.12/tempfile.py:940: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmphn58qouz'>
  _warnings.warn(warn_message, ResourceWarning)

Singularity output

(base) Ξ (login02.sc.uni-leipzig.de) mb97hape-software/openfold-3 git:(main) ▶ srun --partition=paula --time=01:00:00 --gres=gpu:1 --mem=32G --ntasks=1 --cpus-per-task=8 singularity run --env TORCH_EXTENSIONS_DIR="/work2/mb97hape-software/openfold-3/cache" --env TRITON_CACHE_DIR="/work2/mb97hape-software/openfold-3/cache/triton" --nv /work2/mb97hape-software/sif_containers/openfold3_stable.sif run_openfold predict --query-json examples/example_inference_inputs/query_ubiquitin.json --output-dir output --inference-ckpt-path /work2/mb97hape-software/openfold-3/cache/weights/of3-p2-155k.pt
srun: job 21065591 queued and waiting for resources
srun: job 21065591 has been allocated resources

==========
== CUDA ==
==========

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

/opt/openfold3/openfold3/core/utils/checkpoint_loading_utils.py:49: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(ckpt_path)
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:03 remaining: 00:00]
/opt/openfold3/openfold3/core/data/tools/colabfold_msa_server.py:331: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  tar_gz.extractall(path)
No complexes found for paired MSA generation. Skipping...
/opt/conda/envs/openfold3/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=3376680) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|██████████| 1/1 [00:03<00:00,  3.32s/it]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
SLURM auto-requeueing enabled. Setting signal handlers.
/opt/conda/envs/openfold3/lib/python3.12/site-packages/torch/utils/data/dataloader.py:617: UserWarning: This DataLoader will create 10 worker processes in total. Our suggested max number of worker in current system is 8, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(
[rank: 0] Seed set to 42
/opt/conda/envs/openfold3/lib/python3.12/site-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/opt/conda/envs/openfold3/lib/python3.12/site-packages/biotite/structure/io/pdbx/convert.py:912: DeprecationWarning: `include_bonds` parameter is deprecated, intra-residue are always written, if available
  warnings.warn(

==================================================
    PREDICTION SUMMARY (COMPLETE)    
==================================================
Total Queries Processed: 1
  - Successful Queries:  1
  - Failed Queries:      0
==================================================

Predicting ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 0:00:15 • 0:00:00 0.00it/s 
Cleaning up MSA directories...
/opt/conda/envs/openfold3/lib/python3.12/tempfile.py:940: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmporwm6ewv'>
  _warnings.warn(warn_message, ResourceWarning)

---

Regarding your questions

setup_openfold

After installing pytest in the environment and rerunning the setup, the integration test also failed. I skipped that in the original run, because i thought "we will see" :D. However, apart from some warnings from torch (e.g. 2026-03-30 09:52:27 [ WARNING] No version_tensor is found for this checkpoint. Assuming the user knows checkpoints are parameters are compatible, continuing... (experiment_runner.py:664)) i get the same errors as in the first post in this issue.

Command

(openfold3) (base) Ξ (login01.sc.uni-leipzig.de) mb97hape-software/openfold-3 git:(main) ▶ setup_openfold

Stack trace
(I'm not familiar with good practise for using expandable sections. to make it "easier" to oversee this post, I decided here for a expandable section)

PyTest output

(openfold3) (base) Ξ (login01.sc.uni-leipzig.de) mb97hape-software/openfold-3 git:(main) ▶ setup_openfold       
Setting up OpenFold cache directory...
Please specify the OpenFold cache directory (default: /home/sc.uni-leipzig.de/mb97hape/.openfold3): /work2/mb97hape-software/openfold-3/cache
OpenFold3 parameters may already be installed at: /work2/mb97hape-software/openfold-3/cache/weights
Do you want to:
1) Use existing parameters (skip download)
2) Download to a new location
3) Re-download to existing location
Enter your choice (1/2/3): 1
Using existing parameters at: /work2/mb97hape-software/openfold-3/cache/weights
Starting Biotite CCD setup...
Biotite CCD file at /work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/biotite/structure/info/components.bcif is up-to-date with s3://openfold3-data/components.bcif, skipping.
Run integration tests? (yes/no)yes
Running integration tests...
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.12.11, pytest-9.0.2, pluggy-1.6.0 -- /work2/mb97hape-software/openfold-3/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /work2/mb97hape-software/openfold-3/openfold3
configfile: ../pyproject.toml
collected 2 items                                                                                                                                                                            

openfold3/tests/test_inference_full.py::test_inference_run[query_set0] 

--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
2026-03-30 09:52:27 [ WARNING] No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing... (experiment_runner.py:664)
FAILED                                                                                                                                                                                 [ 50%]
openfold3/tests/test_inference_full.py::test_inference_run[query_set1] 
--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
2026-03-30 09:54:24 [ WARNING] No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing... (experiment_runner.py:664)
FAILED                                                                                                                                                                                 [100%]

========================================================================================== FAILURES ==========================================================================================
_______________________________________________________________________________ test_inference_run[query_set0] _______________________________________________________________________________

self = <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x1534235d9820>, timeout = 5.0

    def _try_get_data(self, timeout=_utils.MP_STATUS_CHECK_INTERVAL):
        # Tries to fetch data from `self._data_queue` once for a given timeout.
        # This can also be used as inner loop of fetching without timeout, with
        # the sender status as the loop condition.
        #
        # This raises a `RuntimeError` if any worker died expectedly. This error
        # can come from either the SIGCHLD handler in `_utils/signal_handling.py`
        # (only for non-Windows platforms), or the manual check below on errors
        # and timeouts.
        #
        # Returns a 2-tuple:
        #   (bool: whether successfully get data, any: data if successful else None)
        try:
>           data = self._data_queue.get(timeout=timeout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1310: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py:113: in get
    if not self._poll(timeout):
           ^^^^^^^^^^^^^^^^^^^
/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py:257: in poll
    return self._poll(timeout)
           ^^^^^^^^^^^^^^^^^^^
/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py:440: in _poll
    r = wait([self], timeout)
        ^^^^^^^^^^^^^^^^^^^^^
/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py:1136: in wait
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/selectors.py:415: in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

signum = 17
frame = <frame at 0x1533f62e4f40, file '/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/selectors.py', line 416, code select>

    def handler(signum, frame) -> None:
        # This following call uses `waitid` with WNOHANG from C side. Therefore,
        # Python can still get and update the process status successfully.
>       _error_if_any_worker_fails()
E       RuntimeError: DataLoader worker (pid 3385908) is killed by signal: Segmentation fault.

.venv/lib/python3.12/site-packages/torch/utils/data/_utils/signal_handling.py:73: RuntimeError

The above exception was the direct cause of the following exception:

tmp_path = PosixPath('/tmp/pytest-of-mb97hape/pytest-0/test_inference_run_query_set0_0')
query_set = InferenceQuerySet(seeds=[42], queries={'query1': Query(query_name='query1', chains=[Chain(molecule_type=<MoleculeType....y_chain_ids=None, sdf_file_path=None)], use_msas=True, use_paired_msas=True, use_main_msas=True, covalent_bonds=None)})

    @skip_unless_cuda_available()
    @pytest.mark.parametrize("query_set", [protein_only_query, protein_and_ligand_query])
    def test_inference_run(tmp_path, query_set):
        # Trigger validation logic to replace the cache path
        with patch("builtins.input", return_value="no"):
            # your test code that calls _maybe_download_parameters
            experiment_config = InferenceExperimentConfig.model_validate({})
        expt_runner = InferenceExperimentRunner(
            experiment_config, num_diffusion_samples=1, output_dir=tmp_path
        )
        try:
            expt_runner.setup()
        except ValueError as e:
            # If called from setup script, fail the test
            if "is not a valid file or directory" in str(e):
                if os.environ.get("OPENFOLD_SETUP_SCRIPT") == "1":
                    pytest.fail(
                        "No checkpoint files found after running setup script. "
                        "Please check that the download completed successfully."
                    )
                else:
                    logger.warning(
                        "No checkpoint files found, skipping for now. "
                        "Please use scripts/setup_openfold3.sh to download the weights."
                    )
                    pytest.skip("No checkpoint files available")
            else:
                raise
    
>       expt_runner.run(query_set)

openfold3/tests/test_inference_full.py:108: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
openfold3/entry_points/experiment_runner.py:696: in run
    self.trainer.predict(
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:941: in predict
    return call._call_and_handle_interrupt(
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/call.py:49: in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:990: in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path, weights_only=weights_only)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:1079: in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:1118: in _run_stage
    return self.predict_loop.run()
           ^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/utilities.py:179: in _decorator
    return loop_run(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/prediction_loop.py:122: in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
                                       ^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fetchers.py:134: in __next__
    batch = super().__next__()
            ^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fetchers.py:61: in __next__
    batch = next(self.iterator)
            ^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/combined_loader.py:341: in __next__
    out = next(self._iterator)
          ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/combined_loader.py:142: in __next__
    out = next(self.iterators[0])
          ^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:741: in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1524: in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1483: in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x1534235d9820>, timeout = 5.0

    def _try_get_data(self, timeout=_utils.MP_STATUS_CHECK_INTERVAL):
        # Tries to fetch data from `self._data_queue` once for a given timeout.
        # This can also be used as inner loop of fetching without timeout, with
        # the sender status as the loop condition.
        #
        # This raises a `RuntimeError` if any worker died expectedly. This error
        # can come from either the SIGCHLD handler in `_utils/signal_handling.py`
        # (only for non-Windows platforms), or the manual check below on errors
        # and timeouts.
        #
        # Returns a 2-tuple:
        #   (bool: whether successfully get data, any: data if successful else None)
        try:
            data = self._data_queue.get(timeout=timeout)
            return (True, data)
        except Exception as e:
            # At timeout and error, we manually check whether any worker has
            # failed. Note that this is the only mechanism for Windows to detect
            # worker failures.
            failed_workers = []
            for worker_id, w in enumerate(self._workers):
                if self._workers_status[worker_id] and not w.is_alive():
                    failed_workers.append(w)
                    self._mark_worker_as_unavailable(worker_id)
            if len(failed_workers) > 0:
                pids_str = ", ".join(str(w.pid) for w in failed_workers)
>               raise RuntimeError(
                    f"DataLoader worker (pid(s) {pids_str}) exited unexpectedly"
                ) from e
E               RuntimeError: DataLoader worker (pid(s) 3385908) exited unexpectedly

.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1323: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
Predicting: |          | 0/? [00:00<?, ?it/s]
------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------
No inference_ckpt_path or inference_ckpt_name provided, selecting default checkpoint.
Set diffusion samples to 1
Setting RLIMIT_STACK to -1 (was -1)
Loading weights from /work2/mb97hape-software/openfold-3/cache/weights/of3-p2-155k.pt
No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
Beginning inference prediction
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
Seeding DataModule DatasetMode.prediction generator with 42
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 5 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 5 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
ERROR: Unexpected segmentation fault encountered in worker.

------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------
INFO     openfold3.entry_points.validator:validator.py:434 No inference_ckpt_path or inference_ckpt_name provided, selecting default checkpoint.
INFO     openfold3.entry_points.experiment_runner:experiment_runner.py:596 Set diffusion samples to 1
INFO     /work2/mb97hape-software/openfold-3/openfold3/core/utils/script_utils.py:script_utils.py:53 Setting RLIMIT_STACK to -1 (was -1)
INFO     openfold3.entry_points.experiment_runner:experiment_runner.py:678 Loading weights from /work2/mb97hape-software/openfold-3/cache/weights/of3-p2-155k.pt
WARNING  openfold3.entry_points.experiment_runner:experiment_runner.py:664 No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
INFO     openfold3.entry_points.experiment_runner:experiment_runner.py:695 Beginning inference prediction
INFO     pytorch_lightning.utilities.rank_zero:setup.py:164 GPU available: True (cuda), used: True
INFO     pytorch_lightning.utilities.rank_zero:setup.py:167 TPU available: False, using: 0 TPU cores
INFO     pytorch_lightning.utilities.rank_zero:logger_connector.py:91 💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
INFO     pytorch_lightning.utilities.rank_zero:callback_connector.py:190 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
INFO     pytorch_lightning.accelerators.cuda:cuda.py:61 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
INFO     openfold3.core.data.framework.data_module:data_module.py:425 Seeding DataModule DatasetMode.prediction generator with 42
_______________________________________________________________________________ test_inference_run[query_set1] _______________________________________________________________________________

self = <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x1533f600f8f0>, timeout = 5.0

    def _try_get_data(self, timeout=_utils.MP_STATUS_CHECK_INTERVAL):
        # Tries to fetch data from `self._data_queue` once for a given timeout.
        # This can also be used as inner loop of fetching without timeout, with
        # the sender status as the loop condition.
        #
        # This raises a `RuntimeError` if any worker died expectedly. This error
        # can come from either the SIGCHLD handler in `_utils/signal_handling.py`
        # (only for non-Windows platforms), or the manual check below on errors
        # and timeouts.
        #
        # Returns a 2-tuple:
        #   (bool: whether successfully get data, any: data if successful else None)
        try:
>           data = self._data_queue.get(timeout=timeout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1310: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.queues.Queue object at 0x1534236dfb00>, block = True, timeout = 4.999998750165105

    def get(self, block=True, timeout=None):
        if self._closed:
            raise ValueError(f"Queue {self!r} is closed")
        if block and timeout is None:
            with self._rlock:
                res = self._recv_bytes()
            self._sem.release()
        else:
            if block:
                deadline = time.monotonic() + timeout
            if not self._rlock.acquire(block, timeout):
                raise Empty
            try:
                if block:
                    timeout = deadline - time.monotonic()
                    if not self._poll(timeout):
>                       raise Empty
E                       _queue.Empty

/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py:114: Empty

The above exception was the direct cause of the following exception:

tmp_path = PosixPath('/tmp/pytest-of-mb97hape/pytest-0/test_inference_run_query_set1_0')
query_set = InferenceQuerySet(seeds=[42], queries={'query1': Query(query_name='query1', chains=[Chain(molecule_type=<MoleculeType....y_chain_ids=None, sdf_file_path=None)], use_msas=True, use_paired_msas=True, use_main_msas=True, covalent_bonds=None)})

    @skip_unless_cuda_available()
    @pytest.mark.parametrize("query_set", [protein_only_query, protein_and_ligand_query])
    def test_inference_run(tmp_path, query_set):
        # Trigger validation logic to replace the cache path
        with patch("builtins.input", return_value="no"):
            # your test code that calls _maybe_download_parameters
            experiment_config = InferenceExperimentConfig.model_validate({})
        expt_runner = InferenceExperimentRunner(
            experiment_config, num_diffusion_samples=1, output_dir=tmp_path
        )
        try:
            expt_runner.setup()
        except ValueError as e:
            # If called from setup script, fail the test
            if "is not a valid file or directory" in str(e):
                if os.environ.get("OPENFOLD_SETUP_SCRIPT") == "1":
                    pytest.fail(
                        "No checkpoint files found after running setup script. "
                        "Please check that the download completed successfully."
                    )
                else:
                    logger.warning(
                        "No checkpoint files found, skipping for now. "
                        "Please use scripts/setup_openfold3.sh to download the weights."
                    )
                    pytest.skip("No checkpoint files available")
            else:
                raise
    
>       expt_runner.run(query_set)

openfold3/tests/test_inference_full.py:108: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
openfold3/entry_points/experiment_runner.py:696: in run
    self.trainer.predict(
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:941: in predict
    return call._call_and_handle_interrupt(
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/call.py:49: in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:990: in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path, weights_only=weights_only)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:1079: in _run
    results = self._run_stage()
              ^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/trainer/trainer.py:1118: in _run_stage
    return self.predict_loop.run()
           ^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/utilities.py:179: in _decorator
    return loop_run(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/prediction_loop.py:122: in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
                                       ^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fetchers.py:134: in __next__
    batch = super().__next__()
            ^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/loops/fetchers.py:61: in __next__
    batch = next(self.iterator)
            ^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/combined_loader.py:341: in __next__
    out = next(self._iterator)
          ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/combined_loader.py:142: in __next__
    out = next(self.iterators[0])
          ^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:741: in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1524: in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1483: in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x1533f600f8f0>, timeout = 5.0

    def _try_get_data(self, timeout=_utils.MP_STATUS_CHECK_INTERVAL):
        # Tries to fetch data from `self._data_queue` once for a given timeout.
        # This can also be used as inner loop of fetching without timeout, with
        # the sender status as the loop condition.
        #
        # This raises a `RuntimeError` if any worker died expectedly. This error
        # can come from either the SIGCHLD handler in `_utils/signal_handling.py`
        # (only for non-Windows platforms), or the manual check below on errors
        # and timeouts.
        #
        # Returns a 2-tuple:
        #   (bool: whether successfully get data, any: data if successful else None)
        try:
            data = self._data_queue.get(timeout=timeout)
            return (True, data)
        except Exception as e:
            # At timeout and error, we manually check whether any worker has
            # failed. Note that this is the only mechanism for Windows to detect
            # worker failures.
            failed_workers = []
            for worker_id, w in enumerate(self._workers):
                if self._workers_status[worker_id] and not w.is_alive():
                    failed_workers.append(w)
                    self._mark_worker_as_unavailable(worker_id)
            if len(failed_workers) > 0:
                pids_str = ", ".join(str(w.pid) for w in failed_workers)
>               raise RuntimeError(
                    f"DataLoader worker (pid(s) {pids_str}) exited unexpectedly"
                ) from e
E               RuntimeError: DataLoader worker (pid(s) 3387152) exited unexpectedly

.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1323: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
Predicting: |          | 0/? [00:00<?, ?it/s]
------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------
No inference_ckpt_path or inference_ckpt_name provided, selecting default checkpoint.
Set diffusion samples to 1
Setting RLIMIT_STACK to -1 (was -1)
Loading weights from /work2/mb97hape-software/openfold-3/cache/weights/of3-p2-155k.pt
No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
Beginning inference prediction
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
Seeding DataModule DatasetMode.prediction generator with 42
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 5 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 10 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 8 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 7 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 15 hydrogen atoms.
Removed 13 hydrogen atoms.
Removed 11 hydrogen atoms.
Removed 5 hydrogen atoms.
Removed 9 hydrogen atoms.
Removed 15 hydrogen atoms.
ERROR: Unexpected segmentation fault encountered in worker.

------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------
INFO     openfold3.entry_points.validator:validator.py:434 No inference_ckpt_path or inference_ckpt_name provided, selecting default checkpoint.
INFO     openfold3.entry_points.experiment_runner:experiment_runner.py:596 Set diffusion samples to 1
INFO     /work2/mb97hape-software/openfold-3/openfold3/core/utils/script_utils.py:script_utils.py:53 Setting RLIMIT_STACK to -1 (was -1)
INFO     openfold3.entry_points.experiment_runner:experiment_runner.py:678 Loading weights from /work2/mb97hape-software/openfold-3/cache/weights/of3-p2-155k.pt
WARNING  openfold3.entry_points.experiment_runner:experiment_runner.py:664 No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
INFO     openfold3.entry_points.experiment_runner:experiment_runner.py:695 Beginning inference prediction
INFO     pytorch_lightning.utilities.rank_zero:setup.py:164 GPU available: True (cuda), used: True
INFO     pytorch_lightning.utilities.rank_zero:setup.py:167 TPU available: False, using: 0 TPU cores
INFO     pytorch_lightning.utilities.rank_zero:logger_connector.py:91 💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
INFO     pytorch_lightning.utilities.rank_zero:callback_connector.py:190 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
INFO     pytorch_lightning.accelerators.cuda:cuda.py:61 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
INFO     openfold3.core.data.framework.data_module:data_module.py:425 Seeding DataModule DatasetMode.prediction generator with 42
====================================================================================== warnings summary ======================================================================================
tests/test_inference_full.py::test_inference_run[query_set0]
tests/test_inference_full.py::test_inference_run[query_set1]
  /work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/lightning_fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python3 /work2/mb97hape-software/openfold-3/.venv/bin/setup ...

tests/test_inference_full.py::test_inference_run[query_set0]
tests/test_inference_full.py::test_inference_run[query_set1]
  /work2/mb97hape-software/openfold-3/openfold3/core/data/framework/data_module.py:571: UserWarning: Expected MSA file for chain A,B of type PROTEIN in query query1, but no MSA files found. A dummy MSA with only the query sequence will be used for this chain.
    self.inference_config.query_set = augment_main_msa_with_query_sequence(

tests/test_inference_full.py::test_inference_run[query_set0]
tests/test_inference_full.py::test_inference_run[query_set1]
  /work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.

tests/test_inference_full.py: 20 warnings
  /home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=3384283) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ===================================================================================
FAILED openfold3/tests/test_inference_full.py::test_inference_run[query_set0] - RuntimeError: DataLoader worker (pid(s) 3385908) exited unexpectedly
FAILED openfold3/tests/test_inference_full.py::test_inference_run[query_set1] - RuntimeError: DataLoader worker (pid(s) 3387152) exited unexpectedly
========================================================================= 2 failed, 26 warnings in 257.95s (0:04:17) =========================================================================
/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:67: PytestUnraisableExceptionWarning: Exception ignored in: <function _ConnectionBase.__del__ at 0x1534daec8b80>

Traceback (most recent call last):
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 133, in __del__
    self._close()
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 377, in _close
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor

Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
  warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
/work2/mb97hape-software/openfold-3/.venv/lib/python3.12/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread QueueFeederThread

Traceback (most recent call last):
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 259, in _feed
    reader_close()
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 178, in close
    self._close()
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/connection.py", line 377, in _close
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/home/sc.uni-leipzig.de/mb97hape/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 291, in _feed
    queue_sem.release()
ValueError: semaphore or lock released too many times

Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
  warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))
Predicting: |          | 0/? [02:10<?, ?it/s]
Integration tests failed. Please check the output above.

---

Removing multiprocessing

When I remove multiprocessing using the option you mentioned, the error still occurs, but without showing the error for the dataloader as in the first post. The core simply dumped

Command
The command was used with the low_mem.yaml:

# Model changes for low memory 
model_update:
  presets: 
    - predict
    - low_mem
    
data_module_args: {num_workers: 0}

Ξ software/openfold-3 git:(main) ▶ source .venv/bin/activate
(openfold3) Ξ software/openfold-3 git:(main) ▶ run_openfold predict \
    --query_json=examples/example_inference_inputs/query_ubiquitin.json \
    --inference-ckpt-path weights/of3-p2-155k.pt \
    --runner_yaml examples/example_runner_yamls/low_mem.yml

Stack trace

(openfold3) Ξ software/openfold-3 git:(main) ▶ run_openfold predict --query_json=examples/example_inference_inputs/query_ubiquitin.json --inference-ckpt-path weights/of3-p2-155k.pt --runner_yaml examples/example_runner_yamls/low_mem.yml
WARNING:openfold3.entry_points.experiment_runner:No version_tensor is found for this checkpoint.Assuming the user knows checkpoints are parameters are compatible, continuing...
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Using output directory: /tmp/of3_colabfold_msas for ColabFold MSAs.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Mapping file /tmp/of3_colabfold_msas/mappings/seq_to_rep_id.json already exists. Appending new sequences.
WARNING:openfold3.core.data.tools.colabfold_msa_server:Mapping file /tmp/of3_colabfold_msas/mappings/rep_id_to_seq.json already exists. Appending new sequences.
Submitting 1 sequences to the Colabfold MSA server for main MSAs...
No complexes found for paired MSA generation. Skipping...
/home/iwe34/.local/share/uv/python/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=2846681) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
Preprocessing templates: 100%|███████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 206.37it/s]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/media/data/software/openfold-3/.venv/lib/python3.13/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'predict_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=23` in the `DataLoader` to improve performance.
Predicting: |                                                                                                             | 0/? [00:00<?, ?it/s]
[1]    2846681 segmentation fault (core dumped)  run_openfold predict  --inference-ckpt-path weights/of3-p2-155k.pt

[jnwei]

Thank you for the extensive stacktraces @Bloeci

Hmm.. interesting. I recall @dwhswenson also had a dataloader core dump when he was working on the openfold3 build with uv... but I'm not sure if we got to the bottom of that issue. One thing I notice is that there seems to be a mismatch between the python versions used in your local virtual environment set up with uv (3.13, the second log) and the pytest log (3.12, the first log). But maybe you intentionally tested different python versions.

I am glad to hear that at least the docker version is working. The confidence scores for ubiquitin are about in the range I would expect. I would take the confidence scores with a grain of salt, they are more useful when comparing relative confidences rather than as an absolute value on their own.

Hopefully the installation process will be more smooth once we work out the pixi build, coming in the next month or so. Thank you for your patience.