cuDNN, cuFFT, and cuBLAS Errors

[joshuacuellar1]

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

GIT_VERSION:v2.14.0-rc1-21-g4dacf3f368e VERSION:2.14.0

Custom code

No

OS platform and distribution

WSL2 Linux Ubuntu 22

Mobile device

No response

Python version

3.10, but I can try different versions

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

CUDA version: 11.8, cuDNN version: 8.7

GPU model and memory

NVIDIA Geforce GTX 1660 Ti, 8GB Memory

Current behavior?

When I run the GPU test from the TensorFlow install instructions, I get several errors and warnings.
I don't care about the NUMA stuff, but the first 3 errors are that TensorFlow was not able to load cuDNN. I would really like to be able to use it to speed up training some RNNs and FFNNs. I do get my GPU in the list of physical devices, so I can still train, but not as fast as with cuDNN.

Standalone code to reproduce the issue

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Relevant log output

2023-10-09 13:36:23.355516: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-10-09 13:36:23.355674: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-10-09 13:36:23.355933: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-10-09 13:36:23.413225: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-09 13:36:25.872586: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-10-09 13:36:25.916952: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-10-09 13:36:25.917025: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

[SuryanarayanaY]

Hi @Ke293-x2Ek-Qe-7-aE-B ,

Starting from TF2.14 tensorflow provides CUDA package which can install all the cuDNN,cuFFT and cubLas libraries.

You can use pip install tensorflow[and-cuda] command for that.

Please try this command let us know if it helps. Thankyou!

[joshuacuellar1]

@SuryanarayanaY I did not know that it now came bundled with cuDNN. I installed tensorflow with the [and-cuda] part, though, but I also installed cuda toolkit and cuDNN separately. I will try just installing the cuda toolkit and then installing tensorflow[and-cuda].
Also, is there a way to install tensorflow for GPU without it coming with cuDNN? If I just pip install tensorflow, will that install with GPU support, just without cuDNN, so that I can manually install them? I don't really need to, but I am curious if it can be installed that way too.

[joshuacuellar1]

@SuryanarayanaY I tried several times, reinstalling Ubuntu, but it still doesn't work.

[AthiemoneZero]

I also have the same issue, and this seems not to be due to cuda environment as I rebulid cuda and cudnn to make them suit for tf-2.14.0.

This is log out I find:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2023-10-11 18:21:57.387396: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2023-10-11 18:21:57.415774: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2023-10-11 18:21:57.415847: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2023-10-11 18:21:57.415877: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-10-11 18:21:57.421400: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-10-11 18:21:58.155058: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2023-10-11 18:21:59.113217: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:65:00.0/numa_node Your kernel may have been built without NUMA support. 2023-10-11 18:21:59.152044: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:65:00.0/numa_node Your kernel may have been built without NUMA support. 2023-10-11 18:21:59.152153: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:880] could not open file to read NUMA node: /sys/bus/pci/devices/0000:65:00.0/numa_node Your kernel may have been built without NUMA support. [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

[joshuacuellar1]

@AthiemoneZero Because it still does output a GPU device at the bottom of the log, I am training on GPU, just without cuDNN. It will be slower, but it is better than nothing or training on CPU.

[AthiemoneZero]

@AthiemoneZero Because it still does output a GPU device at the bottom of the log, I am training on GPU, just without cuDNN. It will be slower, but it is better than nothing or training on CPU.

Yeah. But I just found that when I downgrade to 2.13.0 version, errors in register won't appear again. It looks like this:

(TF) ephys3@ZhouLab-Ephy3:~$ python3 -c "import tensorrt as trt;import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2023-10-11 20:39:12.097457: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-10-11 20:39:12.130250: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-11 20:39:13.856721: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:65:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-10-11 20:39:13.870767: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:65:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-10-11 20:39:13.870941: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:65:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Although I haven't figured out how to solve NUMA node error, I found some clues from another issue (as I operated all above in WSL Ubuntu). This bug seems not to be significant as explaination from NVIDIA forums . So I guess errors in register might have something with the latest version and errors in NUMA might be caused by OS enviroment. Hope this information would help some guys.

[joshuacuellar1]

@AthiemoneZero I tried downgrading as well, but it didn't work for me. The NUMA errors are (as stated in the error message) because the kernel provided by Microsoft for WSL2 is not built with NUMA support. I tried cloning the repo (here) and building from source my own with NUMA support, but that didn't work, so I am just ignoring those errors for now.

[AthiemoneZero]

@Ke293-x2Ek-Qe-7-aE-B I rebuilt all in an independent conda environment as TF. My steps were to create a TF env with python 3.9.8 and tried python3 -m pip install tensorflow[and-cuda] --user according to instruction. Following these I tried python3 -m pip install tensorflow[and-cuda]=2.13.0 --user and found it solved some bug.

[joshuacuellar1]

@AthiemoneZero Thanks for the instructions. I'll try and see if it works on my system. I have been using python 3.10, so maybe that's why it didn't work. Did you have to install the CUDA toolkit?

[AthiemoneZero]

@Ke293-x2Ek-Qe-7-aE-B I didnt execute conda install cuda-toolkit here. I guess [and-cuda] argument help me install some dependencies.

[AthiemoneZero]

But I did double check version of cuda and cudnn. For this I even downgrade them again and again.

[joshuacuellar1]

@AthiemoneZero Usually, I would install the CUDA toolkit according to these instructions (here), then install cuDNN according to these instructions (here). I installed CUDA toolkit version 11.8 and cuDNN version 8.7, because they are the latest supported by TensorFlow, according to their support table here. I guess using [and-cuda] installs all of that for you.

[AthiemoneZero]

@Ke293-x2Ek-Qe-7-aE-B Apologize for my misunderstanding. I did the same in installing cuda toolkit as what you described above before I went directly to debug tf_gpu. I made sure my gpu and cuda could perform well as I have tried another task smoothly using cuda but without tf. What I concerned is some dependencies of tf have to be pre-installed in a conda env and this might be treated by [and-cuda] (my naive guess

[joshuacuellar1]

@AthiemoneZero I always install CUDA toolkit and cuDNN globally for the whole system, and then install TensorFlow in a miniconda environment. This doesn't work anymore with the newest versions of TensorFlow, so I'll try your instructions. It does make sense to install everything in a conda env, I just hadn't thought of that since my other method had worked in the past. Thanks for sharing what you did to make it work.

[AthiemoneZero]

@Ke293-x2Ek-Qe-7-aE-B You're welcomed. BTW, I also followed the instruction to configure development including suitable version of bazel and clang-16, just before all my operation digging into conda env.

[joshuacuellar1]

@AthiemoneZero Thanks, but it didn't work.

[FaisalAlj]

Hello,

I'm experiencing the same issue, even though I meticulously followed all the instructions for setting up CUDA 11.8 and CuDNN 8.7. The error messages I'm encountering are as follows:

Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered.
Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered.
Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered.

I've tried this with different versions of Python. Surprisingly, when I used Python 3.11, TensorFlow 2.13 was installed without these errors. However, when I used Python 3.10 or 3.9, I ended up with TensorFlow 2.14 and the aforementioned errors.

I've come across information suggesting that I may not need to manually install CUDA and CuDNN, as [and-cuda] should handle the installation of these components automatically.

Could someone please guide me on the correct approach to resolve this issue? I've tried various methods, but unfortunately, none of them have yielded a working solution.

P.S. I'm using conda in WSL 2 on Windows 11.

[nkinnaird]

I am having the same issue as FaisalAlj above, on Windows 10 with the same versions of CUDA and CuDNN. The package tensorflow[and-cuda] is not found by pip. I've tried different versions of python and tensorflow without success. In my case I'm using virtualenv rather than conda.

Edit 1:
I appear to be able to install tensorflow[and-cuda] as long as I use quotes around the package, like: pip install "tensorflow[and-cuda]".

Edit 2:
I still appear to be getting these messages however, so I'm not sure I've installed things correctly.

[SuryanarayanaY]

Hi @Ke293-x2Ek-Qe-7-aE-B ,

I have checked the installation on colab(linx environment) and observed same logs as per attached gist.

These logs seems generated from XLA compiler but GPU is able to detectable. Similar issue #62002 and already bought to Engineering team attention.

CC: @learning-to-play

[johnnkp]

@belitskiy @ddunl Is there any update on the bug fix?

[YanivZeg]

I see the same warnings after installing tensorflow with pip install tensorflow[and-cuda].

E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
12441 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
12441 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

I wasted some hours to try figure this out but then later when I gave up I trained a keras model and saw this among the later prints:

 cuda_dnn.cc:529] Loaded cuDNN version 90300

So does it mean that it actually work and we get a confusing message at init?

[rivershah]

Yes that indicated it worked @YanivZeg. But this is not 100% depending on system config of other users. Till tf team fix this, I am afraid we are all collectively going to be wasting time.

Sorry you wasted precious hours. Tensorflow team any response?

[auc88]

I was running into this issue within the tensorflow image: tensorflow/tensorflow:2.18.0-gpu-jupyter
Doing pip install tensorflow[with-cuda] within the container fixed the problem and starts training from GPU now

[lakshyaverma2414]

Please help , I am using wsl ubuntu, I installed cuda 12 and cudnn 12.6 , I created virtual environment in home directory, everything works perfect but 3 warnings pops up

• cudnn factory unable to register
• cuda factory unable to register
• cuffty factory unable to register

Can anyone help
@tensorflow-jenkins

[CamiloMartinezM]

Sadly the only version that doesn't show those warnings is 2.16.1 as far as I've tested.

[lakshyaverma2414]

Sadly the only version that doesn't show those warnings is 2.16.1 as far as I've tested.

Could you please explain, what actually cuda factory registration actually is ?

[CamiloMartinezM]

Sadly the only version that doesn't show those warnings is 2.16.1 as far as I've tested.

Could you please explain, what actually cuda factory registration actually is ?

I have no idea what those warnings mean, there are other reported issues on this GitHub repo that explain it, but I'm lazy to read that. But I do know that even if those warnings appear, the GPU is recognized and you can train models with it. I don't know if other functionalities are the ones that are affected by those or if they mean performance issues.

[DeveloperXRTECH]

Ran the classification problem (garment ID) in parallel on an internal machine installation (tensorflow 2.18) and collab (tensorflow 2.17). Received warnings in 2.18 but not in 2.17. Otherwise, the results are essentially the same. The prediction results in the internal calculations with version 2.18 are, in many cases, significantly better.

[LunaMeerkats]

I would appreciate this being fixed :(

[lakshyaverma2414]

I would appreciate this being fixed :(

What version are you using ?

[DeveloperXRTECH]

After testing, I concluded that the warnings were just warnings. I have ignored them with a performance on par or better than collab. Suboptimal use of GPU RAM is a potential problem. But so far, 12 GB GPU RAM has provided plenty of headroom.

[johnnkp]

The message disappeared in tf-nightly 2.20.0.dev20250305. It can be installed by the following command:
pip install tf-nightly[and-cuda]

[DebangshuPramanik]

I tried this mentioned fix of using the nightly version, but it didn't work for me. Would have to do with the fact that I'm using linux? Here is the installation errors I received for cuda. It says I'd have to downgrade versions, but I want to use the most recent, theoretically best, version if possible instead.

By the way, I installed everything from the terminal into my python virtual environment.

[johnnkp]

To install together with PyTorch, use the command pip install tf-nightly.

[mazurkin]

tensorflow 2.19 installed with poetry as

[tool.poetry.group.tensorflow.dependencies]
tensorflow = {version = "2.19.0", extras = ["and-cuda"]}
tensorboard = {version = "2.19.0"}
tensorflow-datasets = {version = "4.9.6"}

still reports the same error

$ python fashion-cnn.py 
2025-03-17 08:30:57.442335: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742214657.453348   33850 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742214657.456598   33850 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1742214657.465319   33850 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1742214657.465336   33850 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1742214657.465338   33850 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1742214657.465340   33850 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
<class 'numpy.ndarray'>
I0000 00:00:1742214659.705067   33850 gpu_device.cc:2019] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 10118 MB memory:  -> device: 0, name: NVIDIA RTX 3500 Ada Generation Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.9

but GPU is used and the code works

[lakshyaverma2414]

Install this : pip install tf-nightly

This works absolutely fine with no error flags !

[mazurkin]

Install this : pip install tf-nightly
This works absolutely fine with no error flags !

Thanks @lakshyaverma2414, I will be waiting for 2.20 release version. I need a stable version.

[samyzhh]

tensorflow==2.15.0 ===>>> cuda-12.* (cuda-12.1, cuda-12.2)
tensorflow==2.14.0 ===>>> cuda-12.* (cuda-12.1, cuda-12.2)
tensorflow==2.13.0 ===>>> cuda-11.* (cuda-11.8 etc)
Installing the corresponding tensorflow based on cuda version can solve this problem

[mathbunnyru]

Hi, I'm installing tensorflow[and-cuda] using pip inside a Docker container without GPU: https://github.com/jupyter/docker-stacks/blob/main/images/tensorflow-notebook/cuda/Dockerfile
The reason for that is that I want to provide an image to be used by someone with a GPU.

I have a simple snippet which I run inside an image to test tensorflow works properly:

import tensorflow as tf

print(tf.constant("Hello, TensorFlow"))
print(tf.reduce_sum(tf.random.normal([1000, 1000])))

If I use version version 2.17.1, everything works just fine.
Later versions give an error:

Executing the command: python /home/jovyan/data/unit_tensorflow.py
2025-04-04 01:52:37.835709: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1743731557.947063       7 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1743731557.983104       7 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1743731558.226643       7 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1743731558.226689       7 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1743731558.226700       7 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1743731558.226712       7 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2025-04-04 01:52:38.253996: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-04-04 01:52:44.102925: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
tf.Tensor(b'Hello, TensorFlow', shape=(), dtype=string)
tf.Tensor(901.8375, shape=(), dtype=float32)

Can anyone tell me if this is now expected or how to fix this?
Once again, it was completely fine in 2.17.1.

Ref: jupyter/docker-stacks#2263