CUDA detection failed for text-generation-webui #54

knkski · 2023-10-21T23:59:24Z

First off, thanks for the great project.

I'm trying to run the text-generation-webui project, and hitting a CUDA error. This is on an older card, a 1080 TI, so I possibly just need to upgrade it? Here's what I'm trying to do and the error:

$ nix --extra-experimental-features nix-command --extra-experimental-features flakes run .#textgen-nvidia
warning: Using saved setting for 'extra-substituters = https://ai.cachix.org' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = ai.cachix.org-1:N9dzRK+alWwoKXQlnn0H6aUx0lU/mspIoz8hMvGvbbc=' from ~/.local/share/nix/trusted-settings.json.
False
/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: /run/opengl-driver/lib:/nix/store/izaqzlav74p7q86bjwk8201wab14q4cs-cudatoolkit-11.8.0/lib did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('/home/me/.nix-profile/etc/xdg'), PosixPath('/nix/var/nix/profiles/default/etc/xdg')}
The following directories listed in your path were found to be non-existent: {PosixPath('/nix/var/nix/profiles/default/share/pixmaps'), PosixPath('/home/me/.nix-profile/share/pixmaps'), PosixPath('/nix/var/nix/profiles/default/share/icons'), PosixPath('/home/me/.nix-profile/share/icons'), PosixPath('/home/me/.icons')}
The following directories listed in your path were found to be non-existent: {PosixPath('nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos'), PosixPath('nixos-config=/etc/nixos/configuration.nix')}
The following directories listed in your path were found to be non-existent: {PosixPath('/etc/profiles/per-user/me/info'), PosixPath('/home/me/.nix-profile/share/info'), PosixPath('/home/me/.nix-profile/info'), PosixPath('/nix/var/nix/profiles/default/share/info'), PosixPath('/run/current-system/sw/info'), PosixPath('/nix/var/nix/profiles/default/info')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/me/.nix-profile/lib/gtk-2.0'), PosixPath('/nix/var/nix/profiles/default/lib/gtk-2.0'), PosixPath('/home/me/.nix-profile/lib/gtk-4.0'), PosixPath('/nix/var/nix/profiles/default/lib/gtk-4.0'), PosixPath('/home/me/.nix-profile/lib/gtk-3.0'), PosixPath('/nix/var/nix/profiles/default/lib/gtk-3.0')}
The following directories listed in your path were found to be non-existent: {PosixPath('/nix/var/nix/profiles/default/lib/mozilla/plugins'), PosixPath('/etc/profiles/per-user/me/lib/mozilla/plugins'), PosixPath('/home/me/.nix-profile/lib/mozilla/plugins'), PosixPath('/run/current-system/sw/lib/mozilla/plugins')}
The following directories listed in your path were found to be non-existent: {PosixPath('/nix/var/nix/profiles/default/share/terminfo'), PosixPath('/home/me/.nix-profile/share/terminfo')}
The following directories listed in your path were found to be non-existent: {PosixPath('/nix/var/nix/profiles/default/lib/mozilla/plugins'), PosixPath('/etc/profiles/per-user/me/lib/mozilla/plugins'), PosixPath('/home/me/.nix-profile/lib/mozilla/plugins'), PosixPath('/run/current-system/sw/lib/mozilla/plugins')}
The following directories listed in your path were found to be non-existent: {PosixPath('/nix/var/nix/profiles/default'), PosixPath('/home/me/.nix-profile')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/me/.nix-profile/lib/libexec'), PosixPath('/nix/var/nix/profiles/default/lib/libexec'), PosixPath('/run/current-system/sw/lib/libexec'), PosixPath('/etc/profiles/per-user/me/lib/libexec')}
The following directories listed in your path were found to be non-existent: {PosixPath('/etc/profiles/per-user/me/lib/kde4/plugins'), PosixPath('/run/current-system/sw/lib/kde4/plugins'), PosixPath('/nix/var/nix/profiles/default/lib/qt4/plugins'), PosixPath('/etc/profiles/per-user/me/lib/qt4/plugins'), PosixPath('/run/current-system/sw/lib/qt4/plugins'), PosixPath('/home/me/.nix-profile/lib/kde4/plugins'), PosixPath('/nix/var/nix/profiles/default/lib/kde4/plugins'), PosixPath('/home/me/.nix-profile/lib/qt4/plugins')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/nix/store/fgpx31cf48rag3yjmrgyw2kjvc12pi5m-cuda-native-redist-11.8/lib/libcudart.so.11.0'), PosixPath('/nix/store/fgpx31cf48rag3yjmrgyw2kjvc12pi5m-cuda-native-redist-11.8/lib/libcudart.so')}.. We select the PyTorch default libcudart.so, which is {torch.version.cuda},but this might missmatch with the CUDA version that is needed for bitsandbytes.To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variableFor example, if you want to use the CUDA version 122BNB_CUDA_VERSION=122 python ...OR set the environmental variable in your .bashrc: export BNB_CUDA_VERSION=122In the case of a manual override, make sure you set the LD_LIBRARY_PATH, e.g.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2
  warn(msg)
DEBUG: Possible options found for libcudart.so: {PosixPath('/nix/store/fgpx31cf48rag3yjmrgyw2kjvc12pi5m-cuda-native-redist-11.8/lib/libcudart.so.11.0'), PosixPath('/nix/store/fgpx31cf48rag3yjmrgyw2kjvc12pi5m-cuda-native-redist-11.8/lib/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 6.1.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!                     If you run into issues with 8-bit matmul, you can try 4-bit quantization: https://huggingface.co/blog/4bit-transformers-bitsandbytes
  warn(msg)
CUDA SETUP: Required library version not found: libbitsandbytes_cuda118_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries
5. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source:
git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=118 make cuda11x_nomatmul
python setup.py install
CUDA SETUP: Setup Failed!
Traceback (most recent call last):
  File "/nix/store/ypar8406iyb6r22n755ygvfbplwjs050-textgen-patchedSrc/server.py", line 30, in <module>
    from modules import (
  File "/nix/store/ypar8406iyb6r22n755ygvfbplwjs050-textgen-patchedSrc/modules/chat.py", line 18, in <module>
    from modules.text_generation import (
  File "/nix/store/ypar8406iyb6r22n755ygvfbplwjs050-textgen-patchedSrc/modules/text_generation.py", line 24, in <module>
    from modules.models import clear_torch_cache, local_rank
  File "/nix/store/ypar8406iyb6r22n755ygvfbplwjs050-textgen-patchedSrc/modules/models.py", line 10, in <module>
    from accelerate import infer_auto_device_map, init_empty_weights
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 131, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in <module>
    import bitsandbytes as bnb
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/nix/store/300ld78f8vr4gxi57pp2pbjnm91wpv55-python3-3.10.12-env/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Here's the output from nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   28C    P8    10W / 280W |    546MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

If I run the bitsandbytes commands myself, the package is built and installs just fine, so I'm not sure why it's failing as part of the larger build.

The text was updated successfully, but these errors were encountered:

MatthewCroughan · 2023-10-22T13:30:44Z

Yeah, bitsandbytes is a hard library, if I understand correctly it compiles for a specific "compute capability" like most cuda stuff, here's the logs from the much superior llama.cpp

ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6

My hunch is that we would probably need to compile bitsandbytes for the "compute capability" of the 1080ti which is probably not 8.6 like mine.

MatthewCroughan · 2023-10-22T13:33:17Z

@knkski as another suggestion, here's my nvidia-smi, try matching the cuda version and driver version and seeing what the runtime behavior is:

nvidia-smi
Sun Oct 22 14:32:32 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   44C    P8    23W / 350W |     21MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2168      G   ...xorg-server-1.20.14/bin/X       10MiB |
|    0   N/A  N/A      2277      G   ...hell-43.3/bin/gnome-shell        8MiB |
+-----------------------------------------------------------------------------+

max-privatevoid · 2023-10-22T18:44:26Z

The GTX 10 series is CC 6.1. Below CC 7.5, bitsandbytes switches to the "nocublaslt" version of one of its libraries, which we don't include currently.

Modifying preBuild for bitsandbytes should allow us to compile both variants of the library. The Makefile contains a bunch different targets, we probably want to match this to our CUDA version.

https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/python-modules/bitsandbytes/default.nix#L69-L74
https://github.com/TimDettmers/bitsandbytes/blob/18e827d666fa2b70a12d539ccedc17aa51b2c97c/Makefile#L86

knkski · 2023-10-22T21:25:42Z

@MatthewCroughan @max-privatevoid Thanks for the quick responses! I was able to get it going with a workaround following @max-privatevoid's suggestion. I copied the upstream bitsandbytes default.nix file to a new packages/bitsandbytes/default.nix file in this repo, then applied this diff:

-    ''make CUDA_VERSION=${cudaVersion} cuda${cudaMajorVersion}x''
+    ''make CUDA_VERSION=${cudaVersion} cuda${cudaMajorVersion}x_nomatmul''

It compiled and is running successfully 🎉

I don't have enough experience here to polish that up into a proper PR, but I can at least confirm that it works.

Also, if anyone hits issues with the above, I previously got it working (albeit quite slowly) by just commenting out the bitsandbytes dependency. Everything ran fine, except for obviously the models couldn't get quantized.

Closing this issue since there's a workaround and the fix probably needs to happen in upstream nixpkgs, but feel free to reopen if that's incorrect.

max-privatevoid · 2023-10-22T23:37:24Z

Reopened for #57

knkski closed this as completed Oct 22, 2023

max-privatevoid reopened this Oct 22, 2023

max-privatevoid mentioned this issue Oct 22, 2023

support bitsandbytes on older Nvidia GPUs (CC < 7.5) #57

Merged

MatthewCroughan closed this as completed in #57 Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA detection failed for text-generation-webui #54

CUDA detection failed for text-generation-webui #54

knkski commented Oct 21, 2023

MatthewCroughan commented Oct 22, 2023

MatthewCroughan commented Oct 22, 2023

max-privatevoid commented Oct 22, 2023

knkski commented Oct 22, 2023

max-privatevoid commented Oct 22, 2023

CUDA detection failed for text-generation-webui #54

CUDA detection failed for text-generation-webui #54

Comments

knkski commented Oct 21, 2023

MatthewCroughan commented Oct 22, 2023

MatthewCroughan commented Oct 22, 2023

max-privatevoid commented Oct 22, 2023

knkski commented Oct 22, 2023

max-privatevoid commented Oct 22, 2023