Error named symbol not found at line 116 in file bitsandbytes/csrc/ops.cu

[RDouglasSharp]

check_bnb_install.py output:

bin bitsandbytes/bitsandbytes/libbitsandbytes_cuda121.so
CUDA SETUP: CUDA runtime path found: miniconda3/envs/adapt/lib/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary bitsandbytes/bitsandbytes/libbitsandbytes_cuda121.so...
Error named symbol not found at line 116 in file bitsandbytes/csrc/ops.cu

(side note)
Line 116 of bitsandbytes/csrc/ops.cu is CUDA_CHECK_RETURN(cudaPeekAtLastError())

[rhulha]

Same (on the new H100 GPU):
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /usr/local/lib/python3.8/dist-packages/bitsandbytes-0.38.1-py3.8.egg/bitsandbytes/libbitsandbytes_cuda118.so
CUDA SETUP: CUDA runtime path found: /usr/lib/x86_64-linux-gnu/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 9.0
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes-0.38.1-py3.8.egg/bitsandbytes/libbitsandbytes_cuda118.so...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++
/usr/local/lib/python3.8/dist-packages/bitsandbytes-0.38.1-py3.8.egg/bitsandbytes/libbitsandbytes_cuda118.so

+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++
/home/ubuntu/bitsandbytes/build/lib/bitsandbytes/libbitsandbytes_cuda118.so
/home/ubuntu/bitsandbytes/bitsandbytes/libbitsandbytes_cuda118.so

++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
++++++++++ /usr/lib/x86_64-linux-gnu/ CUDA PATHS +++++++++++
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcudart.so
/usr/lib/x86_64-linux-gnu/stubs/libcuda.so

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['9.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
+ library is importable
+ CUDA function is callable

WARNING: Please be sure to sanitize sensible info from any such env vars!

Error named symbol not found at line 116 in file /home/ubuntu/bitsandbytes/csrc/ops.cu

[Daryl149]

Fix: #566

[maximegmd]

Fix: #566

Not it did not fix, I am still getting this issue with latest version, except line is 216 now.

[andrewssobral]

Hello everyone, I am also getting the same issue, tried @Daryl149 but it not works for me.
Any solution? Please see below what I did.

First, here's the output of my nvidia-smi:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA TITAN V                  Off| 00000000:09:00.0 Off |                  N/A |
| 35%   51C    P8               28W / 250W|      0MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA TITAN RTX                Off| 00000000:42:00.0 Off |                  N/A |
| 41%   43C    P8               21W / 280W|      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

Here's my trials:

Trial 1. On the Ubuntu 22.04, I created an empty Python 3.10 virtualenv and did pip install autotrain-advanced and autotrain setup, then, when I run python -m bitsandbytes it shows me at the end Error named symbol not found at line 117 in file /home/bitsandbytes/csrc/ops.cu

Trial 2. Used a pre-built docker container from nvidia, this one: nvcr.io/nvidia/cuda:11.7.1-cudnn8-devel-ubuntu20.04:
docker run -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/cuda:11.7.1-cudnn8-devel-ubuntu20.04 bash
then, installed python3 python3-dev python3-pip python3-venv among others several packages needed...
then, I ran pip install autotrain-advancedand autotrain setup, everything seems ok.
then, when I ran python -m bitsandbytes I get the same error

(env) root@ba9485273a5f:/home# python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++
/usr/local/cuda-11.7/compat/libcuda.so
/usr/local/cuda-11.7/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudart.so

++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['7.0', '7.5']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable


WARNING: Please be sure to sanitize sensible info from any such env vars!

Error named symbol not found at line 117 in file /home/bitsandbytes/csrc/ops.cu

Trial 3: Build and install bitsandbytes from source

python -m pip uninstall bitsandbytes
git clone --recursive https://github.com/timdettmers/bitsandbytes.git
CUDA_VERSION=117 make cuda11x
python setup.py install
python -m bitsandbytes

Same error !

also tested with:

CUDA_VERSION=120 make cuda120
CUDA_VERSION=121 make cuda121

Same error!

Honestly , I this error is ruining my day...I need help please!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.