# Why bitsandbytes fails its CUDA setup check, and how to fix it > Source: https://netray.co/troubleshoot/bitsandbytes-cuda-setup-failed > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer bitsandbytes needs to locate the exact CUDA runtime shared library at import time and load a matching precompiled binary for that version. The setup fails most often because LD_LIBRARY_PATH points at a different CUDA installation than the one it detected, or because an older bitsandbytes version could not auto-detect the GPU correctly. Upgrading to the latest bitsandbytes and running its built-in diagnostic resolves the majority of cases. ## Error Title bitsandbytes cannot find or detect a working CUDA setup ## Error Signature RuntimeError: CUDA Setup failed despite GPU being available ## Alternate Signatures - UserWarning: The installed version of bitsandbytes was compiled without GPU support - libbitsandbytes_cuda118.so: cannot open shared object file: No such file or directory ## Category installation ## Stack - pytorch - cuda ## Affects bitsandbytes used for 4-bit and 8-bit quantized loading and QLoRA fine-tuning, especially on machines with multiple CUDA installations or containers missing runtime libraries. ## Quick Fix ### Fix it in a few minutes **Steps:** - Upgrade to the latest bitsandbytes: pip install -U bitsandbytes, since recent versions detect CUDA far more reliably. - Run the built-in diagnostic: python -m bitsandbytes, which reports exactly what CUDA library it found and why. - Check for conflicting CUDA paths: echo $LD_LIBRARY_PATH and remove entries pointing at CUDA versions you are not using. - If the diagnostic still cannot find libcudart, locate it manually with find / -name 'libcudart.so*' and add its directory to LD_LIBRARY_PATH. - Re-run your training or inference script to confirm quantized loading works. ## Symptoms - Warning that the installed bitsandbytes was compiled without GPU support even though a GPU is present - RuntimeError naming CUDA Setup failed despite a working GPU visible in nvidia-smi - Error naming a specific libbitsandbytes_cudaXXX.so file that cannot be found - 4-bit or 8-bit quantized model loading silently falls back to CPU or errors out ## Causes ### LD_LIBRARY_PATH points at the wrong CUDA installation **Likelihood:** most-common **Explanation:** bitsandbytes selects a precompiled shared library file named after a specific CUDA version, such as libbitsandbytes_cuda118.so, based on what it detects on the system at import time. When multiple CUDA toolkit versions coexist and LD_LIBRARY_PATH lists an older or unrelated one first, bitsandbytes loads the wrong binary or fails to find any matching one at all. **Fix:** Clean up LD_LIBRARY_PATH to only include the CUDA version you actually intend bitsandbytes to use, then re-run python -m bitsandbytes to confirm it now detects the correct library. **Commands:** - echo $LD_LIBRARY_PATH - export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH ### An older bitsandbytes version could not reliably auto-detect the GPU **Likelihood:** common **Explanation:** Earlier bitsandbytes releases had much less robust CUDA auto-detection and sometimes shipped a CPU-only compiled binary by default in certain pip resolution scenarios, requiring manual environment variable overrides to force GPU mode. Recent releases rewrote this detection logic to be far more reliable out of the box. **Fix:** Upgrade to the latest bitsandbytes release with pip install -U bitsandbytes before doing any other troubleshooting; this alone resolves a large share of reported setup failures. **Commands:** - pip install -U bitsandbytes - python -m bitsandbytes ### CUDA_HOME or the CUDA library path is unset or invalid **Likelihood:** common **Explanation:** bitsandbytes's detection logic checks common environment variables and standard install paths to locate libcudart. If CUDA_HOME is unset, or set to a path where the CUDA toolkit was later removed or moved, the library search comes up empty even though a working CUDA runtime exists elsewhere on the system. **Fix:** Set CUDA_HOME to your actual CUDA installation directory and ensure its lib64 subdirectory is included in LD_LIBRARY_PATH before importing bitsandbytes. ### Running inside a container missing the CUDA runtime libraries **Likelihood:** occasional **Explanation:** A container built from a slim base image, or one launched without the NVIDIA Container Toolkit runtime properly configured, may not expose the CUDA shared libraries bitsandbytes needs, even though the GPU device itself is visible via nvidia-smi inside the container. **Fix:** Use an nvidia/cuda base image with the runtime or devel variant matching your needs, and confirm the container was launched with --gpus all and the NVIDIA Container Toolkit installed on the host. ### Unsupported, very old GPU architecture **Likelihood:** rare **Explanation:** Recent bitsandbytes releases have dropped support for very old GPU compute capabilities in favor of the CUDA generations found on modern data center and workstation cards. On genuinely old hardware, no combination of environment variables fixes the detection failure because the compiled kernels simply do not target that architecture. **Fix:** Check the bitsandbytes documentation for the minimum supported compute capability, and if your GPU falls below it, pin an older bitsandbytes release known to support that architecture or plan a hardware upgrade. ## Diagnostics ### Run bitsandbytes's own built-in diagnostic **Command:** python -m bitsandbytes **Interpretation:** Recent versions print exactly which CUDA library they found, its path, and whether GPU support is active, making this the single fastest way to pinpoint the cause. ### Check what CUDA paths are currently exposed to the process **Command:** echo $LD_LIBRARY_PATH **Interpretation:** Multiple CUDA version directories listed here, especially with an older one first, is the most common root cause and should be cleaned up to a single correct path. ### Locate all CUDA runtime libraries actually present on disk **Command:** find / -name "libcudart.so*" 2>/dev/null **Interpretation:** Confirms which CUDA versions are physically installed on the machine, which you can then reconcile against what LD_LIBRARY_PATH and CUDA_HOME are pointing to. ### Confirm the GPU itself is visible outside of bitsandbytes **Command:** nvidia-smi **Interpretation:** If this fails too, the problem is at the driver level, not bitsandbytes; resolve driver visibility first before revisiting bitsandbytes detection. ## Prevention - Pin bitsandbytes to a known-working version in requirements.txt rather than letting it float across environment rebuilds. - Keep only one CUDA toolkit version's paths in LD_LIBRARY_PATH per environment to avoid ambiguous detection. - Bake a verified bitsandbytes setup into your training container image and run python -m bitsandbytes as a build-time health check. - When adding a new GPU generation to your fleet, verify bitsandbytes compatibility before assuming existing quantized training scripts will just work. ## When To Escalate If quantized fine-tuning or inference needs to run reliably across many heterogeneous nodes, or in an air-gapped environment where you cannot freely pip install -U to chase the latest fix, build and validate one canonical container image with a confirmed working bitsandbytes setup rather than debugging each node. ## Faq ### What is the fastest way to diagnose a bitsandbytes CUDA setup failure? **Answer:** Run python -m bitsandbytes. Recent releases include a built-in diagnostic that reports exactly which CUDA library version it detected, where it found it, and whether GPU acceleration is active. This single command replaces most manual environment variable inspection and should be your first troubleshooting step. ### Does upgrading bitsandbytes really fix most of these errors? **Answer:** Yes, in a large share of reported cases. Bitsandbytes's CUDA detection logic has improved significantly across recent releases, and many setup failures reported against older versions simply do not reproduce after upgrading with pip install -U bitsandbytes, before any further environment changes. ### Why does bitsandbytes need a specific CUDA library file at all? **Answer:** Bitsandbytes ships precompiled kernels for 8-bit and 4-bit quantized operations that are compiled against a specific CUDA version, named in the shared library filename such as libbitsandbytes_cuda121.so. At import time it must find and load the file matching your actual CUDA runtime, which is why version mismatches cause detection failures rather than silent fallback. ### Can I use bitsandbytes inside a Docker container? **Answer:** Yes, but the container needs the CUDA runtime libraries present, typically via an nvidia/cuda base image, and must be launched with GPU access enabled through the NVIDIA Container Toolkit and the --gpus all flag. A slim base image without CUDA runtime libraries will reproduce the same detection failure seen on bare metal. ## Search terms bitsandbytes cuda setup failed, bitsandbytes compiled without gpu support, libbitsandbytes_cuda not found, bitsandbytes LD_LIBRARY_PATH, bitsandbytes qlora cuda error, python -m bitsandbytes diagnostic --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json