Why bitsandbytes fails its CUDA setup check, and how to fix it
RuntimeError: CUDA Setup failed despite GPU being available
Also appears as
- UserWarning: The installed version of bitsandbytes was compiled without GPU support
- libbitsandbytes_cuda118.so: cannot open shared object file: No such file or directory
Short answer
bitsandbytes needs to locate the exact CUDA runtime shared library at import time and load a matching precompiled binary for that version. The setup fails most often because LD_LIBRARY_PATH points at a different CUDA installation than the one it detected, or because an older bitsandbytes version could not auto-detect the GPU correctly. Upgrading to the latest bitsandbytes and running its built-in diagnostic resolves the majority of cases.
Affects: bitsandbytes used for 4-bit and 8-bit quantized loading and QLoRA fine-tuning, especially on machines with multiple CUDA installations or containers missing runtime libraries.
Fix it in a few minutes
- 1Upgrade to the latest bitsandbytes: pip install -U bitsandbytes, since recent versions detect CUDA far more reliably.
- 2Run the built-in diagnostic: python -m bitsandbytes, which reports exactly what CUDA library it found and why.
- 3Check for conflicting CUDA paths: echo $LD_LIBRARY_PATH and remove entries pointing at CUDA versions you are not using.
- 4If the diagnostic still cannot find libcudart, locate it manually with find / -name 'libcudart.so*' and add its directory to LD_LIBRARY_PATH.
- 5Re-run your training or inference script to confirm quantized loading works.
How to confirm this is your problem
- Warning that the installed bitsandbytes was compiled without GPU support even though a GPU is present
- RuntimeError naming CUDA Setup failed despite a working GPU visible in nvidia-smi
- Error naming a specific libbitsandbytes_cudaXXX.so file that cannot be found
- 4-bit or 8-bit quantized model loading silently falls back to CPU or errors out
Root causes and fixes
LD_LIBRARY_PATH points at the wrong CUDA installation
bitsandbytes selects a precompiled shared library file named after a specific CUDA version, such as libbitsandbytes_cuda118.so, based on what it detects on the system at import time. When multiple CUDA toolkit versions coexist and LD_LIBRARY_PATH lists an older or unrelated one first, bitsandbytes loads the wrong binary or fails to find any matching one at all.
Fix: Clean up LD_LIBRARY_PATH to only include the CUDA version you actually intend bitsandbytes to use, then re-run python -m bitsandbytes to confirm it now detects the correct library.
echo $LD_LIBRARY_PATH export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH
An older bitsandbytes version could not reliably auto-detect the GPU
Earlier bitsandbytes releases had much less robust CUDA auto-detection and sometimes shipped a CPU-only compiled binary by default in certain pip resolution scenarios, requiring manual environment variable overrides to force GPU mode. Recent releases rewrote this detection logic to be far more reliable out of the box.
Fix: Upgrade to the latest bitsandbytes release with pip install -U bitsandbytes before doing any other troubleshooting; this alone resolves a large share of reported setup failures.
pip install -U bitsandbytes python -m bitsandbytes
CUDA_HOME or the CUDA library path is unset or invalid
bitsandbytes's detection logic checks common environment variables and standard install paths to locate libcudart. If CUDA_HOME is unset, or set to a path where the CUDA toolkit was later removed or moved, the library search comes up empty even though a working CUDA runtime exists elsewhere on the system.
Fix: Set CUDA_HOME to your actual CUDA installation directory and ensure its lib64 subdirectory is included in LD_LIBRARY_PATH before importing bitsandbytes.
Running inside a container missing the CUDA runtime libraries
A container built from a slim base image, or one launched without the NVIDIA Container Toolkit runtime properly configured, may not expose the CUDA shared libraries bitsandbytes needs, even though the GPU device itself is visible via nvidia-smi inside the container.
Fix: Use an nvidia/cuda base image with the runtime or devel variant matching your needs, and confirm the container was launched with --gpus all and the NVIDIA Container Toolkit installed on the host.
Unsupported, very old GPU architecture
Recent bitsandbytes releases have dropped support for very old GPU compute capabilities in favor of the CUDA generations found on modern data center and workstation cards. On genuinely old hardware, no combination of environment variables fixes the detection failure because the compiled kernels simply do not target that architecture.
Fix: Check the bitsandbytes documentation for the minimum supported compute capability, and if your GPU falls below it, pin an older bitsandbytes release known to support that architecture or plan a hardware upgrade.
Diagnostic commands
Run bitsandbytes's own built-in diagnostic
python -m bitsandbytes
Recent versions print exactly which CUDA library they found, its path, and whether GPU support is active, making this the single fastest way to pinpoint the cause.
Check what CUDA paths are currently exposed to the process
echo $LD_LIBRARY_PATH
Multiple CUDA version directories listed here, especially with an older one first, is the most common root cause and should be cleaned up to a single correct path.
Locate all CUDA runtime libraries actually present on disk
find / -name "libcudart.so*" 2>/dev/null
Confirms which CUDA versions are physically installed on the machine, which you can then reconcile against what LD_LIBRARY_PATH and CUDA_HOME are pointing to.
Confirm the GPU itself is visible outside of bitsandbytes
nvidia-smi
If this fails too, the problem is at the driver level, not bitsandbytes; resolve driver visibility first before revisiting bitsandbytes detection.
Stopping it from happening again
- Pin bitsandbytes to a known-working version in requirements.txt rather than letting it float across environment rebuilds.
- Keep only one CUDA toolkit version's paths in LD_LIBRARY_PATH per environment to avoid ambiguous detection.
- Bake a verified bitsandbytes setup into your training container image and run python -m bitsandbytes as a build-time health check.
- When adding a new GPU generation to your fleet, verify bitsandbytes compatibility before assuming existing quantized training scripts will just work.
When this becomes an architecture problem
If quantized fine-tuning or inference needs to run reliably across many heterogeneous nodes, or in an air-gapped environment where you cannot freely pip install -U to chase the latest fix, build and validate one canonical container image with a confirmed working bitsandbytes setup rather than debugging each node.
Frequently asked questions
What is the fastest way to diagnose a bitsandbytes CUDA setup failure?
Run python -m bitsandbytes. Recent releases include a built-in diagnostic that reports exactly which CUDA library version it detected, where it found it, and whether GPU acceleration is active. This single command replaces most manual environment variable inspection and should be your first troubleshooting step.
Does upgrading bitsandbytes really fix most of these errors?
Yes, in a large share of reported cases. Bitsandbytes's CUDA detection logic has improved significantly across recent releases, and many setup failures reported against older versions simply do not reproduce after upgrading with pip install -U bitsandbytes, before any further environment changes.
Why does bitsandbytes need a specific CUDA library file at all?
Bitsandbytes ships precompiled kernels for 8-bit and 4-bit quantized operations that are compiled against a specific CUDA version, named in the shared library filename such as libbitsandbytes_cuda121.so. At import time it must find and load the file matching your actual CUDA runtime, which is why version mismatches cause detection failures rather than silent fallback.
Can I use bitsandbytes inside a Docker container?
Yes, but the container needs the CUDA runtime libraries present, typically via an nvidia/cuda base image, and must be launched with GPU access enabled through the NVIDIA Container Toolkit and the --gpus all flag. A slim base image without CUDA runtime libraries will reproduce the same detection failure seen on bare metal.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Free ToolQLoRA vs Full Fine-Tuning Cost Calculator
See the GPU memory footprint, GPU-hour requirement, and dollar cost gap between QLoRA and full fine-tuning for the same model size and dataset.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Related problems
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
torch.cuda.is_available() returns False even though a GPU is present
torch.cuda.is_available() returning False almost always means either the installed torch wheel is a CPU-only build, or the process cannot see the GPU due to a driver, container, or environment variable problem. Checking torch.version.cuda for None immediately tells you whether you have a CPU-only wheel, which is the single most common cause and the fastest thing to rule out.
FlashAttention install fails during compilation or gets killed
FlashAttention's pip install compiles CUDA kernels from source unless an exact prebuilt wheel exists for your torch, CUDA, Python, and C++ ABI combination, and that compilation is extremely RAM-hungry per parallel job. The build gets silently OOM-killed on machines without enough memory unless you limit MAX_JOBS, and separately fails if your CUDA toolkit does not match the version torch itself was built against.
cuDNN version mismatch or library loading error in PyTorch
PyTorch wheels bundle their own cuDNN version internally, so a separately installed system-wide cuDNN is usually unnecessary and often the actual cause of this error. When LD_LIBRARY_PATH exposes a different cuDNN version than the one torch was compiled against, torch loads the wrong one at runtime and throws a version incompatibility error. Removing the manual cuDNN path and letting torch use its bundled copy resolves most cases.
GuideLLM Quantization: AWQ vs GPTQ vs FP8 vs GGUF
AWQ, GPTQ, FP8, and GGUF compared for production LLM serving: memory savings, throughput impact, quality loss, and which format fits which deployment.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.