Why PyTorch complains about a CUDA version mismatch, and how to fix it
RuntimeError: The NVIDIA driver on your system is too old (found version 11000). Please update your GPU driver by downloading and installing a new version
Also appears as
- RuntimeError: CUDA error: no kernel image is available for execution on the device
- UserWarning: CUDA initialization: CUDA unknown error, this may be due to an incorrectly set up environment
Short answer
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
Affects: Any PyTorch install where pip pulled a CUDA 12.x wheel onto a machine with an older driver; common right after `pip install torch` with no version pin.
Fix it in a few minutes
- 1Run nvidia-smi and note the CUDA Version shown in the top right corner. That is the maximum CUDA runtime your driver supports.
- 2Go to pytorch.org's install matrix and pick a torch build whose CUDA version is equal to or lower than that number.
- 3Uninstall the current torch with pip uninstall torch torchvision torchaudio.
- 4Reinstall using the exact --index-url pytorch.org gives you for that CUDA version, e.g. pip install torch --index-url https://download.pytorch.org/whl/cu121.
- 5Verify with python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)".
How to confirm this is your problem
- torch.cuda.is_available() returns False even though nvidia-smi shows a GPU
- RuntimeError mentioning the driver is too old, with a specific found version number
- CUDA operations fail immediately on first .cuda() call, before any real compute happens
- The error appears right after a fresh pip install of torch with no CUDA version pinned
Root causes and fixes
Installed torch wheel targets a newer CUDA runtime than the driver supports
A plain pip install torch pulls the latest default wheel, which today usually targets CUDA 12.x. If the machine's driver predates that CUDA generation, the driver refuses to run the bundled runtime because it cannot guarantee compatibility, so torch throws the version error before doing any GPU work.
Fix: Check nvidia-smi's top right CUDA Version, then reinstall torch from the matching CUDA index (cu118, cu121, cu124, etc.) using the exact --index-url from pytorch.org, not a bare pip install.
nvidia-smi pip uninstall -y torch torchvision torchaudio pip install torch --index-url https://download.pytorch.org/whl/cu121
Driver was never updated after a GPU or OS upgrade
Servers imaged months or years ago often keep the original driver even after the OS or GPU hardware changes. New torch releases assume recent CUDA generations, so an old driver that was fine for last year's torch build stops satisfying this year's default wheel.
Fix: Update the NVIDIA driver package for your distro (apt, dnf, or the official .run installer) to the latest production branch, reboot, then reconfirm with nvidia-smi before reinstalling torch.
Confusing nvcc's toolkit version with the driver's supported CUDA version
nvcc --version reports the locally installed CUDA toolkit used for compiling custom kernels. It has nothing to do with which CUDA runtime PyTorch uses at inference time, because torch wheels vendor their own cudart, cublas, and cudnn. Engineers chase the wrong number and upgrade the toolkit instead of the driver.
Fix: Ignore nvcc --version for this problem entirely. Only nvidia-smi's CUDA Version field and torch.version.cuda matter for choosing a compatible wheel.
Container image ships a newer CUDA toolkit than the host driver allows
Base images like nvidia/cuda:12.4-runtime bundle a CUDA runtime that still has to pass the host driver's compatibility check via NVIDIA's forward-compatibility layer. On a host with an old driver and no compat package installed, containers built for newer CUDA fail the same way.
Fix: Either update the host driver or install NVIDIA's CUDA forward-compatibility package on the host, and prefer base images pinned to a CUDA version the fleet's oldest driver supports.
Diagnostic commands
Check the driver's maximum supported CUDA version
nvidia-smi
The CUDA Version field top right is the ceiling your driver supports. Any torch wheel built for a CUDA version at or below this number will work; anything higher triggers the driver-too-old error.
Check the locally installed toolkit (usually irrelevant to torch)
nvcc --version
This is the compiler toolkit version, only relevant if you are compiling CUDA extensions yourself. It has no bearing on which CUDA runtime a pip-installed torch wheel uses.
Check which CUDA runtime torch itself was built against
python -c "import torch; print(torch.__version__, torch.version.cuda)"
torch.version.cuda shows the bundled runtime version. Compare it against nvidia-smi's CUDA Version; if torch.version.cuda is higher, that is the mismatch causing the failure.
Confirm the GPU is actually reachable after reinstalling
python -c "import torch; print(torch.cuda.is_available())"
True confirms the fix worked. False after a matching reinstall usually points to a deeper driver install problem rather than a version mismatch.
Stopping it from happening again
- Pin the exact torch version and --index-url in requirements.txt so CI and production always resolve the same CUDA build.
- Track the driver version deployed across your GPU fleet and treat driver upgrades as a planned, tested change, not incidental.
- Before bumping torch in any project, check its CUDA requirement against the oldest driver version in your fleet.
- In Docker, pin the base image's CUDA version to something your least-updated host driver can run, and document the minimum required driver.
When this becomes an architecture problem
If you manage a fleet with inconsistent driver versions across nodes, or you are air-gapped and cannot pull new drivers on demand, this stops being a one-machine fix and becomes a driver standardization and image management problem worth solving at the platform level.
Frequently asked questions
Do I need to match nvcc's CUDA version to my PyTorch version?
No. PyTorch wheels bundle their own CUDA runtime (cudart, cublas, cudnn), so the locally installed toolkit that nvcc reports is irrelevant for running pip-installed torch. nvcc only matters if you are compiling custom CUDA extensions yourself, such as FlashAttention or a custom kernel, where the toolkit version must align with your compiler and PyTorch's build.
What does the CUDA Version field in nvidia-smi actually mean?
It is the maximum CUDA runtime version your currently installed driver supports, not a CUDA toolkit that is installed. Any application, including a PyTorch wheel, whose bundled CUDA runtime is at or below that number will run correctly. A higher bundled version triggers the driver-too-old error.
Can I fix this without updating my GPU driver?
Yes, in most cases. Instead of upgrading the driver, reinstall torch using an older CUDA build (for example cu118 instead of cu124) that your current driver already supports. This is usually faster and lower-risk than a driver upgrade on shared infrastructure.
Why did this suddenly appear after I upgraded transformers or vllm?
Upgrading a library like transformers or vllm often pulls in a newer torch as a dependency automatically, silently changing your CUDA requirement. Always check pip's install log for a torch version bump whenever an unrelated package upgrade breaks GPU detection.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Self-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolNVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Related problems
nvidia-smi command not found or fails to communicate with the driver
nvidia-smi not found or unable to communicate almost always means the NVIDIA kernel module never loaded, and the two most common reasons are that the driver was never installed, or the driver is installed but Secure Boot is blocking the unsigned kernel module from loading. WSL2 users hit a different variant: the driver must be installed on the Windows host, never inside the Linux guest.
torch.cuda.is_available() returns False even though a GPU is present
torch.cuda.is_available() returning False almost always means either the installed torch wheel is a CPU-only build, or the process cannot see the GPU due to a driver, container, or environment variable problem. Checking torch.version.cuda for None immediately tells you whether you have a CPU-only wheel, which is the single most common cause and the fastest thing to rule out.
cuDNN version mismatch or library loading error in PyTorch
PyTorch wheels bundle their own cuDNN version internally, so a separately installed system-wide cuDNN is usually unnecessary and often the actual cause of this error. When LD_LIBRARY_PATH exposes a different cuDNN version than the one torch was compiled against, torch loads the wrong one at runtime and throws a version incompatibility error. Removing the manual cuDNN path and letting torch use its bundled copy resolves most cases.
NVIDIA driver installation fails on Ubuntu
The single most common reason NVIDIA driver installation fails on Ubuntu is Secure Boot rejecting the unsigned or self-signed kernel module at load time, since most machines now ship with Secure Boot enabled out of the box. The fix is enrolling the MOK key the installer generates, or disabling Secure Boot in the BIOS, then clearing any lingering nouveau or mixed-install conflicts before rebooting.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.