# Why PyTorch throws a cuDNN version error, and how to fix it

> Source: https://netray.co/troubleshoot/cudnn-version-error
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

PyTorch wheels bundle their own cuDNN version internally, so a separately installed system-wide cuDNN is usually unnecessary and often the actual cause of this error. When LD_LIBRARY_PATH exposes a different cuDNN version than the one torch was compiled against, torch loads the wrong one at runtime and throws a version incompatibility error. Removing the manual cuDNN path and letting torch use its bundled copy resolves most cases.

## Error Title

cuDNN version mismatch or library loading error in PyTorch

## Error Signature

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (8, 9, 0) but linked against (8, 5, 0)

## Alternate Signatures

- libcudnn.so.8: cannot open shared object file: No such file or directory
- RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

## Category

installation

## Stack

- pytorch
- cuda

## Affects

PyTorch installs on machines with a manually installed system-wide cuDNN, or environments where multiple ML frameworks each bundle a conflicting cuDNN version.

## Quick Fix

### Fix it in a few minutes

**Steps:**

- Check which cuDNN version torch reports internally: python -c "import torch; print(torch.backends.cudnn.version())".
- Check for a conflicting system-wide cuDNN on the library path: echo $LD_LIBRARY_PATH and ldconfig -p | grep cudnn.
- Remove any manually added cuDNN directories from LD_LIBRARY_PATH so torch falls back to its own bundled copy.
- If another framework like TensorFlow is imported in the same process and needs its own cuDNN, isolate the two into separate environments rather than sharing a process.
- Re-run your script and confirm torch.backends.cudnn.is_available() returns True with no version warning.

## Symptoms

- RuntimeError explicitly naming two different cuDNN version numbers as compiled against versus linked against
- libcudnn.so.8 or a similarly named file reported as missing despite cuDNN appearing installed
- CUDNN_STATUS_NOT_INITIALIZED error on the first convolution or attention operation
- The same code works in one virtual environment but fails in another on the identical machine

## Causes

### A manually installed system-wide cuDNN shadows torch's bundled copy

**Likelihood:** most-common

**Explanation:** PyTorch wheels already include a specific cuDNN version compiled and tested against that exact torch release. When a different cuDNN version is separately installed system-wide, for example copied manually into /usr/local/cuda, and its directory appears in LD_LIBRARY_PATH ahead of torch's bundled path, the dynamic linker loads the system version instead, which was never validated against that torch build.

**Fix:** Remove the manually installed cuDNN directory from LD_LIBRARY_PATH entirely and let torch load its own bundled cuDNN, which is almost always sufficient since torch does not require a separate system install.

**Commands:**

- echo $LD_LIBRARY_PATH
- python -c "import torch; print(torch.backends.cudnn.version())"

### Multiple environments each with a different cuDNN package are active

**Likelihood:** common

**Explanation:** Conda environments and pip virtual environments can each install their own cudnn package independently. If a shell session accidentally activates the wrong environment, or a script is launched with an inherited environment variable pointing at a different environment's cuDNN, the version torch expects and the version actually loaded diverge silently.

**Fix:** Deactivate all environments and start from a clean shell, then activate only the single intended environment, verifying with which python and pip show torch before running the script again.

### Manually copied cuDNN files into the CUDA toolkit directory

**Likelihood:** common

**Explanation:** Some older tutorials instruct users to manually download cuDNN from NVIDIA and copy its header and library files directly into the CUDA toolkit's include and lib64 directories. This global system change affects every CUDA application on the machine, including ones like torch that never needed it, and any mismatch between this manually installed version and what torch expects produces the incompatibility error.

**Fix:** Remove the manually copied cuDNN files from the CUDA toolkit directory unless another application on the same machine specifically requires them, since torch itself does not need a system cuDNN install.

### A different framework imported earlier in the process pins a conflicting cuDNN

**Likelihood:** occasional

**Explanation:** If TensorFlow, JAX, or another CUDA-aware library is imported before torch in the same Python process and that library dynamically loads its own bundled or system cuDNN first, torch's subsequent attempt to initialize CUDA operations can conflict with the already-loaded library version in memory.

**Fix:** Avoid importing multiple heavy CUDA frameworks in the same process where possible, or explicitly control import order and verify each framework's expected cuDNN version is compatible before combining them.

### Partial or corrupted manual cuDNN installation missing library symlinks

**Likelihood:** rare

**Explanation:** A manual cuDNN install that copies the versioned library files (e.g. libcudnn.so.8.9.0) but omits creating the standard unversioned symlink (libcudnn.so) that some tools expect can produce a missing library error even though the actual file exists on disk somewhere.

**Fix:** Recreate the expected symlinks pointing to the actual versioned library file, or simply remove the partial manual install and rely on torch's bundled cuDNN instead.

## Diagnostics

### Check the cuDNN version torch itself reports

**Command:** python -c "import torch; print(torch.backends.cudnn.version())"

**Interpretation:** This is the version torch is actually using at runtime; compare it against the version named in any error message to confirm which cuDNN got loaded.

### Check for system-installed cuDNN libraries that might conflict

**Command:** ldconfig -p | grep cudnn

**Interpretation:** Any entries here indicate a system-wide cuDNN install exists outside of torch's bundled copy, which is the most likely source of a version conflict.

### Check what library paths are exposed to the current process

**Command:** echo $LD_LIBRARY_PATH

**Interpretation:** A manually added CUDA or cuDNN path listed here, especially ahead of the standard system paths, is the classic cause of loading the wrong cuDNN version.

### Confirm cuDNN is usable at all after cleaning up the environment

**Command:** python -c "import torch; print(torch.backends.cudnn.is_available())"

**Interpretation:** True with no version warning confirms the fix worked; False or a repeated warning means another conflicting path or environment is still active.

## Prevention

- Avoid manually installing a system-wide cuDNN unless a specific non-torch application explicitly requires it.
- Keep LD_LIBRARY_PATH minimal and free of manually added CUDA or cuDNN directories in your shell profile and Dockerfiles.
- Isolate different CUDA-aware frameworks like TensorFlow and PyTorch into separate virtual environments or containers rather than one shared process.
- Document any legitimate reason for a manual cuDNN install on a given machine so future engineers do not remove it blindly or reintroduce the same conflict elsewhere.

## When To Escalate

If multiple teams share GPU machines with conflicting framework and cuDNN requirements that cannot be resolved through environment isolation alone, this becomes an infrastructure segmentation decision, such as moving to per-team containers or dedicated nodes, rather than an ongoing environment variable troubleshooting exercise.

## Faq

### Do I need to install cuDNN separately for PyTorch to work?

**Answer:** No. PyTorch wheels already bundle a specific, tested cuDNN version internally. A separate system-wide cuDNN install is usually unnecessary for PyTorch alone, and when one exists, it is frequently the actual cause of version mismatch errors rather than a requirement for torch to function.

### What does the compiled against versus linked against error actually mean?

**Answer:** It means torch was built and tested against one specific cuDNN version, but at runtime the dynamic linker loaded a different cuDNN version found elsewhere on the library search path, typically from a manually installed system-wide copy. The two versions are not guaranteed to be compatible, so torch raises this error rather than risk incorrect results.

### How do I remove a conflicting system cuDNN safely?

**Answer:** First confirm no other application on the machine specifically depends on that system-wide cuDNN install. Then remove its directory from LD_LIBRARY_PATH, or uninstall the package if it was installed via a package manager, and re-run python -c "import torch; print(torch.backends.cudnn.version())" to confirm torch now uses its own bundled version cleanly.

### Can this error happen even in a fresh virtual environment?

**Answer:** Yes, if LD_LIBRARY_PATH is set globally in your shell profile rather than per-environment, a fresh virtual environment still inherits the same conflicting system path. Check your shell startup files (.bashrc, .profile) for any CUDA or cuDNN path exports that persist across environments.

## Search terms

cudnn version mismatch pytorch, cudnn compiled against linked against, libcudnn.so.8 cannot open shared object file, cudnn_status_not_initialized, torch.backends.cudnn.version, pytorch cudnn error fix

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json