# Why nvidia-container-cli throws initialization errors, and how to fix them > Source: https://netray.co/troubleshoot/nvidia-container-toolkit-errors > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer nvidia-container-cli initialization errors mean the host's NVIDIA kernel module failed to load or the driver's supported CUDA version does not meet the minimum your container image requires. Check nvidia-smi on the bare host first; if it fails there, fix the kernel module or driver before touching Docker. If the host is healthy, compare its CUDA support against your image's requirement and either upgrade the driver or use an older image tag. ## Error Title nvidia-container-cli errors when starting a GPU container ## Error Signature nvidia-container-cli: initialization error: driver error: failed to process request ## Alternate Signatures - Error response from daemon: OCI runtime create failed: ... nvidia-container-cli: mount error - nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.4 ## Category containers ## Stack - docker - nvidia-driver - cuda ## Affects Any host running NVIDIA Container Toolkit 1.13 or later with Docker or containerd, most common after a driver or kernel update ## Quick Fix ### Fastest path to a working GPU container **Steps:** - Run nvidia-smi directly on the host (not in a container) to rule out a broken driver first. - Check lsmod | grep nvidia; if empty, run sudo modprobe nvidia or rebuild via DKMS. - Note the CUDA version shown in nvidia-smi's header, that is your driver's ceiling. - Compare it to your image's required CUDA version and switch to a compatible tag if needed. - Re-run sudo nvidia-ctk runtime configure --runtime=docker and restart Docker. - Retest with docker run --rm --gpus all nvidia-smi. ## Symptoms - container fails at 'nvidia-container-cli: initialization error' - error mentions 'unsatisfied condition: cuda>=X.X' - GPU containers worked yesterday and fail after a routine host update - OCI runtime create failed mentioning nvidia-container-cli ## Causes ### Host kernel module not loaded or driver/kernel version mismatch after an update **Likelihood:** most-common **Explanation:** nvidia-container-cli talks directly to the host's NVIDIA kernel driver to set up device nodes; if the driver kernel module failed to load, often after an unattended kernel upgrade that did not rebuild the module via DKMS, the initialization call fails before it ever reaches your container, so every single container GPU launch fails identically regardless of image. **Fix:** Check lsmod for nvidia, and if missing, reload it or reinstall the driver via DKMS so it matches the running kernel; reboot if a new kernel is active and the module cannot load without one. **Commands:** - lsmod | grep nvidia - sudo modprobe nvidia - dkms status - sudo apt-get install --reinstall nvidia-driver-535 ### Unsatisfied CUDA version condition between the image and the host driver **Likelihood:** common **Explanation:** CUDA base images embed a minimum driver requirement label such as cuda>=12.4; nvidia-container-cli checks the host driver's supported CUDA version against that label before it hands the container anything, and refuses to start if your driver is older than what the image demands, which is a safety check, not a bug. **Fix:** Upgrade the host driver to a version supporting the required CUDA release, or switch to an older, compatible CUDA base image tag; nvidia-smi's header shows the maximum CUDA version your current driver supports. **Commands:** - nvidia-smi - docker run --rm --gpus all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi ### Toolkit and Docker/containerd version incompatibility after an upgrade **Likelihood:** occasional **Explanation:** Docker, containerd, and the NVIDIA Container Toolkit are updated independently by the OS package manager, and a toolkit release can ship a config schema or CLI interface that an older containerd shim does not understand, producing OCI runtime create failures that look like GPU errors but are actually a plumbing mismatch. **Fix:** Update all three components together (docker-ce, containerd.io, nvidia-container-toolkit) from their official repositories in the same maintenance window, then re-run nvidia-ctk runtime configure. **Commands:** - apt list --installed | grep -E 'docker|containerd|nvidia-container' - sudo apt-get update && sudo apt-get install --only-upgrade docker-ce containerd.io nvidia-container-toolkit ### Corrupted or partial toolkit install leaving stale config **Likelihood:** rare **Explanation:** An interrupted apt install or a manual package removal can leave the NVIDIA container runtime config half-written or referencing binaries that no longer exist, causing nvidia-container-cli to fail in ways unrelated to actual driver or CUDA compatibility. **Fix:** Purge and reinstall the toolkit package cleanly, then regenerate config with nvidia-ctk, rather than hand-editing the existing config file. **Commands:** - sudo apt-get purge nvidia-container-toolkit - sudo apt-get install nvidia-container-toolkit - sudo nvidia-ctk runtime configure --runtime=docker ## Diagnostics ### Check the kernel module is loaded **Command:** lsmod | grep nvidia **Interpretation:** No output means the driver kernel module never loaded; that has to be fixed before any container-level troubleshooting matters. ### Check host CUDA capability **Command:** nvidia-smi **Interpretation:** The top-right corner shows the CUDA version your driver supports; compare it against your base image's required version. ### Check toolkit config integrity **Command:** cat /etc/nvidia-container-runtime/config.toml **Interpretation:** Look for missing or malformed paths; a truncated file usually means a failed or interrupted package install. ### Run the CLI directly for a verbose error **Command:** nvidia-container-cli -k -d /dev/tty info **Interpretation:** Surfaces the exact initialization failure reason instead of Docker's wrapped, less specific message. ## Prevention - Pin nvidia-container-toolkit, docker-ce, and containerd.io to tested versions instead of unattended upgrades - Rebuild and verify the DKMS driver module after every kernel upgrade before assuming GPU hosts are healthy - Add a post-update health check that runs nvidia-smi and a --gpus all test container before marking a node ready - Track host driver CUDA support against the CUDA base image versions used across your fleet ## When To Escalate If this keeps recurring after unattended kernel or driver upgrades on a fleet of GPU nodes, it is worth locking down update policy and building an automated post-patch GPU health check rather than manually diagnosing the same failure on every node. ## Faq ### What does 'unsatisfied condition: cuda>=12.4' actually mean? **Answer:** Every CUDA base image carries a label declaring the minimum driver-supported CUDA version it needs to run correctly. nvidia-container-cli checks this label against what your installed host driver actually supports before allowing the container to start. If your driver only supports an older CUDA version than the image requires, the container is refused rather than allowed to start and fail unpredictably later. Check the supported version in the header of nvidia-smi's output and either upgrade the driver or use an older image tag. ### Why did GPU containers break after a routine apt upgrade? **Answer:** Unattended kernel upgrades are the most common cause. When the kernel changes, the NVIDIA driver's kernel module needs to be rebuilt against the new kernel headers via DKMS; if that rebuild fails silently or the DKMS package was not installed, the module never loads for the new kernel, and every GPU container fails at initialization even though the driver package itself still shows as installed. ### Is nvidia-container-cli part of Docker or NVIDIA's toolkit? **Answer:** It ships as part of the NVIDIA Container Toolkit, not Docker itself. Docker calls into it through the registered nvidia OCI runtime whenever a container requests GPU access. Because it operates below Docker, its errors reference driver and CUDA compatibility details rather than Docker concepts, which is why the messages can look unfamiliar even to experienced Docker users. ### Should I reinstall the driver or just the toolkit? **Answer:** Start by checking whether nvidia-smi works on the host at all outside of any container. If it fails there too, the driver or kernel module is the problem and needs reinstalling or a DKMS rebuild. If nvidia-smi works fine on the host but containers fail, the issue is isolated to the toolkit or its configuration, and reinstalling nvidia-container-toolkit and re-running nvidia-ctk runtime configure is the right next step. ## Search terms nvidia-container-cli initialization error, nvidia-container-cli unsatisfied condition cuda, oci runtime create failed nvidia-container-cli, nvidia container toolkit error, docker gpu driver error failed to process request, nvidia-container-cli mount error --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json