Why the NVIDIA GPU Operator fails to install, and how to fix it

The GPU Operator expects to manage the NVIDIA driver itself inside a driver container and load the kernel module from there; if the node image already has a driver installed via apt or a cloud provider's base image, the kernel module is already loaded and owns the GPU, so the Operator's driver daemonset fails to load its own module because the device is busy, and the two installations conflict rather than coexist.

Fix: Either use a node image with no preinstalled NVIDIA driver and let the Operator manage it fully, or set driver.enabled=false in the Operator's Helm values so it uses the host's existing driver instead of trying to install its own.

helm show values nvidia/gpu-operator | grep -A3 driver:
helm upgrade gpu-operator nvidia/gpu-operator -n gpu-operator --set driver.enabled=false

The Operator relies on Node Feature Discovery to label which nodes have NVIDIA PCI devices so it knows where to schedule driver and device plugin daemonsets; if NFD is not running, misconfigured, or excluded by a node's taints, GPU nodes never get labeled and the Operator's components never get scheduled onto them at all.

Fix: Confirm NFD worker pods are Running on every GPU node and check for the NVIDIA PCI presence label; reinstall or reconfigure NFD if labels are missing.

kubectl get pods -n gpu-operator -l app=nvidia-node-feature-discovery
kubectl get nodes --show-labels | grep pci-10de

The Operator's container-toolkit daemonset writes runtime configuration into containerd's or Docker's config file to register the nvidia runtime automatically; if the cluster uses a nonstandard containerd config path, a read-only filesystem, or a runtime the Operator does not recognize, that automatic registration silently fails and no container on the node can request a GPU even though the driver loaded fine.

Fix: Check the containerd config path the Operator expects matches your actual node image, and inspect the container-toolkit daemonset logs for explicit registration errors; override the config path via Helm values if your distribution uses a nonstandard location.

kubectl logs -n gpu-operator -l app=nvidia-container-toolkit-daemonset
cat /etc/containerd/config.toml | grep nvidia

When UEFI Secure Boot is enabled on a node, the kernel refuses to load any kernel module that is not cryptographically signed with a key enrolled in the machine owner key database, and the Operator's default driver container ships unsigned modules for many configurations, causing the driver daemonset to fail to load with a signature verification error.

Fix: Either disable Secure Boot on GPU nodes, or use the Operator's precompiled signed driver variant and enroll the appropriate MOK key as documented for your distribution.

mokutil --sb-state