# Why the NVIDIA GPU Operator fails to install, and how to fix it > Source: https://netray.co/troubleshoot/gpu-operator-installation-issues > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer GPU Operator installation problems almost always come from a preinstalled host driver conflicting with the Operator's own driver container, or from Node Feature Discovery never labeling GPU nodes so downstream components stay unscheduled. Check kubectl get pods -n gpu-operator first to see which subsystem is failing, then confirm whether the node has a preexisting driver and whether NFD applied the expected NVIDIA labels. ## Error Title NVIDIA GPU Operator pods stuck installing or crashlooping ## Error Signature nvidia-driver-daemonset pod: Error: failed to load kernel module nvidia: Device or resource busy ## Alternate Signatures - gpu-feature-discovery CrashLoopBackOff - Error: could not load NVIDIA driver: version mismatch with pre-installed driver ## Category containers ## Stack - kubernetes - nvidia-driver - docker ## Affects Kubernetes clusters installing the NVIDIA GPU Operator via Helm, most common on nodes with a preinstalled host driver ## Quick Fix ### Diagnose the failing subsystem fast **Steps:** - Run kubectl get pods -n gpu-operator to see which component (driver, toolkit, device plugin, NFD) is failing. - If the driver daemonset shows 'Device or resource busy', check for a preinstalled driver with dpkg -l | grep nvidia-driver. - Either remove the preinstalled driver or set driver.enabled=false in the Operator's Helm values. - If nothing schedules onto the GPU node at all, check Node Feature Discovery labels with kubectl get nodes --show-labels. - If the driver fails with a signature error, check mokutil --sb-state for Secure Boot before assuming a driver bug. ## Symptoms - nvidia-driver-daemonset pods stuck in CrashLoopBackOff - gpu-feature-discovery or device-plugin pods never start - Helm install of gpu-operator completes but no node reports nvidia.com/gpu - driver daemonset logs show 'Device or resource busy' or a signature error ## Causes ### Preinstalled host driver conflicts with the Operator's own driver container **Likelihood:** most-common **Explanation:** The GPU Operator expects to manage the NVIDIA driver itself inside a driver container and load the kernel module from there; if the node image already has a driver installed via apt or a cloud provider's base image, the kernel module is already loaded and owns the GPU, so the Operator's driver daemonset fails to load its own module because the device is busy, and the two installations conflict rather than coexist. **Fix:** Either use a node image with no preinstalled NVIDIA driver and let the Operator manage it fully, or set driver.enabled=false in the Operator's Helm values so it uses the host's existing driver instead of trying to install its own. **Commands:** - helm show values nvidia/gpu-operator | grep -A3 driver: - helm upgrade gpu-operator nvidia/gpu-operator -n gpu-operator --set driver.enabled=false ### Node Feature Discovery not detecting or mislabeling GPU nodes **Likelihood:** common **Explanation:** The Operator relies on Node Feature Discovery to label which nodes have NVIDIA PCI devices so it knows where to schedule driver and device plugin daemonsets; if NFD is not running, misconfigured, or excluded by a node's taints, GPU nodes never get labeled and the Operator's components never get scheduled onto them at all. **Fix:** Confirm NFD worker pods are Running on every GPU node and check for the NVIDIA PCI presence label; reinstall or reconfigure NFD if labels are missing. **Commands:** - kubectl get pods -n gpu-operator -l app=nvidia-node-feature-discovery - kubectl get nodes --show-labels | grep pci-10de ### Container runtime not configured or mismatched (containerd vs Docker) for the Operator's toolkit container **Likelihood:** occasional **Explanation:** The Operator's container-toolkit daemonset writes runtime configuration into containerd's or Docker's config file to register the nvidia runtime automatically; if the cluster uses a nonstandard containerd config path, a read-only filesystem, or a runtime the Operator does not recognize, that automatic registration silently fails and no container on the node can request a GPU even though the driver loaded fine. **Fix:** Check the containerd config path the Operator expects matches your actual node image, and inspect the container-toolkit daemonset logs for explicit registration errors; override the config path via Helm values if your distribution uses a nonstandard location. **Commands:** - kubectl logs -n gpu-operator -l app=nvidia-container-toolkit-daemonset - cat /etc/containerd/config.toml | grep nvidia ### Secure Boot enabled without signed driver modules **Likelihood:** rare **Explanation:** When UEFI Secure Boot is enabled on a node, the kernel refuses to load any kernel module that is not cryptographically signed with a key enrolled in the machine owner key database, and the Operator's default driver container ships unsigned modules for many configurations, causing the driver daemonset to fail to load with a signature verification error. **Fix:** Either disable Secure Boot on GPU nodes, or use the Operator's precompiled signed driver variant and enroll the appropriate MOK key as documented for your distribution. **Commands:** - mokutil --sb-state ## Diagnostics ### Check Operator pod status across the namespace **Command:** kubectl get pods -n gpu-operator **Interpretation:** Any pod in CrashLoopBackOff or Init:Error points directly to which subsystem, driver, toolkit, device plugin, or NFD, is failing. ### Read the driver daemonset logs **Command:** kubectl logs -n gpu-operator -l app=nvidia-driver-daemonset --tail=100 **Interpretation:** Device or resource busy means a conflicting preinstalled driver; signature verification failures point to Secure Boot. ### Confirm NFD labeled the node **Command:** kubectl get node --show-labels **Interpretation:** Missing NVIDIA-related labels means NFD never detected the GPU hardware, so nothing downstream gets scheduled. ### Check for a preinstalled driver conflicting with the Operator **Command:** lsmod | grep nvidia && dpkg -l | grep nvidia-driver **Interpretation:** If a driver package is already installed outside the Operator's management, it needs to be removed or the Operator set to skip driver installation. ## Prevention - Use a minimal node image with no preinstalled NVIDIA driver when planning to run the GPU Operator - Decide upfront whether the Operator or the OS image owns the driver, and set driver.enabled accordingly, never both - Verify Node Feature Discovery is healthy before troubleshooting any other Operator component - Document Secure Boot state per node pool since it changes the required driver installation path ## When To Escalate If Operator install problems keep recurring across a heterogeneous fleet of node images and driver versions, standardizing on a single golden GPU node image with a known driver ownership model is a platform decision worth making once, rather than resolving conflicts per node pool. ## Faq ### Should I let the GPU Operator install the driver, or use the host's preinstalled driver? **Answer:** For most clusters, let the Operator manage the driver fully by using a node image with no NVIDIA driver preinstalled, since this gives you centralized version control and upgrade management through Helm values. Only set driver.enabled=false and rely on a preinstalled host driver if your node image is managed by a process outside your control, such as a locked-down cloud provider image, where you cannot remove the existing driver. ### What does 'Device or resource busy' mean in the driver daemonset logs? **Answer:** It means the NVIDIA kernel module is already loaded on the node, almost always because a driver was preinstalled through the OS package manager or a base image, and the Operator's own driver container is trying to load a second, potentially different, copy of the module into a kernel slot that is already occupied. The fix is to remove the preinstalled driver or tell the Operator to skip driver management for that node pool. ### Why do GPU Operator pods never even get scheduled onto a GPU node? **Answer:** This usually points to Node Feature Discovery. The Operator's daemonsets use nodeSelectors tied to labels that NFD applies after detecting NVIDIA PCI hardware; if NFD is not running, is misconfigured, or the node has a taint NFD's pods do not tolerate, those labels never appear, and every downstream Operator component that depends on them stays unscheduled. ### Does Secure Boot actually block the GPU Operator, or just slow it down? **Answer:** It fully blocks the default driver installation path. With Secure Boot enabled, the kernel will refuse to load any unsigned kernel module, and the Operator's standard driver container includes modules that may not be signed for your specific configuration. You either need to disable Secure Boot on GPU nodes or explicitly use the Operator's signed driver option with a properly enrolled MOK key. ## Search terms gpu operator installation failed, nvidia-driver-daemonset crashloopbackoff, gpu operator device or resource busy, node feature discovery gpu not detected, gpu operator secure boot signature error, helm gpu-operator driver conflict --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json