# Why a Kubernetes GPU pod gets stuck Pending, and how to fix it

> Source: https://netray.co/troubleshoot/kubernetes-gpu-pod-pending
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

A GPU pod stays Pending when no node advertises the nvidia.com/gpu resource because the device plugin is down or missing, the pod requests more GPUs than any single node has, or a taint, toleration, or nodeSelector mismatch blocks placement on the GPU pool. Always start with kubectl describe pod, since the Events section states the exact blocking reason rather than leaving you to guess between these causes.

## Error Title

Kubernetes GPU pod stuck in Pending

## Error Signature

0/5 nodes are available: 5 Insufficient nvidia.com/gpu

## Alternate Signatures

- pod has unbound immediate PersistentVolumeClaims
- 0/3 nodes are available: 3 node(s) had untolerated taint {nvidia.com/gpu: present}

## Category

containers

## Stack

- kubernetes
- docker
- nvidia-driver

## Affects

Kubernetes 1.24 and later with GPU worker nodes, any cluster using the NVIDIA device plugin or GPU Operator

## Quick Fix

### Diagnose it in under two minutes

**Steps:**

- Run kubectl describe pod <name> and read the Events section at the bottom for the exact reason.
- If it says Insufficient nvidia.com/gpu, run kubectl describe node <gpu-node> to check Allocatable for that resource.
- If nvidia.com/gpu is missing from Allocatable, check the device plugin or GPU Operator pods for CrashLoopBackOff.
- If it mentions an untolerated taint, add a matching tolerations block to the pod spec.
- If the request exceeds any single node's GPU count, reduce it or redesign as a multi-node job.

## Symptoms

- kubectl get pods shows STATUS Pending indefinitely
- kubectl describe pod shows 'Insufficient nvidia.com/gpu'
- kubectl describe pod shows untolerated taint messages
- nvidia.com/gpu missing entirely from node Allocatable output

## Causes

### NVIDIA device plugin is not running so no node advertises nvidia.com/gpu

**Likelihood:** most-common

**Explanation:** Kubernetes only knows a node has GPUs if a device plugin DaemonSet is running on that node and registers the nvidia.com/gpu extended resource with the kubelet; without it, the scheduler sees zero allocatable GPUs on every node, and any pod requesting nvidia.com/gpu in its resource limits will never find a fit, no matter how many physical GPUs exist.

**Fix:** Verify the nvidia-device-plugin or GPU Operator DaemonSet pods are Running on your GPU nodes, and check node capacity for the nvidia.com/gpu entry; reinstall the plugin or operator if it is missing or crashlooping.

**Commands:**

- kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
- kubectl describe node <gpu-node> | grep -A5 Capacity
- kubectl get pods -n gpu-operator

### Pod requests more GPUs than any single node has

**Likelihood:** common

**Explanation:** GPU requests in Kubernetes are not divisible or schedulable across nodes; a pod asking for nvidia.com/gpu: 4 can only land on a node with at least 4 free GPUs advertised, so on a cluster of 2-GPU nodes that pod stays Pending forever even though the cluster's total GPU count is far higher.

**Fix:** Reduce the pod's GPU request to fit within a single node's capacity, or move to a multi-node job that uses gang scheduling or an MPI operator to span nodes instead of one pod claiming all GPUs.

**Commands:**

- kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

### Missing node labels/taints or the pod lacks the matching toleration

**Likelihood:** common

**Explanation:** GPU node pools are commonly tainted so only GPU-aware workloads land on expensive hardware; if the pod spec has no matching toleration, or a required nodeSelector or affinity label is missing, the scheduler correctly refuses to place it and it stays Pending indefinitely.

**Fix:** Add the matching toleration block and nodeSelector or affinity to the pod spec so it targets the tainted GPU pool, confirmed against the actual taint and label keys shown by describing the node.

**Commands:**

- kubectl describe node <gpu-node> | grep -A3 Taints
- kubectl get nodes --show-labels | grep gpu

### Insufficient CPU or memory alongside the GPU request causes bin-packing failure

**Likelihood:** occasional

**Explanation:** A pod can request a GPU that is available but also request more CPU or memory than remains free on that specific node after other pods have been scheduled, and Kubernetes will not partially satisfy a pod's resource requests, so it stays Pending on a resource dimension that has nothing to do with the GPU at all.

**Fix:** Check the full describe pod events for the actual reason string, not just the GPU count, and either lower CPU or memory requests or free capacity on the target node.

**Commands:**

- kubectl describe pod <pod-name> | grep -A10 Events

## Diagnostics

### Read the full scheduling failure reason

**Command:** kubectl describe pod <pod-name>

**Interpretation:** The Events section at the bottom states exactly which resource dimension failed (GPU, CPU, memory, taints, or PVC binding); do not guess, read this first.

### Confirm GPUs are advertised as allocatable

**Command:** kubectl describe node <gpu-node> | grep -A5 'Allocatable:'

**Interpretation:** If nvidia.com/gpu is absent or zero here, the device plugin is not registering GPUs on that node regardless of physical hardware.

### Check device plugin health

**Command:** kubectl get pods -n kube-system -o wide | grep nvidia

**Interpretation:** CrashLoopBackOff or ImagePullBackOff on the device plugin daemonset means no node in the cluster can report GPU capacity.

### Check taints on GPU nodes

**Command:** kubectl get nodes -o json | jq '.items[].spec.taints'

**Interpretation:** Confirms whether your GPU pool is tainted and whether your pod's tolerations actually match the taint key, value, and effect exactly.

## Prevention

- Run a GPU Operator or device plugin health check as part of node bootstrapping before marking a node Ready for scheduling
- Standardize taints, tolerations, and nodeSelectors in a Helm chart or Kustomize base so every GPU workload spec is consistent
- Set resource requests that reflect real per-node GPU counts, documented in a cluster capacity sheet
- Alert on device plugin DaemonSet pod restarts, since a silent crash there causes GPU scheduling failures cluster-wide

## When To Escalate

If pods are Pending because your largest node genuinely does not have enough GPUs for the workload, this is a hardware capacity and cluster topology decision, more nodes, bigger nodes, or multi-node parallelism, not a config fix, and is worth planning deliberately rather than resizing requests reactively.

## Faq

### Why does kubectl describe node show no nvidia.com/gpu at all?

**Answer:** This means the NVIDIA device plugin or GPU Operator is either not installed, not running on that specific node, or has crashed. The extended resource nvidia.com/gpu only appears in a node's Allocatable and Capacity sections after the device plugin daemonset successfully registers with that node's kubelet. Check for the device plugin daemonset and inspect its logs if it is not in a Running state.

### My pod requests 1 GPU and the cluster has plenty free, why is it still Pending?

**Answer:** Check the full event message with kubectl describe pod rather than assuming it is purely a GPU count issue. Common causes at this stage are an untolerated taint on the GPU node pool, a nodeSelector or affinity rule referencing a label the node does not actually have, or a co-scheduled CPU or memory request that cannot fit on the specific node with a free GPU. The scheduler reports the real blocking reason in the Events section.

### Can a single pod use GPUs from two different nodes?

**Answer:** No. A Kubernetes pod's resource requests are satisfied entirely within one node; GPUs cannot be split across nodes for a single pod. Multi-node GPU workloads need multiple pods, typically one per node, coordinated through a framework like the Kubeflow MPI Operator or a custom Job, with NCCL or a similar library handling the actual cross-node communication.

### Do I need both the NVIDIA device plugin and the GPU Operator?

**Answer:** No, they overlap. The GPU Operator is the more complete option: it installs and manages the driver, container toolkit, device plugin, node feature discovery, and monitoring as one coordinated set of components. The device plugin alone only handles resource advertisement and assumes the driver and toolkit are already installed some other way. Most new clusters should use the GPU Operator rather than installing the device plugin by itself.

## Search terms

kubernetes gpu pod pending, insufficient nvidia.com/gpu, kubernetes gpu not scheduling, nvidia device plugin not running, untolerated taint nvidia.com/gpu, kubectl describe pod insufficient gpu, kubernetes gpu node not ready

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json