Why Kubernetes persistent volumes fail for model weights, and how to fix it
Warning FailedMount ... MountVolume.SetUp failed for volume "model-weights": rpc error: code = Internal
Also appears as
- Multi-Attach error for volume: Volume is already exclusively attached to one node
- pod has unbound immediate PersistentVolumeClaims
Short answer
PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode.
Affects: Kubernetes deployments serving LLM weights from a PersistentVolume, especially with more than one replica
Fix it in a few steps
- 1Run kubectl describe pvc <name> and check for Multi-Attach or unbound PVC events.
- 2Check kubectl get pv <pv-name> -o jsonpath for accessModes to see what the volume actually supports.
- 3If it is ReadWriteOnce and you need multiple replicas, provision from a ReadOnlyMany-capable storage class instead.
- 4Mount the volume with readOnly true in the pod's volumeMounts, since inference never writes to weights.
- 5Set the storage class's volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches.
How to confirm this is your problem
- additional replicas fail with Multi-Attach error for volume
- pod has unbound immediate PersistentVolumeClaims
- MountVolume.SetUp failed rpc error on a subset of nodes
- model loads successfully but takes far longer than expected from the mounted volume
Root causes and fixes
Access mode is ReadWriteOnce but multiple pod replicas need to read the same weights
The default and most widely available PersistentVolume access mode, ReadWriteOnce, only allows the volume to be mounted read-write by a single node at a time; scaling an inference deployment to multiple replicas across different nodes with a ReadWriteOnce claim causes every replica beyond the first to fail scheduling or mounting, since the volume is already exclusively attached elsewhere.
Fix: Use a storage class and volume that supports ReadOnlyMany, or ReadWriteMany, for weights that many replicas need to read simultaneously, and mount it read-only, since inference replicas never need to write to the weights directory.
kubectl get pv <pv-name> -o jsonpath='{.spec.accessModes}'
kubectl get storageclassStorage class does not support the required access mode at all
Not every storage backend or CSI driver supports ReadOnlyMany or ReadWriteMany; many default cloud block storage classes are strictly ReadWriteOnce by design, so simply changing the accessModes field in your PVC spec does nothing if the underlying storage class and CSI driver never advertise that capability.
Fix: Check your CSI driver's documentation for which access modes it actually supports, and if ReadOnlyMany is unavailable, switch to a network filesystem class, such as NFS or a cloud file-storage tier, that explicitly supports shared read access.
kubectl describe storageclass <name>
Storage throughput is too low for the model size, causing slow pod startup rather than an outright error
A PersistentVolume can mount successfully and still be a serving bottleneck if the underlying storage class delivers low sequential read throughput; loading a 70B parameter model from a class rated for typical database workloads rather than large sequential reads can turn a mount success into a multi-minute delay before the pod becomes ready.
Fix: Choose a storage class rated for high throughput sequential reads for model weight volumes, and benchmark actual throughput rather than relying on the class name alone.
dd if=/mnt/model-weights/testfile of=/dev/null bs=1M status=progress
PVC and pod are in different availability zones on cloud-managed clusters
Many cloud block storage volumes are zone-locked, meaning a PersistentVolume provisioned in one availability zone cannot be attached to a node in a different zone; if the pod gets scheduled to a node in a different zone than the volume, the mount fails even though both the PVC and the pod appear healthy individually.
Fix: Use a zone-aware storage class with a volumeBindingMode of WaitForFirstConsumer so the volume is provisioned in the same zone as the pod that first claims it, rather than an arbitrary zone chosen at PVC creation time.
kubectl get storageclass <name> -o jsonpath='{.volumeBindingMode}'Diagnostic commands
Check the PVC's actual status and events
kubectl describe pvc <name>
Look for Multi-Attach errors or pending binding events; these point directly at access mode or zone mismatches.
Check what access modes the underlying PV actually supports
kubectl get pv <pv-name> -o jsonpath='{.spec.accessModes}'If this shows only ReadWriteOnce and you need multiple replicas, this volume type cannot support your scaling plan as configured.
Check pod scheduling and mount events together
kubectl describe pod <pod-name> | grep -A10 Events
FailedMount or FailedAttachVolume events reveal whether the issue is zone affinity, exclusive attachment, or a CSI driver error.
Benchmark storage throughput directly
kubectl exec <pod> -- dd if=/mnt/model-weights/testfile of=/dev/null bs=1M status=progress
Confirms whether a successful mount is still a performance bottleneck for large model loads.
Stopping it from happening again
- Choose ReadOnlyMany-capable storage classes upfront for any model weight volume served by multiple replicas
- Mount weight volumes read-only in the pod spec, since inference workloads never need write access to weights
- Use volumeBindingMode WaitForFirstConsumer to avoid zone mismatches between PVCs and pods
- Benchmark storage class throughput against your largest model size before committing to it in production
When this becomes an architecture problem
If your storage backend genuinely cannot support ReadOnlyMany at the scale you need, or throughput remains a bottleneck even on higher tiers, this becomes a storage architecture decision, a dedicated model artifact cache, node-local NVMe with a sync sidecar, or a different CSI driver entirely, rather than a PVC spec tweak.
Frequently asked questions
Can multiple pods read from the same PersistentVolume at once?
Only if the underlying storage class and CSI driver support the ReadOnlyMany or ReadWriteMany access mode; the common default, ReadWriteOnce, restricts the volume to a single node at a time regardless of how many pods you try to schedule. Check kubectl get pv for the volume's actual accessModes, and if it only lists ReadWriteOnce, you need to provision the claim from a different storage class to support multiple concurrent readers.
What causes a Multi-Attach error for a volume in Kubernetes?
It means a ReadWriteOnce volume is already mounted by a pod on one node, and a second pod, often a new replica or a rescheduled pod after a node failure, is trying to attach the same volume from a different node. Kubernetes refuses the second attachment to protect data integrity. The fix is either to use an access mode that supports multiple readers, or to ensure only one pod at a time actually needs write access to that specific volume.
Should model weights be mounted read-only or read-write in the pod spec?
Read-only, in almost all serving scenarios. Inference workloads load weights into GPU memory once at startup and never modify the files themselves, so mounting the volume with readOnly true in the volumeMounts section is both safer and, on some storage backends, a requirement for using ReadOnlyMany access mode with multiple simultaneous readers.
Why does my pod mount its volume fine on one node but fail to schedule on another?
This is commonly a zone mismatch on cloud-managed clusters, where the PersistentVolume was provisioned in a specific availability zone and cannot attach to nodes in a different zone. Setting the storage class's volumeBindingMode to WaitForFirstConsumer, instead of provisioning immediately at PVC creation, defers volume creation until a pod is actually scheduled, ensuring the volume lands in the same zone as that pod.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Kubernetes Cluster Cost Calculator
Estimate the true monthly and annual cost of a Kubernetes cluster, including compute, control plane fees, managed service premiums, and utilization waste.
Free ToolOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
Free ToolDocument Ingestion Pipeline Estimator
Estimate total pipeline time from document count, OCR share, and embedding throughput, so ingestion timelines stop being a guess in the project plan.
Related problems
Kubernetes GPU pod stuck in Pending
A GPU pod stays Pending when no node advertises the nvidia.com/gpu resource because the device plugin is down or missing, the pod requests more GPUs than any single node has, or a taint, toleration, or nodeSelector mismatch blocks placement on the GPU pool. Always start with kubectl describe pod, since the Events section states the exact blocking reason rather than leaving you to guess between these causes.
LLM container image is tens of gigabytes and slow to pull
LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.
Model takes minutes to load every time a container restarts
A model that reloads slowly on every container restart almost always means the weight directory is not backed by a persistent volume, so each restart re-downloads or cold-reads the full checkpoint instead of hitting a warm cache. Mount a persistent volume for the model cache path, set HF_HUB_OFFLINE=1 once weights are local, and if load time is still slow, benchmark your storage backend's raw throughput since network storage is a common hidden bottleneck.
Kubernetes readiness probe fails while the model is still loading
LLM service pods get killed or marked unready during startup because default Kubernetes readiness and liveness probes assume a service starts in seconds, while loading multi-gigabyte weights into GPU memory can take minutes. Add a startupProbe sized with a failureThreshold times periodSeconds budget that comfortably exceeds your worst-case load time; Kubernetes suppresses readiness and liveness checks entirely until the startup probe succeeds, which stops premature restarts without needing a fragile fixed initialDelaySeconds guess.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.