# Why Kubernetes persistent volumes fail for model weights, and how to fix it > Source: https://netray.co/troubleshoot/persistent-volume-model-weights > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode. ## Error Title Kubernetes PersistentVolumeClaim errors when serving model weights ## Error Signature Warning FailedMount ... MountVolume.SetUp failed for volume "model-weights": rpc error: code = Internal ## Alternate Signatures - Multi-Attach error for volume: Volume is already exclusively attached to one node - pod has unbound immediate PersistentVolumeClaims ## Category containers ## Stack - kubernetes - docker - huggingface ## Affects Kubernetes deployments serving LLM weights from a PersistentVolume, especially with more than one replica ## Quick Fix ### Fix it in a few steps **Steps:** - Run kubectl describe pvc and check for Multi-Attach or unbound PVC events. - Check kubectl get pv -o jsonpath for accessModes to see what the volume actually supports. - If it is ReadWriteOnce and you need multiple replicas, provision from a ReadOnlyMany-capable storage class instead. - Mount the volume with readOnly true in the pod's volumeMounts, since inference never writes to weights. - Set the storage class's volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches. ## Symptoms - additional replicas fail with Multi-Attach error for volume - pod has unbound immediate PersistentVolumeClaims - MountVolume.SetUp failed rpc error on a subset of nodes - model loads successfully but takes far longer than expected from the mounted volume ## Causes ### Access mode is ReadWriteOnce but multiple pod replicas need to read the same weights **Likelihood:** most-common **Explanation:** The default and most widely available PersistentVolume access mode, ReadWriteOnce, only allows the volume to be mounted read-write by a single node at a time; scaling an inference deployment to multiple replicas across different nodes with a ReadWriteOnce claim causes every replica beyond the first to fail scheduling or mounting, since the volume is already exclusively attached elsewhere. **Fix:** Use a storage class and volume that supports ReadOnlyMany, or ReadWriteMany, for weights that many replicas need to read simultaneously, and mount it read-only, since inference replicas never need to write to the weights directory. **Commands:** - kubectl get pv -o jsonpath='{.spec.accessModes}' - kubectl get storageclass ### Storage class does not support the required access mode at all **Likelihood:** common **Explanation:** Not every storage backend or CSI driver supports ReadOnlyMany or ReadWriteMany; many default cloud block storage classes are strictly ReadWriteOnce by design, so simply changing the accessModes field in your PVC spec does nothing if the underlying storage class and CSI driver never advertise that capability. **Fix:** Check your CSI driver's documentation for which access modes it actually supports, and if ReadOnlyMany is unavailable, switch to a network filesystem class, such as NFS or a cloud file-storage tier, that explicitly supports shared read access. **Commands:** - kubectl describe storageclass ### Storage throughput is too low for the model size, causing slow pod startup rather than an outright error **Likelihood:** common **Explanation:** A PersistentVolume can mount successfully and still be a serving bottleneck if the underlying storage class delivers low sequential read throughput; loading a 70B parameter model from a class rated for typical database workloads rather than large sequential reads can turn a mount success into a multi-minute delay before the pod becomes ready. **Fix:** Choose a storage class rated for high throughput sequential reads for model weight volumes, and benchmark actual throughput rather than relying on the class name alone. **Commands:** - dd if=/mnt/model-weights/testfile of=/dev/null bs=1M status=progress ### PVC and pod are in different availability zones on cloud-managed clusters **Likelihood:** occasional **Explanation:** Many cloud block storage volumes are zone-locked, meaning a PersistentVolume provisioned in one availability zone cannot be attached to a node in a different zone; if the pod gets scheduled to a node in a different zone than the volume, the mount fails even though both the PVC and the pod appear healthy individually. **Fix:** Use a zone-aware storage class with a volumeBindingMode of WaitForFirstConsumer so the volume is provisioned in the same zone as the pod that first claims it, rather than an arbitrary zone chosen at PVC creation time. **Commands:** - kubectl get storageclass -o jsonpath='{.volumeBindingMode}' ## Diagnostics ### Check the PVC's actual status and events **Command:** kubectl describe pvc **Interpretation:** Look for Multi-Attach errors or pending binding events; these point directly at access mode or zone mismatches. ### Check what access modes the underlying PV actually supports **Command:** kubectl get pv -o jsonpath='{.spec.accessModes}' **Interpretation:** If this shows only ReadWriteOnce and you need multiple replicas, this volume type cannot support your scaling plan as configured. ### Check pod scheduling and mount events together **Command:** kubectl describe pod | grep -A10 Events **Interpretation:** FailedMount or FailedAttachVolume events reveal whether the issue is zone affinity, exclusive attachment, or a CSI driver error. ### Benchmark storage throughput directly **Command:** kubectl exec -- dd if=/mnt/model-weights/testfile of=/dev/null bs=1M status=progress **Interpretation:** Confirms whether a successful mount is still a performance bottleneck for large model loads. ## Prevention - Choose ReadOnlyMany-capable storage classes upfront for any model weight volume served by multiple replicas - Mount weight volumes read-only in the pod spec, since inference workloads never need write access to weights - Use volumeBindingMode WaitForFirstConsumer to avoid zone mismatches between PVCs and pods - Benchmark storage class throughput against your largest model size before committing to it in production ## When To Escalate If your storage backend genuinely cannot support ReadOnlyMany at the scale you need, or throughput remains a bottleneck even on higher tiers, this becomes a storage architecture decision, a dedicated model artifact cache, node-local NVMe with a sync sidecar, or a different CSI driver entirely, rather than a PVC spec tweak. ## Faq ### Can multiple pods read from the same PersistentVolume at once? **Answer:** Only if the underlying storage class and CSI driver support the ReadOnlyMany or ReadWriteMany access mode; the common default, ReadWriteOnce, restricts the volume to a single node at a time regardless of how many pods you try to schedule. Check kubectl get pv for the volume's actual accessModes, and if it only lists ReadWriteOnce, you need to provision the claim from a different storage class to support multiple concurrent readers. ### What causes a Multi-Attach error for a volume in Kubernetes? **Answer:** It means a ReadWriteOnce volume is already mounted by a pod on one node, and a second pod, often a new replica or a rescheduled pod after a node failure, is trying to attach the same volume from a different node. Kubernetes refuses the second attachment to protect data integrity. The fix is either to use an access mode that supports multiple readers, or to ensure only one pod at a time actually needs write access to that specific volume. ### Should model weights be mounted read-only or read-write in the pod spec? **Answer:** Read-only, in almost all serving scenarios. Inference workloads load weights into GPU memory once at startup and never modify the files themselves, so mounting the volume with readOnly true in the volumeMounts section is both safer and, on some storage backends, a requirement for using ReadOnlyMany access mode with multiple simultaneous readers. ### Why does my pod mount its volume fine on one node but fail to schedule on another? **Answer:** This is commonly a zone mismatch on cloud-managed clusters, where the PersistentVolume was provisioned in a specific availability zone and cannot attach to nodes in a different zone. Setting the storage class's volumeBindingMode to WaitForFirstConsumer, instead of provisioning immediately at PVC creation, defers volume creation until a pod is actually scheduled, ensuring the volume lands in the same zone as that pod. ## Search terms kubernetes pvc model weights multi-attach error, persistentvolumeclaim readonlymany model weights, kubernetes pod unbound immediate persistentvolumeclaims, mountvolume setup failed rpc error, storage class readwriteonce vs readonlymany, pvc zone mismatch kubernetes --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json