# Why your model loads slowly inside a container every time it restarts, and how to fix it > Source: https://netray.co/troubleshoot/model-loading-slow-in-container > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer A model that reloads slowly on every container restart almost always means the weight directory is not backed by a persistent volume, so each restart re-downloads or cold-reads the full checkpoint instead of hitting a warm cache. Mount a persistent volume for the model cache path, set HF_HUB_OFFLINE=1 once weights are local, and if load time is still slow, benchmark your storage backend's raw throughput since network storage is a common hidden bottleneck. ## Error Title Model takes minutes to load every time a container restarts ## Error Signature Loading checkpoint shards: 100%|... [took 8+ minutes] ## Alternate Signatures - INFO vllm: Loading model weights took 420.35 seconds - pod restart causes multi-minute delay before readiness probe passes ## Category containers ## Stack - docker - kubernetes - huggingface - vllm ## Affects Any containerized LLM serving deployment, especially on Kubernetes with pod autoscaling or frequent restarts ## Quick Fix ### Fastest path to fast restarts **Steps:** - Check whether the model cache path is mounted from a persistent volume with kubectl get pod -o jsonpath. - If it is emptyDir or unmounted, add a PersistentVolumeClaim or hostPath mount at that path. - Set HF_HOME to the mounted path and HF_HUB_OFFLINE=1 once weights are cached there. - Benchmark raw read throughput on the storage backend with dd to confirm it is not the bottleneck. - For frequent restarts, add an init container that prefetches weights before the main container starts. ## Symptoms - every pod restart re-triggers a multi-minute model load - readiness or startup probes time out specifically after a restart, not on first deploy - load time is the same whether the model was just loaded minutes ago or not at all - logs show weight download progress on restarts, not just at first launch ## Causes ### Weights are re-downloaded or re-pulled from a cold cache on every container restart **Likelihood:** most-common **Explanation:** If the Hugging Face cache directory or model volume is ephemeral, part of the container's writable layer rather than a persistent mount, every restart, pod reschedule, or autoscale event starts from zero: the full multi-gigabyte weight download happens again before inference can start, even though nothing about the model changed. **Fix:** Mount a persistent volume or node-local cache directory for HF_HOME so weights survive container restarts, and set HF_HUB_OFFLINE=1 once weights are cached to skip hub metadata checks entirely. **Commands:** - export HF_HOME=/mnt/model-cache - export HF_HUB_OFFLINE=1 - docker run -v model-cache:/mnt/model-cache myimage ### Page cache is lost on container restart, forcing a cold read from disk or network storage **Likelihood:** common **Explanation:** The Linux page cache that made a previous load fast lives in host RAM, not in the container; when a pod is rescheduled to a different node, common with Kubernetes autoscaling or node drains, that node has never read these weight files before, so even a fast local disk read still has to traverse the actual storage backend cold, and if that backend is slow network storage, load time balloons further. **Fix:** For workloads that restart frequently, prefer node-local NVMe caching or a warm pool of pinned nodes for large models, and consider a prefetch init container that primes the cache before the main container starts serving traffic. **Commands:** ### Network storage throughput is the actual bottleneck, not the application **Likelihood:** common **Explanation:** A 70B parameter model in FP16 is on the order of 140GB; reading that from an NFS or generic network block store with modest throughput can take several minutes purely on I/O, regardless of CPU, GPU, or how efficient the loading code is, and this often gets misdiagnosed as an application or framework problem. **Fix:** Benchmark raw read throughput from the actual storage backend independent of model loading, and if it is the bottleneck, move to a storage class or backend rated for high sequential throughput, or use a local NVMe cache in front of it. **Commands:** - dd if=/mnt/models/testfile of=/dev/null bs=1M status=progress ### No volume cache at all, model directory lives inside the container's ephemeral filesystem **Likelihood:** occasional **Explanation:** Some Dockerfiles or deployment manifests never define a volume for the model cache path at all, meaning weights are downloaded into the container's own writable layer each run; this is functionally identical to having no cache, and it also means disk usage accumulates against the container's own storage quota rather than a shared, cleanable volume. **Fix:** Explicitly define a PersistentVolumeClaim or named volume mounted at the model cache path in every deployment manifest, never leaving it as an implicit ephemeral path. **Commands:** ## Diagnostics ### Time the actual load phase **Command:** grep -i 'loading' pod-logs.txt **Interpretation:** Frameworks like vLLM and transformers log an explicit load duration; compare it across restarts to see if it is consistently slow, storage-bound, or only slow on cold nodes, cache-bound. ### Check whether the cache volume is actually persistent **Command:** kubectl get pod -o jsonpath='{.spec.volumes}' **Interpretation:** If the model path maps to emptyDir or no volume at all rather than a PVC or hostPath, the cache never survives a restart by design. ### Benchmark raw storage throughput **Command:** dd if=/mnt/models/ of=/dev/null bs=1M status=progress **Interpretation:** Sub-500MB/s sustained reads on a large model directory point to storage, not application code, as the real bottleneck. ### Check if offline mode is set **Command:** echo $HF_HUB_OFFLINE **Interpretation:** If unset, the loader may spend extra time on hub metadata or version checks even when weights are already cached locally. ## Prevention - Always mount model weight directories from a persistent volume, never the container's writable layer - Set HF_HUB_OFFLINE=1 and HF_HUB_DISABLE_TELEMETRY=1 once weights are locally cached to remove network round trips from the load path - Benchmark your storage backend's throughput against your model size before choosing it, not after latency complaints - Use an init container to prefetch and warm weights onto local node storage before the main serving container starts ## When To Escalate If model load time is dominated by storage throughput rather than caching gaps, and multiple models or replicas need fast cold starts simultaneously, this becomes a storage architecture decision, local NVMe tiering, a dedicated model artifact store, or a warm pod pool, rather than something a volume mount alone will fix. ## Faq ### Why is my model slow to load only after a Kubernetes pod restart, not the first time? **Answer:** This is almost always a caching problem, not a code problem. If the model directory is not backed by a persistent volume, every restart starts the download from scratch. Even with a persistent volume, if the pod lands on a different node than before, that node's page cache and any local disk cache are cold, so the read has to go all the way back to the underlying storage backend, which is slower than a warm local read. ### Does HF_HUB_OFFLINE actually make loading faster, or just prevent network errors? **Answer:** Both. With HF_HUB_OFFLINE=1, the Hugging Face libraries skip contacting the hub to check for newer file versions or metadata, which removes network round trips from the load path entirely and also prevents failures when the hub is unreachable in an air-gapped or restricted network. It only helps once weights are already present locally; it does not download anything itself. ### What is the fastest way to make repeated container restarts load a large model quickly? **Answer:** Mount the model cache directory from a persistent volume so weights survive restarts, prefer node-local NVMe-backed storage classes over generic network storage for that volume, set HF_HUB_OFFLINE=1, and where restarts are frequent, consider pinning replicas to a warm pool of nodes that already have the weights in local page cache rather than letting the scheduler place them on cold nodes. ### Is it worth keeping the model loaded in a separate long-running process instead of reloading per container? **Answer:** For very frequently restarted or scaled services, yes; some teams run a persistent model server process on the node and have short-lived containers proxy to it, avoiding reload entirely. For most production serving setups, though, a well-cached persistent volume plus offline mode gets load time down to a level where a dedicated always-on process is unnecessary complexity. ## Search terms model loading slow in container, vllm loading model weights took seconds, kubernetes pod restart slow model load, hf_hub_offline model cache, container cold start llm weights, loading checkpoint shards slow, persistent volume model cache kubernetes --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json