# Why your LLM container image is so large, and how to shrink it > Source: https://netray.co/troubleshoot/container-image-too-large-llm > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code. ## Error Title LLM container image is tens of gigabytes and slow to pull ## Error Signature Error response from daemon: no space left on device (during docker pull) ## Alternate Signatures - image size 45GB, docker push timing out - kubelet: Failed to pull image: context deadline exceeded ## Category containers ## Stack - docker - kubernetes - huggingface ## Affects Any containerized LLM serving image that bakes model weights into a layer, common with 7B parameter models and larger ## Quick Fix ### Shrink it fast **Steps:** - Run docker history to find which layer is largest, usually the weights COPY. - Delete any COPY or ADD instruction that adds model weight files to the image. - Mount weights instead, from a volume, object storage sync, or a Hugging Face cache directory, at container startup. - Switch the final stage's CUDA base image from -devel to -runtime if you are not compiling CUDA code at build time. - Convert the Dockerfile to a multi-stage build so builder-only tools never reach the final image. - Rebuild and compare size with docker images against the previous version. ## Symptoms - docker pull or push takes many minutes even on a fast network - no space left on device errors on GPU nodes with many pods - kubelet Failed to pull image: context deadline exceeded - every new pod replica re-downloads the full multi-gigabyte image ## Causes ### Model weights baked directly into the image layer instead of mounted at runtime **Likelihood:** most-common **Explanation:** Copying a multi-gigabyte safetensors or GGUF checkpoint into the image with a COPY instruction embeds it permanently in a layer; every rebuild that touches any earlier layer re-uploads the entire weight blob, image pulls take minutes even on fast networks, and node disk pressure grows because every replica caches a redundant full copy of the weights. **Fix:** Remove weights from the Dockerfile entirely and mount them from an external volume, object storage sync, or a Hugging Face cache directory at container startup instead, keeping the image itself limited to code and the inference runtime. **Commands:** - docker history --format "{{.Size}}\t{{.CreatedBy}}" - du -sh /var/lib/docker/overlay2/* ### No multi-stage build, so build tools and intermediate artifacts ship in the final image **Likelihood:** common **Explanation:** A single-stage Dockerfile that installs compilers, CUDA devel headers, pip build caches, and test dependencies leaves all of it in the final layer even though none of it is needed at runtime, often adding several gigabytes of dead weight that has nothing to do with the model itself. **Fix:** Split the Dockerfile into a builder stage that compiles wheels or extensions and a slim final stage that copies only the built artifacts and runtime dependencies forward with COPY --from=builder. **Commands:** - docker build --target builder -t myimage:builder . - docker build -t myimage:runtime . ### Using a devel CUDA base image instead of runtime for production serving **Likelihood:** common **Explanation:** NVIDIA's devel CUDA images include the full toolkit, nvcc, headers, and static libraries needed to compile CUDA code, while the runtime images ship only the shared libraries needed to execute already-compiled CUDA binaries; using devel in production adds gigabytes of compiler tooling that inference never touches. **Fix:** Use the devel image only in a build stage if you compile custom CUDA extensions, and switch the final serving stage to the matching runtime tag. **Commands:** - docker pull nvidia/cuda:12.4.1-runtime-ubuntu22.04 ### No layer caching strategy, so pip/apt caches and duplicate dependency layers pile up **Likelihood:** occasional **Explanation:** Installing Python packages without cleaning pip's cache, or running apt-get install without cleaning apt lists afterward, leaves downloaded package archives sitting in the image layer permanently, which on a typical ML stack with CUDA-linked wheels can add a gigabyte or more of pure waste. **Fix:** Add --no-cache-dir to pip installs and clean apt lists in the same RUN instruction they were created in, since layer size is fixed once a RUN completes, not once the Dockerfile finishes. **Commands:** - RUN pip install --no-cache-dir -r requirements.txt - RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/* ## Diagnostics ### See which layers are largest **Command:** docker history --format "table {{.Size}}\t{{.CreatedBy}}" **Interpretation:** Layers over a gigabyte usually correspond to a weights COPY, a pip install without cache cleanup, or an apt-get without cleanup; target the biggest offenders first. ### Check total image size **Command:** docker images **Interpretation:** Compare against a baseline: a code-plus-runtime image for a serving stack should typically be under a few gigabytes without weights. ### Inspect what is actually inside a layer **Command:** docker run --rm du -sh /app /root/.cache **Interpretation:** Reveals whether weights, pip caches, or build artifacts ended up somewhere unexpected inside the final filesystem. ### Check node disk pressure from cached images **Command:** kubectl describe node | grep -A3 Conditions **Interpretation:** DiskPressure=True on GPU nodes is often caused by every replica pulling and caching its own multi-gigabyte image with baked-in weights. ## Prevention - Never COPY model weights into a Dockerfile; treat weights as data, mounted or synced at runtime, always - Adopt multi-stage builds as the default template for every ML serving image, not an afterthought - Standardize on runtime, not devel, CUDA base images for anything that does not compile CUDA code at build time - Set up a registry image size budget and fail CI if an image exceeds it, catching regressions before they reach production ## When To Escalate If image size problems keep recurring across many models and teams, or you are about to scale from a handful of pods to a large fleet where redundant multi-gigabyte pulls become a real bandwidth and disk cost, it is worth designing a shared weight-caching architecture rather than fixing one Dockerfile at a time. ## Faq ### Should model weights ever be baked into a container image? **Answer:** Generally no, for anything beyond a small demo. Weights change independently of code, are often many gigabytes, and baking them into an image layer means every code change forces a full weight re-upload and every replica caches a redundant copy on its node's disk. The standard pattern is to keep weights in object storage or a shared volume and have the container pull or mount them at startup, so the image itself stays small and stable. ### What is the actual size difference between a CUDA devel and runtime image? **Answer:** The devel variant includes the nvcc compiler, headers, and static libraries needed to build CUDA code from source, which typically adds several gigabytes over the runtime variant that ships only the shared libraries needed to run already-compiled CUDA binaries. Unless your image compiles custom CUDA kernels or extensions at build time, the runtime tag is sufficient and meaningfully smaller. ### Why does my image keep growing even after I delete files in a later RUN step? **Answer:** Docker layers are immutable once written; deleting a file in a later layer only hides it from the final filesystem view, it does not remove the bytes from the image, which are still present in the earlier layer. The fix is to install, use, and clean up in the same RUN instruction, or use a multi-stage build so the artifact never enters the final image's layer history at all. ### How much does image size actually affect cold-start time in Kubernetes? **Answer:** Significantly, especially on autoscaled GPU pools where new nodes pull the image fresh with no cache. A multi-gigabyte image with baked-in weights can take minutes to pull before the container even starts, directly adding to pod startup latency and to how long autoscaling takes to add capacity under load. Separating weights from the image and mounting them from fast shared storage removes this bottleneck entirely. ## Search terms docker image too large llm, reduce llm container image size, cuda devel vs runtime image size, no space left on device docker pull, multi-stage docker build cuda, kubelet failed to pull image context deadline exceeded, model weights in docker image --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json