Why your LLM container image is so large, and how to shrink it
Error response from daemon: no space left on device (during docker pull)
Also appears as
- image size 45GB, docker push timing out
- kubelet: Failed to pull image: context deadline exceeded
Short answer
LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.
Affects: Any containerized LLM serving image that bakes model weights into a layer, common with 7B parameter models and larger
Shrink it fast
- 1Run docker history <image> to find which layer is largest, usually the weights COPY.
- 2Delete any COPY or ADD instruction that adds model weight files to the image.
- 3Mount weights instead, from a volume, object storage sync, or a Hugging Face cache directory, at container startup.
- 4Switch the final stage's CUDA base image from -devel to -runtime if you are not compiling CUDA code at build time.
- 5Convert the Dockerfile to a multi-stage build so builder-only tools never reach the final image.
- 6Rebuild and compare size with docker images <image> against the previous version.
How to confirm this is your problem
- docker pull or push takes many minutes even on a fast network
- no space left on device errors on GPU nodes with many pods
- kubelet Failed to pull image: context deadline exceeded
- every new pod replica re-downloads the full multi-gigabyte image
Root causes and fixes
Model weights baked directly into the image layer instead of mounted at runtime
Copying a multi-gigabyte safetensors or GGUF checkpoint into the image with a COPY instruction embeds it permanently in a layer; every rebuild that touches any earlier layer re-uploads the entire weight blob, image pulls take minutes even on fast networks, and node disk pressure grows because every replica caches a redundant full copy of the weights.
Fix: Remove weights from the Dockerfile entirely and mount them from an external volume, object storage sync, or a Hugging Face cache directory at container startup instead, keeping the image itself limited to code and the inference runtime.
docker history <image> --format "{{.Size}}\t{{.CreatedBy}}"
du -sh /var/lib/docker/overlay2/*No multi-stage build, so build tools and intermediate artifacts ship in the final image
A single-stage Dockerfile that installs compilers, CUDA devel headers, pip build caches, and test dependencies leaves all of it in the final layer even though none of it is needed at runtime, often adding several gigabytes of dead weight that has nothing to do with the model itself.
Fix: Split the Dockerfile into a builder stage that compiles wheels or extensions and a slim final stage that copies only the built artifacts and runtime dependencies forward with COPY --from=builder.
docker build --target builder -t myimage:builder . docker build -t myimage:runtime .
Using a devel CUDA base image instead of runtime for production serving
NVIDIA's devel CUDA images include the full toolkit, nvcc, headers, and static libraries needed to compile CUDA code, while the runtime images ship only the shared libraries needed to execute already-compiled CUDA binaries; using devel in production adds gigabytes of compiler tooling that inference never touches.
Fix: Use the devel image only in a build stage if you compile custom CUDA extensions, and switch the final serving stage to the matching runtime tag.
docker pull nvidia/cuda:12.4.1-runtime-ubuntu22.04
No layer caching strategy, so pip/apt caches and duplicate dependency layers pile up
Installing Python packages without cleaning pip's cache, or running apt-get install without cleaning apt lists afterward, leaves downloaded package archives sitting in the image layer permanently, which on a typical ML stack with CUDA-linked wheels can add a gigabyte or more of pure waste.
Fix: Add --no-cache-dir to pip installs and clean apt lists in the same RUN instruction they were created in, since layer size is fixed once a RUN completes, not once the Dockerfile finishes.
RUN pip install --no-cache-dir -r requirements.txt RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
Diagnostic commands
See which layers are largest
docker history <image> --format "table {{.Size}}\t{{.CreatedBy}}"Layers over a gigabyte usually correspond to a weights COPY, a pip install without cache cleanup, or an apt-get without cleanup; target the biggest offenders first.
Check total image size
docker images <image>
Compare against a baseline: a code-plus-runtime image for a serving stack should typically be under a few gigabytes without weights.
Inspect what is actually inside a layer
docker run --rm <image> du -sh /app /root/.cache
Reveals whether weights, pip caches, or build artifacts ended up somewhere unexpected inside the final filesystem.
Check node disk pressure from cached images
kubectl describe node <node> | grep -A3 Conditions
DiskPressure=True on GPU nodes is often caused by every replica pulling and caching its own multi-gigabyte image with baked-in weights.
Stopping it from happening again
- Never COPY model weights into a Dockerfile; treat weights as data, mounted or synced at runtime, always
- Adopt multi-stage builds as the default template for every ML serving image, not an afterthought
- Standardize on runtime, not devel, CUDA base images for anything that does not compile CUDA code at build time
- Set up a registry image size budget and fail CI if an image exceeds it, catching regressions before they reach production
When this becomes an architecture problem
If image size problems keep recurring across many models and teams, or you are about to scale from a handful of pods to a large fleet where redundant multi-gigabyte pulls become a real bandwidth and disk cost, it is worth designing a shared weight-caching architecture rather than fixing one Dockerfile at a time.
Frequently asked questions
Should model weights ever be baked into a container image?
Generally no, for anything beyond a small demo. Weights change independently of code, are often many gigabytes, and baking them into an image layer means every code change forces a full weight re-upload and every replica caches a redundant copy on its node's disk. The standard pattern is to keep weights in object storage or a shared volume and have the container pull or mount them at startup, so the image itself stays small and stable.
What is the actual size difference between a CUDA devel and runtime image?
The devel variant includes the nvcc compiler, headers, and static libraries needed to build CUDA code from source, which typically adds several gigabytes over the runtime variant that ships only the shared libraries needed to run already-compiled CUDA binaries. Unless your image compiles custom CUDA kernels or extensions at build time, the runtime tag is sufficient and meaningfully smaller.
Why does my image keep growing even after I delete files in a later RUN step?
Docker layers are immutable once written; deleting a file in a later layer only hides it from the final filesystem view, it does not remove the bytes from the image, which are still present in the earlier layer. The fix is to install, use, and clean up in the same RUN instruction, or use a multi-stage build so the artifact never enters the final image's layer history at all.
How much does image size actually affect cold-start time in Kubernetes?
Significantly, especially on autoscaled GPU pools where new nodes pull the image fresh with no cache. A multi-gigabyte image with baked-in weights can take minutes to pull before the container even starts, directly adding to pod startup latency and to how long autoscaling takes to add capacity under load. Separating weights from the image and mounting them from fast shared storage removes this bottleneck entirely.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
On-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
Free ToolDocument Ingestion Pipeline Estimator
Estimate total pipeline time from document count, OCR share, and embedding throughput, so ingestion timelines stop being a guess in the project plan.
Free ToolKubernetes Cluster Cost Calculator
Estimate the true monthly and annual cost of a Kubernetes cluster, including compute, control plane fees, managed service premiums, and utilization waste.
Related problems
Model takes minutes to load every time a container restarts
A model that reloads slowly on every container restart almost always means the weight directory is not backed by a persistent volume, so each restart re-downloads or cold-reads the full checkpoint instead of hitting a warm cache. Mount a persistent volume for the model cache path, set HF_HUB_OFFLINE=1 once weights are local, and if load time is still slow, benchmark your storage backend's raw throughput since network storage is a common hidden bottleneck.
Kubernetes PersistentVolumeClaim errors when serving model weights
PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode.
Setting up a container registry for air-gapped Kubernetes deployments
Air-gapped Kubernetes clusters cannot reach public registries or the Hugging Face Hub, so image pulls fail at DNS resolution unless a local registry mirror is stood up ahead of time and populated from a connected staging environment. Mirror container images and model weights as two distinct pipeline steps, distribute the internal registry's CA certificate to every node, and sign mirrored artifacts so provenance, not just reachability, is auditable, which is a mandatory control in ITAR and CMMC environments.
HuggingFace model download is extremely slow or stalls partway through
Slow or stalled HuggingFace downloads are usually caused by huggingface_hub's default transfer path not using parallel chunked downloads, a corporate proxy or firewall throttling or dropping long-lived connections, or genuinely insufficient bandwidth for a hundreds-of-gigabytes model. Enable hf_transfer for a much faster Rust-based parallel downloader, rely on the client's built-in resume behavior rather than restarting from zero, and for regulated or air-gapped sites, download once and mirror internally instead of pulling repeatedly over the internet.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideAir-Gapped LLM Deployment Patterns That Actually Work
Air-gapped LLM deployment patterns that work: offline model transfer, update workflows, monitoring without telemetry, and CMMC-ready architectures.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.