Why vLLM runs out of memory during startup, and how to fix it
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X GiB (GPU 0; Y GiB total capacity; Z GiB already allocated)
Also appears as
- RuntimeError: CUDA error: out of memory (during engine initialization or profiling run)
- torch.OutOfMemoryError during Loading model weights or Profiling run
Short answer
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
Affects: vLLM 0.3 and later, any GPU; the OOM happens during weight loading or the initial memory-profiling pass, before the server accepts a single request
Fastest path to a clean startup
- 1Check nvidia-smi for other processes holding VRAM before you even launch vLLM, and kill anything unnecessary.
- 2Lower --gpu-memory-utilization from a too-aggressive value, for example from 0.95 to 0.85, to leave headroom for CUDA context and fragmentation.
- 3Lower --max-model-len to reduce the KV cache pool size vLLM tries to preallocate.
- 4If weights alone barely fit, switch to a quantized checkpoint with --quantization awq, gptq, or fp8, or add --tensor-parallel-size across more GPUs.
- 5Relaunch and watch the log for Loading model weights followed by the KV cache allocation line completing without error.
How to confirm this is your problem
- Crash happens during a clearly logged phase like model loading or the memory profiling run, never after Uvicorn running.
- Error reports a specific requested allocation size that's larger than what nvidia-smi shows as free.
- Same command works on an otherwise-identical GPU with more VRAM.
- Reducing --max-model-len or --gpu-memory-utilization makes the exact same model start successfully.
Root causes and fixes
gpu_memory_utilization set too high for what's actually free
vLLM computes its target memory budget as gpu_memory_utilization times total GPU memory, not currently-free memory. If other processes, CUDA context overhead, or memory fragmentation have already consumed part of that total, the actual allocation attempt exceeds what's really available and PyTorch raises OOM during the profiling run.
Fix: Drop --gpu-memory-utilization by 0.05 to 0.10 to build in headroom for CUDA context and fragmentation, then retest.
nvidia-smi vllm serve MODEL_ID --gpu-memory-utilization 0.85
KV cache pool preallocated for a max_model_len that leaves no slack after weights
After loading weights, vLLM runs a profiling pass to determine how much memory is left for a KV cache pool sized to fit max_model_len tokens per sequence times max_num_seqs. If that combination is large relative to remaining VRAM, the profiling allocation itself OOMs.
Fix: Lower --max-model-len and/or --max-num-seqs so the KV cache pool vLLM tries to preallocate fits the actual remaining VRAM.
vllm serve MODEL_ID --max-model-len 8192 --max-num-seqs 64
Weights alone consume nearly all VRAM at the requested dtype
Serving a large model in fp16 or bf16 on a card sized just barely for the raw parameter count leaves almost nothing for KV cache or activation buffers, so even a minimal cache pool allocation fails.
Fix: Use a quantized checkpoint such as AWQ, GPTQ, or FP8 to shrink weight memory, or shard weights across more GPUs with --tensor-parallel-size.
vllm serve MODEL_ID --quantization awq vllm serve MODEL_ID --tensor-parallel-size 2
Other processes already holding GPU memory
The OOM math assumes the process has exclusive access to the fraction of VRAM implied by gpu_memory_utilization. Any other resident CUDA context, such as notebooks, a display server, or a zombie previous vLLM process, reduces true free memory below what vLLM calculated should be available.
Fix: Check nvidia-smi for other memory-resident processes and terminate them before starting vLLM.
nvidia-smi
CUDA graph capture adding extra fixed memory overhead near the limit
vLLM captures CUDA graphs for common batch sizes to speed up decoding, which reserves additional fixed memory on top of weights and KV cache. On GPUs already close to their limit, this overhead alone can tip a borderline startup into OOM.
Fix: Pass --enforce-eager to skip CUDA graph capture as a diagnostic step, trading some throughput for a smaller memory footprint.
vllm serve MODEL_ID --enforce-eager
Diagnostic commands
Check free VRAM immediately before launch
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
If used memory is non-trivial before vLLM even starts, another process is eating into the budget vLLM's gpu_memory_utilization math assumes is free.
Watch which startup phase fails
vllm serve MODEL_ID --max-model-len 4096 2>&1 | tee startup.log
If the OOM happens during Loading model weights, it's a weights-vs-VRAM problem; if it happens during the memory profiling or KV cache allocation step, it's a max-model-len or max-num-seqs sizing problem.
Test with a minimal, known-safe config
vllm serve MODEL_ID --gpu-memory-utilization 0.7 --max-model-len 2048 --enforce-eager
If this succeeds, walk the values back up, utilization first, then max-model-len, then remove enforce-eager, one at a time to find the actual ceiling on your hardware.
Stopping it from happening again
- Size GPUs against a KV cache calculator using your real max_model_len and expected concurrency before deployment, not the model's advertised maximum context.
- Leave a buffer below 1.0 for gpu_memory_utilization, typically 0.85 to 0.95, to absorb fragmentation and CUDA context overhead.
- Add a pre-deployment smoke test that launches with production flags in staging before every model or vLLM version change.
- Monitor nvidia-smi memory usage over time in production to catch creeping fragmentation before it causes a restart failure.
When this becomes an architecture problem
If you've dropped utilization, capped context, quantized weights, and disabled CUDA graphs and it still won't start, the model plus your required context and concurrency simply doesn't fit on this GPU: that's a hardware sizing decision such as bigger cards, more GPUs, or a smaller model, not a flag to keep tuning.
Frequently asked questions
Why does the OOM happen at startup instead of when real traffic arrives?
vLLM preallocates its entire KV cache pool up front during a profiling pass right after loading weights, specifically so it never has to OOM mid-request later. That design trades a startup-time failure for guaranteed stability once the server is actually running.
Is enforce-eager a permanent fix?
Treat it as a diagnostic and stopgap. It removes CUDA graph capture overhead, which does free some memory, but you lose some decoding throughput. Once you've found headroom, prefer fixing the underlying sizing, such as max-model-len, quantization, or more GPUs, over running eager mode permanently.
Why did the same command work last week and OOM today?
Usually another process is now resident on the GPU, such as a notebook, a second server, or a zombie process from a previous crash, or memory has fragmented over multiple restarts without a full driver reset. Check nvidia-smi for unexpected occupants first.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Related problems
vLLM: model's max seq len is larger than the KV cache can hold
vLLM preallocates a fixed KV cache pool sized by gpu_memory_utilization and refuses to start a context length whose worst case (batch x max sequence length) doesn't fit in that pool. Fix it by raising --gpu-memory-utilization toward 0.9-0.95, lowering --max-model-len to what you actually need, or adding a GPU/quantizing weights to leave more headroom for cache.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
vLLM server won't start (port in use, auth, VRAM, or unsupported architecture)
vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.
GPU memory stays full after inference finishes
This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideThe LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.