# Why vLLM runs out of memory during startup, and how to fix it > Source: https://netray.co/troubleshoot/vllm-oom-at-startup > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs. ## Error Title vLLM runs out of memory during startup, before serving any requests ## Error Signature torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X GiB (GPU 0; Y GiB total capacity; Z GiB already allocated) ## Alternate Signatures - RuntimeError: CUDA error: out of memory (during engine initialization or profiling run) - torch.OutOfMemoryError during Loading model weights or Profiling run ## Category serving ## Stack - vllm - cuda - pytorch ## Affects vLLM 0.3 and later, any GPU; the OOM happens during weight loading or the initial memory-profiling pass, before the server accepts a single request ## Quick Fix ### Fastest path to a clean startup **Steps:** - Check nvidia-smi for other processes holding VRAM before you even launch vLLM, and kill anything unnecessary. - Lower --gpu-memory-utilization from a too-aggressive value, for example from 0.95 to 0.85, to leave headroom for CUDA context and fragmentation. - Lower --max-model-len to reduce the KV cache pool size vLLM tries to preallocate. - If weights alone barely fit, switch to a quantized checkpoint with --quantization awq, gptq, or fp8, or add --tensor-parallel-size across more GPUs. - Relaunch and watch the log for Loading model weights followed by the KV cache allocation line completing without error. ## Symptoms - Crash happens during a clearly logged phase like model loading or the memory profiling run, never after Uvicorn running. - Error reports a specific requested allocation size that's larger than what nvidia-smi shows as free. - Same command works on an otherwise-identical GPU with more VRAM. - Reducing --max-model-len or --gpu-memory-utilization makes the exact same model start successfully. ## Causes ### gpu_memory_utilization set too high for what's actually free **Likelihood:** most-common **Explanation:** vLLM computes its target memory budget as gpu_memory_utilization times total GPU memory, not currently-free memory. If other processes, CUDA context overhead, or memory fragmentation have already consumed part of that total, the actual allocation attempt exceeds what's really available and PyTorch raises OOM during the profiling run. **Fix:** Drop --gpu-memory-utilization by 0.05 to 0.10 to build in headroom for CUDA context and fragmentation, then retest. **Commands:** - nvidia-smi - vllm serve MODEL_ID --gpu-memory-utilization 0.85 ### KV cache pool preallocated for a max_model_len that leaves no slack after weights **Likelihood:** common **Explanation:** After loading weights, vLLM runs a profiling pass to determine how much memory is left for a KV cache pool sized to fit max_model_len tokens per sequence times max_num_seqs. If that combination is large relative to remaining VRAM, the profiling allocation itself OOMs. **Fix:** Lower --max-model-len and/or --max-num-seqs so the KV cache pool vLLM tries to preallocate fits the actual remaining VRAM. **Commands:** - vllm serve MODEL_ID --max-model-len 8192 --max-num-seqs 64 ### Weights alone consume nearly all VRAM at the requested dtype **Likelihood:** common **Explanation:** Serving a large model in fp16 or bf16 on a card sized just barely for the raw parameter count leaves almost nothing for KV cache or activation buffers, so even a minimal cache pool allocation fails. **Fix:** Use a quantized checkpoint such as AWQ, GPTQ, or FP8 to shrink weight memory, or shard weights across more GPUs with --tensor-parallel-size. **Commands:** - vllm serve MODEL_ID --quantization awq - vllm serve MODEL_ID --tensor-parallel-size 2 ### Other processes already holding GPU memory **Likelihood:** occasional **Explanation:** The OOM math assumes the process has exclusive access to the fraction of VRAM implied by gpu_memory_utilization. Any other resident CUDA context, such as notebooks, a display server, or a zombie previous vLLM process, reduces true free memory below what vLLM calculated should be available. **Fix:** Check nvidia-smi for other memory-resident processes and terminate them before starting vLLM. **Commands:** - nvidia-smi ### CUDA graph capture adding extra fixed memory overhead near the limit **Likelihood:** rare **Explanation:** vLLM captures CUDA graphs for common batch sizes to speed up decoding, which reserves additional fixed memory on top of weights and KV cache. On GPUs already close to their limit, this overhead alone can tip a borderline startup into OOM. **Fix:** Pass --enforce-eager to skip CUDA graph capture as a diagnostic step, trading some throughput for a smaller memory footprint. **Commands:** - vllm serve MODEL_ID --enforce-eager ## Diagnostics ### Check free VRAM immediately before launch **Command:** nvidia-smi --query-gpu=memory.used,memory.total --format=csv **Interpretation:** If used memory is non-trivial before vLLM even starts, another process is eating into the budget vLLM's gpu_memory_utilization math assumes is free. ### Watch which startup phase fails **Command:** vllm serve MODEL_ID --max-model-len 4096 2>&1 | tee startup.log **Interpretation:** If the OOM happens during Loading model weights, it's a weights-vs-VRAM problem; if it happens during the memory profiling or KV cache allocation step, it's a max-model-len or max-num-seqs sizing problem. ### Test with a minimal, known-safe config **Command:** vllm serve MODEL_ID --gpu-memory-utilization 0.7 --max-model-len 2048 --enforce-eager **Interpretation:** If this succeeds, walk the values back up, utilization first, then max-model-len, then remove enforce-eager, one at a time to find the actual ceiling on your hardware. ## Prevention - Size GPUs against a KV cache calculator using your real max_model_len and expected concurrency before deployment, not the model's advertised maximum context. - Leave a buffer below 1.0 for gpu_memory_utilization, typically 0.85 to 0.95, to absorb fragmentation and CUDA context overhead. - Add a pre-deployment smoke test that launches with production flags in staging before every model or vLLM version change. - Monitor nvidia-smi memory usage over time in production to catch creeping fragmentation before it causes a restart failure. ## When To Escalate If you've dropped utilization, capped context, quantized weights, and disabled CUDA graphs and it still won't start, the model plus your required context and concurrency simply doesn't fit on this GPU: that's a hardware sizing decision such as bigger cards, more GPUs, or a smaller model, not a flag to keep tuning. ## Faq ### Why does the OOM happen at startup instead of when real traffic arrives? **Answer:** vLLM preallocates its entire KV cache pool up front during a profiling pass right after loading weights, specifically so it never has to OOM mid-request later. That design trades a startup-time failure for guaranteed stability once the server is actually running. ### Is enforce-eager a permanent fix? **Answer:** Treat it as a diagnostic and stopgap. It removes CUDA graph capture overhead, which does free some memory, but you lose some decoding throughput. Once you've found headroom, prefer fixing the underlying sizing, such as max-model-len, quantization, or more GPUs, over running eager mode permanently. ### Why did the same command work last week and OOM today? **Answer:** Usually another process is now resident on the GPU, such as a notebook, a second server, or a zombie process from a previous crash, or memory has fragmented over multiple restarts without a full driver reset. Check nvidia-smi for unexpected occupants first. ## Search terms vllm oom at startup, vllm cuda out of memory loading model, vllm out of memory profiling run, vllm gpu_memory_utilization oom, vllm crashes on startup memory, vllm enforce eager oom --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json