# Why vLLM says there is no memory for the KV cache, and how to fix it > Source: https://netray.co/troubleshoot/vllm-kv-cache-insufficient-memory > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache. ## Error Title vLLM fails to start because there is not enough memory for the KV cache ## Error Signature ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine ## Alternate Signatures - RuntimeError: To serve at least one request with the models max seq len, (X GiB KV cache is needed - torch.OutOfMemoryError: KV cache allocation failed ## Category gpu-memory ## Stack - vllm - cuda - pytorch ## Affects vLLM 0.4 and later on any CUDA GPU, most common serving 13B+ models at long max_model_len on 24-48 GB cards ## Quick Fix ### Get vLLM serving again in one restart **Steps:** - Raise the memory budget: add --gpu-memory-utilization 0.95 (from the 0.9 default) so vLLM reserves more of the physical VRAM for weights plus KV cache. - Lower --max-model-len to the context length you actually need in production, not the model's architectural maximum, since KV cache size scales directly with it. - If the model is large relative to the GPU, switch to a quantized checkpoint (AWQ, GPTQ, or FP8) with --quantization so more VRAM is freed for the cache pool. - For multi-GPU boxes, add --tensor-parallel-size N so weight memory is sharded and each GPU has a larger share left over for KV cache blocks. - Restart and confirm in the startup logs that the reported number of KV cache blocks is greater than zero. ## Symptoms - vllm serve process exits during startup, before it ever accepts a request - Log explicitly states there is no memory available for cache blocks or that even one request cannot be served at max_model_len - Same model serves fine with a shorter --max-model-len on the identical GPU - Reducing gpu_memory_utilization makes the error worse, raising it toward 0.9-0.95 resolves it ## Causes ### max_model_len is set to the model's full architectural context (often 32k-128k) which requires far more KV cache than the GPU has left after weights **Likelihood:** most-common **Explanation:** KV cache size per sequence scales as 2 x num_layers x num_kv_heads x head_dim x max_model_len x bytes_per_element. A model with a 128k native context can require tens of gigabytes of KV cache for a single maximum-length sequence, which frequently exceeds what remains after loading the weights themselves. **Fix:** Set --max-model-len to the longest context your application actually uses, for example 8192 or 16384, rather than accepting the model's default maximum. **Commands:** - vllm serve MODEL --max-model-len 8192 --gpu-memory-utilization 0.9 ### gpu_memory_utilization is left at a conservative default while the physical GPU has more headroom **Likelihood:** common **Explanation:** vLLM's default of 0.9 intentionally leaves 10 percent of VRAM unreserved as a safety margin for CUDA context and fragmentation. On GPUs shared with other processes or with driver overhead already present, that margin is fine, but on a dedicated inference GPU it can be the difference between zero and enough KV cache blocks. **Fix:** Increase --gpu-memory-utilization toward 0.93-0.95 on a GPU dedicated solely to this vLLM process, watching nvidia-smi to confirm no other process needs the freed headroom. **Commands:** - vllm serve MODEL --gpu-memory-utilization 0.95 ### Full-precision weights leave too little of the memory budget for any KV cache **Likelihood:** common **Explanation:** Weights and KV cache draw from the same gpu_memory_utilization pool. If fp16 weights alone consume 80-90 percent of VRAM, there is arithmetically almost nothing left for cache blocks, regardless of how max_model_len is tuned. **Fix:** Serve a 4-bit or FP8 quantized checkpoint so weight memory shrinks by roughly 2-4x, freeing proportionally more of the budget for KV cache. **Commands:** - vllm serve MODEL --quantization awq ### Tensor-parallel size is smaller than it should be for the model and GPU count available **Likelihood:** occasional **Explanation:** Without tensor parallelism, all weights and all KV cache for every request live on one GPU. If additional GPUs are present but --tensor-parallel-size is left at 1, the single GPU is starved for cache space while the others sit unused. **Fix:** Set --tensor-parallel-size to match the number of GPUs allocated to this model so both weights and KV cache are sharded, multiplying the effective memory budget. **Commands:** - vllm serve MODEL --tensor-parallel-size 2 --gpu-memory-utilization 0.9 ### Another process on the same GPU (a monitoring agent, a second model server, or a stale vLLM worker) is holding memory outside vLLM's accounting **Likelihood:** rare **Explanation:** vLLM's gpu_memory_utilization is computed against total device memory at process start, but memory already held by unrelated processes reduces what is genuinely free, so vLLM's internal math can look correct while the real headroom is smaller than assumed. **Fix:** Confirm with nvidia-smi that vLLM's process is the only significant consumer on the GPU before tuning utilization further. **Commands:** - nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv ## Diagnostics ### Read the exact KV cache requirement vLLM reports **Command:** vllm serve MODEL --max-model-len 8192 2>&1 | grep -i 'kv cache\|gpu blocks' **Interpretation:** The startup log states the number of GPU blocks available and the number needed per sequence; zero or a very low block count confirms this is a capacity problem, not a bug. ### Check real free memory versus what vLLM assumes **Command:** nvidia-smi --query-gpu=memory.used,memory.total,memory.free --format=csv **Interpretation:** Compare memory.free against (1 - gpu_memory_utilization) x memory.total; if actual free memory is lower than expected, another process is consuming the headroom vLLM planned around. ### Estimate KV cache size for your target context **Command:** python -c "layers=32; kv_heads=8; head_dim=128; ctx=8192; bytes_elem=2; print(2*layers*kv_heads*head_dim*ctx*bytes_elem/1e9, 'GB per sequence')" **Interpretation:** Multiply this per-sequence figure by your target concurrent request count to see the true KV cache budget you need, then compare against what is left after weights on your GPU. ## Prevention - Set --max-model-len explicitly to your application's real requirement in every deployment config; never leave it at the model default. - Size gpu_memory_utilization per GPU model as part of your deployment template, not per incident. - Load-test with your expected concurrent request count before production cutover, since KV cache exhaustion under load looks different from startup failure. - Track free VRAM per GPU node in monitoring so a co-located process cannot silently eat vLLM's planned headroom. ## When To Escalate If your production concurrency and context length genuinely require more KV cache than any single GPU can provide even after quantization and a tuned max_model_len, that is a multi-GPU tensor-parallel or multi-node serving architecture decision, not a flag to tune further. ## Faq ### What does gpu_memory_utilization actually control? **Answer:** It sets the fraction of total GPU memory vLLM is allowed to reserve for the combination of model weights and the KV cache block pool, default 0.9. vLLM computes available cache memory as (gpu_memory_utilization x total_memory) minus the memory the weights already consume, then allocates that remainder as KV cache blocks. ### Why does lowering max_model_len fix an out-of-memory startup failure? **Answer:** KV cache memory per sequence is directly proportional to max_model_len. Halving max_model_len roughly halves the KV cache memory vLLM must reserve for even a single maximum-length request, which is often enough to bring the requirement back under the available budget without touching hardware. ### Is it safe to set gpu_memory_utilization close to 1.0? **Answer:** Not recommended. Values above roughly 0.95 leave too little headroom for CUDA context overhead and short-lived allocations, risking a runtime OOM later under load even though startup succeeds. 0.9 to 0.95 is the practical safe range for a GPU dedicated to one vLLM process. ### Does quantization reduce KV cache memory too, or only weight memory? **Answer:** Standard AWQ/GPTQ/FP8 quantization reduces weight memory only; KV cache dtype is controlled separately (vLLM supports FP8 KV cache on supported hardware). Quantizing weights still helps indirectly because it frees more of the shared gpu_memory_utilization budget for the cache pool. ## Search terms vllm no available memory for cache blocks, vllm kv cache out of memory, vllm gpu_memory_utilization error, vllm max_model_len oom, vllm cannot serve at least one request, kv cache size calculation llm, vllm startup out of memory --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json