Why vLLM says the model's max seq len is larger than the KV cache, and how to fix it
ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (16384). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.
Also appears as
- RuntimeError: To serve at least one request with the model's max seq len (X), (Y) GPU KV cache blocks are needed, which is more than the total number of available KV cache blocks (Z)
Short answer
vLLM preallocates a fixed KV cache pool sized by gpu_memory_utilization and refuses to start a context length whose worst case (batch x max sequence length) doesn't fit in that pool. Fix it by raising --gpu-memory-utilization toward 0.9-0.95, lowering --max-model-len to what you actually need, or adding a GPU/quantizing weights to leave more headroom for cache.
Affects: vLLM 0.4 and later, any GPU, most common on 24GB and 48GB cards serving 32k+ context models
Fastest path to a working server
- 1Re-launch with a higher --gpu-memory-utilization, e.g. 0.90 or 0.95, checking your current value first since some deployments already leave it low.
- 2If that alone doesn't fix it, cap context to what you need with --max-model-len 8192 (or whatever fits) instead of the model's full trained context.
- 3If you need the full context length, quantize weights with --quantization awq, gptq, or fp8, or add a GPU with --tensor-parallel-size 2 to free VRAM for cache.
- 4Re-run the server and confirm it logs the KV cache size without raising the ValueError.
How to confirm this is your problem
- Server crashes immediately on startup, before accepting any HTTP requests.
- Error mentions two numbers: the model's configured max sequence length and a smaller number of tokens the KV cache can hold.
- Happens more often after upgrading to a model or config with a longer max_position_embeddings.
- Works fine when you pass a shorter --max-model-len override.
Root causes and fixes
KV cache pool sized by gpu_memory_utilization is too small for the requested context
vLLM reserves a fixed fraction of GPU memory (often around 90 percent by default) for weights, activations, and a paged KV cache block pool sized at startup. It computes the max number of concurrent tokens that pool can hold and compares that to max_model_len. If the requested max length needs more cache blocks than the pool has, it refuses to start rather than silently truncate context.
Fix: Increase --gpu-memory-utilization to give the cache pool more room, or decrease --max-model-len to what your workload actually needs.
vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.95 --max-model-len 8192
Model's default context length is inherited even though you don't need it
When --max-model-len isn't set, vLLM uses the model config's max_position_embeddings, which for modern models is often 32k to 128k tokens. That full length, multiplied through the KV cache math of layers x heads x head_dim x bytes, can exceed available VRAM long before you ever send a genuinely long prompt.
Fix: Explicitly pass --max-model-len set to your real workload's maximum, not the model's advertised maximum context.
vllm serve MODEL_ID --max-model-len 16384
Not enough total GPUs or VRAM for both weights and the requested cache size
Large models leave little headroom on a single GPU; even at 0.95 utilization the cache pool may still be too small once weights and CUDA graph buffers are subtracted from total VRAM.
Fix: Add tensor parallelism across more GPUs, or move to fp8, AWQ, or GPTQ quantized weights to shrink the weight footprint and leave more VRAM for cache.
vllm serve MODEL_ID --tensor-parallel-size 2 vllm serve MODEL_ID --quantization awq
Other processes already holding VRAM before vLLM starts
A previous vLLM process that didn't shut down cleanly, a Jupyter kernel, or another service can hold VRAM that shrinks the effective free memory vLLM measures at startup, making its cache-size calculation land below what max_model_len requires.
Fix: Kill stray GPU processes before starting the server and confirm free VRAM with nvidia-smi first.
nvidia-smi
swap-space set to zero on a workload that needs CPU offload for the cache
The swap-space flag controls CPU RAM used to offload KV cache blocks under pressure and defaults to a few GB; setting it to zero removes that overflow room, making the GPU-only cache pool a hard ceiling with no slack.
Fix: Leave --swap-space at its default or raise it slightly rather than disabling it entirely.
vllm serve MODEL_ID --swap-space 4
Diagnostic commands
Check free VRAM before launch
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
If used memory is already high before you start vLLM, another process is eating your headroom; free it first.
Print the model's configured context length
python -c "from transformers import AutoConfig; c = AutoConfig.from_pretrained('MODEL_ID'); print(c.max_position_embeddings)"Compare this to the --max-model-len you're passing; if you omitted the flag, vLLM is using this full value as its target.
Start vLLM with logging to see the computed cache size
vllm serve MODEL_ID --max-model-len 4096 2>&1 | grep -i "kv cache"
The logged 'GPU KV cache size' in tokens tells you exactly how much headroom you have at a given gpu_memory_utilization; raise or lower max_model_len relative to that number.
Stopping it from happening again
- Always set --max-model-len explicitly to your application's real maximum prompt plus completion length, not the model default.
- Size GPUs using a KV cache calculator before deployment so this never surprises you in production.
- Pin --gpu-memory-utilization in your deployment config so it isn't left at a default that changes between vLLM versions.
- Load-test with your longest expected prompts in staging before promoting a new model or context length change to production.
When this becomes an architecture problem
If you've maxed out gpu_memory_utilization, right-sized max_model_len, and quantized weights, and you still can't fit the context length your product requires, that's a hardware sizing problem, not a config problem: you need more GPUs, larger-VRAM cards, or a KV-cache-efficient model architecture, which is worth a capacity-planning pass.
Frequently asked questions
What does gpu_memory_utilization actually control?
It's the fraction of total GPU memory vLLM is allowed to use for everything: model weights, activation buffers, CUDA graphs, and the paged KV cache pool. Raising it, typically toward 0.9 to 0.95, gives the KV cache pool more room without changing your model or context length, which is why it's usually the first lever to pull for this error.
Should I just set max_model_len as high as possible?
No. Set it to the actual maximum prompt plus completion length your application will send. A needlessly high value forces vLLM to reserve cache capacity for worst-case concurrent long sequences it will never actually see, wasting VRAM you could use for more concurrent users.
Does quantization fix this even though the error is about KV cache, not weights?
Indirectly yes: quantizing weights with AWQ, GPTQ, or FP8 shrinks the memory the model occupies, which leaves more of your gpu_memory_utilization budget available for the KV cache pool, effectively raising the max context or concurrency you can fit.
Can I just disable this check?
No, and you shouldn't want to. It isn't an arbitrary limit, it's vLLM telling you honestly that a full-length request would run out of VRAM mid-generation. Bypassing it would just move the failure from startup to a mid-request crash.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Related problems
vLLM fails to start because there is not enough memory for the KV cache
vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.
vLLM runs out of memory during startup, before serving any requests
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
How to reduce VRAM usage for LLM inference
VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideLLM Quantization: AWQ vs GPTQ vs FP8 vs GGUF
AWQ, GPTQ, FP8, and GGUF compared for production LLM serving: memory savings, throughput impact, quality loss, and which format fits which deployment.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.