Why long-context requests are so much slower, and what to do about it
latency increases sharply as prompt length or context window grows
Also appears as
- 128k context requests are far slower and more expensive than short prompts
- long documents in the prompt make the model take much longer to respond
Short answer
Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.
Affects: Any transformer-based model at long context lengths (32k and up), most pronounced approaching a model's advertised maximum (e.g. 128k), regardless of serving framework
Manage long-context cost, do not just absorb it
- 1Measure prefill time and KV cache usage specifically at your actual production context lengths, not just at a short benchmark prompt.
- 2Enable chunked prefill so long-context requests do not monopolize a scheduling step and starve other concurrent requests of decode progress.
- 3Reduce max-model-len to the context length you actually need in practice, rather than leaving it at the model's advertised maximum, to reclaim KV cache headroom.
- 4Where possible, trim or summarize context before it reaches the model (retrieval filtering, chunk selection) instead of sending the maximum available context on every request.
- 5If long context is a hard product requirement at meaningful concurrency, size hardware for the KV cache memory that specific context length demands, not for a short-prompt benchmark.
How to confirm this is your problem
- Latency grows disproportionately as prompt length increases, not just linearly
- Maximum supported concurrency drops sharply as average context length grows
- GPU memory fills up quickly and preemption/OOM appears only with long-context traffic
- Cost per request at long context is far higher than a naive per-token estimate would suggest
Root causes and fixes
Attention compute cost scales roughly quadratically with sequence length during prefill
Standard self-attention computes a score between every pair of tokens in the sequence, so the compute cost grows with the square of sequence length: doubling context length roughly quadruples the attention compute in the prefill pass. FlashAttention and similar kernels reduce the memory overhead of this computation but do not change its fundamental quadratic compute scaling, so prefill time at 128k tokens is dramatically more than eight times the prefill time at 16k tokens.
Fix: Treat prefill time as a first-class latency budget item that scales nonlinearly with context, not a fixed cost; measure it directly at your real context lengths rather than extrapolating linearly from short-prompt numbers.
KV cache memory pressure scales linearly with context length, multiplied across every concurrent request
Each token in context requires storing key and value vectors for every layer and attention head, so KV cache size per sequence grows linearly with context length. At high concurrency, total KV cache demand is concurrency times context length times per-token KV size, meaning long-context traffic consumes GPU memory far faster than short-context traffic at the same request count, directly reducing how many concurrent long-context requests a given GPU can hold.
Fix: Calculate KV cache memory needs at your actual context length and target concurrency before deployment, and reduce max-model-len to reclaim memory if the full advertised context is not actually needed.
No chunked prefill, so one long request's prefill blocks decode progress for other in-flight requests
Without chunking, a scheduler processes an entire long prompt's prefill as one uninterrupted step, during which other concurrent requests' decode steps do not advance, creating latency spikes for unrelated users whenever a long-context request arrives.
Fix: Enable chunked prefill so long prefills are broken into pieces interleaved with other requests' decode steps, smoothing out latency across concurrent traffic.
vllm serve MODEL_NAME --enable-chunked-prefill --max-num-batched-tokens 8192
128k context is treated as a fixed operating point rather than a worst case, oversizing hardware or crashing under real concurrency
Teams often size capacity based on the model's advertised maximum context or a single-request benchmark at that length, then discover in production that even a modest number of concurrent long-context requests exceeds available KV cache memory, since the true cost is concurrency multiplied by context length, not context length alone.
Fix: Size hardware and concurrency limits based on your actual distribution of context lengths in production traffic, with explicit headroom for the tail of long-context requests, rather than assuming average-case cost.
Retrieval or RAG pipelines send more context than the task actually needs
Sending maximum available retrieved chunks by default, rather than the minimum context that answers the query, pushes every request toward the expensive end of the quadratic prefill and linear KV cache cost curve without a corresponding accuracy benefit.
Fix: Tune retrieval to return fewer, more relevant chunks and re-rank before sending to the model, measuring accuracy against context size to find the minimum context that preserves answer quality.
Diagnostic commands
Measure prefill time at realistic production context lengths
time curl -s localhost:8000/v1/completions -d '{"prompt":"<32k-token prompt>","max_tokens":1}'Compare prefill time at short, medium, and long context lengths from your actual traffic; a clearly superlinear growth curve confirms the quadratic attention cost is the driver, not a config issue.
Check KV cache memory usage as context length grows
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1
Memory usage growing roughly in proportion to context length at fixed concurrency confirms the linear KV cache scaling is the source of reduced concurrency at long context.
Verify chunked prefill is enabled
curl -s localhost:8000/v1/models | python -m json.tool
If chunked prefill is off, other concurrent requests are likely experiencing latency spikes whenever long-context requests arrive, independent of the long request's own latency.
Stopping it from happening again
- Size capacity based on the realistic distribution of context lengths in production, not the advertised max context or a short-prompt benchmark
- Enable chunked prefill by default on any server that mixes short and long context requests
- Track prefill time and KV cache usage as separate metrics broken out by context length bucket
- Push context minimization into the retrieval/RAG layer rather than treating the model's max context as a target to fill
When this becomes an architecture problem
If your product genuinely requires long-context requests (large document analysis, long conversation history) at meaningful concurrency, and chunked prefill plus right-sized max-model-len still cannot hit your latency and cost targets, that is a hardware capacity and architecture decision: you need more GPU memory per replica, more replicas, or a model with a more efficient attention mechanism for long context, not further serving-parameter tuning.
Frequently asked questions
Why isn't 128k context just eight times the cost of 16k context?
Because attention compute in prefill scales roughly quadratically with sequence length, not linearly. Going from 16k to 128k tokens is an 8x increase in length but can be closer to a 64x increase in raw attention compute, on top of the 8x increase in KV cache memory per sequence, which is why long-context requests are disproportionately expensive.
Does FlashAttention fix the long-context slowdown?
FlashAttention dramatically reduces the memory bandwidth and memory footprint overhead of computing attention, which is a major real-world speedup, but it does not change the underlying quadratic compute scaling with sequence length. Long contexts are still fundamentally more expensive to prefill than short ones, just less wastefully so.
Should I always max out max-model-len to support the largest possible requests?
No. max-model-len directly determines how much KV cache memory the server reserves per sequence, so setting it far beyond what your traffic actually needs reduces the concurrency you can support at any context length, including short ones. Set it to your real p99 requirement plus margin, not the model's theoretical maximum.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolRAG Context Window Budget Calculator
Allocate your context window across system prompt, retrieved chunks, and conversation history, then see window utilization and the real cost of every RAG query.
Free ToolModel Context Window Planner
Allocate a fixed context window across system prompt, retrieved chunks, conversation history, and reserved output, then see exactly how much retrieval headroom is left.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Related problems
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
High time to first token (TTFT) on LLM inference requests
High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases.
vLLM: model's max seq len is larger than the KV cache can hold
vLLM preallocates a fixed KV cache pool sized by gpu_memory_utilization and refuses to start a context length whose worst case (batch x max sequence length) doesn't fit in that pool. Fix it by raising --gpu-memory-utilization toward 0.9-0.95, lowering --max-model-len to what you actually need, or adding a GPU/quantizing weights to leave more headroom for cache.
vLLM fails to start because there is not enough memory for the KV cache
vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuideRAG Chunking Strategies: Fixed, Semantic, Structural, and Late
Compare RAG chunking strategies, fixed-size, semantic, structural, and late chunking, with concrete guidance on chunk size, overlap, and when each wins.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.