# Why long-context requests are so much slower, and what to do about it

> Source: https://netray.co/troubleshoot/long-context-slow-performance
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.

## Error Title

Inference gets much slower as context length grows toward 32k, 64k, or 128k tokens

## Error Signature

latency increases sharply as prompt length or context window grows

## Alternate Signatures

- 128k context requests are far slower and more expensive than short prompts
- long documents in the prompt make the model take much longer to respond

## Category

performance

## Stack

- vllm
- pytorch
- cuda

## Affects

Any transformer-based model at long context lengths (32k and up), most pronounced approaching a model's advertised maximum (e.g. 128k), regardless of serving framework

## Quick Fix

### Manage long-context cost, do not just absorb it

**Steps:**

- Measure prefill time and KV cache usage specifically at your actual production context lengths, not just at a short benchmark prompt.
- Enable chunked prefill so long-context requests do not monopolize a scheduling step and starve other concurrent requests of decode progress.
- Reduce max-model-len to the context length you actually need in practice, rather than leaving it at the model's advertised maximum, to reclaim KV cache headroom.
- Where possible, trim or summarize context before it reaches the model (retrieval filtering, chunk selection) instead of sending the maximum available context on every request.
- If long context is a hard product requirement at meaningful concurrency, size hardware for the KV cache memory that specific context length demands, not for a short-prompt benchmark.

## Symptoms

- Latency grows disproportionately as prompt length increases, not just linearly
- Maximum supported concurrency drops sharply as average context length grows
- GPU memory fills up quickly and preemption/OOM appears only with long-context traffic
- Cost per request at long context is far higher than a naive per-token estimate would suggest

## Causes

### Attention compute cost scales roughly quadratically with sequence length during prefill

**Likelihood:** most-common

**Explanation:** Standard self-attention computes a score between every pair of tokens in the sequence, so the compute cost grows with the square of sequence length: doubling context length roughly quadruples the attention compute in the prefill pass. FlashAttention and similar kernels reduce the memory overhead of this computation but do not change its fundamental quadratic compute scaling, so prefill time at 128k tokens is dramatically more than eight times the prefill time at 16k tokens.

**Fix:** Treat prefill time as a first-class latency budget item that scales nonlinearly with context, not a fixed cost; measure it directly at your real context lengths rather than extrapolating linearly from short-prompt numbers.

### KV cache memory pressure scales linearly with context length, multiplied across every concurrent request

**Likelihood:** common

**Explanation:** Each token in context requires storing key and value vectors for every layer and attention head, so KV cache size per sequence grows linearly with context length. At high concurrency, total KV cache demand is concurrency times context length times per-token KV size, meaning long-context traffic consumes GPU memory far faster than short-context traffic at the same request count, directly reducing how many concurrent long-context requests a given GPU can hold.

**Fix:** Calculate KV cache memory needs at your actual context length and target concurrency before deployment, and reduce max-model-len to reclaim memory if the full advertised context is not actually needed.

### No chunked prefill, so one long request's prefill blocks decode progress for other in-flight requests

**Likelihood:** common

**Explanation:** Without chunking, a scheduler processes an entire long prompt's prefill as one uninterrupted step, during which other concurrent requests' decode steps do not advance, creating latency spikes for unrelated users whenever a long-context request arrives.

**Fix:** Enable chunked prefill so long prefills are broken into pieces interleaved with other requests' decode steps, smoothing out latency across concurrent traffic.

**Commands:**

- vllm serve MODEL_NAME --enable-chunked-prefill --max-num-batched-tokens 8192

### 128k context is treated as a fixed operating point rather than a worst case, oversizing hardware or crashing under real concurrency

**Likelihood:** occasional

**Explanation:** Teams often size capacity based on the model's advertised maximum context or a single-request benchmark at that length, then discover in production that even a modest number of concurrent long-context requests exceeds available KV cache memory, since the true cost is concurrency multiplied by context length, not context length alone.

**Fix:** Size hardware and concurrency limits based on your actual distribution of context lengths in production traffic, with explicit headroom for the tail of long-context requests, rather than assuming average-case cost.

### Retrieval or RAG pipelines send more context than the task actually needs

**Likelihood:** rare

**Explanation:** Sending maximum available retrieved chunks by default, rather than the minimum context that answers the query, pushes every request toward the expensive end of the quadratic prefill and linear KV cache cost curve without a corresponding accuracy benefit.

**Fix:** Tune retrieval to return fewer, more relevant chunks and re-rank before sending to the model, measuring accuracy against context size to find the minimum context that preserves answer quality.

## Diagnostics

### Measure prefill time at realistic production context lengths

**Command:** time curl -s localhost:8000/v1/completions -d '{"prompt":"<32k-token prompt>","max_tokens":1}'

**Interpretation:** Compare prefill time at short, medium, and long context lengths from your actual traffic; a clearly superlinear growth curve confirms the quadratic attention cost is the driver, not a config issue.

### Check KV cache memory usage as context length grows

**Command:** nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1

**Interpretation:** Memory usage growing roughly in proportion to context length at fixed concurrency confirms the linear KV cache scaling is the source of reduced concurrency at long context.

### Verify chunked prefill is enabled

**Command:** curl -s localhost:8000/v1/models | python -m json.tool

**Interpretation:** If chunked prefill is off, other concurrent requests are likely experiencing latency spikes whenever long-context requests arrive, independent of the long request's own latency.

## Prevention

- Size capacity based on the realistic distribution of context lengths in production, not the advertised max context or a short-prompt benchmark
- Enable chunked prefill by default on any server that mixes short and long context requests
- Track prefill time and KV cache usage as separate metrics broken out by context length bucket
- Push context minimization into the retrieval/RAG layer rather than treating the model's max context as a target to fill

## When To Escalate

If your product genuinely requires long-context requests (large document analysis, long conversation history) at meaningful concurrency, and chunked prefill plus right-sized max-model-len still cannot hit your latency and cost targets, that is a hardware capacity and architecture decision: you need more GPU memory per replica, more replicas, or a model with a more efficient attention mechanism for long context, not further serving-parameter tuning.

## Faq

### Why isn't 128k context just eight times the cost of 16k context?

**Answer:** Because attention compute in prefill scales roughly quadratically with sequence length, not linearly. Going from 16k to 128k tokens is an 8x increase in length but can be closer to a 64x increase in raw attention compute, on top of the 8x increase in KV cache memory per sequence, which is why long-context requests are disproportionately expensive.

### Does FlashAttention fix the long-context slowdown?

**Answer:** FlashAttention dramatically reduces the memory bandwidth and memory footprint overhead of computing attention, which is a major real-world speedup, but it does not change the underlying quadratic compute scaling with sequence length. Long contexts are still fundamentally more expensive to prefill than short ones, just less wastefully so.

### Should I always max out max-model-len to support the largest possible requests?

**Answer:** No. max-model-len directly determines how much KV cache memory the server reserves per sequence, so setting it far beyond what your traffic actually needs reduces the concurrency you can support at any context length, including short ones. Set it to your real p99 requirement plus margin, not the model's theoretical maximum.

## Search terms

long context slow inference, 128k context performance, attention quadratic cost, kv cache long context, chunked prefill, context length latency, why is long context expensive

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json