Why LLM serving throughput collapses under load, and how to prevent it
throughput drops sharply and latency spikes once concurrent requests pass a threshold
Also appears as
- vllm requests start getting preempted under heavy load
- server throughput falls off a cliff instead of scaling with more requests
Short answer
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
Affects: vLLM and similar continuous-batching servers under sustained high concurrency, most common with long contexts or many concurrent long-generation requests
Stop the preemption spiral
- 1Check server logs/metrics for preemption or recompute events; a rising preemption count that correlates with the throughput drop confirms KV cache exhaustion.
- 2Lower max-num-seqs so fewer sequences are admitted concurrently, keeping combined KV cache usage safely under the memory ceiling.
- 3Reduce max-model-len or enforce per-request output length limits if a small number of very long generations are consuming disproportionate KV cache.
- 4Enable admission control / request queueing with a hard concurrency cap in front of the server so it never accepts more than it can hold in memory at once.
- 5If the workload genuinely needs the higher concurrency, add GPU memory (larger GPU or additional replica) rather than pushing the existing server past its KV cache ceiling.
How to confirm this is your problem
- Throughput scales with concurrency up to a point, then drops sharply instead of plateauing
- Latency for in-flight requests spikes and becomes highly variable under heavy load
- Server logs show preemption, eviction, or recompute events increasing right before the collapse
- The same request rate that worked fine yesterday collapses today after prompt lengths or output lengths grew
Root causes and fixes
KV cache exhaustion triggers preemption and recompute
Every in-flight sequence holds its KV cache in GPU memory for the duration of generation, and total KV cache demand grows with concurrency, context length, and output length together. Once demand exceeds the memory the scheduler reserved for KV cache, it must preempt some sequences, evicting their cache. When those sequences resume, the model must recompute their prefill from scratch, which consumes GPU cycles that produce zero new output tokens, so measured throughput drops even though the GPU is fully busy.
Fix: Reduce max-num-seqs and/or max-model-len so the worst-case combined KV cache footprint stays under the memory the server reserves for it, and monitor preemption count as a leading indicator rather than waiting for throughput to visibly drop.
vllm serve MODEL_NAME --max-num-seqs 128 --max-model-len 8192 --gpu-memory-utilization 0.9
No admission control, so the server accepts more concurrent requests than memory can hold
Without a hard limit in front of the server, a traffic spike simply pushes more requests into the scheduler than it can serve without preemption. The server tries to be helpful by accepting everything, which is exactly the behavior that triggers the preemption spiral described above.
Fix: Put a concurrency-limiting gateway or queue in front of the model server, capped below the point where preemption starts, and return backpressure (429/queueing) to callers beyond that limit instead of letting the server thrash.
Queue explosion: incoming request rate exceeds sustainable service rate, backlog grows unbounded
If the arrival rate of new requests exceeds the rate the server can actually complete them at (its true steady-state throughput), the queue length grows without bound, and average wait time grows with it, compounding the perceived throughput and latency problem even if per-request GPU processing is unaffected.
Fix: Set a maximum queue depth and reject or shed load beyond it, size replica count to the sustained arrival rate rather than peak burst rate, and add autoscaling with enough lead time to absorb bursts.
Memory thrashing from KV cache fragmentation across many short-lived sequences
In servers without paged/block-based KV cache management, memory can fragment as sequences of varying lengths are allocated and freed, reducing effectively usable capacity below the raw free memory and causing preemption at lower concurrency than expected.
Fix: Use a serving framework with paged attention/block-based KV cache management (vLLM, SGLang) which allocates KV cache in fixed-size blocks specifically to avoid this fragmentation.
A small number of very long-context or long-generation requests consume disproportionate KV cache
KV cache size per sequence scales with sequence length, so a handful of requests with unusually long prompts or generation limits can consume as much KV cache as dozens of typical requests, crowding out capacity for everyone else and triggering preemption at a lower total request count than expected.
Fix: Enforce a maximum output token limit and/or a separate queue or rate limit for long-context requests so they cannot silently consume the majority of KV cache capacity.
Diagnostic commands
Check preemption/recompute counters
curl -s localhost:8000/metrics | grep -iE 'preempt|recompute'
A nonzero and rising preemption counter that lines up with the throughput drop is a direct confirmation of KV cache exhaustion as the root cause.
Watch GPU memory usage approach the reserved ceiling
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1
Memory usage flatlining near the gpu-memory-utilization ceiling right as throughput collapses confirms the server is out of room for additional KV cache blocks.
Measure queue depth over time under sustained load
curl -s localhost:8000/metrics | grep -iE 'num_requests_waiting'
A queue that grows continuously rather than stabilizing indicates arrival rate exceeds service rate; the fix is capacity or backpressure, not more tuning of the current server.
Stopping it from happening again
- Load test to find the concurrency point where preemption begins, and set max-num-seqs comfortably below it
- Alert on preemption/recompute counts, not just on latency, since preemption is the earlier and more actionable signal
- Enforce per-request max output length limits so a few requests cannot monopolize KV cache
- Size replica count and autoscaling to sustained arrival rate with margin for realistic burst patterns
When this becomes an architecture problem
If admission control and KV cache-aware sizing are already in place and the sustained request rate still exceeds what a single GPU (or your current fleet) can serve without preemption, the fix is horizontal scaling (more replicas), tensor parallelism across GPUs for a single larger-capacity server, or reducing per-request context/output length at the product level, all of which are capacity-planning decisions rather than configuration tuning.
Frequently asked questions
Why does throughput drop instead of just leveling off under heavy load?
Leveling off is what you would expect from a purely compute-bound system reaching saturation. The drop happens because preemption forces recomputation of previously-completed work, meaning some GPU cycles under heavy load produce zero net new tokens, actively reducing effective throughput rather than merely capping it.
Will increasing max-num-seqs help throughput under load?
Usually the opposite. Raising it past the point your KV cache memory supports admits more concurrent sequences than can fit, which increases preemption frequency and can make the collapse worse, not better. Lowering it to match measured KV cache capacity is typically the correct direction.
Is this the same problem as an out-of-memory crash?
No. OOM is a hard failure; preemption is the server's built-in mechanism to avoid OOM by gracefully evicting sequences under memory pressure. The tradeoff is that preemption itself has a throughput cost, so a server that never crashes can still silently underperform badly if it preempts constantly.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Related problems
vLLM fails to start because there is not enough memory for the KV cache
vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
Inference gets much slower as context length grows toward 32k, 64k, or 128k tokens
Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.
Not sure how to tune batch size for LLM inference throughput vs latency
Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.