# Why LLM serving throughput collapses under load, and how to prevent it > Source: https://netray.co/troubleshoot/throughput-collapses-under-load > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue. ## Error Title LLM serving throughput collapses once load increases past a certain point ## Error Signature throughput drops sharply and latency spikes once concurrent requests pass a threshold ## Alternate Signatures - vllm requests start getting preempted under heavy load - server throughput falls off a cliff instead of scaling with more requests ## Category performance ## Stack - vllm - sglang - cuda ## Affects vLLM and similar continuous-batching servers under sustained high concurrency, most common with long contexts or many concurrent long-generation requests ## Quick Fix ### Stop the preemption spiral **Steps:** - Check server logs/metrics for preemption or recompute events; a rising preemption count that correlates with the throughput drop confirms KV cache exhaustion. - Lower max-num-seqs so fewer sequences are admitted concurrently, keeping combined KV cache usage safely under the memory ceiling. - Reduce max-model-len or enforce per-request output length limits if a small number of very long generations are consuming disproportionate KV cache. - Enable admission control / request queueing with a hard concurrency cap in front of the server so it never accepts more than it can hold in memory at once. - If the workload genuinely needs the higher concurrency, add GPU memory (larger GPU or additional replica) rather than pushing the existing server past its KV cache ceiling. ## Symptoms - Throughput scales with concurrency up to a point, then drops sharply instead of plateauing - Latency for in-flight requests spikes and becomes highly variable under heavy load - Server logs show preemption, eviction, or recompute events increasing right before the collapse - The same request rate that worked fine yesterday collapses today after prompt lengths or output lengths grew ## Causes ### KV cache exhaustion triggers preemption and recompute **Likelihood:** most-common **Explanation:** Every in-flight sequence holds its KV cache in GPU memory for the duration of generation, and total KV cache demand grows with concurrency, context length, and output length together. Once demand exceeds the memory the scheduler reserved for KV cache, it must preempt some sequences, evicting their cache. When those sequences resume, the model must recompute their prefill from scratch, which consumes GPU cycles that produce zero new output tokens, so measured throughput drops even though the GPU is fully busy. **Fix:** Reduce max-num-seqs and/or max-model-len so the worst-case combined KV cache footprint stays under the memory the server reserves for it, and monitor preemption count as a leading indicator rather than waiting for throughput to visibly drop. **Commands:** - vllm serve MODEL_NAME --max-num-seqs 128 --max-model-len 8192 --gpu-memory-utilization 0.9 ### No admission control, so the server accepts more concurrent requests than memory can hold **Likelihood:** common **Explanation:** Without a hard limit in front of the server, a traffic spike simply pushes more requests into the scheduler than it can serve without preemption. The server tries to be helpful by accepting everything, which is exactly the behavior that triggers the preemption spiral described above. **Fix:** Put a concurrency-limiting gateway or queue in front of the model server, capped below the point where preemption starts, and return backpressure (429/queueing) to callers beyond that limit instead of letting the server thrash. ### Queue explosion: incoming request rate exceeds sustainable service rate, backlog grows unbounded **Likelihood:** common **Explanation:** If the arrival rate of new requests exceeds the rate the server can actually complete them at (its true steady-state throughput), the queue length grows without bound, and average wait time grows with it, compounding the perceived throughput and latency problem even if per-request GPU processing is unaffected. **Fix:** Set a maximum queue depth and reject or shed load beyond it, size replica count to the sustained arrival rate rather than peak burst rate, and add autoscaling with enough lead time to absorb bursts. ### Memory thrashing from KV cache fragmentation across many short-lived sequences **Likelihood:** occasional **Explanation:** In servers without paged/block-based KV cache management, memory can fragment as sequences of varying lengths are allocated and freed, reducing effectively usable capacity below the raw free memory and causing preemption at lower concurrency than expected. **Fix:** Use a serving framework with paged attention/block-based KV cache management (vLLM, SGLang) which allocates KV cache in fixed-size blocks specifically to avoid this fragmentation. ### A small number of very long-context or long-generation requests consume disproportionate KV cache **Likelihood:** rare **Explanation:** KV cache size per sequence scales with sequence length, so a handful of requests with unusually long prompts or generation limits can consume as much KV cache as dozens of typical requests, crowding out capacity for everyone else and triggering preemption at a lower total request count than expected. **Fix:** Enforce a maximum output token limit and/or a separate queue or rate limit for long-context requests so they cannot silently consume the majority of KV cache capacity. ## Diagnostics ### Check preemption/recompute counters **Command:** curl -s localhost:8000/metrics | grep -iE 'preempt|recompute' **Interpretation:** A nonzero and rising preemption counter that lines up with the throughput drop is a direct confirmation of KV cache exhaustion as the root cause. ### Watch GPU memory usage approach the reserved ceiling **Command:** nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1 **Interpretation:** Memory usage flatlining near the gpu-memory-utilization ceiling right as throughput collapses confirms the server is out of room for additional KV cache blocks. ### Measure queue depth over time under sustained load **Command:** curl -s localhost:8000/metrics | grep -iE 'num_requests_waiting' **Interpretation:** A queue that grows continuously rather than stabilizing indicates arrival rate exceeds service rate; the fix is capacity or backpressure, not more tuning of the current server. ## Prevention - Load test to find the concurrency point where preemption begins, and set max-num-seqs comfortably below it - Alert on preemption/recompute counts, not just on latency, since preemption is the earlier and more actionable signal - Enforce per-request max output length limits so a few requests cannot monopolize KV cache - Size replica count and autoscaling to sustained arrival rate with margin for realistic burst patterns ## When To Escalate If admission control and KV cache-aware sizing are already in place and the sustained request rate still exceeds what a single GPU (or your current fleet) can serve without preemption, the fix is horizontal scaling (more replicas), tensor parallelism across GPUs for a single larger-capacity server, or reducing per-request context/output length at the product level, all of which are capacity-planning decisions rather than configuration tuning. ## Faq ### Why does throughput drop instead of just leveling off under heavy load? **Answer:** Leveling off is what you would expect from a purely compute-bound system reaching saturation. The drop happens because preemption forces recomputation of previously-completed work, meaning some GPU cycles under heavy load produce zero net new tokens, actively reducing effective throughput rather than merely capping it. ### Will increasing max-num-seqs help throughput under load? **Answer:** Usually the opposite. Raising it past the point your KV cache memory supports admits more concurrent sequences than can fit, which increases preemption frequency and can make the collapse worse, not better. Lowering it to match measured KV cache capacity is typically the correct direction. ### Is this the same problem as an out-of-memory crash? **Answer:** No. OOM is a hard failure; preemption is the server's built-in mechanism to avoid OOM by gracefully evicting sequences under memory pressure. The tradeoff is that preemption itself has a throughput cost, so a server that never crashes can still silently underperform badly if it preempts constantly. ## Search terms throughput collapses under load, vllm preemption, kv cache exhaustion, queue explosion llm serving, vllm recompute throughput drop, admission control llm server --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json