# Why vLLM throughput is far below expected, and how to tune it > Source: https://netray.co/troubleshoot/vllm-low-throughput > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Low vLLM throughput almost always traces back to max-num-seqs capping concurrent batching too low, chunked prefill being disabled so long prompts stall the decode batch, an unintended dtype that doesn't use tensor cores efficiently, CPU-bound tokenization or preprocessing, or requests spilling into swap. Diagnose with nvidia-smi and vLLM's own throughput logs before changing anything. ## Error Title vLLM throughput is far below expected tokens per second ## Error Signature Avg generation throughput: 12.3 tokens/s (far below expected for the GPU and model combination) ## Alternate Signatures - GPU utilization stuck around 20-40% while requests queue up in vLLM logs - Time to first token and inter-token latency both much higher than benchmark numbers for the same model and GPU ## Category serving ## Stack - vllm - cuda ## Affects vLLM 0.4 and later, any GPU; most common when serving with default flags on a model or workload that needs tuning for concurrency, prefill, or dtype ## Quick Fix ### Tuning pass, in order of impact **Steps:** - Raise --max-num-seqs, since the default is often conservative, so more requests can be batched concurrently for the same forward pass. - Enable chunked prefill with --enable-chunked-prefill so long prompt prefill doesn't block decode steps for other in-flight requests. - Confirm --dtype matches the checkpoint's native precision, such as bfloat16 or float16, rather than falling back to float32. - Check GPU utilization with nvidia-smi during load; if it's low while requests queue, the bottleneck is likely request arrival rate, tokenization, or network, not the model itself. - If VRAM allows, raise --gpu-memory-utilization to grow the KV cache pool and support more concurrent sequences. ## Symptoms - Tokens per second reported in vLLM logs or metrics is far below published benchmarks for the same model and GPU pairing. - nvidia-smi shows GPU utilization well under 80-90 percent even under sustained load. - Latency increases sharply once more than a handful of concurrent requests arrive. - Long-prompt requests appear to stall throughput for all other concurrent users. ## Causes ### max-num-seqs or max-num-batched-tokens capped too low for available VRAM **Likelihood:** most-common **Explanation:** vLLM's continuous batching only helps if enough sequences are actually admitted into the running batch at once. A conservative default or manually-set low cap means the GPU sits idle between small batches instead of saturating with a larger concurrent batch, wasting the parallelism the paged KV cache was built to exploit. **Fix:** Raise --max-num-seqs, and optionally --max-num-batched-tokens, toward the largest value your KV cache pool can support, checking GPU utilization as you increase it. **Commands:** - vllm serve MODEL_ID --max-num-seqs 256 - nvidia-smi dmon -s u ### Chunked prefill disabled, so long prompts block decode steps **Likelihood:** common **Explanation:** Without chunked prefill, a long incoming prompt's entire prefill computation runs as one blocking step, delaying token generation for every other sequence already being decoded in that batch. This creates head-of-line blocking that tanks aggregate throughput under mixed short and long prompt traffic. **Fix:** Enable --enable-chunked-prefill so long prefills are split into pieces interleaved with ongoing decode steps. **Commands:** - vllm serve MODEL_ID --enable-chunked-prefill ### Wrong or fallback dtype not using tensor cores efficiently **Likelihood:** common **Explanation:** If --dtype isn't explicitly set to match the checkpoint, or an unsupported combination silently falls back toward slower compute paths, matrix multiplications run far slower because they don't hit the GPU's fp16 or bf16 tensor core throughput, even though the model still works correctly. **Fix:** Explicitly set --dtype bfloat16 or --dtype float16 matching the checkpoint's training precision, and confirm it's actually applied in the startup logs. **Commands:** - vllm serve MODEL_ID --dtype bfloat16 ### CPU-bound tokenization or request preprocessing ahead of the GPU **Likelihood:** occasional **Explanation:** With very high request rates or a slow tokenizer implementation, the CPU-side path, including tokenizing, detokenizing, and JSON serialization in the OpenAI-compatible API layer, can become the bottleneck, leaving the GPU underfed even though the model itself could run faster. **Fix:** Profile CPU usage during load; if a single core is pegged while the GPU sits idle, add more API server workers or check for a slow, non-fast tokenizer being loaded. **Commands:** - top - python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('MODEL_ID'); print(t.is_fast)" ### KV cache pool too small, forcing requests into swap or aggressive preemption **Likelihood:** rare **Explanation:** When the KV cache pool fills up under high concurrency, vLLM either swaps blocks to CPU RAM or preempts and recomputes lower-priority sequences. Both are far slower than serving purely from GPU memory and show up as throughput cliffs precisely when concurrency rises. **Fix:** Increase --gpu-memory-utilization or reduce --max-model-len to grow the effective KV cache pool so fewer requests need swap or preemption under load. **Commands:** - vllm serve MODEL_ID --gpu-memory-utilization 0.92 ## Diagnostics ### Watch GPU utilization during a load test **Command:** nvidia-smi dmon -s u **Interpretation:** Sustained utilization well under 80-90 percent during heavy concurrent load points to a CPU, batching, or network bottleneck rather than the model being compute-bound. ### Check vLLM's own throughput and latency logging **Command:** vllm serve MODEL_ID ... 2>&1 | grep -i "throughput" **Interpretation:** Compare the logged tokens per second against published benchmark numbers for the same model and GPU; a large gap confirms a config issue rather than a hardware ceiling. ### Confirm the dtype actually in use **Command:** vllm serve MODEL_ID 2>&1 | grep -i dtype **Interpretation:** If it doesn't match the checkpoint's native precision, fix --dtype explicitly and relaunch. ### Check current batching and concurrency settings **Command:** vllm serve --help | grep -i "max-num-seqs" **Interpretation:** Confirm these flags are set to values appropriate for your VRAM headroom, not left at overly conservative defaults. ## Prevention - Benchmark each new model and GPU combination against published or self-measured baseline throughput before production rollout, so regressions are caught immediately. - Load-test with a realistic mix of short and long prompts, not just uniform synthetic requests, to catch prefill head-of-line blocking early. - Document the tuned flags, including max-num-seqs, chunked-prefill, and dtype, per model in your deployment manifests so they survive redeploys. - Monitor GPU utilization and per-request latency continuously in production to catch throughput regressions from config drift. ## When To Escalate If you've tuned batching, prefill, and dtype and GPU utilization is genuinely pinned near 100 percent with throughput still below product requirements, that's no longer a configuration problem: it's a capacity problem needing more or faster GPUs, a smaller or quantized model, or speculative decoding, all of which are architecture decisions worth planning deliberately. ## Faq ### What's the single highest-leverage flag for throughput? **Answer:** For most under-tuned deployments it's max-num-seqs: raising it lets vLLM's continuous batching admit more concurrent sequences per forward pass, directly increasing GPU utilization and aggregate tokens per second, as long as your KV cache pool has room. ### Does chunked prefill help single-request latency too? **Answer:** It mainly helps aggregate throughput and fairness under concurrent mixed-length traffic by preventing long prompts from blocking others. For a single isolated request it has minimal effect on that request's own latency. ### How do I know if I'm CPU-bound versus GPU-bound? **Answer:** Watch nvidia-smi during a load test: if GPU utilization stays low while requests queue and a CPU core is pegged, you're CPU-bound on tokenization or API serialization. If GPU utilization is high and throughput is still low, look at batching and dtype settings instead. ## Search terms vllm low throughput, vllm slow tokens per second, vllm max-num-seqs tuning, vllm chunked prefill, vllm gpu utilization low, vllm performance tuning, increase vllm throughput --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json