# How to tune LLM inference batch size for the right throughput vs latency tradeoff > Source: https://netray.co/troubleshoot/inference-batch-size-tuning > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default. ## Error Title Not sure how to tune batch size for LLM inference throughput vs latency ## Error Signature increasing max-num-seqs or batch size does not improve throughput as expected ## Alternate Signatures - unsure what batch size to use for production llm serving - higher batch size makes latency worse without helping throughput ## Category performance ## Stack - vllm - pytorch - cuda ## Affects Any continuous-batching LLM server (vLLM, TGI, SGLang) tuning max-num-seqs or equivalent batch limits ## Quick Fix ### Find your knee, then set the limit **Steps:** - Run a load test sweeping concurrency (batch size) from low to high, recording both aggregate throughput and p50/p99 per-request latency at each level. - Plot or tabulate throughput and latency together; identify the point where throughput gains per added concurrent request flatten while latency keeps climbing, that is the knee. - Set max-num-seqs (or equivalent) at or just below that knee if latency matters, or push further past it only if pure throughput is the priority and latency SLO allows it. - Re-run the sweep whenever prompt length distribution, output length, or model changes meaningfully, since the knee shifts with all three. - Validate the chosen setting under realistic mixed-length traffic, not just uniform short prompts, since long prompts consume disproportionate batch capacity. ## Symptoms - Throughput keeps climbing as concurrency rises, but individual request latency also keeps climbing without a clear plateau - A batch size that worked well in testing produces missed latency SLOs once real, more varied traffic arrives - Increasing max-num-seqs past a certain point causes preemption or memory pressure instead of more throughput - No clear answer for what batch size to configure, settings are copied from another deployment or a default ## Causes ### Batch size set without measuring the actual throughput/latency tradeoff for this workload **Likelihood:** most-common **Explanation:** The relationship between batch size and both throughput and latency is workload-specific, driven by model size, GPU, and prompt/output length distribution, so a value that works well for one deployment can be badly miscalibrated for another. Copying a default or a number from a blog post skips the one step that actually determines the right setting: measuring your own throughput-latency curve. **Fix:** Run a concurrency sweep against your real model, hardware, and representative traffic, and choose max-num-seqs based on where your latency SLO is crossed, not a borrowed number. ### Batch size set above the point where added concurrency still helps throughput meaningfully **Likelihood:** common **Explanation:** Past a certain concurrency, additional in-flight sequences contend for the same fixed compute and memory bandwidth, so each additional sequence adds proportionally less new throughput while still adding queueing and compute contention that raises every request's latency, a classic diminishing-returns curve. **Fix:** Identify the knee via a concurrency sweep, and cap max-num-seqs there rather than at the theoretical maximum the GPU's memory could support. ### SLO-driven sizing not applied, batch size chosen for peak throughput instead of the latency budget **Likelihood:** common **Explanation:** Maximizing aggregate throughput and meeting a p99 latency SLO are frequently in direct tension; optimizing purely for the former by maximizing batch size will often violate the latter, especially for interactive or streaming use cases where users notice per-token latency directly. **Fix:** Start from your actual latency SLO (e.g. p99 time-per-output-token under X ms) and work backward to the maximum batch size that satisfies it, treating throughput as the outcome of that constraint rather than the primary target. ### Batch size tuned only against uniform short prompts, not the real mixed-length distribution **Likelihood:** occasional **Explanation:** A small number of long-prompt or long-generation requests within a batch can consume disproportionate compute and KV cache relative to short ones, so a batch size validated only against short, uniform test prompts can behave very differently, and worse, once real traffic with a long tail of prompt lengths arrives. **Fix:** Include a realistic mix of short and long prompts/generations in the load test used to determine batch size, not a synthetic uniform benchmark. ### Confusing batch size (max-num-seqs) with max-num-batched-tokens, leading to mismatched expectations **Likelihood:** rare **Explanation:** Some servers expose both a sequence-count limit and a token-count-per-step limit; tuning only one while leaving the other at a restrictive default can cap effective throughput in a way that looks like a batch size problem but is actually the other parameter constraining it. **Fix:** Check both the max concurrent sequences setting and the max tokens processed per scheduling step, and tune them together rather than assuming a single parameter controls batching. **Commands:** - vllm serve MODEL_NAME --max-num-seqs 128 --max-num-batched-tokens 8192 ## Diagnostics ### Sweep concurrency and record throughput plus p50/p99 latency **Command:** for c in 4 8 16 32 64 128; do echo concurrency=$c; done **Interpretation:** Tabulating throughput and latency together across this sweep is the only reliable way to find the knee; eyeballing a single concurrency level tells you nothing about the tradeoff shape. ### Check current server-side batching configuration **Command:** curl -s localhost:8000/v1/models | python -m json.tool **Interpretation:** Confirm what max-num-seqs and max-num-batched-tokens are actually set to, since a mismatch between assumed and actual config is a common source of confusing benchmark results. ### Monitor GPU utilization and memory alongside the sweep **Command:** nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1 **Interpretation:** If utilization is already near 100 percent well before the latency knee appears, you are compute-bound and further batch size increases will mostly add latency; if memory approaches its ceiling first, you are at risk of preemption before you are compute-bound. ## Prevention - Treat batch size as a measured, workload-specific setting, not a value to copy from documentation or another deployment - Define your latency SLO before tuning throughput, and size batch limits to satisfy it - Re-run the concurrency sweep after any change to model, hardware, or typical prompt/output length - Test with representative mixed-length traffic, not uniform synthetic prompts ## When To Escalate If the measured knee of the throughput/latency curve still cannot deliver both your required throughput and your latency SLO on current hardware, that is a capacity problem, not a tuning problem: you need more GPUs (horizontal scaling or tensor parallelism), a smaller/faster model, or a relaxed SLO, all of which are architecture decisions beyond batch size configuration. ## Faq ### Is there a universal 'good' batch size for LLM serving? **Answer:** No. It depends on model size, GPU, prompt and output length distribution, and your specific latency SLO, all of which vary by deployment. Any specific number quoted online is only a starting point for your own measurement, not a value to adopt directly. ### Should I always maximize batch size for the best cost per token? **Answer:** Maximizing batch size generally improves throughput and therefore cost efficiency up to the knee of the curve, but pushing past it risks preemption from KV cache exhaustion, which actively hurts both cost and latency. The cost-optimal point is close to, but not necessarily at, the same knee that satisfies your latency SLO. ### How often should batch size settings be revisited? **Answer:** Any time the model, GPU, or the shape of production traffic (average prompt length, output length, concurrency pattern) changes meaningfully. A setting tuned for one traffic mix can be meaningfully wrong for another, so treat it as something to re-validate periodically, not a one-time decision. ## Search terms llm batch size tuning, max-num-seqs vllm, throughput vs latency llm, inference batch size optimization, vllm concurrency sizing, slo driven batch sizing --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json