Why GPU utilization is low during LLM inference, and how to raise it
nvidia-smi shows gpu utilization under 30 percent while the model is serving requests
Also appears as
- gpu sits idle most of the time during inference despite steady request traffic
- throughput does not improve even though the gpu is not maxed out
Short answer
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
Affects: Self-hosted LLM serving on any GPU, most common in early production deployments before load testing
Raise utilization without adding hardware
- 1Confirm how many concurrent requests are actually in flight at the server at any moment; a single synchronous client loop will never fill the GPU regardless of server config.
- 2Check the server's max-num-seqs (or equivalent) setting; if it is small, raise it so more sequences can be batched together during decode.
- 3Switch client code to async/concurrent request issuance (asyncio, thread pool, or a load-testing tool) instead of one request at a time in a for loop.
- 4Confirm tokenization is happening on the server side inside the batching scheduler, not as a separate blocking client-side step per request.
- 5Re-measure utilization at increasing concurrency levels until it plateaus; that plateau is your real per-GPU capacity.
How to confirm this is your problem
- nvidia-smi shows utilization well under 50 percent during active serving
- Adding more requests does not increase throughput proportionally
- Latency per request stays roughly flat regardless of load, suggesting nothing is actually contending for the GPU
- CPU usage on the client or gateway is high while GPU usage is low
Root causes and fixes
Request concurrency at the server is too low to fill the batch scheduler
Continuous batching can only combine as many sequences as are actually in flight. If your traffic pattern rarely has more than one or two concurrent requests reaching the server, the scheduler has nothing to batch and each decode step processes almost nothing, leaving compute idle between memory-bound reads.
Fix: Verify actual concurrent in-flight requests using server metrics, and if it is consistently low, that is a traffic/architecture issue (single client, no queueing) rather than a serving config issue; consider whether the workload genuinely needs a dedicated GPU or should share one via a gateway that batches across callers.
Client code issues requests synchronously, one at a time
A for-loop calling a blocking HTTP client waits for each response before sending the next request, so no matter how well-configured the server's batching is, the server never sees more than one request at a time. This is the single most common reason a correctly configured vLLM server still shows low utilization in a naive benchmark.
Fix: Rewrite the client to issue requests concurrently, using asyncio with aiohttp, a thread pool, or a proper load-testing tool that opens many connections at once.
python -c "import asyncio, aiohttp" # confirm async http client available
Tokenization or network I/O is the actual bottleneck, not the GPU
If tokenization runs as a separate blocking step per request on a slow CPU, or the network path between client and server adds significant round-trip time, the GPU can sit idle waiting for input, especially at low request rates, even though the server itself is capable of much higher throughput.
Fix: Profile request latency broken into network, tokenization, and GPU compute segments; if network or tokenization dominates, address those directly (colocate client and server, use faster tokenizers, batch tokenization) rather than tuning GPU settings.
max-num-seqs (or equivalent max concurrent sequences setting) is set too low
Even with plenty of concurrent client requests, the server will refuse to batch more than its configured maximum concurrent sequences, queueing the rest. A conservative default or a value copied from a smaller deployment caps utilization well below what the GPU's memory actually supports.
Fix: Raise max-num-seqs (vLLM) or the equivalent batch size limit, bounded by available KV cache memory for your context length, and re-measure.
vllm serve MODEL_NAME --max-num-seqs 256 --gpu-memory-utilization 0.9
Model is too small for the GPU, so even full batching cannot saturate compute
A small model (a few billion parameters) on a large, fast GPU may simply not have enough arithmetic work per token to keep the GPU busy even at high concurrency, because the memory-bandwidth-bound decode phase finishes each step faster than the GPU's compute units can be meaningfully occupied.
Fix: This is often fine and not actually a problem: confirm whether real-world latency and cost targets are being met before treating idle compute as an issue, or consider serving multiple models on the same GPU to use the spare capacity.
Diagnostic commands
Sample GPU utilization continuously during a load test
nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv -l 1
If utilization stays low even while a proper concurrent load test is running, the bottleneck is server-side config (max-num-seqs) or the model being too small for the hardware, not the client.
Count actual concurrent in-flight requests at the server
curl -s localhost:8000/metrics | grep -E 'num_requests_running|num_requests_waiting'
If num_requests_running rarely exceeds 1-2, the client is not sending enough concurrent traffic to exercise batching, regardless of server configuration.
Compare synchronous vs concurrent client benchmarks
python -m pip show locust
Re-run your benchmark with a proper concurrent load tool (Locust, k6, vegeta) instead of a sequential loop; if utilization jumps dramatically, the original client code was the bottleneck all along.
Stopping it from happening again
- Always load test with a concurrent client tool, never a sequential for-loop, before drawing conclusions about server performance
- Set max-num-seqs based on measured KV cache headroom, not a copied default
- Track num_requests_running as a first-class metric alongside GPU utilization
- Re-validate utilization after any client library change, since async-to-sync regressions are easy to introduce accidentally
When this becomes an architecture problem
If concurrency, client code, and max-num-seqs are all confirmed correct and utilization is still low, and the model is genuinely too small to saturate the GPU at your realistic traffic volume, the right move is capacity consolidation (serve multiple models per GPU, or move to a smaller/cheaper GPU) rather than further inference tuning; that is a hardware and fleet-design decision.
Frequently asked questions
Is low GPU utilization always a problem?
Not necessarily. If latency and cost targets are already met, spare compute is not automatically wasted money, especially if the GPU is shared across multiple models or workloads. It becomes a problem when you are also missing throughput or cost targets, which is the common case worth investigating.
Why does my load test show low utilization even with hundreds of requests queued?
Check whether those requests were actually sent concurrently or queued client-side and dispatched one at a time. A common mistake is generating a list of hundreds of requests but iterating through them with a blocking call, which produces the same low-concurrency pattern as a single-user test.
Should I just raise max-num-seqs as high as possible?
No. Setting it far beyond what your KV cache memory can support at your typical context length causes preemption and recompute under load instead of higher throughput. Size it based on measured KV cache capacity, not an arbitrarily large number.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Concurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Related problems
LLM inference is much slower in production than in benchmarks
Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
Not sure how to tune batch size for LLM inference throughput vs latency
Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.
High time to first token (TTFT) on LLM inference requests
High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.