# Why LLM inference is much slower in production than your benchmarks, and how to fix it

> Source: https://netray.co/troubleshoot/llm-inference-slow-in-production
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.

## Error Title

LLM inference is much slower in production than in benchmarks

## Error Signature

vllm serving is way slower in production than the benchmark numbers showed

## Alternate Signatures

- tokens per second drops once real traffic hits the endpoint
- inference is fast in a single curl test but slow under the load balancer

## Category

performance

## Stack

- vllm
- pytorch
- cuda

## Affects

Any self-hosted serving stack (vLLM, TGI, TensorRT-LLM, plain transformers), most visible on 7B-70B parameter models running on 24GB-80GB GPUs

## Quick Fix

### Find the bottleneck in one pass

**Steps:**

- Confirm the model is loaded in bfloat16 or float16, not float32: check the serving command or model config for dtype/torch_dtype.
- Confirm you are running a continuous-batching server (vLLM, TGI, SGLang) and not a naive for-loop over requests.
- Run nvidia-smi during a burst of concurrent requests: if GPU utilization stays under 40-50 percent, the bottleneck is upstream (batching, network, or CPU), not the GPU.
- Time the request end to end versus GPU-only time (server logs usually separate these); a large gap points to tokenization, guardrails, or serialization overhead outside the model call.
- If GPU memory bandwidth is the limiter (large model, small GPU), apply quantization (AWQ, GPTQ, or FP8) or move to a GPU with more memory bandwidth for the concurrency you need.

## Symptoms

- Single test requests feel fine but latency climbs once concurrent users are added
- Tokens per second in production is a fraction of the number quoted in the model or serving framework's benchmarks
- nvidia-smi shows low utilization even while users report the service is slow
- Response times vary wildly between requests of similar length

## Causes

### Model loaded in FP32 (full precision) instead of BF16/FP16

**Likelihood:** most-common

**Explanation:** FP32 doubles the bytes that must move from GPU memory for every weight and activation, and it does not use the tensor core paths that modern GPUs are optimized for. Inference is memory-bandwidth-bound during decode, so doubling the bytes read per token roughly halves throughput, and you also lose the tensor-core speedup that BF16/FP16 unlock.

**Fix:** Load the model with bfloat16 (fp16 on older GPUs without bf16 support). In vLLM this is the default when the checkpoint supports it; verify with --dtype bfloat16 explicitly rather than relying on an unexamined default.

**Commands:**

- python -c "import torch; print(torch.cuda.is_bf16_supported())"
- vllm serve MODEL_NAME --dtype bfloat16

### No continuous batching, requests are effectively served one at a time

**Likelihood:** common

**Explanation:** Decoding one token at a time for one request leaves most of the GPU's compute idle because the workload is memory-bound, not compute-bound, at batch size 1. A continuous-batching scheduler interleaves the decode steps of many requests so the same memory read serves many sequences per step, which is where the real throughput gain comes from.

**Fix:** Serve with vLLM, TGI, or SGLang instead of a hand-rolled loop calling model.generate() per request. Confirm --max-num-seqs is set high enough to admit multiple concurrent requests rather than defaulting to a low value.

**Commands:**

- vllm serve MODEL_NAME --max-num-seqs 256

### Serving on hardware whose memory bandwidth cannot support the model size and concurrency in use

**Likelihood:** common

**Explanation:** Decode-phase throughput is bounded by how fast the GPU can stream weights (and KV cache) from HBM. A 70B model on a 24GB card forced into aggressive offload, or any model pushed far beyond the concurrency its memory bandwidth supports, will show low and inconsistent throughput regardless of software tuning.

**Fix:** Right-size the GPU to the model and target concurrency, or apply quantization to shrink the memory footprint and the bytes moved per token. Use a sizing calculator before committing to hardware.

### CPU-bound pre/post-processing (tokenization, guardrails, JSON assembly) serialized around the GPU call

**Likelihood:** occasional

**Explanation:** Python-level tokenization, regex-based guardrails, or synchronous logging/formatting steps run on the CPU and, if not overlapped with GPU work, add straight-line latency to every request. This shows up as high end-to-end latency with normal GPU utilization, because the GPU is idle while the CPU step runs.

**Fix:** Profile the request handler separately from the model call. Move tokenization to the batch scheduler (vLLM does this internally), and make guardrail/post-processing steps async or run them concurrently with the next request's prefill.

### Model not quantized when VRAM is tight, forcing CPU offload or paging

**Likelihood:** rare

**Explanation:** When a model barely fits or does not fit in VRAM, frameworks may offload layers to CPU RAM or swap KV cache pages more aggressively, both of which are orders of magnitude slower than pure GPU execution because of the PCIe transfer cost.

**Fix:** Apply AWQ, GPTQ, or FP8 quantization to shrink the model until it comfortably fits with headroom for KV cache, instead of relying on offload.

## Diagnostics

### Check GPU utilization under concurrent load

**Command:** nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used --format=csv -l 1

**Interpretation:** Utilization consistently under 40-50 percent during a burst of concurrent requests points to a bottleneck outside the GPU (batching config, network, or CPU preprocessing). Utilization near 90-100 percent means the GPU itself is the limiter and you need quantization or more/better hardware.

### Confirm the serving dtype

**Command:** curl -s localhost:8000/v1/models | python -m json.tool

**Interpretation:** Some serving frameworks report the active dtype in the models endpoint or startup logs. If it shows float32, that is the first fix regardless of anything else.

### Compare single-request vs concurrent latency

**Command:** for i in 1 2 4 8 16; do echo concurrency=$i; done

**Interpretation:** Run the same prompt at increasing concurrency levels and record p50/p99 latency. If per-request latency barely rises with concurrency, batching is working; if it rises linearly, requests are being serialized.

## Prevention

- Load test with realistic concurrent traffic before launch, not a single-request smoke test
- Pin the serving dtype explicitly in your deployment config instead of trusting a framework default
- Add a GPU utilization and p99 latency dashboard to your monitoring from day one
- Re-run the sizing exercise whenever you change model, concurrency target, or context length

## When To Escalate

If you have confirmed correct dtype, continuous batching, and reasonable GPU utilization but throughput still cannot meet your SLO at the concurrency you need, the problem is capacity, not configuration: you need a bigger or additional GPU, tensor parallelism across multiple GPUs, or a smaller/quantized model, which is an architecture and hardware sizing decision rather than a tuning fix.

## Faq

### Why is my model fast in a Jupyter notebook but slow behind an API?

**Answer:** A notebook typically runs one request at a time with no network, serialization, or guardrail overhead, and often no continuous batching either. Production adds concurrent traffic, HTTP overhead, and pre/post-processing, all of which the notebook test never exercises. Benchmark with the actual serving stack and realistic concurrency, not a bare model.generate() call.

### Does upgrading to a newer GPU always fix slow inference?

**Answer:** Only if the current GPU's memory bandwidth or capacity is actually the bottleneck. If the real issue is batch size 1, FP32 weights, or CPU-bound preprocessing, a faster GPU will still be underutilized. Fix software-level issues first, then re-measure before spending on hardware.

### What is the single highest-leverage fix for slow production inference?

**Answer:** In practice, enabling continuous batching at a sensible max-num-seqs value combined with BF16/FP16 weights closes most of the gap between benchmark and production throughput. Both are configuration changes, not hardware changes, and should be verified before any capacity investment.

## Search terms

llm inference slow production, vllm slow throughput, low tokens per second production, gpu underutilized inference, continuous batching vllm, fp32 vs bf16 inference speed, inference latency high production, why is my llm slow

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json