How to reduce VRAM usage for LLM inference without hurting quality

Total inference VRAM use is the sum of three largely independent costs: weights (fixed, set at model choice and precision), KV cache (variable, grows with concurrent requests and context length), and activation/workspace memory (variable, grows with batch size and sequence length within a single forward pass). Treating VRAM as one undifferentiated number instead of these three budgets makes it hard to know which lever actually reduces usage.

Fix: Break down current usage into weights, KV cache, and activations using framework instrumentation or the underlying math, then attack whichever is largest rather than guessing at a single global setting.

python -c "params=7e9; bytes_per_param=2; print('weights GB', params*bytes_per_param/1e9)"

Weight memory is fixed and often the largest single line item; every bit of precision above what your accuracy requirements actually need is memory spent for no benefit. 4-bit quantization (AWQ, GPTQ) typically cuts weight memory to roughly a quarter of bf16 with modest accuracy impact, freeing that memory for KV cache or higher concurrency instead.

Fix: Benchmark a 4-bit quantized version of your model against your actual evaluation set; if quality holds, the freed memory can be redirected to context length or concurrency, which usually matters more to users than the last percent of raw accuracy.

vllm serve MODEL --quantization awq

KV cache memory scales linearly with the configured maximum context length, since the serving engine must reserve enough cache capacity for the worst-case sequence. A model with a 128k native context configured at that maximum reserves dramatically more KV cache per sequence than one capped at, say, 8k, even if actual usage rarely approaches the maximum.

Fix: Set max_model_len to a value that comfortably covers your real traffic's 95th or 99th percentile context length, not the model's theoretical ceiling, and monitor actual usage to validate the choice.

vllm serve MODEL --max-model-len 8192

Without an explicit cap on concurrent sequences, a serving engine will keep admitting new requests until KV cache memory is exhausted, meaning VRAM usage under peak traffic is effectively unbounded by the request pattern rather than by a deliberate operational limit.

Fix: Set an explicit maximum concurrent sequence or batch size limit appropriate to your GPU's KV cache budget, and use request queuing or rejection for traffic beyond that limit rather than letting the process OOM.

vllm serve MODEL --max-num-seqs 64

Standard (non-fused) attention implementations compute and store the full attention score matrix, which scales quadratically with sequence length and can dominate activation memory for long-context requests; fused kernels like FlashAttention avoid materializing this matrix entirely.

Fix: Confirm FlashAttention or an equivalent fused kernel is active (most modern serving frameworks enable it by default when the GPU and library versions support it) rather than falling back to a naive attention path.

python -c "import torch; print(torch.backends.cuda.flash_sdp_enabled())"