# How to reduce VRAM usage for LLM inference without hurting quality

> Source: https://netray.co/troubleshoot/reduce-vram-usage-llm-inference
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.

## Error Title

How to reduce VRAM usage for LLM inference

## Error Signature

Need to reduce GPU memory usage for LLM inference without a specific error, or: torch.cuda.OutOfMemoryError under production load that a smaller test load did not trigger

## Alternate Signatures

- How to fit a larger model on my GPU
- vLLM/transformers using more VRAM than expected in production

## Category

gpu-memory

## Stack

- vllm
- pytorch
- huggingface
- transformers

## Affects

Any on-prem LLM serving deployment where VRAM is the binding constraint on model size, context length, or concurrency

## Quick Fix

### Cut VRAM usage in order of impact

**Steps:**

Quantize the weights first, since they are usually the largest fixed cost: AWQ or GPTQ 4-bit cuts weight memory roughly 4x versus bf16 with a small accuracy cost.

Cap --max-model-len (vLLM) or max_length (transformers) to your application's real context need rather than the model's architectural maximum, since KV cache scales directly with it.

Reduce max concurrent sequences or batch size if KV cache for many simultaneous requests is the dominant cost, since KV cache also scales linearly with concurrent batch size.

Use FlashAttention or an equivalent memory-efficient attention kernel, which avoids materializing the full attention matrix and meaningfully cuts activation memory for long sequences.

If still short, add tensor parallelism across a second GPU so both weights and KV cache are sharded rather than trying to fit everything on one device.

## Symptoms

- Model serves fine at low concurrency or short context but OOMs as production traffic or context length increases
- VRAM usage is close to the GPU's total capacity even during light load, leaving no headroom for traffic spikes
- Reducing batch size or max_model_len resolves the OOM but at a concurrency or context ceiling that is too low for the product requirement
- Same model at a different quantization level (or on a bigger GPU) does not exhibit the problem

## Causes

### Weight memory, KV cache memory, and activation memory are all drawing from the same fixed VRAM pool without an explicit budget for each

**Likelihood:** most-common

**Explanation:** Total inference VRAM use is the sum of three largely independent costs: weights (fixed, set at model choice and precision), KV cache (variable, grows with concurrent requests and context length), and activation/workspace memory (variable, grows with batch size and sequence length within a single forward pass). Treating VRAM as one undifferentiated number instead of these three budgets makes it hard to know which lever actually reduces usage.

**Fix:** Break down current usage into weights, KV cache, and activations using framework instrumentation or the underlying math, then attack whichever is largest rather than guessing at a single global setting.

**Commands:**

- python -c "params=7e9; bytes_per_param=2; print('weights GB', params*bytes_per_param/1e9)"

### Weights are kept in full bf16/fp16 precision when a quantized variant would serve the same requests

**Likelihood:** common

**Explanation:** Weight memory is fixed and often the largest single line item; every bit of precision above what your accuracy requirements actually need is memory spent for no benefit. 4-bit quantization (AWQ, GPTQ) typically cuts weight memory to roughly a quarter of bf16 with modest accuracy impact, freeing that memory for KV cache or higher concurrency instead.

**Fix:** Benchmark a 4-bit quantized version of your model against your actual evaluation set; if quality holds, the freed memory can be redirected to context length or concurrency, which usually matters more to users than the last percent of raw accuracy.

**Commands:**

- vllm serve MODEL --quantization awq

### max_model_len is set to the model's architectural maximum context rather than what the application uses

**Likelihood:** common

**Explanation:** KV cache memory scales linearly with the configured maximum context length, since the serving engine must reserve enough cache capacity for the worst-case sequence. A model with a 128k native context configured at that maximum reserves dramatically more KV cache per sequence than one capped at, say, 8k, even if actual usage rarely approaches the maximum.

**Fix:** Set max_model_len to a value that comfortably covers your real traffic's 95th or 99th percentile context length, not the model's theoretical ceiling, and monitor actual usage to validate the choice.

**Commands:**

- vllm serve MODEL --max-model-len 8192

### Batch size or maximum concurrent sequences is unbounded, letting KV cache grow unpredictably under load

**Likelihood:** occasional

**Explanation:** Without an explicit cap on concurrent sequences, a serving engine will keep admitting new requests until KV cache memory is exhausted, meaning VRAM usage under peak traffic is effectively unbounded by the request pattern rather than by a deliberate operational limit.

**Fix:** Set an explicit maximum concurrent sequence or batch size limit appropriate to your GPU's KV cache budget, and use request queuing or rejection for traffic beyond that limit rather than letting the process OOM.

**Commands:**

- vllm serve MODEL --max-num-seqs 64

### The attention implementation is a naive or non-optimized kernel that materializes large intermediate tensors

**Likelihood:** rare

**Explanation:** Standard (non-fused) attention implementations compute and store the full attention score matrix, which scales quadratically with sequence length and can dominate activation memory for long-context requests; fused kernels like FlashAttention avoid materializing this matrix entirely.

**Fix:** Confirm FlashAttention or an equivalent fused kernel is active (most modern serving frameworks enable it by default when the GPU and library versions support it) rather than falling back to a naive attention path.

**Commands:**

- python -c "import torch; print(torch.backends.cuda.flash_sdp_enabled())"

## Diagnostics

### Break down current VRAM usage by category

**Command:** nvidia-smi --query-gpu=memory.used,memory.total --format=csv

**Interpretation:** Comparing total used against your independently calculated weight memory reveals how much is KV cache plus activations, telling you which budget to target next.

### Check actual context length distribution in production traffic

**Command:** grep -o 'prompt_tokens\":[0-9]*' access.log | sort -n | uniq -c | tail -20

**Interpretation:** If your 99th percentile actual context length is far below max_model_len, you have unused KV cache reservation that can be reclaimed by lowering the configured maximum.

### Confirm FlashAttention or equivalent is active

**Command:** python -c "import torch; print(torch.backends.cuda.flash_sdp_enabled(), torch.backends.cuda.mem_efficient_sdp_enabled())"

**Interpretation:** Both should report True on supported GPUs (Ampere and later); False indicates a fallback path that costs more activation memory than necessary for long sequences.

## Prevention

- Track weight, KV cache, and activation memory as separate metrics in your monitoring rather than one aggregate VRAM number.
- Set max_model_len and max concurrency from measured production traffic percentiles, revisited quarterly, not from defaults.
- Evaluate quantized model variants as a standing part of your deployment checklist, not only when hardware runs out.
- Load test at realistic concurrency and context length before production cutover to reveal the true VRAM ceiling.

## When To Escalate

If you have already quantized the weights, capped context length and concurrency to real traffic needs, and confirmed efficient attention kernels are active, and VRAM is still the binding constraint on serving the traffic your business requires, that is a signal to add GPUs (tensor parallelism) or move to a larger-memory GPU tier rather than continuing to tune software settings.

## Faq

### What uses the most VRAM: weights, KV cache, or activations?

**Answer:** For most production serving configurations, weights are the largest fixed cost (fully determined by model size and precision), while KV cache is the largest variable cost that grows with concurrency and context length. Activation memory during inference is typically the smallest of the three unless sequences are extremely long, since inference does not need to retain activations for a backward pass the way training does.

### Does quantizing the KV cache help as much as quantizing the weights?

**Answer:** KV cache quantization (for example FP8 KV cache, supported on some hardware in vLLM) roughly halves KV cache memory, which matters most in high-concurrency, long-context deployments where KV cache dominates VRAM usage. It is a smaller absolute saving than weight quantization in typical deployments but becomes increasingly valuable as concurrency and context length grow.

### Is there a formula I can use to estimate my VRAM budget?

**Answer:** Yes: total VRAM needed is approximately (params x bytes_per_param) for weights, plus (2 x num_layers x num_kv_heads x head_dim x total_tokens_across_all_concurrent_sequences x bytes_per_element) for KV cache, plus a smaller activation term that depends on your attention kernel and batch size. Plugging in your specific model config and target concurrency gives a reasonably accurate planning number.

### Should I reduce max_model_len even if some users occasionally need longer context?

**Answer:** Consider routing long-context requests to a separate, appropriately sized serving pool rather than reserving worst-case KV cache for every request on your main pool. This keeps the common case efficient while still supporting the occasional long-context need through a deliberately provisioned path.

## Search terms

reduce vram usage llm inference, lower gpu memory llm serving, vllm max_model_len memory, llm inference memory optimization, kv cache memory reduction, quantize model to save vram, flashattention memory savings

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json