vLLM Production Deployment: A Practitioner's Guide
vLLM is the default choice for self-hosted LLM inference in 2026 because PagedAttention and continuous batching solve the two problems that made earlier serving stacks fall over under real traffic: wasted KV cache memory and idle GPU time between requests. But running vllm serve with default flags on a single GPU is not a production deployment. Production means picking the right --gpu-memory-utilization, --max-num-seqs, and --max-model-len for your actual traffic shape, understanding how continuous batching interacts with your latency SLAs, and instrumenting the metrics that tell you when you are about to fall over before your users do. This guide covers what changes between a demo and a deployment that survives a Tuesday afternoon traffic spike.
PagedAttention and Continuous Batching, in Practice
PagedAttention treats the KV cache like virtual memory: it allocates cache in fixed-size blocks rather than one contiguous chunk per sequence, which eliminates the internal fragmentation that used to waste 60 to 80 percent of KV cache memory in naive serving implementations. Continuous batching (also called in-flight batching) means vLLM does not wait for a full batch to finish before starting new requests; as soon as any sequence in the batch finishes generating, its slot is freed and a queued request steps in on the next iteration. The combined effect is that GPU utilization stays high even under bursty, variable-length request patterns, which is the normal traffic shape for a chat or agent workload and the exact case that static batching handled badly.
- PagedAttention blocks are typically 16 tokens each, configurable via --block-size
- Continuous batching means throughput scales with concurrent requests, not with a fixed batch window
- Preemption can occur under memory pressure: vLLM will swap or recompute a sequence's KV cache if it needs the space, which shows up as latency spikes worth monitoring
- Prefix caching (on by default in recent versions) reuses KV cache blocks across requests sharing a prompt prefix, which matters a lot for system-prompt-heavy or RAG workloads
The Config Flags That Actually Matter
Most vLLM production incidents trace back to three flags left at defaults. --gpu-memory-utilization (default 0.9) controls how much of the GPU's VRAM vLLM reserves for weights plus KV cache; set it too high and you get OOM crashes under load, too low and you starve concurrent capacity. --max-num-seqs caps how many sequences run concurrently in one batch, which is your primary throughput-versus-latency lever: higher values raise throughput but increase per-request latency as requests compete for compute each iteration. --max-model-len should match your actual context need, not the model's advertised maximum, because KV cache allocation scales with it and an unnecessarily large value silently shrinks your concurrent capacity. Add --enable-chunked-prefill for mixed short-generation and long-context workloads, since it prevents a single long prompt from blocking the decode step of everything else in the batch.
- --gpu-memory-utilization: start at 0.85-0.90, back off if you see OOM under peak concurrency
- --max-num-seqs: tune against your p95 latency budget, not against a throughput benchmark alone
- --max-model-len: set to your real p99 context length plus headroom, never the model card maximum by default
- --enable-prefix-caching and --enable-chunked-prefill: both on by default in current vLLM, verify they are actually active in your version
- --tensor-parallel-size: set to match GPU count for models that do not fit on one card, covered in more detail in the multi-GPU serving guide
Sizing for Real Traffic, Not Benchmark Traffic
Public vLLM benchmarks report tokens per second under sustained maximum load, which is not how production traffic arrives. Real traffic has a request-rate distribution, a prompt-length distribution, and a generation-length distribution, and the interaction between all three determines your actual capacity. Load test with a traffic replay that matches your production logs, not a synthetic uniform-length benchmark. Pay particular attention to time-to-first-token (TTFT) under concurrent load, since TTFT degrades nonlinearly as queue depth grows, well before throughput saturates. A server that reports 4000 tokens per second aggregate throughput can still have unacceptable TTFT at 40 concurrent users if your prompts are long and your --max-num-seqs is set for a short-prompt workload.
Deployment Topology and Rolling Updates
Run vLLM behind a load balancer with multiple replicas rather than one large instance, even if a single GPU has capacity, because a single vLLM process is a single point of failure and model reloads take minutes. Use vLLM's OpenAI-compatible API server mode so client code is portable across providers. For rolling updates, drain connections gracefully: vLLM does not natively support zero-downtime weight swaps, so blue-green deployment at the load balancer level, not in-place restarts, is the pattern that avoids dropped requests during a model or config change. Health checks should verify the model actually responds to a real inference request, not just that the HTTP port is open, since a hung CUDA context can leave the port responsive while inference silently stalls.
- Multiple smaller replicas beat one large instance for fault isolation and rolling deploys
- Blue-green or canary at the load balancer, not in-place process restarts
- Health checks must run a real short inference call, not just a TCP probe
- Pin vLLM and CUDA driver versions together; minor version mismatches are a common source of silent perf regressions
How Netray Deploys vLLM for On-Prem Clients
Netray builds vLLM deployments on customer-owned GPUs as part of on-prem inference engagements, which means every config flag gets tuned against the client's actual traffic replay rather than a public benchmark. We size --max-num-seqs and --gpu-memory-utilization against measured TTFT and inter-token latency targets, set up multi-replica topology with health checks that catch stalled CUDA contexts, and hand over a Grafana dashboard tracking the metrics covered in our LLM observability guide. For regulated manufacturing and defense clients this means production-grade inference entirely inside your network boundary, with no request ever leaving to a third-party API.
Frequently Asked Questions
What GPU memory utilization should I set for vLLM in production?
Start at 0.85 to 0.90 and load test at your expected peak concurrency. --gpu-memory-utilization controls how much VRAM vLLM reserves for model weights plus KV cache; setting it too high risks OOM crashes when KV cache demand spikes under load, while setting it too low wastes concurrent capacity you paid for. Back it off in 0.05 increments if you see out-of-memory errors during load testing, and always test with your real prompt-length distribution, not a synthetic benchmark.
How does vLLM continuous batching differ from static batching?
Static batching waits for a fixed batch to fully complete before starting new requests, which wastes GPU cycles once shorter sequences in the batch finish generating. Continuous batching (in-flight batching) frees a sequence's slot the moment it completes and immediately admits a queued request on the next decode step, keeping GPU utilization high under the bursty, variable-length traffic that real production workloads produce. This is the single biggest throughput improvement vLLM delivers over naive serving loops.
Do I need multiple vLLM replicas or is one large instance enough?
Use multiple replicas behind a load balancer even if one GPU has spare capacity. A single vLLM process is a single point of failure, model or config changes require a restart that takes minutes, and vLLM has no native zero-downtime weight swap. Multiple smaller replicas give you fault isolation and let you do blue-green or canary deployment at the load balancer level instead of dropping requests during every rolling update.
What is the most common mistake in vLLM production deployments?
Leaving --max-model-len at the model's advertised maximum context length instead of your actual measured need. KV cache allocation scales directly with max-model-len, so an unnecessarily large value silently shrinks how many concurrent sequences the same GPU memory can support, cutting your real throughput without any error message. Set it to your measured p99 context length plus a safety margin, not the model card number.
Key Takeaways
- 1PagedAttention and Continuous Batching, in Practice: PagedAttention treats the KV cache like virtual memory: it allocates cache in fixed-size blocks rather than one contiguous chunk per sequence, which eliminates the internal fragmentation that used to waste 60 to 80 percent of KV cache memory in naive serving implementations. Continuous batching (also called in-flight batching) means vLLM does not wait for a full batch to finish before starting new requests; as soon as any sequence in the batch finishes generating, its slot is freed and a queued request steps in on the next iteration.
- 2The Config Flags That Actually Matter: Most vLLM production incidents trace back to three flags left at defaults. --gpu-memory-utilization (default 0.9) controls how much of the GPU's VRAM vLLM reserves for weights plus KV cache; set it too high and you get OOM crashes under load, too low and you starve concurrent capacity.
- 3Sizing for Real Traffic, Not Benchmark Traffic: Public vLLM benchmarks report tokens per second under sustained maximum load, which is not how production traffic arrives. Real traffic has a request-rate distribution, a prompt-length distribution, and a generation-length distribution, and the interaction between all three determines your actual capacity.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Terms used in this article
Deploying vLLM in production and want the config validated against your real traffic? Netray will benchmark your workload and tune the deployment before it goes live.
Related Resources
KV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
AI & AutomationLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
AI & AutomationSGLang vs vLLM: An Honest Serving Comparison
SGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.