What Is vLLM?
Also known as: vLLM Inference Server, PagedAttention
Definition
vLLM is an open-source inference server for large language models that uses PagedAttention memory management and continuous batching to serve many concurrent users at high throughput, exposing an OpenAI-compatible HTTP API.
vLLM Explained
The core innovation is PagedAttention. Earlier serving stacks allocated a contiguous KV cache block per request sized to the maximum possible sequence length, wasting most of it because real sequences are shorter. PagedAttention borrows the idea of virtual memory paging, storing the cache in fixed-size blocks that are allocated on demand and referenced through a block table. Memory waste drops dramatically, and the recovered capacity translates directly into more concurrent requests on the same GPU.
Continuous batching is the second pillar. Rather than assembling a batch, running it to completion, and starting the next, vLLM maintains a running batch and swaps finished sequences out for waiting ones at every decode step. Short requests are no longer stuck behind long ones, GPU utilization stays high, and tail latency improves substantially compared with static batching under mixed workloads.
Operationally, vLLM is straightforward to adopt because it presents an OpenAI-compatible REST interface. Applications written against that API can be pointed at an internal endpoint by changing a base URL and key, which makes migrating a prototype from a cloud provider to on-prem hardware a configuration change rather than a rewrite. It supports tensor parallelism across GPUs, common quantization formats, streaming responses, and serving multiple LoRA adapters over a shared base model.
It is not the only choice and not always the best one. TensorRT-LLM can deliver higher throughput on NVIDIA hardware at the cost of a compilation step and more operational complexity. llama.cpp and Ollama are far simpler for single-user or CPU-bound edge scenarios. vLLM occupies the practical middle: strong multi-user throughput with reasonable operational burden, which is why it has become a common default for on-prem enterprise serving.
Why It Matters
- PagedAttention recovers KV cache memory that other servers waste, often multiplying concurrent user capacity on identical hardware.
- OpenAI-compatible API means moving a working prototype from cloud to on-prem is a base-URL change rather than an application rewrite.
- Continuous batching keeps short requests from queueing behind long ones, which is what makes shared department-wide use feel responsive.
- Multi-LoRA serving lets one base model in memory back several department-specific behaviors, avoiding a separate deployment per variant.
In Practice
A team replaced a naive serving script with vLLM on unchanged hardware and saw concurrent capacity rise several times over, purely from KV cache efficiency and continuous batching. The planned GPU purchase was deferred two quarters. Measure your current server's memory waste before assuming you need more cards.
Frequently Asked Questions
Is vLLM suitable for air-gapped deployment?
Yes. vLLM is open source and runs entirely offline once the model weights and Python dependencies are staged locally. The usual preparation is to mirror the container image, the package index, and the weight files into the enclave through your approved transfer process. There is no license server, telemetry requirement, or outbound call needed for normal operation.
How does vLLM compare to Ollama?
Ollama optimizes for single-user simplicity: one command pulls and runs a model, with good CPU and consumer GPU support. vLLM optimizes for multi-user throughput on server GPUs through PagedAttention and continuous batching. Use Ollama for workstations, edge devices, and experimentation; use vLLM when a shared service must handle many concurrent users reliably.
Related Terms
Inference
Inference is the phase in which a trained model is run to produce output from new input, as opposed to training, where the model's weights are being learned. In production, inference is where essentially all AI compute cost is spent.
GPU VRAM
GPU VRAM is the high-bandwidth memory physically attached to a graphics processor. For AI serving it must simultaneously hold the model weights, the key-value cache for every active request, and activation buffers, which makes it the binding constraint on most deployments.
Tensor Parallelism
Tensor parallelism is a technique that splits the weight matrices inside each model layer across several GPUs, so every card holds a slice and computes part of each operation, letting a model run that no single GPU could hold.
Go Deeper
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
Working with vLLM in a live environment? Our engineers do this every day - and our AI agents automate most of it.
Try it here, free
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Keep exploring
Free tools
- vLLM Throughput EstimatorEstimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
- vLLM Docker Compose Generator for a Private ERP LLMFill in the model, context length and GPU count to get a docker-compose file that serves an OpenAI-compatible endpoint inside your network.
- GPU Sizing Calculator for LLM InferenceWork out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Guides
- vLLM Production Deployment: A Practitioner's GuideDeploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
- LLM Batching and Throughput Tuning: A Field GuideTune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
- SGLang vs vLLM: An Honest Production Serving ComparisonSGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.
Troubleshooting
- Why LLM inference is much slower in production than your benchmarks, and how to fix itProduction LLM inference is almost always slower than benchmarks because of FP32 weights, batch size 1, wrong hardware, or CPU-bound pre/post-processing. Here is the fix order.
- Why vLLM's OpenAI-compatible API returns 404, and how to fix the requestThis 404 is almost always a wrong route or a model name mismatch. Match your request's model field and URL path to what /v1/models actually returns.