What Is Inference?
Also known as: Model Inference, Serving
Definition
Inference is the phase in which a trained model is run to produce output from new input, as opposed to training, where the model's weights are being learned. In production, inference is where essentially all AI compute cost is spent.
Inference Explained
LLM inference has two distinct phases with different performance characteristics. Prefill processes the entire input prompt in parallel and is compute-bound, so a long prompt costs a burst of GPU work and determines time to first token. Decode then generates output one token at a time, each step reading all model weights from memory, which makes it memory-bandwidth-bound. This asymmetry is why prompt length and output length affect cost and latency in quite different ways.
The key-value cache is what keeps decode tractable. Rather than recomputing attention over the whole sequence at every step, the server caches per-token key and value tensors and appends to them. The cache grows linearly with sequence length and with the number of concurrent requests, and it lives in VRAM alongside the weights. On a busy server the KV cache can consume as much memory as the model itself, which is the single most common cause of unexpected out-of-memory failures.
Batching is the lever that turns hardware into throughput. Because decode is bandwidth-bound, processing several requests together amortizes the cost of reading weights across all of them. Continuous batching, where finished sequences leave the batch and new ones join mid-flight rather than waiting for the whole batch to complete, raises utilization dramatically compared with static batching and is standard in modern serving stacks such as vLLM and TensorRT-LLM.
Capacity planning should be expressed in the metrics users actually feel. Time to first token governs perceived responsiveness and is driven by prefill and queueing. Inter-token latency governs how fast text streams. Aggregate tokens per second governs how many people the box serves. These trade against each other: larger batches raise total throughput while lengthening individual latencies, so define a service level first and size hardware to meet it under realistic concurrency.
Why It Matters
- Inference, not training, is where nearly all ongoing AI cost lands, so serving efficiency dominates the total cost of ownership.
- KV cache growth under concurrency is the leading cause of production out-of-memory failures that never appear in single-user testing.
- Continuous batching can multiply effective throughput on the same hardware, often deferring a costly GPU expansion by many months.
- Latency targets and throughput targets conflict, so a defined service level is a prerequisite for any credible hardware sizing exercise.
In Practice
A pilot served eight engineers comfortably on one GPU. Opening it to two hundred users failed not because average load exceeded capacity but because concurrent long-context requests exhausted KV cache memory during a morning peak. Adding a queue with a concurrency cap fixed availability without any new hardware.
Frequently Asked Questions
What is the difference between training and inference?
Training adjusts model weights using gradient descent over large datasets and is a one-time or periodic capital-intensive activity. Inference uses the finished weights to produce output for a specific input and changes nothing permanent. Training needs far more memory per parameter because it stores gradients and optimizer state, while inference needs only the weights plus activations and cache.
How do I estimate how many users one GPU can serve?
Start with VRAM: subtract model weight size from card capacity, and divide the remainder by per-request KV cache size at your typical context length to get maximum concurrent requests. Then check throughput: measure tokens per second under that concurrency and compare against expected demand. Real user counts are usually several times concurrent request counts because people spend most of their time reading.
Related Terms
vLLM
vLLM is an open-source inference server for large language models that uses PagedAttention memory management and continuous batching to serve many concurrent users at high throughput, exposing an OpenAI-compatible HTTP API.
GPU VRAM
GPU VRAM is the high-bandwidth memory physically attached to a graphics processor. For AI serving it must simultaneously hold the model weights, the key-value cache for every active request, and activation buffers, which makes it the binding constraint on most deployments.
Token
A token is the basic unit of text a language model processes, usually a word fragment rather than a whole word. In English, one token averages roughly four characters, so about 100 tokens covers 75 words.
Go Deeper
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Working with Inference in a live environment? Our engineers do this every day - and our AI agents automate most of it.