What Is GPU VRAM?
Also known as: Video RAM, GPU Memory
Definition
GPU VRAM is the high-bandwidth memory physically attached to a graphics processor. For AI serving it must simultaneously hold the model weights, the key-value cache for every active request, and activation buffers, which makes it the binding constraint on most deployments.
GPU VRAM Explained
VRAM is the primary sizing constraint because a model cannot run at speed unless its weights fit. Weight footprint is straightforward to calculate: parameter count multiplied by bytes per parameter, which is two at 16-bit and roughly half a byte plus overhead at 4-bit. A 70-billion-parameter model therefore needs about 140 GB at 16-bit or around 35 to 40 GB at 4-bit. Data center accelerators commonly ship with 40, 80, 94, or more gigabytes per card.
Weights are only the first term. KV cache is the second and it is dynamic, growing with both context length and the number of concurrent requests. Activation buffers and the CUDA context add a further working allowance. A realistic plan reserves roughly ten to twenty percent headroom beyond the calculated total, because fragmentation and transient spikes cause failures well before nominal capacity is reached.
Bandwidth matters as much as capacity and is quoted far less often in procurement discussions. Token generation is memory-bandwidth-bound: every decode step reads the full set of active weights. A card with high capacity but modest bandwidth will hold a big model and generate tokens slowly. This is why cards using high-bandwidth memory dominate serving deployments and why consumer cards with similar advertised capacity underperform on throughput.
When a model exceeds a single card, you have three options. Quantize to fit, which is usually tried first. Split across GPUs using tensor parallelism, which requires fast interconnect between cards to avoid a communication bottleneck. Or offload layers to system RAM or NVMe, which works functionally but is typically an order of magnitude slower and is rarely acceptable for interactive use. Most enterprise designs choose quantization first and multi-GPU second.
Why It Matters
- Determines which models you can run at all, making it the first hard gate in any on-prem AI hardware decision.
- Remaining VRAM after weights sets concurrent user capacity through KV cache, so it drives how many people one server supports.
- Bandwidth, not just capacity, governs tokens per second, and specification sheets that emphasize capacity alone mislead buyers.
- Insufficient headroom causes intermittent out-of-memory failures under peak load that never reproduce in single-user testing.
In Practice
Worked example: a 48 GB card running a 4-bit 70B model uses roughly 40 GB for weights. That leaves about 8 GB, minus overhead, for KV cache. At a few hundred megabytes per request with long ERP context, realistic concurrency is a handful of simultaneous requests, not the dozens the raw card capacity suggests.
Frequently Asked Questions
How much VRAM do I need to run a 70B model?
At 4-bit quantization, weights alone need roughly 35 to 40 GB, so a single 48 GB or 80 GB card can host the model. For useful multi-user serving, plan for 80 GB or more so there is substantial memory left for KV cache. At 16-bit precision the same model needs about 140 GB and therefore multiple GPUs with tensor parallelism.
Can I use system RAM if I run out of VRAM?
Yes, most serving frameworks can offload layers to system memory or disk, and the model will run. Throughput typically falls by an order of magnitude because every offloaded layer crosses the PCIe bus each decode step. Offloading is reasonable for batch jobs run overnight and generally unacceptable for interactive users waiting at a terminal.
Related Terms
Quantization
Quantization is the technique of storing a model's weights at lower numeric precision, for example 8-bit or 4-bit integers instead of 16-bit floats, which sharply reduces GPU memory use and often increases speed at a modest accuracy cost.
Tensor Parallelism
Tensor parallelism is a technique that splits the weight matrices inside each model layer across several GPUs, so every card holds a slice and computes part of each operation, letting a model run that no single GPU could hold.
Inference
Inference is the phase in which a trained model is run to produce output from new input, as opposed to training, where the model's weights are being learned. In production, inference is where essentially all AI compute cost is spent.
Go Deeper
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Working with GPU VRAM in a live environment? Our engineers do this every day - and our AI agents automate most of it.