Why a model is too large for a single GPU, and how to actually fit it
ValueError: The model size exceeds the available GPU memory. Consider using a smaller model, quantization, or model parallelism
Also appears as
- torch.cuda.OutOfMemoryError: CUDA out of memory (model does not fit even at batch size 1)
- OSError: Not enough free disk/GPU memory to load the entire model
Short answer
This is a hard arithmetic ceiling, not a bug: a 70B parameter model needs about 140 GB in bf16, which exceeds every single current GPU. There are exactly three valid fixes: quantize the weights to reduce bytes-per-parameter, shard the model across multiple GPUs with tensor or pipeline parallelism, or choose a smaller model that fits your single-GPU budget at the precision you need.
Affects: Any model whose parameter count x bytes-per-parameter exceeds a single GPU's VRAM, most commonly 70B+ dense models on a single 24-80 GB GPU
Choose the right fix for your constraint
- 1If you have multiple GPUs available, shard the model with tensor parallelism: vllm serve MODEL --tensor-parallel-size N where N is your GPU count, so weights and KV cache both split across devices.
- 2If you are constrained to one GPU, quantize: 4-bit AWQ or GPTQ roughly quarters weight memory, often enough to bring a 70B model under 40 GB.
- 3If quantization alone is not enough, combine both: a 4-bit quantized 70B model split across 2 smaller GPUs.
- 4If neither is available, pick a smaller open-weight model in the same family (for example an 8B or 30B variant) and validate it meets your accuracy bar before assuming you need the larger one.
- 5Recalculate params_in_billions x bytes_per_param against your actual available VRAM (single GPU or aggregate) before committing to a deployment plan.
How to confirm this is your problem
- OOM occurs even at batch size 1 and the shortest possible sequence length, ruling out KV cache or activation memory as the cause
- The error occurs during weight loading itself, before the first forward pass
- nvidia-smi shows the single GPU's total capacity is smaller than the checkpoint's on-disk size at that precision
- A smaller model from the same family loads and serves without issue on the identical hardware
Root causes and fixes
The chosen model's parameter count times bytes-per-parameter genuinely exceeds the GPU's total VRAM
This is simple arithmetic: a 70B model in bf16 needs about 140 GB (70e9 x 2 bytes), which is larger than every single GPU on the market as of 2026 (H200 tops out at 141 GB, and that alone would leave zero room for KV cache). No configuration change fixes this; only reducing bytes-per-parameter (quantization) or spreading parameters across more than one GPU (parallelism) changes the equation.
Fix: Decide between quantization (same GPU count, less precision) and multi-GPU sharding (same precision, more hardware) based on whether accuracy or hardware budget is the harder constraint for your use case.
python -c "print(70e9*2/1e9, 'GB needed in bf16')"
The deployment defaulted to single-GPU serving even though multiple GPUs are physically present in the node
Serving frameworks do not automatically detect and use multiple GPUs unless explicitly told to via a tensor-parallel-size or device_map argument; without it, all frameworks attempt to fit the full model on the first visible device and fail identically to a genuine single-GPU box.
Fix: Explicitly configure tensor parallelism to match your physical GPU count so weight memory is actually sharded rather than replicated onto one device.
vllm serve MODEL --tensor-parallel-size 4
A full-precision checkpoint was chosen when a pre-quantized version of the same model exists
Most popular open-weight model releases have community or official AWQ, GPTQ, or GGUF quantized variants published alongside the full-precision weights; defaulting to the fp16/bf16 repo when a 4-bit variant would fit your hardware is a common avoidable choice, especially for teams new to a model family.
Fix: Search the model's HuggingFace repo or model card for an official or well-regarded community quantized variant before assuming full precision is required.
huggingface-cli search MODEL-AWQ
The model was selected based on benchmark leaderboard position without checking hardware fit for the deployment budget
Choosing the largest or highest-scoring open-weight model in a family without first checking whether its memory footprint fits the actual deployment hardware is a planning gap rather than a technical one, and it surfaces as this exact error the first time someone tries to load it.
Fix: Build model selection around a hardware-fit constraint from the start: shortlist models whose parameter count fits your GPU budget at your acceptable quantization level, then compare accuracy within that shortlist.
CPU or disk offload was expected to handle the shortfall automatically but was not configured
Some loaders (accelerate's device_map="auto" with offload_folder set) can spill excess weights to CPU RAM or disk when GPU memory runs out, but this requires explicit configuration; without it, the loader fails outright rather than silently offloading.
Fix: If offload is an acceptable tradeoff for your latency requirements, configure it explicitly rather than assuming it happens by default; otherwise treat this as the same hard capacity ceiling as the other causes.
model = AutoModelForCausalLM.from_pretrained(MODEL, device_map="auto", offload_folder="./offload")
Diagnostic commands
Confirm the exact memory requirement at your target precision
python -c "params=70e9; print('fp32', params*4/1e9, 'fp16/bf16', params*2/1e9, 'int8', params*1/1e9, 'int4', params*0.5/1e9, 'GB')"Compare each precision's requirement against your actual single-GPU VRAM; this tells you immediately which precisions are even theoretically possible on your hardware.
Check aggregate VRAM across all GPUs in the node
nvidia-smi --query-gpu=memory.total --format=csv,noheader | paste -sd+ | bc
If the model fits in aggregate multi-GPU memory but not on one device, tensor parallelism (not quantization) is the correct fix; if it does not fit even in aggregate, quantization or a smaller model is required regardless of GPU count.
List available pre-quantized variants for the model
huggingface-cli search MODEL
Official or community AWQ/GPTQ/GGUF repos are usually named with the base model plus a quantization suffix; finding one avoids doing quantization yourself and gives a known-good starting point.
Stopping it from happening again
- Make hardware-fit at your required precision a hard filter during model selection, before any accuracy benchmarking.
- Maintain a reference table mapping model size and precision to minimum GPU configuration for your team's common deployment targets.
- Default to checking for an existing quantized release before deciding full precision is necessary.
- Size new hardware purchases against your actual model shortlist's memory requirements, not generic GPU specs.
When this becomes an architecture problem
When your accuracy requirements genuinely need a model whose full-precision footprint exceeds your realistic multi-GPU budget even after quantization, that is a capacity planning and hardware procurement decision (how many GPUs, which interconnect, which node topology) rather than something to resolve with another configuration flag.
Frequently asked questions
How do I know if I need quantization or multiple GPUs?
Compare the model's memory requirement at your target precision against your single-GPU VRAM. If it fits within a modest multiple of your total available GPUs' combined VRAM, tensor parallelism preserves full accuracy at the cost of more hardware. If it does not fit even across your available GPUs, quantization is necessary regardless of GPU count, since it reduces the underlying memory requirement rather than just distributing it.
Is a 4-bit quantized 70B model as good as the full-precision version?
Well-calibrated 4-bit methods like AWQ typically show a small quality gap versus bf16, often under 1-2 points on standard benchmarks, and the gap has narrowed further with newer calibration techniques. For most production use cases the tradeoff is worth the roughly 4x memory reduction; validate on your specific task before committing, since sensitivity varies by task type.
Can I just add more GPUs to any model to make it fit?
Yes for any model, as long as the serving framework supports tensor or pipeline parallelism for that architecture, which vLLM, SGLang, and TensorRT-LLM all do for mainstream open-weight model families. The practical limit becomes interconnect bandwidth (NVLink versus PCIe) and cost, not whether it is technically possible.
What is the practical GPU memory ceiling I should plan around in 2026?
Single-GPU options top out around 141 GB (H200) at the high end, with 80 GB (H100/A100) and 48 GB (L40S) common in enterprise deployments, and 24 GB (RTX 4090) or 32 GB (RTX 5090) common in workstation and edge setups. Plan model and quantization choices against whichever of these tiers matches your actual procurement budget.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
NVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Free ToolMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Related problems
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
vLLM tensor-parallel-size must divide the number of attention heads
vLLM shards attention heads and KV heads evenly across GPUs for tensor parallelism, so --tensor-parallel-size must be a divisor of the model's head count and must equal the number of GPUs vLLM can actually see. Pick a TP size from the model's valid divisors, commonly 1, 2, 4, or 8, and make sure CUDA_VISIBLE_DEVICES exposes exactly that many GPUs.
How to reduce VRAM usage for LLM inference
VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideNVIDIA H100 vs H200 vs B200 for Enterprise AI in 2026
Compare NVIDIA H100, H200, and B200 GPUs on specs, price, availability, and performance per dollar for enterprise LLM inference and training in 2026.
GuideMulti-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.