# Why a model is too large for a single GPU, and how to actually fit it

> Source: https://netray.co/troubleshoot/model-too-large-for-single-gpu
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

This is a hard arithmetic ceiling, not a bug: a 70B parameter model needs about 140 GB in bf16, which exceeds every single current GPU. There are exactly three valid fixes: quantize the weights to reduce bytes-per-parameter, shard the model across multiple GPUs with tensor or pipeline parallelism, or choose a smaller model that fits your single-GPU budget at the precision you need.

## Error Title

The model is too large to fit on a single GPU

## Error Signature

ValueError: The model size exceeds the available GPU memory. Consider using a smaller model, quantization, or model parallelism

## Alternate Signatures

- torch.cuda.OutOfMemoryError: CUDA out of memory (model does not fit even at batch size 1)
- OSError: Not enough free disk/GPU memory to load the entire model

## Category

gpu-memory

## Stack

- pytorch
- vllm
- transformers
- accelerate

## Affects

Any model whose parameter count x bytes-per-parameter exceeds a single GPU's VRAM, most commonly 70B+ dense models on a single 24-80 GB GPU

## Quick Fix

### Choose the right fix for your constraint

**Steps:**

- If you have multiple GPUs available, shard the model with tensor parallelism: vllm serve MODEL --tensor-parallel-size N where N is your GPU count, so weights and KV cache both split across devices.
- If you are constrained to one GPU, quantize: 4-bit AWQ or GPTQ roughly quarters weight memory, often enough to bring a 70B model under 40 GB.
- If quantization alone is not enough, combine both: a 4-bit quantized 70B model split across 2 smaller GPUs.
- If neither is available, pick a smaller open-weight model in the same family (for example an 8B or 30B variant) and validate it meets your accuracy bar before assuming you need the larger one.
- Recalculate params_in_billions x bytes_per_param against your actual available VRAM (single GPU or aggregate) before committing to a deployment plan.

## Symptoms

- OOM occurs even at batch size 1 and the shortest possible sequence length, ruling out KV cache or activation memory as the cause
- The error occurs during weight loading itself, before the first forward pass
- nvidia-smi shows the single GPU's total capacity is smaller than the checkpoint's on-disk size at that precision
- A smaller model from the same family loads and serves without issue on the identical hardware

## Causes

### The chosen model's parameter count times bytes-per-parameter genuinely exceeds the GPU's total VRAM

**Likelihood:** most-common

**Explanation:** This is simple arithmetic: a 70B model in bf16 needs about 140 GB (70e9 x 2 bytes), which is larger than every single GPU on the market as of 2026 (H200 tops out at 141 GB, and that alone would leave zero room for KV cache). No configuration change fixes this; only reducing bytes-per-parameter (quantization) or spreading parameters across more than one GPU (parallelism) changes the equation.

**Fix:** Decide between quantization (same GPU count, less precision) and multi-GPU sharding (same precision, more hardware) based on whether accuracy or hardware budget is the harder constraint for your use case.

**Commands:**

- python -c "print(70e9*2/1e9, 'GB needed in bf16')"

### The deployment defaulted to single-GPU serving even though multiple GPUs are physically present in the node

**Likelihood:** common

**Explanation:** Serving frameworks do not automatically detect and use multiple GPUs unless explicitly told to via a tensor-parallel-size or device_map argument; without it, all frameworks attempt to fit the full model on the first visible device and fail identically to a genuine single-GPU box.

**Fix:** Explicitly configure tensor parallelism to match your physical GPU count so weight memory is actually sharded rather than replicated onto one device.

**Commands:**

- vllm serve MODEL --tensor-parallel-size 4

### A full-precision checkpoint was chosen when a pre-quantized version of the same model exists

**Likelihood:** common

**Explanation:** Most popular open-weight model releases have community or official AWQ, GPTQ, or GGUF quantized variants published alongside the full-precision weights; defaulting to the fp16/bf16 repo when a 4-bit variant would fit your hardware is a common avoidable choice, especially for teams new to a model family.

**Fix:** Search the model's HuggingFace repo or model card for an official or well-regarded community quantized variant before assuming full precision is required.

**Commands:**

- huggingface-cli search MODEL-AWQ

### The model was selected based on benchmark leaderboard position without checking hardware fit for the deployment budget

**Likelihood:** occasional

**Explanation:** Choosing the largest or highest-scoring open-weight model in a family without first checking whether its memory footprint fits the actual deployment hardware is a planning gap rather than a technical one, and it surfaces as this exact error the first time someone tries to load it.

**Fix:** Build model selection around a hardware-fit constraint from the start: shortlist models whose parameter count fits your GPU budget at your acceptable quantization level, then compare accuracy within that shortlist.

### CPU or disk offload was expected to handle the shortfall automatically but was not configured

**Likelihood:** rare

**Explanation:** Some loaders (accelerate's device_map="auto" with offload_folder set) can spill excess weights to CPU RAM or disk when GPU memory runs out, but this requires explicit configuration; without it, the loader fails outright rather than silently offloading.

**Fix:** If offload is an acceptable tradeoff for your latency requirements, configure it explicitly rather than assuming it happens by default; otherwise treat this as the same hard capacity ceiling as the other causes.

**Commands:**

- model = AutoModelForCausalLM.from_pretrained(MODEL, device_map="auto", offload_folder="./offload")

## Diagnostics

### Confirm the exact memory requirement at your target precision

**Command:** python -c "params=70e9; print('fp32', params*4/1e9, 'fp16/bf16', params*2/1e9, 'int8', params*1/1e9, 'int4', params*0.5/1e9, 'GB')"

**Interpretation:** Compare each precision's requirement against your actual single-GPU VRAM; this tells you immediately which precisions are even theoretically possible on your hardware.

### Check aggregate VRAM across all GPUs in the node

**Command:** nvidia-smi --query-gpu=memory.total --format=csv,noheader | paste -sd+ | bc

**Interpretation:** If the model fits in aggregate multi-GPU memory but not on one device, tensor parallelism (not quantization) is the correct fix; if it does not fit even in aggregate, quantization or a smaller model is required regardless of GPU count.

### List available pre-quantized variants for the model

**Command:** huggingface-cli search MODEL

**Interpretation:** Official or community AWQ/GPTQ/GGUF repos are usually named with the base model plus a quantization suffix; finding one avoids doing quantization yourself and gives a known-good starting point.

## Prevention

- Make hardware-fit at your required precision a hard filter during model selection, before any accuracy benchmarking.
- Maintain a reference table mapping model size and precision to minimum GPU configuration for your team's common deployment targets.
- Default to checking for an existing quantized release before deciding full precision is necessary.
- Size new hardware purchases against your actual model shortlist's memory requirements, not generic GPU specs.

## When To Escalate

When your accuracy requirements genuinely need a model whose full-precision footprint exceeds your realistic multi-GPU budget even after quantization, that is a capacity planning and hardware procurement decision (how many GPUs, which interconnect, which node topology) rather than something to resolve with another configuration flag.

## Faq

### How do I know if I need quantization or multiple GPUs?

**Answer:** Compare the model's memory requirement at your target precision against your single-GPU VRAM. If it fits within a modest multiple of your total available GPUs' combined VRAM, tensor parallelism preserves full accuracy at the cost of more hardware. If it does not fit even across your available GPUs, quantization is necessary regardless of GPU count, since it reduces the underlying memory requirement rather than just distributing it.

### Is a 4-bit quantized 70B model as good as the full-precision version?

**Answer:** Well-calibrated 4-bit methods like AWQ typically show a small quality gap versus bf16, often under 1-2 points on standard benchmarks, and the gap has narrowed further with newer calibration techniques. For most production use cases the tradeoff is worth the roughly 4x memory reduction; validate on your specific task before committing, since sensitivity varies by task type.

### Can I just add more GPUs to any model to make it fit?

**Answer:** Yes for any model, as long as the serving framework supports tensor or pipeline parallelism for that architecture, which vLLM, SGLang, and TensorRT-LLM all do for mainstream open-weight model families. The practical limit becomes interconnect bandwidth (NVLink versus PCIe) and cost, not whether it is technically possible.

### What is the practical GPU memory ceiling I should plan around in 2026?

**Answer:** Single-GPU options top out around 141 GB (H200) at the high end, with 80 GB (H100/A100) and 48 GB (L40S) common in enterprise deployments, and 24 GB (RTX 4090) or 32 GB (RTX 5090) common in workstation and edge setups. Plan model and quantization choices against whichever of these tiers matches your actual procurement budget.

## Search terms

model too large for gpu, 70b model gpu memory, model does not fit on single gpu, tensor parallelism vs quantization, how much vram for large language model, llm model size exceeds gpu memory, choose gpu for llm deployment

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json