Why you get CUDA out of memory just loading a model, and how to actually fix it
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 23.69 GiB total capacity; 21.84 GiB already allocated; 412.00 MiB free; 22.06 GiB reserved in total by PyTorch)
Also appears as
- RuntimeError: CUDA out of memory. Tried to allocate 4.50 GiB (GPU 0; 79.15 GiB total capacity)
- OSError: Can't load the model for 'meta-llama/Llama-3.1-70B'. CUDA out of memory
Short answer
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
Affects: Any CUDA GPU, most common loading 13B+ parameter models in fp16/bf16 on 24 GB cards (RTX 4090, L4, L40S) or 70B+ models on a single 80 GB A100/H100
Fastest path to a loaded model
- 1Compute the real requirement first: params_in_billions x 2 bytes for fp16/bf16, or x 1 byte for INT8, or x 0.5 byte for INT4/AWQ/GPTQ, then add roughly 10-20 percent for activation and framework overhead.
- 2If the number exceeds your single GPU's VRAM, load a pre-quantized checkpoint (AWQ or GPTQ 4-bit, or GGUF for llama.cpp/Ollama) instead of the full-precision weights.
- 3If you must keep full precision, split the model across GPUs: vllm serve MODEL --tensor-parallel-size 2 (or 4, 8) matching your GPU count.
- 4For HuggingFace transformers, set device_map="auto" and load_in_4bit=True (bitsandbytes) so the loader shards and quantizes automatically instead of failing on a single device.
- 5Re-run and confirm with nvidia-smi that memory used is now below total capacity with headroom for KV cache.
How to confirm this is your problem
- Process crashes during model.from_pretrained() or vllm serve startup, before any inference request is served
- nvidia-smi shows the GPU had free memory moments before the crash, then a spike to full
- Error names a Tried to allocate size close to the size of one model shard or layer, not the whole model
- Works fine on a smaller model checkpoint of the same family
Root causes and fixes
Loading full-precision (fp16/bf16) weights for a model larger than the GPU's VRAM
Every parameter costs 2 bytes in fp16/bf16. A 7B model needs about 14 GB, a 13B model about 26 GB, a 70B model about 140 GB, before any KV cache or activation memory. If that number alone exceeds your card's VRAM, the load fails deterministically regardless of batch size or context length.
Fix: Quantize to 4-bit (AWQ, GPTQ) or 8-bit before loading, or split the model with tensor parallelism across enough GPUs to cover the parameter memory.
python -c "print(7e9*2/1e9, 'GB for a 7B model in fp16')" vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --quantization awq
Default device_map loads the entire model onto GPU 0 even when multiple GPUs are present
HuggingFace transformers defaults to placing all weights on the first visible CUDA device unless device_map is explicitly set. On a multi-GPU box this means GPU 0 tries to hold 100 percent of the weights while the other GPUs sit idle, so the single-device OOM happens even though the cluster has enough aggregate VRAM.
Fix: Set device_map="auto" so accelerate shards layers across all visible GPUs, or use vllm tensor-parallel-size for proper tensor-sharded loading rather than layer-sharded loading.
model = AutoModelForCausalLM.from_pretrained(MODEL, device_map="auto", torch_dtype=torch.bfloat16)
Another process (a stale Python session, a Jupyter kernel, or a previous crashed server) still holds GPU memory
PyTorch does not always release CUDA memory back to the driver when a process exits abnormally (killed, OOM-killed itself, or ctrl-C mid-allocation). The zombie process keeps its CUDA context and allocation alive until the driver reclaims it, silently reducing the VRAM available to your new load.
Fix: Find and kill any leftover process holding the GPU before retrying the load.
nvidia-smi --query-compute-apps=pid,used_memory --format=csv kill -9 <pid>
CUDA_VISIBLE_DEVICES or a container GPU limit is silently restricting you to one small GPU instead of the full node
In Docker or Kubernetes, a misconfigured device request or an inherited CUDA_VISIBLE_DEVICES environment variable can mask all but one GPU from the process, so a model sized for the whole node tries to load onto a single card and fails.
Fix: Check the visible device list inside the container matches the physical GPU count before blaming the model size.
python -c "import torch; print(torch.cuda.device_count())" echo $CUDA_VISIBLE_DEVICES
The checkpoint itself is the wrong precision, for example an fp32 checkpoint was downloaded instead of the fp16/bf16 release
Some repos publish both fp32 and fp16 weight files. Pulling the fp32 variant by accident doubles the memory requirement (4 bytes per parameter instead of 2), which can push an otherwise-fitting model past the VRAM ceiling.
Fix: Confirm the dtype of the downloaded checkpoint and explicitly request torch_dtype=torch.bfloat16 on load so PyTorch casts down rather than keeping fp32 weights resident.
python -c "import torch; sd=torch.load('pytorch_model.bin', map_location='cpu'); print(next(iter(sd.values())).dtype)"Diagnostic commands
Check total and used VRAM before loading
nvidia-smi --query-gpu=memory.used,memory.total,memory.free --format=csv
If memory.used is already non-trivial before you start your process, something else is holding the GPU; free that first. If memory.total itself is smaller than params_in_billions x 2 GB, no software fix will make full-precision loading work on this card.
Confirm how many GPUs the process can actually see
python -c "import torch; print(torch.cuda.device_count(), [torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())])"
If this reports fewer devices than the physical GPU count on the box, you have a visibility or container configuration problem, not a genuine capacity problem.
Estimate the checkpoint's on-disk size as a proxy for VRAM need
du -sh ./model-weights/*.safetensors
Total safetensors size roughly equals the resident weight memory at that precision. If this number alone exceeds your GPU's VRAM, quantization or multi-GPU sharding is mandatory, not optional.
Stopping it from happening again
- Before choosing hardware, calculate params_in_billions x bytes_per_param for your target precision and compare against actual GPU VRAM, not the marketing number.
- Standardize on a quantization format (AWQ or FP8) for anything over 13B parameters in production so loading is deterministic across environments.
- Pin exact model revisions and precision in deployment configs so a stray fp32 download does not silently double memory needs.
- Add a pre-flight VRAM check in your deployment scripts that fails fast with a clear message instead of letting the OOM traceback be the first signal.
When this becomes an architecture problem
If the model family you need to run does not fit on your current GPU even at 4-bit quantization (for example a 400B+ dense model on single-node hardware), this is a hardware sizing decision, not a config fix. That is the point to size a multi-GPU node or a smaller open-weight model against your actual accuracy requirements before buying more hardware.
Frequently asked questions
How much VRAM does a 70B parameter model actually need?
In bf16/fp16, weights alone need about 140 GB (70 billion x 2 bytes). Add roughly 10-20 GB for KV cache and activations depending on batch size and context length, so a realistic full-precision deployment needs at least 160 GB of aggregate VRAM, typically two 80 GB or two 96 GB GPUs with tensor parallelism. At 4-bit (AWQ/GPTQ), the same model fits in about 35-40 GB, workable on a single 48 GB card.
Does quantization hurt output quality?
Well-calibrated 4-bit methods like AWQ and GPTQ typically cost a small, usually under 1-2 point, drop in standard benchmark scores versus fp16, while cutting memory by roughly 4x. FP8 on Hopper/Blackwell GPUs is closer to lossless. The tradeoff is almost always worth it for inference; for fine-tuning source weights, keep a higher-precision copy.
Why does the error say 'Tried to allocate 2 GiB' when my model is 140 GB?
PyTorch allocates memory incrementally as it loads tensors layer by layer, so the failure surfaces on whichever individual allocation first has no room left, not on the full model size. The reported number is the size of that one allocation, not the total memory the model needs.
Can I just add a second GPU instead of quantizing?
Yes, if you configure tensor parallelism so weights are actually sharded across both GPUs, for example vllm --tensor-parallel-size 2. Simply having a second GPU physically present does nothing unless the serving framework is told to shard onto it; transformers, vLLM, and SGLang all require an explicit parallelism argument.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Free ToolMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Related problems
The model is too large to fit on a single GPU
This is a hard arithmetic ceiling, not a bug: a 70B parameter model needs about 140 GB in bf16, which exceeds every single current GPU. There are exactly three valid fixes: quantize the weights to reduce bytes-per-parameter, shard the model across multiple GPUs with tensor or pipeline parallelism, or choose a smaller model that fits your single-GPU budget at the precision you need.
vLLM runs out of memory during startup, before serving any requests
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
How to reduce VRAM usage for LLM inference
VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideLLM Quantization: AWQ vs GPTQ vs FP8 vs GGUF
AWQ, GPTQ, FP8, and GGUF compared for production LLM serving: memory savings, throughput impact, quality loss, and which format fits which deployment.
GuideMulti-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.