# Why CUDA runs out of memory during fine-tuning even when inference works fine

> Source: https://netray.co/troubleshoot/torch-cuda-outofmemoryerror-fine-tuning
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Training needs far more memory than inference for the same model because it must hold weights, gradients, optimizer states, and activations simultaneously. AdamW alone adds about 8 bytes per parameter for its two fp32 moment buffers, so full fine-tuning of a 7B model can need 60-70+ GB versus about 14 GB for inference of the same weights. The fix is LoRA/QLoRA to shrink trainable parameters, gradient checkpointing to shrink activation memory, or a paged/8-bit optimizer to shrink optimizer state.

## Error Title

CUDA out of memory during fine-tuning

## Error Signature

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.24 GiB. GPU 0 has a total capacity of 79.15 GiB of which 623.12 MiB is free

## Alternate Signatures

- RuntimeError: CUDA error: out of memory during backward pass
- OutOfMemoryError: CUDA out of memory when calling optimizer.step()

## Category

gpu-memory

## Stack

- pytorch
- transformers
- accelerate
- deepspeed
- huggingface

## Affects

Full fine-tuning and LoRA/QLoRA training with HuggingFace Trainer, TRL, or Axolotl on any CUDA GPU, most common with full-parameter fine-tuning of 7B+ models on a single 40-80 GB GPU

## Quick Fix

### Cut training memory without changing your dataset or model

**Steps:**

- Switch from full fine-tuning to LoRA or QLoRA so only adapter parameters (often under 1 percent of the model) carry gradients and optimizer state.
- Enable gradient checkpointing (gradient_checkpointing=True in TrainingArguments) to trade compute time for a large activation memory reduction.
- Use a memory-efficient optimizer: bitsandbytes paged_adamw_8bit instead of the default AdamW, which cuts optimizer state from 8 bytes/param to about 2 bytes/param.
- Reduce per_device_train_batch_size and raise gradient_accumulation_steps to keep the same effective batch size at a fraction of the activation memory.
- If still short, add DeepSpeed ZeRO Stage 2 or 3 to shard optimizer state and gradients across GPUs instead of replicating them on each device.

## Symptoms

- Model loads fine and the first forward pass succeeds, but OOM hits on the backward pass or optimizer.step()
- OOM appears a few steps or a few epochs into training, not immediately, as memory creeps up
- Reducing batch size to 1 still fails, pointing at optimizer state or weight memory rather than activations
- Works with LoRA but fails with full fine-tuning on the identical hardware and dataset

## Causes

### Full fine-tuning with AdamW keeps four copies of the trainable parameters in memory at once

**Likelihood:** most-common

**Explanation:** Mixed-precision full fine-tuning needs fp16/bf16 weights (2 bytes), an fp32 master copy (4 bytes), and AdamW's two fp32 moment buffers, m and v (4 bytes each, 8 bytes total). That is roughly 14-18 bytes per trainable parameter before activations, versus 2 bytes for pure inference, which is why the identical GPU that serves a model comfortably cannot fine-tune it in full precision.

**Fix:** Switch to LoRA or QLoRA so the frozen base weights need no optimizer state at all, and only the small adapter matrices carry the 8 bytes/param AdamW overhead.

**Commands:**

- pip install peft
- from peft import LoraConfig, get_peft_model
- model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"]))

### Activation memory grows with batch size, sequence length, and number of layers, and is not checkpointed

**Likelihood:** common

**Explanation:** Every layer's forward-pass activations must be kept in memory until the backward pass consumes them, and that memory scales roughly linearly with batch_size x sequence_length x hidden_size x num_layers. Long context fine-tuning or large batches multiply this quickly, and without checkpointing it often dwarfs the weight and optimizer memory combined.

**Fix:** Enable gradient checkpointing so only a subset of activations are kept and the rest are recomputed during backward, typically cutting activation memory by 60-80 percent at the cost of extra compute.

**Commands:**

- training_args = TrainingArguments(gradient_checkpointing=True, per_device_train_batch_size=1, gradient_accumulation_steps=16)

### Default AdamW keeps optimizer states in fp32 even under mixed-precision training

**Likelihood:** common

**Explanation:** Mixed precision only affects the forward/backward compute dtype; by default the optimizer's internal moment buffers remain fp32 for numerical stability, which is where the 8 bytes/param figure comes from. This is invisible in the training script but shows up directly in memory profiles.

**Fix:** Use bitsandbytes 8-bit or paged AdamW variants, which quantize optimizer states to roughly 2 bytes/param with minimal convergence impact for most fine-tuning runs.

**Commands:**

- from transformers import TrainingArguments
- TrainingArguments(optim="paged_adamw_8bit")

### Evaluation runs during training accumulate memory because eval batches are not detached from the autograd graph

**Likelihood:** occasional

**Explanation:** If an eval loop calls the model without wrapping it in torch.no_grad() or model.eval() context correctly, PyTorch keeps building the autograd graph for eval batches too, silently adding to memory that never gets freed until the run ends or crashes.

**Fix:** Wrap all evaluation and generation calls in torch.no_grad() and confirm model.eval() is set, so no gradient graph is retained for inference-only forward passes.

**Commands:**

- with torch.no_grad():
-     outputs = model(**eval_batch)

### Dataset examples with unusually long sequences spike activation memory on specific batches

**Likelihood:** rare

**Explanation:** If sequence lengths are not capped or sorted, a rare very-long example in an otherwise short-sequence dataset can produce a batch whose activation memory is many times the typical batch, causing an OOM that appears random and step-dependent rather than consistent.

**Fix:** Set a hard max_seq_length truncation and consider length-based bucketing/sorting so batches contain similarly sized sequences instead of one outlier forcing worst-case memory on every step.

**Commands:**

- tokenizer(text, truncation=True, max_length=2048)

## Diagnostics

### Profile memory by training phase

**Command:** python -c "import torch; print(torch.cuda.memory_allocated()/1e9, torch.cuda.memory_reserved()/1e9)" # call after forward, after backward, after optimizer.step()

**Interpretation:** A large jump specifically at optimizer.step() confirms optimizer state is the bottleneck; a jump during backward with a huge batch points to activations; a jump during model load points to base weights.

### Check trainable parameter count versus total

**Command:** python -c "print(sum(p.numel() for p in model.parameters() if p.requires_grad), sum(p.numel() for p in model.parameters()))"

**Interpretation:** If trainable count equals total count, you are doing full fine-tuning and paying the full 8 bytes/param optimizer cost; if it is a small fraction, LoRA is already active and the OOM is more likely activation-driven.

### Watch memory over the first few training steps

**Command:** nvidia-smi --query-gpu=memory.used --format=csv -l 1

**Interpretation:** Memory that climbs steadily rather than plateauing after step 2-3 suggests a leak (often ungathered eval graphs or growing logging buffers) rather than a fixed-size configuration problem.

## Prevention

- Default new fine-tuning projects to LoRA/QLoRA and only justify full fine-tuning when adapter-based training demonstrably underperforms on your task.
- Always enable gradient checkpointing for anything above a 7B base model as a standing default, not an emergency fix.
- Calculate expected memory (weights plus optimizer plus activations) before a training run using your batch size and sequence length, not after it crashes.
- Cap max_seq_length explicitly in the tokenizer rather than relying on the dataset to be well-behaved.

## When To Escalate

When even LoRA plus gradient checkpointing plus 8-bit optimizer still cannot fit your target batch size and sequence length on the largest single GPU you have, that is a signal to move to multi-GPU training with DeepSpeed ZeRO or FSDP, which is an infrastructure decision worth planning rather than patching around.

## Faq

### Why does full fine-tuning need so much more memory than inference?

**Answer:** Inference only needs the weights resident, about 2 bytes per parameter in bf16. Training additionally needs gradients (another 2 bytes/param), an fp32 master weight copy (4 bytes/param), and AdamW's two fp32 moment buffers (8 bytes/param combined), plus activation memory that scales with batch size and sequence length. The combined multiplier is commonly 12-20x the inference footprint.

### Does LoRA really avoid all that overhead?

**Answer:** Yes, because LoRA freezes the base model weights entirely; only the small low-rank adapter matrices, typically well under 1 percent of total parameters, require gradients and optimizer state. The base weights still need to be resident in memory (or in 4-bit with QLoRA), but the optimizer state that dominates full fine-tuning memory shrinks by roughly two orders of magnitude.

### Is gradient checkpointing safe to leave on permanently?

**Answer:** Yes for most workloads. It recomputes activations during the backward pass instead of storing them, typically adding 20-30 percent to wall-clock training time in exchange for a large activation memory reduction. Leave it on by default for any model where memory is a binding constraint, and only disable it if you have confirmed spare VRAM and want the throughput back.

### Can I fine-tune a 70B model on a single 80 GB GPU?

**Answer:** Not with full-parameter fine-tuning; the optimizer state and gradients alone exceed 80 GB before activations. QLoRA (4-bit base weights plus LoRA adapters) makes this feasible on a single 80 GB GPU for moderate batch sizes and sequence lengths, which is exactly the technique that made single-GPU fine-tuning of large models practical.

## Search terms

torch.cuda.OutOfMemoryError fine-tuning, cuda out of memory training llm, adamw memory per parameter, lora vs full fine-tuning memory, gradient checkpointing out of memory, fine-tuning oom backward pass, qlora memory requirements, optimizer state memory llm

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json