# Why your fine-tuning loss becomes NaN, and how to fix it

> Source: https://netray.co/troubleshoot/nan-loss-during-training
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

NaN loss during training is most often caused by fp16 numeric overflow in gradients or activations, which bf16 avoids because of its wider exponent range. Other common causes are a learning rate spike (especially right after warmup), a small number of corrupt or malformed training samples, and unsafe division or log operations in a custom loss function. Switch to bf16 first if your hardware supports it, then check for corrupt samples and unstable LR.

## Error Title

Loss becomes NaN during fine-tuning

## Error Signature

loss: nan

## Alternate Signatures

- RuntimeError: Function 'LogSoftmaxBackward0' returned nan values
- grad_norm: nan

## Category

fine-tuning

## Stack

- pytorch
- huggingface
- transformers
- accelerate

## Affects

Fine-tuning with fp16 mixed precision most commonly, also occurs with bf16 and full fp32 training

## Quick Fix

### Fastest path to a stable loss

**Steps:**

- If your GPU supports it (Ampere or newer), switch from fp16 to bf16 mixed precision, since bf16's wider exponent range eliminates most overflow-driven NaNs immediately.
- If you must use fp16, enable gradient clipping (max_grad_norm=1.0) and confirm the loss scaler is not stuck at a very high scale value.
- Lower the learning rate and/or lengthen the warmup period, since a rate spike right after warmup ends is a common trigger for the first NaN step.
- Scan your dataset for empty strings, all-padding sequences, or extremely long outlier sequences and filter them out before training.
- If using a custom loss function, add epsilon values to any division or log operation and clamp inputs to avoid log(0) or division by zero.

## Symptoms

- Loss value logs as literally nan starting at a specific step, often early in training or right after a learning rate change
- Gradient norm (grad_norm) also shows nan or inf at the same step the loss goes NaN
- Training was stable for many steps and then suddenly diverges to NaN with no configuration change
- Loss scaler value (if using fp16 with dynamic scaling) has grown very large just before the NaN appears
- Model outputs become garbage or repetitive tokens immediately after the NaN step if training continues

## Causes

### fp16 numeric overflow in activations or gradients

**Likelihood:** most-common

**Explanation:** fp16 has a much smaller representable exponent range than fp32 or bf16. During backpropagation, gradient values or intermediate activations (especially in attention softmax or large logit values) can exceed fp16's maximum representable magnitude, overflowing to inf, and any subsequent arithmetic on an inf value produces NaN.

**Fix:** Switch to bf16 mixed precision on any Ampere-or-newer GPU (A100, H100, RTX 30-series and later), which has the same exponent range as fp32 and essentially eliminates this overflow class of NaN.

**Commands:**

- accelerate launch --mixed_precision bf16 train.py

### Learning rate spike, often right after warmup ends

**Likelihood:** common

**Explanation:** If the learning rate schedule ramps up too aggressively or peaks higher than the model can tolerate at that point in training, a single large weight update can push activations into an unstable numeric range. This is especially common the first time the rate hits its full peak value right after a short warmup period.

**Fix:** Extend the warmup period (more steps at a lower rate before reaching peak LR) and lower the peak learning rate, particularly for full fine-tuning where sensitivity to LR spikes is higher than with LoRA.

### Corrupt or malformed training samples

**Likelihood:** common

**Explanation:** A small number of pathological examples (empty strings after tokenization, sequences that are entirely padding with no real content, or encoding errors that produce out-of-vocabulary token ids) can produce degenerate loss values on the specific batch that contains them, and that single bad batch's NaN then propagates through the optimizer state to every subsequent step.

**Fix:** Filter the dataset for empty or near-empty sequences after tokenization, validate that all token ids fall within the tokenizer's actual vocabulary size, and log which batch index triggers the first NaN to identify the specific sample.

**Commands:**

- python -c "import datasets; ds=datasets.load_from_disk('train'); print(min(len(x['input_ids']) for x in ds))"

### Unsafe division or log operation in a custom loss function

**Likelihood:** occasional

**Explanation:** Custom loss implementations (weighted cross-entropy, custom regularization terms, or manual perplexity calculations) sometimes divide by a count that can be zero for certain batches, or take log of a value that can reach exactly zero due to floating point underflow. Either operation produces NaN or inf that then contaminates the entire loss graph.

**Fix:** Add a small epsilon (like 1e-8) to any denominator or log argument in custom loss code, and use torch.clamp to keep values within a safe numeric range before the operation.

### Gradient accumulation interacting badly with loss scaling

**Likelihood:** rare

**Explanation:** When using gradient accumulation with fp16 dynamic loss scaling, the scaler can grow the scale factor during a run of stable steps and then overflow on the very next step if the true gradient magnitude was borderline, particularly right at an accumulation boundary where accumulated gradients are finally unscaled and applied.

**Fix:** Reduce the initial loss scale or disable dynamic scale growth if this pattern recurs, or switch to bf16 which does not require loss scaling at all and removes this failure mode entirely.

## Diagnostics

### Check which precision mode is active

**Command:** python -c "from transformers import TrainingArguments; print(TrainingArguments.fp16, TrainingArguments.bf16)"

**Interpretation:** Confirms whether fp16 or bf16 is active in your current TrainingArguments. If fp16 is True and your GPU supports bf16, switching removes the most common source of NaN outright.

### Log gradient norm every step to find the exact failure point

**Command:** python -c "print(trainer.state.log_history)"

**Interpretation:** Find the exact step where grad_norm first shows nan or inf. If it correlates with a learning rate schedule transition (end of warmup, restart), that points to an LR-driven spike rather than a data issue.

### Isolate the batch that first produces NaN

**Command:** python -c "for i,b in enumerate(dl):\n    out=model(**b)\n    if torch.isnan(out.loss): print(i); break"

**Interpretation:** If a specific batch index reliably reproduces the NaN, inspect that batch's raw text and token ids directly for empty sequences, encoding errors, or unexpectedly extreme lengths.

### Check for inf or NaN in model weights after the failure

**Command:** python -c "print(any(torch.isnan(p).any() for p in model.parameters()))"

**Interpretation:** If weights themselves already contain NaN, the run must be restarted from the last good checkpoint since NaN weights cannot recover on their own; if weights are still clean, the NaN is confined to the current batch's loss computation.

## Prevention

- Default to bf16 mixed precision on any hardware that supports it instead of fp16.
- Always enable gradient clipping (max_grad_norm around 1.0) as a cheap safeguard against rare gradient spikes.
- Validate and clean datasets (no empty sequences, no out-of-vocabulary tokens, reasonable length bounds) before every training run, not just once at dataset creation.
- Checkpoint frequently (every few hundred steps) so a NaN failure only costs a small amount of retraining time.
- Add automated NaN detection that halts training and dumps the offending batch immediately rather than letting corrupted optimizer state silently continue.

## When To Escalate

If NaN losses recur across multiple datasets and configurations even after switching to bf16, clipping gradients, and cleaning data, the issue may be in a custom model modification, a specific base model's numerical sensitivity, or the training framework version itself, and is worth a deeper systematic review. If this is happening in a production fine-tuning pipeline that customers or internal teams depend on, building automated NaN detection and rollback into the pipeline is worth doing properly rather than firefighting each occurrence.

## Faq

### Does switching from fp16 to bf16 fix NaN loss?

**Answer:** In most cases yes, because bf16 has the same exponent range as fp32 and does not overflow the way fp16 can during backpropagation. bf16 requires Ampere-generation or newer NVIDIA GPUs (A100, H100, RTX 30-series and later); older GPUs cannot use it and need other mitigations like gradient clipping and lower learning rates.

### Can I recover a training run after the loss becomes NaN?

**Answer:** Not from the point of failure itself, since NaN values in the optimizer state and model weights do not resolve on their own. You need to resume from the last saved checkpoint before the NaN occurred and address the underlying cause (precision, learning rate, or data) before continuing.

### How do I find which training example is causing NaN loss?

**Answer:** Log the batch index alongside the loss value at every step, then re-run inference on just that batch outside the training loop to isolate whether a specific sample (empty text, corrupt tokens, or extreme length) is responsible versus a general precision or learning rate issue.

### Is gradient clipping enough to prevent NaN loss on its own?

**Answer:** Gradient clipping helps but does not fully prevent NaN, since it clips the gradient after it exists and cannot fix an already-overflowed fp16 value or a NaN produced inside a custom loss calculation. Combine clipping with bf16 precision and data validation for reliable protection.

## Search terms

NaN loss fine-tuning, loss nan training LLM, fp16 overflow training, bf16 vs fp16 stability, grad_norm nan, gradient clipping NaN loss, corrupt training samples NaN, loss scaler overflow fp16

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json