# Why your fine-tuning loss won't go down, and how to fix it

> Source: https://netray.co/troubleshoot/training-loss-not-decreasing
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Training loss that stays flat is most often caused by a LoRA adapter that targets the wrong modules (missing q_proj/k_proj/v_proj/o_proj), a learning rate that is too low for LoRA or too high and bouncing, or labels that were never masked so the model is trying to learn the prompt tokens as if they were random noise. Check target_modules first, then the label mask, then the learning rate.

## Error Title

Training loss not decreasing during fine-tuning

## Error Signature

loss: 2.31, 2.30, 2.31, 2.30, 2.31 (loss flat across steps, not converging)

## Alternate Signatures

- train_loss stays the same value for the entire run
- eval_loss barely moves after 3 epochs

## Category

fine-tuning

## Stack

- pytorch
- huggingface
- transformers
- accelerate

## Affects

LoRA and full fine-tuning with HuggingFace transformers, PEFT, and TRL, any base model, any GPU

## Quick Fix

### Fastest path to a moving loss curve

**Steps:**

Print your LoraConfig.target_modules and confirm it includes q_proj, k_proj, v_proj, and o_proj (not just a partial list like just q_proj and v_proj on a model that needs all four).

Print one training batch and check that labels contains -100 for every prompt/instruction token and only real token ids for the response tokens; if labels equals input_ids everywhere, masking was skipped.

Confirm the LoRA layers actually have requires_grad=True by running print([n for n,p in model.named_parameters() if p.requires_grad]) and verifying it is non-empty and matches the adapter layers.

Set the learning rate explicitly: 1e-4 to 2e-4 for LoRA on a 7B-70B model, 1e-5 to 2e-5 for full fine-tuning, and rerun for 50 steps to see if loss moves at all.

Verify the chat template applied to your data matches the base model's expected template (tokenizer.apply_chat_template), since a mismatched template can make the loss objective nearly impossible to optimize.

## Symptoms

- Loss value barely changes from the first logged step to the last
- Loss oscillates in a narrow band (for example 2.28 to 2.34) without a downward trend
- Eval loss is identical to train loss and both are flat
- Generated samples during or after training look unchanged from the base model
- Gradient norm logged by the trainer is exactly 0.0 or NaN every step

## Causes

### LoRA target_modules does not cover the attention projections

**Likelihood:** most-common

**Explanation:** If target_modules only lists one or two projection names, or uses a name that does not match the actual module names in this architecture (they vary between Llama, Qwen, and Mistral-style models), PEFT silently attaches adapters to nothing meaningful. The frozen base model then dominates the forward pass and there are effectively no useful trainable parameters influencing the loss.

**Fix:** Set target_modules to the full attention block for the architecture: typically q_proj, k_proj, v_proj, o_proj, and optionally gate_proj, up_proj, down_proj for MLP coverage. Use target_modules="all-linear" in recent PEFT versions if you are unsure of exact names.

**Commands:**

- python -c "from peft import LoraConfig; print(LoraConfig(target_modules=['q_proj','k_proj','v_proj','o_proj']))"
- python -c "import torch; m=torch.load('adapter_model.bin', map_location='cpu'); print(list(m.keys())[:10])"

### Labels not masked, so the model is trained to predict the prompt

**Likelihood:** common

**Explanation:** Many training scripts build labels by simply cloning input_ids. Without setting the prompt portion of labels to -100, the loss function computes cross-entropy over prompt tokens too, which are highly variable and unpredictable from the model's perspective. This inflates and flattens the loss because the model can never really learn to predict someone else's instruction text.

**Fix:** Mask every prompt and system token with -100 in labels, leaving only the assistant response tokens (plus EOS) as real targets. Most SFT trainers (TRL's SFTTrainer with a proper data collator) do this automatically if you pass the correct response template.

**Commands:**

- python -c "import torch; b=next(iter(train_dataloader)); print((b['labels']==-100).float().mean())"

### Learning rate too low for the optimizer and rank chosen

**Likelihood:** common

**Explanation:** LoRA adds a small number of new parameters initialized near zero, and a learning rate tuned for full fine-tuning (like 2e-5) is often an order of magnitude too small to move those parameters meaningfully within a short run, especially with a cosine schedule and warmup that keeps the effective rate low for the first several hundred steps.

**Fix:** For LoRA, start at 1e-4 to 2e-4 with rank 8-32. For full fine-tuning, use 1e-5 to 2e-5. Watch the first 20-50 steps; if loss has not started dropping, increase the rate by 3-5x before assuming something else is broken.

### Base weights frozen but adapter never attached to the forward pass

**Likelihood:** occasional

**Explanation:** This happens when a model is loaded once for tokenization or inference and a second, unwrapped copy is passed to the trainer, or when get_peft_model() is called but the returned object is discarded instead of being the one actually trained. The optimizer then has zero real parameters to update even though training appears to run.

**Fix:** Confirm model.print_trainable_parameters() reports a non-zero, sensible percentage (typically 0.1% to 2% of total parameters for LoRA) immediately after wrapping with get_peft_model, and make sure that exact object reference is what gets passed to the Trainer.

### Wrong or missing chat template causing malformed training examples

**Likelihood:** rare

**Explanation:** If you format data with a generic template (or none at all) that doesn't match the special tokens the base model expects, the model sees training sequences that look nothing like anything in its pretraining distribution. Loss can flatten at a high value because every example is effectively out-of-distribution noise rather than a learnable pattern.

**Fix:** Use tokenizer.apply_chat_template with the model's own tokenizer config, and manually inspect 2-3 fully rendered training strings to confirm special tokens (like <|im_start|> or [INST]) appear correctly before training.

## Diagnostics

### Check trainable parameter count

**Command:** python -c "m.print_trainable_parameters()"

**Interpretation:** Should show a nonzero count, typically 0.1%-2% of total params for LoRA on a 7B-70B model. A count of 0 or a percentage near 0.0000% means the adapter is not actually attached to trainable weights.

### Inspect one batch's label mask

**Command:** python -c "b=next(iter(dl)); print((b['labels']==-100).float().mean().item())"

**Interpretation:** A value near 0.0 means almost nothing is masked (labels equal input_ids, which is wrong). A value between roughly 0.5 and 0.9 is typical for instruction data where the prompt is longer than the response.

### Confirm gradients are flowing

**Command:** python -c "print(sum(p.grad.abs().sum().item() for p in model.parameters() if p.grad is not None))"

**Interpretation:** A nonzero sum after one backward pass confirms gradients exist. A value of exactly 0.0 across all parameters means either everything is frozen or the loss graph is disconnected from the trainable weights.

### Log learning rate schedule at each step

**Command:** python -c "print(trainer.lr_scheduler.get_last_lr())"

**Interpretation:** If the effective rate is far lower than expected during the first 100-200 steps due to a long warmup, loss will look flat even though training is technically correct; shortening warmup or raising base LR resolves it.

## Prevention

- Run a 20-50 step smoke test on a tiny data subset before committing to a full run, and confirm loss visibly drops within that window.
- Log print_trainable_parameters() and a sample of the label mask automatically at the start of every training script.
- Pin PEFT, transformers, and TRL versions together in a lockfile since target_modules defaults have changed across versions.
- Keep a known-good reference config (rank, alpha, target_modules, learning rate) per model family and diff new configs against it.
- Version-control your chat template and data formatting function alongside the training script, not just the raw dataset.

## When To Escalate

If loss moves correctly on a small smoke test but the full run still plateaus after fixing masking, targets, and LR, the problem is likely data quality or label noise at scale rather than configuration, and that calls for a systematic dataset audit rather than more hyperparameter guessing. If you are fine-tuning multiple model families or need this to work reliably across a training pipeline serving several teams, that is an architecture and process problem worth bringing in outside help for.

## Faq

### What learning rate should I use for LoRA fine-tuning?

**Answer:** Start at 1e-4 to 2e-4 for LoRA with rank 8-32 on a 7B-70B model. This is roughly 5-10x higher than typical full fine-tuning rates because LoRA updates a much smaller set of parameters that need a larger step to move meaningfully within a short training run.

### How do I know if my labels are masked correctly?

**Answer:** Load one training batch and check that a meaningful fraction of the labels tensor equals -100 (the ignore index), corresponding to prompt and system tokens. If labels is identical to input_ids with no -100 values, masking was never applied and the model is being trained to predict the prompt itself.

### Does target_modules matter that much for LoRA?

**Answer:** Yes. Missing even one of the four attention projections (q_proj, k_proj, v_proj, o_proj) meaningfully reduces what the adapter can express, and using a name that doesn't exist in that architecture means PEFT attaches to nothing, silently producing a run that looks like training but changes almost nothing.

### Is a flat loss always a bug, or could the task just be hard?

**Answer:** A genuinely hard task still shows some downward trend, even if slow, within the first 100-200 steps. A loss that is bit-for-bit flat or oscillating in a tiny band with zero trend is a strong signal of a configuration bug (masking, targets, or frozen weights) rather than task difficulty.

## Search terms

training loss not decreasing, loss stuck fine-tuning, LoRA loss flat, loss not converging LLM, target_modules attention projections, labels not masked training, learning rate LoRA fine-tuning, print_trainable_parameters zero

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json