Why your fine-tuning loss won't go down, and how to fix it
loss: 2.31, 2.30, 2.31, 2.30, 2.31 (loss flat across steps, not converging)
Also appears as
- train_loss stays the same value for the entire run
- eval_loss barely moves after 3 epochs
Short answer
Training loss that stays flat is most often caused by a LoRA adapter that targets the wrong modules (missing q_proj/k_proj/v_proj/o_proj), a learning rate that is too low for LoRA or too high and bouncing, or labels that were never masked so the model is trying to learn the prompt tokens as if they were random noise. Check target_modules first, then the label mask, then the learning rate.
Affects: LoRA and full fine-tuning with HuggingFace transformers, PEFT, and TRL, any base model, any GPU
Fastest path to a moving loss curve
- 1Print your LoraConfig.target_modules and confirm it includes q_proj, k_proj, v_proj, and o_proj (not just a partial list like just q_proj and v_proj on a model that needs all four).
- 2Print one training batch and check that labels contains -100 for every prompt/instruction token and only real token ids for the response tokens; if labels equals input_ids everywhere, masking was skipped.
- 3Confirm the LoRA layers actually have requires_grad=True by running print([n for n,p in model.named_parameters() if p.requires_grad]) and verifying it is non-empty and matches the adapter layers.
- 4Set the learning rate explicitly: 1e-4 to 2e-4 for LoRA on a 7B-70B model, 1e-5 to 2e-5 for full fine-tuning, and rerun for 50 steps to see if loss moves at all.
- 5Verify the chat template applied to your data matches the base model's expected template (tokenizer.apply_chat_template), since a mismatched template can make the loss objective nearly impossible to optimize.
How to confirm this is your problem
- Loss value barely changes from the first logged step to the last
- Loss oscillates in a narrow band (for example 2.28 to 2.34) without a downward trend
- Eval loss is identical to train loss and both are flat
- Generated samples during or after training look unchanged from the base model
- Gradient norm logged by the trainer is exactly 0.0 or NaN every step
Root causes and fixes
LoRA target_modules does not cover the attention projections
If target_modules only lists one or two projection names, or uses a name that does not match the actual module names in this architecture (they vary between Llama, Qwen, and Mistral-style models), PEFT silently attaches adapters to nothing meaningful. The frozen base model then dominates the forward pass and there are effectively no useful trainable parameters influencing the loss.
Fix: Set target_modules to the full attention block for the architecture: typically q_proj, k_proj, v_proj, o_proj, and optionally gate_proj, up_proj, down_proj for MLP coverage. Use target_modules="all-linear" in recent PEFT versions if you are unsure of exact names.
python -c "from peft import LoraConfig; print(LoraConfig(target_modules=['q_proj','k_proj','v_proj','o_proj']))"
python -c "import torch; m=torch.load('adapter_model.bin', map_location='cpu'); print(list(m.keys())[:10])"Labels not masked, so the model is trained to predict the prompt
Many training scripts build labels by simply cloning input_ids. Without setting the prompt portion of labels to -100, the loss function computes cross-entropy over prompt tokens too, which are highly variable and unpredictable from the model's perspective. This inflates and flattens the loss because the model can never really learn to predict someone else's instruction text.
Fix: Mask every prompt and system token with -100 in labels, leaving only the assistant response tokens (plus EOS) as real targets. Most SFT trainers (TRL's SFTTrainer with a proper data collator) do this automatically if you pass the correct response template.
python -c "import torch; b=next(iter(train_dataloader)); print((b['labels']==-100).float().mean())"
Learning rate too low for the optimizer and rank chosen
LoRA adds a small number of new parameters initialized near zero, and a learning rate tuned for full fine-tuning (like 2e-5) is often an order of magnitude too small to move those parameters meaningfully within a short run, especially with a cosine schedule and warmup that keeps the effective rate low for the first several hundred steps.
Fix: For LoRA, start at 1e-4 to 2e-4 with rank 8-32. For full fine-tuning, use 1e-5 to 2e-5. Watch the first 20-50 steps; if loss has not started dropping, increase the rate by 3-5x before assuming something else is broken.
Base weights frozen but adapter never attached to the forward pass
This happens when a model is loaded once for tokenization or inference and a second, unwrapped copy is passed to the trainer, or when get_peft_model() is called but the returned object is discarded instead of being the one actually trained. The optimizer then has zero real parameters to update even though training appears to run.
Fix: Confirm model.print_trainable_parameters() reports a non-zero, sensible percentage (typically 0.1% to 2% of total parameters for LoRA) immediately after wrapping with get_peft_model, and make sure that exact object reference is what gets passed to the Trainer.
Wrong or missing chat template causing malformed training examples
If you format data with a generic template (or none at all) that doesn't match the special tokens the base model expects, the model sees training sequences that look nothing like anything in its pretraining distribution. Loss can flatten at a high value because every example is effectively out-of-distribution noise rather than a learnable pattern.
Fix: Use tokenizer.apply_chat_template with the model's own tokenizer config, and manually inspect 2-3 fully rendered training strings to confirm special tokens (like <|im_start|> or [INST]) appear correctly before training.
Diagnostic commands
Check trainable parameter count
python -c "m.print_trainable_parameters()"
Should show a nonzero count, typically 0.1%-2% of total params for LoRA on a 7B-70B model. A count of 0 or a percentage near 0.0000% means the adapter is not actually attached to trainable weights.
Inspect one batch's label mask
python -c "b=next(iter(dl)); print((b['labels']==-100).float().mean().item())"
A value near 0.0 means almost nothing is masked (labels equal input_ids, which is wrong). A value between roughly 0.5 and 0.9 is typical for instruction data where the prompt is longer than the response.
Confirm gradients are flowing
python -c "print(sum(p.grad.abs().sum().item() for p in model.parameters() if p.grad is not None))"
A nonzero sum after one backward pass confirms gradients exist. A value of exactly 0.0 across all parameters means either everything is frozen or the loss graph is disconnected from the trainable weights.
Log learning rate schedule at each step
python -c "print(trainer.lr_scheduler.get_last_lr())"
If the effective rate is far lower than expected during the first 100-200 steps due to a long warmup, loss will look flat even though training is technically correct; shortening warmup or raising base LR resolves it.
Stopping it from happening again
- Run a 20-50 step smoke test on a tiny data subset before committing to a full run, and confirm loss visibly drops within that window.
- Log print_trainable_parameters() and a sample of the label mask automatically at the start of every training script.
- Pin PEFT, transformers, and TRL versions together in a lockfile since target_modules defaults have changed across versions.
- Keep a known-good reference config (rank, alpha, target_modules, learning rate) per model family and diff new configs against it.
- Version-control your chat template and data formatting function alongside the training script, not just the raw dataset.
When this becomes an architecture problem
If loss moves correctly on a small smoke test but the full run still plateaus after fixing masking, targets, and LR, the problem is likely data quality or label noise at scale rather than configuration, and that calls for a systematic dataset audit rather than more hyperparameter guessing. If you are fine-tuning multiple model families or need this to work reliably across a training pipeline serving several teams, that is an architecture and process problem worth bringing in outside help for.
Frequently asked questions
What learning rate should I use for LoRA fine-tuning?
Start at 1e-4 to 2e-4 for LoRA with rank 8-32 on a 7B-70B model. This is roughly 5-10x higher than typical full fine-tuning rates because LoRA updates a much smaller set of parameters that need a larger step to move meaningfully within a short training run.
How do I know if my labels are masked correctly?
Load one training batch and check that a meaningful fraction of the labels tensor equals -100 (the ignore index), corresponding to prompt and system tokens. If labels is identical to input_ids with no -100 values, masking was never applied and the model is being trained to predict the prompt itself.
Does target_modules matter that much for LoRA?
Yes. Missing even one of the four attention projections (q_proj, k_proj, v_proj, o_proj) meaningfully reduces what the adapter can express, and using a name that doesn't exist in that architecture means PEFT attaches to nothing, silently producing a run that looks like training but changes almost nothing.
Is a flat loss always a bug, or could the task just be hard?
A genuinely hard task still shows some downward trend, even if slow, within the first 100-200 steps. A loss that is bit-for-bit flat or oscillating in a tiny band with zero trend is a strong signal of a configuration bug (masking, targets, or frozen weights) rather than task difficulty.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolQLoRA vs Full Fine-Tuning Cost Calculator
See the GPU memory footprint, GPU-hour requirement, and dollar cost gap between QLoRA and full fine-tuning for the same model size and dataset.
Free ToolFine-Tuning Dataset Size Estimator
Estimate the number of training examples, total tokens, and human curation hours needed for a fine-tuning dataset based on task complexity and quality bar.
Free ToolFine-Tuning GPU-Hours Estimator
Estimate GPU-hours for LoRA, QLoRA, and full fine-tuning on the same model size and dataset, so you can compare method tradeoffs before choosing.
Related problems
Model forgets general knowledge after fine-tuning (catastrophic forgetting)
Catastrophic forgetting happens when fine-tuning overwrites the general capabilities the base model already had, and it is driven by a learning rate that is too high, too many epochs over a narrow dataset, or a rank that gives the adapter too much capacity relative to the data. Fix it by lowering the LoRA rank, adding a learning rate decay schedule, mixing in general-purpose replay data, or simply training fewer epochs.
Loss becomes NaN during fine-tuning
NaN loss during training is most often caused by fp16 numeric overflow in gradients or activations, which bf16 avoids because of its wider exponent range. Other common causes are a learning rate spike (especially right after warmup), a small number of corrupt or malformed training samples, and unsafe division or log operations in a custom loss function. Switch to bf16 first if your hardware supports it, then check for corrupt samples and unstable LR.
Tokenizer padding and truncation errors during training
Padding errors during training happen because many base models ship without a defined pad token at all, because the common workaround of setting pad_token equal to eos_token teaches the model that end-of-sequence and padding look identical (so it can learn to never emit a real stop signal), or because left-padding is used when right-padding was needed for the training collator, or vice versa. Add a distinct pad token when possible, and always right-pad for causal LM training.
Training dataset format errors during fine-tuning
Dataset format errors happen because the trainer expects a specific schema (either a messages list of role/content dicts, or a prompt/completion pair, or a single text field) and your JSONL doesn't match it, because samples are missing an EOS token so the model never learns to stop generating, or because a fixed max_length silently truncates long examples and cuts off labels partway through the intended response. Confirm your exact schema against what SFTTrainer or your data collator expects before training.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
GuideBuilding Fine-Tuning Datasets From Enterprise Data
Build fine-tuning datasets from enterprise data: instruction formats, deduplication methods, PII scrubbing, and quality filtering that actually works.
GuideFine-Tuning LLMs On-Prem with Enterprise Data
Fine-tune LLMs on-prem with enterprise data: LoRA vs full fine-tuning, dataset prep, GPU requirements, eval, and when RAG beats tuning altogether.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.