Why fine-tuning makes your model forget general skills, and how to prevent it
model answers basic general-knowledge questions incorrectly after fine-tuning, though it did fine before
Also appears as
- fine-tuned model only performs well on the exact training examples and fails everywhere else
- model has lost instruction-following ability outside the fine-tuning domain
Short answer
Catastrophic forgetting happens when fine-tuning overwrites the general capabilities the base model already had, and it is driven by a learning rate that is too high, too many epochs over a narrow dataset, or a rank that gives the adapter too much capacity relative to the data. Fix it by lowering the LoRA rank, adding a learning rate decay schedule, mixing in general-purpose replay data, or simply training fewer epochs.
Affects: LoRA and full fine-tuning on any base model, most pronounced with narrow or small domain-specific datasets
Fastest path to keeping general ability
- 1Cut epochs first: try 1-2 epochs instead of 3-5 and re-evaluate on both your domain task and a general benchmark.
- 2Lower the LoRA rank (try 8 or 16 instead of 64 or 128) so the adapter has less capacity to overwrite broad knowledge.
- 3Add a cosine or linear learning rate decay with warmup instead of a constant rate, so late-stage updates are smaller.
- 4Mix 5-15% general-purpose instruction data (replay data) into the fine-tuning set alongside your domain-specific examples.
- 5Evaluate on a held-out general capability set (not just your domain eval) at each epoch checkpoint and stop at the best tradeoff.
How to confirm this is your problem
- Model answers domain-specific questions well but fails simple general questions it previously answered correctly
- Instruction-following degrades outside the exact style or format seen in fine-tuning data
- Model repeats phrasing or structure from training examples even for unrelated prompts
- Performance on a general benchmark (MMLU-style or similar) drops noticeably versus the base model
- Behavior becomes rigid or repetitive compared to the base model's more varied responses
Root causes and fixes
Learning rate too high for the amount and diversity of data
A high learning rate makes large weight updates per step. On a small or narrow dataset, those large updates push the model's parameters far from their pretrained values in directions that only help the narrow task, effectively overwriting the broader representations that gave the base model its general skill.
Fix: Drop the learning rate by 2-5x from what you'd use on a large diverse dataset, and add warmup plus decay so the largest updates happen briefly at the start, not throughout the whole run.
Too many epochs over a narrow dataset
Each additional epoch over the same narrow examples reinforces the same narrow patterns more strongly. Past a certain point (often 2-3 epochs for small instruction sets), further epochs stop improving domain performance and instead specialize the model's weights so tightly to the training distribution that unrelated capabilities degrade.
Fix: Train for 1-3 epochs and evaluate after each one on both domain and general tasks; pick the checkpoint that best balances both rather than assuming more epochs is always better.
LoRA rank too high relative to dataset size
A higher rank gives the adapter more free parameters to fit the training data. On a small dataset this extra capacity is used to memorize training examples rather than learn generalizable patterns, and the larger weight delta it applies at inference time interferes more with the base model's existing representations.
Fix: Reduce rank to 8-16 for datasets under a few thousand examples, and only increase rank if you have tens of thousands of diverse examples that justify the added capacity.
Training data lacks diversity, covering only one narrow style or task
If every training example uses the same format, tone, and narrow topic, the model has no counterexamples pulling it back toward general behavior, so gradient updates consistently push in one direction. This directional pressure compounds across steps and epochs, amplifying drift away from the base model's broader distribution.
Fix: Add replay data: mix in 5-15% general-purpose instruction-following examples (unrelated to your domain) so gradients periodically reinforce broad behavior alongside the domain-specific signal.
Full fine-tuning without any regularization or replay strategy
Full fine-tuning updates every parameter in the model, which gives it the most capacity to both learn the new task and forget old ones. Without techniques like elastic weight consolidation, replay buffers, or a much lower learning rate than LoRA typically needs, full fine-tuning on a narrow dataset forgets faster and more thoroughly than a rank-limited adapter would.
Fix: Prefer LoRA or QLoRA for narrow domain adaptation tasks; if full fine-tuning is required, use a very low learning rate (1e-6 to 5e-6), fewer epochs, and a larger, more diverse dataset.
Diagnostic commands
Compare general benchmark scores before and after fine-tuning
lm-eval --model hf --model_args pretrained=BASE_MODEL --tasks mmlu,gsm8k
Run the same eval suite on the base model and the fine-tuned model. A drop of more than a few points on tasks unrelated to your fine-tuning domain confirms forgetting rather than expected specialization tradeoffs.
Check checkpoint-by-checkpoint domain vs general score
python eval_script.py --checkpoint checkpoint-epoch-1 --checkpoint checkpoint-epoch-3
Plotting domain accuracy against general accuracy across saved epoch checkpoints usually shows domain performance rising while general performance falls; pick the checkpoint before the crossover gets severe.
Inspect effective learning rate over training
python -c "print(trainer.state.log_history[-5:])"
Look at the logged learning rate values across the run. A rate that stays at its peak value for most of training (no decay) is a common contributor to overwriting general knowledge late in the run.
Stopping it from happening again
- Always hold out a general-capability eval set, not just a domain-specific one, and check both after every fine-tuning run.
- Default to LoRA with rank 8-16 for narrow domain tasks and only raise rank when the data volume and diversity justify it.
- Build a small (5-15%) replay dataset of general instruction examples that you mix into every fine-tuning job by default.
- Save and evaluate a checkpoint after every epoch instead of only the final one, so you can roll back if forgetting appears late.
- Treat epoch count as a tuned hyperparameter, not a fixed default; most narrow instruction-tuning tasks need 1-3 epochs, not 5-10.
When this becomes an architecture problem
If your use case genuinely requires both deep domain specialization and preserved general ability at production quality, that tension usually needs a deliberate data strategy (curated replay ratios, staged training, or multiple adapters swapped at inference) rather than a single hyperparameter tweak, and is worth planning with someone who has built that tradeoff before. If forgetting reappears across every configuration you try, the underlying dataset composition itself needs redesigning.
Frequently asked questions
How many epochs should I fine-tune for to avoid forgetting?
Most narrow instruction-tuning or domain-adaptation datasets should use 1-3 epochs. Evaluate general capability after each epoch; if you see meaningful degradation on unrelated tasks by epoch 2 or 3, stop earlier or reduce the learning rate rather than continuing.
Does LoRA prevent catastrophic forgetting compared to full fine-tuning?
LoRA reduces forgetting because it constrains updates to a low-rank subspace rather than modifying every weight, but it does not eliminate the risk. A high rank, high learning rate, or many epochs on narrow data can still cause significant forgetting even with LoRA.
What is replay data and how much should I use?
Replay data is general-purpose instruction examples unrelated to your fine-tuning domain, mixed into the training set to keep gradient updates from pulling entirely in one narrow direction. A ratio of 5-15% of total training examples is a reasonable starting point for most domain adaptation jobs.
Can I fix catastrophic forgetting after training is already done?
Not by editing the trained weights directly. The practical fix is to retrain with a lower learning rate, fewer epochs, a lower rank, or added replay data, then compare the new checkpoint's general and domain scores against the original to confirm the tradeoff improved.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolFine-Tuning Dataset Size Estimator
Estimate the number of training examples, total tokens, and human curation hours needed for a fine-tuning dataset based on task complexity and quality bar.
Free ToolQLoRA vs Full Fine-Tuning Cost Calculator
See the GPU memory footprint, GPU-hour requirement, and dollar cost gap between QLoRA and full fine-tuning for the same model size and dataset.
Free ToolOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Related problems
Training loss not decreasing during fine-tuning
Training loss that stays flat is most often caused by a LoRA adapter that targets the wrong modules (missing q_proj/k_proj/v_proj/o_proj), a learning rate that is too low for LoRA or too high and bouncing, or labels that were never masked so the model is trying to learn the prompt tokens as if they were random noise. Check target_modules first, then the label mask, then the learning rate.
LLM overfits on a small fine-tuning dataset
Overfitting on a small dataset shows up as training loss continuing to drop while evaluation loss rises after a few epochs, meaning the model is memorizing training examples rather than learning generalizable patterns. Fix it by holding out a genuine evaluation split, tracking eval loss every epoch, stopping early at the point where eval loss stops improving, and reducing epochs, rank, or learning rate if the crossover happens very early.
Fine-tuned model scores worse than the base model
A fine-tuned model that scores worse than its own base model almost always means the evaluation is contaminated (test examples leaked into training) or unfair (a genuinely improved model getting compared under a broken harness), the inference prompt format doesn't match the exact format used during training, or the fine-tuning process optimized for surface style and tone rather than the underlying capability the benchmark actually measures. Check inference prompt formatting first, since it is the single most common cause.
Training dataset format errors during fine-tuning
Dataset format errors happen because the trainer expects a specific schema (either a messages list of role/content dicts, or a prompt/completion pair, or a single text field) and your JSONL doesn't match it, because samples are missing an EOS token so the model never learns to stop generating, or because a fixed max_length silently truncates long examples and cuts off labels partway through the intended response. Confirm your exact schema against what SFTTrainer or your data collator expects before training.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
GuideHow to Evaluate a Fine-Tuned Model Before Production
Evaluate a fine-tuned model before production: held-out eval sets, task-specific metrics, calibrated LLM-as-judge setups, and regression testing.
GuideBuilding Fine-Tuning Datasets From Enterprise Data
Build fine-tuning datasets from enterprise data: instruction formats, deduplication methods, PII scrubbing, and quality filtering that actually works.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.