Why your fine-tuned model scores worse than the base model, and how to fix it
fine-tuned model underperforms the base model on the same evaluation benchmark
Also appears as
- custom model produces worse answers than the untuned checkpoint it started from
- benchmark accuracy dropped after fine-tuning instead of improving
Short answer
A fine-tuned model that scores worse than its own base model almost always means the evaluation is contaminated (test examples leaked into training) or unfair (a genuinely improved model getting compared under a broken harness), the inference prompt format doesn't match the exact format used during training, or the fine-tuning process optimized for surface style and tone rather than the underlying capability the benchmark actually measures. Check inference prompt formatting first, since it is the single most common cause.
Affects: Any fine-tuned model evaluated against its own base model, LoRA or full fine-tuning
Fastest path to finding the regression
- 1Confirm the exact prompt format, chat template, and special tokens used at inference time match what was used during training, character for character where possible.
- 2Check for eval contamination: verify none of your benchmark's test examples (or close paraphrases of them) exist in your fine-tuning training data.
- 3Compare a handful of individual failing examples side by side between base and fine-tuned outputs to see if the fine-tuned model is producing a different style/format that a rigid eval parser is scoring as wrong even though the content is reasonable.
- 4Re-run the exact same eval harness and prompt format against the unmodified base model to get a true apples-to-apples baseline, since informal recollection of the base model's score is often wrong.
- 5If the gap persists after ruling out format and contamination issues, review whether your training data emphasized tone, verbosity, or style over the specific capability the benchmark measures.
How to confirm this is your problem
- The same benchmark or eval harness gives the fine-tuned model a lower score than the base model checkpoint it was tuned from
- Individual failing examples show the fine-tuned model producing reasonable-looking answers that don't match the eval's expected exact format
- The gap is large on automated benchmarks but human spot-checking of the same outputs looks fine or even better
- Base model score used for comparison was taken from a different source (a leaderboard) rather than re-run with your own exact eval harness
- Fine-tuned model performs noticeably better on your specific domain examples but worse on the general benchmark used for comparison
Root causes and fixes
Prompt format mismatch between training and inference
If training data was formatted with one chat template or set of special tokens (system/user/assistant markers, specific delimiters) and the evaluation harness queries the model with a different format at inference time, the model receives an input distribution it was never actually trained to handle well, producing worse outputs even though the underlying fine-tuning itself may have gone fine. This is especially easy to introduce when the eval harness uses a generic default template rather than the model-specific one.
Fix: Extract the exact chat template and special token structure used during training (from your training script or tokenizer_config.json) and confirm the evaluation harness uses that identical format, not a generic or default one, when constructing prompts for the fine-tuned model.
python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('MODEL'); print(t.chat_template)"Evaluation contamination or an unfair baseline comparison
If the benchmark being used for comparison was never actually re-run against the true base model with the exact same harness (relying instead on a remembered or published number from a different evaluation setup), the comparison isn't measuring the same thing at all. Separately, if training data accidentally includes examples that overlap with or closely resemble the benchmark's test set, the reported baseline score for the base model can look artificially low or high relative to what a clean re-run would show.
Fix: Re-run the identical evaluation harness, prompt format, and scoring logic against both the true unmodified base model and the fine-tuned model in the same session, and separately audit your training data for any overlap with benchmark test examples.
Fine-tuning optimized for style and tone rather than the underlying capability
If training data consistently used a particular tone, verbosity level, or response structure (for example, always brief and direct) that differs from what the benchmark's scoring expects (for example, showing reasoning steps before an answer), the model can genuinely become better at your intended style while scoring worse on a benchmark that specifically rewards a different structural pattern, even though the underlying knowledge and capability may be equal or better.
Fix: Inspect several individually failing examples closely: if the fine-tuned model's actual content is correct but formatted differently than the benchmark expects (missing a required reasoning trace, wrong output structure), address the training data's format to match the target capability's expected structure rather than assuming the model regressed on knowledge.
Catastrophic forgetting affecting the specific skills the benchmark tests
If fine-tuning caused the model to overwrite general capabilities in favor of the narrow domain it was trained on (a distinct and separate problem from prompt formatting), and the benchmark used for comparison happens to test general capabilities rather than the fine-tuned domain, the score drop reflects genuine forgetting rather than a measurement artifact.
Fix: Determine whether the benchmark measures general capability or your specific fine-tuning domain; if general capability, address this as catastrophic forgetting (lower learning rate, fewer epochs, added replay data) rather than as a formatting or contamination issue.
Bug in the evaluation harness's answer extraction or scoring logic
Some evaluation harnesses extract the final answer from model output using regex or string matching tuned to the base model's typical output style (for example, expecting the answer immediately after a specific phrase). If fine-tuning shifted the model's typical phrasing even slightly, the extraction logic can fail to find a correctly-formatted answer that's actually present, scoring a correct response as wrong purely due to a parsing mismatch.
Fix: Manually inspect the raw model output versus what the harness extracted as the final answer for several scored-as-wrong examples, and adjust or make more robust the answer extraction logic if it's failing to parse otherwise-correct fine-tuned outputs.
Diagnostic commands
Compare chat templates used at train time versus eval time
python -c "print(train_tokenizer.chat_template == eval_tokenizer.chat_template)"
Should print True. If False, the prompt formatting between training and evaluation differs, which is the most common root cause of a fine-tuned model scoring worse than its base.
Re-run base model through the exact same eval harness
python eval_harness.py --model BASE_MODEL --eval-set benchmark.jsonl
Compare this freshly-computed base model score against whatever score you were using as your comparison point; if they differ meaningfully, your original baseline number was not measured the same way and the whole comparison needs to be redone fairly.
Manually review individual failing examples
python -c "print(fine_tuned_output, extracted_answer, expected_answer)"
If the raw fine_tuned_output looks reasonable but extracted_answer is empty or wrong, the failure is in the eval harness's parsing logic, not the model's actual capability.
Search training data for benchmark test set overlap
python -c "# check n-gram or embedding similarity between training examples and benchmark test examples"
Significant overlap indicates evaluation contamination, meaning the base-versus-fine-tuned comparison on this specific benchmark is not trustworthy and a different, clean eval set should be used instead.
Stopping it from happening again
- Always re-run the exact same evaluation harness against both the base and fine-tuned model in the same session rather than trusting a remembered or externally published base score.
- Store and version the exact chat template and prompt format used during training alongside the model checkpoint so evaluation code always matches.
- Audit training data for overlap with any benchmark you plan to evaluate against before training, not after seeing a surprising result.
- Evaluate on both a general capability benchmark and a domain-specific one so you can distinguish real forgetting from expected specialization tradeoffs.
- Manually spot-check a sample of scored outputs against raw model text, not just the automated score, before drawing conclusions about a regression.
When this becomes an architecture problem
If you've ruled out prompt formatting, contamination, and parsing bugs, and the fine-tuned model still genuinely underperforms on capabilities you need preserved, that points to a deeper training design issue (learning rate, data composition, or task scope) that benefits from a structured before/after evaluation methodology rather than one-off debugging. If this keeps happening across multiple fine-tuning projects, building a standardized, contamination-checked evaluation pipeline that runs automatically after every training job is worth investing in properly.
Frequently asked questions
Why would a fine-tuned model score worse than the base model it came from?
The most common reason is a mismatch between the prompt format used at inference time and the format the model was actually trained on, which makes the fine-tuned model receive an input structure it never learned to handle. Evaluation contamination, unfair baseline comparisons, and tuning for style over the benchmark's actual measured capability are the next most common causes.
How do I know if my evaluation is contaminated?
Check whether any of your fine-tuning training examples overlap with or closely paraphrase examples in your benchmark's test set, using text similarity or embedding comparison. If overlap exists, the evaluation is not measuring generalization fairly, and comparisons based on it are not reliable.
Could my fine-tuned model actually be better even though it scores lower?
Yes. If the benchmark's scoring logic expects a specific answer format or structure and your fine-tuning shifted the model's typical output style, a genuinely equal or better model can score lower simply because automated extraction fails to find the answer in the new format. Manual review of individual examples is the way to check this.
Should I always re-run the base model's score myself before comparing?
Yes. Published or remembered base model scores are frequently measured with different prompt formats, few-shot settings, or scoring logic than your own harness uses, making direct comparison misleading. Always re-run the identical harness against the true base model in the same session as the fine-tuned model for a fair comparison.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Fine-Tuning Dataset Size Estimator
Estimate the number of training examples, total tokens, and human curation hours needed for a fine-tuning dataset based on task complexity and quality bar.
Free ToolLoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolQLoRA vs Full Fine-Tuning Cost Calculator
See the GPU memory footprint, GPU-hour requirement, and dollar cost gap between QLoRA and full fine-tuning for the same model size and dataset.
Free ToolOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Related problems
Model forgets general knowledge after fine-tuning (catastrophic forgetting)
Catastrophic forgetting happens when fine-tuning overwrites the general capabilities the base model already had, and it is driven by a learning rate that is too high, too many epochs over a narrow dataset, or a rank that gives the adapter too much capacity relative to the data. Fix it by lowering the LoRA rank, adding a learning rate decay schedule, mixing in general-purpose replay data, or simply training fewer epochs.
LLM overfits on a small fine-tuning dataset
Overfitting on a small dataset shows up as training loss continuing to drop while evaluation loss rises after a few epochs, meaning the model is memorizing training examples rather than learning generalizable patterns. Fix it by holding out a genuine evaluation split, tracking eval loss every epoch, stopping early at the point where eval loss stops improving, and reducing epochs, rank, or learning rate if the crossover happens very early.
Training dataset format errors during fine-tuning
Dataset format errors happen because the trainer expects a specific schema (either a messages list of role/content dicts, or a prompt/completion pair, or a single text field) and your JSONL doesn't match it, because samples are missing an EOS token so the model never learns to stop generating, or because a fixed max_length silently truncates long examples and cuts off labels partway through the intended response. Confirm your exact schema against what SFTTrainer or your data collator expects before training.
Tokenizer mismatch causing garbage output or wrong special tokens
Garbage or repetitive output with weights that loaded without error almost always means the tokenizer does not exactly match the model, either because the vocabulary size or token IDs differ from what the model was trained on, or because special tokens like BOS/EOS/chat markers are mapped to the wrong IDs. The fix is to always load the tokenizer from the exact same repo and revision as the model weights, never mix files between repos, and verify the chat template and special token IDs match the model card.
GuideHow to Evaluate a Fine-Tuned Model Before Production
Evaluate a fine-tuned model before production: held-out eval sets, task-specific metrics, calibrated LLM-as-judge setups, and regression testing.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
GuideBuilding Fine-Tuning Datasets From Enterprise Data
Build fine-tuning datasets from enterprise data: instruction formats, deduplication methods, PII scrubbing, and quality filtering that actually works.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.