# Why your fine-tuned model scores worse than the base model, and how to fix it > Source: https://netray.co/troubleshoot/fine-tuned-model-worse-than-base > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer A fine-tuned model that scores worse than its own base model almost always means the evaluation is contaminated (test examples leaked into training) or unfair (a genuinely improved model getting compared under a broken harness), the inference prompt format doesn't match the exact format used during training, or the fine-tuning process optimized for surface style and tone rather than the underlying capability the benchmark actually measures. Check inference prompt formatting first, since it is the single most common cause. ## Error Title Fine-tuned model scores worse than the base model ## Error Signature fine-tuned model underperforms the base model on the same evaluation benchmark ## Alternate Signatures - custom model produces worse answers than the untuned checkpoint it started from - benchmark accuracy dropped after fine-tuning instead of improving ## Category fine-tuning ## Stack - pytorch - huggingface - transformers ## Affects Any fine-tuned model evaluated against its own base model, LoRA or full fine-tuning ## Quick Fix ### Fastest path to finding the regression **Steps:** Confirm the exact prompt format, chat template, and special tokens used at inference time match what was used during training, character for character where possible. Check for eval contamination: verify none of your benchmark's test examples (or close paraphrases of them) exist in your fine-tuning training data. Compare a handful of individual failing examples side by side between base and fine-tuned outputs to see if the fine-tuned model is producing a different style/format that a rigid eval parser is scoring as wrong even though the content is reasonable. Re-run the exact same eval harness and prompt format against the unmodified base model to get a true apples-to-apples baseline, since informal recollection of the base model's score is often wrong. If the gap persists after ruling out format and contamination issues, review whether your training data emphasized tone, verbosity, or style over the specific capability the benchmark measures. ## Symptoms - The same benchmark or eval harness gives the fine-tuned model a lower score than the base model checkpoint it was tuned from - Individual failing examples show the fine-tuned model producing reasonable-looking answers that don't match the eval's expected exact format - The gap is large on automated benchmarks but human spot-checking of the same outputs looks fine or even better - Base model score used for comparison was taken from a different source (a leaderboard) rather than re-run with your own exact eval harness - Fine-tuned model performs noticeably better on your specific domain examples but worse on the general benchmark used for comparison ## Causes ### Prompt format mismatch between training and inference **Likelihood:** most-common **Explanation:** If training data was formatted with one chat template or set of special tokens (system/user/assistant markers, specific delimiters) and the evaluation harness queries the model with a different format at inference time, the model receives an input distribution it was never actually trained to handle well, producing worse outputs even though the underlying fine-tuning itself may have gone fine. This is especially easy to introduce when the eval harness uses a generic default template rather than the model-specific one. **Fix:** Extract the exact chat template and special token structure used during training (from your training script or tokenizer_config.json) and confirm the evaluation harness uses that identical format, not a generic or default one, when constructing prompts for the fine-tuned model. **Commands:** - python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('MODEL'); print(t.chat_template)" ### Evaluation contamination or an unfair baseline comparison **Likelihood:** common **Explanation:** If the benchmark being used for comparison was never actually re-run against the true base model with the exact same harness (relying instead on a remembered or published number from a different evaluation setup), the comparison isn't measuring the same thing at all. Separately, if training data accidentally includes examples that overlap with or closely resemble the benchmark's test set, the reported baseline score for the base model can look artificially low or high relative to what a clean re-run would show. **Fix:** Re-run the identical evaluation harness, prompt format, and scoring logic against both the true unmodified base model and the fine-tuned model in the same session, and separately audit your training data for any overlap with benchmark test examples. ### Fine-tuning optimized for style and tone rather than the underlying capability **Likelihood:** common **Explanation:** If training data consistently used a particular tone, verbosity level, or response structure (for example, always brief and direct) that differs from what the benchmark's scoring expects (for example, showing reasoning steps before an answer), the model can genuinely become better at your intended style while scoring worse on a benchmark that specifically rewards a different structural pattern, even though the underlying knowledge and capability may be equal or better. **Fix:** Inspect several individually failing examples closely: if the fine-tuned model's actual content is correct but formatted differently than the benchmark expects (missing a required reasoning trace, wrong output structure), address the training data's format to match the target capability's expected structure rather than assuming the model regressed on knowledge. ### Catastrophic forgetting affecting the specific skills the benchmark tests **Likelihood:** occasional **Explanation:** If fine-tuning caused the model to overwrite general capabilities in favor of the narrow domain it was trained on (a distinct and separate problem from prompt formatting), and the benchmark used for comparison happens to test general capabilities rather than the fine-tuned domain, the score drop reflects genuine forgetting rather than a measurement artifact. **Fix:** Determine whether the benchmark measures general capability or your specific fine-tuning domain; if general capability, address this as catastrophic forgetting (lower learning rate, fewer epochs, added replay data) rather than as a formatting or contamination issue. ### Bug in the evaluation harness's answer extraction or scoring logic **Likelihood:** rare **Explanation:** Some evaluation harnesses extract the final answer from model output using regex or string matching tuned to the base model's typical output style (for example, expecting the answer immediately after a specific phrase). If fine-tuning shifted the model's typical phrasing even slightly, the extraction logic can fail to find a correctly-formatted answer that's actually present, scoring a correct response as wrong purely due to a parsing mismatch. **Fix:** Manually inspect the raw model output versus what the harness extracted as the final answer for several scored-as-wrong examples, and adjust or make more robust the answer extraction logic if it's failing to parse otherwise-correct fine-tuned outputs. ## Diagnostics ### Compare chat templates used at train time versus eval time **Command:** python -c "print(train_tokenizer.chat_template == eval_tokenizer.chat_template)" **Interpretation:** Should print True. If False, the prompt formatting between training and evaluation differs, which is the most common root cause of a fine-tuned model scoring worse than its base. ### Re-run base model through the exact same eval harness **Command:** python eval_harness.py --model BASE_MODEL --eval-set benchmark.jsonl **Interpretation:** Compare this freshly-computed base model score against whatever score you were using as your comparison point; if they differ meaningfully, your original baseline number was not measured the same way and the whole comparison needs to be redone fairly. ### Manually review individual failing examples **Command:** python -c "print(fine_tuned_output, extracted_answer, expected_answer)" **Interpretation:** If the raw fine_tuned_output looks reasonable but extracted_answer is empty or wrong, the failure is in the eval harness's parsing logic, not the model's actual capability. ### Search training data for benchmark test set overlap **Command:** python -c "# check n-gram or embedding similarity between training examples and benchmark test examples" **Interpretation:** Significant overlap indicates evaluation contamination, meaning the base-versus-fine-tuned comparison on this specific benchmark is not trustworthy and a different, clean eval set should be used instead. ## Prevention - Always re-run the exact same evaluation harness against both the base and fine-tuned model in the same session rather than trusting a remembered or externally published base score. - Store and version the exact chat template and prompt format used during training alongside the model checkpoint so evaluation code always matches. - Audit training data for overlap with any benchmark you plan to evaluate against before training, not after seeing a surprising result. - Evaluate on both a general capability benchmark and a domain-specific one so you can distinguish real forgetting from expected specialization tradeoffs. - Manually spot-check a sample of scored outputs against raw model text, not just the automated score, before drawing conclusions about a regression. ## When To Escalate If you've ruled out prompt formatting, contamination, and parsing bugs, and the fine-tuned model still genuinely underperforms on capabilities you need preserved, that points to a deeper training design issue (learning rate, data composition, or task scope) that benefits from a structured before/after evaluation methodology rather than one-off debugging. If this keeps happening across multiple fine-tuning projects, building a standardized, contamination-checked evaluation pipeline that runs automatically after every training job is worth investing in properly. ## Faq ### Why would a fine-tuned model score worse than the base model it came from? **Answer:** The most common reason is a mismatch between the prompt format used at inference time and the format the model was actually trained on, which makes the fine-tuned model receive an input structure it never learned to handle. Evaluation contamination, unfair baseline comparisons, and tuning for style over the benchmark's actual measured capability are the next most common causes. ### How do I know if my evaluation is contaminated? **Answer:** Check whether any of your fine-tuning training examples overlap with or closely paraphrase examples in your benchmark's test set, using text similarity or embedding comparison. If overlap exists, the evaluation is not measuring generalization fairly, and comparisons based on it are not reliable. ### Could my fine-tuned model actually be better even though it scores lower? **Answer:** Yes. If the benchmark's scoring logic expects a specific answer format or structure and your fine-tuning shifted the model's typical output style, a genuinely equal or better model can score lower simply because automated extraction fails to find the answer in the new format. Manual review of individual examples is the way to check this. ### Should I always re-run the base model's score myself before comparing? **Answer:** Yes. Published or remembered base model scores are frequently measured with different prompt formats, few-shot settings, or scoring logic than your own harness uses, making direct comparison misleading. Always re-run the identical harness against the true base model in the same session as the fine-tuned model for a fair comparison. ## Search terms fine-tuned model worse than base, fine-tuning regression benchmark, eval contamination fine-tuning, prompt format mismatch inference, chat template training vs eval, fine-tuned model lower score, benchmark score dropped after fine-tuning, fine-tuning overfits style not capability --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json