# Why quality dropped after quantizing your model, and how to recover it > Source: https://netray.co/troubleshoot/quality-degraded-after-quantization > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Quality degradation after quantization usually comes from choosing too aggressive a quantization level for the model size and task, quantizing layers that are unusually sensitive to precision loss (often attention output projections and the final layers), or from trusting perplexity as the only quality signal when perplexity can look nearly unchanged while task-specific accuracy drops meaningfully. FP8 is close to lossless for most models and tasks, while INT4 methods carry real risk that must be validated with a task-specific eval set before shipping, not assumed safe from a perplexity number alone. ## Error Title Model output quality dropped noticeably after quantization ## Error Signature quantized model gives noticeably worse answers than the full precision version ## Alternate Signatures - int4 model performs much worse than fp16 on our tasks - quantization made the model less accurate even though perplexity looked fine ## Category performance ## Stack - pytorch - huggingface - cuda ## Affects Any quantized open-weight model (AWQ, GPTQ, GGUF, INT4, FP8), most pronounced on smaller models and reasoning-heavy or precise-formatting tasks ## Quick Fix ### Recover quality without giving up quantization **Steps:** Confirm which quantization method and bit width is in use (FP8, AWQ/GPTQ INT4, or GGUF at a given quant level); FP8 is the safest starting point for quality-sensitive workloads. Build or use a task-specific eval set that reflects your real production task, not a generic benchmark or raw perplexity, and score both the full-precision and quantized model on it. If INT4 is causing a measurable quality drop, try a higher-precision fallback for the most sensitive layers if your quantization tool supports mixed-precision quantization. If the calibration dataset used for quantization (for AWQ/GPTQ) is not representative of your actual traffic distribution, requantize with a more representative calibration set. If quality still does not meet the bar at INT4, move to FP8 or a smaller/larger unquantized model rather than continuing to tune INT4 settings indefinitely. ## Symptoms - Quantized model gives wrong or lower-quality answers on tasks the full-precision model handled correctly - Perplexity or generic benchmark scores look nearly identical between quantized and full-precision versions, but real usage feels worse - Quality loss is more noticeable on structured output, reasoning, or precise numeric tasks than on open-ended chat - Degradation is inconsistent, some prompts are fine, others are clearly worse ## Causes ### Quantization method too aggressive for the model size and task (INT4 without validation) **Likelihood:** most-common **Explanation:** Lower bit widths (INT4) compress weight representation more aggressively than FP8, and the resulting rounding error compounds differently depending on model size and task; smaller models and tasks requiring precise multi-step reasoning or exact formatting are generally more sensitive to this compounding error than larger models or open-ended generation tasks. **Fix:** For quality-sensitive or precision-sensitive tasks, prefer FP8 as a first quantization step, and only move to INT4 methods (AWQ, GPTQ) after validating quality on a task-specific eval set, not assuming it is safe by default. ### Certain layers are unusually sensitive to quantization but were quantized uniformly **Likelihood:** common **Explanation:** Not all layers contribute equally to output quality under quantization; attention output projections, embedding/unembedding layers, and certain outlier-heavy activation layers are commonly more sensitive to precision loss than typical feed-forward weights, so uniform quantization across all layers can degrade quality more than a mixed-precision approach that protects the most sensitive layers. **Fix:** Use a quantization method or tool that supports mixed-precision or per-layer sensitivity analysis, keeping known-sensitive layers at higher precision while quantizing the rest more aggressively. ### Evaluating with perplexity alone instead of a task-specific eval set **Likelihood:** common **Explanation:** Perplexity measures how well the model predicts held-out text in aggregate, which can remain nearly unchanged after quantization even when the model's accuracy on a specific downstream task (structured extraction, multi-step reasoning, exact tool-call formatting) degrades meaningfully, because perplexity averages over many tokens and can mask localized but consequential errors. **Fix:** Build a small, representative task-specific eval set (real prompts and expected outputs from your actual use case) and score both full-precision and quantized models on it directly, treating perplexity as at most a rough sanity check, not the deciding metric. ### Calibration dataset for AWQ/GPTQ not representative of production traffic **Likelihood:** occasional **Explanation:** AWQ and GPTQ use a calibration dataset to determine which weights and activations matter most and should be preserved with higher fidelity; if that calibration set does not resemble your actual traffic (different domain, different formatting, different language), the quantization can end up optimized for the wrong distribution, degrading quality specifically on your real use case while looking fine on generic benchmarks. **Fix:** Requantize using a calibration dataset built from representative samples of your actual production prompts and expected outputs rather than a generic public calibration set. ### Quantization applied without re-validating after a model or framework upgrade **Likelihood:** rare **Explanation:** A quantization recipe validated against one model checkpoint or framework version can behave differently after a model update or a change in the quantization library's implementation details, silently reintroducing quality loss that was previously fixed or absent. **Fix:** Re-run the task-specific eval any time the base model checkpoint, quantization library version, or calibration process changes, rather than assuming a previously validated recipe still holds. ## Diagnostics ### Run a task-specific eval set against both full-precision and quantized checkpoints **Command:** python -c "print('score both checkpoints on your real eval set and diff the results')" **Interpretation:** A meaningful gap in task-specific accuracy despite similar perplexity confirms the quality issue is real and specific to your task, not a generic degradation visible in standard benchmarks. ### Compare per-example outputs between full-precision and quantized models **Command:** diff <(python fp16_infer.py) <(python quantized_infer.py) **Interpretation:** Categorizing which specific prompt types show degradation (structured output, long reasoning chains, numeric precision) points directly at which layers or quantization settings need adjustment. ### Check which quantization method and bit width is actually deployed **Command:** curl -s localhost:8000/v1/models | python -m json.tool **Interpretation:** Confirm the deployed method matches what was validated; a mismatch between the tested and deployed quantization config is a common and easily overlooked cause of unexpected quality drops. ## Prevention - Always validate quantization against a task-specific eval set before shipping, never perplexity alone - Build the calibration dataset for AWQ/GPTQ from representative production traffic - Start from FP8 for quality-sensitive workloads and only move to INT4 after explicit validation - Re-validate quality any time the base model, quantization library, or calibration set changes ## When To Escalate If a task-specific eval confirms that no quantization method, including FP8, meets the required quality bar for the workload, the real fix is not further quantization tuning but reconsidering the model choice (a different base model that quantizes more gracefully for your task) or accepting full precision at higher cost/memory for that specific workload, which is a model-selection and capacity tradeoff rather than a quantization configuration issue. ## Faq ### Is FP8 always safe for quality? **Answer:** FP8 is close to lossless for most models and tasks and is a reasonable default starting point, but 'close to lossless' is not the same as 'guaranteed lossless' for every task. It should still be validated against a task-specific eval set for anything quality-critical, even though the risk is meaningfully lower than with INT4 methods. ### Why did perplexity look fine but our actual task got worse? **Answer:** Perplexity averages prediction quality across many tokens and can stay nearly flat even when quantization introduces errors concentrated in a specific type of output, like exact numeric values, JSON formatting, or multi-step reasoning chains, that a task-specific eval would catch but an aggregate language-modeling metric would not. ### Should we quantize every layer the same way? **Answer:** Not necessarily. Some layers, particularly attention output projections and layers handling high-variance activations, are more sensitive to precision loss than typical feed-forward weights. Mixed-precision quantization that protects the most sensitive layers while compressing the rest more aggressively often preserves more quality than uniform quantization at the same average bit width. ## Search terms quality degraded after quantization, int4 quantization accuracy loss, awq gptq quality comparison, quantization worse output quality, fp8 vs int4 quality, perplexity vs task accuracy quantization --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json