Why quality dropped after quantizing your model, and how to recover it
quantized model gives noticeably worse answers than the full precision version
Also appears as
- int4 model performs much worse than fp16 on our tasks
- quantization made the model less accurate even though perplexity looked fine
Short answer
Quality degradation after quantization usually comes from choosing too aggressive a quantization level for the model size and task, quantizing layers that are unusually sensitive to precision loss (often attention output projections and the final layers), or from trusting perplexity as the only quality signal when perplexity can look nearly unchanged while task-specific accuracy drops meaningfully. FP8 is close to lossless for most models and tasks, while INT4 methods carry real risk that must be validated with a task-specific eval set before shipping, not assumed safe from a perplexity number alone.
Affects: Any quantized open-weight model (AWQ, GPTQ, GGUF, INT4, FP8), most pronounced on smaller models and reasoning-heavy or precise-formatting tasks
Recover quality without giving up quantization
- 1Confirm which quantization method and bit width is in use (FP8, AWQ/GPTQ INT4, or GGUF at a given quant level); FP8 is the safest starting point for quality-sensitive workloads.
- 2Build or use a task-specific eval set that reflects your real production task, not a generic benchmark or raw perplexity, and score both the full-precision and quantized model on it.
- 3If INT4 is causing a measurable quality drop, try a higher-precision fallback for the most sensitive layers if your quantization tool supports mixed-precision quantization.
- 4If the calibration dataset used for quantization (for AWQ/GPTQ) is not representative of your actual traffic distribution, requantize with a more representative calibration set.
- 5If quality still does not meet the bar at INT4, move to FP8 or a smaller/larger unquantized model rather than continuing to tune INT4 settings indefinitely.
How to confirm this is your problem
- Quantized model gives wrong or lower-quality answers on tasks the full-precision model handled correctly
- Perplexity or generic benchmark scores look nearly identical between quantized and full-precision versions, but real usage feels worse
- Quality loss is more noticeable on structured output, reasoning, or precise numeric tasks than on open-ended chat
- Degradation is inconsistent, some prompts are fine, others are clearly worse
Root causes and fixes
Quantization method too aggressive for the model size and task (INT4 without validation)
Lower bit widths (INT4) compress weight representation more aggressively than FP8, and the resulting rounding error compounds differently depending on model size and task; smaller models and tasks requiring precise multi-step reasoning or exact formatting are generally more sensitive to this compounding error than larger models or open-ended generation tasks.
Fix: For quality-sensitive or precision-sensitive tasks, prefer FP8 as a first quantization step, and only move to INT4 methods (AWQ, GPTQ) after validating quality on a task-specific eval set, not assuming it is safe by default.
Certain layers are unusually sensitive to quantization but were quantized uniformly
Not all layers contribute equally to output quality under quantization; attention output projections, embedding/unembedding layers, and certain outlier-heavy activation layers are commonly more sensitive to precision loss than typical feed-forward weights, so uniform quantization across all layers can degrade quality more than a mixed-precision approach that protects the most sensitive layers.
Fix: Use a quantization method or tool that supports mixed-precision or per-layer sensitivity analysis, keeping known-sensitive layers at higher precision while quantizing the rest more aggressively.
Evaluating with perplexity alone instead of a task-specific eval set
Perplexity measures how well the model predicts held-out text in aggregate, which can remain nearly unchanged after quantization even when the model's accuracy on a specific downstream task (structured extraction, multi-step reasoning, exact tool-call formatting) degrades meaningfully, because perplexity averages over many tokens and can mask localized but consequential errors.
Fix: Build a small, representative task-specific eval set (real prompts and expected outputs from your actual use case) and score both full-precision and quantized models on it directly, treating perplexity as at most a rough sanity check, not the deciding metric.
Calibration dataset for AWQ/GPTQ not representative of production traffic
AWQ and GPTQ use a calibration dataset to determine which weights and activations matter most and should be preserved with higher fidelity; if that calibration set does not resemble your actual traffic (different domain, different formatting, different language), the quantization can end up optimized for the wrong distribution, degrading quality specifically on your real use case while looking fine on generic benchmarks.
Fix: Requantize using a calibration dataset built from representative samples of your actual production prompts and expected outputs rather than a generic public calibration set.
Quantization applied without re-validating after a model or framework upgrade
A quantization recipe validated against one model checkpoint or framework version can behave differently after a model update or a change in the quantization library's implementation details, silently reintroducing quality loss that was previously fixed or absent.
Fix: Re-run the task-specific eval any time the base model checkpoint, quantization library version, or calibration process changes, rather than assuming a previously validated recipe still holds.
Diagnostic commands
Run a task-specific eval set against both full-precision and quantized checkpoints
python -c "print('score both checkpoints on your real eval set and diff the results')"A meaningful gap in task-specific accuracy despite similar perplexity confirms the quality issue is real and specific to your task, not a generic degradation visible in standard benchmarks.
Compare per-example outputs between full-precision and quantized models
diff <(python fp16_infer.py) <(python quantized_infer.py)
Categorizing which specific prompt types show degradation (structured output, long reasoning chains, numeric precision) points directly at which layers or quantization settings need adjustment.
Check which quantization method and bit width is actually deployed
curl -s localhost:8000/v1/models | python -m json.tool
Confirm the deployed method matches what was validated; a mismatch between the tested and deployed quantization config is a common and easily overlooked cause of unexpected quality drops.
Stopping it from happening again
- Always validate quantization against a task-specific eval set before shipping, never perplexity alone
- Build the calibration dataset for AWQ/GPTQ from representative production traffic
- Start from FP8 for quality-sensitive workloads and only move to INT4 after explicit validation
- Re-validate quality any time the base model, quantization library, or calibration set changes
When this becomes an architecture problem
If a task-specific eval confirms that no quantization method, including FP8, meets the required quality bar for the workload, the real fix is not further quantization tuning but reconsidering the model choice (a different base model that quantizes more gracefully for your task) or accepting full precision at higher cost/memory for that specific workload, which is a model-selection and capacity tradeoff rather than a quantization configuration issue.
Frequently asked questions
Is FP8 always safe for quality?
FP8 is close to lossless for most models and tasks and is a reasonable default starting point, but 'close to lossless' is not the same as 'guaranteed lossless' for every task. It should still be validated against a task-specific eval set for anything quality-critical, even though the risk is meaningfully lower than with INT4 methods.
Why did perplexity look fine but our actual task got worse?
Perplexity averages prediction quality across many tokens and can stay nearly flat even when quantization introduces errors concentrated in a specific type of output, like exact numeric values, JSON formatting, or multi-step reasoning chains, that a task-specific eval would catch but an aggregate language-modeling metric would not.
Should we quantize every layer the same way?
Not necessarily. Some layers, particularly attention output projections and layers handling high-variance activations, are more sensitive to precision loss than typical feed-forward weights. Mixed-precision quantization that protects the most sensitive layers while compressing the rest more aggressively often preserves more quality than uniform quantization at the same average bit width.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Free ToolOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Related problems
GGUF quantization damages model output quality
GGUF quantization below roughly 4 bits per weight (Q2_K, Q3_K_S) trades accuracy aggressively for size and speed, and on smaller models or reasoning-heavy tasks this shows up as incoherent, repetitive, or factually unreliable output. Q4_K_M and Q5_K_M are the widely used sweet spots that keep most of the quality of the full-precision model while still cutting memory roughly in half or more, and Q2/Q3 should be reserved for cases where fitting in VRAM matters more than output quality.
Same prompt produces different outputs across requests or replicas
Inconsistent outputs for an apparently identical prompt usually come from one of three sources: sampling is not actually deterministic (temperature above zero and no fixed seed), continuous batching introduces small floating-point nondeterminism because the exact batch composition changes token-level numerics run to run, or different replicas behind a load balancer are quietly running different quantization or even different model revisions. Full bit-for-bit determinism is hard to guarantee in batched GPU inference, but the practical fix is to control sampling explicitly and make sure all replicas are provably running the same model artifact.
Cost per token for self-hosted LLM inference is higher than expected
Cost per token is dominated by GPU utilization far more than by hardware choice: an underutilized GPU serving one request at a time can cost more per token than a well-tuned smaller GPU serving at full continuous-batching concurrency. Before concluding self-hosting is not worth it, check utilization, whether the model is right-sized for the task, whether quantization and prefix/response caching are in use, and whether the comparison to an API is even apples-to-apples once amortization is accounted for.
Fine-tuned model scores worse than the base model
A fine-tuned model that scores worse than its own base model almost always means the evaluation is contaminated (test examples leaked into training) or unfair (a genuinely improved model getting compared under a broken harness), the inference prompt format doesn't match the exact format used during training, or the fine-tuning process optimized for surface style and tone rather than the underlying capability the benchmark actually measures. Check inference prompt formatting first, since it is the single most common cause.
GuideLLM Quantization: AWQ vs GPTQ vs FP8 vs GGUF
AWQ, GPTQ, FP8, and GGUF compared for production LLM serving: memory savings, throughput impact, quality loss, and which format fits which deployment.
GuideHow to Evaluate a Fine-Tuned Model Before Production
Evaluate a fine-tuned model before production: held-out eval sets, task-specific metrics, calibrated LLM-as-judge setups, and regression testing.
GuideThe 2026 Open-Weight LLM Landscape: A Practical Map
A practical map of the 2026 open-weight LLM landscape: model families, license terms, and which model fits your VRAM budget and use case.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.