# Why the same prompt produces different outputs, and how to make responses consistent > Source: https://netray.co/troubleshoot/inconsistent-model-outputs > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Inconsistent outputs for an apparently identical prompt usually come from one of three sources: sampling is not actually deterministic (temperature above zero and no fixed seed), continuous batching introduces small floating-point nondeterminism because the exact batch composition changes token-level numerics run to run, or different replicas behind a load balancer are quietly running different quantization or even different model revisions. Full bit-for-bit determinism is hard to guarantee in batched GPU inference, but the practical fix is to control sampling explicitly and make sure all replicas are provably running the same model artifact. ## Error Title Same prompt produces different outputs across requests or replicas ## Error Signature identical prompt returns different responses each time it is sent ## Alternate Signatures - outputs differ between production replicas for the same input - responses are not reproducible even with the same prompt and settings ## Category performance ## Stack - vllm - pytorch - cuda ## Affects Any self-hosted LLM serving deployment, especially multi-replica production setups and anything using sampling-based generation ## Quick Fix ### Isolate the source of variation **Steps:** Set temperature to 0 (or use greedy/deterministic decoding) and fix a random seed if the API supports one, then retest for consistency. If outputs are still inconsistent at temperature 0, send the same prompt directly to a single replica repeatedly (bypassing the load balancer) to rule out cross-replica differences. Compare the model checkpoint hash, quantization method, and serving framework version across all replicas to confirm they are provably identical. If a single replica is still nondeterministic at temperature 0, check whether continuous batching's variable batch composition is the source, since exact floating-point results can shift slightly depending on what else is in the batch. Decide how much determinism you actually need: for most production use cases, tight variance rather than bit-for-bit identical output across all conditions is a more realistic and sufficient target. ## Symptoms - Sending the exact same prompt twice produces noticeably different text - Automated evaluation scores fluctuate run to run without any code or data changes - Some users report different answers to the same question depending on which server instance handled the request - Outputs are consistent in a single-replica dev environment but vary in the multi-replica production deployment ## Causes ### Sampling temperature above zero with no fixed seed **Likelihood:** most-common **Explanation:** Any temperature greater than zero introduces randomness into token selection by design, sampling from a probability distribution rather than always picking the highest-probability token, so different runs will legitimately produce different text even with everything else held constant. This is intentional model behavior, not a bug, and is the most common reason for perceived inconsistency. **Fix:** Set temperature to 0 for deterministic, repeatable output where consistency matters more than diversity, or set a fixed random seed per request if the framework supports it and diversity is still desired. **Commands:** - curl -s localhost:8000/v1/completions -d '{"prompt":"...","temperature":0}' ### Different replicas behind a load balancer are running different quantization or model revisions **Likelihood:** common **Explanation:** If replicas were deployed at different times, from different build pipelines, or with slightly different configuration, one replica might be serving an FP16 checkpoint while another serves an AWQ or GPTQ quantized version, or replicas may be pinned to different model revisions entirely, producing genuinely different outputs for the same prompt depending purely on which replica handled the request. **Fix:** Pin the exact model revision and quantization method in a single source of truth (a deployment manifest or container image tag), and add a startup check that verifies all replicas report the same checkpoint hash and quantization before joining the load balancer pool. ### Batching nondeterminism from continuous batching's varying batch composition **Likelihood:** common **Explanation:** GPU floating-point operations are not strictly associative, and the exact order and grouping of operations in a batched matrix multiplication can differ depending on what other sequences happen to be in the same batch at that moment, producing tiny numerical differences that can occasionally tip a token selection at the margin, even at temperature 0, especially near-tied logits. **Fix:** Accept that bit-for-bit determinism under continuous batching is difficult to guarantee even at temperature 0, and instead validate consistency using semantic similarity or task-level correctness rather than exact string matching, unless the framework offers a specific deterministic batching mode. ### No fixed seed passed when true reproducibility is required for compliance or debugging **Likelihood:** occasional **Explanation:** Even with temperature 0 largely removing sampling randomness, some code paths (top-k/top-p filtering edge cases, or explicit sampling for creative use cases) still depend on a random seed, and if the API call does not pass one explicitly, the framework may use a different seed each time, reintroducing variability that could otherwise be eliminated. **Fix:** Explicitly pass a fixed seed parameter on every request where reproducibility must be auditable, and document that this is required for any compliance-sensitive use case. ### Different serving framework versions across replicas with slightly different default sampling or numeric behavior **Likelihood:** rare **Explanation:** Version drift between replicas (one running an older vLLM build, another a newer one) can change default numeric behavior, kernel implementations, or even default sampling parameters, producing outputs that differ across replicas even when the request and model checkpoint are otherwise identical. **Fix:** Pin the exact serving framework version in the container image and treat any replica running a different version as a deployment bug to be fixed, not tolerated. ## Diagnostics ### Test the same prompt at temperature 0 against a single replica repeatedly **Command:** for i in 1 2 3; do curl -s localhost:8000/v1/completions -d '{"prompt":"test","temperature":0,"max_tokens":20}'; done **Interpretation:** If outputs still differ at temperature 0 on a single replica, the source is batching nondeterminism or a code path not respecting temperature 0, not cross-replica drift or sampling randomness in the usual sense. ### Compare model and quantization metadata across all replicas **Command:** curl -s localhost:8000/v1/models | python -m json.tool **Interpretation:** Any difference in reported model revision, dtype, or quantization method between replicas is a direct confirmation that cross-replica drift, not model behavior, explains the inconsistency users are seeing. ### Check serving framework version on each replica **Command:** pip show vllm | grep -i version **Interpretation:** Mismatched versions across replicas point to a deployment pipeline issue (inconsistent image tags or rollout) that should be fixed at the CI/CD level, not worked around at the application level. ## Prevention - Pin exact model revision, quantization method, and serving framework version in a single deployment manifest used by every replica - Add a startup health check that verifies checkpoint hash and config match across the replica pool before serving traffic - Set temperature and seed explicitly on every request rather than relying on framework defaults - Evaluate output consistency with semantic or task-level metrics rather than exact string equality where batching nondeterminism is expected ## When To Escalate If strict bit-for-bit reproducibility is a hard compliance or audit requirement (e.g. regulated industries needing to reproduce exact historical outputs), that goes beyond configuration and into architecture: it typically requires disabling continuous batching for the affected workload, running single-sequence deterministic inference, or maintaining a separate reproducibility-mode deployment path, all of which trade throughput for guaranteed determinism. ## Faq ### Can I get perfectly deterministic output from a GPU-batched LLM server? **Answer:** Close to it at temperature 0 with a fixed seed and a single pinned model/quantization, but not perfectly guaranteed, because continuous batching's floating-point nondeterminism can still occasionally shift results at the margin. True bit-for-bit determinism generally requires giving up batching efficiency, which is a real tradeoff to make consciously rather than assume away. ### Does quantization make outputs less consistent? **Answer:** Quantization itself does not add run-to-run randomness, a correctly quantized model at temperature 0 is just as internally deterministic as full precision. The inconsistency problem arises when different replicas are quantized differently or run different quantization methods, producing systematically different (not randomly different) outputs between replicas. ### Why do outputs vary even though our load balancer routes to identical replica images? **Answer:** If the images are provably identical and temperature is 0, remaining variation is most likely continuous batching's floating-point nondeterminism, since exact numeric results can depend on what else is in the batch at inference time. This is a known limitation of high-throughput batched serving, not a deployment misconfiguration. ## Search terms inconsistent llm outputs, same prompt different response, llm output not reproducible, vllm nondeterministic output, temperature 0 still different output, batching nondeterminism llm --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json