# AI Inference Latency Calculator: Predict Self-Hosted LLM Response Times > Source: https://netray.co/tools/ai-inference-latency-calculator > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization. ## Category on-prem-ai ## Tool **Kind:** calculator **Inputs:** #### Model size **Key:** model_params_b **Type:** number **Default:** 70 **Min:** 1 **Max:** 700 **Step:** 1 **Unit:** B params **Help:** Total parameter count in billions. For mixture-of-experts models use active parameters per token. #### Weight precision **Key:** quantization_bytes **Type:** select **Default:** 2 **Unit:** bytes/param **Help:** Lower precision shrinks the memory read per token, which is the main driver of decode speed. **Options:** ##### FP16 / BF16 (2 bytes per parameter) **Value:** 2 ##### FP8 (1 byte per parameter) **Value:** 1 ##### INT4 / AWQ (0.5 bytes per parameter) **Value:** 0.5 #### GPU memory bandwidth **Key:** gpu_memory_bandwidth_gbs **Type:** select **Default:** 3350 **Unit:** GB/s **Help:** Decode is memory-bandwidth bound, so this matters far more than raw FLOPS for single-stream latency. **Options:** ##### L40S class (864 GB/s) **Value:** 864 ##### A100 80GB (2,039 GB/s) **Value:** 2039 ##### H100 SXM (3,350 GB/s) **Value:** 3350 ##### H200 SXM (4,800 GB/s) **Value:** 4800 #### GPUs serving the model **Key:** gpu_count **Type:** number **Default:** 4 **Min:** 1 **Max:** 64 **Step:** 1 **Unit:** GPUs **Help:** Tensor-parallel degree for one replica. Scaling is sub-linear because of interconnect overhead. #### Achieved bandwidth efficiency **Key:** efficiency_pct **Type:** slider **Default:** 65 **Min:** 20 **Max:** 90 **Step:** 5 **Unit:** % **Help:** Real serving stacks reach 55-75% of theoretical bandwidth after kernel and interconnect overhead. #### Input tokens per request **Key:** input_tokens **Type:** number **Default:** 2000 **Min:** 50 **Max:** 200000 **Step:** 50 **Unit:** tokens **Help:** Full prompt length including retrieved context. Drives prefill time and time to first token. #### Output tokens per response **Key:** output_tokens **Type:** number **Default:** 500 **Min:** 10 **Max:** 32000 **Step:** 10 **Unit:** tokens **Help:** Generated response length. Decode time scales almost linearly with this number. **Outputs:** #### Working memory footprint **Key:** model_memory_gb **Expr:** model_params_b * quantization_bytes * 1.2 **Format:** number **Description:** Weights plus roughly 20% for KV cache, activations, and runtime overhead, in gigabytes. #### Decode throughput **Key:** tokens_per_sec **Expr:** gpu_memory_bandwidth_gbs * gpu_count * efficiency_pct / 100 / max(model_memory_gb, 1) **Format:** number **Description:** Tokens generated per second for a single stream. Batched serving raises aggregate throughput. **Primary:** true #### Time to first token (ms) **Key:** ttft_ms **Expr:** input_tokens / max(tokens_per_sec * 40, 1) * 1000 **Format:** number **Description:** Prefill time, which processes prompt tokens in parallel far faster than decode. #### Generation time (seconds) **Key:** generation_sec **Expr:** output_tokens / max(tokens_per_sec, 1) **Format:** number **Description:** Time spent producing the response after the first token appears. #### Total response time (seconds) **Key:** total_response_sec **Expr:** ttft_ms / 1000 + generation_sec **Format:** number **Description:** End-to-end wall clock a user experiences for one complete answer. **Primary:** true **Disclaimer:** Rough planning estimates only. Real latency depends on serving engine, batch size, speculative decoding, and interconnect topology. Benchmark your actual stack before committing to an SLA. ## Content **Intro:** This free AI inference latency calculator estimates how fast a self-hosted language model will respond on your hardware, and it is built for platform engineers and IT leaders sizing on-prem GPU clusters. Enter model size, weight precision, GPU class and count, achieved bandwidth efficiency, and prompt and response lengths, and the tool returns memory footprint, decode throughput, time to first token, and total response time. Decode is bound by memory bandwidth, not compute, which is why a smaller quantized model on modest GPUs often feels dramatically faster than a large model on expensive ones. **Sections:** #### How the latency model works **Body:** Generating each token requires reading the model weights from GPU memory, so single-stream decode speed is approximately aggregate memory bandwidth divided by the bytes that must be read per token. A 70B model at FP16 occupies about 140 GB of weights, and adding roughly 20% for KV cache and activations gives a 168 GB working footprint. Four H100 SXM GPUs provide 13,400 GB/s of theoretical bandwidth; at a realistic 65% efficiency that is 8,710 GB/s, or about 52 tokens per second. Prefill processes prompt tokens in parallel and runs far faster per token, modeled here at roughly forty times decode speed, so a 2,000-token prompt yields about 960 milliseconds to first token. Adding 500 output tokens gives a total response near 10.6 seconds. **Bullets:** - Decode throughput scales with memory bandwidth, so quantization from FP16 to INT4 can deliver a near 4x speedup. - Time to first token dominates perceived responsiveness in chat interfaces, even when total generation time is longer. - Tensor parallelism across more GPUs adds bandwidth but also interconnect overhead, so scaling is sub-linear beyond four to eight GPUs. - Continuous batching raises aggregate cluster throughput substantially while leaving single-stream latency roughly unchanged. #### Benchmarks used in this tool **Body:** The efficiency default of 65% reflects what mature serving engines achieve in practice after kernel launch overhead, memory access patterns, and cross-GPU communication. Vendors quote theoretical bandwidth; nobody reaches it. The 20% memory overhead factor covers KV cache at moderate concurrency, CUDA context, and activation buffers, and it will be conservative for very long contexts and high batch sizes. The prefill-to-decode ratio of forty is typical for prompts in the low thousands of tokens on modern serving stacks. If you run extremely long prompts, prefill becomes compute-bound instead and time to first token grows faster than this model predicts. #### Interpreting the result against user expectations **Body:** Time to first token is what users judge. Under 500 milliseconds feels instant, one to two seconds feels responsive, and beyond four seconds people assume the system is broken. Total response time matters less if you stream tokens, because a reader consumes roughly 15 to 20 tokens per second and anything faster than that reads as smooth. If your calculated throughput falls below 20 tokens per second, streaming will visibly stutter and you should quantize, shrink the model, or add GPUs. If the memory footprint exceeds the aggregate VRAM of your GPU count, the configuration will not load at all and no amount of tuning will rescue it. #### How Netray sizes and tunes on-prem inference **Body:** Netray deploys open-weight models inside customer networks for aerospace, defense, and electronics manufacturers who cannot send data to a public API. We benchmark candidate models on your actual prompts rather than public leaderboards, then tune the serving stack: quantization level, tensor parallel degree, batching policy, and KV cache configuration. Frequently the right answer is a small quantized model that answers ERP and quality questions in under two seconds, not a flagship model that takes fifteen. We integrate the result directly with SyteLine, LN, M3, and ServiceMax so answers are grounded in live operational data rather than a stale document dump. **Cta:** Get a benchmarked inference architecture sized for your latency targets, your models, and your security constraints. **Faq:** #### Why is inference bound by memory bandwidth rather than compute? **Answer:** During decode the model produces one token at a time, and every parameter must be read from GPU memory for each token. That means the arithmetic intensity is extremely low: the GPU spends most of its time waiting on memory rather than calculating. Prefill is different because it processes many prompt tokens simultaneously and does become compute-bound. This is why memory bandwidth specifications predict single-stream chat latency far better than FLOPS numbers do. #### How much does quantization actually hurt answer quality? **Answer:** Modern INT4 methods such as AWQ and GPTQ typically cost one to three percent on standard benchmarks while cutting memory reads by four times, which roughly quadruples decode speed. FP8 is close to lossless on most tasks. The degradation is not uniform: mathematical reasoning and long-chain tool use suffer more than extraction or summarization. Always validate quantized models against your own golden question set rather than trusting published averages. #### Should I add GPUs or use a smaller model to hit my latency target? **Answer:** Try the smaller model first. Doubling GPUs adds bandwidth but interconnect overhead means real gains are usually 1.5 to 1.7 times, and you pay for capital, power, and cooling permanently. Moving from a 70B model to a well-chosen 8B or 14B model quantized to INT4 can be ten times faster on identical hardware. If a task-tuned small model passes your evaluation bar, it is almost always the better answer for production inference. ## Search terms LLM inference latency calculator, tokens per second calculator, time to first token estimator, self-hosted LLM performance, GPU memory bandwidth inference, LLM quantization performance impact, on-prem LLM throughput sizing --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json