AI Inference Latency Calculator: Predict Self-Hosted LLM Response Times
This free AI inference latency calculator estimates how fast a self-hosted language model will respond on your hardware, and it is built for platform engineers and IT leaders sizing on-prem GPU clusters. Enter model size, weight precision, GPU class and count, achieved bandwidth efficiency, and prompt and response lengths, and the tool returns memory footprint, decode throughput, time to first token, and total response time. Decode is bound by memory bandwidth, not compute, which is why a smaller quantized model on modest GPUs often feels dramatically faster than a large model on expensive ones.
Your numbers
Total parameter count in billions. For mixture-of-experts models use active parameters per token.
Lower precision shrinks the memory read per token, which is the main driver of decode speed.
Decode is memory-bandwidth bound, so this matters far more than raw FLOPS for single-stream latency.
Tensor-parallel degree for one replica. Scaling is sub-linear because of interconnect overhead.
Real serving stacks reach 55-75% of theoretical bandwidth after kernel and interconnect overhead.
Full prompt length including retrieved context. Drives prefill time and time to first token.
Generated response length. Decode time scales almost linearly with this number.
Your results
Rough planning estimates only. Real latency depends on serving engine, batch size, speculative decoding, and interconnect topology. Benchmark your actual stack before committing to an SLA.
Get your full inference performance report
We will email you a personalized latency and throughput analysis with quantization, batching, and GPU sizing options, and a Netray AI infrastructure specialist will follow up with a benchmark plan.
No spam. Your results stay private. Unsubscribe anytime.
How the latency model works
Generating each token requires reading the model weights from GPU memory, so single-stream decode speed is approximately aggregate memory bandwidth divided by the bytes that must be read per token. A 70B model at FP16 occupies about 140 GB of weights, and adding roughly 20% for KV cache and activations gives a 168 GB working footprint. Four H100 SXM GPUs provide 13,400 GB/s of theoretical bandwidth; at a realistic 65% efficiency that is 8,710 GB/s, or about 52 tokens per second. Prefill processes prompt tokens in parallel and runs far faster per token, modeled here at roughly forty times decode speed, so a 2,000-token prompt yields about 960 milliseconds to first token. Adding 500 output tokens gives a total response near 10.6 seconds.
- Decode throughput scales with memory bandwidth, so quantization from FP16 to INT4 can deliver a near 4x speedup.
- Time to first token dominates perceived responsiveness in chat interfaces, even when total generation time is longer.
- Tensor parallelism across more GPUs adds bandwidth but also interconnect overhead, so scaling is sub-linear beyond four to eight GPUs.
- Continuous batching raises aggregate cluster throughput substantially while leaving single-stream latency roughly unchanged.
Benchmarks used in this tool
The efficiency default of 65% reflects what mature serving engines achieve in practice after kernel launch overhead, memory access patterns, and cross-GPU communication. Vendors quote theoretical bandwidth; nobody reaches it. The 20% memory overhead factor covers KV cache at moderate concurrency, CUDA context, and activation buffers, and it will be conservative for very long contexts and high batch sizes. The prefill-to-decode ratio of forty is typical for prompts in the low thousands of tokens on modern serving stacks. If you run extremely long prompts, prefill becomes compute-bound instead and time to first token grows faster than this model predicts.
Interpreting the result against user expectations
Time to first token is what users judge. Under 500 milliseconds feels instant, one to two seconds feels responsive, and beyond four seconds people assume the system is broken. Total response time matters less if you stream tokens, because a reader consumes roughly 15 to 20 tokens per second and anything faster than that reads as smooth. If your calculated throughput falls below 20 tokens per second, streaming will visibly stutter and you should quantize, shrink the model, or add GPUs. If the memory footprint exceeds the aggregate VRAM of your GPU count, the configuration will not load at all and no amount of tuning will rescue it.
How Netray sizes and tunes on-prem inference
Netray deploys open-weight models inside customer networks for aerospace, defense, and electronics manufacturers who cannot send data to a public API. We benchmark candidate models on your actual prompts rather than public leaderboards, then tune the serving stack: quantization level, tensor parallel degree, batching policy, and KV cache configuration. Frequently the right answer is a small quantized model that answers ERP and quality questions in under two seconds, not a flagship model that takes fifteen. We integrate the result directly with SyteLine, LN, M3, and ServiceMax so answers are grounded in live operational data rather than a stale document dump.
Frequently Asked Questions
Why is inference bound by memory bandwidth rather than compute?
During decode the model produces one token at a time, and every parameter must be read from GPU memory for each token. That means the arithmetic intensity is extremely low: the GPU spends most of its time waiting on memory rather than calculating. Prefill is different because it processes many prompt tokens simultaneously and does become compute-bound. This is why memory bandwidth specifications predict single-stream chat latency far better than FLOPS numbers do.
How much does quantization actually hurt answer quality?
Modern INT4 methods such as AWQ and GPTQ typically cost one to three percent on standard benchmarks while cutting memory reads by four times, which roughly quadruples decode speed. FP8 is close to lossless on most tasks. The degradation is not uniform: mathematical reasoning and long-chain tool use suffer more than extraction or summarization. Always validate quantized models against your own golden question set rather than trusting published averages.
Should I add GPUs or use a smaller model to hit my latency target?
Try the smaller model first. Doubling GPUs adds bandwidth but interconnect overhead means real gains are usually 1.5 to 1.7 times, and you pay for capital, power, and cooling permanently. Moving from a 70B model to a well-chosen 8B or 14B model quantized to INT4 can be ten times faster on identical hardware. If a task-tuned small model passes your evaluation bar, it is almost always the better answer for production inference.
Get a benchmarked inference architecture sized for your latency targets, your models, and your security constraints.
Related Tools
GPU Cluster Utilization Calculator
Turn GPU capital, amortization, and operating cost into an effective cost per productive GPU hour, and find the utilization threshold where owning beats renting.
On-Prem AIModel Context Window Planner
Allocate a fixed context window across system prompt, retrieved chunks, conversation history, and reserved output, then see exactly how much retrieval headroom is left.
On-Prem AIAI Data Center Power and Cooling Calculator
Convert GPU count and class into facility electrical load, required cooling capacity, and annual energy cost before you commit to an on-prem AI build.
Go Deeper
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.