LLM Gateway Queueing Delay Calculator
Inference services feel fine until utilization climbs, then waits explode. This calculator uses a simple queueing model to show average and 95th percentile wait for a given arrival rate, service time, replicas and slots, so you can see where the cliff is.
Your numbers
Full generation time for one request at typical load.
Effective parallel streams with continuous batching before latency degrades.
Your results
Pools all slots into one fast server, which understates waiting versus a true multi-server queue, and assumes random arrivals. Use it for direction and load test before committing hardware.
95th percentile queue wait vs arrival rate
Sweeps Peak request arrival rate from 10 to 150, holding your other inputs fixed.
Get an inference load test plan
Optional: we can outline a load test for your models and GPUs.
Optional. The tool and every export work without this. No spam.
How to use it
Set the peak requests per minute from gateway logs, not the daily average. Enter average service time per request, including generation of the full answer. Replicas and slots give capacity: each slot is one request being generated in parallel. Read utilization first. Below roughly sixty to seventy percent the queue is short; above eighty-five percent waits grow quickly. The chart shows the curve as arrivals rise.
The model
Capacity mu = replicas x slots / service time. Utilization rho = arrivals / mu. In an M/M/1 queue, average wait = rho / (mu - lambda) and the probability of waiting longer than t is rho x exp(-(mu - lambda) t), so the 95th percentile wait is ln(rho / 0.05) / (mu - lambda). Pooling all slots into one fast server is an approximation that is optimistic compared with separate servers.
- Waits scale with 1 / (1 - utilization).
- Long generations raise service time and shrink capacity.
- Batching changes slots, so measure them.
Using the result
Plan for a peak utilization around sixty to seventy percent, then add replicas or reduce service time through shorter outputs, smaller models or caching. Set request timeouts and queue limits so overload sheds load instead of piling up. Separate interactive chat from batch jobs such as nightly document summarization, or the batch work will push chat waits up. Always confirm with a load test using real prompt lengths, since slot counts depend on memory and context size.
Frequently Asked Questions
What is a slot?
One request being generated in parallel. With continuous batching a GPU can serve many at once, but latency per request rises as slots fill, so the usable count comes from load testing.
Why does wait explode near full utilization?
Random arrivals create bursts. When there is little spare capacity, the backlog from a burst clears slowly, so queue time rises sharply as utilization approaches one.
Why cap utilization at 0.995?
At or above one the model has no steady state and the wait is unbounded. The cap keeps outputs finite, but treat any utilization near one as overload.
Is this accurate for multi-GPU servers?
It is directional. Real systems have batching effects and uneven request lengths. Use it to choose a test range, then measure.
Netray sizes and load tests on-prem inference for ERP assistants. We can help you find the real slot count for your GPUs and models.
Related Tools
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AIConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
On-Prem AIAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
ERP OperationsERP AI Agent API Load Impact Calculator
Convert user count and calls per question into peak ERP calls per second, concurrent sessions and rate limit headroom before you connect an agent.
On-Prem AIAI Agent Stack Availability and Downtime Calculator
Multiply component availabilities in an ERP AI stack into end-to-end uptime and hours of downtime per year, and see what a second GPU node buys.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- GPU Sizing Calculator for LLM Inference
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- LLM API vs Self-Hosted Cost Calculator
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator