LLM Gateway Queueing Delay Calculator
Inference services feel fine until utilization climbs, then waits explode. This calculator uses a simple queueing model to show average and 95th percentile wait for a given arrival rate, service time, replicas and slots, so you can see where the cliff is.
Your numbers
Full generation time for one request at typical load.
Effective parallel streams with continuous batching before latency degrades.
Your results
Pools all slots into one fast server, which understates waiting versus a true multi-server queue, and assumes random arrivals. Use it for direction and load test before committing hardware.
95th percentile queue wait vs arrival rate
Sweeps Peak request arrival rate from 10 to 150, holding your other inputs fixed.
Get an inference load test plan
Optional: we can outline a load test for your models and GPUs.
Optional. The tool and every export work without this. No spam.
How to use it
Set the peak requests per minute from gateway logs, not the daily average. Enter average service time per request, including generation of the full answer. Replicas and slots give capacity: each slot is one request being generated in parallel. Read utilization first. Below roughly sixty to seventy percent the queue is short; above eighty-five percent waits grow quickly. The chart shows the curve as arrivals rise.
The model
Capacity mu = replicas x slots / service time. Utilization rho = arrivals / mu. In an M/M/1 queue, average wait = rho / (mu - lambda) and the probability of waiting longer than t is rho x exp(-(mu - lambda) t), so the 95th percentile wait is ln(rho / 0.05) / (mu - lambda). Pooling all slots into one fast server is an approximation that is optimistic compared with separate servers.
- Waits scale with 1 / (1 - utilization).
- Long generations raise service time and shrink capacity.
- Batching changes slots, so measure them.
Using the result
Plan for a peak utilization around sixty to seventy percent, then add replicas or reduce service time through shorter outputs, smaller models or caching. Set request timeouts and queue limits so overload sheds load instead of piling up. Separate interactive chat from batch jobs such as nightly document summarization, or the batch work will push chat waits up. Always confirm with a load test using real prompt lengths, since slot counts depend on memory and context size.
Frequently Asked Questions
What is a slot?
One request being generated in parallel. With continuous batching a GPU can serve many at once, but latency per request rises as slots fill, so the usable count comes from load testing.
Why does wait explode near full utilization?
Random arrivals create bursts. When there is little spare capacity, the backlog from a burst clears slowly, so queue time rises sharply as utilization approaches one.
Why cap utilization at 0.995?
At or above one the model has no steady state and the wait is unbounded. The cap keeps outputs finite, but treat any utilization near one as overload.
Is this accurate for multi-GPU servers?
It is directional. Real systems have batching effects and uneven request lengths. Use it to choose a test range, then measure.
Netray sizes and load tests on-prem inference for ERP assistants. We can help you find the real slot count for your GPUs and models.
Related Tools
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AIConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
On-Prem AIAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
ERP OperationsERP AI Agent API Load Impact Calculator
Convert user count and calls per question into peak ERP calls per second, concurrent sessions and rate limit headroom before you connect an agent.
On-Prem AIAI Agent Stack Availability and Downtime Calculator
Multiply component availabilities in an ERP AI stack into end-to-end uptime and hours of downtime per year, and see what a second GPU node buys.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- GPU Sizing Calculator for LLM Inference
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- LLM API vs Self-Hosted Cost Calculator
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator
Keep exploring
Free tools
- Expedite Cost vs Delay Cost CalculatorDecide whether paying to expedite is cheaper than waiting, using fees, price premium, order value and the daily cost of delay.
- ERP Selection Delay Cost CalculatorPut a monthly price on stalling an ERP selection: benefits you do not capture, the legacy system you keep paying for, and price drift on the quote.
- GPU Sizing Calculator for LLM InferenceWork out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
ERP fixes and how-tos
- How NetSuite scheduled script queueing actually worksUnderstand why NetSuite Scheduled Scripts sit in Queued or Pending status, how account-level concurrency limits and deployment priority affect start time, and how to design self-rescheduling scripts that do not compound the queue.
- How to Add AI to a Self-Hosted Ignition Gateway Without CloudPrivate AI for self-hosted Ignition: Tag Historian tables, alarm journal, Named Queries, WebDev, role mapping, local GPUs and an audit trail. No cloud needed.
- How to Add a Private AI Layer to Oracle EBS 12.2 On-PremReference architecture for a private AI layer on Oracle EBS 12.2: Active Data Guard replica, ISG REST, MO and responsibility security, local GPUs and audit.
Guides
- The LLM Inference Cost Optimization PlaybookCut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
- Infor ION Connect API Gateway Setup GuideConfigure Infor ION Connect API Gateway with OAuth2 authentication, endpoint registration, rate limiting, and external system integration for CloudSuite.
- LLM Observability: TTFT, ITL, Throughput, and GPU DashboardsLLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
AI on your ERP
- AI Agents Over Infor ION API, BODs, and the Infor Data LakeBuild AI agents on Infor ION API Gateway, BODs, and Data Lake, alongside or instead of Infor's own GenAI features, with your choice of model and hosting.
- NetSuite Procurement With AI for Purchase Orders, Receipts and Vendor BillsAI for NetSuite procurement: query purchase orders, item receipts and vendor bills with SuiteQL, flag three-way match breaks, and draft vendor follow-ups.
Integrations
- How to add a mobile app to abas ERPBuild or add a mobile app for abas ERP: barcode scanning, goods receipt, stock moves and approvals over EDP or REST, with offline queueing and device control.
- How to connect Power BI to SAP Business OneConnect Power BI to SAP Business One: SQL Server or HANA direct, Service Layer OData, gateway setup, key manufacturing dashboards and natural-language Q&A.
Troubleshooting
- Why time to first token is high on your LLM endpoint, and how to fix itHigh TTFT usually comes from long prefill prompts, no prefix caching, cold starts, or queueing delay. Here is how to diagnose and fix each one.
- Why GPU utilization is low during LLM inference, and how to raise itLow GPU utilization during LLM inference usually means concurrency is too low, the client is synchronous, or tokenization/network is the real bottleneck. Fix order inside.