# LLM Gateway Queueing Delay Calculator

> Source: https://netray.co/tools/llm-gateway-queueing-delay-calculator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Use queueing theory to see how request wait times explode as an inference service approaches full utilization.

## Category

on-prem-ai

## Audience

- IT manager
- Infrastructure
- AI platform team

## Tool

**Kind:** calculator

**Inputs:**

#### Peak request arrival rate

**Key:** arrivals

**Type:** slider

**Default:** 90

**Min:** 10

**Max:** 150

**Step:** 5

**Unit:** requests/min

#### Average time to serve one request

**Key:** service_s

**Type:** number

**Default:** 6

**Min:** 0.1

**Step:** 0.5

**Unit:** seconds

**Help:** Full generation time for one request at typical load.

#### Model replicas

**Key:** replicas

**Type:** number

**Default:** 2

**Min:** 1

**Step:** 1

**Unit:** replicas

#### Concurrent requests per replica

**Key:** slots

**Type:** number

**Default:** 8

**Min:** 1

**Step:** 1

**Unit:** slots

**Help:** Effective parallel streams with continuous batching before latency degrades.

**Outputs:**

#### Total service capacity

**Key:** mu

**Expr:** replicas * slots / service_s * 60

**Format:** decimal

**Description:** Requests per minute the pool can complete.

#### Utilization

**Key:** rho

**Expr:** arrivals / mu

**Format:** decimal

**Primary:** true

**Description:** At or above 1.0 the queue grows without limit.

#### Utilization used (capped at 0.995)

**Key:** rho_c

**Expr:** min(rho, 0.995)

**Format:** decimal

#### Average queue wait

**Key:** wq

**Expr:** rho_c / (mu / 60 * (1 - rho_c))

**Format:** number

**Description:** Seconds, single-pooled-server approximation.

#### 95th percentile queue wait

**Key:** p95

**Expr:** max(0, ln(rho_c / 0.05) / (mu / 60 * (1 - rho_c)))

**Format:** number

**Primary:** true

**Description:** Seconds.

#### Average time to answer

**Key:** total

**Expr:** wq + service_s

**Format:** number

**Description:** Seconds, queue plus service.

**Chart:**

#### 95th percentile queue wait vs arrival rate

**Type:** line

**X:** arrivals

**From:** 10

**To:** 150

**Steps:** 28

**Series:**

- p95

**Disclaimer:** Pools all slots into one fast server, which understates waiting versus a true multi-server queue, and assumes random arrivals. Use it for direction and load test before committing hardware.

## Content

**Intro:** Inference services feel fine until utilization climbs, then waits explode. This calculator uses a simple queueing model to show average and 95th percentile wait for a given arrival rate, service time, replicas and slots, so you can see where the cliff is.

**Sections:**

#### How to use it

**Body:** Set the peak requests per minute from gateway logs, not the daily average. Enter average service time per request, including generation of the full answer. Replicas and slots give capacity: each slot is one request being generated in parallel. Read utilization first. Below roughly sixty to seventy percent the queue is short; above eighty-five percent waits grow quickly. The chart shows the curve as arrivals rise.

#### The model

**Body:** Capacity mu = replicas x slots / service time. Utilization rho = arrivals / mu. In an M/M/1 queue, average wait = rho / (mu - lambda) and the probability of waiting longer than t is rho x exp(-(mu - lambda) t), so the 95th percentile wait is ln(rho / 0.05) / (mu - lambda). Pooling all slots into one fast server is an approximation that is optimistic compared with separate servers.

**Bullets:**

- Waits scale with 1 / (1 - utilization).
- Long generations raise service time and shrink capacity.
- Batching changes slots, so measure them.

#### Using the result

**Body:** Plan for a peak utilization around sixty to seventy percent, then add replicas or reduce service time through shorter outputs, smaller models or caching. Set request timeouts and queue limits so overload sheds load instead of piling up. Separate interactive chat from batch jobs such as nightly document summarization, or the batch work will push chat waits up. Always confirm with a load test using real prompt lengths, since slot counts depend on memory and context size.

**Cta:** Netray sizes and load tests on-prem inference for ERP assistants. We can help you find the real slot count for your GPUs and models.

**Faq:**

#### What is a slot?

**Answer:** One request being generated in parallel. With continuous batching a GPU can serve many at once, but latency per request rises as slots fill, so the usable count comes from load testing.

#### Why does wait explode near full utilization?

**Answer:** Random arrivals create bursts. When there is little spare capacity, the backlog from a burst clears slowly, so queue time rises sharply as utilization approaches one.

#### Why cap utilization at 0.995?

**Answer:** At or above one the model has no steady state and the wait is unbounded. The cap keeps outputs finite, but treat any utilization near one as overload.

#### Is this accurate for multi-GPU servers?

**Answer:** It is directional. Real systems have batching effects and uneven request lengths. Use it to choose a test range, then measure.

## Search terms

llm queueing delay calculator, llm gateway capacity planning, inference utilization wait time, mm1 queue calculator, ai inference p95 wait time, how many llm replicas do i need, llm request queue latency

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json