# vLLM Throughput Estimator: Model Continuous Batching Performance

> Source: https://netray.co/tools/vllm-throughput-estimator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.

## Category

on-prem-ai

## Tool

**Kind:** calculator

**Inputs:**

#### Model size class

**Key:** model_footprint_gb

**Type:** select

**Default:** 140

**Unit:** GB

**Help:** Weight memory footprint at FP16. Quantized deployments should use a smaller equivalent GB figure.

**Options:**

##### 7B dense model (14 GB weights)

**Value:** 14

##### 13B dense model (26 GB weights)

**Value:** 26

##### 34B dense model (68 GB weights)

**Value:** 68

##### 70B dense model (140 GB weights)

**Value:** 140

#### GPU class

**Key:** gpu_class

**Type:** select

**Default:** 3350

**Unit:** GB/s

**Help:** Memory bandwidth per GPU, the dominant factor in decode speed for a memory-bound workload.

**Options:**

##### L40S (864 GB/s)

**Value:** 864

##### A100 80GB SXM (2,039 GB/s)

**Value:** 2039

##### H100 SXM (3,350 GB/s)

**Value:** 3350

##### H200 SXM (4,800 GB/s)

**Value:** 4800

#### GPUs serving this replica

**Key:** gpu_count

**Type:** number

**Default:** 2

**Min:** 1

**Max:** 16

**Step:** 1

**Unit:** GPUs

**Help:** Tensor-parallel degree for one model replica behind the vLLM scheduler.

#### Concurrent sequences in flight

**Key:** batch_size

**Type:** number

**Default:** 32

**Min:** 1

**Max:** 256

**Step:** 1

**Unit:** sequences

**Help:** How many requests vLLM's continuous batching scheduler is actively decoding at once.

#### Average output tokens per response

**Key:** avg_output_tokens

**Type:** number

**Default:** 400

**Min:** 10

**Max:** 8000

**Step:** 10

**Unit:** tokens

**Help:** Generated response length, used to convert aggregate throughput into requests per minute.

#### Achieved bandwidth efficiency

**Key:** efficiency_pct

**Type:** slider

**Default:** 55

**Min:** 20

**Max:** 85

**Step:** 5

**Unit:** %

**Help:** Real vLLM deployments reach 45-65% of theoretical bandwidth after PagedAttention and scheduling overhead.

**Outputs:**

#### Single-stream decode speed

**Key:** single_stream_tps

**Expr:** gpu_class * gpu_count * efficiency_pct / 100 / model_footprint_gb

**Format:** number

**Description:** Tokens per second for one isolated request before continuous batching gains are applied.

#### Effective batching concurrency

**Key:** effective_concurrency

**Expr:** min(batch_size, 40)

**Format:** number

**Description:** Continuous batching gains saturate once the workload turns compute-bound, typically near 32-48 sequences.

#### Aggregate throughput

**Key:** aggregate_tokens_per_sec

**Expr:** single_stream_tps * effective_concurrency * 0.82

**Format:** number

**Description:** Cluster-wide tokens per second across all concurrent sequences with scheduling overhead applied.

**Primary:** true

#### Sustained requests per minute

**Key:** requests_per_min

**Expr:** aggregate_tokens_per_sec * 60 / max(avg_output_tokens, 1)

**Format:** number

**Description:** How many complete responses of your target length this replica can sustain per minute.

**Primary:** true

#### Effective throughput per GPU

**Key:** tokens_per_sec_per_gpu

**Expr:** aggregate_tokens_per_sec / max(gpu_count, 1)

**Format:** number

**Description:** Aggregate throughput divided across GPU count, useful for comparing hardware options apples-to-apples.

**Disclaimer:** Planning estimates only. Real vLLM throughput depends on serving engine version, prefix caching, chunked prefill, quantization, and request mix. Benchmark your actual stack and prompts before setting an SLA.

## Content

**Intro:** This free vLLM throughput estimator translates model size, GPU class, and batch depth into a realistic tokens-per-second figure for a self-hosted deployment, and it is built for platform engineers sizing on-prem inference clusters. Continuous batching is the single biggest throughput lever in modern serving stacks like vLLM, TensorRT-LLM, and SGLang, because it lets the GPU work on dozens of requests simultaneously instead of idling between decode steps for one user at a time. Enter your model footprint, GPU class, and how many sequences you expect in flight, and the tool returns single-stream speed, aggregate cluster throughput, and sustained requests per minute.

**Sections:**

#### How continuous batching changes the math

**Body:** A single isolated request is memory-bandwidth bound: the GPU reads every weight from memory for every generated token, so a 70B model on two H100s tops out around single-digit tens of tokens per second per stream. Continuous batching changes the unit economics by processing many sequences per forward pass, amortizing that memory read across dozens of concurrent decodes. The gain is not linear, it flattens once the workload shifts from memory-bound to compute-bound, which is why this estimator caps effective concurrency near 40 sequences and applies a scheduling overhead factor rather than assuming perfect scaling.

**Bullets:**

- Single-stream decode speed is dominated by memory bandwidth divided by model size, not GPU compute (FLOPS).
- Continuous batching (vLLM's core innovation) lets one GPU serve many users concurrently for a small latency cost per user.
- Throughput gains from batching saturate around 32-48 concurrent sequences on typical mid-size deployments.
- PagedAttention reduces KV cache fragmentation, which is what makes high concurrency practical in the first place.

#### Reading the result against your SLA

**Body:** Aggregate tokens per second is a capacity number, not a latency number: it tells you how much total work the replica can absorb, not how fast any one user's response streams back. Convert it to requests per minute using your typical response length, then compare that against your peak concurrent user count. If requests per minute falls short of peak demand, you need more replicas, a smaller or more quantized model, or a faster GPU class, not just a bigger batch size, because batch size beyond the saturation point buys little additional throughput while degrading per-user latency.

#### Where teams get vLLM sizing wrong

**Body:** The most common mistake is sizing off a vendor's peak-benchmark tokens-per-second figure, which is measured at maximum batch saturation with short prompts and rarely reflects production traffic mixed with long context and bursty load. The second is ignoring that quantization changes the model footprint input directly: an FP8 70B model reads roughly 70 GB instead of 140 GB, which can meaningfully change how many GPUs you need for the same throughput target. Always re-run sizing after a quantization decision, not before it.

**Bullets:**

- Benchmark with your actual prompt and response length distribution, not a synthetic short-prompt test.
- Re-check GPU count needed after any quantization change, since it directly shrinks the memory footprint input.
- Reserve headroom for traffic bursts; sizing at exactly average load produces visible latency spikes at peak.

#### How Netray sizes production vLLM deployments

**Body:** Netray deploys open-weight models on vLLM, SGLang, and TensorRT-LLM inside customer networks for manufacturers and defense contractors who cannot send inference traffic to a public API. We benchmark candidate models and quantization levels against your actual production prompts, tune batching and scheduling parameters, and validate throughput against your real peak concurrency before you commit to hardware. Engagements typically begin with a two-week benchmarking sprint that produces a costed, right-sized serving architecture.

**Cta:** Get a benchmarked vLLM sizing plan tuned to your models, your prompts, and your peak concurrency target.

**Faq:**

#### Why does adding more GPUs not scale throughput linearly?

**Answer:** Tensor parallelism across GPUs adds interconnect communication overhead at every layer boundary, so doubling GPUs rarely doubles throughput. In practice you gain roughly 1.5 to 1.8 times per doubling on NVLink-connected systems and less over PCIe. Past four to eight GPUs for a single replica, it is usually more efficient to run additional independent replicas behind a load balancer than to keep growing one tensor-parallel group.

#### What is a realistic bandwidth efficiency to assume for vLLM?

**Answer:** Most production vLLM deployments achieve 45-65% of theoretical GPU memory bandwidth after accounting for kernel launch overhead, PagedAttention block management, and scheduling. Newer versions with chunked prefill and prefix caching push toward the higher end, especially for workloads with repeated system prompts. Treat anything above 70% as optimistic unless you have measured it on your own hardware and model.

#### Does batch size have a downside once throughput saturates?

**Answer:** Yes. Past the saturation point, adding more concurrent sequences no longer meaningfully raises aggregate throughput but does increase the time each individual user waits for their tokens, since the GPU is now dividing attention across more work than it can process in parallel. The practical target is the highest batch depth that still meets your per-user latency SLA, not the highest batch depth the GPU can technically accept.

#### How much does quantization change these throughput numbers?

**Answer:** Substantially. Moving a 70B model from FP16 to FP8 halves the memory footprint, which roughly doubles single-stream decode speed and lets more concurrent sequences fit in the same VRAM. Moving to INT4 can quadruple decode speed relative to FP16. Always re-run this estimator with the quantized model size before finalizing a hardware order.

## Search terms

vLLM throughput calculator, LLM tokens per second estimator, continuous batching throughput, vLLM GPU sizing calculator, paged attention throughput, LLM serving throughput calculator, tokens per second per GPU, self-hosted LLM performance

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json