Concurrent Users Per GPU Calculator
This free concurrent users per GPU calculator answers the question every capacity planner eventually asks: how many people can actually use this thing at once? It is built for IT directors and platform engineers sizing internal AI assistants and chat applications on self-hosted hardware. Enter your model's memory footprint, GPU class, and throughput characteristics, and the tool returns both actively-streaming capacity and total connected user capacity, which are very different numbers because most connected users are reading or typing rather than actively generating a response at any given instant.
Your numbers
Total memory available on the GPU class you are sizing.
Weights only, after any quantization. A 70B model at FP8 is about 70 GB, at INT4 about 35 GB.
Cache memory per concurrent sequence at your typical context length. Use the KV cache calculator for a precise figure.
Memory held back for CUDA context, activations, and a safety margin against out-of-memory errors.
Total tokens per second the GPU sustains across all active streams under continuous batching.
Decode speed below which streaming visibly stutters. 15-25 tokens/sec matches typical human reading speed.
Time a connected user spends reading or typing between their requests, when they consume no GPU capacity.
How long one response takes to stream back to a user while they are actively consuming GPU capacity.
Your results
Planning estimates only. Real capacity depends on request bursts, prompt length variance, and scheduler fairness policy. Load-test your actual serving stack before committing to a user count in an SLA.
Get your full user capacity report
We will email you a personalized concurrency breakdown across GPU classes and quantization options, and a Netray infrastructure specialist will follow up with a load-testing plan.
No spam. Your results stay private. Unsubscribe anytime.
Two limits, and the second matters more
Capacity is bound by two independent constraints. Memory limits how many sequences can hold a KV cache slot at once. Throughput limits how many sequences can decode fast enough to stay above an acceptable speed while sharing the GPU's compute. Whichever number is lower is your real ceiling on actively streaming users; buying more VRAM does nothing if throughput is the binding constraint, and vice versa. This tool computes both explicitly so you know which lever to pull.
- Memory-bound limit: VRAM available for cache divided by cache needed per user.
- Throughput-bound limit: aggregate tokens per second divided by minimum acceptable speed per user.
- The lower of the two is the real ceiling on simultaneously streaming responses.
- Fixing the wrong constraint (adding VRAM when throughput is the limit) wastes budget without adding capacity.
Why connected users vastly exceed active streams
A chat interface with 200 logged-in users rarely has 200 people simultaneously receiving tokens. Most of that population is reading a previous response, formulating their next question, or simply has the tab open without interacting. The ratio between think time and generation time defines a duty cycle: if a user spends 12 seconds thinking for every 8 seconds of active generation, only 40% of their connected time consumes GPU capacity, so one GPU that can actively stream to 10 users simultaneously can realistically serve 25 total connected users. This is the single most common reason a proof-of-concept that felt fine with 5 testers falls over at 150 real users, because nobody modeled the duty cycle.
Sizing for peak, not average
Total connected user capacity assumes a steady mix of thinking and generating users, which holds reasonably well for organic usage but breaks during synchronized peaks, such as everyone opening the assistant right after a company-wide announcement. Size hardware against your peak 15-minute window, not your daily average, and add explicit queueing behavior so requests wait gracefully rather than fail outright when that peak exceeds capacity. A rate limiter with a visible queue position beats a silent timeout every time.
How Netray sizes AI capacity for real organizations
Netray builds internal AI assistants for manufacturers and defense contractors where sizing has to survive contact with actual employee behavior, not a demo with five friendly testers. We model realistic think-time and generation-time ratios from comparable deployments, validate against load tests before go-live, and design graceful degradation for peak load rather than hard failures. Engagements typically include a capacity model tied to your actual headcount and expected adoption curve.
Frequently Asked Questions
Why is total connected user capacity so much higher than active stream capacity?
Because streaming a response is a small fraction of a user's total session time. Most of the time a person has your AI assistant open, they are reading the last answer or composing the next question, consuming zero GPU capacity. The ratio of think time to generation time determines the multiplier: with typical chat usage patterns, one GPU that can actively stream to 8-12 users at once can often serve 200 or more total connected users without anyone noticing degraded performance.
What happens when actual usage exceeds the calculated capacity?
Behavior depends entirely on how the serving stack is configured. A well-designed deployment queues excess requests and shows users their position, adding a few seconds of wait during peak load. A poorly configured one either times out silently, degrading every user's experience simultaneously as the scheduler thrashes across too many concurrent sequences, or crashes the server outright from an out-of-memory error. Always configure explicit admission control rather than letting the serving engine accept unlimited concurrent requests.
Should I size for average load or peak load?
Peak, always. AI assistant usage is not smooth throughout the day, it clusters around specific triggers: shift starts, post-meeting periods, or organization-wide announcements. Sizing for daily average load routinely leaves a deployment that feels fine most of the day but becomes unusable during the 15-minute windows that actually matter to adoption. Measure your peak 15-minute request rate from pilot data and size against that, with headroom.
Does adding more GPUs proportionally increase user capacity?
Running independent model replicas across additional GPUs scales user capacity close to linearly, since each replica serves its own pool of users behind a load balancer. This is different from tensor parallelism, which splits one model across GPUs to fit a larger model or reduce single-stream latency, and scales sub-linearly due to interconnect overhead. For pure user-count scaling, prefer more independent replicas over a larger tensor-parallel group once a model comfortably fits on one to two GPUs.
Get a capacity model sized to your real user population, not a pilot-scale assumption.
Related Tools
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
On-Prem AIvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AILLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Go Deeper
LLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
The IT Director Playbook for On-Prem AI
The IT director playbook for on-prem AI: hardware sizing, model selection, security hardening, and rollout steps for running LLMs inside your firewall.