LLM Serving Capacity Planner: From Peak Users to GPU Count
This free LLM serving capacity planner converts a peak concurrent user target directly into a GPU count, the number that ultimately becomes a purchase order or cloud budget line. It is built for infrastructure leads sizing a new deployment from a rollout target rather than reverse-engineering capacity from hardware already on hand. Enter your expected peak actively-streaming users, response length, minimum acceptable speed, and per-GPU throughput, and the tool returns the raw GPU requirement, a redundancy-adjusted hardware order, and the daily capacity that hardware delivers.
Your numbers
The worst-case number of users receiving tokens simultaneously, not total connected or logged-in users.
Typical generated response length for your primary use case.
Decode speed floor below which the experience feels broken. 15-25 matches typical reading speed.
Sustained tokens per second one GPU delivers under continuous batching at your model and quantization.
Extra capacity above the raw calculated minimum, for traffic spikes, failover, and maintenance windows.
Your results
Planning estimate assuming worst-case simultaneous peak streaming. Real traffic is rarely perfectly synchronized, so this sizing is intentionally conservative. Validate with a load test before finalizing a hardware order.
Get your full capacity and hardware plan
We will email you a personalized GPU sizing breakdown with redundancy scenarios and a daily capacity projection, and a Netray infrastructure specialist will follow up with a rollout plan.
Optional. The tool and every export work without this. No spam.
Sizing from worst-case simultaneous load
The core calculation assumes every peak user is streaming a response at the exact same moment, which is deliberately conservative: it is the scenario that determines whether your SLA holds during the worst 60 seconds of your busiest day, not your average traffic pattern. Required throughput is peak users multiplied by minimum acceptable per-user speed; dividing by what one GPU sustains gives the raw GPU count. This worst-case framing protects you from the failure mode where a system performs beautifully in testing and then visibly degrades the first time real synchronized load arrives.
- Peak concurrent users should reflect actively-streaming load, not total logged-in or connected user count.
- Minimum acceptable speed per user, not average speed, protects the experience of every user during peak load.
- This method deliberately over-provisions relative to average load, which is the correct bias for a launch.
- Revisit sizing quarterly as adoption grows; peak load rarely stays flat after a successful rollout.
Why redundancy is not optional
The raw GPU count is a bare minimum with zero margin for a failed node, a maintenance window, or a traffic spike above your peak estimate. A 20% redundancy buffer is a reasonable starting point for most internal deployments; regulated or customer-facing workloads with a formal SLA typically warrant 30-50% to comfortably absorb an N+1 failover scenario without degrading service. Skipping redundancy is the single most common reason a capacity plan that looked adequate on paper fails during its first real incident.
Turning GPU count into a daily capacity check
Daily token and response capacity is a useful sanity check in the opposite direction: does the hardware this calculation recommends make sense against your total expected daily volume, not just peak instantaneous load? If daily capacity vastly exceeds realistic total daily usage, your peak assumption may be too aggressive, or your minimum acceptable speed target may be stricter than necessary. If daily capacity is uncomfortably close to expected volume, you have no room for growth and should revisit the redundancy buffer upward.
How Netray plans capacity for production rollouts
Netray sizes AI infrastructure for manufacturers rolling out internal assistants and ERP copilots across hundreds to thousands of employees, where getting the launch-day capacity number wrong either wastes budget or produces a visibly broken first impression. We build capacity models from comparable deployment data, validate with load testing before go-live, and design headroom that matches your actual redundancy requirements rather than a generic percentage. Engagements typically include a capacity plan tied to your specific rollout timeline.
Frequently Asked Questions
How do I estimate peak concurrent users before launch?
Start from your total target user population and apply a concurrency assumption from comparable deployments: internal chat assistants commonly see 5-15% of the logged-in population actively streaming at true peak moments, such as right after a company announcement or at shift start. If you have pilot data, use its observed peak-to-total ratio directly rather than an industry rule of thumb, since usage patterns vary meaningfully by organization and use case.
Is 20% redundancy enough for a production deployment?
For most internal tools without a formal external SLA, 20-25% is a reasonable starting buffer. For customer-facing deployments, regulated workloads with committed response time SLAs, or any system where an outage has real business cost, 30-50% is more appropriate, and you should also plan for N+1 at the individual GPU node level so a single hardware failure does not take down an entire serving tier.
Should capacity planning use average or minimum acceptable per-user speed?
Always minimum acceptable, never average. Average speed hides the fact that at true peak concurrency, per-stream decode speed for every user drops as the GPU's batching scheduler divides throughput across more simultaneous work. Sizing against a minimum floor guarantees every user stays above an acceptable experience even during the worst moment, rather than guaranteeing only that the average user has a good experience while some subset suffers visibly degraded speed.
What happens if actual peak load exceeds this sizing?
With a well-configured admission control and queueing policy, excess requests wait briefly rather than degrading service for everyone already being served, which is a graceful failure mode. Without admission control, the serving engine typically accepts all requests and per-user throughput degrades across the board simultaneously, which is a much worse user experience. Capacity planning and admission control configuration should always be designed together, not treated as separate concerns.
Get a validated capacity plan and hardware order sized to your actual rollout timeline and peak load.
Related Tools
Concurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
On-Prem AIvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AIGPU Cluster Utilization Calculator
Turn GPU capital, amortization, and operating cost into an effective cost per productive GPU hour, and find the utilization threshold where owning beats renting.
On-Prem AIMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- GPU Sizing Calculator for LLM Inference
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- LLM API vs Self-Hosted Cost Calculator
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator
Go Deeper
LLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Keep exploring
Free tools
- LLM Latency Budget PlannerBreak total response time into time-to-first-token, generation time, and network overhead, then see your exact margin or shortfall against a target SLA.
- AI Hardware Refresh PlannerWeigh your current GPU fleet's remaining book value against the cost of refreshing to a newer generation, factoring in performance-per-watt gains and power savings.
- Embedding Model Serving Cost CalculatorEstimate the GPU hours and dollar cost to embed your daily document volume, factoring in embedding model size, batching efficiency, and re-indexing overhead.
ERP fixes and how-tos
- How to Use the MRP Planner Workbench in SyteLineHow planners review SyteLine MRP output in the planner workbench: read exception messages, firm and release planned orders, and reschedule supply safely.
- How to add a private AI layer to an on-prem CMMSHow to add a private AI assistant to an on-prem CMMS: read-only data access, work order retrieval, local models, permissions, approval-based write-back.
Runs with your ERP
- Delfoi Planner with Oracle NetSuite: finite schedulingHow Delfoi Planner adds finite, constraint-based scheduling on top of NetSuite: reading work orders, routings and demand, writing achievable dates back.
- Delfoi Planner with SAP S/4HANA: finite scheduling for production ordersHow Delfoi Planner adds finite scheduling and a drag-and-drop Gantt to SAP S/4HANA, which data flows, how to set it up, what breaks and where private AI helps.
Guides
- Multi-GPU LLM Serving: Tensor Parallelism vs Pipeline ParallelismMulti-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
- SGLang vs vLLM: An Honest Production Serving ComparisonSGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.
- On-Prem LLM Inference Hardware in 2026: A Practitioner's RoundupOn-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Troubleshooting
- Why GPU utilization is low during LLM inference, and how to raise itLow GPU utilization during LLM inference usually means concurrency is too low, the client is synchronous, or tokenization/network is the real bottleneck. Fix order inside.
- Why vLLM's multi-LoRA serving fails, and how to fix rank and capacity mismatchesvLLM multi-LoRA needs --enable-lora plus max-lora-rank, max-loras, and registered lora-modules sized to your adapters. Fix rank and count mismatches here.