# LLM Serving Capacity Planner: From Peak Users to GPU Count > Source: https://netray.co/tools/llm-serving-capacity-planner > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Free Tools ## Short answer Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers. ## Category on-prem-ai ## Tool **Kind:** calculator **Inputs:** #### Peak actively-streaming users **Key:** peak_concurrent_users **Type:** number **Default:** 150 **Min:** 1 **Max:** 20000 **Step:** 5 **Unit:** users **Help:** The worst-case number of users receiving tokens simultaneously, not total connected or logged-in users. #### Average tokens per response **Key:** avg_tokens_per_response **Type:** number **Default:** 450 **Min:** 10 **Max:** 16000 **Step:** 10 **Unit:** tokens **Help:** Typical generated response length for your primary use case. #### Minimum acceptable speed per user **Key:** target_tokens_per_sec_per_user **Type:** number **Default:** 25 **Min:** 5 **Max:** 200 **Step:** 5 **Unit:** tokens/sec **Help:** Decode speed floor below which the experience feels broken. 15-25 matches typical reading speed. #### Per-GPU aggregate throughput **Key:** gpu_aggregate_throughput_tps **Type:** number **Default:** 1800 **Min:** 50 **Max:** 20000 **Step:** 50 **Unit:** tokens/sec **Help:** Sustained tokens per second one GPU delivers under continuous batching at your model and quantization. #### Redundancy and headroom buffer **Key:** redundancy_pct **Type:** slider **Default:** 20 **Min:** 0 **Max:** 100 **Step:** 5 **Unit:** % **Help:** Extra capacity above the raw calculated minimum, for traffic spikes, failover, and maintenance windows. **Outputs:** #### Required simultaneous throughput **Key:** required_streaming_throughput_tps **Expr:** peak_concurrent_users * target_tokens_per_sec_per_user **Format:** number **Description:** Total tokens per second needed if every peak user streams at your minimum acceptable speed at once. #### Raw GPU count needed **Key:** gpus_required_raw **Expr:** required_streaming_throughput_tps / max(gpu_aggregate_throughput_tps, 1) **Format:** number **Description:** GPUs needed to cover peak load with zero redundancy margin. #### GPUs to provision **Key:** gpus_required_with_redundancy **Expr:** ceil(gpus_required_raw * (1 + redundancy_pct / 100)) **Format:** number **Description:** The actual hardware order: raw requirement plus your redundancy and headroom buffer, rounded up. **Primary:** true #### Daily token capacity (millions) **Key:** max_tokens_served_per_day_m **Expr:** gpu_aggregate_throughput_tps * gpus_required_with_redundancy * 86400 / 1000000 **Format:** number **Description:** Total tokens this cluster can serve in 24 hours at full theoretical throughput. #### Daily response capacity **Key:** daily_response_capacity **Expr:** max_tokens_served_per_day_m * 1000000 / max(avg_tokens_per_response, 1) **Format:** number **Description:** How many complete responses of your typical length this cluster can generate per day at full utilization. **Primary:** true **Disclaimer:** Planning estimate assuming worst-case simultaneous peak streaming. Real traffic is rarely perfectly synchronized, so this sizing is intentionally conservative. Validate with a load test before finalizing a hardware order. ## Content **Intro:** This free LLM serving capacity planner converts a peak concurrent user target directly into a GPU count, the number that ultimately becomes a purchase order or cloud budget line. It is built for infrastructure leads sizing a new deployment from a rollout target rather than reverse-engineering capacity from hardware already on hand. Enter your expected peak actively-streaming users, response length, minimum acceptable speed, and per-GPU throughput, and the tool returns the raw GPU requirement, a redundancy-adjusted hardware order, and the daily capacity that hardware delivers. **Sections:** #### Sizing from worst-case simultaneous load **Body:** The core calculation assumes every peak user is streaming a response at the exact same moment, which is deliberately conservative: it is the scenario that determines whether your SLA holds during the worst 60 seconds of your busiest day, not your average traffic pattern. Required throughput is peak users multiplied by minimum acceptable per-user speed; dividing by what one GPU sustains gives the raw GPU count. This worst-case framing protects you from the failure mode where a system performs beautifully in testing and then visibly degrades the first time real synchronized load arrives. **Bullets:** - Peak concurrent users should reflect actively-streaming load, not total logged-in or connected user count. - Minimum acceptable speed per user, not average speed, protects the experience of every user during peak load. - This method deliberately over-provisions relative to average load, which is the correct bias for a launch. - Revisit sizing quarterly as adoption grows; peak load rarely stays flat after a successful rollout. #### Why redundancy is not optional **Body:** The raw GPU count is a bare minimum with zero margin for a failed node, a maintenance window, or a traffic spike above your peak estimate. A 20% redundancy buffer is a reasonable starting point for most internal deployments; regulated or customer-facing workloads with a formal SLA typically warrant 30-50% to comfortably absorb an N+1 failover scenario without degrading service. Skipping redundancy is the single most common reason a capacity plan that looked adequate on paper fails during its first real incident. #### Turning GPU count into a daily capacity check **Body:** Daily token and response capacity is a useful sanity check in the opposite direction: does the hardware this calculation recommends make sense against your total expected daily volume, not just peak instantaneous load? If daily capacity vastly exceeds realistic total daily usage, your peak assumption may be too aggressive, or your minimum acceptable speed target may be stricter than necessary. If daily capacity is uncomfortably close to expected volume, you have no room for growth and should revisit the redundancy buffer upward. #### How Netray plans capacity for production rollouts **Body:** Netray sizes AI infrastructure for manufacturers rolling out internal assistants and ERP copilots across hundreds to thousands of employees, where getting the launch-day capacity number wrong either wastes budget or produces a visibly broken first impression. We build capacity models from comparable deployment data, validate with load testing before go-live, and design headroom that matches your actual redundancy requirements rather than a generic percentage. Engagements typically include a capacity plan tied to your specific rollout timeline. **Cta:** Get a validated capacity plan and hardware order sized to your actual rollout timeline and peak load. **Faq:** #### How do I estimate peak concurrent users before launch? **Answer:** Start from your total target user population and apply a concurrency assumption from comparable deployments: internal chat assistants commonly see 5-15% of the logged-in population actively streaming at true peak moments, such as right after a company announcement or at shift start. If you have pilot data, use its observed peak-to-total ratio directly rather than an industry rule of thumb, since usage patterns vary meaningfully by organization and use case. #### Is 20% redundancy enough for a production deployment? **Answer:** For most internal tools without a formal external SLA, 20-25% is a reasonable starting buffer. For customer-facing deployments, regulated workloads with committed response time SLAs, or any system where an outage has real business cost, 30-50% is more appropriate, and you should also plan for N+1 at the individual GPU node level so a single hardware failure does not take down an entire serving tier. #### Should capacity planning use average or minimum acceptable per-user speed? **Answer:** Always minimum acceptable, never average. Average speed hides the fact that at true peak concurrency, per-stream decode speed for every user drops as the GPU's batching scheduler divides throughput across more simultaneous work. Sizing against a minimum floor guarantees every user stays above an acceptable experience even during the worst moment, rather than guaranteeing only that the average user has a good experience while some subset suffers visibly degraded speed. #### What happens if actual peak load exceeds this sizing? **Answer:** With a well-configured admission control and queueing policy, excess requests wait briefly rather than degrading service for everyone already being served, which is a graceful failure mode. Without admission control, the serving engine typically accepts all requests and per-user throughput degrades across the board simultaneously, which is a much worse user experience. Capacity planning and admission control configuration should always be designed together, not treated as separate concerns. ## Search terms LLM serving capacity planner, GPU count from user load, self-hosted LLM capacity planning, AI serving cluster sizing, LLM peak load planning, N+1 redundancy GPU sizing, on-prem AI capacity calculator, enterprise LLM deployment sizing --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json