GPU Sizing Calculator for LLM Inference Workloads
This calculator tells infrastructure teams how many GPUs they need to serve a chosen open-weight LLM to their user base at acceptable latency. Sizing GPU inference capacity has two independent constraints: the model must fit in aggregate VRAM, and the cluster must sustain your peak token throughput. Most sizing mistakes come from checking only one. Enter your model class, quantization, hardware, and usage profile, and the tool computes both constraints and recommends the binding one - the number you should actually build to.
Your numbers
Parameter count of the open-weight model you plan to serve.
Bytes per parameter. INT8 halves memory vs FP16 with minimal quality loss.
VRAM per GPU in your target hardware class.
Users actively generating at peak, not total licensed seats. Typically 5-10% of seats.
How often an active user triggers a generation during peak usage.
Chat answers average 300-700 tokens; document drafting runs 1,000+.
Your results
Estimates only. Real throughput depends on serving stack (vLLM, TensorRT-LLM), batch sizes, context lengths, and interconnect. Validate with a load test before purchasing.
Get your full GPU sizing report
We will email a personalized sizing breakdown with serving-stack recommendations for your model and workload, and a Netray infrastructure specialist will follow up to review it with you.
No spam. Your results stay private. Unsubscribe anytime.
The two constraints that drive GPU count
Memory: a model needs roughly parameters times bytes-per-parameter in VRAM, plus 15-25% overhead for KV cache, activation buffers, and the serving runtime - the calculator uses 20%. We also reserve 10% of each card's VRAM as headroom, because running GPUs at 100% memory causes fragmentation and out-of-memory crashes under load spikes. Throughput: modern batched serving stacks like vLLM deliver aggregate output throughput that scales roughly inversely with parameter count on 80GB-class hardware - our heuristic is 40,000 divided by model size in billions, so about 570 tokens/second for a 70B model at INT8 and 5,000 tokens/second for an 8B model. Your GPU count is whichever constraint demands more cards.
Benchmarks used in this model
The throughput heuristic is calibrated against published vLLM and TensorRT-LLM benchmarks on H100-class hardware with continuous batching at moderate context lengths (4K-8K). Real numbers move with your stack and traffic shape, but the heuristic lands within about 30% of measured results across common deployments, which is adequate for capacity planning before a proof of concept.
- 8B model, INT8, one H100: roughly 4,000-6,000 output tokens/sec batched
- 70B model, INT8, tensor-parallel across 2x H100: roughly 400-700 tokens/sec
- Concurrent active users are typically 5-10% of total licensed seats
- Reserve 10% VRAM headroom per card to avoid OOM under load spikes
Interpreting the recommendation
If memory is the binding constraint, quantization is your cheapest lever: dropping from FP16 to INT8 halves the footprint with negligible quality loss for most enterprise tasks, and INT4/AWQ shrinks it further with a small accuracy trade-off worth evaluating. If throughput binds, consider whether a smaller model fine-tuned on your domain could serve most requests, routing only hard queries to the large model - a pattern that routinely cuts GPU requirements 50-70%. Either way, buy for the recommendation plus one node of growth headroom, not for peak fantasy load.
How Netray turns sizing into a running cluster
Netray builds and tunes on-prem inference clusters for manufacturers, including air-gapped environments for defense programs. We validate this sizing with a real load test on your candidate model and traffic profile before you issue a purchase order, then stand up the serving stack - vLLM or TensorRT-LLM, routing, observability, and autoscaling policies - so day-one performance matches the plan. If the numbers say two GPUs but procurement says H200s, we will tell you which spec actually serves your users best per dollar.
Frequently Asked Questions
Why does the calculator recommend fewer GPUs than my vendor quoted?
Hardware vendors size conservatively and often quote for FP16 serving with generous headroom, which can double or triple the GPU count. Quantization to INT8 or INT4 dramatically reduces both memory and cost with minimal quality impact for most enterprise use cases. That said, if you plan heavy fine-tuning on the same cluster, long 100K+ contexts, or strict sub-second latency targets, add capacity beyond this estimate.
How do I estimate concurrent users if we have not launched yet?
Start from total intended seats and apply a 5-10% peak concurrency factor - a 500-seat rollout typically peaks at 25-50 simultaneously active generators. Internal copilots skew lower (5%); customer-facing or agentic workloads that fire many automated requests skew much higher and should be modeled from expected request rates instead of users.
Does this work for fine-tuning workloads too?
No - this calculator sizes inference only. Fine-tuning has very different memory dynamics: full fine-tuning needs roughly 16-20 bytes per parameter for optimizer states, while LoRA-style parameter-efficient tuning fits in close to inference memory plus gradient overhead. Many teams fine-tune in a cloud burst or on a separate node, then deploy the resulting weights to the on-prem inference cluster this tool sizes.
Get your GPU count, then have Netray validate it with a real load test before you buy hardware.
Related Tools
Self-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
On-Prem AIOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
On-Prem AIOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Go Deeper
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.