On-Prem GPU Sizing Calculator for ERP AI
Buying GPUs for a private ERP assistant starts with memory, not speed. This calculator estimates whether a model fits, how much cache concurrent chats need, and how many cards that implies.
Your numbers
Share of named users with an active generation at the busiest moment. 3 to 10 percent is common for chat.
Prompt plus retrieved data plus output
About 130 MB for an 8B model and 320 MB for a 70B model with grouped query attention at FP16 cache. Check your model config.
Your results
A memory-based first estimate only. Throughput, latency targets, batching, prompt length distribution and the serving engine (vLLM, Ollama and others) determine real capacity. Validate with a load test on the target hardware before purchase.
GPUs needed vs named users
Sweeps Named users from 50 to 5000, holding your other inputs fixed.
Validate the sizing with a load test
We can run your candidate model against realistic ERP prompts and report real throughput and latency before you buy hardware.
Optional. The tool and every export work without this. No spam.
How to use it
Pick the model class you expect to run and the precision you will serve it at. FP8 is a common production choice, while INT4 trades some quality for a much smaller footprint. Enter named users and the share active at peak. For ERP question answering, 3 to 10 percent concurrency is typical, because users read the answer before asking again. Set average context to match your retrieval design, since retrieved rows and chunks dominate it.
The memory model
Weights memory is parameters times bytes per parameter. A 70B model at FP8 needs about 70 GB for weights alone. The KV cache grows with every concurrent token: memory per request is context length times cache size per token. Add a runtime overhead for activations, CUDA graphs and fragmentation, and keep around 10 percent of each card free. The calculator divides total memory by 90 percent of card memory and then rounds up to a power of two for tensor parallelism.
- Quantizing the KV cache to FP8 roughly halves cache memory
- Prefix caching helps when many requests share a long system prompt
- Mixture of experts models need all expert weights in memory even though few are active per token
Reading the result
With the defaults, a 70B model at FP8 for 500 named users needs weights of roughly 70 GB plus a few GB of cache, which lands on two 80 GB cards after headroom. The chart shows the step pattern as users grow: cache is small relative to weights, so adding users adds GPUs slowly until context or concurrency is large. If the answer is one GPU with very little spare memory, consider a second for resilience.
Common mistakes
Sizing on weights alone is the most common error, and it fails the moment users send long retrieved contexts. Another is ignoring availability: a single node is a single point of failure for a tool people will rely on. Also check power, cooling and rack space, which often limit an on-prem build more than the card price does. Test throughput under realistic prompts before ordering hardware.
Frequently Asked Questions
Does a smaller model need fewer GPUs?
Yes, weights memory scales with parameter count, so an 8B model fits on one 24 GB card at INT4 or FP8. Whether quality is enough for your ERP questions is a separate test you should run before deciding.
Why not just size for all users at once?
Chat is bursty. Users spend most of the time reading or working, so concurrency of 3 to 10 percent is typical. Batch jobs and agents that run unattended can raise it, so size those separately.
Is this accurate for every model?
It is a planning estimate. Cache per token varies with layers, key value heads and head dimension. Read your model configuration and replace the default KV cache figure for a tighter number.
Do I need an NVIDIA GPU?
Most serving engines support NVIDIA best, but AMD and other accelerators are available. The memory arithmetic is the same, and the main differences are software support and availability in your region.
If you want a sizing validated with a load test on your own prompts and documents, we can benchmark candidate models on your hardware or a short-term rented node.
Related Tools
ChatGPT vs On-Prem LLM TCO Calculator
Compare yearly cost of a hosted LLM API with a self-hosted GPU server for the same workload and find the break-even volume.
AI Agents & AutomationLLM Token Cost Calculator for ERP Chat
Estimate what an ERP chat assistant costs per month on a pay-per-token API, including retrieved context and output length.
On-Prem AIGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
AI Agents & AutomationRAG Readiness Assessment for ERP Documents
Ten questions that show whether your documents and ERP attachments are ready to feed a retrieval-based AI assistant.
More On-Prem AI tools
See all- On-Prem LLM Total Cost of Ownership Calculator
- On-Prem vs Cloud AI Cost Comparison Calculator
- Self-Hosted LLM Hardware Estimator
- Air-Gapped AI Readiness Assessment
- LLM API vs Self-Hosted Cost Calculator
- RAG Infrastructure Sizing Calculator
- Open-Weight Model Selector
- On-Prem AI ROI Calculator
- On-Prem AI Deployment Checklist