# On-Prem GPU Sizing Calculator for ERP AI

> Source: https://netray.co/tools/on-prem-gpu-sizing-calculator-for-erp-ai
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Work out how much GPU memory and how many cards you need to serve an open-weight model to your ERP users on-prem.

## Category

on-prem-ai

## Audience

- CIO
- Infrastructure manager
- IT manager

## Tool

**Kind:** calculator

**Inputs:**

#### Model size

**Key:** params

**Type:** select

**Default:** 70

**Options:**

##### 8B parameters

**Value:** 8

##### 14B parameters

**Value:** 14

##### 32B parameters

**Value:** 32

##### 70B parameters

**Value:** 70

##### 120B parameters

**Value:** 120

#### Weight precision

**Key:** bytes

**Type:** select

**Default:** 1

**Options:**

##### FP16 / BF16 (2 bytes)

**Value:** 2

##### FP8 / INT8 (1 byte)

**Value:** 1

##### INT4 (0.5 bytes)

**Value:** 0.5

#### Named users

**Key:** named

**Type:** number

**Default:** 500

**Min:** 1

**Step:** 10

**Unit:** users

#### Peak concurrency

**Key:** conc_pct

**Type:** slider

**Default:** 5

**Min:** 1

**Max:** 50

**Step:** 1

**Unit:** %

**Help:** Share of named users with an active generation at the busiest moment. 3 to 10 percent is common for chat.

#### Average context per request

**Key:** ctx

**Type:** number

**Default:** 6000

**Min:** 500

**Step:** 500

**Unit:** tokens

**Help:** Prompt plus retrieved data plus output

#### KV cache per 1,000 tokens

**Key:** kv_mb

**Type:** number

**Default:** 320

**Min:** 10

**Step:** 10

**Unit:** MB

**Help:** About 130 MB for an 8B model and 320 MB for a 70B model with grouped query attention at FP16 cache. Check your model config.

#### Memory per GPU

**Key:** gpu_gb

**Type:** select

**Default:** 80

**Options:**

##### 24 GB

**Value:** 24

##### 48 GB

**Value:** 48

##### 80 GB

**Value:** 80

##### 141 GB

**Value:** 141

#### Runtime overhead and headroom

**Key:** overhead

**Type:** number

**Default:** 20

**Min:** 0

**Max:** 60

**Step:** 5

**Unit:** %

**Outputs:**

#### Peak concurrent requests

**Key:** conc

**Expr:** max(1, ceil(named * conc_pct / 100))

**Format:** integer

#### Model weights

**Key:** w_gb

**Expr:** params * bytes

**Format:** decimal

**Description:** Billions of parameters times bytes per parameter gives gigabytes

#### KV cache at peak

**Key:** kv_gb

**Expr:** conc * ctx / 1000 * kv_mb / 1024

**Format:** decimal

#### Total memory needed

**Key:** total_gb

**Expr:** (w_gb + kv_gb) * (1 + overhead / 100)

**Format:** decimal

**Primary:** true

#### Minimum GPUs by memory

**Key:** min_gpu

**Expr:** ceil(total_gb / (gpu_gb * 0.9))

**Format:** integer

**Primary:** true

#### GPUs rounded to tensor parallel size

**Key:** tp_gpu

**Expr:** pow(2, ceil(ln(min_gpu) / ln(2)))

**Format:** integer

**Description:** Tensor parallel groups are normally 1, 2, 4 or 8 GPUs

**Chart:**

#### GPUs needed vs named users

**Type:** line

**X:** named

**From:** 50

**To:** 5000

**Steps:** 30

**Series:**

- min_gpu

**Disclaimer:** A memory-based first estimate only. Throughput, latency targets, batching, prompt length distribution and the serving engine (vLLM, Ollama and others) determine real capacity. Validate with a load test on the target hardware before purchase.

## Content

**Intro:** Buying GPUs for a private ERP assistant starts with memory, not speed. This calculator estimates whether a model fits, how much cache concurrent chats need, and how many cards that implies.

**Sections:**

#### How to use it

**Body:** Pick the model class you expect to run and the precision you will serve it at. FP8 is a common production choice, while INT4 trades some quality for a much smaller footprint. Enter named users and the share active at peak. For ERP question answering, 3 to 10 percent concurrency is typical, because users read the answer before asking again. Set average context to match your retrieval design, since retrieved rows and chunks dominate it.

#### The memory model

**Body:** Weights memory is parameters times bytes per parameter. A 70B model at FP8 needs about 70 GB for weights alone. The KV cache grows with every concurrent token: memory per request is context length times cache size per token. Add a runtime overhead for activations, CUDA graphs and fragmentation, and keep around 10 percent of each card free. The calculator divides total memory by 90 percent of card memory and then rounds up to a power of two for tensor parallelism.

**Bullets:**

- Quantizing the KV cache to FP8 roughly halves cache memory
- Prefix caching helps when many requests share a long system prompt
- Mixture of experts models need all expert weights in memory even though few are active per token

#### Reading the result

**Body:** With the defaults, a 70B model at FP8 for 500 named users needs weights of roughly 70 GB plus a few GB of cache, which lands on two 80 GB cards after headroom. The chart shows the step pattern as users grow: cache is small relative to weights, so adding users adds GPUs slowly until context or concurrency is large. If the answer is one GPU with very little spare memory, consider a second for resilience.

#### Common mistakes

**Body:** Sizing on weights alone is the most common error, and it fails the moment users send long retrieved contexts. Another is ignoring availability: a single node is a single point of failure for a tool people will rely on. Also check power, cooling and rack space, which often limit an on-prem build more than the card price does. Test throughput under realistic prompts before ordering hardware.

**Cta:** If you want a sizing validated with a load test on your own prompts and documents, we can benchmark candidate models on your hardware or a short-term rented node.

**Faq:**

#### Does a smaller model need fewer GPUs?

**Answer:** Yes, weights memory scales with parameter count, so an 8B model fits on one 24 GB card at INT4 or FP8. Whether quality is enough for your ERP questions is a separate test you should run before deciding.

#### Why not just size for all users at once?

**Answer:** Chat is bursty. Users spend most of the time reading or working, so concurrency of 3 to 10 percent is typical. Batch jobs and agents that run unattended can raise it, so size those separately.

#### Is this accurate for every model?

**Answer:** It is a planning estimate. Cache per token varies with layers, key value heads and head dimension. Read your model configuration and replace the default KV cache figure for a tighter number.

#### Do I need an NVIDIA GPU?

**Answer:** Most serving engines support NVIDIA best, but AMD and other accelerators are available. The memory arithmetic is the same, and the main differences are software support and availability in your region.

## Search terms

gpu sizing calculator llm, how many gpus for llama 70b, on prem llm hardware requirements, vllm gpu memory calculator, kv cache size calculator, gpu sizing for private chatgpt, llm vram requirements calculator, on premise ai server sizing

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json