# Self-Hosted LLM Hardware Estimator: VRAM, GPUs, and Budget

> Source: https://netray.co/tools/self-hosted-llm-hardware-estimator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.

## Category

on-prem-ai

## Tool

**Kind:** calculator

**Inputs:**

#### Model size

**Key:** model_size_b

**Type:** select

**Default:** 32

**Help:** Parameter count of the open-weight model you plan to host.

**Options:**

##### 8B (small, fast)

**Value:** 8

##### 14B (balanced)

**Value:** 14

##### 32B (strong reasoning)

**Value:** 32

##### 70B (near-frontier)

**Value:** 70

##### 120B (large MoE class)

**Value:** 120

#### Serving precision

**Key:** precision

**Type:** select

**Default:** 1

**Help:** Bytes per parameter; INT8/FP8 is the enterprise default in 2026.

**Options:**

##### FP16 / BF16 (2 bytes/param)

**Value:** 2

##### INT8 / FP8 (1 byte/param)

**Value:** 1

##### INT4 / AWQ (0.55 bytes/param)

**Value:** 0.55

#### Concurrent sessions

**Key:** concurrent_sessions

**Type:** slider

**Default:** 25

**Min:** 1

**Max:** 500

**Step:** 1

**Unit:** sessions

**Help:** Simultaneous active conversations the server must hold in memory.

#### Context window per session

**Key:** context_length

**Type:** select

**Default:** 32

**Help:** Longer contexts inflate KV cache memory linearly.

**Options:**

##### 8K tokens (chat)

**Value:** 8

##### 32K tokens (documents/RAG)

**Value:** 32

##### 128K tokens (long documents)

**Value:** 128

**Outputs:**

#### Model weights memory

**Key:** weights_memory_gb

**Expr:** model_size_b * precision

**Format:** number

**Description:** VRAM consumed by model weights alone, in GB.

#### KV cache memory

**Key:** kv_cache_gb

**Expr:** round(concurrent_sessions * context_length * model_size_b * 0.002)

**Format:** number

**Description:** Attention cache for all concurrent sessions at full context, in GB (assumes GQA).

#### Total VRAM required

**Key:** total_vram_gb

**Expr:** round((weights_memory_gb + kv_cache_gb) * 1.1)

**Format:** number

**Description:** Weights plus KV cache plus 10% runtime overhead, in GB.

**Primary:** true

#### 80GB-class GPUs needed

**Key:** gpus_80gb_needed

**Expr:** ceil(total_vram_gb / 72)

**Format:** number

**Description:** H100/A100 cards required, keeping 10% VRAM headroom per card.

#### Estimated server budget

**Key:** est_hardware_cost

**Expr:** gpus_80gb_needed * 32000 + 18000

**Format:** currency

**Description:** GPUs at ~$32K each plus ~$18K for chassis, CPU, RAM, and storage.

**Primary:** true

**Disclaimer:** Estimates only. KV cache varies with attention architecture (GQA/MLA), and vendor pricing moves quarterly. Confirm with a proof-of-concept deployment before purchasing.

## Content

**Intro:** This estimator computes the hardware you need to self-host an open-weight LLM - total VRAM, GPU count, and a realistic server budget - for platform teams planning an on-prem deployment. The most common sizing mistake is budgeting for model weights alone and forgetting the KV cache, the per-session attention memory that grows linearly with concurrent users and context length and frequently exceeds the weights themselves. Enter your model, precision, concurrency, and context window, and the tool produces a defensible hardware spec you can take to procurement.

**Sections:**

#### The methodology: weights plus KV cache plus overhead

**Body:** Weights memory is parameters times bytes per parameter: a 32B model at INT8 needs 32 GB. KV cache is where deployments blow their budgets - each active session holds attention keys and values for every token in its context, and the estimator models this at roughly 2 MB per 1K tokens per billion parameters for modern grouped-query-attention (GQA) models. At the defaults - 25 concurrent sessions at 32K context on a 32B model - that is about 51 GB of cache, larger than the weights. We add 10% for the serving runtime and CUDA overhead, then divide by usable VRAM (90% of each 80GB card) to get GPU count.

#### Benchmark assumptions behind the numbers

**Body:** The defaults reflect 2025-2026 open-weight architectures and street pricing. Older non-GQA models cache roughly 4x more per token, and newer MLA-style attention caches substantially less - so treat the estimate as a mid-range planning figure.

**Bullets:**

- GQA KV cache: ~2 MB per 1K context tokens per billion parameters at FP16 cache precision
- Usable VRAM per card: 90% of nominal, to avoid fragmentation OOMs
- 80GB-class GPU street price: ~$32K including vendor margin
- Base server platform (CPU, RAM, NVMe, NIC): ~$18K per node

#### How to shrink the spec if the number shocks you

**Body:** Three levers cut hardware requirements fastest. First, quantize: INT8 serving is effectively lossless for most enterprise tasks and halves weight memory versus FP16; INT4 halves it again with a small, measurable quality cost. Second, cap the context window - most chat sessions never exceed 8K tokens, so provisioning every session for 128K wastes enormous cache memory; serving stacks can enforce per-request limits. Third, question concurrency: sessions idle between turns can have their cache evicted and recomputed, so the number that matters is sessions actively generating, which is usually far below open connections.

#### From estimate to running hardware with Netray

**Body:** Netray specifies, procures, and deploys self-hosted LLM servers for aerospace, defense, and discrete manufacturers - from single-node pilots to multi-node air-gapped clusters. We validate this estimate against your real traffic with a short proof of concept, negotiate hardware through our vendor relationships, and deliver a hardened serving stack with quantized models, monitoring, and failover configured. Clients typically go from estimate to production inference in six to ten weeks.

**Cta:** Get your hardware spec, then let Netray validate it with a proof of concept before you commit capex.

**Faq:**

#### Why is the KV cache sometimes bigger than the model itself?

**Answer:** Because it scales with users and context, not just model size. Weights are fixed, but every concurrent session stores keys and values for each token in its context window. Twenty-five sessions at 32K tokens is 800K cached token positions - on a 32B model that is roughly 50 GB even with grouped-query attention. This is why long-context enterprise deployments need far more VRAM than a quick weights-only calculation suggests.

#### Can I use cheaper 24GB or 48GB GPUs instead of 80GB cards?

**Answer:** Often, yes - especially for models at 32B and below with INT4/INT8 quantization. Four 48GB L40S cards cost roughly half of two H100s while offering similar total VRAM, at the price of lower memory bandwidth and slower token generation. For latency-tolerant internal workloads that trade is frequently worth it. Divide the estimator's total VRAM figure by 90% of your candidate card's memory to get the alternative count.

#### Does this estimate cover fine-tuning hardware?

**Answer:** No - it sizes inference serving only. LoRA-style fine-tuning can usually run on the same class of hardware with reduced concurrency, but full fine-tuning requires roughly 8-10x the weight memory for gradients and optimizer states. Most Netray clients fine-tune periodically on a separate schedule (or in a secure cloud burst where policy allows) and deploy the resulting adapters onto the inference hardware this tool estimates.

## Search terms

self-hosted llm hardware requirements, vram calculator llm, hardware to run llama 70b, kv cache memory calculator, llm server hardware estimator, gpu requirements open source llm, enterprise llm hosting hardware

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json