Self-Hosted LLM Hardware Estimator: VRAM, GPUs, and Budget
This estimator computes the hardware you need to self-host an open-weight LLM - total VRAM, GPU count, and a realistic server budget - for platform teams planning an on-prem deployment. The most common sizing mistake is budgeting for model weights alone and forgetting the KV cache, the per-session attention memory that grows linearly with concurrent users and context length and frequently exceeds the weights themselves. Enter your model, precision, concurrency, and context window, and the tool produces a defensible hardware spec you can take to procurement.
Your numbers
Parameter count of the open-weight model you plan to host.
Bytes per parameter; INT8/FP8 is the enterprise default in 2026.
Simultaneous active conversations the server must hold in memory.
Longer contexts inflate KV cache memory linearly.
Your results
Estimates only. KV cache varies with attention architecture (GQA/MLA), and vendor pricing moves quarterly. Confirm with a proof-of-concept deployment before purchasing.
Get your full hardware sizing report
We will email a personalized hardware specification with current vendor pricing for your model and workload, and a Netray specialist will follow up to walk through procurement options.
Optional. The tool and every export work without this. No spam.
The methodology: weights plus KV cache plus overhead
Weights memory is parameters times bytes per parameter: a 32B model at INT8 needs 32 GB. KV cache is where deployments blow their budgets - each active session holds attention keys and values for every token in its context, and the estimator models this at roughly 2 MB per 1K tokens per billion parameters for modern grouped-query-attention (GQA) models. At the defaults - 25 concurrent sessions at 32K context on a 32B model - that is about 51 GB of cache, larger than the weights. We add 10% for the serving runtime and CUDA overhead, then divide by usable VRAM (90% of each 80GB card) to get GPU count.
Benchmark assumptions behind the numbers
The defaults reflect 2025-2026 open-weight architectures and street pricing. Older non-GQA models cache roughly 4x more per token, and newer MLA-style attention caches substantially less - so treat the estimate as a mid-range planning figure.
- GQA KV cache: ~2 MB per 1K context tokens per billion parameters at FP16 cache precision
- Usable VRAM per card: 90% of nominal, to avoid fragmentation OOMs
- 80GB-class GPU street price: ~$32K including vendor margin
- Base server platform (CPU, RAM, NVMe, NIC): ~$18K per node
How to shrink the spec if the number shocks you
Three levers cut hardware requirements fastest. First, quantize: INT8 serving is effectively lossless for most enterprise tasks and halves weight memory versus FP16; INT4 halves it again with a small, measurable quality cost. Second, cap the context window - most chat sessions never exceed 8K tokens, so provisioning every session for 128K wastes enormous cache memory; serving stacks can enforce per-request limits. Third, question concurrency: sessions idle between turns can have their cache evicted and recomputed, so the number that matters is sessions actively generating, which is usually far below open connections.
From estimate to running hardware with Netray
Netray specifies, procures, and deploys self-hosted LLM servers for aerospace, defense, and discrete manufacturers - from single-node pilots to multi-node air-gapped clusters. We validate this estimate against your real traffic with a short proof of concept, negotiate hardware through our vendor relationships, and deliver a hardened serving stack with quantized models, monitoring, and failover configured. Clients typically go from estimate to production inference in six to ten weeks.
Frequently Asked Questions
Why is the KV cache sometimes bigger than the model itself?
Because it scales with users and context, not just model size. Weights are fixed, but every concurrent session stores keys and values for each token in its context window. Twenty-five sessions at 32K tokens is 800K cached token positions - on a 32B model that is roughly 50 GB even with grouped-query attention. This is why long-context enterprise deployments need far more VRAM than a quick weights-only calculation suggests.
Can I use cheaper 24GB or 48GB GPUs instead of 80GB cards?
Often, yes - especially for models at 32B and below with INT4/INT8 quantization. Four 48GB L40S cards cost roughly half of two H100s while offering similar total VRAM, at the price of lower memory bandwidth and slower token generation. For latency-tolerant internal workloads that trade is frequently worth it. Divide the estimator's total VRAM figure by 90% of your candidate card's memory to get the alternative count.
Does this estimate cover fine-tuning hardware?
No - it sizes inference serving only. LoRA-style fine-tuning can usually run on the same class of hardware with reduced concurrency, but full fine-tuning requires roughly 8-10x the weight memory for gradients and optimizer states. Most Netray clients fine-tune periodically on a separate schedule (or in a secure cloud burst where policy allows) and deploy the resulting adapters onto the inference hardware this tool estimates.
Get your hardware spec, then let Netray validate it with a proof of concept before you commit capex.
Related Tools
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
On-Prem AIOn-Prem LLM Total Cost of Ownership Calculator
Model the full multi-year cost of running LLMs on your own hardware, including GPU capex, power, cooling, support contracts, and operations staffing.
On-Prem AIOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
More On-Prem AI tools
See all- On-Prem vs Cloud AI Cost Comparison Calculator
- Air-Gapped AI Readiness Assessment
- LLM API vs Self-Hosted Cost Calculator
- RAG Infrastructure Sizing Calculator
- On-Prem AI ROI Calculator
- On-Prem AI Deployment Checklist
- LLM Token Cost Calculator
- AI Data Center Power and Cooling Calculator
- Model Context Window Planner
Go Deeper
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Keep exploring
Free tools
- LLM API vs Self-Hosted Cost CalculatorModel your API bill from requests and token mix, compare it against an all-in self-hosted monthly cost, and see monthly and annual savings.
- Self-Hosted Odoo AI Assistant Sizing and Value CalculatorEstimate GPU count, token volume, index size, hours saved and payback for a private on-prem AI assistant on Self-Hosted Odoo.
- Llama 4 Hardware Requirements CalculatorEstimate VRAM, GPU count, and capital cost to run Llama 4 Scout or Maverick on your own hardware, accounting for full mixture-of-experts weight loading and KV cache growth.
ERP fixes and how-tos
- Self-Hosted Odoo AI Assistant: Private Architecture With a Local ModelAdd a private AI assistant to self-hosted Odoo: external API with user API keys, record rules respected, PostgreSQL replica, local model and audit trail.
- How to Add AI to a Self-Hosted Ignition Gateway Without CloudPrivate AI for self-hosted Ignition: Tag Historian tables, alarm journal, Named Queries, WebDev, role mapping, local GPUs and an audit trail. No cloud needed.
Guides
- The IT Director Playbook for On-Prem AI: Hardware, Models, Security, RolloutThe IT director playbook for on-prem AI: hardware sizing, model selection, security hardening, and rollout steps for running LLMs inside your firewall.
- On-Prem LLM Deployment Architecture: The Complete Reference GuideReference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
- On-Prem LLM Inference Hardware in 2026: A Practitioner's RoundupOn-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
AI on your ERP
- SAP Joule or a Private LLM Beside SAP: An Honest ComparisonAn honest comparison of SAP Joule and Business AI against a private, self-hosted LLM beside SAP: what each covers, where they overlap, and where CIOs run both.
- SAP BTP and a Private LLM: Where the Generative AI Hub Fits and Where It Doesn'tWhere SAP BTP's Generative AI Hub fits and where a self-hosted LLM belongs instead. CAP, Integration Suite, and an architecture decision framework.
Troubleshooting
- How to reduce VRAM usage for LLM inference without hurting qualityReduce LLM inference VRAM by quantizing weights, capping max context length and concurrency, and using FlashAttention, in that order of impact.
- Why your cost per token is too high, and the biggest levers to fix itHigh cost per token almost always traces back to low GPU utilization, the wrong model size, or no quantization or caching. Here is how to find the biggest lever.