On-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Designing an on-prem GPU cluster is a three-way tradeoff between node configuration, network fabric, and storage throughput, and getting any one of the three wrong throttles the other two no matter how much you spend. A cluster with eight H100s per node and a gigabit management network cannot train anything beyond single-node jobs. A cluster with fast InfiniBand and slow local storage stalls on checkpoint writes. The right design starts from the workload: single-node inference serving needs almost none of the interconnect investment that multi-node fine-tuning or pretraining requires, and most enterprise buyers overspend on the wrong layer because a vendor quote bundled it in.
Node Sizing: How Many GPUs Per Server
The standard enterprise building block in 2026 is an 8-GPU node, either HGX H100, HGX H200, or the newer B200/GB200 platforms, with NVLink and NVSwitch providing full GPU-to-GPU bandwidth inside the chassis. An 8x H100 SXM node draws roughly 10 to 10.5 kW fully loaded and needs 2 to 4 TB of system RAM and dual high core count CPUs to avoid data loader bottlenecks starving the GPUs. For inference-only serving of models under 70B parameters, 2 or 4 GPU nodes with PCIe cards are often more cost effective and easier to right-size against fluctuating demand than committing to 8-GPU blocks.
- 8x H100/H200 SXM node: NVLink/NVSwitch full-mesh bandwidth, ~10-10.5 kW per node, best for training and large-model serving
- 2x-4x GPU PCIe nodes: lower cost per node, no NVLink between GPUs, adequate for single-model inference under 70B parameters
- Size system RAM at 2x aggregate GPU memory minimum to avoid CPU-side data loader bottlenecks during training
- Reserve 15-20 percent headroom on CPU cores and PCIe lanes for storage and network I/O, not just the GPUs
Networking: InfiniBand vs RoCE and When It Matters
For single-node inference, the fabric barely matters since all GPU traffic stays inside the NVLink domain. For multi-node training and fine-tuning, the fabric is the single biggest lever on effective utilization. NVIDIA Quantum-2 InfiniBand NDR at 400 Gb/s per port remains the reference architecture for large training clusters because of its low latency and GPUDirect RDMA maturity, but it requires a dedicated switch fabric, specialized cabling, and staff who know how to operate it. RoCEv2 over 400 GbE Ethernet closes most of the latency gap in well-tuned deployments and lets IT teams reuse existing Ethernet operational knowledge, at the cost of more careful congestion control tuning to avoid tail latency spikes under load.
- InfiniBand NDR (400 Gb/s): lowest latency, mature GPUDirect RDMA, standard choice for multi-node pretraining clusters
- RoCEv2 over 400GbE: 80-90 percent of IB performance when tuned, reuses existing Ethernet operations skills
- Rail-optimized topology (one NIC per GPU, non-blocking leaf-spine) is required to avoid GPUs starving each other during collective operations
- Single-node serving clusters can skip high-speed interconnect investment entirely and use standard 25/100GbE for management traffic
Storage: Local NVMe vs Parallel Filesystem
Training clusters need two storage tiers. Local NVMe on each node handles checkpoint scratch space and active dataset shards, sized for at least 2x your largest checkpoint to allow safe overlapping writes. A shared parallel filesystem such as WEKA, VAST Data, or Lustre serves the training corpus and holds durable checkpoints, and it needs to sustain 50 to 150+ GB/s aggregate read throughput for an 8-node cluster to keep GPUs fed during data-heavy epochs. Undersized shared storage is the most common bottleneck we see in customer clusters: GPUs sit at 40 to 60 percent utilization not because of compute limits but because the storage layer cannot deliver batches fast enough, especially with small-file heavy datasets.
- Local NVMe scratch: size for 2x largest checkpoint per node, use for active shards and fast checkpoint writes
- Shared parallel filesystem (WEKA, VAST, Lustre, or NVIDIA-certified alternatives) for durable corpus and checkpoint storage
- Target 50-150+ GB/s aggregate read throughput for an 8-node H100/H200 training cluster to avoid GPU starvation
- Checkpoint frequency tradeoff: more frequent checkpoints reduce restart cost but increase storage write pressure during long runs
Control Plane and Cluster Orchestration
Slurm remains the dominant scheduler for dedicated training clusters because of its mature gang scheduling and topology-aware placement, while Kubernetes with the NVIDIA GPU Operator is more common where the cluster serves mixed inference and batch workloads alongside other applications. Whichever you choose, instrument GPU utilization, network fabric errors, and storage latency from day one, because a cluster that looks healthy on GPU temperature dashboards can still be running at half its effective throughput. Budget for a jump host, out-of-band management (IPMI/BMC) on every node, and a documented node-drain procedure before the first production job runs.
How Netray Designs and Deploys On-Prem GPU Clusters
Netray designs GPU cluster architecture against the actual workload, not a vendor's default bill of materials, sizing nodes, fabric, and storage as one system for regulated manufacturers and defense suppliers who need the cluster fully inside their own network boundary. We run a capacity assessment against your real model sizes and concurrency targets, specify the InfiniBand or RoCE fabric only where multi-node throughput justifies the cost, and validate storage throughput before hardware ships rather than discovering the bottleneck after go-live. Every design includes the orchestration layer, monitoring, and a documented runbook your own team can operate without a standing consulting retainer.
Frequently Asked Questions
How many GPUs do I need for an on-prem AI cluster?
It depends entirely on the workload. Serving a single 70B model to moderate concurrency fits on 2 to 4 GPUs with enough memory headroom. Multi-model serving or fine-tuning workloads typically need at least one full 8-GPU node. Multi-node pretraining or large-scale fine-tuning starts at 2 to 4 nodes (16-32 GPUs) and scales from there. Size against your concurrent user count and model parameter count, not a round number.
Do I need InfiniBand for an on-prem GPU cluster?
Only if you are running multi-node training or fine-tuning jobs that require GPU-to-GPU communication across nodes. Single-node inference serving, even with a 70B or larger model spread across 8 GPUs, uses NVLink inside the chassis and does not need InfiniBand. If you plan multi-node pretraining or large distributed fine-tuning runs, InfiniBand NDR or a well-tuned RoCEv2 fabric materially changes effective GPU utilization.
What storage throughput does a GPU training cluster need?
For an 8-node H100 or H200 training cluster, target 50 to 150+ GB/s aggregate read throughput from shared storage to avoid starving the GPUs during data-heavy epochs. Undersized storage is the most common cause of low GPU utilization in customer deployments, often masked because GPU temperature and power draw look normal even while utilization sits at 40 to 60 percent.
Key Takeaways
- 1Node Sizing: How Many GPUs Per Server: The standard enterprise building block in 2026 is an 8-GPU node, either HGX H100, HGX H200, or the newer B200/GB200 platforms, with NVLink and NVSwitch providing full GPU-to-GPU bandwidth inside the chassis. An 8x H100 SXM node draws roughly 10 to 10.5 kW fully loaded and needs 2 to 4 TB of system RAM and dual high core count CPUs to avoid data loader bottlenecks starving the GPUs.
- 2Networking: InfiniBand vs RoCE and When It Matters: For single-node inference, the fabric barely matters since all GPU traffic stays inside the NVLink domain. For multi-node training and fine-tuning, the fabric is the single biggest lever on effective utilization.
- 3Storage: Local NVMe vs Parallel Filesystem: Training clusters need two storage tiers. Local NVMe on each node handles checkpoint scratch space and active dataset shards, sized for at least 2x your largest checkpoint to allow safe overlapping writes.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Multi-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
Free ToolGPU Cluster Buildout Cost Calculator
Turn GPU count and class into a full cluster budget covering server chassis, networking fabric, power and cooling capex, and install, with a true cost per GPU.
Terms used in this article
Planning an on-prem GPU cluster and want the node, network, and storage sizing validated against your real workload before you buy? Netray will run the capacity assessment and deliver a bill of materials you can defend to finance and IT.
Related Resources
NVIDIA H100 vs H200 vs B200 for Enterprise AI in 2026
Compare NVIDIA H100, H200, and B200 GPUs on specs, price, availability, and performance per dollar for enterprise LLM inference and training in 2026.
AI & AutomationMulti-Node LLM Training Infrastructure: Networking and Storage
Multi-node LLM training infrastructure explained: InfiniBand vs RoCE tradeoffs, storage throughput needs, and cluster topology for enterprise fine-tuning.
AI & AutomationAI Datacenter Power and Cooling Planning for GPU Racks
Plan AI datacenter power and cooling for GPU racks: density thresholds, liquid cooling triggers, PUE targets, and real 2026 numbers for H100 to B200 racks.