AI & Automation5 min readNetray Engineering TeamUpdated March 4, 2025

Enterprise GPU Cluster Planning for AI Workloads: Sizing, Cost, and Design

Enterprise GPU cluster planning is the process of sizing GPU compute, networking, storage, power, and cooling to match your actual AI workload profile - inference, fine-tuning, or both - before committing capital. Most mid-market manufacturers need far less hardware than vendors suggest: a well-designed 2-4 node cluster covers LLM inference, RAG, and LoRA fine-tuning for a 1,000-person company. This guide walks through workload profiling, GPU selection (L40S vs A100 vs H100), interconnect decisions, facility requirements, and realistic three-year cost models so you buy the cluster you need, not the one on the sales sheet.

Profile Workloads Before Picking GPUs

Cluster sizing starts with three numbers: peak concurrent users, target tokens per second per user, and the largest model you will serve. Inference-only shops serving quantized 8B-32B models can standardize on NVIDIA L40S (48 GB, ~$8,000-$11,000 per card) - excellent FP8 throughput without HBM pricing. If you plan LoRA fine-tuning on 70B models or need maximum batch throughput, H100 80 GB (or H200 141 GB) with NVLink is worth the premium. A100 80 GB cards remain a strong value on the secondary market at $12,000-$18,000. Rule of thumb: a 70B model at FP8 with 32K context and 40 concurrent streams needs roughly 4x H100; the same load on a 14B model fits on 2x L40S.

Inference on 8B-32B quantized models: L40S nodes deliver the best cost per token for most ERP automation workloads
70B-class serving or fine-tuning: 4x-8x H100 80 GB with NVLink; SXM form factor over PCIe for multi-GPU jobs
Budget alternative: refurbished A100 80 GB clusters at 40-50% of H100 pricing with mature driver support
Always benchmark with vLLM against your real prompt mix - synthetic benchmarks overstate throughput by 20-40%

Networking and Storage Decisions That Matter

For pure inference clusters up to 4 nodes, 25/100 GbE is sufficient - inference traffic is API calls, not gradient exchange. The moment multi-node training or tensor-parallel serving across nodes enters the picture, you need 400 Gb InfiniBand NDR or RoCEv2 with proper lossless configuration, which adds $3,000-$8,000 per node in NICs plus switch costs. Storage is simpler than vendors claim: model weights and vector indexes live happily on NVMe (a 70B FP8 checkpoint is ~70 GB), while training datasets want a fast NFS or parallel filesystem. A 2-node inference cluster does not need a $200,000 storage array; a pair of NVMe-backed servers with ZFS replication covers it.

Power, Cooling, and Facility Reality Checks

GPU density breaks assumptions built for typical enterprise racks. An 8x H100 HGX server draws up to 10.2 kW; three of them exceed what most manufacturing-floor server rooms deliver per rack (often 5-15 kW). Before ordering, audit rack power (208V/415V three-phase availability), UPS capacity, and cooling. Air cooling handles up to roughly 30-40 kW per rack with rear-door heat exchangers; beyond that you are into direct liquid cooling territory. Many manufacturers place the cluster in an existing MDF with 2-3 nodes per rack and skip facility upgrades entirely - a deliberate design choice that saves $100,000+ in electrical work and keeps the CMMC boundary small.

Three-Year Cost Model: On-Prem vs Cloud GPUs

The crossover math is straightforward. A 4x L40S inference node costs roughly $60,000-$75,000 capitalized, plus $8,000-$12,000 per year in power, support, and spares. The equivalent cloud capacity (4x L40S-class instances running 24/7) runs $85,000-$130,000 per year on major clouds. If utilization exceeds roughly 30-40% sustained, on-prem wins within 18 months - and that is before counting egress fees or the compliance cost of processing CUI in a commercial cloud. Reserve cloud burst capacity for rare heavy fine-tuning jobs rather than sizing on-prem hardware for peak training demand.

4x L40S inference node: $60,000-$75,000 capex, breakeven vs cloud in 12-18 months at moderate utilization
4x H100 node: $250,000-$300,000 capex, justified when serving 70B models or running frequent fine-tunes
Budget 10-15% of hardware cost annually for power, cooling, warranty, and spare parts
Use cloud GPUs only for burst training jobs on non-CUI data; keep steady-state inference on-prem

How Netray Right-Sizes GPU Clusters for Manufacturers

Netray runs a two-week cluster planning engagement: we instrument your candidate workloads, replay real prompt traffic through vLLM on rented benchmark hardware, and deliver a bill of materials with measured tokens-per-second data behind every line item. Clients routinely cut planned hardware spend 30-40% versus vendor-proposed configurations because we size against observed concurrency, not theoretical peaks. We then deploy the inference stack, integrate it with SyteLine, Infor LN, or M3 workflows, and hand over runbooks. One aerospace supplier avoided a $400,000 H100 purchase after benchmarking showed a $95,000 L40S cluster met their 2-second latency SLA with 35% headroom.

Frequently Asked Questions

How many GPUs do I need to run a 70B LLM for my company?

A 70B model quantized to FP8 needs roughly 70-80 GB of VRAM for weights plus KV cache. In practice, 4x H100 80 GB serves 30-50 concurrent users at 32K context with sub-2-second first-token latency. If you can use a 14B-32B model instead - which handles most ERP and document tasks well - 2x L40S 48 GB covers the same user count at a fraction of the cost.

Is it cheaper to buy GPUs or rent them in the cloud?

For sustained workloads above roughly 30-40% utilization, buying wins. A $70,000 L40S inference node matches cloud capacity that costs $85,000-$130,000 per year, so breakeven arrives in 12-18 months and the hardware serves 4-5 years. Cloud makes sense for bursty training jobs or short experiments. For regulated manufacturers, on-prem also avoids the compliance overhead of processing CUI on commercial cloud GPUs.

Do I need InfiniBand for an AI inference cluster?

No - not for inference-only clusters. Inference traffic is API requests and responses, so standard 25/100 GbE networking is sufficient for clusters up to 4 nodes. InfiniBand or 400 Gb RoCEv2 becomes necessary only for multi-node training or tensor-parallel serving where a single model spans servers and GPUs exchange activations every layer. Skipping InfiniBand on an inference cluster saves $20,000-$50,000 in NICs and switching.

Key Takeaways

1Profile Workloads Before Picking GPUs: Cluster sizing starts with three numbers: peak concurrent users, target tokens per second per user, and the largest model you will serve. Inference-only shops serving quantized 8B-32B models can standardize on NVIDIA L40S (48 GB, ~$8,000-$11,000 per card) - excellent FP8 throughput without HBM pricing.
2Networking and Storage Decisions That Matter: For pure inference clusters up to 4 nodes, 25/100 GbE is sufficient - inference traffic is API calls, not gradient exchange. The moment multi-node training or tensor-parallel serving across nodes enters the picture, you need 400 Gb InfiniBand NDR or RoCEv2 with proper lossless configuration, which adds $3,000-$8,000 per node in NICs plus switch costs.
3Power, Cooling, and Facility Reality Checks: GPU density breaks assumptions built for typical enterprise racks. An 8x H100 HGX server draws up to 10.2 kW; three of them exceed what most manufacturing-floor server rooms deliver per rack (often 5-15 kW).

Stop guessing at GPU counts - get a benchmarked cluster plan and bill of materials from Netray before you sign a hardware PO.

Talk to an Expert See a Demo

Related Resources

AI & Automation