AI Infrastructure
The rack that replaces the API bill.
On-prem AI infrastructure is normal infrastructure with sharper choices: which GPUs, what interconnect, how the serving layer squeezes them, and what keeps it observable at 2am. NetRay designs it, stands it up, and leaves your team able to run it.
Right-sizing beats future-proofing
The most common infrastructure mistake is buying for imagined year-three scale: a cluster sized for a workload that was never benchmarked, depreciating while the pilot runs on 10% of it. Our design rule is the opposite: size for the benchmarked workload plus honest growth headroom, on hardware that racks into what you already operate.
For most first deployments that means one to four rack units: a single GPU server for a departmental assistant, or a small node group for org-wide serving plus a fine-tuning lane. Scale-out is a procurement, not a redesign, when the serving layer is built right from day one.
The choices that actually matter
GPU memory is the binding constraint (it decides which models fit and at what quantization), so we spec memory first, compute second. After that:
- GPU class: datacenter cards for sustained multi-user serving; workstation-class cards are legitimate for pilots and small teams
- Interconnect: only multi-GPU model sharding needs the fast lanes; single-model-per-GPU serving does not, and skipping it saves real money
- Storage: fast local NVMe for model weights and KV caches; your existing SAN/NAS for corpora and indexes
- Networking: standard 10/25GbE covers most inference traffic; the exotic fabrics are for training clusters you probably don't need yet
- Power and cooling: a 4-GPU node is an ordinary server-room load, planned, not discovered
The software layer is where the performance lives
Identical hardware varies 3 to 5x in delivered throughput depending on the serving stack. Continuous batching, paged attention, quantization strategy, and context-cache tuning in vLLM or TensorRT-LLM are the difference between one GPU serving forty users comfortably and three GPUs struggling with the same load. This tuning is where our deployments earn their hardware budget back.
Operations: boring by design
The finished system plugs into what you already run: metrics to your monitoring stack, logs to your SIEM, deployments through your existing pipeline, and a model registry with signed artifacts and staged promotion. We document runbooks for the six things that will actually happen (model update, GPU failure, quality regression report, capacity increase, security patch, full restore) and train your team on each. Infrastructure you can't hand off is a hostage situation, not a deliverable.
Straight answers
Which GPUs should we buy?
The ones the benchmark says, in the memory size the model demands, from the vendor lane your procurement supports. Datacenter-class cards for sustained serving, workstation-class for pilots; specific SKUs shift quarter to quarter, which is why we spec from a live benchmark rather than publishing a shopping list that ages in months.
Can this run in our existing virtualization environment?
Frequently, yes: GPU passthrough or vGPU on your existing hypervisor cluster is a legitimate serving platform for small and mid-size deployments, and it keeps operations inside tooling your team already knows. Bare metal wins at higher scale.
What about training infrastructure vs inference?
Different problems. Inference wants memory and throughput per watt; training wants raw compute and fast interconnect. Most enterprises need serious inference plus a modest fine-tuning lane (LoRA runs happily on a single node), not a training cluster. We design for that reality.
How do we avoid buying hardware that's obsolete in a year?
By buying for benchmarked workloads with modest headroom instead of speculation, and by keeping the serving layer portable so next year's GPUs slot in beside this year's. Hardware refresh becomes routine capacity planning, not a rebuild.
Spec the rack from a benchmark, not a brochure.
Workload in, bill of materials out: benchmarked model choices, hardware spec, serving design, and the three-year TCO.
Design my AI infrastructureFree AI workshops available. A founder reads every message.