AI & Automation5 min readNetray Engineering TeamUpdated March 4, 2025

On-Prem LLM Deployment Architecture: The Complete Reference Guide

An on-prem LLM deployment architecture is the stack of GPU compute, inference serving, retrieval infrastructure, and security controls needed to run large language models entirely inside your own data center. For aerospace, defense, and discrete manufacturers bound by ITAR, DFARS 252.204-7012, or CMMC 2.0, keeping prompts and model weights off third-party clouds is often not a preference but a compliance requirement. This guide lays out a proven reference architecture - from the inference layer through RAG, orchestration, and network segmentation - so your IT team can plan hardware, software, and security zones before spending a dollar on GPUs.

The Four Layers of an On-Prem LLM Stack

A production on-prem LLM architecture separates cleanly into four layers. The compute layer holds GPU servers - typically 2-8 nodes with NVIDIA L40S, A100, or H100 cards. The serving layer runs an inference engine such as vLLM 0.6+, NVIDIA Triton, or TGI, exposing an OpenAI-compatible API behind a load balancer. The retrieval layer pairs a vector database (Qdrant, Milvus, or pgvector) with an embedding service and document ingestion pipeline. The application layer hosts agents, chat UIs, and ERP integrations that call the serving layer. Keeping these layers on separate VLANs with explicit firewall rules is what makes the architecture auditable under NIST SP 800-171 controls.

Compute layer: GPU nodes sized for model memory - a 70B model at FP8 needs roughly 80 GB of VRAM plus KV cache headroom
Serving layer: vLLM or Triton behind an internal load balancer with an OpenAI-compatible /v1/chat/completions endpoint
Retrieval layer: vector database, embedding model (e.g., BGE-M3 or nomic-embed), and a document chunking pipeline
Application layer: agent orchestration, ERP connectors to SyteLine or Infor LN, and role-scoped user interfaces

Sizing Hardware: From Pilot to Production

Most manufacturers overspend on their first cluster. A realistic pilot serving 50-100 concurrent users on a quantized 8B-14B model runs comfortably on a single server with 2x NVIDIA L40S (48 GB each), which lands around $35,000-$50,000 fully configured. Production deployments serving a 70B-class model with sub-2-second first-token latency typically need 4x H100 80 GB per inference node, pushing per-node cost to $250,000-$300,000. Plan 20-30% headroom for KV cache growth as context windows expand from 8K to 32K tokens in real workloads. Power and cooling matter: an 8x H100 HGX node draws up to 10 kW, so verify rack power budgets and CRAC capacity before ordering hardware, not after.

RAG Pipeline Design for ERP and Engineering Data

Retrieval-augmented generation is where on-prem LLMs earn their keep in manufacturing. The pipeline ingests ERP records, quality documents, AS9100D procedures, and engineering drawings metadata; chunks them at 256-512 tokens with overlap; embeds them; and stores vectors with rich metadata filters. The critical design decision is metadata-first filtering: queries against SyteLine item masters or Infor LN production orders should filter by site, item class, and effectivity date before vector similarity ever runs. This keeps retrieval precision above 85% in practice and prevents cross-site data leakage between divisions - essential when one plant handles ITAR-controlled programs and another does not.

Security Zones and Compliance Boundaries

Regulated manufacturers should treat the LLM stack as a CUI enclave. Place GPU and serving nodes in a dedicated security zone with no outbound internet access, pull model weights through a scanned staging host, and log every prompt and completion to an append-only store for audit. Map controls directly to CMMC 2.0 practices: AC.L2-3.1.3 for CUI flow enforcement, AU.L2-3.3.1 for audit logging, and SC.L2-3.13.1 for boundary protection. Air-gapped variants of this architecture replace the staging host with removable-media transfer procedures governed by a two-person rule.

Isolate inference nodes on a dedicated VLAN with deny-by-default egress and no direct internet route
Log all prompts, completions, and retrieval hits to immutable storage with 1-year minimum retention
Verify model weight checksums (SHA-256) against publisher hashes before loading into the serving layer
Map each architecture control to a specific CMMC 2.0 or NIST SP 800-171 practice in your SSP

How Netray Designs and Delivers On-Prem LLM Architectures

Netray delivers this reference architecture as a fixed-scope engagement: hardware bill of materials, network design, inference stack deployment, RAG pipeline wired to your ERP, and a CMMC-mapped control matrix. Our AI agents then run on the stack - automating SyteLine data entry, quoting, and quality documentation. Typical results: a 6-8 week pilot-to-production timeline, inference cost per query 60-80% below API pricing at equivalent volume, and an architecture that passes C3PAO assessment scrutiny because every control is documented from day one. We size clusters against your actual concurrency data, not vendor spreadsheets, which routinely cuts initial hardware spend by 30-40%.

Frequently Asked Questions

What hardware do I need to run an LLM on-premises?

For a pilot serving 50-100 users on an 8B-14B quantized model, one server with 2x NVIDIA L40S 48 GB GPUs (roughly $35,000-$50,000) is sufficient. Production 70B-class deployments typically require 4x H100 80 GB per inference node at $250,000-$300,000 per node. Add 20-30% VRAM headroom for KV cache as context lengths grow, and verify rack power - dense GPU nodes draw 5-10 kW each.

Is on-prem LLM deployment required for ITAR compliance?

ITAR does not mandate on-prem deployment, but it prohibits foreign-person access to technical data, which most public AI APIs cannot guarantee. On-prem deployment is the most defensible pattern because prompts, retrieved documents, and model weights never leave your controlled environment. Some manufacturers use FedRAMP High or AWS GovCloud alternatives, but on-prem remains the simplest architecture to document in an ITAR compliance program.

How long does an on-prem LLM deployment take?

With hardware in hand, a competent team can stand up inference serving and a basic RAG pipeline in 2-3 weeks. Realistic end-to-end timelines - procurement, network segmentation, security review, ERP integration, and user rollout - run 8-16 weeks for a first production use case. GPU lead times are the usual bottleneck: H100 systems can take 4-12 weeks to arrive, so order hardware early.

Key Takeaways

1The Four Layers of an On-Prem LLM Stack: A production on-prem LLM architecture separates cleanly into four layers. The compute layer holds GPU servers - typically 2-8 nodes with NVIDIA L40S, A100, or H100 cards.
2Sizing Hardware: From Pilot to Production: Most manufacturers overspend on their first cluster. A realistic pilot serving 50-100 concurrent users on a quantized 8B-14B model runs comfortably on a single server with 2x NVIDIA L40S (48 GB each), which lands around $35,000-$50,000 fully configured.
3RAG Pipeline Design for ERP and Engineering Data: Retrieval-augmented generation is where on-prem LLMs earn their keep in manufacturing. The pipeline ingests ERP records, quality documents, AS9100D procedures, and engineering drawings metadata; chunks them at 256-512 tokens with overlap; embeds them; and stores vectors with rich metadata filters.

Get a reference architecture and hardware bill of materials tailored to your plant - book an on-prem LLM architecture review with Netray.

Talk to an Expert See a Demo

Related Resources

AI & Automation