# The IT Director Playbook for On-Prem AI: Hardware, Models, Security, Rollout > Source: https://netray.co/resources/it-director-on-prem-ai-playbook > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: Knowledge Base ## Short answer The IT director playbook for on-prem AI: hardware sizing, model selection, security hardening, and rollout steps for running LLMs inside your firewall. ## Category ai-automation ## Content **Intro:** Running on-prem AI as an IT director means standing up four layers: GPU inference hardware, a serving stack, a model portfolio, and the security controls that make auditors comfortable. The good news is the stack has standardized. A single server with two NVIDIA L40S or RTX 6000 Ada GPUs, vLLM serving a quantized Llama 3.3 70B, an OpenAI-compatible API endpoint, and SSO-gated access covers the majority of enterprise workloads for a 200-2,000 employee manufacturer. This playbook covers sizing, model choice, hardening, and a rollout sequence that avoids the common failure of infrastructure without adoption. **Sections:** #### Hardware Sizing: What to Actually Buy **Body:** Size for concurrent users and model class, not peak theoretical demand. For most mid-market manufacturers, one inference server handles company-wide chat, RAG, and agent workloads. A dual L40S configuration (96GB combined VRAM) serves a 4-bit quantized 70B model with continuous batching to 100-300 concurrent users at acceptable latency. Smaller task models - Llama 3.1 8B or Qwen 2.5 14B for classification and extraction - run alongside on spare VRAM or a second modest GPU. **Bullets:** - Entry tier ($15K-$25K): single RTX 6000 Ada 48GB - runs 8B-32B models, good for pilots and agents - Standard tier ($35K-$50K): dual L40S server - quantized 70B models for company-wide inference - Performance tier ($250K-plus): H100 or H200 nodes - only justified for fine-tuning or very high concurrency - Always spec 2x NVMe storage headroom for model weights, embeddings, and vector indexes #### The Serving Stack and Model Portfolio **Body:** Standardize on vLLM as the inference server exposing an OpenAI-compatible API - it gives you continuous batching, paged attention, and drop-in compatibility with virtually every AI framework and client library. Ollama is fine for developer laptops but not for shared production serving. Run a small portfolio rather than one model: a 70B generalist for chat and drafting, an 8B-14B workhorse for high-volume extraction and routing, and a dedicated embedding model such as BGE-M3 for retrieval. Quantize with AWQ or GPTQ to 4-bit for the large model; quality loss is under 2 percent on enterprise tasks while VRAM drops by roughly three quarters. Version-pin everything and maintain an evaluation set of 100-300 real prompts from your business so model upgrades are gated by measured regression testing, not release-note optimism. #### Security Hardening for Compliance Environments **Body:** Treat the inference endpoint like any tier-1 internal application, then add AI-specific controls. For defense contractors, on-prem inference keeps CUI inside the CMMC 2.0 assessed boundary - but only if the surrounding controls hold up to NIST SP 800-171 scrutiny. **Bullets:** - Gate access via SSO/OIDC with group-based entitlements; no shared API keys in scripts - Log every prompt, retrieval, and agent action to your SIEM with user identity attached - Network-segment the GPU server; block all outbound internet so models cannot exfiltrate even if compromised - Apply role-based filtering at the retrieval layer so RAG never surfaces documents a user cannot already open #### Rollout Sequence and Where Netray Fits **Body:** The standard failure mode is infrastructure without adoption: a GPU server that hums at 5 percent utilization because nothing useful was built on it. Sequence rollout in 30-day phases - internal chat and document Q&A first, RAG over controlled document sets second, ERP-connected agents third - each phase with named users and measured usage targets. Netray delivers this playbook as a service: we spec and stage the hardware, deploy the hardened vLLM stack, integrate SSO and SIEM logging, and build the first ERP-connected agents against SyteLine, LN, or M3. Typical engagements reach production in 6-8 weeks with 60-plus percent weekly active usage by month three, and your team receives full runbooks and admin training so you own the stack, not us. **Cta:** Download nothing, guess nothing - book a Netray on-prem AI architecture session and get a sized bill of materials and rollout plan for your environment. **Faq:** #### What hardware do you need to run an LLM on-premises? **Answer:** For a mid-market company, a single server with two NVIDIA L40S GPUs (96GB combined VRAM, roughly $35K-$50K installed) serves a 4-bit quantized 70B model to 100-300 concurrent users via vLLM. A single RTX 6000 Ada at $15K-$25K handles 8B-32B models for pilots and agent workloads. H100-class nodes at $250K-plus are only necessary for fine-tuning at scale or very high concurrency. #### What is the best software stack for on-prem AI? **Answer:** The 2026 standard is vLLM serving quantized open-weight models (Llama 3.3 70B, Qwen 2.5) behind an OpenAI-compatible API, with SSO-gated access, SIEM logging, and a vector database for retrieval. Use AWQ or GPTQ 4-bit quantization to fit large models on mid-range GPUs. Keep Ollama for developer experimentation only. Maintain an evaluation set of real business prompts so model upgrades are regression-tested before deployment. #### Is on-prem AI compliant with CMMC 2.0? **Answer:** On-prem AI is the most defensible architecture for CMMC 2.0 because prompts and data never leave your assessed boundary. Compliance still depends on surrounding controls mapped to NIST SP 800-171: SSO-based access control, full audit logging of prompts and agent actions, network segmentation with blocked outbound traffic from the GPU server, and role-based retrieval filtering so CUI is only surfaced to authorized users. ## Related Slugs - why-on-prem-ai-is-back-2026 - defense-cio-ai-briefing-2026 - ai-governance-framework-manufacturers ## Search terms on-prem AI playbook, on-premise LLM hardware requirements, vLLM enterprise deployment, IT director AI infrastructure, self-hosted LLM security --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json