The Deployment Landscape: On-Prem, IL5, IL6, and True Air Gap
There are three distinct scenarios that people conflate. First, connected-but-controlled: CUI enclaves at defense contractors, governed by CMMC and NIST SP 800-171, where on-prem inference with tightly restricted egress suffices. Second, classified cloud: DoD Impact Level 5 (CUI and national security systems) and IL6 (Secret) regions, where Azure OpenAI variants and AWS Secret Region services now hold authorizations, useful for programs already operating there.
Third, true air gap: SCIFs and closed research networks with zero external connectivity, where everything, model weights, serving software, GPU drivers, embedding models, must arrive via approved media transfer under the facility's cross-domain and media control procedures. Architecture and cost differ sharply across the three; scoping the wrong one wastes months.
Hardware Sizing Without Cloud Elasticity
Air-gapped deployments cannot burst to the cloud, so size for peak concurrency up front. Rules of thumb from production deployments: a 4-bit-quantized 70B model needs roughly 48-64 GB of VRAM and serves 5-15 concurrent users acceptably on 2x NVIDIA L40S; an 8B model runs on a single 24 GB card and handles extraction and classification at high throughput.
- Pilot tier: one server, 2x L40S (96 GB VRAM total), 70B quantized model, 5-15 users, roughly $35,000-$50,000
- Production tier: 2 nodes with 4x L40S or 2x H100 each for redundancy and 30-60 users, $120,000-$250,000
- Add 10-20 percent VRAM headroom for embedding models, rerankers, and KV-cache growth at long context
- Plan physical logistics early: power (2-4 kW per GPU node), cooling, and rack space inside the accredited facility
Getting Models and Updates Across the Air Gap
Model weights are large (a 70B model is 40-140 GB depending on quantization) and update cadence matters, so define the transfer pipeline as a first-class process, not an afterthought. The defensible pattern mirrors existing software supply chain controls: download on a connected staging system, verify checksums against publisher-signed hashes, scan per your media protection procedures, then transfer via approved one-way media or data diode.
- Verify SHA-256 hashes of weight files against the publisher's signed manifest before and after transfer
- Mirror the full dependency chain offline: vLLM, CUDA, drivers, and Python packages via an internal repository
- Batch updates quarterly; chasing every model release across an air gap burns accreditation goodwill
- Document the pipeline in your SSP or accreditation package as part of configuration and media management
Try it here, free
Air-Gapped LLM Deployment Checklist
A practical control checklist for deploying and maintaining large language models in a fully air-gapped environment, from initial staging through ongoing patching and drift detection.
Accreditation: What Your ISSM and AO Will Ask
Whether the authority is a C3PAO, an ISSM under NISPOM (32 CFR Part 117), or a government Authorizing Official running RMF, the questions converge. Where do prompts and outputs persist, and at what classification? How is the model prevented from becoming an uncontrolled aggregation point, since retrieval indexes concentrate data that was previously compartmented?
Who audits prompt logs, and how are they marked? Answer these by design: enforce source-document permissions in the retrieval layer so the model cannot surface content beyond a user's authorization, store logs at the enclave's classification with standard marking, and disable any telemetry or phone-home behavior in serving software. Deployments that treat the LLM as just another accredited application, with an SBOM, baseline configuration, and audit story, get approved; those pitched as magic boxes stall.
How Netray Supports Air-Gapped AI Programs
Netray delivers turnkey air-gapped AI stacks for defense manufacturers and cleared facilities: GPU hardware specification, offline installation media with hash-verified model weights and a fully mirrored dependency chain, permission-aware retrieval over ERP and document data, and the accreditation artifacts, SBOM, configuration baseline, data flow diagrams, and audit logging design, your ISSM or AO needs. Our stacks run SyteLine, Infor LN, and document intelligence workloads with zero external connectivity and quarterly offline update packages. Clients have taken Netray-built systems through NISPOM and CMMC reviews without AI-related findings, while giving engineers natural-language search over decades of controlled technical data for the first time.
Frequently asked questions
Can you run a large language model completely offline?
Yes. Open-weight models such as Llama 3.1 70B or Qwen 2.5 72B run fully offline on local GPU servers using serving software like vLLM, with no internet dependency at inference time. You download weights once on a connected staging system, verify hashes, transfer via approved media, and mirror all software dependencies internally. A quantized 70B model serves 5-15 concurrent users on roughly $35,000-$50,000 of GPU hardware.
What DoD impact level is required for AI on classified data?
Secret-level workloads require IL6-authorized environments, such as Azure Government Secret or AWS Secret Region, or accredited on-prem systems within a cleared facility under RMF. CUI and national security systems below Secret map to IL5. For contractor-owned air-gapped networks, the governing framework is NISPOM (32 CFR Part 117) with the facility ISSM and the cognizant Authorizing Official approving the system.
How do you update AI models in an air-gapped environment?
Through a controlled offline transfer pipeline: download weights and dependencies on a connected staging system, verify SHA-256 hashes against publisher-signed manifests, scan per media protection procedures, then move them via approved one-way media or a data diode. Production teams batch updates quarterly rather than chasing each release, and document the pipeline in the SSP or accreditation package as part of configuration management.
Planning AI inside a closed or classified network? Netray delivers the full offline stack plus the accreditation package.