Air-Gapped Model Update and Patching Workflows
Updating a model behind an air gap is not a smaller version of a normal deployment pipeline, it is a different discipline built around one constraint: nothing crosses the boundary without a documented, verifiable chain of custody. A defense manufacturer running Llama 3.3 70B or Qwen3 on an isolated network cannot pull a new checkpoint from a public registry at 2 a.m. the way a cloud-connected team can. Every update, whether a quantized weight refresh, a vLLM version bump, or a security patch to the inference server, has to travel through a controlled transfer process, get verified against a known-good hash, and roll out in stages that catch a bad artifact before it reaches production. Skip any one of those steps and you either stall your AI program behind manual review for months, or you introduce an unverified binary into a network that was air-gapped specifically to keep unverified binaries out.
Why Air-Gapped Updates Need Their Own Process
Connected environments patch on a cadence measured in days because a vulnerability scanner flags a CVE and a package manager pulls the fix automatically. Air-gapped environments patch on a cadence measured in weeks or months because every artifact has to be requested, reviewed, transferred, and verified by a person with the authority to do so. That lag is not a flaw to engineer away, it is the tradeoff you accepted when you chose isolation over connectivity. The mistake most programs make is treating the lag as an excuse to skip process rigor, when it should be the opposite: because you cannot patch quickly, you need a documented procedure that makes every patch you do apply defensible to an auditor and safe to roll back.
- Expect a 2 to 6 week request-to-production cycle for a routine model or inference-server patch
- Track known CVEs against your deployed stack even when you cannot patch immediately, so exposure is documented
- Assign a named patch owner who requests, verifies, and signs off on every artifact crossing the boundary
- Maintain a patch calendar separate from your connected-environment cadence, reviewed monthly
Building the Secure Transfer Procedure
The transfer procedure is the part auditors care about most, because it is where an attacker or a mistake most easily introduces something you did not intend. Use write-once media or a vetted data diode rather than reusable USB drives, and scan every artifact with an offline malware scanner on a dedicated transfer workstation that never touches the isolated network directly. Require a documented manifest listing every file, its source, its hash, and the person who requested it, signed before the media crosses the boundary. Treat model weights, tokenizer files, configuration, and serving-engine binaries as separate artifacts each requiring their own verification, not one bundled zip file nobody actually inspects.
- Dedicated transfer workstation, air-gapped from both networks except during a scanning window
- Write-once media or a one-way data diode instead of reusable removable drives
- Signed manifest listing every artifact, source, hash, and requesting individual before transfer
- Separate verification per artifact type: weights, tokenizer, configs, and serving binaries individually checked
Hash Verification and Provenance Chains
A hash check only proves the file was not corrupted or altered after the hash was recorded, so provenance matters as much as the hash itself. Capture the published SHA256 or SHA512 hash from the model publisher's own signed release notes before the artifact ever leaves a connected network, not after it arrives at the boundary where a compromised source could hand you a matching hash for a tampered file. Where the publisher supports cryptographic signing, verify the signature against a known public key, not just a checksum. Log every verification result with a timestamp and the verifying operator, and reject any artifact that fails, no exceptions, even under deployment deadline pressure.
- Capture publisher hashes from the original signed source before transfer, never after
- Verify cryptographic signatures where available, not just checksums, against a known public key
- Log every verification attempt, pass or fail, with timestamp and operator identity
- Zero-exception rejection policy for any artifact that fails verification, regardless of schedule pressure
Staged Rollout: Canary to Production
Never promote a verified artifact directly to every production node. Stand up a canary node that mirrors production hardware and software versions, deploy the update there first, and rerun your golden evaluation set against it before touching anything users depend on. Compare accuracy, latency, and output format against the previous version, because a quantization change or serving-engine upgrade can silently shift behavior even when the model weights are identical. Only after the canary passes should the update move to a limited production ring, then full rollout, with each stage separated by enough time to observe real usage rather than synthetic tests alone.
- Canary node matching production hardware, software version, and quantization settings exactly
- Full golden evaluation suite rerun on canary before any production promotion decision
- Limited production ring for 48 to 72 hours before expanding to all nodes
- Explicit go or no-go decision recorded by the patch owner at each stage, not an automatic timer
Rollback and Version Pinning
Pin the exact model version, quantization method, and serving-engine build in your deployment configuration, and never track a floating latest tag inside an air-gapped environment where you cannot audit what changed upstream. Keep the previous two known-good versions available on production storage at all times so a rollback is a configuration change, not an emergency re-transfer across the boundary. Rehearse the rollback procedure at least once per quarter on the canary node so the first real rollback is not also the first time anyone has run the command. Document the rollback trigger conditions in advance: a measurable accuracy drop, an error-rate spike, or a security finding are the three that should authorize an immediate revert without waiting for a change board.
How Netray Runs Air-Gapped Update Programs
Netray builds the transfer procedure, hash verification chain, and staged rollout pipeline as part of the initial deployment for aerospace and defense clients, so the update process exists before the first production patch is ever needed rather than being improvised under pressure. We maintain a documented patch calendar against your deployed model and serving stack, track known CVEs even when immediate patching is not possible, and rehearse rollback on your canary hardware quarterly. For clients running Llama, Qwen3, or gpt-oss models inside a CMMC-assessed enclave, every artifact that crosses your boundary carries a signed verification record our team and yours can both produce on demand during an audit.
Frequently Asked Questions
How often should air-gapped AI models be updated?
There is no universal cadence, but most defense manufacturers land on a monthly review cycle: check for published CVEs and model or serving-engine updates monthly, and execute a transfer only when there is a documented security or accuracy reason to do so. Patching for its own sake inside an air gap adds risk exposure during every transfer without a corresponding benefit, so tie each update to a specific, recorded justification.
What is a data diode and do we need one?
A data diode is hardware that physically enforces one-way data flow, typically used to move verified artifacts into a higher-security network without any possible path back out. It is not required for every air-gapped deployment, but it is common in CUI and ITAR-boundary environments where the cost of an accidental or malicious reverse data flow is unacceptable. Write-once media with a strict manifest process is an acceptable lighter-weight alternative for many manufacturing environments.
How do you verify a model checkpoint has not been tampered with?
Capture the publisher's official hash and, where available, cryptographic signature from a trusted connected source before the artifact ever approaches your air gap. Verify that hash and signature on your dedicated transfer workstation against a known public key, log the result with a timestamp and operator identity, and reject the artifact outright on any mismatch. A hash captured after the file has already crossed an untrusted path proves nothing.
Can quantized model updates skip staged rollout?
No. A quantization change, even to a model whose full-precision weights are unchanged, can shift output behavior, latency, and memory footprint enough to matter in production. Always run quantized updates through a canary node with your full evaluation suite before promoting them, and treat a quantization method change with the same rigor as a full model version upgrade rather than as a routine configuration tweak.
Key Takeaways
- 1Why Air-Gapped Updates Need Their Own Process: Connected environments patch on a cadence measured in days because a vulnerability scanner flags a CVE and a package manager pulls the fix automatically. Air-gapped environments patch on a cadence measured in weeks or months because every artifact has to be requested, reviewed, transferred, and verified by a person with the authority to do so.
- 2Building the Secure Transfer Procedure: The transfer procedure is the part auditors care about most, because it is where an attacker or a mistake most easily introduces something you did not intend. Use write-once media or a vetted data diode rather than reusable USB drives, and scan every artifact with an offline malware scanner on a dedicated transfer workstation that never touches the isolated network directly.
- 3Hash Verification and Provenance Chains: A hash check only proves the file was not corrupted or altered after the hash was recorded, so provenance matters as much as the hash itself. Capture the published SHA256 or SHA512 hash from the model publisher's own signed release notes before the artifact ever leaves a connected network, not after it arrives at the boundary where a compromised source could hand you a matching hash for a tampered file.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Air-Gapped LLM Deployment Checklist
A practical control checklist for deploying and maintaining large language models in a fully air-gapped environment, from initial staging through ongoing patching and drift detection.
Free ToolOn-Prem AI Security Hardening Checklist
A practical control checklist for securing self-hosted language models, covering model provenance, network isolation, data governance, host hardening, and audit readiness.
Free ToolAir-Gapped AI Readiness Assessment
A 10-question assessment that scores how prepared your organization is to deploy and operate LLMs inside an air-gapped or classified enclave.
Terms used in this article
Need a documented patching workflow for an air-gapped AI deployment? Netray will build the transfer, verification, and rollout process before your next update is due.
Related Resources
Securing Model Weights in the Enterprise
Secure model weights end to end: custody controls, encryption at rest, access policies, and exfiltration prevention for regulated AI deployments.
AI & AutomationITAR and CMMC Handling of AI Workloads
How ITAR and CMMC apply to AI workloads: technical data boundaries, CUI handling, assessed environments, and where on-prem AI is the only option.
AI & AutomationAir-Gapped LLM Deployment Patterns That Actually Work
Air-gapped LLM deployment patterns that work: offline model transfer, update workflows, monitoring without telemetry, and CMMC-ready architectures.