# How to set up a container registry for air-gapped Kubernetes deployments > Source: https://netray.co/troubleshoot/air-gapped-container-registry-setup > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Air-gapped Kubernetes clusters cannot reach public registries or the Hugging Face Hub, so image pulls fail at DNS resolution unless a local registry mirror is stood up ahead of time and populated from a connected staging environment. Mirror container images and model weights as two distinct pipeline steps, distribute the internal registry's CA certificate to every node, and sign mirrored artifacts so provenance, not just reachability, is auditable, which is a mandatory control in ITAR and CMMC environments. ## Error Title Setting up a container registry for air-gapped Kubernetes deployments ## Error Signature Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: no such host ## Alternate Signatures - ImagePullBackOff: rpc error: code = Unknown desc = failed to pull and unpack image - x509: certificate signed by unknown authority (private registry) ## Category containers ## Stack - docker - kubernetes - huggingface ## Affects Any air-gapped or network-restricted Kubernetes or Docker environment, mandatory in ITAR and CMMC-regulated deployments ## Quick Fix ### Get a working air-gapped pull path **Steps:** - Stand up an internal registry (Harbor or a private Docker Registry) reachable from every node inside the secure enclave. - From a connected staging environment, pull, tag, and push every required container image into the internal registry. - Separately, download required model weights from the Hugging Face Hub and sync them into internal object storage or an OCI artifact store. - Distribute the internal registry's CA certificate to every node and add it to the container runtime's trust store. - Repoint deployment manifests to the internal registry address and set HF_HUB_OFFLINE=1 for weight loading. - Sign mirrored images with cosign and enforce signature verification at admission before allowing them to deploy. ## Symptoms - dial tcp: lookup registry-1.docker.io: no such host on every pull attempt - ImagePullBackOff on all nodes for any public image reference - x509: certificate signed by unknown authority when pulling from the internal registry - containers start but fail trying to reach huggingface.co for model weights ## Causes ### No local registry mirror exists, and nodes have no path to the public internet at all **Likelihood:** most-common **Explanation:** Air-gapped networks by design have no route to public registries like Docker Hub, ghcr.io, or the Hugging Face Hub, so any image pull or model download that assumes internet access fails at DNS resolution before it ever reaches an authentication or compatibility problem; this is expected behavior, not a misconfiguration, and the deployment needs its own internal registry populated ahead of time. **Fix:** Stand up an internal registry, such as Harbor or a private Docker Registry mirrored into the secure enclave, mirror every required image and model artifact into it from a connected staging environment, and repoint all deployment manifests to pull from the internal registry address. **Commands:** - docker pull myimage:tag - docker tag myimage:tag internal-registry.local/myimage:tag - docker push internal-registry.local/myimage:tag ### Model weights were never mirrored alongside container images, only the images themselves **Likelihood:** common **Explanation:** Teams frequently mirror container images into an internal registry but forget that model weights are a separate artifact entirely, typically downloaded from the Hugging Face Hub at container startup; without a plan to mirror those weight files into internal object storage as well, the image pulls fine but the container then fails trying to reach a hub it cannot route to. **Fix:** Treat model weights as a first-class artifact in your mirroring pipeline, downloading them once in a connected staging environment and pushing them into internal object storage or a registry-compatible OCI artifact store, then configuring containers to load from that internal source with HF_HUB_OFFLINE=1. **Commands:** - huggingface-cli download --local-dir /staging/models/ - aws s3 sync /staging/models/ s3://internal-bucket/models/ ### Private registry uses a certificate the container runtime does not trust **Likelihood:** occasional **Explanation:** Internal registries commonly use certificates signed by an internal certificate authority rather than a public one, and unless that CA's certificate is explicitly added to the trust store of every node's container runtime, pulls fail with an x509 unknown authority error even though the registry itself is reachable and otherwise healthy. **Fix:** Distribute the internal CA certificate to every node and configure the container runtime to trust it, either by adding it to the OS trust store or via runtime-specific registry certificate configuration. **Commands:** - sudo cp internal-ca.crt /usr/local/share/ca-certificates/ - sudo update-ca-certificates - sudo systemctl restart containerd ### No pull-through cache or signing policy, so image provenance cannot be verified after mirroring **Likelihood:** occasional **Explanation:** Simply copying images into an internal registry without a signing and verification policy means there is no cryptographic guarantee that what got mirrored matches the original upstream artifact, which is a specific problem in ITAR and CMMC environments where supply chain integrity of every deployed artifact must be demonstrable, not just its availability. **Fix:** Sign images during the mirroring process with a tool like cosign, and configure admission control, such as a Kubernetes admission webhook, to reject unsigned images at deploy time, closing the provenance gap rather than only closing the network gap. **Commands:** - cosign sign --key cosign.key internal-registry.local/myimage:tag - cosign verify --key cosign.pub internal-registry.local/myimage:tag ## Diagnostics ### Confirm nodes truly have no external route **Command:** curl -v https://registry-1.docker.io/v2/ **Interpretation:** A DNS resolution or connection timeout confirms genuine air-gap; a certificate or auth error instead means there is a route but a trust or credential problem. ### Check whether the internal registry is reachable and healthy **Command:** curl -v https://internal-registry.local/v2/_catalog **Interpretation:** A 200 response with a repository list confirms the registry itself works; failures here point to the registry, not the air-gap. ### Check the container runtime's configured registry mirrors **Command:** cat /etc/containerd/config.toml | grep -A5 registry **Interpretation:** Confirms whether nodes are actually configured to redirect pulls to the internal registry rather than the public default. ### Verify the internal CA is trusted by the runtime **Command:** openssl s_client -connect internal-registry.local:443 -showcerts **Interpretation:** Certificate verify errors here confirm the CA trust chain is the specific blocker, not network reachability. ## Prevention - Build the internal registry and mirroring pipeline before any application deployment begins, not reactively after the first failed pull - Mirror model weights as a distinct pipeline step alongside container images, never assume they travel together - Distribute internal CA certificates to every node's container runtime trust store as part of node provisioning - Sign every mirrored artifact and enforce signature verification at admission, since air-gapping solves reachability but not provenance ## When To Escalate Air-gapped registry and artifact mirroring is not a one-time script; in ITAR, CMMC, or similar regulated environments it is a mandatory, auditable pipeline covering images, model weights, and their provenance, and designing that pipeline correctly from the start is exactly the kind of architecture work worth bringing in outside expertise for rather than improvising per deployment. ## Faq ### Why is a local registry mandatory in ITAR or CMMC environments, not just convenient? **Answer:** ITAR and CMMC compliance require that controlled technical data and the software processing it never traverse networks or infrastructure outside the approved boundary, and pulling images or model weights directly from public registries or the Hugging Face Hub would mean data flows in and out of that boundary in ways that cannot be audited or controlled. A local, mirrored registry inside the secure enclave is how you maintain a fully auditable, air-gapped supply chain for every artifact the deployment depends on. ### Do I need to mirror model weights separately from container images? **Answer:** Yes. Container images and model weights are different artifact types with different typical sources; images usually come from registries like Docker Hub or a private registry, while weights are commonly pulled from the Hugging Face Hub at runtime. An air-gapped pipeline needs to explicitly mirror both into the secure enclave, since mirroring only the container images still leaves the application trying to reach the public Hugging Face Hub for weights and failing. ### What does the x509 certificate signed by unknown authority error mean for a private registry? **Answer:** It means your container runtime does not trust the certificate authority that signed your internal registry's TLS certificate, which is common when organizations use an internal CA rather than a public one. The registry itself is reachable and working; the fix is distributing the internal CA's certificate to every node and adding it to the container runtime's trust store, not changing anything about the registry configuration itself. ### Is image signing actually necessary if the registry is already air-gapped? **Answer:** Yes, they solve different problems. Air-gapping controls where an artifact can be reached from, but it says nothing about whether the artifact that got mirrored into your internal registry is actually the exact, untampered artifact you intended to mirror. Signing images with a tool like cosign and verifying signatures at deployment time closes that provenance gap, which is specifically what frameworks like CMMC expect you to demonstrate for supply chain integrity. ## Search terms air-gapped container registry setup, kubernetes air gapped image pull, mirror docker images air gapped network, private registry x509 unknown authority, huggingface offline air gapped model weights, itar cmmc container registry compliance --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json