Why a model checkpoint is corrupted, and how to verify and fix it
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
Also appears as
- _pickle.UnpicklingError: invalid load key
- RuntimeError: storage has wrong size, expected ... got ...
- Model loads without error but produces nonsensical output on every input
Short answer
A corrupted checkpoint means the bytes on disk do not match the original artifact the model author published, whether from an interrupted download, a bad copy between systems, disk-level bit rot, or a failed write during a save operation. There is no reliable way to repair a corrupted deep learning checkpoint; the fix is always to verify the file against a known-good hash or size and re-obtain a clean copy, then build a verification step into your pipeline so the same failure does not silently recur.
Affects: Any checkpoint format (PyTorch .bin/.pt, safetensors, GGUF) transferred, cached, or stored across an unreliable path; especially relevant for air-gapped transfers
Fix it in a few minutes
- 1Compare the file's size in bytes against the size listed on the source (HuggingFace repo page or your internal artifact registry).
- 2If a checksum (sha256) is published for the artifact, compute and compare it: sha256sum <file>.
- 3If sizes or hashes do not match, delete the local file entirely and re-download or re-copy it from the original source rather than attempting to patch it.
- 4For multi-shard models, check every shard individually, since only one shard among many can be corrupted while the rest load fine.
- 5After obtaining a clean copy, re-run your standard load-and-smoke-test step before treating the model as ready to serve.
How to confirm this is your problem
- Loading throws a low-level deserialization error (zip archive, pickle, or safetensors header error) rather than a model-specific error
- The model loads without any exception but produces completely nonsensical output that does not resemble a real language model's failure modes
- The failure is specific to one file among several shards, or specific to one copy of the model on one machine but not another
- File size on disk is smaller than, or subtly different from, the size reported at the original source
Root causes and fixes
File was only partially downloaded or partially copied before being treated as complete
An interrupted network transfer, a script that does not check download completion before proceeding, or a copy operation killed partway through all leave a file that is present on disk but structurally incomplete. Binary checkpoint formats (pickle-based .bin/.pt, zip-based PyTorch archives, safetensors) all require the complete byte stream to parse correctly; a truncated file fails at whatever point the reader runs out of expected bytes.
Fix: Always verify file size (and ideally checksum) against the source immediately after any download or copy operation, before marking that transfer as successful in your pipeline, and re-transfer if there is any mismatch.
ls -la model.bin sha256sum model.bin
Bit rot, disk errors, or filesystem corruption altered bytes on storage over time
Long-lived storage, especially on aging disks, degraded RAID arrays, or filesystems without integrity checking (no checksumming filesystem, no ECC memory), can silently flip bits over months or years of storage. A checkpoint that loaded fine when first downloaded can fail later purely from storage degradation, with no code or environment change involved at all.
Fix: Store critical model artifacts on storage with integrity guarantees where possible (checksumming filesystems, RAID with scrubbing, cloud object storage with built-in integrity verification), and periodically re-verify archived checkpoints' checksums against your original record.
sha256sum -c checksums.sha256
Transfer between an internet-connected staging host and an air-gapped target altered or truncated the file
Manual transfer methods used to move artifacts across an air gap (certain removable media formats, some file transfer utilities with size limits, network shares with unusual encoding settings) can truncate or subtly alter binary content in ways that are not obvious until the model is loaded and fails or misbehaves on the target side.
Fix: Use a transfer method proven for large binary files (tar archives with an accompanying checksum manifest) and verify the checksum on the receiving side immediately after transfer, before deleting the source copy.
tar -cf model-bundle.tar model/ && sha256sum model-bundle.tar
A save or export operation was interrupted mid-write (out of disk space, process killed, power loss)
When a training or export job writes a checkpoint and is killed by an out-of-memory reaper, hits a full disk, or loses power partway through torch.save or safe_save, the resulting file can be left in a partially-written state that looks present but is structurally incomplete, identical in symptom to a truncated download.
Fix: Write checkpoints to a temporary filename and atomically rename to the final name only after the write completes successfully, and always confirm free disk space before starting a large checkpoint save.
df -h
Wrong file entirely was referenced (stale cache pointing at an old or unrelated file with a coincidentally similar name)
In pipelines with aggressive local caching or symlink-based model directories, a stale cache entry or dangling symlink can point at leftover data from a previous, unrelated experiment, producing load failures or bizarre output that has nothing to do with the current model's actual integrity.
Fix: Verify the exact path being loaded resolves to the file you expect, especially in setups using symlinks or content-addressed caches, and clear stale cache directories when switching between model versions.
readlink -f model.bin
Diagnostic commands
Verify file size against the published source
ls -la model.safetensors
Compare directly against the byte count shown on the HuggingFace repo page or your internal artifact registry's manifest. Any difference, even by a few bytes, confirms corruption or truncation.
Compute and compare a checksum
sha256sum model.safetensors
Match this against a published or previously recorded hash. A mismatch is definitive proof of corruption regardless of how similar the file size appears.
Attempt a minimal load in isolation
python -c "from safetensors.torch import load_file; sd=load_file('model.safetensors'); print(len(sd))"A clean tensor count confirms the file structure is intact. An exception here reproduces the corruption independent of the full model-loading pipeline, narrowing the problem to the file itself.
Stopping it from happening again
- Publish and store checksums (sha256) alongside every model artifact in your internal registry, and verify them automatically as part of any download, copy, or transfer step.
- Use atomic write patterns (write to temp file, rename on success) for any process that saves checkpoints, so a killed process never leaves a half-written file mistaken for a complete one.
- For air-gapped transfers, standardize on a single verified transfer method (checksummed tar bundle) rather than ad hoc file copying.
- Periodically re-verify long-term archived checkpoints against their original checksums to catch storage-level bit rot before it is discovered at deployment time.
When this becomes an architecture problem
If corruption keeps recurring across your model supply chain rather than being an isolated incident, the fix is a proper artifact integrity process end to end (checksum at source, checksum at every hop, checksum before serving) rather than re-downloading and hoping each time. This is a supply-chain control worth designing deliberately, especially for regulated environments where model provenance and integrity are audit requirements.
Frequently asked questions
Can I repair a corrupted checkpoint file instead of re-downloading it?
No, deep learning checkpoint formats (pickle, zip-based PyTorch archives, safetensors, GGUF) are not designed to be partially repaired. The only reliable fix is obtaining a complete, verified copy of the file from a trusted source.
How do I know if it is corruption versus a genuinely different but valid model?
Compare the file's checksum against the value published by the original source (HuggingFace model card, your internal registry's manifest at time of ingestion). A checksum match with load failure suggests a code or environment bug rather than corruption; a mismatch confirms the file itself is the problem.
Why did the model load without error but produce nonsensical output?
Not all corruption produces a hard parsing error. If enough of the file structure is intact for the loader to succeed but the underlying tensor bytes are altered (partial truncation padded with zeros, a bit-flipped region within otherwise valid data), the model can load 'successfully' while its weights are silently wrong, producing degraded or nonsensical output as the visible symptom.
Should I keep a local backup of every model I download?
For any model in active production use, yes, keep a verified, checksummed copy in your own internal storage rather than relying on being able to re-download it on demand, especially important for air-gapped or regulated environments where the original source may not always be reachable later.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Air-Gapped LLM Deployment Checklist
A practical control checklist for deploying and maintaining large language models in a fully air-gapped environment, from initial staging through ongoing patching and drift detection.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Related problems
safetensors header too large or invalid header error loading model weights
A safetensors HeaderTooLarge or InvalidHeaderDeserialization error means the file's first bytes are not the expected binary length-prefixed JSON header, almost always because the file on disk is not the actual model weights but a truncated partial download or a small Git LFS pointer text file that was never smudged into the real binary. The fix is to verify the file size matches what the Hub reports and re-download it properly, either with huggingface_hub or with git lfs pull, not to try to repair the file in place.
HuggingFace model download is extremely slow or stalls partway through
Slow or stalled HuggingFace downloads are usually caused by huggingface_hub's default transfer path not using parallel chunked downloads, a corporate proxy or firewall throttling or dropping long-lived connections, or genuinely insufficient bandwidth for a hundreds-of-gigabytes model. Enable hf_transfer for a much faster Rust-based parallel downloader, rely on the client's built-in resume behavior rather than restarting from zero, and for regulated or air-gapped sites, download once and mirror internally instead of pulling repeatedly over the internet.
Model loading fails offline or in an air-gapped environment despite having local files
Passing a local path to from_pretrained does not guarantee an offline load, because transformers and related libraries (tokenizers, some model configs, auto-mapping code) can still issue background network calls to check for updates, fetch a referenced remote component, or resolve auto_map entries that point back at the original HuggingFace repo. The fix is to set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 explicitly, use a complete local snapshot directory (not just the weights file), and verify no config field still references a remote repo ID.
Model revision or commit not found when pinning a specific version
A revision not found error means the exact commit hash, branch name, or tag you specified does not exist in that repository, most often because it was copied from a different repo, mistyped, or refers to a commit that was later force-pushed away or a tag that was deleted or renamed by the repo maintainer. Fix it by listing the repo's actual available revisions and re-pinning to a real, current one, and build your own immutable mirror if you need guarantees beyond what the source repo's maintainers commit to preserving.
GuideSecuring Model Weights in the Enterprise
Secure model weights end to end: custody controls, encryption at rest, access policies, and exfiltration prevention for regulated AI deployments.
GuideAir-Gapped Model Updates: A Patching Guide
Air-gapped model updates for enterprise AI: secure transfer procedures, hash verification, and staged rollout so patches never introduce risk.
GuideAir-Gapped LLM Deployment Patterns That Actually Work
Air-gapped LLM deployment patterns that work: offline model transfer, update workflows, monitoring without telemetry, and CMMC-ready architectures.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.