# Why a model checkpoint is corrupted, and how to verify and fix it

> Source: https://netray.co/troubleshoot/corrupted-model-checkpoint
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

A corrupted checkpoint means the bytes on disk do not match the original artifact the model author published, whether from an interrupted download, a bad copy between systems, disk-level bit rot, or a failed write during a save operation. There is no reliable way to repair a corrupted deep learning checkpoint; the fix is always to verify the file against a known-good hash or size and re-obtain a clean copy, then build a verification step into your pipeline so the same failure does not silently recur.

## Error Title

Corrupted model checkpoint fails to load or loads with garbage weights

## Error Signature

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

## Alternate Signatures

- _pickle.UnpicklingError: invalid load key
- RuntimeError: storage has wrong size, expected ... got ...
- Model loads without error but produces nonsensical output on every input

## Category

model-loading

## Stack

- pytorch
- huggingface
- transformers

## Affects

Any checkpoint format (PyTorch .bin/.pt, safetensors, GGUF) transferred, cached, or stored across an unreliable path; especially relevant for air-gapped transfers

## Quick Fix

### Fix it in a few minutes

**Steps:**

- Compare the file's size in bytes against the size listed on the source (HuggingFace repo page or your internal artifact registry).
- If a checksum (sha256) is published for the artifact, compute and compare it: sha256sum <file>.
- If sizes or hashes do not match, delete the local file entirely and re-download or re-copy it from the original source rather than attempting to patch it.
- For multi-shard models, check every shard individually, since only one shard among many can be corrupted while the rest load fine.
- After obtaining a clean copy, re-run your standard load-and-smoke-test step before treating the model as ready to serve.

## Symptoms

- Loading throws a low-level deserialization error (zip archive, pickle, or safetensors header error) rather than a model-specific error
- The model loads without any exception but produces completely nonsensical output that does not resemble a real language model's failure modes
- The failure is specific to one file among several shards, or specific to one copy of the model on one machine but not another
- File size on disk is smaller than, or subtly different from, the size reported at the original source

## Causes

### File was only partially downloaded or partially copied before being treated as complete

**Likelihood:** most-common

**Explanation:** An interrupted network transfer, a script that does not check download completion before proceeding, or a copy operation killed partway through all leave a file that is present on disk but structurally incomplete. Binary checkpoint formats (pickle-based .bin/.pt, zip-based PyTorch archives, safetensors) all require the complete byte stream to parse correctly; a truncated file fails at whatever point the reader runs out of expected bytes.

**Fix:** Always verify file size (and ideally checksum) against the source immediately after any download or copy operation, before marking that transfer as successful in your pipeline, and re-transfer if there is any mismatch.

**Commands:**

- ls -la model.bin
- sha256sum model.bin

### Bit rot, disk errors, or filesystem corruption altered bytes on storage over time

**Likelihood:** common

**Explanation:** Long-lived storage, especially on aging disks, degraded RAID arrays, or filesystems without integrity checking (no checksumming filesystem, no ECC memory), can silently flip bits over months or years of storage. A checkpoint that loaded fine when first downloaded can fail later purely from storage degradation, with no code or environment change involved at all.

**Fix:** Store critical model artifacts on storage with integrity guarantees where possible (checksumming filesystems, RAID with scrubbing, cloud object storage with built-in integrity verification), and periodically re-verify archived checkpoints' checksums against your original record.

**Commands:**

- sha256sum -c checksums.sha256

### Transfer between an internet-connected staging host and an air-gapped target altered or truncated the file

**Likelihood:** occasional

**Explanation:** Manual transfer methods used to move artifacts across an air gap (certain removable media formats, some file transfer utilities with size limits, network shares with unusual encoding settings) can truncate or subtly alter binary content in ways that are not obvious until the model is loaded and fails or misbehaves on the target side.

**Fix:** Use a transfer method proven for large binary files (tar archives with an accompanying checksum manifest) and verify the checksum on the receiving side immediately after transfer, before deleting the source copy.

**Commands:**

- tar -cf model-bundle.tar model/ && sha256sum model-bundle.tar

### A save or export operation was interrupted mid-write (out of disk space, process killed, power loss)

**Likelihood:** occasional

**Explanation:** When a training or export job writes a checkpoint and is killed by an out-of-memory reaper, hits a full disk, or loses power partway through torch.save or safe_save, the resulting file can be left in a partially-written state that looks present but is structurally incomplete, identical in symptom to a truncated download.

**Fix:** Write checkpoints to a temporary filename and atomically rename to the final name only after the write completes successfully, and always confirm free disk space before starting a large checkpoint save.

**Commands:**

- df -h

### Wrong file entirely was referenced (stale cache pointing at an old or unrelated file with a coincidentally similar name)

**Likelihood:** rare

**Explanation:** In pipelines with aggressive local caching or symlink-based model directories, a stale cache entry or dangling symlink can point at leftover data from a previous, unrelated experiment, producing load failures or bizarre output that has nothing to do with the current model's actual integrity.

**Fix:** Verify the exact path being loaded resolves to the file you expect, especially in setups using symlinks or content-addressed caches, and clear stale cache directories when switching between model versions.

**Commands:**

- readlink -f model.bin

## Diagnostics

### Verify file size against the published source

**Command:** ls -la model.safetensors

**Interpretation:** Compare directly against the byte count shown on the HuggingFace repo page or your internal artifact registry's manifest. Any difference, even by a few bytes, confirms corruption or truncation.

### Compute and compare a checksum

**Command:** sha256sum model.safetensors

**Interpretation:** Match this against a published or previously recorded hash. A mismatch is definitive proof of corruption regardless of how similar the file size appears.

### Attempt a minimal load in isolation

**Command:** python -c "from safetensors.torch import load_file; sd=load_file('model.safetensors'); print(len(sd))"

**Interpretation:** A clean tensor count confirms the file structure is intact. An exception here reproduces the corruption independent of the full model-loading pipeline, narrowing the problem to the file itself.

## Prevention

Publish and store checksums (sha256) alongside every model artifact in your internal registry, and verify them automatically as part of any download, copy, or transfer step.

Use atomic write patterns (write to temp file, rename on success) for any process that saves checkpoints, so a killed process never leaves a half-written file mistaken for a complete one.

For air-gapped transfers, standardize on a single verified transfer method (checksummed tar bundle) rather than ad hoc file copying.

Periodically re-verify long-term archived checkpoints against their original checksums to catch storage-level bit rot before it is discovered at deployment time.

## When To Escalate

If corruption keeps recurring across your model supply chain rather than being an isolated incident, the fix is a proper artifact integrity process end to end (checksum at source, checksum at every hop, checksum before serving) rather than re-downloading and hoping each time. This is a supply-chain control worth designing deliberately, especially for regulated environments where model provenance and integrity are audit requirements.

## Faq

### Can I repair a corrupted checkpoint file instead of re-downloading it?

**Answer:** No, deep learning checkpoint formats (pickle, zip-based PyTorch archives, safetensors, GGUF) are not designed to be partially repaired. The only reliable fix is obtaining a complete, verified copy of the file from a trusted source.

### How do I know if it is corruption versus a genuinely different but valid model?

**Answer:** Compare the file's checksum against the value published by the original source (HuggingFace model card, your internal registry's manifest at time of ingestion). A checksum match with load failure suggests a code or environment bug rather than corruption; a mismatch confirms the file itself is the problem.

### Why did the model load without error but produce nonsensical output?

**Answer:** Not all corruption produces a hard parsing error. If enough of the file structure is intact for the loader to succeed but the underlying tensor bytes are altered (partial truncation padded with zeros, a bit-flipped region within otherwise valid data), the model can load 'successfully' while its weights are silently wrong, producing degraded or nonsensical output as the visible symptom.

### Should I keep a local backup of every model I download?

**Answer:** For any model in active production use, yes, keep a verified, checksummed copy in your own internal storage rather than relying on being able to re-download it on demand, especially important for air-gapped or regulated environments where the original source may not always be reachable later.

## Search terms

corrupted model checkpoint, PytorchStreamReader failed reading zip archive, invalid load key pickle error, verify model checksum huggingface, checkpoint file corrupted, storage has wrong size expected

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json