# Why safetensors throws HeaderTooLarge or InvalidHeaderDeserialization, and how to fix it > Source: https://netray.co/troubleshoot/safetensors-header-too-large > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer A safetensors HeaderTooLarge or InvalidHeaderDeserialization error means the file's first bytes are not the expected binary length-prefixed JSON header, almost always because the file on disk is not the actual model weights but a truncated partial download or a small Git LFS pointer text file that was never smudged into the real binary. The fix is to verify the file size matches what the Hub reports and re-download it properly, either with huggingface_hub or with git lfs pull, not to try to repair the file in place. ## Error Title safetensors header too large or invalid header error loading model weights ## Error Signature safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge ## Alternate Signatures - safetensors_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization - SafetensorError: Metadata not found - Error while deserializing header: HeaderTooLarge (attempted to allocate ... bytes) ## Category model-loading ## Stack - huggingface - transformers - pytorch ## Affects Any model shipped in .safetensors format, transformers/safetensors of any recent version, most common after interrupted downloads or git clone without LFS smudge ## Quick Fix ### Fix it in a few minutes **Steps:** - Check the file size: ls -la for the .safetensors file. If it is a few hundred bytes to a few KB, it is an LFS pointer or a failed download, not real weights. - If you used git clone on a HuggingFace repo, run git lfs install once, then git lfs pull inside the repo to fetch the actual binary content. - If you used snapshot_download or from_pretrained, delete the specific corrupted file from the local cache (under ~/.cache/huggingface/hub/) and retry, letting it re-download cleanly. - Prefer huggingface_hub's snapshot_download or hf download with resume_download enabled instead of raw git clone for large model repos. - After re-download, confirm the file loads: python -c "from safetensors import safe_open; safe_open('model.safetensors','pt')". ## Symptoms - Loading fails specifically on one shard file (model-00003-of-00007.safetensors) while others loaded fine - The problematic file's size on disk is dramatically smaller than sibling shard files or than the size listed on the HuggingFace repo page - cat or head on the file shows plaintext like 'version https://git-lfs.github.com/spec/v1' instead of binary content - The error occurs consistently on every load attempt from that specific local file, not intermittently ## Causes ### File on disk is an unsmudged Git LFS pointer, not the real binary weights **Likelihood:** most-common **Explanation:** HuggingFace repos store large files via Git LFS. A plain git clone without git-lfs installed, or with LFS smudge filters disabled (common in CI containers, Docker builds, or air-gapped mirrors), downloads only the tiny LFS pointer text file for each large asset. safetensors then tries to parse that pointer's opening bytes as a binary header length and fails immediately. **Fix:** Install git-lfs (apt-get install git-lfs or equivalent), run git lfs install, then git lfs pull from inside the cloned repo to replace every pointer file with the real binary content. **Commands:** - git lfs install - git lfs pull - git lfs ls-files ### Download was interrupted or truncated (network drop, disk full, killed process) **Likelihood:** common **Explanation:** safetensors reads a 8-byte length prefix followed by a JSON header of that exact length. If the download stopped partway through, the file ends abruptly and either the length prefix points past the actual file size or the JSON itself is cut off mid-object, producing HeaderTooLarge or a deserialization failure. **Fix:** Delete the partial file and re-download with a resumable client. huggingface_hub's snapshot_download and hf download resume broken transfers automatically by default in current versions; verify final file size against the repo's listed size before trusting it. **Commands:** - rm ~/.cache/huggingface/hub/models--*/blobs/ - python -c "from huggingface_hub import snapshot_download; snapshot_download('org/model')" ### Disk quota, out-of-space, or filesystem error silently truncated the write **Likelihood:** occasional **Explanation:** On a full disk or a filesystem hitting a quota limit, some write paths fail silently or truncate rather than raising an obvious I/O error, leaving a partially-written safetensors file that looks superficially present but has an incomplete or absent header. **Fix:** Check available disk space before large downloads (a 70B model in FP16 needs well over 140GB), and after download always verify file size against the expected value rather than assuming presence means completeness. **Commands:** - df -h - du -sh ~/.cache/huggingface/hub/models--org--model ### File was copied or synced with a tool that mangled binary content (e.g. text-mode transfer, wrong rsync flags, antivirus quarantine) **Likelihood:** rare **Explanation:** Certain file transfer tools or misconfigured sync jobs (FTP in ASCII mode, some corporate proxy content scanners, misconfigured rsync excludes) can alter or truncate binary files during transfer between an internet-connected staging host and an air-gapped target, producing the same header corruption symptom. **Fix:** Transfer model artifacts as a single verified archive (tar plus a checksum file) rather than ad hoc file-by-file copies, and validate the checksum on the receiving side before use. **Commands:** - sha256sum model.safetensors ## Diagnostics ### Check actual vs expected file size **Command:** ls -la model.safetensors && curl -sI https://huggingface.co/org/model/resolve/main/model.safetensors | grep -i content-length **Interpretation:** If the local size is smaller than the Content-Length reported by the Hub, the file is truncated or a pointer. They should match exactly for a good download. ### Peek at the first bytes to detect an LFS pointer **Command:** head -c 200 model.safetensors **Interpretation:** Readable text starting with 'version https://git-lfs.github.com' confirms this is an LFS pointer, not binary weights. Binary garbage in the terminal is expected and fine for a real safetensors file. ### Validate the file opens correctly **Command:** python -c "from safetensors import safe_open; f=safe_open('model.safetensors','pt'); print(len(f.keys()))" **Interpretation:** A tensor key count greater than zero confirms the header parsed successfully. Any exception here reproduces the original bug and confirms the file itself, not the loading code, is the problem. ## Prevention - Always use huggingface_hub's snapshot_download or hf download CLI for model weights instead of raw git clone, since it handles LFS content and resumable downloads correctly by default. - For internal mirrors serving air-gapped environments, checksum every artifact (sha256) at mirror time and re-verify at the consuming end, never trust file presence alone. - In CI/build images, explicitly install git-lfs and confirm git lfs pull before treating a cloned model repo as usable. - Set a minimum free disk space check in deployment automation before starting multi-shard model downloads. ## When To Escalate If this keeps recurring across an air-gapped mirror pipeline rather than being a one-off bad download, the fix is a proper internal artifact registry with checksum verification built into the transfer process, not repeated manual re-downloads. That is worth designing once as part of your model supply chain rather than firefighting per incident. ## Faq ### Can I fix a truncated safetensors file without re-downloading? **Answer:** No. Safetensors files are not designed to be patched or repaired; the header and tensor data must be complete and consistent. The only reliable fix is obtaining a complete copy of the file, either by re-downloading or by copying a verified good copy from another location. ### Why did some shards load fine but one specific shard fails? **Answer:** Multi-shard models download each file independently, so a network blip or interruption during one specific file's transfer will only corrupt that shard while others complete normally. Check that one file's size and checksum specifically against the Hub's listing. ### Is this the same underlying issue as a 'corrupted checkpoint' error? **Answer:** Yes, HeaderTooLarge is one specific manifestation of a corrupted or incomplete checkpoint file. See the general corrupted checkpoint troubleshooting page for broader verification steps covering both safetensors and other formats like GGUF or PyTorch .bin. ### Does converting a .bin checkpoint to safetensors avoid this problem? **Answer:** No, the format is not the cause here; an interrupted or LFS-pointer download of a .bin file fails just as badly, just with a different error message from torch.load. The root cause and fix (verify size, re-download or LFS pull) are the same regardless of weight format. ## Search terms safetensors HeaderTooLarge, InvalidHeaderDeserialization, safetensors error while deserializing header, git lfs pointer file safetensors, corrupted safetensors file, safetensors truncated download, huggingface model download corrupted --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json