# Why your training container crashes with a shared memory error, and how to fix it

> Source: https://netray.co/troubleshoot/docker-shared-memory-error
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

PyTorch DataLoader workers crash with a bus error inside Docker or Kubernetes because the default container shared memory allocation is only 64MB, far too small for multi-worker data loading. Set --shm-size explicitly in Docker, or add a Memory-medium emptyDir volume at /dev/shm in Kubernetes, sized to your batch size and worker count, and the crash disappears without touching your training code.

## Error Title

Dataloader crashes with a shared memory error inside a container

## Error Signature

RuntimeError: DataLoader worker (pid 1234) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory

## Alternate Signatures

- OSError: [Errno 28] No space left on device (writing to /dev/shm)
- torch.multiprocessing.shared_memory ... Bus error

## Category

containers

## Stack

- docker
- kubernetes
- pytorch

## Affects

PyTorch DataLoader with num_workers greater than 0 running inside Docker or Kubernetes, most common during fine-tuning and large batch training

## Quick Fix

### Fix it in 60 seconds

**Steps:**

- Confirm the cause: set num_workers=0 temporarily; if the crash disappears, shared memory is confirmed as the root cause.
- In Docker, rerun with --shm-size=8g added to your existing docker run command.
- In Kubernetes, add an emptyDir volume with medium Memory and a sizeLimit mounted at /dev/shm in the pod spec.
- Restore your original num_workers setting and rerun training.
- If it still crashes, increase shm-size further or check df -h /dev/shm during the run to see actual usage.

## Symptoms

- DataLoader worker killed by signal Bus error
- training crashes only when num_workers is greater than 0
- same code works with num_workers=0 but crashes with num_workers=4 or higher
- OSError no space left on device referencing /dev/shm

## Causes

### Docker's default 64MB /dev/shm is too small for multi-worker dataloaders

**Likelihood:** most-common

**Explanation:** Docker allocates a default shared memory size of 64MB for /dev/shm inside a container, which PyTorch's DataLoader workers use to pass tensors between worker processes and the main process; with several workers and reasonably sized batches, this default fills up almost instantly, and the kernel kills the worker process with a bus error rather than letting it silently corrupt memory.

**Fix:** Increase the container's shared memory allocation explicitly at run time, since the 64MB default is a Docker safety default, not a hard system limit.

**Commands:**

- docker run --shm-size=8g --gpus all myimage
- docker run --ipc=host --gpus all myimage

### Kubernetes emptyDir /dev/shm is not backed by memory medium, or has no size limit set

**Likelihood:** common

**Explanation:** Kubernetes pods get their own tiny default /dev/shm similar to Docker unless you explicitly mount an emptyDir volume at /dev/shm with medium Memory and a sizeLimit; without this, the pod inherits the same small shared memory ceiling as a default Docker container, and any dataloader with multiple workers hits the same bus error inside Kubernetes.

**Fix:** Add an emptyDir volume with medium Memory and an explicit sizeLimit mounted at /dev/shm in the pod spec, sized to comfortably exceed your batch size times worker count.

**Commands:**

### Too many DataLoader workers relative to batch size and available shared memory

**Likelihood:** occasional

**Explanation:** Each DataLoader worker holds its own share of in-flight batches in shared memory simultaneously; setting num_workers very high without a corresponding increase in shm-size multiplies memory pressure linearly, so a shm-size that was fine at 4 workers can still be exhausted at 16 workers even on the same batch size.

**Fix:** Either raise shm-size further or reduce num_workers, and treat the two as a joint tuning knob rather than adjusting only one.

**Commands:**

- python -c "import torch; print(torch.utils.data.DataLoader)"

### Using pin_memory with very large batches multiplies shared memory pressure further

**Likelihood:** rare

**Explanation:** pin_memory=True causes the DataLoader to allocate page-locked host memory for faster GPU transfer, and combined with shared memory tensors from multiple workers, very large batch or sequence lengths for LLM training data can push total shared memory usage well beyond what a modestly sized shm allocation anticipated.

**Fix:** When training with very large batches or long sequences, size /dev/shm generously (16GB or more) or reduce pin_memory usage if shared memory remains constrained.

**Commands:**

## Diagnostics

### Check the container's current shared memory size

**Command:** df -h /dev/shm

**Interpretation:** The default Docker allocation shows as 64M; if your workload needs more than what is shown as available, this confirms the exhaustion.

### Reproduce with a minimal worker count

**Command:** python train.py --num_workers 0

**Interpretation:** If the crash disappears with zero workers, it confirms shared memory, not a data bug, is the root cause, since num_workers=0 does not use inter-process shared memory.

### Check Kubernetes pod's shm mount

**Command:** kubectl exec <pod> -- df -h /dev/shm

**Interpretation:** Confirms whether the pod actually has an emptyDir Memory volume mounted at /dev/shm or is stuck with the tiny default.

### Watch shared memory usage live during training

**Command:** watch -n1 df -h /dev/shm

**Interpretation:** Rising usage that plateaus near the limit right before the crash confirms shm-size, not a code bug, is the bottleneck.

## Prevention

- Set --shm-size explicitly on every training container, never rely on Docker's 64MB default
- Add an explicit Memory-medium emptyDir for /dev/shm in every Kubernetes training pod spec
- Size shm relative to batch_size times num_workers times sample size, not a fixed guess
- Add a startup check that logs df -h /dev/shm so shm exhaustion is visible in logs before a crash, not just after

## When To Escalate

If you are running many concurrent training jobs across a shared cluster and each needs a different shm-size tuned to its batch size and worker count, standardizing a training pod template with a generous, monitored shm allocation is worth doing once at the platform level rather than tuning per job.

## Faq

### Why does Docker default to only 64MB for /dev/shm?

**Answer:** It is a conservative default set by Docker to avoid one container consuming excessive host memory through shared memory allocation by default; most simple containerized applications never need more than this. Machine learning workloads that use multiprocessing to parallelize data loading are a common exception, since PyTorch's DataLoader workers pass batches between processes via shared memory, and that usage pattern was not what the small default was designed around.

### What's the difference between --shm-size and --ipc=host?

**Answer:** --shm-size=8g sets an explicit, isolated size for the container's own /dev/shm, keeping it contained and predictable. --ipc=host instead shares the host's entire IPC namespace and shared memory with the container, effectively removing any practical size limit but also removing the isolation between the container and host processes. For production training jobs, an explicit --shm-size is usually preferred over --ipc=host for isolation reasons.

### How do I set shared memory size for a training pod in Kubernetes?

**Answer:** Kubernetes has no shm-size flag equivalent; instead you mount an emptyDir volume with medium Memory and a sizeLimit at the /dev/shm path in both the pod's volumes and the container's volumeMounts sections. Without this explicit volume, the pod's /dev/shm defaults to the same small size as an unconfigured Docker container, and multi-worker DataLoaders will hit the same bus error.

### What size should I set for /dev/shm during LLM fine-tuning?

**Answer:** There is no universal number since it scales with your batch size, sequence length, and num_workers, but a common practical starting point for fine-tuning workloads is somewhere between 4GB and 16GB, then watching df -h /dev/shm during an actual run to see how close usage gets to the limit. It is safer to allocate generously, since unused shared memory does not consume host RAM until it is actually written to.

## Search terms

dataloader worker killed by signal bus error, shared memory error pytorch docker, dev shm too small docker, docker shm-size training crash, kubernetes emptydir medium memory dev shm, no space left on device dev shm

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json