Why multi-node training hangs with no error, and the four things that actually cause it
Training hangs indefinitely after 'Rendezvous complete' with no error, no progress, and no crash across nodes
Also appears as
- torchrun hangs at 'Setting worker0 host as' and never proceeds to step 1
- Multi-node job stuck at NCCL init, single-node training on the same code works fine
Short answer
A multi-node job that hangs with no error almost always means not every rank actually joined the process group: a mismatched world size, a wrong MASTER_ADDR or MASTER_PORT, a firewall blocking the ephemeral ports NCCL negotiates after rendezvous, or a node that silently OOMed are the four most common causes. Because NCCL blocks silently while waiting for missing ranks, there is often no error at all until you manually intervene or hit a long default timeout.
Affects: torchrun, torch.distributed, DeepSpeed, and accelerate multi-node launches on any cluster without a managed job scheduler handling rendezvous automatically
Verify every rank actually joined before debugging anything else
- 1Confirm the total process count launched across all nodes equals world_size exactly: nproc_per_node times number of nodes, with no node accidentally launching zero or double.
- 2Confirm MASTER_ADDR resolves to the same reachable IP from every node (ping and getent hosts from each machine), and that MASTER_PORT is open and not already bound by another job.
- 3Test connectivity on the wider ephemeral port range NCCL uses after the initial handshake, not just the MASTER_PORT itself, since firewalls often only open one port.
- 4Check dmesg on every node for an OOM kill; a node that lost its training process silently will leave the rest of the cluster waiting forever.
- 5Confirm every node is running the identical PyTorch, CUDA, and NCCL version; a silent protocol mismatch can hang instead of erroring.
How to confirm this is your problem
- Single-node, single-GPU or single-node multi-GPU training on the same code works fine
- Logs show rendezvous or worker setup completing, then nothing, no progress bar movement, no error
- Some nodes show GPU utilization climbing while others sit at 0 percent
- The job eventually dies after a long default timeout with a generic NCCL or watchdog error, if it dies at all
Root causes and fixes
World size, nproc_per_node, or total rank count does not match across the launch commands on different nodes
torch.distributed and NCCL both wait for exactly world_size processes to call init_process_group before anything can proceed. If one node's launch script uses a different nproc_per_node, a different node_rank numbering, or simply fails to launch on one machine, the remaining ranks block indefinitely waiting for participants that will never arrive.
Fix: Recompute world_size as nnodes times nproc_per_node and verify the launch command on every node uses the same values, with unique, correctly ordered node_rank per machine. Generate each node's command from one source of truth to prevent drift.
MASTER_ADDR or MASTER_PORT is wrong: an unreachable IP, a hostname that resolves differently per node, or a port already in use
Rank 0 binds a TCP server on MASTER_ADDR:MASTER_PORT and every other rank must connect to that exact address. If MASTER_ADDR is a hostname that resolves differently on some nodes, or the port is already held by a leftover process from a previous crashed run, some ranks silently fail to connect and the whole group hangs.
Fix: Use a literal, verified-reachable IP address for MASTER_ADDR rather than a hostname where possible, confirm the port is free before launch, and kill any stale processes from previous runs first.
ss -ltnp | grep <MASTER_PORT> ping -c 2 <MASTER_ADDR>
A firewall allows the initial MASTER_PORT but blocks the wider ephemeral TCP port range NCCL opens afterward
NCCL's rendezvous only uses MASTER_PORT for initial coordination; the actual communication channels are established on additional dynamically chosen ports. A firewall configured to allow only the single documented port will let rendezvous succeed and then hang the moment real data transfer needs a blocked port.
Fix: Open the full ephemeral port range between nodes, or set NCCL_SOCKET_IFNAME and a bounded port range where supported, or place training nodes on a private network segment with no inter-node firewall at all.
Nodes run different PyTorch, CUDA, or NCCL library versions, causing a protocol-level mismatch instead of an explicit error
NCCL's wire protocol and feature negotiation can differ subtly between versions. Instead of a clean version-mismatch error, some combinations simply fail to complete the handshake and both sides wait, because neither side receives the message it expects in the format it expects.
Fix: Use the same container image, built once, deployed identically to every node, and verify versions match with a one-line python check on each machine before launching a real job.
One node's process was silently killed by the Linux OOM killer without the training framework catching the failure
Data loading with many workers, large prefetch buffers, or an unusually large batch can exhaust host system RAM independent of GPU memory. When the OOM killer terminates the process, the other ranks have no way to detect this since NCCL was not involved in the kill, and they wait on a partner that no longer exists.
Fix: Check dmesg or journalctl for oom-kill entries on every node after a hang, reduce num_workers or prefetch_factor in the data loader, and monitor host RAM alongside GPU VRAM during training.
dmesg | grep -i 'killed process'
Diagnostic commands
Confirm total launched process count matches world size
ps aux | grep train.py | wc -l
Run on each node and sum the results; the total across all nodes must equal nnodes times nproc_per_node exactly, or the mismatch alone explains the hang.
Test reachability of MASTER_ADDR and MASTER_PORT from every worker node
nc -vz <MASTER_ADDR> <MASTER_PORT>
Connection refused or timeout from any worker node means that node cannot reach rank 0 at all, which is sufficient to explain the hang regardless of any other configuration.
Check every node for an OOM kill around the hang time
dmesg -T | grep -i 'killed process'
Any hit here identifies which node lost its process silently; that node needs a memory fix (fewer data loader workers, smaller batch) rather than a networking fix.
Verify identical library versions across nodes
python -c "import torch; print(torch.__version__, torch.version.cuda)"
Run on every node; any node reporting a different version than the rest is a strong candidate for a silent protocol mismatch and should be rebuilt from the same image as the others.
Stopping it from happening again
- Generate every node's launch command from a single script or orchestrator so world_size, node_rank, and MASTER_ADDR can never drift between machines.
- Put training nodes on a dedicated, unfirewalled network segment (or explicitly open the full ephemeral range) rather than relying on a single allowed port.
- Build one container image for the whole cluster and never install packages per node by hand.
- Monitor host system RAM, not just GPU VRAM, since OOM kills at the OS level are a common silent cause of hangs.
When this becomes an architecture problem
If launches routinely need manual per-node fixes to work, or hangs happen unpredictably across different node combinations, the fix is a proper cluster job scheduler (Slurm, or Kubernetes with a training operator) that handles rendezvous, networking, and node health automatically, rather than continuing to hand-roll multi-node launches.
Frequently asked questions
Why does my job work on a single node but hang across multiple nodes?
Single-node training never depends on MASTER_ADDR or MASTER_PORT reachability, cross-node firewall rules, or matching library versions between separate machines, since everything happens in one process group on one host. Any of those factors can silently break a multi-node launch while leaving single-node training completely unaffected.
How long should I wait before assuming a multi-node job is hung, not just slow?
If GPU utilization on every node is at or near 0 percent for several minutes after the rendezvous or setup logs complete, and no progress bar or step log has advanced, treat it as hung rather than slow. A genuinely slow but working job still shows GPU activity and periodic log output.
Does using a hostname instead of an IP for MASTER_ADDR cause hangs?
It can, especially on multi-homed machines, containers, or hosts where DNS resolves differently per node. Using a literal, verified IP address for MASTER_ADDR removes one entire class of hang caused by inconsistent name resolution across the cluster.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Cluster Buildout Cost Calculator
Turn GPU count and class into a full cluster budget covering server chassis, networking fabric, power and cooling capex, and install, with a true cost per GPU.
Free ToolMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolAI Server Rack Power Budget Calculator
Convert GPUs per rack, TDP, host overhead, and facility PUE into a real rack power budget with redundancy, then check it against your available circuit capacity.
Related problems
NCCL error during multi-GPU training or inference
An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.
NCCL collective operation timeout during distributed training
An NCCL timeout means one or more ranks did not reach a collective operation (all-reduce, broadcast, all-gather) within the configured window, almost always because a straggler rank is slow or stuck, not because NCCL is malfunctioning. Raising NCCL_TIMEOUT can mask the symptom, but the durable fix is finding and removing the straggler: a data loading stall, an OOM-crashed rank, or a checkpoint write blocking one process.
InfiniBand not detected, NCCL falls back to slow TCP sockets
NCCL falls back to slow TCP sockets when it cannot find a usable InfiniBand device, most often because the IB kernel modules or rdma-core drivers are not installed or loaded, the fabric's subnet manager is not running so ports stay down, or NCCL environment variables point at the wrong network interface. Checking ibstat to confirm the hardware and fabric are actually up is the first step, before touching any NCCL environment variables.
HuggingFace Accelerate config mismatch causes wrong distributed launch
Accelerate errors during launch almost always mean the saved configuration does not match the current machine's actual GPU count, node count, or distributed type, or that the model was loaded with device_map='auto' inference-style sharding and then also handed to accelerate's training-mode preparation, which are two incompatible placement strategies. Regenerating the config for the current machine, or passing explicit CLI overrides, resolves most cases.
GuideMulti-Node LLM Training Infrastructure: Networking and Storage
Multi-node LLM training infrastructure explained: InfiniBand vs RoCE tradeoffs, storage throughput needs, and cluster topology for enterprise fine-tuning.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
GuideAI Datacenter Power and Cooling Planning for GPU Racks
Plan AI datacenter power and cooling for GPU racks: density thresholds, liquid cooling triggers, PUE targets, and real 2026 numbers for H100 to B200 racks.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.