Why NCCL cannot find InfiniBand and falls back to slow TCP, and how to get it detected again

NCCL's IB transport is built on the ibverbs userspace library, which requires both the appropriate kernel modules and rdma-core, or vendor OFED, to be installed. If either is missing, for example after a fresh OS install, a kernel update that did not rebuild the modules, or a container image built without IB support, ibv_devinfo returns no devices and NCCL silently falls back to sockets without necessarily raising a hard error.

Fix: Install rdma-core (or the vendor's OFED stack) and confirm the relevant kernel modules are loaded, then verify ibv_devinfo lists your HCA before assuming a NCCL-level problem.

ibv_devinfo
lsmod | grep -i mlx
modprobe mlx5_core

InfiniBand fabrics require a subnet manager, either running on a host via opensm or embedded in a managed switch, to assign LIDs and bring ports to an Active state. Without one running anywhere on the fabric, the physical link can show as connected while never reaching the Active state that NCCL requires, which looks identical to a missing device from NCCL's perspective.

Fix: Confirm a subnet manager is running somewhere on the fabric, either the switch's management interface or a designated host running opensm, then recheck ibstat for Active port state.

In multi-homed servers with both InfiniBand and multiple Ethernet interfaces, NCCL's auto-detection can occasionally pick the wrong interface, or a previous engineer may have hardcoded a socket interface override to fix a different problem and left it in place, which forces NCCL to skip its IB path entirely regardless of hardware.

Fix: Unset any stale NCCL_SOCKET_IFNAME or NCCL_IB_HCA overrides and let NCCL auto-detect first, then only set NCCL_IB_HCA explicitly if auto-detection genuinely picks the wrong one on a multi-HCA host.

Docker and Kubernetes isolate device access by default; unless /dev/infiniband and the relevant host network configuration are explicitly passed through, the container has no visibility into IB hardware the host itself can see perfectly well, so NCCL inside the container falls back to sockets even though ibstat on the host shows everything working.

Fix: Mount /dev/infiniband into the container explicitly, use an RDMA-aware Kubernetes device plugin if running under Kubernetes, and verify ibv_devinfo works from inside the container, not just on the host.

Physical layer problems, a failing HCA, a bad cable, or a faulty switch port, can prevent InfiniBand link training from completing even when drivers and the subnet manager are correctly configured, and typically affect only one or a few nodes rather than the whole cluster.

Fix: Swap the cable and, if possible, the switch port for an affected node to isolate whether the fault is in the HCA, the cable, or the switch, and check switch-side port error counters for physical layer errors.