# Why GPU peer-to-peer access does not work between GPUs, and why disabling it is a diagnostic step, not a fix > Source: https://netray.co/troubleshoot/gpu-peer-to-peer-not-working > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer GPU peer-to-peer (P2P) access fails when the PCIe topology, IOMMU or ACS settings, or the GPU model itself does not support a direct memory path between two devices, forcing all transfers through the CPU and host memory instead of GPU-to-GPU. Setting NCCL_P2P_DISABLE=1 is a useful diagnostic to confirm P2P is the problem, but it only removes the crash by falling back to a slower path; it does not restore the P2P bandwidth you actually need for good multi-GPU performance. ## Error Title GPU peer-to-peer (P2P) access not working between GPUs on the same node ## Error Signature RuntimeError: NCCL WARN Cuda failure 'peer access is not supported between these two devices' ## Alternate Signatures - cudaErrorPeerAccessNotEnabled: peer access is not supported between these two devices - Training or serving is dramatically slower with 4 or 8 GPUs than expected, only when P2P is disabled ## Category distributed ## Stack - cuda - nvidia-driver - pytorch ## Affects Any multi-GPU server without NVLink, most common on workstation motherboards, servers with IOMMU or VFIO enabled, and consumer GPUs like the RTX 4090 ## Quick Fix ### Diagnose the topology before deciding P2P cannot be fixed **Steps:** - Run nvidia-smi topo -m to see the actual P2P and NVLink matrix between every GPU pair in the system. - If P2P shows as unsupported and IOMMU is enabled in BIOS or the kernel command line, try disabling IOMMU (or setting iommu=pt) and re-check the topology matrix. - As a diagnostic only, set NCCL_P2P_DISABLE=1 and confirm the error goes away; this tells you P2P is the cause but is not the fix, since it forces slower host-routed transfers. - Check for PCIe ACS redirection with lspci, since some server BIOSes enable it by default and it silently defeats P2P even on capable hardware. - If the GPUs are consumer cards outside an NVLink-capable pairing, treat the lack of P2P as expected behavior and plan tensor or data parallelism around slower interconnect rather than expecting a fix. ## Symptoms - Multi-GPU training or serving is dramatically slower than expected relative to single-GPU throughput times GPU count - nvidia-smi topo -m shows PHB or SOC rather than PIX or NV between GPUs you expect to communicate directly - NCCL logs mention falling back to sockets or failing to open a CUDA IPC handle - The problem appears on one specific server or motherboard but not another with the same GPU model ## Causes ### GPUs are installed in PCIe slots without a direct P2P-capable path, such as behind different CPU sockets or on a consumer board without a PCIe switch **Likelihood:** most-common **Explanation:** P2P DMA requires a topology where the PCIe root complex or a shared switch can route memory transfers directly between two GPUs' address spaces. On dual-socket servers, GPUs attached to different CPU sockets typically cannot P2P at all without a specific interconnect; on consumer boards, GPUs may share a root complex but lack a switch that supports P2P forwarding. **Fix:** Check nvidia-smi topo -m and, where possible, place GPUs that must communicate frequently in slots under the same PCIe switch or CPU socket; for a from-scratch build, choose a server board and GPU count specifically validated for P2P or NVLink. **Commands:** - nvidia-smi topo -m - lspci -tv ### IOMMU is enabled in BIOS or the kernel, which blocks the DMA remapping that P2P transfers rely on **Likelihood:** common **Explanation:** IOMMU (VT-d on Intel, AMD-Vi on AMD) is often enabled by default on server boards for virtualization or security isolation. It intercepts and remaps device DMA, and unless explicitly configured to allow P2P DMA between devices, it silently forces all transfers through the CPU, disabling the fast path even though the PCIe topology would otherwise support it. **Fix:** If virtualization or passthrough is not required, disable IOMMU in BIOS or set iommu=off (or iommu=pt for a middle ground) in the kernel boot parameters, then reboot and re-check the topology matrix. **Commands:** - cat /proc/cmdline | grep -i iommu ### The GPU model itself does not support P2P outside of NVLink, which is common for consumer cards **Likelihood:** common **Explanation:** NVIDIA restricts full PCIe P2P support on several consumer GPU lines regardless of topology, reserving guaranteed high-bandwidth peer access for NVLink-connected or datacenter-class GPUs. On these cards, even a perfect PCIe topology with IOMMU disabled will not enable P2P. **Fix:** Accept that consumer GPUs will route through the host for cross-GPU transfers and plan accordingly (smaller tensor-parallel groups, more reliance on data parallelism), or move to datacenter GPUs (A100, H100, or NVLink-equipped cards) if P2P bandwidth is required. ### PCIe ACS (Access Control Services) is enabled on the root complex or a switch, forcing peer traffic through the CPU for isolation **Likelihood:** occasional **Explanation:** ACS is a PCIe feature that isolates devices from each other for security and IOMMU groupings, commonly enabled by default in server BIOS or by virtualization-focused kernel defaults. It intercepts P2P transactions and routes them back through the root complex, defeating the purpose of P2P without producing an obvious error, just reduced bandwidth or an outright block. **Fix:** Check ACS status with lspci -vvv and disable ACS in BIOS if security requirements allow, or use a documented ACS override kernel parameter if your distribution supports it. **Commands:** - lspci -vvv | grep -A 5 'Access Control Services' ### The system is a virtualized or GPU-passthrough (VFIO) environment where P2P is disabled by design for isolation between virtual machines **Likelihood:** rare **Explanation:** VFIO-based GPU passthrough intentionally isolates each GPU into its own IOMMU group to maintain VM security boundaries, which is fundamentally incompatible with the shared-memory assumptions P2P relies on. This is a deliberate tradeoff of the virtualization layer, not a misconfiguration. **Fix:** If P2P performance is required, run training directly on bare metal rather than through VFIO passthrough VMs, or accept the P2P-disabled performance profile as the cost of the virtualization architecture. ## Diagnostics ### Check the P2P and NVLink topology matrix **Command:** nvidia-smi topo -m **Interpretation:** Look for PIX (same PCIe switch, good) or NV (NVLink, best) versus PHB or SOC (different root complex or socket, P2P unlikely or unavailable) between the GPU pairs you care about. ### Check whether IOMMU is enabled **Command:** cat /proc/cmdline | grep -i iommu **Interpretation:** Presence of intel_iommu=on or amd_iommu=on without a corresponding passthrough allowance suggests IOMMU is actively remapping DMA and may be blocking P2P. ### Check PCIe ACS settings **Command:** lspci -vvv | grep -B 5 -A 10 'Access Control Services' **Interpretation:** An ACSCtl line showing SrcValid or RequestRedir enabled on the relevant bridge indicates ACS is actively redirecting peer traffic through the root complex. ### Confirm the impact of disabling P2P as a controlled test **Command:** NCCL_P2P_DISABLE=1 python your_multi_gpu_script.py **Interpretation:** If the error disappears and throughput drops noticeably compared to expected NVLink or P2P bandwidth, this confirms P2P was both broken and needed; it is not itself a solution, just proof of the diagnosis. ## Prevention - When procuring multi-GPU servers, choose boards and chassis explicitly validated for P2P or NVLink at your intended GPU count, not just a generic multi-GPU claim. - Standardize BIOS settings (IOMMU, ACS) across your fleet and document them, so a new node does not silently regress on P2P support. - Benchmark P2P bandwidth as part of new node acceptance testing, not after a production job underperforms. - For workloads that truly need P2P bandwidth at scale, budget for NVLink or NVSwitch-equipped hardware rather than relying on PCIe topology alone. ## When To Escalate If your current server topology or GPU model fundamentally cannot support P2P, consumer GPUs, cross-socket placement, or a board without a P2P-capable switch, and the workload genuinely needs that bandwidth, this is a hardware procurement decision, not something BIOS or driver settings can fix. ## Faq ### Does NCCL_P2P_DISABLE=1 fix the peer access error? **Answer:** It stops the crash by forcing NCCL to route through the host instead of attempting direct GPU-to-GPU transfers, so in that narrow sense it makes the job run. But it does not restore P2P bandwidth, so multi-GPU throughput will be noticeably lower; treat it as a way to confirm the diagnosis and keep a job running, not as the actual fix. ### Why does nvidia-smi topo -m show PHB instead of PIX for my GPUs? **Answer:** PHB means the GPUs are connected through a PCIe host bridge rather than a shared switch, which usually means they sit on different root complexes, often tied to different CPU sockets on a dual-socket server. That topology typically does not support fast P2P, regardless of driver or IOMMU settings. ### Do consumer GPUs like the RTX 4090 support P2P at all? **Answer:** NVIDIA restricts full P2P support on most current consumer GPU lines outside of specific NVLink-bridged configurations, so you should generally expect P2P to be unavailable or limited on these cards even with ideal PCIe topology and IOMMU disabled. Datacenter GPUs such as the A100 and H100 support P2P and NVLink much more broadly. ## Search terms nccl p2p disable, gpu peer to peer not working, cuda peer access not supported, nvidia-smi topo p2p, iommu disable p2p gpu, pcie acs disable p2p multi gpu --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json