Multi-GPU LLM Serving: Tensor Parallelism vs Pipeline Parallelism

Tensor Parallelism: Splitting Layers, Sharing Every Forward Pass

Tensor parallelism (TP) splits weight matrices within each layer across GPUs, so every GPU computes a portion of every layer and results are combined via an all-reduce operation after each layer. 0 at roughly 128 GB/s) introduces communication overhead that can dominate compute time and tank throughput. In vLLM, set --tensor-parallel-size to the number of GPUs; the framework handles sharding automatically for supported architectures.

TP scales well up to 8 GPUs on a single NVLink-connected node (a DGX or HGX-class server) and is the standard choice for serving a single large dense model, such as a 70B or 405B parameter model, that exceeds one GPU's memory.

• Requires NVLink or equivalent high-bandwidth interconnect; PCIe-only multi-GPU setups suffer badly under TP
• Every layer's forward pass requires an all-reduce, so TP degree is bounded by interconnect topology, not just GPU count
• vLLM's --tensor-parallel-size handles sharding automatically for supported model architectures
• Best fit: single large dense model on one NVLink-connected node, up to 8-way TP on current DGX/HGX hardware

Pipeline Parallelism: Splitting by Layer Range

Pipeline parallelism (PP) assigns contiguous layer ranges to different GPUs or nodes, so a request flows sequentially through GPU 1's layers, then GPU 2's layers, and so on, with far less frequent communication than TP (only activations passed at stage boundaries, not every layer). This makes PP tolerant of slower interconnect and the natural choice for spanning multiple nodes over InfiniBand or even Ethernet where NVLink is not available. The cost is pipeline bubbles: GPUs later in the pipeline sit idle while waiting for the first request to arrive through earlier stages, which continuous batching mitigates but does not eliminate.

PP also complicates request scheduling and generally serves fewer requests per GPU than an equivalent TP setup when NVLink is available, so it is mainly justified when the model does not fit within a single NVLink domain even with TP, common with the largest MoE models like DeepSeek V3 or Kimi K2.

• Only passes activations at stage boundaries, tolerating slower interconnect than TP requires
• Necessary when a model exceeds one NVLink domain, common for very large MoE models spanning multiple nodes
• Pipeline bubble overhead reduces GPU utilization versus TP on the same hardware, mitigated but not eliminated by continuous batching
• vLLM supports combining TP within a node and PP across nodes for the largest models

NCCL, Interconnect, and the Overhead You Do Not See in Benchmarks

Both TP and PP rely on NCCL (NVIDIA Collective Communications Library) for GPU-to-GPU communication, and NCCL performance is entirely bound by the physical interconnect topology. Verify actual NVLink connectivity with nvidia-smi topo -m before assuming a server's GPUs are fully NVLink-connected; some cloud instances and older server configurations have partial NVLink topology where only some GPU pairs get full bandwidth. For multi-node setups, InfiniBand (typically 200-400 Gb/s per link on current hardware) is close to mandatory for TP across nodes; standard Ethernet introduces enough latency that PP or a fully sharded single-node approach usually wins.

Benchmark your actual topology before committing to a TP degree, since a benchmark run on a fully NVLink-connected reference system will not predict performance on hardware with a different or partial interconnect.

Practical Guidance for Choosing a Parallelism Strategy

For most enterprise deployments serving a single dense model up to 70B or so parameters on an 8-GPU NVLink node, tensor parallelism alone is the right answer, no pipeline parallelism needed. For models that exceed a single node's memory even with TP, such as the largest MoE models, combine TP within each node with PP across nodes. If your hardware lacks NVLink entirely, either quantize aggressively enough to fit within fewer GPUs and avoid multi-GPU serving altogether, or accept PP's lower per-GPU utilization as the cost of the interconnect you have.

Do not default to PP because it seems simpler to reason about; on NVLink hardware it is almost always the wrong choice for throughput.

How Netray Designs Multi-GPU Serving Topologies

Netray sizes multi-GPU serving topology against the client's actual GPU fleet interconnect, verified with real topology checks rather than assumed NVLink connectivity, before recommending a tensor or pipeline parallelism strategy. For clients deploying frontier-scale MoE models across multiple nodes, we design the combined TP-within-node, PP-across-node topology and validate it against real request load, not synthetic benchmarks. This is core to our on-prem GPU cluster design work, where getting the interconnect and parallelism strategy wrong is the single most common cause of a client's expensive GPU cluster underperforming its theoretical throughput.