What Is Tensor Parallelism?
Also known as: Model Parallelism, TP
Definition
Tensor parallelism is a technique that splits the weight matrices inside each model layer across several GPUs, so every card holds a slice and computes part of each operation, letting a model run that no single GPU could hold.
Tensor Parallelism Explained
Where pipeline parallelism assigns whole layers to different GPUs, tensor parallelism cuts inside each layer. A large weight matrix is partitioned by rows or columns across the cards; each GPU multiplies its slice against the input and the partial results are combined with a collective operation, typically an all-reduce. Every card participates in every layer, so all GPUs stay busy simultaneously rather than waiting their turn as they would in a pipeline arrangement.
That constant synchronization is the cost. Each layer requires a collective communication step, so latency depends heavily on interconnect quality. High-speed direct GPU-to-GPU links deliver hundreds of gigabytes per second and keep the overhead small. Standard PCIe is substantially slower, and on PCIe-only servers the communication can consume a large share of total inference time, meaning four GPUs deliver far less than four times the capability.
The degree, usually written as the number of ways the model is split, must generally divide evenly into architectural dimensions such as the attention head count. Powers of two, most commonly two, four, or eight, are therefore the practical choices. Serving frameworks including vLLM and TensorRT-LLM implement tensor parallelism behind a single configuration flag, so the engineering effort is small relative to the hardware and topology decisions.
Choose tensor parallelism when a model genuinely does not fit on one card after quantization, or when you need lower latency for a single request than one GPU can deliver. Do not reach for it purely to increase throughput. If the model fits on one card, running several independent single-GPU replicas behind a load balancer almost always yields more aggregate tokens per second than one tensor-parallel instance, because it eliminates the collective communication entirely.
Why It Matters
- Makes the largest open-weight models runnable at full precision on hardware you can actually buy and rack today.
- Reduces single-request latency below what one GPU can achieve, which matters for interactive and agentic workloads.
- Interconnect topology dominates the outcome, so server and switch selection is a first-order decision rather than a detail.
- Costs aggregate throughput compared with independent replicas, so it should be used only when a model truly will not fit.
In Practice
A team bought four GPUs in a PCIe-only chassis expecting near-linear scaling from tensor parallelism and measured well under half the projected throughput. All-reduce traffic across PCIe was the bottleneck. Switching to four independent single-GPU replicas of a quantized model behind a load balancer nearly doubled aggregate tokens per second on identical hardware.
Frequently Asked Questions
When should I use tensor parallelism instead of more replicas?
Use tensor parallelism when the model will not fit in one GPU's memory even after quantization, or when a single request must return faster than one card allows. If the model fits on one card and your goal is serving more users, independent replicas behind a load balancer give higher aggregate throughput because they avoid per-layer collective communication entirely.
Do all GPUs need to be identical for tensor parallelism?
In practice yes. The work is split evenly across cards and every layer synchronizes, so the slowest GPU sets the pace for all of them and mismatched memory capacities complicate partitioning. Production deployments use homogeneous cards within a tensor-parallel group, though separate groups on different hardware can sit behind the same load balancer.
Related Terms
GPU VRAM
GPU VRAM is the high-bandwidth memory physically attached to a graphics processor. For AI serving it must simultaneously hold the model weights, the key-value cache for every active request, and activation buffers, which makes it the binding constraint on most deployments.
vLLM
vLLM is an open-source inference server for large language models that uses PagedAttention memory management and continuous batching to serve many concurrent users at high throughput, exposing an OpenAI-compatible HTTP API.
Inference
Inference is the phase in which a trained model is run to produce output from new input, as opposed to training, where the model's weights are being learned. In production, inference is where essentially all AI compute cost is spent.
Go Deeper
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.
On-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Working with Tensor Parallelism in a live environment? Our engineers do this every day - and our AI agents automate most of it.