# Why pipeline parallelism scales poorly, and why the bubble, not the code, is usually the problem > Source: https://netray.co/troubleshoot/pipeline-parallelism-poor-scaling > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Pipeline parallelism scales poorly when the number of microbatches per training step is too small relative to the number of pipeline stages, because the unavoidable fill and drain bubble at the start and end of each step is proportional to the stage count minus one divided by the number of microbatches. With too few microbatches, GPUs spend a large fraction of every step idle waiting for the pipeline to fill or drain, and adding more stages without also adding more microbatches makes this worse, not better. ## Error Title Pipeline parallelism scales poorly, throughput does not improve with more stages ## Error Signature Pipeline parallel training throughput does not scale with additional stages, GPU utilization drops sharply as pipeline depth increases ## Alternate Signatures - Adding more pipeline parallel stages makes each training step slower per GPU, not faster - GPipe or PipeDream-style pipeline shows large idle gaps between microbatches in the profiler ## Category distributed ## Stack - deepspeed - pytorch - accelerate ## Affects DeepSpeed pipeline parallelism, Megatron-style pipeline parallel training, and any GPipe or PipeDream-style implementation, most visible above 4 pipeline stages ## Quick Fix ### Increase microbatches relative to pipeline stages before changing anything else **Steps:** Calculate your current bubble fraction as (pipeline_stages minus 1) divided by num_microbatches; anything above roughly 25 percent means the bubble is likely dominating your step time. Increase gradient_accumulation_steps or your framework's num_microbatches setting so it is at least 4 times your pipeline stage count, adjusting global batch size accordingly. If activation memory is the reason microbatches are limited, add or increase activation checkpointing to shrink per-microbatch memory footprint, freeing room for more, smaller microbatches. Check that layers are partitioned roughly evenly across stages, since one heavy stage sets the pace for the whole pipeline regardless of microbatch count. Profile with a pipeline trace or torch.profiler to confirm the idle gaps actually shrink after increasing microbatches, rather than assuming it worked. ## Symptoms - GPU utilization graphs show clear idle gaps at the start and end of every training step, worse with more pipeline stages - Throughput per GPU decreases as you increase pipeline parallel degree while holding total GPU count and batch size fixed - Increasing pipeline stages was expected to speed up training but instead made each step slower - The profiler trace shows large blocks of GPU idle time between microbatch forward and backward passes, especially on the first and last stages ## Causes ### Too few microbatches relative to the number of pipeline stages, so the fill and drain bubble dominates total step time **Likelihood:** most-common **Explanation:** Pipeline parallelism only overlaps stages once the pipeline is full: at the start of every step, stage 2 must wait for stage 1 to finish its first microbatch, stage 3 waits for stage 2, and so on, with a mirrored drain at the end. The fraction of step time spent in this unavoidable bubble is roughly the stage count minus one divided by the number of microbatches, so with only as many microbatches as stages, more than half the step can be idle bubble time, and adding stages without adding microbatches makes the ratio worse. **Fix:** Increase the number of microbatches, via gradient accumulation or your framework's num_microbatches setting, to several times your pipeline stage count; a common rule of thumb is 4x or more, so the bubble becomes a small fraction of total step time. ### Layers are partitioned unevenly across pipeline stages, so one heavy stage sets the pace for every other stage **Likelihood:** common **Explanation:** Pipeline parallel frameworks typically split a model into stages by layer count, not actual compute cost per layer. Embedding and output layers can be disproportionately expensive, especially with large vocabularies, or disproportionately cheap compared to a typical transformer block, so an even layer count split can still produce a badly imbalanced compute split, and every stage's throughput is capped by the slowest one. **Fix:** Manually rebalance the partition points based on measured per-layer compute time rather than layer count alone, giving the embedding and output stages fewer transformer blocks to compensate for their extra cost. ### Activation memory limits per-microbatch size, which caps how many microbatches can run without exceeding VRAM, directly worsening the bubble **Likelihood:** common **Explanation:** Smaller microbatches use less activation memory per stage but require more of them to reach a target global batch size; if activation memory is already tight, you may be forced into fewer, larger microbatches purely to fit in VRAM, which is exactly the configuration that maximizes bubble overhead. **Fix:** Add or increase activation checkpointing, recomputing activations during backward instead of storing them, to reduce per-microbatch memory footprint and free headroom for more, smaller microbatches. ### Pipeline parallelism is combined with tensor or data parallelism without accounting for the communication interaction between them **Likelihood:** occasional **Explanation:** In 3D-parallel setups, tensor-parallel all-reduce operations happen within each pipeline stage while pipeline stages themselves communicate activations forward and gradients backward between GPUs; if these are not scheduled to overlap properly, tensor-parallel communication can serialize with pipeline handoffs, adding latency on top of the pipeline bubble rather than hiding it. **Fix:** Use a framework with built-in support for combined 3D parallelism scheduling rather than hand-rolling the combination, since correct overlap of tensor-parallel and pipeline-parallel communication is a nontrivial scheduling problem. ### Slow interconnect between pipeline stages adds latency to every stage handoff **Likelihood:** rare **Explanation:** Every microbatch's activations and gradients must physically move between the GPUs assigned to consecutive pipeline stages; if that link is a slow PCIe path or an oversubscribed network rather than NVLink or InfiniBand, the transfer time itself adds to the bubble on top of the theoretical fill and drain overhead, and this effect is invisible in bubble-fraction math that assumes instant handoffs. **Fix:** Place adjacent pipeline stages on GPUs with the fastest available interconnect between them, same node or NVLink-connected where possible, and reserve cross-node links for the outermost or least-frequent communication boundaries. ## Diagnostics ### Calculate the theoretical bubble fraction for your current config **Command:** python -c "stages=8; microbatches=8; print((stages-1)/microbatches)" **Interpretation:** A result above roughly 0.25 to 0.3 means a large fraction of every step is unavoidable bubble at your current microbatch count; increasing microbatches directly lowers this number. ### Profile per-GPU utilization across a training step **Command:** nvidia-smi dmon -s u -d 1 **Interpretation:** Look for GPUs on the first and last pipeline stages showing long idle stretches at the start and end of each step; this visually confirms bubble overhead versus a genuine compute or communication bottleneck elsewhere. ### Check per-stage compute time balance **Command:** python -m torch.utils.bottleneck train.py **Interpretation:** Compare wall-clock time spent in each pipeline stage; a significantly slower stage, often the embedding or output layer stage, indicates a partitioning imbalance rather than, or in addition to, a bubble problem. ## Prevention - Set microbatch count relative to pipeline depth as a deliberate ratio (4x or higher) in your training configuration, not an afterthought left at the default. - Rebalance pipeline stage boundaries based on measured per-layer timing whenever you change model architecture or vocabulary size, not just layer count. - Budget activation checkpointing into your memory plan from the start so it does not become a forced tradeoff against microbatch count later. - When combining pipeline parallelism with tensor or data parallelism, use a framework with tested 3D-parallel scheduling rather than a custom implementation. ## When To Escalate If you have already maximized microbatch count within memory limits and rebalanced stages, and the bubble is still large because your model is too small to justify the pipeline depth you are using, the right fix is usually reducing pipeline parallel degree and relying more on data or tensor parallelism, which is a parallelism-strategy decision worth an architecture review rather than continued pipeline tuning. ## Faq ### How many microbatches do I actually need for good pipeline parallel efficiency? **Answer:** As a rule of thumb, aim for at least 4 times your pipeline stage count, though the exact number depends on how much bubble overhead you can tolerate. The bubble fraction is the stage count minus one divided by microbatches, so more microbatches always reduces it, subject to the memory and compute cost of managing more, smaller units of work. ### Does pipeline parallelism ever make sense for a small number of GPUs? **Answer:** It can, but the bubble overhead is proportionally worse with few microbatches relative to stages, so pipeline parallelism tends to pay off more at larger scale where you can afford enough microbatches to amortize the bubble. For a small GPU count, tensor or data parallelism alone often scales better with less tuning complexity. ### Why does GPU utilization look fine on average but training is still slow? **Answer:** Average utilization over a whole run can hide short, repeated idle bursts at the start and end of every pipeline step; look at utilization on a per-step or sub-second timescale, and specifically on the first and last pipeline stages, rather than a coarse average, to see the bubble pattern clearly. ## Search terms pipeline parallelism poor scaling, pipeline parallel bubble overhead, gpipe throughput not scaling, pipeline parallel gpu idle, microbatches pipeline parallelism, deepspeed pipeline parallel slow --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json