# Why PyTorch GPU memory fragmentation causes intermittent CUDA out of memory errors > Source: https://netray.co/troubleshoot/gpu-memory-fragmentation-pytorch > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes. ## Error Title PyTorch GPU memory fragmentation causing intermittent OOM ## Error Signature RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU 0; ...; Y MiB free; ...); see documentation for PYTORCH_CUDA_ALLOC_CONF ## Alternate Signatures - torch.cuda.OutOfMemoryError: ... this may be caused by fragmentation. See PYTORCH_CUDA_ALLOC_CONF documentation - CUDA memory allocation failed even though total free memory should be sufficient ## Category gpu-memory ## Stack - pytorch - cuda ## Affects Long-running PyTorch processes with variable tensor shapes, including inference servers, training loops with variable-length batches, and multi-tenant model hosting ## Quick Fix ### Stop fragmentation-driven OOMs **Steps:** Set the environment variable before process start: export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which allows the allocator to grow segments contiguously instead of fragmenting the address space with new fixed-size ones. Restart the affected process; expandable_segments changes allocator behavior going forward, it does not defragment memory already reserved under the old behavior. Normalize input shapes by padding or bucketing batches into a small number of fixed sizes, reducing the variety of allocation sizes the allocator has to manage. If using an older PyTorch, also try PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 (or a value tuned to your typical allocation size) to limit how large a single cached block can be before it is split, reducing certain fragmentation patterns. For serving workloads, schedule periodic process restarts as a standing operational practice, since fragmentation accumulates over the lifetime of a process regardless of these settings. ## Symptoms - Error message explicitly references PYTORCH_CUDA_ALLOC_CONF or fragmentation in its text - Failures are intermittent and correlate with process uptime rather than a fixed workload size - torch.cuda.memory_summary() shows many small free blocks rather than one large contiguous region - The identical allocation succeeds immediately after a process restart ## Causes ### Highly variable allocation sizes over a long process lifetime scatter free memory into many differently-sized, non-adjacent blocks **Likelihood:** most-common **Explanation:** PyTorch's allocator manages memory as segments it requests from CUDA and then subdivides for individual tensors. When allocation sizes vary a lot (different batch sizes, sequence lengths, or intermediate tensor shapes across different requests), freed blocks end up sized for their original tensor rather than for whatever comes next, so new large requests may not find a matching free block even when total free memory is ample. **Fix:** Enable expandable_segments so segments can grow to accommodate new allocation sizes in place instead of forcing the allocator to carve out new differently-sized segments. **Commands:** - export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ### max_split_size_mb is left at its default, allowing large blocks to be split into many small pieces that cannot easily recombine **Likelihood:** common **Explanation:** By default, PyTorch may split a large cached block to satisfy a smaller allocation request, and those split pieces are not automatically recombined when freed, gradually reducing the largest available contiguous block over time even as total free memory stays constant. **Fix:** Set max_split_size_mb to a value close to your largest typical allocation size, preventing the allocator from creating excessively small split fragments that never recombine into something useful for future large allocations. **Commands:** - export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256 ### The process has been running for an extended period without restart, accumulating fragmentation **Likelihood:** common **Explanation:** Fragmentation is a property of the CUDA context's address space and accumulates over the lifetime of a process; a server that has served requests continuously for days has had far more opportunity to fragment than a freshly started one, even at an identical instantaneous memory footprint. **Fix:** Build periodic restarts into your operational runbook for long-lived inference servers, treating fragmentation similarly to a slow memory leak that resets cleanly on restart. ### Multiple distinct workloads with different tensor shape profiles share the same process and CUDA context **Likelihood:** occasional **Explanation:** Running an embedding model alongside a generation model, or batching requests from very different client applications with different sequence length distributions, in the same process multiplies the variety of allocation sizes competing for the same address space, accelerating fragmentation compared to a single, uniform workload. **Fix:** Isolate workloads with meaningfully different shape profiles into separate processes, each with its own CUDA context and address space, even if it means running more small processes instead of one large one. ### An outdated PyTorch version lacks the expandable_segments allocator option entirely **Likelihood:** rare **Explanation:** expandable_segments was added as a targeted fix for this fragmentation class starting with PyTorch 2.0; workloads on older versions cannot use it and are more exposed to fragmentation regardless of shape normalization efforts. **Fix:** Upgrade PyTorch to a current release to gain access to the improved allocator behavior, checking that your CUDA toolkit version remains compatible with the new PyTorch build. **Commands:** - pip install --upgrade torch ## Diagnostics ### Inspect free block sizes directly **Command:** python -c "import torch; print(torch.cuda.memory_summary())" **Interpretation:** The summary table breaks down memory by size class (blocks under 1MB, 1-10MB, and so on); many small free blocks with no large ones present is a direct signature of fragmentation. ### Check whether expandable_segments is active **Command:** python -c "import os; print(os.environ.get('PYTORCH_CUDA_ALLOC_CONF', 'not set'))" **Interpretation:** If unset, this is very likely the single highest-value change available before deeper investigation. ### Reproduce with a controlled shape sweep **Command:** python -c "import torch; a=[torch.randn(i*1000000, device='cuda') for i in range(1,50)]; del a; torch.cuda.empty_cache(); print(torch.cuda.memory_summary())" **Interpretation:** This synthetic test deliberately fragments memory with varying sizes; comparing memory_summary() before and after with expandable_segments toggled demonstrates the effect directly on your own PyTorch version. ## Prevention - Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True as a default environment setting across all training and serving deployments, not an incident response. - Standardize input shapes through padding or bucketing wherever request-level variability allows it. - Build scheduled restarts into operational runbooks for any long-running inference process. - Keep PyTorch versions current to benefit from ongoing allocator improvements around this exact problem. ## When To Escalate If fragmentation persists as a recurring production issue even with expandable_segments enabled, shapes normalized, and restarts scheduled, consider whether the workload's inherent shape diversity (for example, one service handling both short chat completions and long document summarization) should be split into separately provisioned serving pools rather than continuing to fight the allocator. ## Faq ### What exactly does PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True do? **Answer:** It changes PyTorch's CUDA memory allocator so that memory segments can grow contiguously to accommodate new allocation sizes, instead of the allocator creating new, separately sized segments each time a request does not match an existing free block. This directly targets the scenario where varying tensor sizes over a long-running process leave free memory scattered into unusable small gaps. ### Does this setting have any downside? **Answer:** The main tradeoff is a small amount of additional bookkeeping overhead and it requires PyTorch 2.0 or later plus a compatible CUDA driver. For the large majority of workloads experiencing fragmentation-related OOMs, the memory efficiency gain substantially outweighs this cost, which is why it is increasingly recommended as a default rather than a special-case fix. ### Why does the OOM error specifically mention fragmentation and point at this setting? **Answer:** PyTorch's allocator can detect the specific pattern of failing to satisfy an allocation despite sufficient total free memory, which is the signature of fragmentation rather than genuine memory exhaustion, and it surfaces this in the error message with a pointer to the relevant configuration documentation so you do not have to guess at the root cause. ### Will expandable_segments fix an OOM caused by the model genuinely being too large for the GPU? **Answer:** No. If your total free memory is genuinely insufficient for the requested allocation, no allocator setting changes the underlying math; expandable_segments only helps when adequate free memory exists but is unusable due to fragmentation. Confirm with torch.cuda.memory_summary() that total free memory is actually sufficient before expecting this setting to help. ## Search terms PYTORCH_CUDA_ALLOC_CONF expandable_segments, pytorch memory fragmentation oom, cuda out of memory fragmentation, max_split_size_mb, torch.cuda.memory_summary fragmented blocks, intermittent cuda oom long running process --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json