# Why you get CUDA out of memory when nvidia-smi shows free VRAM

> Source: https://netray.co/troubleshoot/cuda-oom-despite-free-vram
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

This almost always means memory fragmentation: the allocator has enough total free memory but no single contiguous block large enough for the requested allocation, because the address space is broken into many small free-and-used segments from prior allocations of different sizes. The fix is enabling PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, reducing allocation size variability, or restarting the process to reset the address space.

## Error Title

CUDA out of memory even though nvidia-smi shows free VRAM

## Error Signature

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 24.00 GiB of which 3.21 GiB is free

## Alternate Signatures

- RuntimeError: CUDA out of memory, but nvidia-smi shows several GB free
- OutOfMemoryError despite low reported memory.used

## Category

gpu-memory

## Stack

- pytorch
- cuda

## Affects

Any CUDA GPU running PyTorch-based inference or training with long-running processes and variable tensor shapes

## Quick Fix

### Resolve a fragmentation-driven OOM

**Steps:**

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True as an environment variable before starting the process; this lets PyTorch grow existing memory segments instead of creating new, differently-sized ones that fragment the address space.

If the error persists, restart the process: fragmentation is a property of the current CUDA context's address space and does not survive a fresh process start.

Reduce variability in tensor shapes where possible, for example by padding batches to a fixed sequence length bucket instead of exact-length batching, so the allocator reuses same-sized blocks more often.

Lower the largest single allocation you are requesting (batch size, sequence length, or KV cache block size) so it can fit in the largest available contiguous free block even under current fragmentation.

Call torch.cuda.empty_cache() to release cached free blocks back to the driver, which can sometimes allow the driver to coalesce free space, though this is not guaranteed to fix fragmentation.

## Symptoms

- nvidia-smi memory.free reports several GB available, larger than the allocation size the error mentions
- Error mentions a modest allocation size (hundreds of MB to a few GB) failing despite apparently ample free memory
- The failure appears intermittently, often correlated with runs that have been serving variable-length requests for a while
- A fresh process restart with an identical workload succeeds where the long-running process failed

## Causes

### Repeated allocation and deallocation of differently sized tensors fragments the CUDA address space into many small non-contiguous free blocks

**Likelihood:** most-common

**Explanation:** PyTorch's caching allocator manages memory in segments; when tensors of many different sizes are allocated and freed over a long-running process (varying batch sizes, sequence lengths, or KV cache blocks), the free memory ends up scattered across many small gaps rather than one large contiguous region, so a new large allocation can fail even though the sum of free memory would be enough.

**Fix:** Enable expandable segments so the allocator grows existing segments in place instead of carving out new fixed-size ones, which substantially reduces this class of fragmentation in PyTorch 2.0 and later.

**Commands:**

- export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
- python your_script.py

### Variable-length sequences in inference or training create allocations of many different sizes back to back

**Likelihood:** common

**Explanation:** Serving requests of wildly different sequence lengths (say 50 tokens then 4000 tokens) forces the allocator to repeatedly find or create blocks of very different sizes, which is a primary driver of fragmentation compared to a workload with consistent batch and sequence shapes.

**Fix:** Bucket requests into a small number of fixed sequence-length tiers (pad to 512, 1024, 2048, and so on) so the allocator reuses a limited set of block sizes instead of a continuous spread.

**Commands:**

- max_length = min(2048, next_bucket_size(actual_length))

### A long-running server process accumulates fragmentation over hours or days without ever restarting

**Likelihood:** common

**Explanation:** Fragmentation is cumulative within a single CUDA context; a process that has been serving thousands of varied requests without restart has had far more opportunity to fragment its address space than one that just started, even at the same instantaneous memory_allocated() level.

**Fix:** Schedule periodic, low-traffic-window restarts of long-lived inference servers as a standing operational practice, treating fragmentation as a slow-accumulating resource similar to memory leaks in other long-lived services.

**Commands:**

- systemctl restart vllm-server

### Multiple models or workloads share one process and interleave differently sized allocations

**Likelihood:** occasional

**Explanation:** Running more than one model, or a model plus an embedding service, in the same Python process means their allocation patterns interleave in the same address space, increasing the variety of block sizes requested and therefore the fragmentation rate compared to a dedicated single-model process.

**Fix:** Isolate distinct workloads (different models, different batch profiles) into separate processes or containers, each with its own CUDA context and address space.

### An older PyTorch version predates the expandable_segments allocator improvement

**Likelihood:** rare

**Explanation:** expandable_segments was introduced in PyTorch 2.0 as a specific fix for this fragmentation pattern; versions before that lack the option entirely, so the same workload can be meaningfully more fragmentation-prone purely due to PyTorch version.

**Fix:** Upgrade to a current PyTorch release that supports expandable_segments, since this is a targeted allocator improvement rather than a general version bump.

**Commands:**

- pip install --upgrade torch

## Diagnostics

### Compare free reported by nvidia-smi against the largest contiguous free block PyTorch can find

**Command:** python -c "import torch; print(torch.cuda.memory_summary())"

**Interpretation:** memory_summary() breaks down allocated, reserved, and free memory by size class; if total free is large but no size class matches your allocation, fragmentation is confirmed as the cause.

### Check whether expandable_segments is currently enabled

**Command:** python -c "import os; print(os.environ.get('PYTORCH_CUDA_ALLOC_CONF'))"

**Interpretation:** If this is unset or does not include expandable_segments:True, that is the first and highest-value change to make before investigating further.

### Confirm the failure disappears after a clean process restart

**Command:** kill <pid>; nvidia-smi

**Interpretation:** If the exact same allocation succeeds immediately after a restart with no other changes, this strongly confirms accumulated fragmentation rather than a genuine capacity shortfall.

## Prevention

- Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True as a standing default in your deployment environment, not a reactive fix.
- Bucket or pad variable-length inputs into a small number of fixed shapes rather than allowing unbounded shape variability.
- Schedule periodic restarts of long-running inference processes as routine maintenance, especially for services handling highly variable request sizes.
- Keep PyTorch and CUDA toolkit versions current to benefit from ongoing allocator improvements.

## When To Escalate

If fragmentation-driven OOMs recur even with expandable_segments enabled, shape bucketing in place, and regular restarts scheduled, the workload's shape variability may genuinely exceed what a single-process allocator can manage gracefully, which is a signal to redesign request batching (for example routing very long and very short requests to separately sized worker pools) rather than continuing to tune allocator flags.

## Faq

### Why does PyTorch say there is not enough memory when nvidia-smi shows free VRAM?

**Answer:** nvidia-smi reports total free memory on the device, but PyTorch's allocator needs a single contiguous block large enough for the requested tensor. If free memory is scattered across many small gaps from prior allocations of different sizes, a request larger than any individual gap fails even though the sum of free memory would be plenty. This is memory fragmentation, and it is one of the most common causes of confusing OOM errors.

### What does PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True actually change?

**Answer:** It changes how PyTorch's caching allocator grows memory segments: instead of allocating new fixed-size blocks that end up scattered across the address space, it allows existing segments to expand contiguously as needed. This substantially reduces the class of fragmentation caused by variable allocation sizes over a long-running process, and is safe to enable by default on PyTorch 2.0 and later.

### Does restarting the process really fix fragmentation, or just hide the symptom?

**Answer:** It genuinely fixes it, at least until fragmentation accumulates again. A process restart destroys the CUDA context entirely, so the new context starts with a clean, unfragmented address space. It is not a permanent architectural fix, but it is a legitimate and commonly used operational mitigation, especially combined with expandable_segments to slow how quickly fragmentation reaccumulates.

### Does fragmentation affect inference and training equally?

**Answer:** Both are affected, but training workloads with highly variable sequence lengths and frequent allocation and deallocation cycles (each training step allocates fresh activation memory) tend to fragment faster than steady-state inference serving with a stable batch shape. Long-running inference servers handling highly variable request lengths, however, can fragment just as severely over time.

## Search terms

cuda out of memory but gpu has free memory, pytorch memory fragmentation, PYTORCH_CUDA_ALLOC_CONF expandable_segments, nvidia-smi shows free memory but oom, torch.cuda.memory_summary fragmentation, cuda oom intermittent long running process

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json