# Why GPU memory stays full after inference finishes, and whether it is actually a leak > Source: https://netray.co/troubleshoot/gpu-memory-not-released-after-inference > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call. ## Error Title GPU memory stays full after inference finishes ## Error Signature nvidia-smi still shows the process holding X GiB of GPU memory after the script has finished running or the request has completed ## Alternate Signatures - CUDA memory not freed after del model - GPU memory usage does not decrease after torch.cuda.empty_cache() ## Category gpu-memory ## Stack - pytorch - cuda ## Affects Any long-running Python process using PyTorch or a serving framework built on it, including vLLM, transformers pipelines, and custom inference servers ## Quick Fix ### Confirm whether this is normal caching or a real leak **Steps:** Compare torch.cuda.memory_allocated() (memory actually in use) against torch.cuda.memory_reserved() (what nvidia-smi shows); a large gap between them is the caching allocator working as designed, not a leak. If you need the memory back for another process on the same GPU, call torch.cuda.empty_cache() to release the unused cached blocks back to the driver. If memory_allocated() itself grows across repeated inference calls (not just memory_reserved()), you have a real leak: check for tensors kept alive by Python references such as accumulating lists, un-detached outputs, or a growing KV cache in a custom serving loop. For a hard reset between workloads on the same process, del the model object, call gc.collect(), then torch.cuda.empty_cache(), in that order. If nothing but ending the process itself frees the memory, the process is the correct unit of isolation: run separate inference jobs in separate processes rather than reusing one long-lived process for unrelated models. ## Symptoms - nvidia-smi memory.used for the process stays constant and high well after the last request finished - torch.cuda.empty_cache() reduces the nvidia-smi number somewhat but not fully - Restarting the process (not just the model) reliably frees the memory, confirming it is process-scoped not model-scoped - memory_allocated() is much lower than memory_reserved() when checked interactively ## Causes ### PyTorch's caching allocator intentionally retains freed GPU memory for reuse rather than returning it to the CUDA driver **Likelihood:** most-common **Explanation:** Requesting memory from the CUDA driver (cudaMalloc) is expensive relative to typical tensor lifetimes, so PyTorch's allocator caches previously freed blocks and reuses them for future allocations of similar size, without a driver round trip. nvidia-smi reports memory reserved by the process's CUDA context, which includes this cache, not just currently-live tensors. **Fix:** This is not something to fix; it is the intended design. If you specifically need the memory available to another process on the same GPU, call torch.cuda.empty_cache() to return the unused cached blocks to the driver. **Commands:** - python -c "import torch; print(torch.cuda.memory_allocated()/1e9, torch.cuda.memory_reserved()/1e9)" ### A reference to output tensors, hidden states, or a KV cache is kept alive somewhere in the serving loop **Likelihood:** common **Explanation:** If a list, a global variable, or a logging call retains a reference to a GPU tensor from each request, Python's garbage collector cannot free the underlying CUDA memory even after the request completes, so memory_allocated() itself climbs request over request rather than plateauing. **Fix:** Audit the serving loop for accumulating structures (metrics buffers, debug logs, caches) that hold onto tensors, and explicitly .detach().cpu() or del any tensor you do not need to keep on GPU past the request. **Commands:** - output = output.detach().cpu() - del hidden_states; import gc; gc.collect() ### Autograd graphs are retained because inference code was not run inside torch.no_grad() **Likelihood:** occasional **Explanation:** Without torch.no_grad(), every forward pass builds a graph of intermediate activations needed for a hypothetical backward pass, and those activations are kept alive as long as anything references the output, multiplying real memory use for inference workloads that will never call backward(). **Fix:** Wrap all inference-only forward passes in torch.no_grad() or use model.eval() combined with torch.inference_mode(), which additionally disables autograd bookkeeping entirely. **Commands:** - with torch.inference_mode(): - output = model.generate(**inputs) ### A CUDA stream or context from a crashed or killed thread is not cleaned up **Likelihood:** occasional **Explanation:** If a worker thread or subprocess is killed (not gracefully exited) while holding CUDA allocations, the parent process's CUDA context can retain those allocations until the process itself terminates, since the driver associates memory with the process, not the thread. **Fix:** Ensure worker processes shut down cleanly (catch signals, call cleanup handlers) rather than being SIGKILLed, and treat a killed worker's memory as unrecoverable until the whole process restarts. **Commands:** - nvidia-smi --query-compute-apps=pid,used_memory --format=csv ### Memory fragmentation makes the allocator hold more reserved memory than the live tensor set would otherwise require **Likelihood:** rare **Explanation:** Repeated allocation and deallocation of differently-sized tensors (variable batch sizes or sequence lengths) can leave the allocator's free blocks too fragmented to satisfy new requests from cache, forcing it to request and retain additional driver memory even though total live tensor memory is unchanged. **Fix:** Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation-driven over-reservation, or standardize batch/sequence shapes where practical. **Commands:** - export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ## Diagnostics ### Separate real usage from cached reservation **Command:** python -c "import torch; print('allocated', torch.cuda.memory_allocated()/1e9); print('reserved', torch.cuda.memory_reserved()/1e9)" **Interpretation:** allocated is live tensor memory; reserved is what nvidia-smi shows. If reserved is much higher than allocated, this is normal caching. If allocated itself is high and climbing across requests, you have a genuine reference leak. ### Track allocated memory across repeated identical requests **Command:** python -c "import torch; [print(i, torch.cuda.memory_allocated()/1e9) for i in range(5)]" # run inside your request loop **Interpretation:** A flat or oscillating number confirms normal steady-state behavior; a monotonically increasing number confirms a leak that needs code-level investigation, not an allocator setting. ### Get a full memory snapshot with allocation history **Command:** python -c "import torch; torch.cuda.memory._record_memory_history(); torch.cuda.memory._dump_snapshot('snapshot.pickle')" **Interpretation:** The snapshot, viewable with PyTorch's memory visualizer, shows exactly which allocation call sites are holding memory, pinpointing the leaking reference instead of guessing. ## Prevention - Always wrap inference-only code in torch.inference_mode() rather than relying on model.eval() alone. - Add a periodic assertion in long-running services that memory_allocated() has not grown beyond an expected ceiling across N requests. - Avoid global accumulator lists or logging calls that capture raw GPU tensors instead of detached, CPU-side summaries. - Treat memory_reserved() as an operational metric for capacity planning and memory_allocated() as the correctness metric for leak detection; do not conflate the two. ## When To Escalate If memory_allocated() itself is provably climbing after code review rules out obvious reference leaks, and the growth correlates with a specific library (a custom KV cache implementation, a third-party inference wrapper) rather than your own code, this becomes a library-level investigation or a case for isolating that component in its own restart-able process rather than a quick fix. ## Faq ### Is it normal for nvidia-smi to show high memory usage even when no requests are running? **Answer:** Yes. PyTorch's caching allocator keeps freed memory reserved for fast reuse rather than returning it to the CUDA driver after every tensor is freed. This shows up in nvidia-smi as memory.used for your process even during idle periods, and is expected behavior, not a bug or leak. ### What is the difference between torch.cuda.memory_allocated() and memory_reserved()? **Answer:** memory_allocated() is the memory currently occupied by live tensors your code is actually using. memory_reserved() is the total memory PyTorch's allocator has claimed from the CUDA driver, including cached blocks kept around for future allocations. nvidia-smi's per-process figure corresponds to memory_reserved(), not memory_allocated(). ### Does torch.cuda.empty_cache() hurt performance if I call it after every request? **Answer:** Yes, calling it frequently defeats the purpose of the caching allocator and forces expensive driver-level memory operations on the next allocation, typically slowing throughput. Only call it when you specifically need to free memory for another process sharing the GPU, not as a routine part of a request handler. ### Why does restarting the process fix memory issues that empty_cache() does not? **Answer:** A process restart tears down the entire CUDA context, unconditionally releasing every allocation including ones the caching allocator or a stuck reference would otherwise keep alive. This is why process-level isolation (one model per process, restartable independently) is a more reliable operational pattern than trying to fully reset state within a single long-lived process. ## Search terms gpu memory not released after inference, pytorch memory not freed, nvidia-smi memory stuck, torch.cuda.empty_cache not working, cuda memory leak inference, memory_allocated vs memory_reserved, gpu memory stays full after script ends --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json