# Why a long-running LLM service slowly leaks memory, and how to fix it > Source: https://netray.co/troubleshoot/llm-service-memory-leak > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Slow memory growth over hours or days in an LLM service is rarely a leak in the model itself, it is almost always one of: a KV cache pool that grows because completed sequences are not being freed correctly, client sessions or connections that are opened but never closed, LoRA adapters that accumulate in memory across many fine-tuned variants without eviction, or memory fragmentation that reduces effectively usable memory even though nothing is technically leaked. Isolating which of these it is requires tracking memory over time correlated with request volume, adapter count, and connection count separately. ## Error Title GPU or host memory usage keeps growing in a long-running LLM service ## Error Signature memory usage climbs steadily over hours or days until the service crashes or is restarted ## Alternate Signatures - gpu memory grows slowly during normal operation until it runs out - llm server needs periodic restarts to stay stable ## Category performance ## Stack - vllm - pytorch - cuda ## Affects Long-running self-hosted LLM servers, especially multi-tenant or multi-LoRA deployments running for days without restart ## Quick Fix ### Isolate the leak before patching anything **Steps:** Graph GPU and host memory usage over several hours alongside request rate; if memory grows even during low-traffic periods, suspect connections or adapters rather than KV cache. Check the number of loaded LoRA adapters over time if you are serving multi-LoRA; if it only grows and never shrinks, adapters are accumulating without eviction. Check client-side and gateway connection counts and open file descriptors; a steadily rising count with flat request volume points to unclosed sessions. Restart the process and immediately compare fresh memory usage to pre-restart usage at the same request volume; a large gap confirms something is not being released during normal operation. If GPU memory specifically grows without a clear cause, check for fragmentation by comparing nvidia-smi's reported used memory against the process's own memory allocator statistics. ## Symptoms - Memory usage graph trends steadily upward over hours or days rather than staying flat at steady traffic - Periodic manual or automated restarts are required to keep the service stable - Memory growth continues even during periods of low or no request traffic - OOM crashes happen at unpredictable times rather than correlating with peak load ## Causes ### KV cache pool grows because completed sequences are not freed correctly **Likelihood:** most-common **Explanation:** If request completion, cancellation, or client disconnect is not handled cleanly, a sequence's KV cache blocks can remain allocated in the pool even after the sequence is logically done, slowly consuming available memory across thousands of requests until none is left for new sequences. **Fix:** Confirm the serving framework's request completion and disconnect handling correctly frees KV cache blocks, check for a known issue in your serving framework version around this, and upgrade if a fix exists; add explicit timeout and cleanup handling for abandoned or cancelled requests. ### Client sessions or HTTP connections opened but never closed **Likelihood:** common **Explanation:** Client libraries that keep persistent connections (HTTP keep-alive, gRPC channels, websocket sessions) will accumulate open connections and their associated buffers if the calling code creates a new client per request instead of reusing or properly closing one, growing host memory and file descriptor usage steadily over time. **Fix:** Reuse a single client/session object across requests instead of creating one per call, and ensure any explicitly created sessions are closed in a finally block or context manager. **Commands:** - python -c "import resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))" ### LoRA adapters accumulate in memory across many fine-tuned variants without eviction **Likelihood:** common **Explanation:** Multi-LoRA serving setups that load adapters on demand but never unload inactive ones will keep adding adapter weights to GPU memory as new adapter IDs are requested, and without an LRU-style eviction policy this grows unbounded as the number of distinct adapters used over the service's lifetime increases. **Fix:** Enable or implement adapter eviction (LRU or similar) so inactive adapters are unloaded from GPU memory after a period of disuse, and cap the maximum number of concurrently loaded adapters explicitly. ### GPU memory fragmentation accumulates over days of varied allocation sizes **Likelihood:** occasional **Explanation:** Repeated allocation and deallocation of GPU memory blocks of varying sizes (varying sequence lengths, adapter sizes, batch compositions) can fragment the memory allocator's free space over time, so that even though total freed memory is sufficient, no single contiguous block is large enough for a new allocation, which looks like a leak but is actually fragmentation. **Fix:** Prefer serving frameworks with block-based/paged memory management for KV cache specifically to avoid this class of fragmentation, and if using raw PyTorch allocation, periodically monitor torch.cuda.memory_stats() for fragmentation indicators. **Commands:** - python -c "import torch; print(torch.cuda.memory_stats())" ### Application-level caches (prompt cache, response cache, embedding cache) grow without a size limit **Likelihood:** rare **Explanation:** Custom caching layers added around the model server, for example an in-process cache of recent prompts or embeddings, that lack an eviction policy or maximum size will grow proportionally with unique traffic over the service's uptime, consuming host or GPU memory slowly but steadily. **Fix:** Add an explicit maximum size and eviction policy (LRU, TTL) to any custom cache in the request path, and monitor its size as a first-class metric. ## Diagnostics ### Track memory usage over a multi-hour window correlated with request volume **Command:** nvidia-smi --query-gpu=memory.used --format=csv -l 60 **Interpretation:** Memory that grows in step with request volume and then plateaus is likely expected caching behavior; memory that keeps growing even at flat or low request volume strongly suggests a real leak somewhere in the request lifecycle. ### Check loaded adapter count over time in multi-LoRA deployments **Command:** curl -s localhost:8000/v1/models | python -m json.tool **Interpretation:** If the number of concurrently loaded adapters only ever increases and never decreases, there is no eviction policy in place and adapter accumulation is a likely contributor to memory growth. ### Check open file descriptor and connection counts on the client and server **Command:** ls /proc/$(pgrep -f vllm)/fd | wc -l **Interpretation:** A file descriptor count that climbs steadily with uptime rather than staying roughly proportional to current active connections points to unclosed client sessions or connections leaking on either side. ## Prevention - Add memory usage, adapter count, and connection count as first-class long-running dashboards, not just point-in-time checks - Reuse HTTP/gRPC clients across requests instead of creating a new one per call - Cap and evict inactive LoRA adapters explicitly rather than relying on unbounded on-demand loading - Schedule a canary restart cadence during low-traffic windows as a safety net while root-causing a suspected leak, without treating it as the permanent fix ## When To Escalate If memory growth persists after ruling out client sessions, adapter accumulation, and application caches, and profiling points to the serving framework's own internals (a genuine bug in KV cache lifecycle management), the fix requires either a framework upgrade, a workaround at the deployment layer (scheduled rolling restarts as a stopgap), or engaging with the framework's maintainers, which moves beyond configuration into infrastructure and vendor management. ## Faq ### Is a scheduled restart an acceptable permanent fix for this? **Answer:** It is a reasonable stopgap to maintain availability while the actual cause is investigated, but it masks the underlying problem and risks dropping in-flight requests at restart time. Treat it as a temporary safety net, not a substitute for finding and fixing the actual source of growth. ### How do I tell a leak apart from normal cache warm-up? **Answer:** Normal cache warm-up plateaus once the cache reaches its working set size and traffic patterns stabilize; a genuine leak keeps growing indefinitely, including during flat or low-traffic periods. Graph memory over a long enough window (many hours to a day) to distinguish a plateau from continued growth. ### Can PyTorch's memory allocator itself cause this? **Answer:** PyTorch's caching allocator holds onto freed memory for reuse rather than returning it to the OS immediately, which can look like a leak in tools that only report process memory, but it usually stabilizes at a working-set size. Use torch.cuda.memory_stats() rather than raw process memory to distinguish allocator caching from an actual leak. ## Search terms llm service memory leak, gpu memory grows over time, vllm memory leak, lora adapter memory accumulation, long running llm server crash, kv cache not freed --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json