Why a long-running LLM service slowly leaks memory, and how to fix it
memory usage climbs steadily over hours or days until the service crashes or is restarted
Also appears as
- gpu memory grows slowly during normal operation until it runs out
- llm server needs periodic restarts to stay stable
Short answer
Slow memory growth over hours or days in an LLM service is rarely a leak in the model itself, it is almost always one of: a KV cache pool that grows because completed sequences are not being freed correctly, client sessions or connections that are opened but never closed, LoRA adapters that accumulate in memory across many fine-tuned variants without eviction, or memory fragmentation that reduces effectively usable memory even though nothing is technically leaked. Isolating which of these it is requires tracking memory over time correlated with request volume, adapter count, and connection count separately.
Affects: Long-running self-hosted LLM servers, especially multi-tenant or multi-LoRA deployments running for days without restart
Isolate the leak before patching anything
- 1Graph GPU and host memory usage over several hours alongside request rate; if memory grows even during low-traffic periods, suspect connections or adapters rather than KV cache.
- 2Check the number of loaded LoRA adapters over time if you are serving multi-LoRA; if it only grows and never shrinks, adapters are accumulating without eviction.
- 3Check client-side and gateway connection counts and open file descriptors; a steadily rising count with flat request volume points to unclosed sessions.
- 4Restart the process and immediately compare fresh memory usage to pre-restart usage at the same request volume; a large gap confirms something is not being released during normal operation.
- 5If GPU memory specifically grows without a clear cause, check for fragmentation by comparing nvidia-smi's reported used memory against the process's own memory allocator statistics.
How to confirm this is your problem
- Memory usage graph trends steadily upward over hours or days rather than staying flat at steady traffic
- Periodic manual or automated restarts are required to keep the service stable
- Memory growth continues even during periods of low or no request traffic
- OOM crashes happen at unpredictable times rather than correlating with peak load
Root causes and fixes
KV cache pool grows because completed sequences are not freed correctly
If request completion, cancellation, or client disconnect is not handled cleanly, a sequence's KV cache blocks can remain allocated in the pool even after the sequence is logically done, slowly consuming available memory across thousands of requests until none is left for new sequences.
Fix: Confirm the serving framework's request completion and disconnect handling correctly frees KV cache blocks, check for a known issue in your serving framework version around this, and upgrade if a fix exists; add explicit timeout and cleanup handling for abandoned or cancelled requests.
Client sessions or HTTP connections opened but never closed
Client libraries that keep persistent connections (HTTP keep-alive, gRPC channels, websocket sessions) will accumulate open connections and their associated buffers if the calling code creates a new client per request instead of reusing or properly closing one, growing host memory and file descriptor usage steadily over time.
Fix: Reuse a single client/session object across requests instead of creating one per call, and ensure any explicitly created sessions are closed in a finally block or context manager.
python -c "import resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))"
LoRA adapters accumulate in memory across many fine-tuned variants without eviction
Multi-LoRA serving setups that load adapters on demand but never unload inactive ones will keep adding adapter weights to GPU memory as new adapter IDs are requested, and without an LRU-style eviction policy this grows unbounded as the number of distinct adapters used over the service's lifetime increases.
Fix: Enable or implement adapter eviction (LRU or similar) so inactive adapters are unloaded from GPU memory after a period of disuse, and cap the maximum number of concurrently loaded adapters explicitly.
GPU memory fragmentation accumulates over days of varied allocation sizes
Repeated allocation and deallocation of GPU memory blocks of varying sizes (varying sequence lengths, adapter sizes, batch compositions) can fragment the memory allocator's free space over time, so that even though total freed memory is sufficient, no single contiguous block is large enough for a new allocation, which looks like a leak but is actually fragmentation.
Fix: Prefer serving frameworks with block-based/paged memory management for KV cache specifically to avoid this class of fragmentation, and if using raw PyTorch allocation, periodically monitor torch.cuda.memory_stats() for fragmentation indicators.
python -c "import torch; print(torch.cuda.memory_stats())"
Application-level caches (prompt cache, response cache, embedding cache) grow without a size limit
Custom caching layers added around the model server, for example an in-process cache of recent prompts or embeddings, that lack an eviction policy or maximum size will grow proportionally with unique traffic over the service's uptime, consuming host or GPU memory slowly but steadily.
Fix: Add an explicit maximum size and eviction policy (LRU, TTL) to any custom cache in the request path, and monitor its size as a first-class metric.
Diagnostic commands
Track memory usage over a multi-hour window correlated with request volume
nvidia-smi --query-gpu=memory.used --format=csv -l 60
Memory that grows in step with request volume and then plateaus is likely expected caching behavior; memory that keeps growing even at flat or low request volume strongly suggests a real leak somewhere in the request lifecycle.
Check loaded adapter count over time in multi-LoRA deployments
curl -s localhost:8000/v1/models | python -m json.tool
If the number of concurrently loaded adapters only ever increases and never decreases, there is no eviction policy in place and adapter accumulation is a likely contributor to memory growth.
Check open file descriptor and connection counts on the client and server
ls /proc/$(pgrep -f vllm)/fd | wc -l
A file descriptor count that climbs steadily with uptime rather than staying roughly proportional to current active connections points to unclosed client sessions or connections leaking on either side.
Stopping it from happening again
- Add memory usage, adapter count, and connection count as first-class long-running dashboards, not just point-in-time checks
- Reuse HTTP/gRPC clients across requests instead of creating a new one per call
- Cap and evict inactive LoRA adapters explicitly rather than relying on unbounded on-demand loading
- Schedule a canary restart cadence during low-traffic windows as a safety net while root-causing a suspected leak, without treating it as the permanent fix
When this becomes an architecture problem
If memory growth persists after ruling out client sessions, adapter accumulation, and application caches, and profiling points to the serving framework's own internals (a genuine bug in KV cache lifecycle management), the fix requires either a framework upgrade, a workaround at the deployment layer (scheduled rolling restarts as a stopgap), or engaging with the framework's maintainers, which moves beyond configuration into infrastructure and vendor management.
Frequently asked questions
Is a scheduled restart an acceptable permanent fix for this?
It is a reasonable stopgap to maintain availability while the actual cause is investigated, but it masks the underlying problem and risks dropping in-flight requests at restart time. Treat it as a temporary safety net, not a substitute for finding and fixing the actual source of growth.
How do I tell a leak apart from normal cache warm-up?
Normal cache warm-up plateaus once the cache reaches its working set size and traffic patterns stabilize; a genuine leak keeps growing indefinitely, including during flat or low-traffic periods. Graph memory over a long enough window (many hours to a day) to distinguish a plateau from continued growth.
Can PyTorch's memory allocator itself cause this?
PyTorch's caching allocator holds onto freed memory for reuse rather than returning it to the OS immediately, which can look like a leak in tools that only report process memory, but it usually stabilizes at a working-set size. Use torch.cuda.memory_stats() rather than raw process memory to distinguish allocator caching from an actual leak.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Related problems
GPU memory stays full after inference finishes
This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call.
PyTorch GPU memory fragmentation causing intermittent OOM
PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes.
vLLM multi-LoRA serving fails to load or apply an adapter
Multi-LoRA serving in vLLM requires the server to be launched with --enable-lora plus explicit capacity flags such as max-lora-rank, max-loras, and max-cpu-loras sized to your actual adapters, and each adapter must be registered by name with --lora-modules so requests can reference it. Mismatches between an adapter's real rank or count and what the server was configured for produce hard failures rather than silent truncation.
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.