Why Ollama runs out of memory, and how to fix it
Error: model requires more system memory (5.4 GiB) than is available (3.9 GiB)
Also appears as
- CUDA error: out of memory
- Error: llama runner process has terminated: signal: killed
- ollama: Killed
Short answer
Ollama out of memory happens when the model's weights plus its KV cache exceed either GPU VRAM or system RAM, and the OOM killer or CUDA allocator terminates the process. The fix depends on which resource is exhausted: reduce quantization or context length for VRAM limits, or reduce concurrent model loads and context for system RAM limits.
Affects: Ollama on GPU and CPU-only hosts, most common when running models close to the size of available VRAM or RAM
Fix it in 60 seconds
- 1Check dmesg or the system logs for an OOM-killer entry to confirm whether this is system RAM or GPU VRAM exhaustion.
- 2Run nvidia-smi to see current VRAM usage versus total before loading the model.
- 3If VRAM is the constraint, switch to a smaller quantization (Q4_K_M instead of Q8_0 or FP16) or a smaller parameter count model.
- 4Reduce num_ctx to lower the KV cache size, since context length directly multiplies memory usage per request.
- 5If multiple models are loaded simultaneously, unload unused ones with ollama stop <model> before loading a new one.
How to confirm this is your problem
- Ollama reports a specific error stating the model needs more memory than is available before it even starts generating
- The ollama process is killed abruptly with no application-level error, visible only as Killed in the terminal or an OOM entry in dmesg
- A model that worked at a shorter context length fails once conversation history grows
- Loading a second model while the first is still resident causes an immediate failure
Root causes and fixes
Model plus KV cache exceeds VRAM
GPU memory has to hold model weights, the KV cache for the current context, and some working memory for the forward pass all at once; KV cache size scales with batch size times context length times the number of layers and attention heads, so a model that fits at a short context can OOM once num_ctx or a long conversation grows the cache beyond free VRAM.
Fix: Reduce num_ctx to the minimum your use case needs, or move to a smaller quantization so more VRAM headroom remains for the cache.
nvidia-smi --query-gpu=memory.used,memory.total --format=csv ollama run <model> --verbose
System RAM insufficient for CPU-offloaded layers
When a model does not fully fit in VRAM, Ollama offloads the remainder to system RAM; if total system RAM is also small relative to the model size, the offloaded portion itself cannot be allocated and the OS OOM killer terminates the process.
Fix: Either free up system RAM by closing other processes, add swap as a temporary buffer, or choose a smaller model/quantization that fits the combined VRAM and RAM budget more comfortably.
free -h dmesg | grep -i 'out of memory'
Multiple models or concurrent requests loaded at once
Ollama can keep more than one model resident in memory, and may also handle concurrent request contexts; each additional resident model or concurrent context adds its own KV cache on top of shared weight memory, which can exhaust VRAM even when a single model alone would fit comfortably.
Fix: Unload models you are not actively using with ollama stop, and if serving concurrent users, explicitly manage or limit the number of parallel model instances Ollama keeps loaded.
ollama ps ollama stop <model>
Very large context request in a single call
A single API call that passes an unusually large prompt, well beyond typical usage, forces a correspondingly large KV cache allocation in one shot; if that allocation alone exceeds free VRAM, the request fails immediately regardless of how well-sized the base model is.
Fix: Cap the maximum prompt size accepted by your application layer before it reaches Ollama, and validate token counts client-side rather than relying on the runtime to reject oversized requests gracefully.
Memory fragmentation after many model swaps
Repeatedly loading and unloading different models in the same long-running process can, in rare cases, leave the CUDA allocator with fragmented free memory such that no single contiguous block large enough for the next model is available, even though the total free VRAM reported by nvidia-smi looks sufficient.
Fix: Restart the Ollama service periodically in long-running deployments that swap models frequently, which resets the allocator to a clean state.
systemctl restart ollama
Diagnostic commands
Check for an OS-level OOM kill
dmesg | grep -i 'out of memory'
A hit here means system RAM, not VRAM, was exhausted and the kernel killed the process; the fix is RAM or offload management, not GPU quantization alone.
Check VRAM headroom before loading
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
If used memory is already high before you load your model, another process or a previously loaded model is consuming the budget you thought was free.
Check system RAM availability
free -h
Low available memory here, especially combined with high swap usage, confirms system RAM is the bottleneck for CPU-offloaded layers.
See what Ollama currently has resident
ollama ps
Multiple models listed as loaded simultaneously explains VRAM pressure that a single model's size alone would not predict.
Stopping it from happening again
- Size models against free VRAM and RAM, not total capacity, leaving headroom for the KV cache at your real target context length.
- Set an explicit, conservative num_ctx default in your application rather than trusting a large runtime default.
- Monitor VRAM and system RAM continuously in production and alert before utilization approaches the ceiling, not after an OOM kill.
- Limit how many distinct models can be resident simultaneously in memory-constrained deployments.
When this becomes an architecture problem
If you are hitting memory limits with a right-sized quantization and a reasonable context length, the hardware is simply undersized for the models and concurrency you need, and the right next step is a proper capacity plan (bigger GPU, multiple GPUs, or a dedicated serving stack with admission control) rather than continuing to trim context length.
Frequently asked questions
Does Ollama tell me how much memory a model needs before loading it?
Ollama estimates memory requirements internally and will sometimes report the shortfall in its error message, but it does not always give a precise breakdown ahead of time. Checking nvidia-smi and free before and after loading is the most reliable way to see the actual footprint.
Will adding swap fix Ollama out of memory errors?
Swap can prevent an immediate OOM kill by giving the OS more room, but disk-backed swap is drastically slower than RAM, so a model that needs swap to run will generate tokens extremely slowly. Treat swap as an emergency buffer, not a real fix.
Why does the same model OOM only sometimes?
This is almost always a context length effect: shorter conversations fit in the available memory budget while longer ones grow the KV cache past it, so the same model and quantization can succeed or fail depending on how much conversation history is being sent.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Related problems
Ollama not using the GPU, falls back to CPU
Ollama falls back to CPU silently, without an obvious error, most often because the NVIDIA driver is missing inside a container, the model does not fit in available VRAM so Ollama offloads some or all layers to system RAM, or the GPU simply is not visible to the process. Check ollama ps for the CPU/GPU split and nvidia-smi for driver visibility before assuming the model itself is slow.
Ollama silently truncates earlier conversation turns
Ollama silently truncates conversation history once the total tokens exceed num_ctx, which defaults to a relatively small value in many client configurations, dropping the oldest turns without any error or warning to the user or the calling application. The fix is to explicitly set num_ctx to a value that matches both your actual conversation length needs and the model's supported maximum, and to monitor token counts rather than assuming the full history is always sent.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
vLLM fails to start because there is not enough memory for the KV cache
vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.