Why Ollama silently forgets earlier turns, and how to fix context truncation
model responses ignore earlier context after a long conversation, as if it forgot the beginning
Also appears as
- num_ctx default of 2048 silently drops earlier turns
- ollama truncates context without warning
- increasing num_ctx has no effect on remembered history
Short answer
Ollama silently truncates conversation history once the total tokens exceed num_ctx, which defaults to a relatively small value in many client configurations, dropping the oldest turns without any error or warning to the user or the calling application. The fix is to explicitly set num_ctx to a value that matches both your actual conversation length needs and the model's supported maximum, and to monitor token counts rather than assuming the full history is always sent.
Affects: Ollama and llama.cpp-based chat applications using default or unset num_ctx values, especially multi-turn conversational use cases
Fix it in 60 seconds
- 1Check the current num_ctx being used by inspecting the Modelfile or the API request parameters: it defaults to a modest value unless explicitly overridden.
- 2Explicitly set num_ctx in your API call or Modelfile to a value that fits your target conversation length, for example 8192 or higher for long conversations.
- 3Confirm the model itself actually supports that context length; setting num_ctx above the model's trained maximum does not add real capability, it can degrade output quality.
- 4Recalculate expected VRAM usage after increasing num_ctx, since a larger context directly increases KV cache memory.
- 5Add client-side logic to count tokens and warn or summarize older turns before they are silently dropped, rather than relying on the runtime to signal truncation.
How to confirm this is your problem
- The model appears to forget instructions, names, or facts established earlier in a long conversation
- Behavior changes noticeably once a conversation crosses a certain length, without any explicit error from the API
- Increasing the visible conversation length in your application does not change the model's apparent memory
- The same system prompt stops being followed after enough turns have accumulated
Root causes and fixes
num_ctx left at its default value
Ollama and many client wrappers apply a default context window unless the caller explicitly overrides it; once the running total of system prompt, conversation history, and new input exceeds that window, the runtime truncates from the oldest end of the context silently, with no error surfaced to the API caller or end user.
Fix: Explicitly set num_ctx in the API request options or in a custom Modelfile PARAMETER num_ctx line to a value large enough for your expected conversation length, rather than relying on the default.
curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "prompt": "hi", "options": {"num_ctx": 8192}}'Application layer not tracking or trimming conversation length itself
Many chat applications simply append every new turn to a growing history array and send the whole thing on every request, trusting the model runtime to handle length; once that grows past num_ctx, the runtime's silent truncation becomes the application's de facto memory management strategy, which produces inconsistent behavior depending on exactly how much text was in earlier turns.
Fix: Implement explicit token counting and either summarize or drop older turns deliberately in the application layer, so truncation is a controlled, visible decision rather than an accidental side effect of hitting num_ctx.
num_ctx set higher than the model's actual trained context length
Setting num_ctx above the maximum context length the model was actually trained or fine-tuned on does not grant real additional memory; instead it can produce degraded, less coherent output on the tokens beyond the model's real training window, which looks similar to truncation but is actually a capability limit rather than a configuration bug.
Fix: Check the model's documented maximum context length in its model card or config and keep num_ctx at or below that value rather than assuming a larger number is always safe.
KV cache memory limit forcing an effective cap below the requested num_ctx
Even if num_ctx is set to a large value, if VRAM cannot hold the resulting KV cache alongside the model weights, some runtimes and configurations will fail to allocate it or fall back to a smaller effective window, which reintroduces truncation-like symptoms despite the configuration looking correct on paper.
Fix: Verify available VRAM against the KV cache size implied by your num_ctx setting, and reduce num_ctx or move to a smaller quantization if the requested context does not actually fit in memory.
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
System prompt growing unboundedly and crowding out conversation history
In applications that append tool definitions, retrieved documents, or accumulating instructions to the system prompt on every turn, the system prompt itself can grow large enough to consume most of the context budget, leaving very little room for actual conversation history even with a generously sized num_ctx.
Fix: Audit what is actually being sent as the system prompt on each request, and cap or periodically prune anything that grows unbounded (like an ever-expanding tool list or retrieved context) separately from the num_ctx setting.
Diagnostic commands
Check the num_ctx actually used in a request
curl http://localhost:11434/api/show -d '{"name": "<model>"}'Shows the model's default parameters including context length; compare this against what your application explicitly passes in the options field of each request.
Count tokens in the full request payload
python -c "import tiktoken; enc = tiktoken.get_encoding('cl100k_base'); print(len(enc.encode(open('payload.txt').read())))"If the token count of your full prompt plus history exceeds the configured num_ctx, truncation is definitely happening on that request, even if the exact tokenizer differs slightly from the target model's own.
Check KV cache memory against VRAM
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
If VRAM is nearly full at your target num_ctx, the runtime may be capping context below what you requested to avoid an out-of-memory failure.
Stopping it from happening again
- Always set num_ctx explicitly rather than relying on the runtime default, and document the chosen value alongside the reason for it.
- Implement application-level token counting and deliberate history trimming or summarization instead of depending on silent runtime truncation.
- Keep num_ctx at or below the model's actual trained context length rather than assuming larger is always better.
- Monitor conversation length in production and alert when typical usage approaches your configured num_ctx limit.
When this becomes an architecture problem
If your application genuinely needs long-running conversations or large retrieved-document context beyond what a single model's practical context window and your VRAM budget can support, that is an architecture problem calling for retrieval-based context management or a model with a larger native context window, not something you can configuration-tune your way out of indefinitely.
Frequently asked questions
Does Ollama warn me when it truncates context?
No. Truncation happens silently inside the runtime with no error or warning surfaced to the API response, which is exactly why it is such a common source of confusing the model forgot bug reports.
What is Ollama's default num_ctx?
It varies by version and how a model was pulled or configured, but it has historically defaulted to a modest value well below what many models actually support. Never assume a specific default; always set num_ctx explicitly for anything beyond casual local testing.
Will setting a very large num_ctx always improve results?
No. Beyond the model's real trained context length, a larger num_ctx does not add genuine memory and can degrade coherence. It also directly increases KV cache memory usage, so an unnecessarily large value can cause out-of-memory failures for no quality benefit.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolModel Context Window Planner
Allocate a fixed context window across system prompt, retrieved chunks, conversation history, and reserved output, then see exactly how much retrieval headroom is left.
Free ToolRAG Context Window Budget Calculator
Allocate your context window across system prompt, retrieved chunks, and conversation history, then see window utilization and the real cost of every RAG query.
Related problems
Ollama runs out of memory loading or running a model
Ollama out of memory happens when the model's weights plus its KV cache exceed either GPU VRAM or system RAM, and the OOM killer or CUDA allocator terminates the process. The fix depends on which resource is exhausted: reduce quantization or context length for VRAM limits, or reduce concurrent model loads and context for system RAM limits.
Ollama generates tokens very slowly
Slow Ollama generation almost always traces back to the model running partly or fully on CPU instead of GPU, either because it does not fit in VRAM, the GPU was never detected, or n_gpu_layers is set too low in a llama.cpp-based config. Check the eval rate in verbose output and the CPU/GPU split in ollama ps before tuning anything else.
Inference gets much slower as context length grows toward 32k, 64k, or 128k tokens
Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.
vLLM: model's max seq len is larger than the KV cache can hold
vLLM preallocates a fixed KV cache pool sized by gpu_memory_utilization and refuses to start a context length whose worst case (batch x max sequence length) doesn't fit in that pool. Fix it by raising --gpu-memory-utilization toward 0.9-0.95, lowering --max-model-len to what you actually need, or adding a GPU/quantizing weights to leave more headroom for cache.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuideThe Model Upgrade Migration Playbook
A playbook for upgrading production LLMs: re-evaluation, prompt regression testing, rollback planning, and avoiding silent quality regressions.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.