# Why Ollama generates tokens slowly, and how to speed it up > Source: https://netray.co/troubleshoot/ollama-slow-generation > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Slow Ollama generation almost always traces back to the model running partly or fully on CPU instead of GPU, either because it does not fit in VRAM, the GPU was never detected, or n_gpu_layers is set too low in a llama.cpp-based config. Check the eval rate in verbose output and the CPU/GPU split in ollama ps before tuning anything else. ## Error Title Ollama generates tokens very slowly ## Error Signature total duration: 45.2s, eval rate: 1.84 tokens/s ## Alternate Signatures - ollama run is extremely slow compared to before - prompt eval rate: 12 tokens/s eval rate: 2 tokens/s ## Category local-runtime ## Stack - ollama - llama-cpp - cuda - nvidia-driver ## Affects Ollama and llama.cpp on any GPU or CPU-only setup, most noticeable on 7B+ models on consumer GPUs ## Quick Fix ### Fastest path to normal speed **Steps:** - Run the same prompt with ollama run --verbose and note the eval rate in tokens per second. - Run ollama ps while the model is loaded and check whether the PROCESSOR column shows a GPU percentage or 100% CPU. - If it shows CPU, work through the ollama-not-using-gpu causes: driver visibility, VRAM headroom, and container GPU passthrough. - If GPU usage is already high but still slow, check nvidia-smi -q -d TEMPERATURE for thermal throttling. - For llama.cpp directly, confirm n_gpu_layers is set high enough to offload the full model rather than a low default. ## Symptoms - Verbose output reports an eval rate far below what the GPU should be capable of for that model size - The same model was fast previously but slowed down after a driver update, a model swap, or a longer prompt - GPU utilization in nvidia-smi bounces near 0% during generation despite the model being loaded - Generation speed drops noticeably as conversation context grows longer ## Causes ### Model running on CPU instead of GPU **Likelihood:** most-common **Explanation:** CPU inference for a multi-billion parameter model is an order of magnitude slower than GPU inference because it lacks the massive parallelism of a GPU's thousands of cores; this happens whenever the GPU is not detected, VRAM is insufficient, or the container lacks GPU passthrough, and it is the single biggest driver of complaints about slow Ollama. **Fix:** Confirm GPU usage with ollama ps and nvidia-smi, then fix driver visibility, container GPU passthrough, or reduce model/quantization size so it fits fully in VRAM. **Commands:** - ollama ps - nvidia-smi ### Model larger than VRAM causing partial layer offload **Likelihood:** common **Explanation:** When a model almost but not quite fits in VRAM, Ollama offloads the layers that do not fit to system RAM and CPU, so every forward pass bounces between GPU and CPU compute for different layers, which is dramatically slower than running fully on either device alone. **Fix:** Move to a smaller quantization of the same model (Q4_K_M instead of Q8_0 or FP16) or reduce num_ctx, since the KV cache also consumes VRAM and competes with model weights for the same budget. **Commands:** - ollama ps - ollama run :q4_K_M ### Thermal throttling under sustained load **Likelihood:** occasional **Explanation:** Consumer GPUs and laptops with inadequate cooling reduce their clock speed automatically once they hit a thermal limit during long generation runs or back-to-back requests, so the first few tokens are fast and throughput degrades progressively the longer the GPU stays busy. **Fix:** Check GPU temperature during a long generation with nvidia-smi, improve case airflow or reapply thermal paste on workstation GPUs, and consider a power limit adjustment if throttling is severe. **Commands:** - nvidia-smi -q -d TEMPERATURE ### n_gpu_layers set too low in a llama.cpp-backed config **Likelihood:** common **Explanation:** When llama.cpp is invoked directly, or through a wrapper that exposes the setting, n_gpu_layers controls how many transformer layers get offloaded to GPU; a low or default value that does not match your actual VRAM budget leaves most layers on CPU even though the GPU has room to spare. **Fix:** Raise n_gpu_layers to a value that fills available VRAM, or set it to a very high number to offload as many layers as possible and let the runtime cap it at what fits. **Commands:** - ./main -m model.gguf -ngl 999 -p "test" ### Long context growing the KV cache faster than expected **Likelihood:** occasional **Explanation:** Every additional token of context adds to the KV cache, and once that cache plus model weights exceed VRAM, the runtime starts trading off cache size or offloading more aggressively, so a conversation that started fast noticeably slows down as it grows longer. **Fix:** Reduce num_ctx to the smallest value your use case actually needs, or summarize/trim older turns instead of keeping full history in every request. **Commands:** - ollama run --verbose ## Diagnostics ### Get exact tokens per second **Command:** ollama run --verbose **Interpretation:** Compare the reported eval rate against published benchmarks for that model size and quantization on similar hardware; a rate an order of magnitude lower points to CPU fallback. ### Check the CPU/GPU processing split **Command:** ollama ps **Interpretation:** A GPU percentage below 100% for a model that should fit fully in VRAM confirms partial CPU offload is the bottleneck. ### Watch GPU utilization live during generation **Command:** nvidia-smi dmon -s u **Interpretation:** Utilization near 0% while tokens are streaming means the GPU is essentially idle and CPU is doing the work; near 100% utilization with still-slow output points to a genuinely undersized GPU for that model. ### Check GPU temperature under load **Command:** nvidia-smi -q -d TEMPERATURE **Interpretation:** Temperatures near the throttle limit during a long run confirm thermal throttling is reducing clocks and therefore tokens per second. ## Prevention - Benchmark expected tokens per second for your model and quantization on your specific GPU before deploying, so a regression is obvious. - Size models to fit fully in VRAM with headroom for the KV cache at your target context length, not just the raw weight size. - Ensure adequate cooling and airflow for GPUs running sustained inference workloads, especially in dense workstation or edge deployments. - Cap num_ctx to a realistic value for your application rather than leaving it at a large default that grows the KV cache unnecessarily. ## When To Escalate If you have confirmed full GPU offload, adequate cooling, and a right-sized quantization and are still not hitting acceptable throughput, the model or context length is simply too large for the GPU you have, and the right move is proper capacity planning across a bigger card, multiple GPUs, or a purpose-built serving stack like vLLM instead of continuing to tune a single-box Ollama setup. ## Faq ### Is Ollama slower than vLLM for the same model? **Answer:** For single-user, single-request workloads on one GPU, per-token speed is often similar since both ultimately run the same model. Ollama is not designed for high-concurrency serving though, and throughput under multiple simultaneous requests will be much lower than a batching-aware server like vLLM. ### Does a longer conversation history slow down every response? **Answer:** Yes, because Ollama has to account for the growing context on each turn, and the KV cache for that context consumes more VRAM as it grows, which can push a model from fully GPU-resident to partially offloaded partway through a long conversation. ### What is a normal tokens per second for a 7B model on a consumer GPU? **Answer:** On a modern 24GB consumer GPU with the model fully offloaded, tens of tokens per second is typical for a 7B class model at a reasonable quantization; single-digit tokens per second on that hardware is a strong signal something is falling back to CPU. ## Search terms ollama slow, ollama tokens per second, ollama eval rate, n_gpu_layers slow, ollama cpu fallback slow, llama.cpp slow generation --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json