# Why Ollama runs on CPU instead of your GPU, and how to fix it

> Source: https://netray.co/troubleshoot/ollama-not-using-gpu
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Ollama falls back to CPU silently, without an obvious error, most often because the NVIDIA driver is missing inside a container, the model does not fit in available VRAM so Ollama offloads some or all layers to system RAM, or the GPU simply is not visible to the process. Check ollama ps for the CPU/GPU split and nvidia-smi for driver visibility before assuming the model itself is slow.

## Error Title

Ollama not using the GPU, falls back to CPU

## Error Signature

level=WARN source=gpu.go msg="no compatible GPUs were discovered"

## Alternate Signatures

- Error: no NVIDIA/AMD GPU detected, falling back to CPU mode
- ollama ps shows 100% CPU, 0% GPU for a loaded model

## Category

local-runtime

## Stack

- ollama
- nvidia-driver
- docker
- cuda

## Affects

Ollama on Linux and Windows with NVIDIA or AMD GPUs, and Ollama running inside Docker or Kubernetes

## Quick Fix

### Fix it in 60 seconds

**Steps:**

- Run nvidia-smi on the host. If it fails or shows no GPU, the driver or GPU is not visible at the OS level and nothing Ollama does will fix that.
- If running in Docker, confirm you started the container with --gpus all and the NVIDIA Container Toolkit is installed.
- Run ollama ps while a model is loaded and check the PROCESSOR column for the CPU/GPU percentage split.
- If the split shows partial GPU usage, the model is larger than free VRAM; try a smaller quantization or a smaller model size.
- Restart the ollama service after fixing driver or container GPU access, since it detects GPUs at startup.

## Symptoms

- Generation is far slower than expected and CPU usage sits near 100% while GPU utilization stays at 0%
- ollama ps shows a percentage split like 100% CPU / 0% GPU next to the loaded model
- The same model was fast on bare metal but became slow after moving to a Docker container
- nvidia-smi shows the GPU as idle even while a prompt is actively generating

## Causes

### NVIDIA driver or CUDA libraries not visible inside the container

**Likelihood:** most-common

**Explanation:** Ollama running inside a Docker container only sees the GPU if the NVIDIA Container Toolkit is installed on the host and the container was launched with GPU passthrough enabled; without it, the container has no path to the driver and Ollama correctly falls back to CPU rather than failing outright.

**Fix:** Install the NVIDIA Container Toolkit on the host, then run the container with --gpus all (Docker) or the correct device plugin resource request (Kubernetes), and verify with nvidia-smi run inside the container.

**Commands:**

- docker run --gpus all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama
- nvidia-smi

### Model is larger than available VRAM

**Likelihood:** common

**Explanation:** Ollama estimates how many transformer layers fit in free VRAM and offloads the rest to system RAM automatically; if the quantized model plus its KV cache exceeds available VRAM, some or all layers run on CPU, which produces much slower tokens per second without any explicit error.

**Fix:** Check ollama ps for the CPU/GPU percentage split, then move to a smaller quantization (for example Q4_K_M instead of Q8_0) or a smaller parameter count model that fits fully in VRAM.

**Commands:**

- ollama ps
- nvidia-smi --query-gpu=memory.used,memory.total --format=csv

### GPU driver installed but the wrong CUDA compute capability or an unsupported GPU

**Likelihood:** occasional

**Explanation:** Very old GPUs, or systems with a driver version too old for the CUDA runtime Ollama bundles, are not recognized during GPU discovery, so Ollama logs a warning and falls back to the CPU backend even though the card is physically present.

**Fix:** Update the NVIDIA driver to a version that matches or exceeds the CUDA toolkit version Ollama ships with, and confirm the GPU is on Ollama's supported compute capability list.

**Commands:**

- nvidia-smi --query-gpu=driver_version,compute_cap --format=csv

### AMD GPU without ROCm support configured

**Likelihood:** occasional

**Explanation:** AMD GPU support in Ollama depends on ROCm being correctly installed and the specific GPU being on the supported HSA override list; unsupported or unlisted AMD cards fall back to CPU even when the ROCm stack is present.

**Fix:** Verify your AMD GPU is on Ollama's supported list and set the HSA_OVERRIDE_GFX_VERSION environment variable if needed for a close-but-unlisted GPU architecture.

**Commands:**

- export HSA_OVERRIDE_GFX_VERSION=10.3.0

### Multiple GPU processes competing for VRAM

**Likelihood:** rare

**Explanation:** If another process (a training job, a second Ollama instance, or another inference server) already holds most of the VRAM, Ollama's own allocation attempt fails or is truncated, and it silently reduces GPU layer offload to fit in what remains.

**Fix:** Check nvidia-smi for other processes holding VRAM and stop or reschedule them, or dedicate a specific GPU to Ollama with CUDA_VISIBLE_DEVICES.

**Commands:**

- nvidia-smi
- export CUDA_VISIBLE_DEVICES=0

## Diagnostics

### Confirm the GPU is visible at all

**Command:** nvidia-smi

**Interpretation:** If this command fails or lists no devices, the problem is at the driver/OS level, not Ollama; fix the driver install before touching Ollama config.

### Check Ollama's own CPU/GPU split

**Command:** ollama ps

**Interpretation:** The PROCESSOR column shows a percentage like 100% GPU, 50%/50%, or 100% CPU for the currently loaded model, telling you directly whether offload is partial or total.

### Check GPU memory headroom

**Command:** nvidia-smi --query-gpu=memory.used,memory.total --format=csv

**Interpretation:** If used is close to total before you even load a model, another process is holding VRAM that Ollama needs.

### Tail the Ollama service logs at startup

**Command:** journalctl -u ollama -n 100 --no-pager

**Interpretation:** Look for lines mentioning gpu.go or CUDA driver discovery; a no compatible GPUs were discovered warning confirms Ollama itself never found the card.

## Prevention

- Always verify nvidia-smi works inside the exact container image you deploy, not just on the host, before shipping to production.
- Size models against actual free VRAM, not total VRAM, accounting for the KV cache and any other GPU workloads on the box.
- Pin the NVIDIA driver and CUDA toolkit versions in your base image so a routine OS update cannot silently break GPU discovery.
- Add a startup health check that asserts ollama ps reports GPU usage before marking a deployment healthy.

## When To Escalate

If you are repeatedly fighting driver and VRAM sizing issues across multiple machines, that points to a hardware and capacity planning gap rather than a one-off config bug, and it is worth sizing GPUs against your real model and context length requirements before buying more hardware.

## Faq

### Does Ollama tell me explicitly when it falls back to CPU?

**Answer:** Not prominently. It logs a warning during GPU discovery and shows the actual split in ollama ps, but there is no loud error, which is why silent CPU fallback is such a common source of unexplained slowness.

### Why is GPU usage 0% even though nvidia-smi sees the card?

**Answer:** Nvidia-smi being able to see the card only confirms the driver is installed at the OS level; Ollama needs its own CUDA discovery to succeed too, and inside containers this additionally requires the NVIDIA Container Toolkit and GPU passthrough flags.

### Can I force Ollama to only use the GPU and fail instead of falling back to CPU?

**Answer:** Ollama does not currently expose a strict GPU-only flag that errors out; the practical approach is to monitor ollama ps in your deployment pipeline and treat a CPU-heavy split as a failed health check.

## Search terms

ollama not using gpu, ollama cpu fallback, ollama gpu.go no compatible gpus, ollama ps gpu split, ollama docker gpu, ollama vram

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json