Why your GPU is not visible inside a Docker container, and how to fix it
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]
Also appears as
- RuntimeError: No CUDA GPUs are available
- nvidia-smi: command not found (inside container)
Short answer
Docker containers cannot see a host GPU unless the NVIDIA Container Toolkit is installed and the nvidia runtime is registered with the daemon, since containers are isolated from host devices by default. The fix is almost always to install nvidia-container-toolkit, run nvidia-ctk runtime configure, restart Docker, and launch with --gpus all. If nvidia-smi already fails on the host itself, the problem is the driver, not Docker.
Affects: Docker Engine 20.10 and later on Linux hosts with an NVIDIA GPU, any CUDA-based image
Fix it in 60 seconds
- 1Confirm the host driver works: run nvidia-smi on the host, not in a container.
- 2Install the toolkit: sudo apt-get install -y nvidia-container-toolkit.
- 3Register the runtime: sudo nvidia-ctk runtime configure --runtime=docker.
- 4Restart Docker: sudo systemctl restart docker.
- 5Test it: docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi.
- 6If that works but your app still can't see the GPU, check your app's own --gpus flag or Compose device reservation.
How to confirm this is your problem
- docker run reports 'could not select device driver "" with capabilities: [[gpu]]'
- nvidia-smi works on the host but not inside any container
- torch.cuda.is_available() returns False only when the code runs in a container
- the same image works on one host but not another
Root causes and fixes
Missing --gpus all flag or NVIDIA Container Toolkit not installed
Docker does not pass through GPUs by default; without the --gpus flag on the run command and the NVIDIA Container Toolkit installed on the host, the container gets a normal Linux namespace with no device nodes or driver libraries mounted in, so any CUDA call fails immediately even though the host GPU is healthy.
Fix: Install nvidia-container-toolkit on the host, restart Docker, then run with --gpus all (or --gpus device=0 for a specific GPU). Confirm with a plain nvidia-smi test container.
sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
NVIDIA runtime not registered in daemon.json
The toolkit can be installed but Docker's daemon.json never got updated to register the nvidia runtime, or a manual edit was reverted by a package update, so docker run --gpus all silently falls back to the default runc runtime which has no GPU support.
Fix: Run nvidia-ctk runtime configure to regenerate /etc/docker/daemon.json with the nvidia runtime entry, restart the daemon, and verify with docker info.
cat /etc/docker/daemon.json sudo nvidia-ctk runtime configure --runtime=docker --set-as-default sudo systemctl restart docker docker info | grep -i runtime
Driver and container toolkit version mismatch
The host NVIDIA driver and the CUDA version baked into the container image must be compatible; an old driver paired with a newer CUDA base image reports the GPU as present but every CUDA API call fails with a driver library version mismatch, which looks identical to no GPU from inside the container.
Fix: Check the driver version on the host against the CUDA version in the image tag, and either upgrade the host driver or pull an older CUDA base image that matches it.
nvidia-smi --query-gpu=driver_version --format=csv docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
Docker rootless mode without the runtime configured for the user namespace
Rootless Docker runs the daemon in a separate user namespace, and the default NVIDIA Container Toolkit setup only registers the runtime for the system daemon, so a rootless install needs its own CDI configuration or GPU access never gets negotiated for that namespace.
Fix: Follow the rootless-specific setup (CDI mode is simplest) and regenerate the CDI spec with nvidia-ctk cdi generate, then reference it explicitly in the run command.
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml docker run --rm --device nvidia.com/gpu=all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
Diagnostic commands
Confirm the host sees the GPU
nvidia-smi
If this fails on the host itself, the problem is the driver, not Docker; fix the host driver before touching containers.
Test the toolkit directly
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
If this fails with the device driver error, the toolkit or daemon.json is the problem; if it succeeds but your app still can't see the GPU, the issue is inside your image or entrypoint.
Check which runtime Docker is using
docker info | grep -i runtime
You should see nvidia listed among the runtimes; if it is absent, nvidia-ctk runtime configure was never run or was reverted.
Check installed toolkit version
dpkg -l | grep nvidia-container
Confirms whether nvidia-container-toolkit is actually installed versus only the driver; a missing package means the whole toolkit needs installing, not just configuring.
Stopping it from happening again
- Pin the NVIDIA Container Toolkit version in your base image build docs alongside the host driver version
- Bake a nvidia-smi smoke test into your CI image build so a broken runtime fails the build, not production
- Document the exact docker run or Compose gpus stanza in your deployment runbook so it is never typed from memory
- Standardize on one CUDA base image tag across the fleet to avoid per-host driver mismatches
When this becomes an architecture problem
If every host in the fleet needs the same toolkit fix and you are about to repeat these steps across dozens of nodes, or you are moving from single Docker hosts to Kubernetes with a GPU Operator, this becomes a platform decision worth designing once rather than debugging per node.
Frequently asked questions
Do I need the NVIDIA Container Toolkit if I already have the NVIDIA driver installed?
Yes. The host driver lets the host itself use the GPU, but Docker containers are isolated by default and have no way to see host devices or driver libraries. The NVIDIA Container Toolkit is a separate component that registers a Docker runtime capable of mounting the GPU device nodes and driver libraries into a container at launch time. Installing only the driver leaves containers with no GPU access at all.
Why does nvidia-smi work on the host but not inside my container?
This is the classic signature of a missing or misconfigured NVIDIA Container Toolkit. The host driver is healthy, but Docker has no runtime configured to pass GPU devices into containers, so every container gets a plain Linux namespace with none of the device files or shared libraries the driver needs. Installing the toolkit and adding --gpus all to the run command resolves it in most cases.
What is the difference between --gpus all and --runtime=nvidia?
--gpus all is the modern Docker flag that requests GPU devices for a specific container and works with recent Docker Engine versions once the toolkit is installed. --runtime=nvidia is the older mechanism that forces the entire container to use the nvidia OCI runtime. Current NVIDIA documentation recommends --gpus for new deployments; --runtime is mainly seen in legacy Compose files.
Does Docker Compose need special configuration for GPU access?
Yes. Compose does not read --gpus from the CLI, so you need a deploy.resources.reservations.devices block in the service definition specifying the nvidia driver and GPU count or all. Without this block, a Compose-launched container behaves exactly like one started without --gpus, and CUDA calls fail even though a plain docker run with --gpus all works fine.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
On-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolKubernetes Cluster Cost Calculator
Estimate the true monthly and annual cost of a Kubernetes cluster, including compute, control plane fees, managed service premiums, and utilization waste.
Related problems
nvidia-container-cli errors when starting a GPU container
nvidia-container-cli initialization errors mean the host's NVIDIA kernel module failed to load or the driver's supported CUDA version does not meet the minimum your container image requires. Check nvidia-smi on the bare host first; if it fails there, fix the kernel module or driver before touching Docker. If the host is healthy, compare its CUDA support against your image's requirement and either upgrade the driver or use an older image tag.
NVIDIA GPU Operator pods stuck installing or crashlooping
GPU Operator installation problems almost always come from a preinstalled host driver conflicting with the Operator's own driver container, or from Node Feature Discovery never labeling GPU nodes so downstream components stay unscheduled. Check kubectl get pods -n gpu-operator first to see which subsystem is failing, then confirm whether the node has a preexisting driver and whether NFD applied the expected NVIDIA labels.
Kubernetes GPU pod stuck in Pending
A GPU pod stays Pending when no node advertises the nvidia.com/gpu resource because the device plugin is down or missing, the pod requests more GPUs than any single node has, or a taint, toleration, or nodeSelector mismatch blocks placement on the GPU pool. Always start with kubectl describe pod, since the Events section states the exact blocking reason rather than leaving you to guess between these causes.
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.