# Why CPU offloading makes LLM inference so slow, and when it is still worth it

> Source: https://netray.co/troubleshoot/cpu-offloading-slow-llm
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

CPU offload trades memory capacity for speed because every offloaded layer's weights must cross the PCIe bus (typically 16-64 GB/s) on every forward pass, versus terabytes-per-second on-GPU HBM bandwidth; this is not a bug, it is the fundamental cost of running more model than your GPU can hold. The real fix is usually to reduce how much needs to be offloaded (quantize first) rather than trying to make offloading itself faster.

## Error Title

LLM inference is extremely slow after enabling CPU offload

## Error Signature

Model runs without an out-of-memory error after enabling device_map="auto" with CPU offload, but generation throughput drops to a fraction of a token per second

## Alternate Signatures

- Inference works but is 10x-50x slower with cpu offload enabled
- accelerate offload_folder makes generation unusably slow

## Category

gpu-memory

## Stack

- pytorch
- transformers
- accelerate
- huggingface

## Affects

Any deployment using CPU or disk offload (accelerate device_map="auto" with offload, llama.cpp partial GPU layers, DeepSpeed ZeRO-Infinity) to fit a model that does not fit entirely in VRAM

## Quick Fix

### Recover throughput or make an informed tradeoff

**Steps:**

Quantize the model to 4-bit or 8-bit first, since this often shrinks it enough to fit entirely in VRAM and eliminates the need for offload altogether, which is almost always faster than any offload configuration.

If offload is still necessary, offload the minimum number of layers possible rather than a large fraction, since throughput degrades roughly in proportion to how much of the model must cross PCIe per token.

Prefer GPU-resident KV cache with only weights offloaded, rather than offloading KV cache itself, since KV cache is read on every single generation step and is far more sensitive to bandwidth than weights.

If your workload can tolerate batch processing instead of interactive latency, batch requests to amortize the PCIe transfer cost across more tokens per transfer.

If throughput remains unacceptable, treat this as confirmation that the workload needs more or bigger GPUs rather than continuing to tune an offload configuration that is fundamentally bandwidth-bound.

## Symptoms

- Model loads and generates correct output with no OOM error, but tokens per second drops by 10x or more compared to full-GPU serving
- GPU utilization (nvidia-smi) is low or spiky, alternating between idle and brief bursts, rather than sustained high utilization
- Increasing batch size does not proportionally improve throughput the way it would on a fully GPU-resident model
- The slowdown is proportional to how many layers or what fraction of weights are offloaded to CPU or disk

## Causes

### Offloaded layers must be transferred over PCIe on every forward pass, and PCIe bandwidth is orders of magnitude lower than GPU HBM bandwidth

**Likelihood:** most-common

**Explanation:** GPU HBM (on-device memory) delivers roughly 2-3 TB/s of bandwidth on modern data center GPUs, while PCIe Gen4/Gen5 delivers roughly 32-64 GB/s between CPU and GPU, a gap of 40-90x. Every offloaded layer's weights must cross that much slower link for every single token generated, so the more of the model lives off-GPU, the more the per-token latency is dominated by data transfer rather than compute.

**Fix:** Minimize the fraction of the model that needs to be offloaded, ideally by quantizing first so more (or all) of the model fits natively in VRAM, since offload cost scales directly with how many bytes must cross PCIe per forward pass.

**Commands:**

- python -c "print(2e12/50e9, 'x faster HBM is than typical PCIe Gen4')"

### device_map="auto" offloaded more layers to CPU than strictly necessary because of a conservative memory margin

**Likelihood:** common

**Explanation:** Accelerate's automatic device mapping reserves some GPU memory headroom by default when deciding how many layers to place on GPU versus CPU, which can be more conservative than your actual available memory allows, pushing more layers to the slow path than a tighter manual configuration would.

**Fix:** Manually specify a max_memory dict per device to allow accelerate to place more layers on GPU, verifying with nvidia-smi that you are not leaving significant unused VRAM headroom while offloading unnecessarily.

**Commands:**

- from accelerate import infer_auto_device_map
- device_map = infer_auto_device_map(model, max_memory={0: "22GiB", "cpu": "64GiB"})

### Disk offload (not just CPU RAM offload) is active, adding storage I/O latency on top of PCIe transfer

**Likelihood:** common

**Explanation:** When CPU RAM itself is insufficient and an offload_folder is configured, weights spill to disk, adding storage read latency (even on NVMe, meaningfully slower than RAM) on top of the PCIe transfer cost, compounding the slowdown further than RAM-only offload.

**Fix:** Ensure enough CPU RAM is available that offload stays in RAM rather than spilling to disk, since RAM-to-GPU offload, while still slow relative to native VRAM, is meaningfully faster than adding a disk round trip on every forward pass.

**Commands:**

- free -h

### KV cache, not just weights, is being kept partly on CPU

**Likelihood:** occasional

**Explanation:** KV cache must be read and updated on every single decoding step for every token generated, making it far more latency-sensitive to offload than weights, which are read once per layer per forward pass; offloading KV cache to CPU multiplies the PCIe traffic per token far more than offloading static weights does.

**Fix:** Keep KV cache entirely GPU-resident even if some weight layers must be offloaded, prioritizing KV cache placement on GPU over weight placement when memory is tight.

### PCIe link is running at a lower generation or lane width than the hardware supports, for example due to a BIOS setting or a shared riser

**Likelihood:** rare

**Explanation:** A GPU installed in a slot negotiated at PCIe Gen3 x8 instead of its supported Gen4/Gen5 x16 halves or quarters available bandwidth without any error being raised, silently making offload performance far worse than the hardware's real capability.

**Fix:** Check the negotiated PCIe link speed and width for the GPU and correct BIOS, riser, or slot configuration issues if it is running below the card's rated specification.

**Commands:**

- nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

## Diagnostics

### Measure actual tokens per second with and without offload

**Command:** python -c "import time; t0=time.time(); model.generate(**inputs, max_new_tokens=100); print(100/(time.time()-t0), 'tokens/sec')"

**Interpretation:** Compare this against a fully GPU-resident configuration of a similarly sized model; a 10-50x gap confirms PCIe-bound offload behavior rather than a misconfiguration elsewhere.

### Watch GPU utilization pattern during generation

**Command:** nvidia-smi dmon -s u -c 20

**Interpretation:** Low, spiky utilization with idle gaps between bursts indicates the GPU is frequently waiting on data transfer from CPU rather than continuously computing, the classic signature of offload-bound throughput.

### Confirm negotiated PCIe link speed matches hardware capability

**Command:** nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csv

**Interpretation:** If current is lower than max, you are losing bandwidth to a configuration issue on top of the inherent offload cost, and fixing that recovers some throughput before any software-level change.

## Prevention

- Treat CPU/disk offload as a capacity-of-last-resort fallback in architecture decisions, not a routine serving configuration for production latency-sensitive workloads.
- Quantize before considering offload, since the memory saved by quantization often eliminates the need for offload entirely.
- Size GPU memory for your model at deployment planning time so offload is never required in the production path.
- If offload is used for occasional large-model access (not production serving), set throughput expectations accordingly rather than being surprised by them.

## When To Escalate

If your production workload requires interactive latency and the model that meets your accuracy bar only fits with CPU offload even after quantization, that is a clear signal you need more or larger GPUs rather than continuing to tune offload settings, since the PCIe bandwidth ceiling is a hardware property, not a software one.

## Faq

### Why is CPU offload so much slower than just running a smaller model?

**Answer:** Offload does not reduce the amount of computation; it adds a PCIe data transfer step before the GPU can use each offloaded layer's weights, on every single forward pass. PCIe bandwidth (tens of GB/s) is roughly 40-90x slower than GPU HBM bandwidth (terabytes/s), so any meaningful fraction of the model living off-GPU dominates total latency, often making offload far slower in practice than simply serving a smaller model that fits entirely in VRAM.

### Is there a way to make CPU offload fast?

**Answer:** Not fundamentally; PCIe bandwidth is a hardware ceiling that no software configuration bypasses. You can minimize the damage by offloading as few layers as possible, keeping KV cache GPU-resident, avoiding disk-level offload, and ensuring the PCIe link is negotiated at its full rated speed, but offload will always be substantially slower than fully GPU-resident serving for the same model.

### Does NVLink change this calculation?

**Answer:** Only between GPUs, not between CPU and GPU; NVLink connects GPU-to-GPU at far higher bandwidth than PCIe (hundreds of GB/s to a few TB/s depending on generation), which is why multi-GPU tensor parallelism does not suffer the same bottleneck as CPU offload. CPU-to-GPU transfer still goes through PCIe (or a CPU-GPU interconnect like NVLink-C2C on select platforms) regardless of how the GPUs are connected to each other.

### Should I use CPU offload for a batch processing job instead of interactive serving?

**Answer:** It is a more reasonable fit there, since batch jobs can amortize the fixed PCIe transfer cost across more tokens and are typically not latency-sensitive on a per-request basis. It is still meaningfully slower than fully GPU-resident inference, so validate that the total job completion time meets your batch window before committing to this architecture.

## Search terms

cpu offload llm slow, accelerate device_map auto slow inference, cpu offloading pytorch performance, pcie bandwidth llm inference, offload_folder slow generation, gpu utilization low during offload, why is cpu offload so slow

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json