CPU Inference for Small Language Models: When It Actually Works
CPU inference for small language models works when three conditions hold together: the model is 8B parameters or smaller, latency requirements tolerate seconds rather than milliseconds, and concurrency is low, typically single-digit simultaneous requests. Outside that envelope, GPUs remain decisively better on cost per token and throughput. The mistake we see most often is enterprises assuming CPU inference is a cheap universal fallback, then discovering that a single Xeon server serving 8B models at meaningful concurrency delivers a fraction of what a used RTX 4090 would, at similar or higher total cost once server-grade CPU pricing is factored in.
Intel AMX and What It Actually Accelerates
Advanced Matrix Extensions (AMX), available on Intel Xeon Sapphire Rapids, Emerald Rapids, and Granite Rapids server CPUs, adds dedicated matrix multiplication units that meaningfully accelerate the INT8 and BF16 matrix operations at the core of transformer inference, delivering 3 to 5x the throughput of the same CPU without AMX on quantized models. This is a real, measurable improvement, not marketing, but it is improving from a low baseline: even AMX-accelerated CPU inference on an 8B model runs at a small fraction of GPU throughput. AMX matters most for organizations that already have Xeon server fleets with spare capacity and want to add small-model inference without a hardware purchase, not as a reason to build new CPU-only infrastructure from scratch.
- AMX requires Sapphire Rapids (4th Gen Xeon) or newer; older Xeon and all AMD EPYC lack this specific acceleration
- Delivers 3-5x throughput improvement over non-AMX CPU inference on INT8/BF16 quantized models
- Best utilized on existing underused server fleets, not as justification for new CPU-only purchases
- Requires software support: llama.cpp, PyTorch with Intel Extension for PyTorch (IPEX), and ONNX Runtime all have AMX paths
llama.cpp and GGUF Quantization for CPU Serving
llama.cpp is the practical software stack for CPU inference, with GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0) trading model size and quality for speed, and AVX-512 plus AMX code paths that make real use of modern server CPU instruction sets. A Llama 3 8B model quantized to Q4_K_M typically runs at 15 to 30 tokens per second on a modern dual-socket Xeon server at batch size 1, versus 80 to 150+ tokens per second for the same model on a single consumer GPU. Ollama wraps llama.cpp with a friendlier interface and is the common on-ramp for teams piloting CPU or edge inference before committing to production infrastructure. Batch size is the critical variable: CPU inference throughput degrades sharply as concurrent requests rise, while GPU throughput scales far more gracefully with batching.
- Llama 3 8B Q4_K_M on dual Xeon: roughly 15-30 tokens/sec at batch size 1
- Same model on a single consumer GPU (RTX 4090): roughly 80-150+ tokens/sec
- CPU throughput degrades sharply with concurrency; GPU throughput scales far better with batching
- GGUF quantization (Q4_K_M typical sweet spot) trades a small quality loss for large speed and memory gains
When CPU Inference Is the Right Call
CPU inference makes sense for low-volume internal tools where a 2 to 5 second response is acceptable, for edge deployments where adding a GPU is impractical due to power, space, or ruggedization constraints, for batch offline processing where wall-clock time matters less than throughput per dollar of idle infrastructure, and for organizations with strict data residency requirements that want to pilot small-model use cases on existing server hardware before committing GPU budget. It is also a reasonable choice for embedding generation and lightweight classification tasks using small encoder models, which are far less compute-intensive than generative inference and run comfortably on CPU even at moderate volume.
When to Skip CPU and Just Buy the GPU
Any workload with more than a handful of concurrent users, any use case with a sub-second latency requirement, and any deployment serving models above roughly 8 to 13B parameters should default to GPU inference. Even a single entry-level data center GPU or a used RTX 4090 workstation card typically delivers better throughput per dollar than a comparably priced server-grade CPU for LLM inference specifically, because CPUs remain fundamentally memory-bandwidth-limited for this workload in a way GPUs are architecturally designed to avoid. The break-even calculation should compare total infrastructure cost per served request, not GPU purchase price alone, since a slow CPU deployment that requires horizontally scaling to many servers to hit a concurrency target can end up costing more than a single GPU server.
How Netray Sizes CPU vs GPU Inference Decisions
Netray evaluates CPU inference as a legitimate option, not a fallback, when the workload profile genuinely fits: low concurrency, latency-tolerant, small model, or edge-constrained. We benchmark against your actual model and expected request pattern rather than published vendor numbers, which routinely overstate real-world throughput at production concurrency levels, and we are equally comfortable telling a client that CPU inference will save money on a specific internal tool as we are recommending a GPU cluster for a customer-facing service. That honesty is what keeps the infrastructure spend matched to the actual requirement instead of over- or under-provisioned.
Frequently Asked Questions
Can you run LLM inference on CPU without a GPU?
Yes, for small models under roughly 8B parameters at low concurrency and with latency tolerance in the range of seconds rather than milliseconds. Using llama.cpp with GGUF quantization on a modern Xeon server with AMX support, a Llama 3 8B model typically runs at 15 to 30 tokens per second at batch size 1, which is workable for low-volume internal tools but far below GPU throughput.
What is Intel AMX and does it help LLM inference?
Advanced Matrix Extensions (AMX) is a hardware feature on Intel Xeon Sapphire Rapids and newer server CPUs that accelerates matrix multiplication for INT8 and BF16 operations, delivering 3 to 5x throughput improvement over non-AMX CPU inference on quantized models. It is real and measurable, but even accelerated CPU inference remains a fraction of GPU throughput, so AMX is best viewed as making existing CPU fleets usable for light inference, not as a GPU replacement.
When should I use CPU instead of GPU for LLM inference?
Use CPU inference for low-volume internal tools tolerant of a few seconds of latency, for edge deployments where power, space, or ruggedization rules out a GPU, for offline batch processing, and for lightweight embedding or classification tasks. Default to GPU for any customer-facing service, sub-second latency requirement, more than a handful of concurrent users, or models above roughly 8 to 13B parameters.
Key Takeaways
- 1Intel AMX and What It Actually Accelerates: Advanced Matrix Extensions (AMX), available on Intel Xeon Sapphire Rapids, Emerald Rapids, and Granite Rapids server CPUs, adds dedicated matrix multiplication units that meaningfully accelerate the INT8 and BF16 matrix operations at the core of transformer inference, delivering 3 to 5x the throughput of the same CPU without AMX on quantized models. This is a real, measurable improvement, not marketing, but it is improving from a low baseline: even AMX-accelerated CPU inference on an 8B model runs at a small fraction of GPU throughput.
- 2llama.cpp and GGUF Quantization for CPU Serving: llama.cpp is the practical software stack for CPU inference, with GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0) trading model size and quality for speed, and AVX-512 plus AMX code paths that make real use of modern server CPU instruction sets. A Llama 3 8B model quantized to Q4_K_M typically runs at 15 to 30 tokens per second on a modern dual-socket Xeon server at batch size 1, versus 80 to 150+ tokens per second for the same model on a single consumer GPU.
- 3When CPU Inference Is the Right Call: CPU inference makes sense for low-volume internal tools where a 2 to 5 second response is acceptable, for edge deployments where adding a GPU is impractical due to power, space, or ruggedization constraints, for batch offline processing where wall-clock time matters less than throughput per dollar of idle infrastructure, and for organizations with strict data residency requirements that want to pilot small-model use cases on existing server hardware before committing GPU budget. It is also a reasonable choice for embedding generation and lightweight classification tasks using small encoder models, which are far less compute-intensive than generative inference and run comfortably on CPU even at moderate volume..
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
CPU-Only LLM Inference Feasibility Calculator
Estimate realistic tokens per second for CPU-only inference from model size, quantization, and system RAM bandwidth, and see whether it clears your target throughput.
Free ToolAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Terms used in this article
Not sure whether your workload actually needs a GPU or can run on CPU inference? Netray will benchmark your model and request pattern and tell you honestly which infrastructure fits.
Related Resources
Edge AI Deployment in Manufacturing Plants: A Practical Guide
Deploy edge AI on plant floors: rugged hardware options, offline operation patterns, and use cases for vision inspection and predictive maintenance.
AI & AutomationApple Silicon for Local LLM Development: A Practical Guide
Apple Silicon for local LLM development: M-series unified memory, MLX vs llama.cpp performance, and where Mac fits versus GPU servers for enterprise teams.
AI & AutomationSmall Language Models for Enterprise: When Smaller Wins
When small language models like Phi-4, Gemma 3, and Qwen3 small variants beat large models on cost, latency, and task-specific enterprise accuracy.