CPU-Only LLM Inference Feasibility Calculator: When You Do Not Need a GPU
This free CPU-only LLM inference feasibility calculator estimates whether a model will run fast enough without a GPU, and it is built for engineers evaluating edge deployments, cost-constrained pilots, or workloads where GPU procurement lead time is the blocker. Enter model size, weight precision, system RAM bandwidth, and your minimum acceptable throughput, and the tool returns estimated tokens per second and whether that clears your target. CPU inference is bound by the same memory bandwidth physics as GPU inference, just at roughly a tenth the bandwidth, which is why it only works for small, aggressively quantized models.
Your numbers
Total parameter count in billions. CPU-only serving is generally practical only under roughly 30B parameters.
CPU serving stacks like llama.cpp rely heavily on aggressive quantization to fit memory bandwidth budgets.
CPU decode speed is bound by memory bandwidth in the same way GPU decode is.
Real CPU serving stacks reach 40-65% of theoretical memory bandwidth after cache and instruction overhead.
Roughly 6-8 tokens per second is comfortable reading speed for a single interactive user.
Your results
Rough planning estimate. Real CPU throughput depends heavily on serving engine, NUMA configuration, and prompt length. Benchmark with llama.cpp or a similar engine on representative hardware before committing to a CPU-only deployment.
Get your full CPU vs GPU feasibility report
We will email you a personalized throughput analysis comparing CPU-only and GPU deployment for your model and workload, and a Netray specialist will follow up with a benchmark plan.
No spam. Your results stay private. Unsubscribe anytime.
How CPU decode speed is calculated
The math mirrors GPU inference: decode speed is approximately memory bandwidth divided by the bytes that must be read per token, which for a dense transformer is close to the full quantized model size. An 8B model quantized to INT4 occupies roughly 4.6 GB with overhead. A quad-channel DDR5 workstation offers about 170 GB/s of theoretical bandwidth; at a realistic 55% efficiency that is 93.5 GB/s, yielding roughly 20 tokens per second, comfortably above typical interactive reading speed. The same model at FP16 would occupy about 18.4 GB and only manage around 5 tokens per second on identical hardware, which feels sluggish for a live chat interface.
Where CPU-only inference genuinely works
This is not a universal GPU replacement, it is a legitimate option for a specific and fairly common set of enterprise use cases: small extraction and classification models, edge devices without GPU slots, air-gapped laptops for field engineers, and low-traffic internal tools where a few seconds of latency is acceptable. It fails for anything requiring a large model, high concurrency, or sub-second responsiveness.
- Small quantized models under roughly 8B parameters are the practical ceiling for responsive single-user CPU inference.
- 8-channel or wider server memory configurations meaningfully outperform desktop dual-channel setups, often 4x the bandwidth.
- INT4 quantization via GGUF formats is close to mandatory for acceptable CPU throughput; FP16 on CPU is rarely usable interactively.
- Concurrency scales poorly on CPU compared to GPU batching, so CPU inference suits single-user or low-traffic deployments best.
Reading your feasibility ratio
A feasibility ratio above 150% gives comfortable headroom for prompt processing overhead and occasional longer contexts. Between 100% and 150% is workable but leaves little margin, and real-world variance in prompt length or concurrent background processes could push you below target. Under 100% means this configuration will not meet your throughput requirement, and the fix is almost always a smaller model or a higher-bandwidth memory configuration, not more CPU cores, since decode is bandwidth-bound rather than compute-bound.
How Netray evaluates CPU versus GPU for edge and constrained deployments
Netray evaluates CPU-only deployment as one option among several for manufacturers running field service, quality inspection, and shop-floor assistants at sites where GPU hardware, power, or budget are constrained. We benchmark candidate small models on representative CPU hardware before recommending an architecture, and we frequently find that a well-chosen 3B to 8B model handles a real production use case at a fraction of the cost and complexity of a GPU deployment. Engagements typically start with a workload and hardware feasibility assessment.
Frequently Asked Questions
What model size is the practical ceiling for CPU-only inference?
For interactive single-user use cases, roughly 8B to 14B parameters at INT4 quantization is the practical ceiling on typical server hardware, delivering somewhere in the range of 10-25 tokens per second depending on memory bandwidth. Larger models technically run on CPU but drop below comfortable reading speed, often under 5 tokens per second, which makes them impractical for anything interactive.
Does adding more CPU cores help throughput?
Only marginally, because decode is bound by memory bandwidth rather than compute. More cores help with prefill, the parallel processing of the prompt before generation begins, and with serving multiple concurrent requests, but they do not meaningfully raise single-stream tokens per second once you have enough cores to saturate the available memory channels.
Is CPU inference viable for RAG or agent workloads?
It can be, for extraction, classification, and routing sub-tasks that use small models, but the reasoning-heavy synthesis step in a RAG pipeline usually benefits from a larger model that CPU throughput cannot serve responsively. A common pattern is running retrieval, reranking, and simple classification on CPU while routing the final synthesis step to a GPU-served model or an API.
How much does prompt length affect CPU feasibility?
Prefill, processing the input prompt, is more compute-bound than decode and scales with prompt length, so very long prompts add meaningful time before generation even starts on CPU hardware. This calculator estimates decode throughput; for prompts beyond a few thousand tokens, add a separate allowance for prefill time when judging real end-to-end responsiveness.
Get a benchmarked feasibility report comparing CPU-only and GPU deployment options for your specific workload.
Related Tools
GPU Server Buy vs Rent Calculator
Turn GPU price, power cost, and cloud hourly rates into an annual owned-versus-rented comparison, plus the utilization rate at which buying starts to win.
On-Prem AIEdge AI Deployment Readiness Assessment
Score your organization across eight dimensions of edge AI readiness, from hardware validation and update strategy to connectivity resilience and fleet monitoring.
On-Prem AISelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Go Deeper
CPU Inference for Small Language Models: When It Works
CPU inference for small language models explained: Intel AMX, llama.cpp, realistic throughput numbers, and when skipping the GPU actually makes sense.
Apple Silicon for Local LLM Development: A Practical Guide
Apple Silicon for local LLM development: M-series unified memory, MLX vs llama.cpp performance, and where Mac fits versus GPU servers for enterprise teams.
Small Language Models for Enterprise: When Smaller Wins
When small language models like Phi-4, Gemma 3, and Qwen3 small variants beat large models on cost, latency, and task-specific enterprise accuracy.