# CPU-Only LLM Inference Feasibility Calculator: When You Do Not Need a GPU

> Source: https://netray.co/tools/cpu-only-llm-inference-feasibility-calculator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Estimate realistic tokens per second for CPU-only inference from model size, quantization, and system RAM bandwidth, and see whether it clears your target throughput.

## Category

on-prem-ai

## Tool

**Kind:** calculator

**Inputs:**

#### Model size

**Key:** model_params_b

**Type:** number

**Default:** 8

**Min:** 0.5

**Max:** 100

**Step:** 0.5

**Unit:** B params

**Help:** Total parameter count in billions. CPU-only serving is generally practical only under roughly 30B parameters.

#### Weight precision

**Key:** quantization_bytes

**Type:** select

**Default:** 0.5

**Unit:** bytes/param

**Help:** CPU serving stacks like llama.cpp rely heavily on aggressive quantization to fit memory bandwidth budgets.

**Options:**

##### FP16 (2 bytes per parameter)

**Value:** 2

##### INT8 (1 byte per parameter)

**Value:** 1

##### INT4 / GGUF Q4 (0.5 bytes per parameter)

**Value:** 0.5

#### System RAM bandwidth

**Key:** ram_bandwidth_gbs

**Type:** select

**Default:** 170

**Unit:** GB/s

**Help:** CPU decode speed is bound by memory bandwidth in the same way GPU decode is.

**Options:**

##### Dual-channel DDR5 desktop (about 80 GB/s)

**Value:** 80

##### Quad-channel DDR5 workstation (about 170 GB/s)

**Value:** 170

##### 8-channel DDR5 server (about 350 GB/s)

**Value:** 350

##### 12-channel DDR5 high-bandwidth server (about 500 GB/s)

**Value:** 500

#### Achieved bandwidth efficiency

**Key:** cpu_efficiency_pct

**Type:** slider

**Default:** 55

**Min:** 30

**Max:** 75

**Step:** 5

**Unit:** %

**Help:** Real CPU serving stacks reach 40-65% of theoretical memory bandwidth after cache and instruction overhead.

#### Minimum acceptable throughput

**Key:** required_tokens_per_sec

**Type:** number

**Default:** 10

**Min:** 1

**Max:** 200

**Step:** 1

**Unit:** tokens/sec

**Help:** Roughly 6-8 tokens per second is comfortable reading speed for a single interactive user.

**Outputs:**

#### Model memory footprint

**Key:** model_memory_gb

**Expr:** model_params_b * quantization_bytes * 1.15

**Format:** number

**Description:** Weights plus roughly 15% overhead for KV cache and runtime buffers, in gigabytes.

#### Effective memory bandwidth

**Key:** effective_bandwidth_gbs

**Expr:** ram_bandwidth_gbs * cpu_efficiency_pct / 100

**Format:** number

**Description:** Realistic sustained bandwidth after real-world CPU serving overhead.

#### Estimated decode throughput

**Key:** estimated_tokens_per_sec

**Expr:** effective_bandwidth_gbs / max(model_memory_gb, 0.1)

**Format:** number

**Description:** Single-stream tokens per second this configuration should sustain.

**Primary:** true

#### Feasibility against your target

**Key:** feasibility_ratio_pct

**Expr:** estimated_tokens_per_sec / max(required_tokens_per_sec, 1) * 100

**Format:** percent

**Description:** Above 100% means this configuration should clear your minimum throughput requirement.

**Primary:** true

#### System RAM required

**Key:** ram_required_gb

**Expr:** model_memory_gb * 1.1

**Format:** number

**Description:** Recommended installed RAM including headroom for the OS and other processes.

**Disclaimer:** Rough planning estimate. Real CPU throughput depends heavily on serving engine, NUMA configuration, and prompt length. Benchmark with llama.cpp or a similar engine on representative hardware before committing to a CPU-only deployment.

## Content

**Intro:** This free CPU-only LLM inference feasibility calculator estimates whether a model will run fast enough without a GPU, and it is built for engineers evaluating edge deployments, cost-constrained pilots, or workloads where GPU procurement lead time is the blocker. Enter model size, weight precision, system RAM bandwidth, and your minimum acceptable throughput, and the tool returns estimated tokens per second and whether that clears your target. CPU inference is bound by the same memory bandwidth physics as GPU inference, just at roughly a tenth the bandwidth, which is why it only works for small, aggressively quantized models.

**Sections:**

#### How CPU decode speed is calculated

**Body:** The math mirrors GPU inference: decode speed is approximately memory bandwidth divided by the bytes that must be read per token, which for a dense transformer is close to the full quantized model size. An 8B model quantized to INT4 occupies roughly 4.6 GB with overhead. A quad-channel DDR5 workstation offers about 170 GB/s of theoretical bandwidth; at a realistic 55% efficiency that is 93.5 GB/s, yielding roughly 20 tokens per second, comfortably above typical interactive reading speed. The same model at FP16 would occupy about 18.4 GB and only manage around 5 tokens per second on identical hardware, which feels sluggish for a live chat interface.

#### Where CPU-only inference genuinely works

**Body:** This is not a universal GPU replacement, it is a legitimate option for a specific and fairly common set of enterprise use cases: small extraction and classification models, edge devices without GPU slots, air-gapped laptops for field engineers, and low-traffic internal tools where a few seconds of latency is acceptable. It fails for anything requiring a large model, high concurrency, or sub-second responsiveness.

**Bullets:**

- Small quantized models under roughly 8B parameters are the practical ceiling for responsive single-user CPU inference.
- 8-channel or wider server memory configurations meaningfully outperform desktop dual-channel setups, often 4x the bandwidth.
- INT4 quantization via GGUF formats is close to mandatory for acceptable CPU throughput; FP16 on CPU is rarely usable interactively.
- Concurrency scales poorly on CPU compared to GPU batching, so CPU inference suits single-user or low-traffic deployments best.

#### Reading your feasibility ratio

**Body:** A feasibility ratio above 150% gives comfortable headroom for prompt processing overhead and occasional longer contexts. Between 100% and 150% is workable but leaves little margin, and real-world variance in prompt length or concurrent background processes could push you below target. Under 100% means this configuration will not meet your throughput requirement, and the fix is almost always a smaller model or a higher-bandwidth memory configuration, not more CPU cores, since decode is bandwidth-bound rather than compute-bound.

#### How Netray evaluates CPU versus GPU for edge and constrained deployments

**Body:** Netray evaluates CPU-only deployment as one option among several for manufacturers running field service, quality inspection, and shop-floor assistants at sites where GPU hardware, power, or budget are constrained. We benchmark candidate small models on representative CPU hardware before recommending an architecture, and we frequently find that a well-chosen 3B to 8B model handles a real production use case at a fraction of the cost and complexity of a GPU deployment. Engagements typically start with a workload and hardware feasibility assessment.

**Cta:** Get a benchmarked feasibility report comparing CPU-only and GPU deployment options for your specific workload.

**Faq:**

#### What model size is the practical ceiling for CPU-only inference?

**Answer:** For interactive single-user use cases, roughly 8B to 14B parameters at INT4 quantization is the practical ceiling on typical server hardware, delivering somewhere in the range of 10-25 tokens per second depending on memory bandwidth. Larger models technically run on CPU but drop below comfortable reading speed, often under 5 tokens per second, which makes them impractical for anything interactive.

#### Does adding more CPU cores help throughput?

**Answer:** Only marginally, because decode is bound by memory bandwidth rather than compute. More cores help with prefill, the parallel processing of the prompt before generation begins, and with serving multiple concurrent requests, but they do not meaningfully raise single-stream tokens per second once you have enough cores to saturate the available memory channels.

#### Is CPU inference viable for RAG or agent workloads?

**Answer:** It can be, for extraction, classification, and routing sub-tasks that use small models, but the reasoning-heavy synthesis step in a RAG pipeline usually benefits from a larger model that CPU throughput cannot serve responsively. A common pattern is running retrieval, reranking, and simple classification on CPU while routing the final synthesis step to a GPU-served model or an API.

#### How much does prompt length affect CPU feasibility?

**Answer:** Prefill, processing the input prompt, is more compute-bound than decode and scales with prompt length, so very long prompts add meaningful time before generation even starts on CPU hardware. This calculator estimates decode throughput; for prompts beyond a few thousand tokens, add a separate allowance for prefill time when judging real end-to-end responsiveness.

## Search terms

CPU only LLM inference, run LLM without GPU, llama.cpp CPU throughput calculator, RAM bandwidth LLM inference, GGUF CPU inference speed, small model CPU feasibility, on-prem LLM without GPU

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json