# Code Model Deployment Sizing Calculator: GPUs Needed for an In-House Coding Assistant

> Source: https://netray.co/tools/code-model-deployment-sizing-calculator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Convert developer headcount and completion volume into the GPU capacity and monthly cost required to self-host a code completion model at acceptable latency.

## Category

on-prem-ai

## Tool

**Kind:** calculator

**Inputs:**

#### Number of developers using the assistant

**Key:** num_developers

**Type:** number

**Default:** 200

**Min:** 1

**Max:** 20000

**Step:** 5

**Unit:** developers

**Help:** Total licensed users, not just those active at any given moment.

#### Completions per developer per day

**Key:** completions_per_dev_day

**Type:** number

**Default:** 250

**Min:** 10

**Max:** 3000

**Step:** 10

**Unit:** completions/day

**Help:** Includes both accepted and shown-but-rejected inline completions plus chat requests.

#### Average tokens per completion (prompt + output)

**Key:** avg_tokens_per_completion

**Type:** number

**Default:** 900

**Min:** 100

**Max:** 8000

**Step:** 50

**Unit:** tokens

**Help:** Includes surrounding code context sent to the model plus the generated completion.

#### Code model size (throughput per GPU)

**Key:** model_size_select

**Type:** select

**Default:** 70

**Unit:** tok/s per GPU

**Help:** Reference decode throughput for a code-specialized model on a single H100-class GPU.

**Options:**

##### 7B, e.g. Qwen2.5-Coder-7B (approx 120 tok/s per GPU)

**Value:** 120

##### 14B, e.g. Qwen2.5-Coder-14B or Codestral-mini (approx 70 tok/s per GPU)

**Value:** 70

##### 22B, e.g. Devstral (approx 45 tok/s per GPU)

**Value:** 45

##### 32B, e.g. Qwen2.5-Coder-32B (approx 30 tok/s per GPU)

**Value:** 30

#### Peak concurrent active share of developers

**Key:** peak_concurrency_pct

**Type:** slider

**Default:** 40

**Min:** 5

**Max:** 100

**Step:** 5

**Unit:** %

**Help:** Share of your developer base actively requesting completions during a peak minute, typically mid-morning.

#### GPU compute cost

**Key:** gpu_cost_per_hour

**Type:** number

**Default:** 2.5

**Min:** 0.1

**Max:** 20

**Step:** 0.1

**Unit:** $/hr

**Help:** Blended cloud GPU rental rate, or amortized on-prem GPU cost per hour including power.

**Outputs:**

#### Daily completion tokens (millions)

**Key:** daily_completion_tokens_m

**Expr:** num_developers * completions_per_dev_day * avg_tokens_per_completion / 1000000

**Format:** number

**Description:** Total tokens processed per day across all developers and completions.

#### Peak concurrent active developers

**Key:** peak_concurrent_devs

**Expr:** num_developers * peak_concurrency_pct / 100

**Format:** number

**Description:** Developers expected to be actively requesting completions at the same peak minute.

#### Required throughput at peak (tokens/sec)

**Key:** required_throughput_tokps

**Expr:** peak_concurrent_devs * avg_tokens_per_completion / 3

**Format:** number

**Description:** Aggregate decode throughput needed assuming each completion should return within about 3 seconds.

#### GPUs needed to hit latency target

**Key:** gpus_needed

**Expr:** ceil(required_throughput_tokps / max(model_size_select, 1))

**Format:** number

**Description:** GPU count required at your selected model size to serve peak concurrent demand within the latency target.

**Primary:** true

#### Monthly GPU hours (always-on)

**Key:** monthly_gpu_hours

**Expr:** gpus_needed * 24 * 30

**Format:** number

**Description:** Total GPU-hours if the cluster runs continuously to serve development-hours traffic reliably.

#### Monthly infrastructure cost

**Key:** monthly_infra_cost

**Expr:** monthly_gpu_hours * gpu_cost_per_hour

**Format:** currency

**Description:** Compute cost to keep this GPU capacity available around the clock.

**Disclaimer:** Planning estimates only. Real throughput depends on serving stack (vLLM, TensorRT-LLM), context length, batching configuration, and prompt caching for repeated file context. Benchmark against your actual codebase before finalizing GPU count.

## Content

**Intro:** This free code model deployment sizing calculator converts developer headcount and completion volume into the GPU capacity needed to self-host an AI coding assistant at acceptable latency, and it is built for engineering leaders evaluating an in-house alternative to a commercial code completion product. Enter developer count, completions per day, average token size, code model size, peak concurrency, and GPU cost, and the tool returns peak throughput requirements, GPUs needed, and monthly infrastructure cost. Self-hosted code models like Qwen2.5-Coder and Devstral have closed most of the capability gap to commercial products on everyday completions, and self-hosting is often the only option for teams whose codebase cannot leave the network at all.

**Sections:**

#### Why peak concurrency drives GPU count, not average volume

**Body:** A code completion assistant is a latency-sensitive interactive workload: developers expect sub-second to low-single-digit-second responses while typing, not a batched overnight job. That means capacity has to be sized for the peak concurrent moment, typically mid-morning when the largest share of the team is actively coding, not the 24-hour average. With 200 developers at 40% peak concurrency and 900 tokens per completion targeting a 3-second response, the required aggregate throughput is substantial even though average daily volume looks modest when spread across 24 hours.

**Bullets:**

- Inline completions need low time-to-first-token above all else; chat-style questions can tolerate more latency
- Prompt caching for repeated file and project context meaningfully reduces effective token processing per completion
- A 7B to 14B code-specialized model is often the sweet spot for latency-sensitive inline completion at scale
- Larger models (22B to 32B) suit chat-style code review and explanation where users tolerate a few seconds more

#### Model size versus latency tradeoff

**Body:** Smaller code models decode faster per GPU but may need more retries or produce lower-quality completions on complex, multi-file reasoning tasks. Qwen2.5-Coder in the 7B and 14B range handles everyday completions, boilerplate, and single-function generation well. Devstral and larger Qwen2.5-Coder variants perform better on multi-step, multi-file agentic coding tasks but need proportionally more GPU throughput to hit the same latency target. Many production deployments run a two-tier setup: a fast small model for inline completions and a larger model reserved for explicit chat and code-review requests where users already expect to wait a few seconds.

#### Reading the GPU count and cost result

**Body:** The GPUs-needed figure assumes always-on capacity sized for peak demand, which is intentionally conservative since idle capacity during off-hours has no way to be reclaimed without autoscaling infrastructure most teams do not build for an internal tool on day one. If the monthly cost looks high relative to a commercial per-seat subscription, compare it honestly: self-hosting wins on data control and predictable cost at scale, not necessarily on raw dollars for a moderate-sized team. The crossover typically favors self-hosting once developer count and completion volume are high enough that per-seat commercial pricing exceeds the amortized GPU cost, or immediately if codebase confidentiality rules out any external code assistant.

#### How Netray deploys self-hosted coding assistants

**Body:** Netray deploys self-hosted AI coding assistants for aerospace, defense, and electronics manufacturers whose codebases, including embedded firmware and ERP customizations, cannot be sent to a third-party code completion service. We benchmark candidate code models against your actual repositories, tune the serving stack for the latency profile inline completion needs, and integrate with your existing IDE tooling. Engagements typically start with a two-week pilot measuring completion acceptance rate against your current tooling before a full rollout.

**Cta:** Get a self-hosted coding assistant sized and benchmarked against your actual codebase and developer count.

**Faq:**

#### Can a self-hosted code model really replace a commercial coding assistant?

**Answer:** For everyday completions, boilerplate, and single-function generation, open code models like Qwen2.5-Coder are competitive with commercial alternatives on most benchmarks and in practice. The gap is more noticeable on complex multi-file refactoring and agentic coding tasks, where the largest commercial models still have an edge. Many teams run a hybrid: self-hosted for daily inline completion, with selective access to a larger frontier or open model for genuinely hard tasks.

#### Why does the calculator assume always-on GPU capacity?

**Answer:** Because inline code completion is a real-time interactive workload; developers will not tolerate a cold-start delay while a GPU spins up mid-keystroke. Some teams reduce cost by scaling down capacity outside working hours if their developer base is concentrated in a single time zone, but that adds operational complexity. The always-on figure here is the conservative, reliable baseline to budget against.

#### How much does prompt caching actually help with code completion?

**Answer:** Significantly, because a large share of the context sent with each completion request is the surrounding file and project context that barely changes between keystrokes. A serving stack with effective prefix caching can avoid reprocessing that shared context on every request, which meaningfully reduces effective throughput demand versus treating every completion as a fully independent prompt.

#### Is Devstral or Qwen2.5-Coder better for our use case?

**Answer:** It depends on the task mix. Qwen2.5-Coder in the 7B to 32B range covers a broad range of languages and completion styles with strong benchmark performance and lower latency at smaller sizes. Devstral is specifically tuned for agentic, multi-step software engineering tasks like navigating a codebase and making coordinated multi-file changes. If your primary use case is inline completion, start with Qwen2.5-Coder; if it is autonomous coding agents, evaluate Devstral alongside it.

## Search terms

code model deployment calculator, self-hosted coding assistant sizing, Qwen Coder deployment, Devstral GPU requirements, Codestral self-hosted cost, AI code assistant infrastructure, on-prem code completion model

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json