Small Language Models
The right model is usually smaller than the demo.
Small language models (roughly 1B to 15B parameters) are the workhorses of on-prem AI: fine-tuned to one job, they match or beat giant models on that job, respond in milliseconds, and run on hardware you can put anywhere, including the shop floor and the edge.
The task-specific inversion
On broad benchmarks, bigger wins. On a single well-defined task with good training data (classify these tickets, extract these fields, answer from this corpus, draft in this format), a fine-tuned small model routinely equals or beats a generic giant, because the task never needed the giant's breadth; it needed the tune.
Enterprises run on exactly these narrow tasks, by the hundreds. The economically sane architecture is a portfolio: small specialists for the high-volume narrow work, a mid-size generalist for open-ended queries, and (where data grades allow) frontier APIs for the rare genuinely hard reasoning. Most volume lands on the cheapest tier.
What SLMs unlock that big models can't
Speed and placement, mostly:
- Interactive latency: milliseconds-class response makes AI feel like software, not a séance
- Edge deployment: on the machine controller, in the vehicle, at the depot bench (the ThermaRay and PCBSpot pattern)
- Massive concurrency per GPU: one card serves an org's worth of narrow-task traffic
- Air-gap friendliness: small artifacts move through signed offline registries easily
- Cheap iteration: retraining a specialist overnight on one GPU keeps it current
The honest limits
Small models fail predictably at long-horizon reasoning, subtle multi-step synthesis, and open-world knowledge, and no amount of tuning fully fixes that. The craft is in the decomposition: route the narrow 90% of traffic to specialists and escalate the hard 10% to a bigger model, with the router itself often being the smallest model in the building. Pretending one small model does everything is how SLM projects get a bad name.
How NetRay builds SLM portfolios
Task inventory, then evaluation sets per task, then the tune-and-benchmark loop: for each task we publish the table (small tuned model vs mid generalist vs frontier reference) and route by evidence. The result is typically that 80%+ of token volume runs on models costing pennies per million tokens on your own hardware, with quality dashboards proving nothing was lost in the trade.
Straight answers
How small is production-viable?
For focused classification and extraction, 1-4B models tuned well are routinely production-grade. For grounded Q&A and drafting, the 7-15B class is the sweet spot. Below 1B lives on true edge devices for single narrow functions.
Do SLMs hallucinate more?
Untuned and unguarded, yes. Tuned to a narrow task and grounded by retrieval with abstention ('not in the corpus'), their error rates on that task compare well with much larger models, and the evaluation harness measures it rather than assuming it.
Can small models run without GPUs entirely?
The smallest ones, yes: quantized 1-4B models run usefully on modern CPUs for low-concurrency workloads, which matters for edge sites and retrofits. GPUs remain the right answer once concurrency climbs.
Is this the same as distillation?
Related: distillation is one way to build a strong small model, by compressing a large teacher's behavior into a small student. We use it when a big model already does your task well and the goal is making that capability cheap and local.
Find the smallest model that does the job.
Bring one high-volume task. We'll show you what a tuned specialist does to your cost and latency, with the benchmark to prove it.
Benchmark a small modelFree AI workshops available. A founder reads every message.