Self-hosted LLMs, done like production software.

What self-hosting actually involves

Downloading an open-weight model is the easy 5%. Production self-hosting is the other 95%: choosing a model that fits your tasks and your hardware, quantizing it without wrecking accuracy, serving it with vLLM or TensorRT-LLM at real concurrency, wiring authentication and audit logging, and keeping it patched without an internet-connected pipeline.

NetRay has shipped this stack inside manufacturing plants, defense electronics facilities, and data centers. The pattern is repeatable: benchmark 3 to 5 candidate models on your actual tasks and your actual hardware, pick the smallest one that clears your quality bar, then engineer the serving layer for your concurrency and latency targets.

Model selection: smaller than you think

Most enterprise workloads (document Q&A, extraction, summarization, internal copilots) are served well by 7B to 70B open-weight models, especially after fine-tuning on your domain. A tuned 8B model frequently beats a generic frontier model on narrow tasks, and it runs on a single server you already know how to rack.

We benchmark on your data before any hardware is purchased. The deliverable is a table: model, quantization level, tokens per second on your target GPU, and accuracy on your evaluation set. You pick from evidence, not from leaderboards.

The serving stack we deploy

The production baseline: vLLM or TensorRT-LLM for inference, an OpenAI-compatible API gateway so your applications integrate with standard tooling, per-user authentication tied to your identity provider, prompt and completion logging to your SIEM, and load-tested autoscaling within your cluster.

• Inference: vLLM / TensorRT-LLM with continuous batching and paged attention
• Quantization: AWQ / GPTQ / FP8 where accuracy budgets allow
• Gateway: OpenAI-compatible API, rate limits, role-based access from your IdP
• Observability: latency, throughput, and quality drift dashboards on your monitoring stack
• Updates: signed model registry, staged rollouts, rollback in minutes

Day-2 operations, not a handoff and a wave

The difference between a demo and infrastructure is what happens in month three: a new model release worth evaluating, a quality regression report from a user, a security patch for the serving layer. We run this with your team (or train them to run it) using the same MLOps discipline we apply in air-gapped environments: signed artifacts, staged promotion, and drift monitoring.