Why Public AI Tools Are a Problem for CUI
204-7012 and, for ITAR data, can be an unauthorized export. Even enterprise AI tiers vary: some provide data-handling commitments, but only a provider authorized for the right impact level, with the right contractual terms and US-person administration, may hold CUI. For most small shops, the simplest policy is clear: no controlled data in any AI tool unless it runs inside your boundary or the provider is explicitly authorized.
Write that policy now, because staff are already using these tools on their own.
What an Inside-the-Boundary Setup Looks Like
A practical stack has four parts. An inference server with one or more GPUs running an open-weight model through vLLM or Ollama. A retrieval layer that indexes your documents and ERP data into a vector store on the same network.
An application layer with login tied to your identity provider, so the AI only retrieves what the user may already see. And logging, so queries, retrieved sources and outputs are recorded for audit. All of it sits in the same enclave segment as your other CUI systems, administered by vetted staff, with encryption at rest and in transit.
No call leaves the boundary.
- Open-weight models such as Llama, Qwen, Mistral, Gemma or gpt-oss class served locally
- Vector index and embeddings stored inside the enclave, since embeddings can leak source content
- Role-aware retrieval that mirrors ERP and document permissions
- Prompt and response logs protected like any other CUI record
Hardware Sizing for a Small Shop
You do not need a data center. A model in the 7B to 32B parameter range handles RFQ summarization, drawing note extraction, clause screening and ERP question answering for a few concurrent users on a single server with one or two professional GPUs. Larger models help with harder reasoning but need more memory and power.
Quantization reduces memory use at some accuracy cost. Budget for the server, GPUs, storage, backup, a UPS, and the staff time to patch and monitor it, because an unpatched AI server is a weak point in an otherwise compliant boundary. Total first-year cost for a small deployment is commonly in the tens of thousands of dollars, not millions.
Try it here, free
AI Model Selection Assessment
Score ten decision factors - data sensitivity, task complexity, volume, latency, and internal capability - to see whether a self-hosted open-weight model fits your workload.
What On-Prem AI Does Well and Badly
It works well on bounded, document-heavy tasks: reading an RFQ package, extracting material and special process callouts, finding similar past jobs, comparing a customer PO to your accepted terms, answering questions about ERP data with the query shown. It is weaker on open-ended reasoning against a frontier hosted model, and it needs maintenance. Accuracy on scanned drawings depends on OCR quality and on tolerances written in tight notation, so keep a human reviewing anything that becomes a price, a commitment or an inspection record.
Anything that writes back to the ERP should require explicit approval.
When It Is Not Worth Building
If you handle little CUI, quote a few jobs a month, or your volume is mostly repeat parts with stable prices, a rules-based quoting tool or better ERP reports may deliver more value for less. If the real bottleneck is messy data, such as inconsistent routings or missing costs, AI will surface that mess, not fix it. Start with one narrow use case, measure estimator hours or schedule-meeting time saved, and expand only if the numbers justify it.
A private deployment is an investment in capability, not a shortcut.
Frequently asked questions
Can I use ChatGPT or Copilot with CUI?
Only if the specific service is authorized for CUI at your required level, with contractual and technical safeguards, and your prime accepts it. Standard consumer and many business tiers are not. For ITAR-controlled technical data, additional US-person and data-residency requirements apply. When uncertain, treat it as not allowed.
How much does it cost to run an LLM on-prem for a small shop?
A small deployment typically needs one GPU server, storage, networking and support time, with first-year costs commonly in the tens of thousands of dollars depending on model size, redundancy and integration work. Ongoing costs are power, patching, monitoring and model updates. Compare that with estimator hours saved and compliance risk avoided.
Do embeddings and vector databases contain CUI?
They can. Embeddings derived from controlled documents may allow partial reconstruction of the source, so assessors and prudent shops treat the vector index as holding the same classification as the documents. Keep it inside the boundary, encrypt it and apply the same access controls.
Netray designs and runs private, on-prem AI for manufacturers and defense contractors, with read-only ERP access by default and human approval on every write-back.