vLLM Docker Compose Generator for a Private ERP LLM
Serving a private model with vLLM is mostly a matter of getting a handful of flags right. This generator writes the compose file for an OpenAI-compatible endpoint so your ERP assistant, agent framework or MCP server can call a model that never leaves your network.
Fill in
Pin a tested release tag in production instead of latest.
Use a local path in an air-gapped site.
docker-compose.yml
services:
erp-llm:
image: vllm/vllm-openai:latest
ipc: host
restart: unless-stopped
ports:
- "8000:8000"
volumes:
- /srv/models:/models:ro
- ./hf-cache:/root/.cache/huggingface
environment:
- HF_HUB_OFFLINE=1
- VLLM_API_KEY=$VLLM_API_KEY
command:
- --model
- /models/Qwen3-32B-AWQ
- --served-model-name
- erp-assistant
- --max-model-len
- "16384"
- --tensor-parallel-size
- "2"
- --gpu-memory-utilization
- "0.90"
- --max-num-seqs
- "32"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 2
capabilities: [gpu]Smoke test (run from another host on the network)
# curl http://HOST:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
# curl http://HOST:8000/v1/chat/completions \
# -H "Content-Type: application/json" \
# -H "Authorization: Bearer $VLLM_API_KEY" \
# -d '{"model": "erp-assistant", "messages": [{"role": "user", "content": "Reply with OK"}]}'Verify flags against the vLLM version you pin, since options evolve between releases. The compose file reads the key from a VLLM_API_KEY variable in your .env file; set it before starting. The NVIDIA Container Toolkit must be installed on the host. Do not expose the port outside the segment that your ERP integration layer uses.
Private LLM stack review
An optional review of your compose setup, GPU sizing and network exposure.
Optional. The tool and every export work without this. No spam.
How to use it
Enter the model path, the number of GPUs and the maximum context you need. Copy the generated YAML into docker-compose.yml, create a .env file with a long random VLLM_API_KEY, and run docker compose up -d. Then use the smoke test block to confirm the model list and a chat completion respond. Mount weights read-only from a host directory so the container never needs internet access.
What the key settings mean
Tensor parallel size splits the model across GPUs and should match the number of GPUs you reserve. GPU memory utilization sets the share of VRAM vLLM may claim for weights and KV cache; 0.90 is a common starting point. Max model length caps the context window and directly drives KV cache memory. Max sequences limits concurrency so a burst of ERP users does not exhaust memory.
- Lower max length first if the container fails to start for memory
- HF_HUB_OFFLINE=1 stops any attempt to reach the internet
- Use a quantized checkpoint if VRAM is tight
- Pin the image tag once tested
Common mistakes
Leaving the image on latest means an unplanned upgrade can change behavior. Forgetting ipc: host or sufficient shared memory causes multi-GPU startup failures. Exposing the port with no API key lets anyone on the network query a model that may be wired to ERP data. Setting max length far above what your prompts need wastes VRAM that could serve more concurrent users.
Frequently Asked Questions
Does this work in an air-gapped network?
Yes, provided the vLLM image and the model weights are transferred in advance through your approved media process. Set offline mode to 1 and use a local model path so nothing tries to reach Hugging Face.
How do I choose the tensor parallel size?
Use the smallest GPU count whose combined VRAM holds the weights plus KV cache for your context and concurrency. It generally needs to divide the model attention head count evenly, so powers of two are the safe choice.
Can ERP tools call this like OpenAI?
Yes. The server exposes OpenAI-compatible routes under /v1, so most agent frameworks and SDKs work by changing the base URL and key. Test tool calling with your model, as support varies by model and chat template.
Is Ollama simpler than vLLM?
Ollama is simpler for a single user or a pilot. vLLM generally handles higher concurrency better on server GPUs. Many teams pilot on Ollama and move to vLLM when many ERP users share one endpoint.
If you want this endpoint wired to your ERP with read-only access, row-level permissions and logging, Netray builds and runs that stack on-prem.
Related Tools
Ollama Modelfile Generator for an ERP Assistant
Build a Modelfile that bakes a grounded, read-only ERP system prompt and sane parameters into a local Ollama model.
On-Prem AIOpen-Weight LLM License Terms Reference
Look up which license each popular open-weight model family ships under and what to check before running it on-prem against ERP data.
Developer ToolsERP AI Agent Read-Only SQL Server Role Generator
Create a least-privilege SQL Server login and role for an AI agent, with table grants, sensitive column denies and a row filter by site.
AI Agents & AutomationText-to-SQL Guardrails Checklist for ERP Data
Check that an LLM that writes SQL over your ERP is limited by database permissions, query validation, cost limits and audit trails.
More Developer Tools tools
See all- SAP Sales Order ABAP SELECT Generator (VBAK, VBAP, VBFA)
- SAP Sales Order Create BAPI ABAP Generator (BAPI_SALESORDER_CREATEFROMDAT2)
- SAP ABAP Open Items SELECT Generator
- SAP ACDOCA SQL Query Generator
- SAP QM Inspection Lot ABAP SELECT Generator
- SAP Quality Notification Pareto SQL Query Generator
- SAP QM Usage Decision BAPI Skeleton
- ABAP Custom Exception Class Generator
- ABAP Unit Test Class Generator
Keep exploring
Free tools
- vLLM Throughput EstimatorEstimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
- GPU Sizing Calculator for LLM InferenceWork out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
- NVIDIA GPU Selector for LLM WorkloadsScore your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Guides
- The LLM Inference Cost Optimization PlaybookCut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
- vLLM Production Deployment: A Practitioner's GuideDeploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
- SGLang vs vLLM: An Honest Production Serving ComparisonSGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.
Troubleshooting
- Why vLLM's OpenAI-compatible API returns 404, and how to fix the requestThis 404 is almost always a wrong route or a model name mismatch. Match your request's model field and URL path to what /v1/models actually returns.
- Why your GPU is not visible inside a Docker container, and how to fix itnvidia-docker not found or "could not select device driver nvidia.com/gpu"? Install the NVIDIA Container Toolkit, register the runtime, use --gpus all.
Software
- vLLMWhat vLLM is, how it serves open-weight LLMs on your own GPUs, how it connects to ERP and RAG stacks, and when Ollama, NIM or TGI fit better.
- Hugging Face Text Generation InferenceWhat Hugging Face Text Generation Inference (TGI) is, how it serves open-weight LLMs, its API and features, and how it compares with vLLM and NIM.