# vLLM Docker Compose Generator for a Private ERP LLM

> Source: https://netray.co/tools/vllm-docker-compose-generator-for-private-erp-llm
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Fill in the model, context length and GPU count to get a docker-compose file that serves an OpenAI-compatible endpoint inside your network.

## Category

developer

## Audience

- IT manager
- DevOps engineer
- ERP architect

## Tool

**Kind:** template

**Output:** code

**Language:** yaml

**Fields:**

#### Service name

**Key:** service

**Type:** text

**Default:** erp-llm

#### vLLM image tag

**Key:** image_tag

**Type:** text

**Default:** latest

**Help:** Pin a tested release tag in production instead of latest.

#### Model path or Hugging Face id

**Key:** model

**Type:** text

**Default:** /models/Qwen3-32B-AWQ

**Help:** Use a local path in an air-gapped site.

#### Served model name

**Key:** served_name

**Type:** text

**Default:** erp-assistant

#### Max model length (tokens)

**Key:** max_len

**Type:** number

**Default:** 16384

#### GPU count (tensor parallel size)

**Key:** gpu_count

**Type:** number

**Default:** 2

#### GPU memory utilization

**Key:** gpu_util

**Type:** text

**Default:** 0.90

#### Max concurrent sequences

**Key:** max_seqs

**Type:** number

**Default:** 32

#### Host port

**Key:** port

**Type:** number

**Default:** 8000

#### Host model directory

**Key:** model_dir

**Type:** text

**Default:** /srv/models

#### Offline mode

**Key:** offline

**Type:** select

**Options:**

- 1
- 0

**Default:** 1

**Blocks:**

#### docker-compose.yml

**Body:** services:
  {{service}}:
    image: vllm/vllm-openai:{{image_tag}}
    ipc: host
    restart: unless-stopped
    ports:
      - "{{port}}:8000"
    volumes:
      - {{model_dir}}:/models:ro
      - ./hf-cache:/root/.cache/huggingface
    environment:
      - HF_HUB_OFFLINE={{offline}}
      - VLLM_API_KEY=$VLLM_API_KEY
    command:
      - --model
      - {{model}}
      - --served-model-name
      - {{served_name}}
      - --max-model-len
      - "{{max_len}}"
      - --tensor-parallel-size
      - "{{gpu_count}}"
      - --gpu-memory-utilization
      - "{{gpu_util}}"
      - --max-num-seqs
      - "{{max_seqs}}"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: {{gpu_count}}
              capabilities: [gpu]

#### Smoke test (run from another host on the network)

**Body:** # curl http://HOST:{{port}}/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
# curl http://HOST:{{port}}/v1/chat/completions \
#   -H "Content-Type: application/json" \
#   -H "Authorization: Bearer $VLLM_API_KEY" \
#   -d '{"model": "{{served_name}}", "messages": [{"role": "user", "content": "Reply with OK"}]}'

**Notes:** Verify flags against the vLLM version you pin, since options evolve between releases. The compose file reads the key from a VLLM_API_KEY variable in your .env file; set it before starting. The NVIDIA Container Toolkit must be installed on the host. Do not expose the port outside the segment that your ERP integration layer uses.

## Content

**Intro:** Serving a private model with vLLM is mostly a matter of getting a handful of flags right. This generator writes the compose file for an OpenAI-compatible endpoint so your ERP assistant, agent framework or MCP server can call a model that never leaves your network.

**Sections:**

#### How to use it

**Body:** Enter the model path, the number of GPUs and the maximum context you need. Copy the generated YAML into docker-compose.yml, create a .env file with a long random VLLM_API_KEY, and run docker compose up -d. Then use the smoke test block to confirm the model list and a chat completion respond. Mount weights read-only from a host directory so the container never needs internet access.

#### What the key settings mean

**Body:** Tensor parallel size splits the model across GPUs and should match the number of GPUs you reserve. GPU memory utilization sets the share of VRAM vLLM may claim for weights and KV cache; 0.90 is a common starting point. Max model length caps the context window and directly drives KV cache memory. Max sequences limits concurrency so a burst of ERP users does not exhaust memory.

**Bullets:**

- Lower max length first if the container fails to start for memory
- HF_HUB_OFFLINE=1 stops any attempt to reach the internet
- Use a quantized checkpoint if VRAM is tight
- Pin the image tag once tested

#### Common mistakes

**Body:** Leaving the image on latest means an unplanned upgrade can change behavior. Forgetting ipc: host or sufficient shared memory causes multi-GPU startup failures. Exposing the port with no API key lets anyone on the network query a model that may be wired to ERP data. Setting max length far above what your prompts need wastes VRAM that could serve more concurrent users.

**Cta:** If you want this endpoint wired to your ERP with read-only access, row-level permissions and logging, Netray builds and runs that stack on-prem.

**Faq:**

#### Does this work in an air-gapped network?

**Answer:** Yes, provided the vLLM image and the model weights are transferred in advance through your approved media process. Set offline mode to 1 and use a local model path so nothing tries to reach Hugging Face.

#### How do I choose the tensor parallel size?

**Answer:** Use the smallest GPU count whose combined VRAM holds the weights plus KV cache for your context and concurrency. It generally needs to divide the model attention head count evenly, so powers of two are the safe choice.

#### Can ERP tools call this like OpenAI?

**Answer:** Yes. The server exposes OpenAI-compatible routes under /v1, so most agent frameworks and SDKs work by changing the base URL and key. Test tool calling with your model, as support varies by model and chat template.

#### Is Ollama simpler than vLLM?

**Answer:** Ollama is simpler for a single user or a pilot. vLLM generally handles higher concurrency better on server GPUs. Many teams pilot on Ollama and move to vLLM when many ERP users share one endpoint.

## Search terms

vllm docker compose, vllm openai compatible server docker, vllm tensor parallel size example, vllm offline air gapped docker, vllm gpu memory utilization, run llm on prem docker compose, vllm api key

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json