Why vLLM can't find a chat template for your model, and how to fix it
ValueError: As of transformers v4.44, default chat template is no longer allowed, so vLLM's chat api requires you to specify a chat template if the tokenizer does not define one
Also appears as
- jinja2.exceptions.TemplateError: chat_template is not defined
- Cannot use chat template functions because tokenizer.chat_template is not set
Short answer
The /v1/chat/completions endpoint needs a Jinja chat template to turn a messages array into the model's expected prompt format, and base pretrained checkpoints plus some older fine-tunes simply don't ship one. Fix it by supplying --chat-template pointing at a template file matching the model family, or by switching to the raw /v1/completions endpoint with a manually formatted prompt.
Affects: vLLM 0.4 and later serving base or non-instruct checkpoints, or older fine-tunes saved before chat templates became mandatory metadata; the /v1/completions endpoint is unaffected
Get chat completions working
- 1Confirm the model is actually instruction-tuned; base or foundation checkpoints have no conversational format and shouldn't be called through /v1/chat/completions at all.
- 2Check whether tokenizer_config.json in the model repo includes a chat_template field; if a sibling instruct checkpoint from the same family has one, that's your reference.
- 3Download or write a Jinja chat template matching the model's expected special tokens and pass it with --chat-template /path/to/template.jinja.
- 4Restart vLLM and re-test /v1/chat/completions with a simple messages array.
- 5If you don't need chat formatting, use /v1/completions directly with a manually constructed prompt string instead.
How to confirm this is your problem
- /v1/completions works fine but /v1/chat/completions errors immediately on the same model.
- Error explicitly mentions chat_template, Jinja, or template not being defined.
- The model is a base or foundation checkpoint, not one labeled Instruct or Chat.
- A related instruct variant of the same model family serves chat completions correctly.
Root causes and fixes
Model is a base pretrained checkpoint with no conversational fine-tuning or template
Base checkpoints are trained purely for next-token prediction on raw text and were never taught a system, user, and assistant turn structure, so their tokenizer config has no chat_template field for vLLM's Jinja renderer to use when formatting the messages array into a single prompt string.
Fix: Use /v1/completions with a manually built prompt instead of /v1/chat/completions, or switch to the model family's instruct or chat variant, which ships a template.
Older fine-tuned checkpoint predates chat_template becoming a required tokenizer field
Before this was standardized in Hugging Face tooling, many custom fine-tunes stored their prompt format only in a README or training script, not in tokenizer_config.json, so newer transformers or vLLM versions that require an explicit template find nothing to fall back to.
Fix: Write a minimal Jinja chat template matching the exact format your fine-tune was trained on, with the correct system, user, and assistant delimiters, and pass it via --chat-template.
vllm serve MODEL_ID --chat-template ./my_template.jinja
Chat template exists but isn't being picked up due to a path or loading issue
If --chat-template points at a file vLLM can't find, such as a relative path issue inside a container, or the tokenizer is loaded from a local snapshot missing tokenizer_config.json, the template that does exist upstream on the Hugging Face repo never reaches the running server.
Fix: Use an absolute path for --chat-template inside the container, and confirm tokenizer_config.json was actually downloaded alongside the weights.
ls -la /models/MODEL_ID/tokenizer_config.json
Custom served-model-name or local snapshot detaches vLLM from the upstream repo's template
When serving from a locally copied or converted checkpoint directory rather than pulling directly from the Hugging Face hub, any template metadata that lived only in the hub repo's tokenizer_config.json may not have been copied into the local snapshot.
Fix: Explicitly copy tokenizer_config.json, or the full tokenizer files, from the source repo into your local model directory, or pass --chat-template explicitly.
Template exists but references a tool or function-calling format the messages array doesn't provide
Some chat templates conditionally render tool-call blocks only when a tools field is present in the request. Malformed or partially specified requests can trigger template rendering errors that look like a missing-template problem but are actually a template logic branch failing.
Fix: Test with a minimal messages-only request, without tools or functions, first to isolate whether the template itself is missing versus erroring on optional fields.
Diagnostic commands
Check whether the tokenizer config defines a chat template
python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('MODEL_ID'); print(t.chat_template is not None)"False confirms the model genuinely has no built-in template; you must supply one or use /v1/completions.
Test the raw completions endpoint as a fallback
curl http://localhost:8000/v1/completions -d "{\"model\":\"MODEL_ID\",\"prompt\":\"Hello\"}"If this works while chat completions fails, the issue is confined to chat templating, not the model or server generally.
Inspect an instruct sibling model's template as a reference
python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('INSTRUCT_MODEL_ID'); print(t.chat_template)"Comparing this against your base model's format tells you what special tokens and structure a compatible template needs.
Stopping it from happening again
- Always verify a model has chat_template defined, or supply one, as part of your model-onboarding checklist, before it reaches production traffic.
- Keep a small library of known-good chat templates per model family in version control alongside your deployment configs.
- Prefer instruct or chat-tuned checkpoints for any workload that needs the chat completions API; reserve base checkpoints for completion-style or fine-tuning workloads.
- Add an automated smoke test that calls /v1/chat/completions after every model swap, not just /v1/models.
When this becomes an architecture problem
If you're regularly deploying custom or internally fine-tuned checkpoints without a consistent chat template process, standardize the fine-tuning pipeline to always emit tokenizer_config.json with a chat_template, so this stops being a recurring per-model fire drill.
Frequently asked questions
What exactly is a chat template?
It's a Jinja2 template stored in the tokenizer configuration that converts a structured messages array, with system, user, and assistant turns, into the single flat prompt string, with the model's specific special tokens, that the model was actually trained to expect.
Can I just use any chat template as a workaround?
No, using the wrong family's template will silently produce a badly formatted prompt with the wrong delimiter tokens, degrading output quality without necessarily erroring. Match the template to the exact model family, ideally its own tokenizer_config.json if a sibling instruct checkpoint exists.
Does this affect the plain /v1/completions endpoint too?
No, /v1/completions takes a raw prompt string directly from the caller and never needs a chat template. It's only /v1/chat/completions that requires one to convert the messages array format.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Open-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Related problems
vLLM ignores tool or function calls, or returns them as plain text
Unlike the hosted OpenAI API, vLLM does not enable tool or function calling by default. You must launch with --enable-auto-tool-choice plus a --tool-call-parser matching your specific model family, and the model itself must have been trained to emit tool-call syntax its parser recognizes. Without both pieces, requests either error out or the model just writes the function call as plain text in its response content.
vLLM keeps generating past the end of turn instead of stopping
Runaway generation almost always means the token the model actually emits to end a turn doesn't match what vLLM is told to stop on, either because generation_config.json's eos_token_id is stale, a custom stop string wasn't passed in the request, or a fine-tune introduced a new end-of-turn token the base config doesn't know about. Fix it by explicitly passing the correct stop token id or stop strings rather than relying on defaults.
vLLM OpenAI-compatible API returns 404 Not Found
This 404 is almost always a client-side mismatch, not a server bug: either the request hit the wrong route, such as missing the /v1 prefix, or the model field in the request body doesn't match the exact served-model-name (or default model repo id) vLLM registered at startup. Fix the URL and model name to match what /v1/models actually reports.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideThe 2026 Open-Weight LLM Landscape: A Practical Map
A practical map of the 2026 open-weight LLM landscape: model families, license terms, and which model fits your VRAM budget and use case.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.