Why vLLM tool and function calling doesn't work, and how to enable it correctly
Assistant message content contains literal function-call JSON text instead of a structured tool_calls field
Also appears as
- ValueError: Cannot use tool calling with the model's default chat template, no tool parser registered
- openai.BadRequestError: 'tool_choice' is not currently supported
Short answer
Unlike the hosted OpenAI API, vLLM does not enable tool or function calling by default. You must launch with --enable-auto-tool-choice plus a --tool-call-parser matching your specific model family, and the model itself must have been trained to emit tool-call syntax its parser recognizes. Without both pieces, requests either error out or the model just writes the function call as plain text in its response content.
Affects: vLLM 0.5 and later; requires a model actually trained for tool use plus explicit server flags, most common when teams assume tool calling works like the hosted OpenAI API by default
Enable tool calling correctly
- 1Confirm the model was actually trained or fine-tuned for tool use; check its model card for tool-calling support and the expected output format.
- 2Launch vLLM with --enable-auto-tool-choice and the matching --tool-call-parser for that model family.
- 3If unsure which parser matches, check vLLM's documented list of supported tool-call parsers for the exact name to pass.
- 4Send a request with tool_choice set to auto, or a specific tool, and a well-formed tools array in the request body.
- 5Inspect the response for a structured tool_calls field rather than plain-text content that merely looks like a function call.
How to confirm this is your problem
- Response content contains something like a JSON blob describing a function call, but the OpenAI-format tool_calls field is empty or absent.
- Server returns an error mentioning tool_choice or tool parser as soon as a request includes a tools array.
- Same request format works fine against the hosted OpenAI API but fails or misbehaves against your self-hosted vLLM endpoint.
- Works for one model on your fleet but not another, even with identical request bodies.
Root causes and fixes
Server launched without --enable-auto-tool-choice and a matching --tool-call-parser
vLLM's OpenAI-compatible layer parses tool calls out of the model's raw text output using a per-model-family parser. Without explicitly enabling auto tool choice and naming the correct parser, vLLM has no logic wired up to detect and extract tool-call syntax, so it just passes the model's raw text straight through as message content.
Fix: Add both --enable-auto-tool-choice and --tool-call-parser matching your model family to the vllm serve command.
vllm serve MODEL_ID --enable-auto-tool-choice --tool-call-parser hermes
Model was never trained to emit tool-call syntax at all
Tool calling isn't a server-side feature alone; it requires the model to have learned during fine-tuning to emit a specific structured format, such as JSON-like function call blocks or special tokens, when it decides to invoke a tool. A base or plain chat model with no tool-use training data has nothing for even a correctly configured parser to extract.
Fix: Confirm on the model's card or release notes that it explicitly supports tool or function calling, and pick a tool-parser value that matches that specific model's documented format.
Wrong tool-call-parser chosen for the model family
Each supported parser expects a specific textual format for tool calls, differing between model families such as Hermes-style, Mistral-style, or Llama 3 JSON style. Pairing the wrong parser with a model means it either fails to match anything and falls through as plain text, or partially misparses the output.
Fix: Match --tool-call-parser exactly to the model family per vLLM's documented compatibility list, not just whichever parser you used for a different model.
vllm serve MODEL_ID --enable-auto-tool-choice --tool-call-parser llama3_json
Chat template doesn't render the tools array into the prompt at all
If the model's chat template doesn't have a branch that formats the tools list and tool-use instructions into the rendered prompt, the model literally never sees what tools are available, regardless of server-side parser configuration, so it can't emit a valid tool call even if it's otherwise capable.
Fix: Check whether the chat template used actually has a conditional branch for tools; if not, supply a template that does, matching the model's trained tool-use format.
tool_choice value not supported by the installed vLLM version
Some tool_choice modes, such as forcing a specific named tool or a required mode, were added in later vLLM releases. An older pinned version can reject or ignore tool_choice values it doesn't yet implement, producing an error or silently falling back to unconstrained generation.
Fix: Check the vLLM changelog for when your desired tool_choice mode was added, and upgrade if you're on an older pinned version.
pip install -U vllm
Diagnostic commands
Check whether the model card documents tool-calling support
Review the model card on Hugging Face for tool or function calling documentation
If the card doesn't mention tool or function calling, or a specific chat format for it, don't expect it to work regardless of server flags.
Confirm the flags vLLM was launched with
ps aux | grep vllm
Both --enable-auto-tool-choice and --tool-call-parser must be present; missing either one means tool calling is effectively off.
Send a minimal tool-calling test request
curl http://localhost:8000/v1/chat/completions -d "{\"model\":\"MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"weather in Boston\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"}}],\"tool_choice\":\"auto\"}"Inspect the response JSON for a populated tool_calls array under choices[0].message; plain text content instead means the parser isn't extracting it.
Stopping it from happening again
- Pick models explicitly documented and benchmarked for tool or function calling before building an agent workflow around them.
- Standardize on documenting the exact enable-auto-tool-choice and tool-call-parser pairing per model in your deployment configs.
- Add an automated integration test that exercises a real tool-calling round trip after every deployment, not just a basic chat completion.
- Keep vLLM version pinned and re-test tool-calling behavior explicitly before any upgrade, since parser support evolves across releases.
When this becomes an architecture problem
If you need reliable structured tool use across a fleet of different open-weight models with inconsistent tool-calling training, that's an application-layer design question, such as standardizing on a single fine-tuned tool-use model or a schema-constrained decoding approach, worth solving deliberately rather than chasing parser compatibility per model.
Frequently asked questions
Why doesn't vLLM support tool calling out of the box like the OpenAI API?
Because vLLM serves arbitrary open-weight models, each with its own trained tool-call output format, there's no single universal parser. You must explicitly tell vLLM which parser matches the specific model you're serving, and enable auto tool choice.
Can I add tool calling to a model that wasn't trained for it?
Not reliably. Prompting a non-tool-trained model to emit JSON in the shape of a tool call can sometimes work loosely, but vLLM's structured tool_calls extraction depends on the model consistently producing the exact format its parser expects, which untrained models won't do reliably.
How do I pick the right tool-call-parser value?
Check vLLM's documentation for the list of supported parsers and which model families each one targets, commonly named after the family such as hermes, mistral, or llama3_json, and match it to your specific model, not just copy a value used for a different model.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Open-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Related problems
vLLM error: no chat template found for this model
The /v1/chat/completions endpoint needs a Jinja chat template to turn a messages array into the model's expected prompt format, and base pretrained checkpoints plus some older fine-tunes simply don't ship one. Fix it by supplying --chat-template pointing at a template file matching the model family, or by switching to the raw /v1/completions endpoint with a manually formatted prompt.
vLLM OpenAI-compatible API returns 404 Not Found
This 404 is almost always a client-side mismatch, not a server bug: either the request hit the wrong route, such as missing the /v1 prefix, or the model field in the request body doesn't match the exact served-model-name (or default model repo id) vLLM registered at startup. Fix the URL and model name to match what /v1/models actually reports.
vLLM keeps generating past the end of turn instead of stopping
Runaway generation almost always means the token the model actually emits to end a turn doesn't match what vLLM is told to stop on, either because generation_config.json's eos_token_id is stale, a custom stop string wasn't passed in the request, or a fine-tune introduced a new end-of-turn token the base config doesn't know about. Fix it by explicitly passing the correct stop token id or stop strings rather than relying on defaults.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideThe 2026 Open-Weight LLM Landscape: A Practical Map
A practical map of the 2026 open-weight LLM landscape: model families, license terms, and which model fits your VRAM budget and use case.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.