# Why vLLM tool and function calling doesn't work, and how to enable it correctly

> Source: https://netray.co/troubleshoot/vllm-tool-calling-not-working
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Unlike the hosted OpenAI API, vLLM does not enable tool or function calling by default. You must launch with --enable-auto-tool-choice plus a --tool-call-parser matching your specific model family, and the model itself must have been trained to emit tool-call syntax its parser recognizes. Without both pieces, requests either error out or the model just writes the function call as plain text in its response content.

## Error Title

vLLM ignores tool or function calls, or returns them as plain text

## Error Signature

Assistant message content contains literal function-call JSON text instead of a structured tool_calls field

## Alternate Signatures

- "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set
- ValueError: Cannot use tool calling with the model's default chat template, no tool parser registered
- openai.BadRequestError: 'tool_choice' is not currently supported

## Category

serving

## Stack

- vllm

## Affects

vLLM 0.5 and later; requires a model actually trained for tool use plus explicit server flags, most common when teams assume tool calling works like the hosted OpenAI API by default

## Quick Fix

### Enable tool calling correctly

**Steps:**

- Confirm the model was actually trained or fine-tuned for tool use; check its model card for tool-calling support and the expected output format.
- Launch vLLM with --enable-auto-tool-choice and the matching --tool-call-parser for that model family.
- If unsure which parser matches, check vLLM's documented list of supported tool-call parsers for the exact name to pass.
- Send a request with tool_choice set to auto, or a specific tool, and a well-formed tools array in the request body.
- Inspect the response for a structured tool_calls field rather than plain-text content that merely looks like a function call.

## Symptoms

- Response content contains something like a JSON blob describing a function call, but the OpenAI-format tool_calls field is empty or absent.
- Server returns an error mentioning tool_choice or tool parser as soon as a request includes a tools array.
- Same request format works fine against the hosted OpenAI API but fails or misbehaves against your self-hosted vLLM endpoint.
- Works for one model on your fleet but not another, even with identical request bodies.

## Causes

### Server launched without --enable-auto-tool-choice and a matching --tool-call-parser

**Likelihood:** most-common

**Explanation:** vLLM's OpenAI-compatible layer parses tool calls out of the model's raw text output using a per-model-family parser. Without explicitly enabling auto tool choice and naming the correct parser, vLLM has no logic wired up to detect and extract tool-call syntax, so it just passes the model's raw text straight through as message content.

**Fix:** Add both --enable-auto-tool-choice and --tool-call-parser matching your model family to the vllm serve command.

**Commands:**

- vllm serve MODEL_ID --enable-auto-tool-choice --tool-call-parser hermes

### Model was never trained to emit tool-call syntax at all

**Likelihood:** common

**Explanation:** Tool calling isn't a server-side feature alone; it requires the model to have learned during fine-tuning to emit a specific structured format, such as JSON-like function call blocks or special tokens, when it decides to invoke a tool. A base or plain chat model with no tool-use training data has nothing for even a correctly configured parser to extract.

**Fix:** Confirm on the model's card or release notes that it explicitly supports tool or function calling, and pick a tool-parser value that matches that specific model's documented format.

### Wrong tool-call-parser chosen for the model family

**Likelihood:** common

**Explanation:** Each supported parser expects a specific textual format for tool calls, differing between model families such as Hermes-style, Mistral-style, or Llama 3 JSON style. Pairing the wrong parser with a model means it either fails to match anything and falls through as plain text, or partially misparses the output.

**Fix:** Match --tool-call-parser exactly to the model family per vLLM's documented compatibility list, not just whichever parser you used for a different model.

**Commands:**

- vllm serve MODEL_ID --enable-auto-tool-choice --tool-call-parser llama3_json

### Chat template doesn't render the tools array into the prompt at all

**Likelihood:** occasional

**Explanation:** If the model's chat template doesn't have a branch that formats the tools list and tool-use instructions into the rendered prompt, the model literally never sees what tools are available, regardless of server-side parser configuration, so it can't emit a valid tool call even if it's otherwise capable.

**Fix:** Check whether the chat template used actually has a conditional branch for tools; if not, supply a template that does, matching the model's trained tool-use format.

### tool_choice value not supported by the installed vLLM version

**Likelihood:** rare

**Explanation:** Some tool_choice modes, such as forcing a specific named tool or a required mode, were added in later vLLM releases. An older pinned version can reject or ignore tool_choice values it doesn't yet implement, producing an error or silently falling back to unconstrained generation.

**Fix:** Check the vLLM changelog for when your desired tool_choice mode was added, and upgrade if you're on an older pinned version.

**Commands:**

- pip install -U vllm

## Diagnostics

### Check whether the model card documents tool-calling support

**Command:** Review the model card on Hugging Face for tool or function calling documentation

**Interpretation:** If the card doesn't mention tool or function calling, or a specific chat format for it, don't expect it to work regardless of server flags.

### Confirm the flags vLLM was launched with

**Command:** ps aux | grep vllm

**Interpretation:** Both --enable-auto-tool-choice and --tool-call-parser must be present; missing either one means tool calling is effectively off.

### Send a minimal tool-calling test request

**Command:** curl http://localhost:8000/v1/chat/completions -d "{\"model\":\"MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"weather in Boston\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\"}}],\"tool_choice\":\"auto\"}"

**Interpretation:** Inspect the response JSON for a populated tool_calls array under choices[0].message; plain text content instead means the parser isn't extracting it.

## Prevention

- Pick models explicitly documented and benchmarked for tool or function calling before building an agent workflow around them.
- Standardize on documenting the exact enable-auto-tool-choice and tool-call-parser pairing per model in your deployment configs.
- Add an automated integration test that exercises a real tool-calling round trip after every deployment, not just a basic chat completion.
- Keep vLLM version pinned and re-test tool-calling behavior explicitly before any upgrade, since parser support evolves across releases.

## When To Escalate

If you need reliable structured tool use across a fleet of different open-weight models with inconsistent tool-calling training, that's an application-layer design question, such as standardizing on a single fine-tuned tool-use model or a schema-constrained decoding approach, worth solving deliberately rather than chasing parser compatibility per model.

## Faq

### Why doesn't vLLM support tool calling out of the box like the OpenAI API?

**Answer:** Because vLLM serves arbitrary open-weight models, each with its own trained tool-call output format, there's no single universal parser. You must explicitly tell vLLM which parser matches the specific model you're serving, and enable auto tool choice.

### Can I add tool calling to a model that wasn't trained for it?

**Answer:** Not reliably. Prompting a non-tool-trained model to emit JSON in the shape of a tool call can sometimes work loosely, but vLLM's structured tool_calls extraction depends on the model consistently producing the exact format its parser expects, which untrained models won't do reliably.

### How do I pick the right tool-call-parser value?

**Answer:** Check vLLM's documentation for the list of supported parsers and which model families each one targets, commonly named after the family such as hermes, mistral, or llama3_json, and match it to your specific model, not just copy a value used for a different model.

## Search terms

vllm tool calling not working, vllm function calling, enable-auto-tool-choice, vllm tool-call-parser, vllm tool_calls empty, vllm function call plain text

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json