[Bug]: Function calling not working properly for Qwen2.5-Coder models

[wizche]

Your current environment

Serving Qwen2.5-Coder-32B-Instruct with the latest docker image vllm-openai:v0.6.4.post1:

docker run --gpus all -e VLLM_LOGGING_LEVEL=DEBUG --ipc=host -p 8002:8002 -ti vllm/vllm-openai:v0.6.4.post1 --model "/models/Qwen2.5-Coder-32B-Instruct" --served-model-name "Qwen25-Coder-32B-Instruct" --port 8002 --tensor-parallel-size 4 --enable-auto-tool-choice --tool-call-parser hermes

Model Input Dumps

No response

🐛 Describe the bug

While running the get_temperature function calling example described in the qwen documentation the function calls are not properly parsed.

What I get from the model generation is the following:

{
    "content": "Sure, I'll get the current temperature in San Francisco and also the temperature for tomorrow.\n\nFirst, let's get the current temperature:\n\n<tools>\n{\"name\": \"get_current_temperature\", \"arguments\": {\"location\": \"San Francisco, California, USA\", \"unit\": \"fahrenheit\"}}\n</tools>\n\nNext, to get the temperature for tomorrow, I'll need to know the date. Since today is 2024-09-30, tomorrow will be 2024-10-01. Let's get the temperature for that date:\n\n<tools>\n{\"name\": \"get_temperature_date\", \"arguments\": {\"location\": \"San Francisco, California, USA\", \"date\": \"2024-10-01\", \"unit\": \"fahrenheit\"}}\n</tools>",
    "role": "assistant",
    "function_call": null,
    "tool_calls": []
}

If I serve the Qwen2.5-7B-Instruct model, using the exact same docker command, it works correctly and tool_calls contains the function call.

I'm not sure if this is just a small bug in the tool call parser or if I am suppose to use a different one than hermes.
Serving the model qwen2.5-coder:32b via ollama results in tool calls working properly.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[cedonley]

Is your code using streaming (i.e. stream=True)?

[wizche]

Is your code using streaming (i.e. stream=True)?

No, without streaming

[bbss]

Coder 32B was not trained on the tool tokens.

[wizche]

Coder 32B was not trained on the tool tokens.

But looking at ollama it seems it is, no? https://ollama.com/library/qwen2.5-coder:32b/blobs/e94a8ecb9327

[bbss]

Yes, it's in the template, but not trained:
https://old.reddit.com/r/LocalLLaMA/comments/1gpw8ls/bug_fixes_in_qwen_25_coder_128k_context_window/lwth34x/

[wizche]

Yes, it's in the template, but not trained: https://old.reddit.com/r/LocalLLaMA/comments/1gpw8ls/bug_fixes_in_qwen_25_coder_128k_context_window/lwth34x/

It works incredibly well given that is not trained for

[bbss]

Yeah the json output looks good, but the tokens used for recognizing tool_call vs just json don't. Possibly ollama does something smart with the parsing? I haven't had it work with vLLM tool calling.