feat: OpenAI-compatible tool calling support#14
Open
timbellay wants to merge 1 commit into
Open
Conversation
- tools + tool_choice fields in ChatCompletionRequest schema - tool role messages with tool_call_id for multi-turn conversations - Tools threaded to tokenizer.apply_chat_template(tools=...) - Output parser for Qwen3 <tool_call> and Mistral [TOOL_CALLS] formats - Structured tool_calls array in response with finish_reason: tool_calls - All 112 existing tests pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Adds tool calling to the chat completions endpoint, enabling use with AI agent frameworks (MCP, LangChain, AgentPack) that expect the OpenAI tools API.
What changed
app/routes/chat.py
tools and tool_choice fields in ChatCompletionRequest
tool role with tool_call_id in ChatMessage for multi-turn tool conversations
_parse_tool_calls() — extracts structured tool calls from raw model output (Qwen3 <tool_call> tags, Mistral [TOOL_CALLS] format)
tool_calls array and nullable content in response, finish_reason: "tool_calls"
app/models/text_chat.py and app/models/qwen_vl.py
tools parameter threaded through count_tokens() and prepare_inputs() to tokenizer.apply_chat_template(tools=...)
HF tokenizers handle prompt formatting per model family automatically — no custom templates needed
How it works
Client sends tools array in the request (OpenAI format)
Tools are passed to apply_chat_template(tools=...) — the model's tokenizer handles injection
Model output is scanned by _parse_tool_calls() for known patterns
If tool calls are detected, response returns structured tool_calls with finish_reason: "tool_calls"
Client executes tools, sends results back as role: "tool" messages
Model generates final natural language response
Tested on
Jetson Thor (CUDA 13, JetPack 7) with Qwen/Qwen3-4B-Instruct-2507:
Single tool call ✅
Parallel multi-tool calls (2 simultaneous) ✅
Full tool result round-trip (tool results → natural language synthesis) ✅
112/112 existing tests pass ✅