server: support named tool_choice, and stop swallowing a trailing tool call into content - #26849
Closed
seyeong-han wants to merge 2 commits into
Closed
server: support named tool_choice, and stop swallowing a trailing tool call into content#26849seyeong-han wants to merge 2 commits into
seyeong-han wants to merge 2 commits into
Conversation
…to auto
A named tool_choice — {"type":"function","function":{"name":"..."}} — was being
dropped. json_value() only understands the string form, so on the object it logged a
type warning and returned the default "auto". Nothing surfaced this to the caller.
For templates whose grammar is lazy under "auto" this is not cosmetic. The Muse
Glimmer handler only arms its grammar once the trigger
<|start|>assistant( to=(?!self<|message|>)(?!user<|message|>)[^<]*?<|message|>)
matches. Asked to call a tool with empty arguments, the model writes
to=skill<|message|><atem:function_calls>
<atem:invoke name="skill"></atem:invoke>
</atem:function_calls>
with no <|start|>assistant header — it never leaves the analysis channel. The trigger
therefore never fires, the grammar never arms, and the call stays buried in
reasoning_content: finish_reason "stop", no tool_calls. Deterministic at temperature 0.
With tool_choice "required" the same prompt works, because that path builds a
non-lazy grammar.
Parse the object form, narrow tools to the named function and map it to "required".
The grammar is then non-lazy and admits only that call, which is the OpenAI semantics
and needs no per-template changes. Malformed input now 400s instead of silently
behaving as "auto".
Tested against muse-glimmer-30B bf16:
named tool_choice + empty args finish_reason stop, no tool_calls -> tool_calls skill {}
named tool_choice + args unchanged, skill {command:invoke, skill_name:deploy}
named tool_choice + no-arg tool unchanged, list_skills {}
tool_choice required x2 unchanged, translate_to_spanish / get_weather
5 malformed tool_choice inputs 400 with specific messages
Note the model still emits the abortive call into reasoning_content; the grammar now
forces a well-formed call afterwards, so the API response is correct. The underlying
channel-transition bug is a model issue and is reported separately.
The model routinely answers the user and calls a tool in one generation:
<prose to the user><|eom|><|start|>assistant to=<tool><|message|>
<atem:function_calls>...</atem:function_calls>
The `final` rule read content with until("<|eot|>"). There is no <|eot|> before
the call, so content ran to the end of the turn and absorbed the tool markup.
The call was never parsed into tool_calls, and the raw ATEM/channel markup was
handed to the caller as chat text.
Measured on tau2-bench telecom (114 tasks, muse-glimmer-30B bf16, user simulator
muse-spark-1.2-aai2): 43 leaked assistant turns across 19 of 113 tasks — 16.8% of
tasks, 1.9% of all assistant turns. Details in P2455728274.
Three malformed shapes were observed, so the parser is made tolerant of all of
them rather than of one:
1. <prose><|eom|><|start|>assistant to=<tool><|message|><atem:function_calls>
— all 43 tau2 leaks
2. <prose>\n<atem:function_calls> — no header, no delimiter
3. to=<tool><|message|><atem:function_calls> inside the analysis channel with
no <|start|> at all — P2455602726 repro 3
Changes, all in common_chat_params_init_muse_glimmer:
- content stops at <|eot|>, <|eom|> or <atem:function_calls>
- the " to=<tool><|message|>" call header is optional
- a turn may be content followed by tool calls, with both the <|eom|> and the
<|start|>assistant delimiters optional
Re-ran the same 19 tasks that leaked, with the fix:
leaked tasks 19/19 (100%) -> 0/19 (0%)
leaked turns 43 -> 0 (of 425 assistant turns)
pass^1 84.2% -> 94.7%
The leak elimination is deterministic. The score movement is 2 tasks on n=19 in
a benchmark with a stochastic user simulator and no control re-run, so it is
suggestive only, not attributable.
Regressions: the three P2455602726 repros and the named/no-arg tool_choice
variants all still return correct tool_calls.
This makes the parser tolerant of malformed output; it does not make the model
emit canonical output. The underlying behaviour is worth fixing model-side.
Author
|
cc @ngxson @pcuenca @albertodepaola — this is the upstream version of the tool-calling fixes we reviewed internally in huggingface/new-model-addition-onyx-llama.cpp#13. Same two commits, rebased onto master; I could not add you as reviewers directly since I do not have write access here. |
|
Hi @seyeong-han, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
ggml-gh-bot
Bot
added
the
draft
github-actions
Bot
removed
the
draft
seyeong-han
marked this pull request as ready for review
August 10, 2026 16:25
aldehir
reviewed
Aug 10, 2026
Comment on lines
+924
to
+926
| // Named choice is "call exactly this function", so narrow `tools` to that one entry | ||
| // and treat it as "required". The grammar then admits only that call, which is the | ||
| // OpenAI semantics and needs no per-template support. |
Contributor
There was a problem hiding this comment.
I am not privy to internal discussions, but AFAIK vLLM does not do this. In vLLM, all tools are kept in the context and the specified tool is forced via grammar constrained decoding. Filtering tools also has negative connotations for prefix caching, as two requests with the same system prompt, same tool definitions, but different tool_choice will trigger reprocessing of the system prompt.
Author
|
This issue has been moved to #26879 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two server-side tool-calling bugs, found while exercising the Muse Glimmer chat handler against a multi-turn agentic benchmark. The first is generic and affects every model; the second is specific to the Muse Glimmer parser.
1. Named
tool_choicewas silently downgraded to"auto"tool_choicemay be a string ("auto"/"none"/"required") or, to force one specific function, an object:json_value()only understands the string form. Given the object it logs a type warning and returns the default"auto", so the request is served as if no choice had been made — nothing surfaces to the caller.That is not cosmetic for templates whose grammar is lazy under
"auto". The Muse Glimmer handler only arms its grammar once this trigger matches:Asked to call a tool with empty arguments, the model writes the call without the
<|start|>assistantheader and never leaves the analysis channel:The trigger never fires, the grammar never arms, and the call stays buried in
reasoning_content—finish_reason: "stop", notool_calls. Deterministic at temperature 0.tool_choice: "required"was unaffected because that path builds a non-lazy grammar.Fix: parse the object form, narrow
toolsto the named function and map it to"required". The grammar is then non-lazy and admits only that call, which is the OpenAI semantics and needs no per-template changes. Malformedtool_choicenow returns 400 instead of silently behaving as"auto".2. A trailing tool call was swallowed into
message.contentThe Muse Glimmer parser read assistant content with
until("<|eot|>"). The model routinely answers the user and calls a tool in one generation, and there is no<|eot|>before the call — so content ran to the end of the generation and absorbed the tool markup, which was then delivered to the user as chat text.On a multi-turn agentic run this leaked raw tool markup into the visible reply on 43 turns across 19 of 113 tasks (~17% of tasks affected).
Fix: make content stop at
<|eom|>/<atem:function_calls>, make theto=<tool><|message|>header optional, and allow content-then-tool-call with both delimiters optional.Testing
Against the Muse Glimmer 30B bf16 weights:
tool_choice, empty argsfinish_reason: stop, notool_callstool_calls: skill {}tool_choice, with argstool_choice, no-parameter tooltool_choice: "required"(2 cases)tool_choiceinputs"auto"Re-running the 19 tasks that leaked tool markup: 19/19 leaking → 0/19, and 43 leaked turns → 0 of 425.