Route structured-output through native combined-mode where supported (skip finalization round-trip)

[tombeckenham]

Background

PR #600 added middleware coverage to the structured-output finalization step. That step exists because, historically, providers couldn't combine tools with schema-constrained output in a single call — so the engine runs the agent loop with tools, then makes a separate finalization call against the structured-output adapter to get the typed answer.

That premise has changed for most providers in the last six months.

Provider landscape (May 2026)

Provider	Native streaming + structured output + tools in one call?	Notes
OpenAI Chat Completions (gpt-4o-2024-08-06+)	Turn-based	Schema applies on the natural final turn; no separate call needed
OpenAI Responses API (gpt-5, o-series, 4.1)	Single-pass interleaved	True interleaved stream — function_call + schema-constrained message items in one SSE
Anthropic (Claude 4.5/4.6/4.7)	Single-pass interleaved	GA 2026-01-29 via output_config.format + tools together
Google Gemini 3.x	Single-pass interleaved	Docs explicitly support combined mode (Gemini 3 only)
Google Gemini 2.x	Unsupported / brittle	Treat as unsupported
xAI Grok 4.x family (grok-4.3, grok-4.20-*)	Yes	Docs explicitly: "structured outputs + tools" gated to Grok 4 family. Tool-call args arrive atomically in one SSE chunk (not token-streamed).
xAI Grok 2/3	One-or-the-other	Schema OR tools, not both
Groq	API-rejected	Docs explicit: "Streaming and tool use are not currently supported with Structured Outputs." Request returns 400. Same family as Ollama.
Ollama	Sampler conflict	GBNF schema mask suppresses tool-call prefix tokens (issue ollama-python#546)
OpenRouter	Depends on upstream	Routes to an underlying provider; combined-mode support inherits from whichever model is selected

Net: the finalization round-trip is strictly overhead for modern OpenAI / Anthropic 4.5+ / Gemini 3+ / Grok 4+ adapters, and load-bearing for Groq, Ollama, Gemini 2.x, Grok 2/3, and legacy model versions.

The Groq finding is notable: even though it's an OpenAI-compatible API hosting "modern" open models, Groq has a hard API-level restriction that puts it firmly in the same bucket as Ollama. This is likely the same sampler-conflict story (constrained decoding vs tool-call grammar) Groq hasn't yet solved.

Cost today

When chat({ outputSchema, stream: true }) runs against a modern provider:

1. Agent loop runs one or more iterations (extra provider call(s)).
2. Finalization runs a separate structured-output call.

For the no-tools case this is doubly wasteful — see the related concern raised in the #600 review (the engine no longer short-circuits when tools.length === 0, so a no-tools streaming structured call now makes two requests where it previously made one).

Proposal

Add a per-adapter capability flag declaring whether the adapter natively supports combining tools + schema-constrained output in a single streaming call. Route accordingly in the engine:

if (outputSchema && adapter.capabilities.nativeToolsWithSchema) {
  // Single combined call. Engine wires outputSchema into the regular
  // chat() / chatStream() request. The model emits tool calls and the
  // schema-constrained final text in one pass. Middleware sees everything
  // through existing phases (init / iteration / afterToolCall).
} else {
  // Current path from #600: agent loop, then runStructuredFinalization().
  // Middleware sees the 'structuredOutput' phase.
}

The flag would likely live in feature-support.ts alongside the existing capability matrix.

Implications

• 'structuredOutput' phase tag stays — it simply stops firing for adapters that handle the combination natively. Backward-compatible.
• onStructuredOutputConfig hook still has independent value (swap model / system-prompt for the final formatting step) but its surface narrows to adapters using the fallback path.
• Adapter-level work needed:
  • OpenAI: wire response_format (Chat Completions) / text.format (Responses) into the regular request when outputSchema is present.
  • Anthropic: wire output_config.format into the regular Messages request; drop the forced-tool-use workaround for Claude 4.5+.
  • Gemini: wire responseSchema into the regular request for Gemini 3.x; keep finalization fallback for 2.x.
  • Grok: wire response_format: json_schema into the regular request for Grok 4.x family; keep finalization fallback for Grok 2/3. Watch for the atomic tool-arg chunking — partial-JSON tool-arg parsing should be skipped for Grok.
  • Groq: keep current finalization path. Also: when sending outputSchema + tools + stream to Groq, the engine must not send response_format: json_schema on the streaming call (Groq returns 400). Use json_object or omit during the agent loop, then enforce schema in the finalization step.
  • Ollama: keep current finalization path (sampler conflict means no path forward).
  • OpenRouter: capability lookup must defer to the resolved upstream model; the flag is per-request, not per-adapter, for this provider.

Validation strategy

• Streaming-path validation contract from feat(ai): structured-output middleware coverage (closes #390) #600 (consumer-side, not engine-side) carries over.
• For Promise path: validation still happens at engine boundary against the accumulated text/JSON output, regardless of which adapter path was taken.

Backward compatibility

• Existing middleware that branches on ctx.phase === 'structuredOutput' continues to work for fallback-path adapters.
• Users on modern adapters silently get a faster path with no behavior change visible to them.
• A migration note in the changeset would call out that 'structuredOutput' phase observability is now adapter-dependent.

Scope notes

• This is not a revert of feat(ai): structured-output middleware coverage (closes #390) #600. The fallback path remains essential for Groq, Ollama, Grok 2/3, Gemini 2.x, and older models, and the middleware plumbing is what makes the dual-path proposal possible.
• Out of scope: changing the StructuredOutputMiddlewareConfig public surface, or removing 'structuredOutput' from ChatMiddlewarePhase.

Follow-ups already noted in #600 review

• Fix the no-tools double-call regression first (Critical Ai poc #1 in PR review). That's a prerequisite — landing this capability-flag work would obscure that bug rather than fix it.
• tools field on StructuredOutputMiddlewareConfig should be Omit-ed structurally (PR review Important chore: migrate from tsup to tsdown #2).

Sources

OpenAI

• Structured model outputs — OpenAI API guide
• Function calling — OpenAI API guide
• Introducing Structured Outputs in the API — OpenAI announcement
• Introduction to Structured Outputs — OpenAI Cookbook
• Migrate to the Responses API — OpenAI
• Streaming API responses — OpenAI
• Using response_format with function calling — OpenAI Community
• Streaming mode with structured output in Responses API — OpenAI Community
• Responses API streaming events guide — OpenAI Community
• openai-python Chat Completions resource source

Anthropic

• Structured outputs — Claude API docs
• Claude Platform API release notes
• Tool use overview — Claude API docs
• Fine-grained tool streaming — Claude API docs
• Anthropic engineering — Advanced tool use
• HN discussion — Structured outputs on the Claude Developer Platform

Google Gemini

• Structured outputs — Gemini API docs
• Function calling with the Gemini API
• Combine built-in tools and function calling
• Generating content reference
• Gemini API changelog / release notes
• Improving Structured Outputs in the Gemini API — Google blog, Nov 5 2025
• Forum thread — pre-Gemini-3 structured-output + function-calling friction

xAI Grok

• Structured Outputs — xAI Docs
• Function Calling — xAI Docs
• Streaming Response — xAI Docs
• Streaming & Synchronous Requests — xAI Docs
• Tools Overview — xAI Docs
• Models catalog — xAI Docs
• xai-sdk-python on GitHub
• Instructor + xAI integration guide

Groq (the API-level holdout)

• Structured Outputs — Groq Docs — explicit restriction quote
• Tool Use — Groq Docs
• Text Generation — Groq Docs
• API Reference — Groq Docs
• LangChain issue #23629 — response_format does not support streaming on Groq
• Groq community — gpt-oss-120b structured-output reliability
• Groq community — OpenAI gpt-oss feature request

Ollama (the sampler-level holdout)

• Ollama API spec — docs/api.md
• Ollama Blog — Structured outputs
• Ollama Blog — Streaming responses with tool calling
• Ollama Docs — Structured Outputs capability
• ollama-python #546 — Tools not working when providing a schema via format (the canonical bug)
• ollama #6002 — JSON Schema conformity using llama.cpp Grammar for Tool Calling
• ollama #13967 — Cloud: oneOf not supported for tool calling
• ollama #11691 — Structured output with OpenAI SDK and gpt-oss not working
• ollama #10937 — format: date-time ignored on /v1/ endpoint
• ollama #8325 — Invalid JSON Schema when using pattern
• ollama #8222 — ToolFunction.Parameters should be json.RawMessage
• ollama #10001 — Improve compatibility with OpenAI json_schema response_format
• llama.cpp grammars (GBNF) — README
• DeepWiki — Ollama OpenAI Compatibility Layer
• Glukhov — Ollama GPT-OSS Structured Output Issues

Related TanStack AI artifacts

• PR feat(ai): structured-output middleware coverage (closes #390) #600 (closes chat({ outputSchema }) runs final structured output request outside middleware #390) — Structured-output middleware coverage
• Issue chat({ outputSchema }) runs final structured output request outside middleware #390 — Original request for middleware visibility into the structured-output finalization step