fix(openai): stream usage non-zero when tools are enabled - #9941
Merged
Conversation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The streaming chat-completions worker for tool-bearing requests
(processTools in core/http/endpoints/openai/chat.go) never forwarded the
cumulative TokenUsage from ComputeChoices to the chunks it placed on the
responses channel. The outer streaming loop's running usage tracker
therefore stayed at the zero value, and the include_usage trailer
reported {prompt_tokens:0, completion_tokens:0, total_tokens:0} whenever
the request carried a `tools` array. Without tools, the alternative
`process` path stamps Usage on every chunk, so that path was unaffected.
Forward the final TokenUsage via a usage-only sentinel chunk (empty
Choices, populated Usage) emitted right before close(responses). The
outer loop's per-chunk Usage capture moves above the empty-Choices skip
so the sentinel updates the tracker without ever reaching the wire,
keeping the existing OpenAI spec contract (intermediate chunks carry no
`usage` field, and the deferred-final-chunk helpers remain Usage-free
per the regression test for issue #8546).
Adds streamUsageFromTokenUsage, usageSentinelChunk, and
applyChunkToUsage helpers with focused Ginkgo coverage plus a flow-level
test that mirrors the outer-loop sequence.
Fixes #9927
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
Replace the usage-only sentinel SSE chunk introduced in the previous
commit with a plain return value. The streaming workers process and
processTools (now extracted as package-level processStream and
processStreamWithTools) return (backend.TokenUsage, error); the outer
ChatEndpoint loop reads the cumulative counts off the existing `ended`
channel (now carrying streamWorkerResult{usage, err}) and builds the
include_usage trailer from a normal Go value after the LOOP exits.
This drops the empty-Choices "skip but capture Usage" rule from the
outer loop and removes the usageSentinelChunk / applyChunkToUsage
helpers entirely. The SSE responses channel is back to a single
purpose: wire chunks only.
processStream and processStreamWithTools move into chat_stream_workers.go
so they can be exercised directly from tests. The chat_stream_usage_test.go
suite now drives the workers with a mocked backend.ModelInferenceFunc
and asserts on the returned TokenUsage. The regression coverage for
issue #9927 is therefore behavioral: reverting the fix (discarding
ComputeChoices' usage return) makes the assertions fail with concrete
count mismatches.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
richiejp
added a commit
to richiejp/LocalAI
that referenced
this pull request
May 22, 2026
Cloud-proxy backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers via a Go gRPC backend, with optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response). Adds a routing-module middleware stack — admission control, content-aware model routing with classifier+rerank candidates, PII detection/redaction (regex + NER), per-user/per-key billing recorder with GORM or in-memory backend — plus the React UI surfaces for the new model-editor cloud-proxy fields and a Traces page that records full passthrough bodies. Subsystems: * cloud-proxy: backend/go/cloud-proxy/ implements the gRPC Forward RPC, OpenAI passthrough, and Anthropic translate (with full tool calling: spec + legacy flat tool_choice forms, tool_result content blocks, temperature/top_p mutex). Schema-side: omitempty discipline on OpenAIRequest so re-marshal doesn't leak union-overlap fields, `json:"-"` on internal staging fields, MaxCompletionTokens alias collapsed to Maxtokens in request middleware. * routing: admission, content-router (embedding cache + classifier + rerank + score), PII (config, redactor, stream filter, event store, OpenAI/Anthropic adapters), billing (StatsBackend interface with gorm/inmem implementations, Recorder with prom counters, LocalUser fallback for no-auth boxes). * middleware: UsageMiddleware records via billing.Recorder with handler-stamp (StampUsage) preferred over body-parse; admission + route-model + usage-stamp + trace middlewares; context-key constants for the routing modules to communicate token counts, models, and correlation IDs to UsageMiddleware. * observability: BackendTrace ring buffer now stores the full request body (capped at 1 MB); MITM proxy emits structured trace events; router classifier failures surface to /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0) with katanemo icon. * docker: .dockerignore excludes in-place llama.cpp clone so the cloud-proxy backend image builds against the pinned commit. * UI: cloud-proxy model templates (Anthropic defaults to translate), ConfigFieldRenderer proxy.mode/proxy.provider dropdowns with descriptions, Traces page renders Request Body pane, useChat.js drops the unconditional max_tokens=contextSize cap. Rebased onto master 0b2ae3c ("fix(openai): stream usage non-zero when tools are enabled mudler#9941"); the master refactor that extracted process/processTools into chat_stream_workers.go is preserved, and the new piiRedactor/piiEvents are threaded into ChatEndpoint without reintroducing the inline workers. UsageMiddleware preserves master's Source / APIKeyID / APIKeyName recording on top of the new billing.Recorder API. statsRecorder gets a SIGTERM Close() handler so the gorm backend's pending batch is drained on graceful exit. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>
richiejp
added a commit
to richiejp/LocalAI
that referenced
this pull request
May 23, 2026
Cloud-proxy backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers via a Go gRPC backend, with optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response). Adds a routing-module middleware stack — admission control, content-aware model routing with classifier+rerank candidates, PII detection/redaction (regex + NER), per-user/per-key billing recorder with GORM or in-memory backend — plus the React UI surfaces for the new model-editor cloud-proxy fields and a Traces page that records full passthrough bodies. Subsystems: * cloud-proxy: backend/go/cloud-proxy/ implements the gRPC Forward RPC, OpenAI passthrough, and Anthropic translate (with full tool calling: spec + legacy flat tool_choice forms, tool_result content blocks, temperature/top_p mutex). Schema-side: omitempty discipline on OpenAIRequest so re-marshal doesn't leak union-overlap fields, `json:"-"` on internal staging fields, MaxCompletionTokens alias collapsed to Maxtokens in request middleware. * routing: admission, content-router (embedding cache + classifier + rerank + score), PII (config, redactor, stream filter, event store, OpenAI/Anthropic adapters), billing (StatsBackend interface with gorm/inmem implementations, Recorder with prom counters, LocalUser fallback for no-auth boxes). * middleware: UsageMiddleware records via billing.Recorder with handler-stamp (StampUsage) preferred over body-parse; admission + route-model + usage-stamp + trace middlewares; context-key constants for the routing modules to communicate token counts, models, and correlation IDs to UsageMiddleware. * observability: BackendTrace ring buffer now stores the full request body (capped at 1 MB); MITM proxy emits structured trace events; router classifier failures surface to /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0) with katanemo icon. * docker: .dockerignore excludes in-place llama.cpp clone so the cloud-proxy backend image builds against the pinned commit. * UI: cloud-proxy model templates (Anthropic defaults to translate), ConfigFieldRenderer proxy.mode/proxy.provider dropdowns with descriptions, Traces page renders Request Body pane, useChat.js drops the unconditional max_tokens=contextSize cap. Rebased onto master 0b2ae3c ("fix(openai): stream usage non-zero when tools are enabled mudler#9941"); the master refactor that extracted process/processTools into chat_stream_workers.go is preserved, and the new piiRedactor/piiEvents are threaded into ChatEndpoint without reintroducing the inline workers. UsageMiddleware preserves master's Source / APIKeyID / APIKeyName recording on top of the new billing.Recorder API. statsRecorder gets a SIGTERM Close() handler so the gorm backend's pending batch is drained on graceful exit. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>
richiejp
added a commit
to richiejp/LocalAI
that referenced
this pull request
May 24, 2026
Cloud-proxy backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers via a Go gRPC backend, with optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response). Adds a routing-module middleware stack — admission control, content-aware model routing with classifier+rerank candidates, PII detection/redaction (regex + NER), per-user/per-key billing recorder with GORM or in-memory backend — plus the React UI surfaces for the new model-editor cloud-proxy fields and a Traces page that records full passthrough bodies. Subsystems: * cloud-proxy: backend/go/cloud-proxy/ implements the gRPC Forward RPC, OpenAI passthrough, and Anthropic translate (with full tool calling: spec + legacy flat tool_choice forms, tool_result content blocks, temperature/top_p mutex). Schema-side: omitempty discipline on OpenAIRequest so re-marshal doesn't leak union-overlap fields, `json:"-"` on internal staging fields, MaxCompletionTokens alias collapsed to Maxtokens in request middleware. * routing: admission, content-router (embedding cache + classifier + rerank + score), PII (config, redactor, stream filter, event store, OpenAI/Anthropic adapters), billing (StatsBackend interface with gorm/inmem implementations, Recorder with prom counters, LocalUser fallback for no-auth boxes). * middleware: UsageMiddleware records via billing.Recorder with handler-stamp (StampUsage) preferred over body-parse; admission + route-model + usage-stamp + trace middlewares; context-key constants for the routing modules to communicate token counts, models, and correlation IDs to UsageMiddleware. * observability: BackendTrace ring buffer now stores the full request body (capped at 1 MB); MITM proxy emits structured trace events; router classifier failures surface to /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0) with katanemo icon. * docker: .dockerignore excludes in-place llama.cpp clone so the cloud-proxy backend image builds against the pinned commit. * UI: cloud-proxy model templates (Anthropic defaults to translate), ConfigFieldRenderer proxy.mode/proxy.provider dropdowns with descriptions, Traces page renders Request Body pane, useChat.js drops the unconditional max_tokens=contextSize cap. Rebased onto master 0b2ae3c ("fix(openai): stream usage non-zero when tools are enabled mudler#9941"); the master refactor that extracted process/processTools into chat_stream_workers.go is preserved, and the new piiRedactor/piiEvents are threaded into ChatEndpoint without reintroducing the inline workers. UsageMiddleware preserves master's Source / APIKeyID / APIKeyName recording on top of the new billing.Recorder API. statsRecorder gets a SIGTERM Close() handler so the gorm backend's pending batch is drained on graceful exit. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>
richiejp
added a commit
to richiejp/LocalAI
that referenced
this pull request
May 24, 2026
Cloud-proxy backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers via a Go gRPC backend, with optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response). Adds a routing-module middleware stack — admission control, content-aware model routing with classifier+rerank candidates, PII detection/redaction (regex + NER), per-user/per-key billing recorder with GORM or in-memory backend — plus the React UI surfaces for the new model-editor cloud-proxy fields and a Traces page that records full passthrough bodies. Subsystems: * cloud-proxy: backend/go/cloud-proxy/ implements the gRPC Forward RPC, OpenAI passthrough, and Anthropic translate (with full tool calling: spec + legacy flat tool_choice forms, tool_result content blocks, temperature/top_p mutex). Schema-side: omitempty discipline on OpenAIRequest so re-marshal doesn't leak union-overlap fields, `json:"-"` on internal staging fields, MaxCompletionTokens alias collapsed to Maxtokens in request middleware. * routing: admission, content-router (embedding cache + classifier + rerank + score), PII (config, redactor, stream filter, event store, OpenAI/Anthropic adapters), billing (StatsBackend interface with gorm/inmem implementations, Recorder with prom counters, LocalUser fallback for no-auth boxes). * middleware: UsageMiddleware records via billing.Recorder with handler-stamp (StampUsage) preferred over body-parse; admission + route-model + usage-stamp + trace middlewares; context-key constants for the routing modules to communicate token counts, models, and correlation IDs to UsageMiddleware. * observability: BackendTrace ring buffer now stores the full request body (capped at 1 MB); MITM proxy emits structured trace events; router classifier failures surface to /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0) with katanemo icon. * docker: .dockerignore excludes in-place llama.cpp clone so the cloud-proxy backend image builds against the pinned commit. * UI: cloud-proxy model templates (Anthropic defaults to translate), ConfigFieldRenderer proxy.mode/proxy.provider dropdowns with descriptions, Traces page renders Request Body pane, useChat.js drops the unconditional max_tokens=contextSize cap. Rebased onto master 0b2ae3c ("fix(openai): stream usage non-zero when tools are enabled mudler#9941"); the master refactor that extracted process/processTools into chat_stream_workers.go is preserved, and the new piiRedactor/piiEvents are threaded into ChatEndpoint without reintroducing the inline workers. UsageMiddleware preserves master's Source / APIKeyID / APIKeyName recording on top of the new billing.Recorder API. statsRecorder gets a SIGTERM Close() handler so the gorm backend's pending batch is drained on graceful exit. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
{prompt_tokens:0, completion_tokens:0, total_tokens:0}in theinclude_usagetrailer whenever the request carried atoolsarray.processTools(the streaming worker for tool-bearing requests) discarded the cumulativeTokenUsagereturned fromComputeChoicesand never stamped it on any chunk it sent on theresponseschannel. The outer loop's running tracker therefore stayed at the zero value. The no-toolsprocesspath stampsUsageon every chunk and was unaffected.TokenUsagevia a usage-only sentinel chunk (emptyChoices, populatedUsage) emitted right beforeclose(responses). Move the outer loop's per-chunkUsagecapture above the empty-Choicesskip so the sentinel updates the tracker without reaching the wire. The OpenAI streaming spec contract is preserved: intermediate chunks still carry nousagefield, andbuildNoActionFinalChunks/buildDeferredToolCallChunksremain Usage-free (the regression contract from issue OpenAI API implementation is broken - shows up only for agentic coding #8546).Changes
core/http/endpoints/openai/chat_emit.go: 3 new helpers —streamUsageFromTokenUsage,usageSentinelChunk,applyChunkToUsage.core/http/endpoints/openai/chat.go:processToolsnow capturesfinalUsagefromComputeChoicesand emits the sentinel; outer loop usesapplyChunkToUsagebefore the empty-Choicesskip;processcallback updated to use the sharedstreamUsageFromTokenUsagehelper.core/http/endpoints/openai/chat_stream_usage_test.go: 7 new Ginkgo specs covering the helpers and the outer-loop flow contract.Test plan
go test ./core/http/endpoints/openai/(105/105 specs pass, was 103)go vet ./core/http/endpoints/... ./core/backend/... ./core/schema/...cleanchat_stream_usage_test.gospec-compliance tests (intermediate chunks must not carryusage,buildNoActionFinalChunks/buildDeferredToolCallChunksreturn Usage-free chunks, etc.) remain greeninclude_usage:true) and confirm the trailer reports non-zero token countsFixes #9927