fix(inference): tolerate null token counts in streamed usage - #6596
Open
VihaanAgarwal wants to merge 1 commit into
Open
fix(inference): tolerate null token counts in streamed usage#6596VihaanAgarwal wants to merge 1 commit into
VihaanAgarwal wants to merge 1 commit into
Conversation
An OpenAI-compatible provider can send a final chunk where usage is present but individual counts are null. CompletionUsage requires ints, so the pydantic error was wrapped as a non-retryable APIConnectionError and tore down the whole AgentSession mid-call. Coerce the three counts to 0, matching what prompt_cached_tokens already does on the adjacent line.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6595
What breaks
On the OpenAI-compatible path (
livekit.agents.inference.llm.LLMStream), a provider can send a final streaming chunk whereusageis present but individual counts arenull. We only guard onif chunk.usage is not None:, then pass the fields straight intoCompletionUsage, wherecompletion_tokens,prompt_tokensandtotal_tokensare all requiredint.The pydantic
ValidationErrorgets caught by the genericexcept Exceptionat the bottom of_runand re-raised asAPIConnectionError(retryable=False). Because it's non-retryable it surfaces as an unrecoverablellm_errorand terminates theAgentSession, so a malformed usage payload drops a live call.The reporter hits this behind a proxy in front of a custom model that occasionally omits
completion_tokens.Fix
Coerce the three counts with
or 0. That's whatprompt_cached_tokenson the adjacent line already does, so this keeps the block internally consistent rather than introducing a new pattern. Complete payloads are unaffected.Same shape as #5404, which fixed the other half of this (usage present but not reached).
Tests
tests/test_inference_llm_usage.pydrivesLLMStreamwith a fake OpenAI stream, so it needs no network or credentials:completion_tokens=Noneyields usage with0and does not raiseprompt_tokens/total_tokensnull is coerced the same wayThe first two fail on
mainwithAPIConnectionErrorand pass with the change.Full unit gate: 1175 passed, 4 skipped.
make type-check,make lintandmake format-checkare clean. The 9 errors in the pytest summary are a pre-existing OTLP metrics-exporter teardown issue that reproduces identically on a clean checkout.