fix(sarvam): emit speech timing for STT metrics - #1744
fix(sarvam): emit speech timing for STT metrics#1744rosetta-livekit-bot[bot] wants to merge 8 commits into
Conversation
…1525) Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: u9g <jason.lernerman@livekit.io>
Agent.llmNode now returns ReadableStream<ChatChunk | string | FlushSentinel>, but the agent_v2 hook overrides and AgentHookAdapter still declared the narrower ChatChunk | string union, so passing super.llmNode as the fallback failed to type-check. Widen the override return types and the adapter's fallback/return signatures to include FlushSentinel. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Brian Yin <brian.yin@livekit.io> Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: u9g <jason.lernerman@livekit.io>
Catch end-call close listener errors to avoid unhandled rejections during shutdown, and make public tool type guards return false for null inputs.
🦋 Changeset detectedLatest commit: c201f8b The changes in this PR will be included in the next version bump. This PR includes changesets to release 34 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔴 Utterance state not reset on WebSocket reconnect causes missed START_OF_SPEECH and stale timing
When the WebSocket reconnects (e.g., after server idle timeout or error), #runWS only resets #speaking to false but does not reset the new utterance-tracking state variables: #pendingEos, #pendingFinalData, #utteranceSpeechEndAudioPos, #utteranceSpeechEndWallTime, #finalReceivedForUtterance, #eosEmittedForUtterance. If the previous session ended with #eosEmittedForUtterance = true (a completed utterance), or #pendingEos = true (an in-flight utterance), the new session inherits this stale state.
For example, after a completed utterance: #eosEmittedForUtterance = true persists. When a new transcript arrives in the new session, the guard at line 960 (!this.#speaking && !this.#pendingEos && !this.#eosEmittedForUtterance) evaluates to false because #eosEmittedForUtterance is still true. This skips emitting START_OF_SPEECH for the new utterance. The transcript then falls through to #sendFinalTranscript(td) at line 982 without a preceding START_OF_SPEECH, breaking the expected event sequence for the downstream voice pipeline (audio_recognition.ts:969-997).
Was this helpful? React with 👍 or 👎 to provide feedback.
| #audioPosition = 0; | ||
| #utteranceStartAudioPos = 0; | ||
| #utteranceSpeechEndAudioPos?: number; |
There was a problem hiding this comment.
🚩 #audioPosition accumulates across WebSocket sessions creating mixed time domains
#audioPosition (line 561) is initialized to 0 in the constructor and monotonically increases across WebSocket reconnections. The server, however, resets its audio timeline on each new session. This means that after a reconnect, the fallback startTime (from #utteranceStartAudioPos = #audioPosition) and endTime (from #utteranceSpeechEndAudioPos = #audioPosition) are in a cumulative client-side time domain, while server-provided speech_start/speech_end values are session-relative. When both sources are mixed (e.g., server provides speech_start but not speech_end), the resulting startTime and endTime can be in different time domains. The downstream consumer at agents/src/voice/audio_recognition.ts:828-834 converts endTime to wall-clock time via endTime * 1000 + inputStartedAt, which would produce an incorrect epoch time if endTime includes accumulated position from previous sessions. This is a design consideration rather than a clear bug, since the server usually provides both or neither timing fields.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const newToolNames = new Set(Object.keys(newToolCtx.functionTools)); | ||
| const toolsAdded = [...newToolNames].filter((name) => !oldToolNames.has(name)); | ||
| const toolsRemoved = [...oldToolNames].filter((name) => !newToolNames.has(name)); |
There was a problem hiding this comment.
🚩 Provider tool additions/removals are not tracked in AgentConfigUpdate during updateTools
In _startSession (agents/src/voice/agent_activity.ts:498-500), the initial AgentConfigUpdate includes both function tool names and provider tool IDs. However, updateTools at agents/src/voice/agent_activity.ts:786-799 only diffs function tool names (Object.keys(newToolCtx.functionTools)), so adding or removing a provider tool at runtime produces no AgentConfigUpdate entry in the chat history. The actual tool update still reaches the realtime session (line 814) and the chat context copy (line 818-819), so the pipeline works correctly — but the history won't reflect the change, which could affect summarization or debugging.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Port of livekit/agents#5995.\n\nSummary:\n- track Sarvam streaming audio position per utterance\n- emit final transcript and end-of-speech timing so STT-based EOU metrics can use provider timing\n- emit Sarvam recognition usage when websocket metrics include audio duration\n\nValidation:\n- pnpm --filter @livekit/agents-plugins-test build\n- pnpm --filter @livekit/agents-plugin-silero build\n- pnpm --filter @livekit/agents-plugin-openai build\n- pnpm --filter @livekit/agents-plugin-sarvam build\n- pnpm --filter @livekit/agents-plugin-sarvam lint
Ported from livekit/agents#5995
Original PR description
Summary
STTstream audio position and speech boundaries soSTART_OF_SPEECH,FINAL_TRANSCRIPT, andEND_OF_SPEECHcarry timing data that LiveKit can use for EOU metrics.FINAL_TRANSCRIPTbeforeEND_OF_SPEECHordering for both transcript-before-EOS and EOS-before-transcript provider ordering.audio_recognition.pyor other STT providers.Test plan
uv run pytest livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.py livekit-plugins/livekit-plugins-sarvam/tests/test_language_probability.py -vuv run ruff check livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.pypython -m py_compile livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.pysarvam.STT(model="saaras:v3"),vad=None, andturn_handling={"turn_detection": "stt"}produced non-zero EOU metrics, includingtranscription_delay=0.146sandend_of_utterance_delay=0.501s.Notes
livekit-plugins-sarvam'ssarvam.STTpath instt.py; it does not changeSTTStreamingor framework turn detection behavior.