fix(sarvam): emit speech timing for STT metrics - #5995
Conversation
Track Sarvam STT speech boundaries in the plugin so LiveKit can compute EOU metrics when using internal VAD with STT turn detection. Co-authored-by: Cursor <cursoragent@cursor.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Keep Sarvam STT final transcripts visible immediately while still using delayed EOS ordering only when speech-end arrives before the transcript. Co-authored-by: Cursor <cursoragent@cursor.com>
| async def _emit_pending_eos_after_timeout(self, timeout: float = 0.1) -> None: | ||
| try: | ||
| if timeout > 0: | ||
| await asyncio.sleep(timeout) | ||
| if self._pending_eos and not self._eos_emitted_for_utterance: | ||
| self._emit_end_of_speech() | ||
| except asyncio.CancelledError: | ||
| raise |
There was a problem hiding this comment.
📝 Info: Orphaned FINAL_TRANSCRIPT can be emitted after END_OF_SPEECH when fallback fires before transcript
When the EOS fallback timer fires before the transcript arrives, _emit_end_of_speech sets _pending_eos=False and _eos_emitted_for_utterance=True. A subsequent late transcript then goes through the else branch at line 1662 (_send_final_transcript called directly), emitting a FINAL_TRANSCRIPT after END_OF_SPEECH without a preceding START_OF_SPEECH. This is tested explicitly in test_late_transcript_after_eos_fallback_is_emitted_after_eos and appears intentional (better to emit a late transcript than lose it), but downstream consumers like audio_recognition.py should handle FINAL events that arrive after EOS gracefully. The existing audio_recognition.py code does handle this since it processes FINAL_TRANSCRIPT independently of speech state.
Was this helpful? React with 👍 or 👎 to provide feedback.
Cancel delayed EOS fallback work during stream shutdown and document the fallback timeout tradeoff for late transcripts. Co-authored-by: Cursor <cursoragent@cursor.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
tinalenguyen
left a comment
tinalenguyen
left a comment
There was a problem hiding this comment.
thank you for the PR! i left a small comment but otherwise i tested it and it lgtm
| return None | ||
| return float(value) | ||
|
|
||
| def _ensure_utterance_timing_state(self) -> None: |
There was a problem hiding this comment.
all these fields are already set in init, ithink we can remove this
There was a problem hiding this comment.
Ah I see, had left it there as a guard for tests that use SpeechStream.new() without calling init. Now removed as suggested, thank you
2 participants
Summary
STTstream audio position and speech boundaries soSTART_OF_SPEECH,FINAL_TRANSCRIPT, andEND_OF_SPEECHcarry timing data that LiveKit can use for EOU metrics.FINAL_TRANSCRIPTbeforeEND_OF_SPEECHordering for both transcript-before-EOS and EOS-before-transcript provider ordering.audio_recognition.pyor other STT providers.Test plan
uv run pytest livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.py livekit-plugins/livekit-plugins-sarvam/tests/test_language_probability.py -vuv run ruff check livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.pypython -m py_compile livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.pysarvam.STT(model="saaras:v3"),vad=None, andturn_handling={"turn_detection": "stt"}produced non-zero EOU metrics, includingtranscription_delay=0.146sandend_of_utterance_delay=0.501s.Notes
livekit-plugins-sarvam'ssarvam.STTpath instt.py; it does not changeSTTStreamingor framework turn detection behavior.