feat(soniox): add TTS speed support - #2005
Conversation
🦋 Changeset detectedLatest commit: 6bc70a5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 37 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Reject non-finite speeds and release completed or failed streams so pooled connections and streaming inputs cannot leak. Co-authored-by: Cursor <cursoragent@cursor.com>
Release local waiters when callers cancel so nonresponsive servers cannot retain pooled stream state. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Keep provider-attempt cleanup from terminating consumer output before the base retry lifecycle completes. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep failed attempts owned until retry completion and drain cancellation before closing the pooled socket. Co-authored-by: Cursor <cursoragent@cursor.com>
Distinguish recoverable attempts from terminal failures so retries remain cancellable without retaining completed streams. Co-authored-by: Cursor <cursoragent@cursor.com>
Centralize provider cleanup after the base retry lifecycle so plugins do not duplicate private attempt policy. Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve both the stream completion and chunked retry regression suites while resolving the add/add test conflict. Co-authored-by: Cursor <cursoragent@cursor.com>
| if (!this.input.closed) { | ||
| this.input.close(); | ||
| } |
There was a problem hiding this comment.
🟡 Agent speech can be truncated when a transient error occurs mid-sentence
During a streaming synthesis attempt, the shared text input is permanently shut off (this.input.close() at plugins/soniox/src/tts.ts:380-382) as soon as an attempt ends, so if the caller is still feeding text when a retryable error happens, the rest of the text is silently dropped and never spoken.
Impact: On a temporary provider hiccup, the agent can speak only the beginning of a reply and silently cut off the remainder, with no error surfaced.
Why closing the input queue on every attempt drops in-flight and future text
The base class feeds text into this.input via pushText (agents/src/tts/tts.ts:529-543), which silently returns once this.input.closed is true. In SynthesizeStream.run() the finally block runs at the end of every attempt — including retryable-failure attempts — and calls this.input.close() to unblock the awaited #sendInput task (plugins/soniox/src/tts.ts:378-387).
Because the base retry loop (agents/src/tts/tts.ts:259-307) then invokes run() again for the retry, and #sendInput on retry only replays #inputCache (plugins/soniox/src/tts.ts:394-406), any text that had not yet been consumed into the cache before the failure — plus any text produced afterward by the still-active updateInputStream/pumpInput producer — is lost. pumpInput keeps calling pushText, but every such call is a no-op once the queue is closed. The result is a truncated cache being replayed on retry and permanent loss of the remainder of the utterance.
This differs from the base contract and from other plugins (e.g. plugins/cartesia/src/tts.ts never closes this.input inside run), where the input queue remains open across the whole logical stream.
Prompt for agents
In plugins/soniox/src/tts.ts, SynthesizeStream.run()'s finally block calls this.input.close() at the end of every attempt in order to unblock the concurrent #sendInput task that run() awaits (await inputTask). However, this.input is the shared, whole-stream input queue fed by the base class's pushText/pumpInput. Closing it on a non-final (retryable-failure) attempt permanently drops (a) any queued input not yet drained into #inputCache and (b) all future text pushed by an still-active updateInputStream producer, because base pushText silently no-ops once this.input.closed is true (agents/src/tts/tts.ts:537-540). On retry, #sendInput replays only the (now-truncated) #inputCache, so the agent can speak a truncated utterance. The fix should terminate the per-attempt #sendInput loop WITHOUT closing the shared input queue on attempts that may be retried — e.g. break the #sendInput for-await via attemptState.cancelled using a wakeup mechanism that does not close this.input, or only close this.input on the final attempt / on cancellation/abort. Ensure the cache continues to accumulate any input that arrives while an attempt is in-flight so retries can replay the complete text.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
speedoption,[0.7, 1.3]validation range, runtime updates, and WebSocket request field.SynthesizeStreamcompletion hook so provider ownership spans the entire logical retry lifecycle.Source diff coverage
The source change from livekit/agents#6339 adds bounded Soniox TTS speed configuration. The JS port preserves its default, validation, update behavior, and initial WebSocket configuration field. JS-specific pooled WebSocket ownership and base-class retry behavior are covered by focused tests.
The Python implementation can pass through encoded formats. The JS TTS interface emits decoded
AudioFrames, so this port explicitly acceptspcm_s16leand rejects unsupported formats instead of silently decoding them incorrectly.Review fixes
TTS.close()cancels and settles them before pooled socket shutdown.Verification
Cue voice E2E
Verified at exact PR head
0d19988a469404d6ed2327bcecd70f0164871ab8in voice mode with a disclosed local protocol-compatible Soniox WebSocket boundary becauseSONIOX_API_KEYwas unavailable.sid_ec11dfb4a2b4.Please verify the Soniox speaking speed.AS_SPEAKING, the assistant conversation item, and return toAS_LISTENING/AS_IDLE; worker output independently reported playout completed without interruption.speed: 1.3, 24 kHzpcm_s16le, 48,000 PCM bytes, terminal response, and socket cleanup; SHA-2568b3e603753d90668c5890686a4fc550367eecfe243d4d709ae87d3a7afcb087f.04a15610aa43d6ee8ee7537c254e1d6ff60add2452b62bb464022dba2fe2b398.a477293786c1eba1ca0a90d8fb4740665f5faceb93971f71192e3882cbf100e0.events.jsonlSHA-256:aaaaf5aa43021a91888b04bdc0b36e3a7775c7abcf8223b8d09f813df091b871.result.jsonSHA-256:12ef54ef40dee1946473821ee5eecc4481916bc42ac101b23afa7f2d62997ad1.Ported from livekit/agents#6339