fix(voice): commit in-flight assistant turn when the session closes mid-playout - #2042
Conversation
…id-playout A room disconnect while the agent is speaking parked the pipeline reply task on a playout promise that can never resolve; AgentActivity.close() then hard-cancelled the task, so the partially-spoken turn was dropped entirely — no ConversationItemAdded, nothing in chatCtx (livekit#2041). Two changes: - close() resolves the current speech's interrupt future before cancelling speech tasks, letting the reply task exit through its existing interruption branch, which already commits the partially-forwarded text with interrupted: true. - The interruption branch's post-clearBuffer waitForPlayout() is raced against the reply abort signal so it cannot park on a dead room; when it aborts, the segment is still marked partially played (first frame was heard) and only the final playback position is lost. Regression test simulates the disconnect with an AudioOutput that starts playback but never reports it finished: pre-fix the assistant item never reaches chat ctx; post-fix it lands as interrupted with the spoken prefix. Full voice suite passes (44 files / 416 tests).
🦋 Changeset detectedLatest commit: 2df522a The changes in this PR will be included in the next version bump. This PR includes changesets to release 37 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ven after abort Codex review: an AudioOutput may report playbackFinished synchronously from clearBuffer(), so the already-aborted path must not discard a settled result — its synchronized transcript is the text actually heard. Rebuilt the helper on the existing utils (isPending, waitForAbort, ThrowsPromise) instead of a hand-rolled listener race.
toubatbrian
left a comment
toubatbrian
left a comment
There was a problem hiding this comment.
Exact-head CI fails throws:check because raceWithAbort() uses isPending() and ThrowsPromise.race() without declaring their Error effect.
Please import type Throws from @livekit/throws-transformer/throws and change the helper return type to Promise<Throws<T | undefined, Error>>, preserving the current settled-promise precedence algorithm. I verified this with focused pre-aborted fulfilled/rejected/pending and abort-first tests; 48 related tests, throws:check, compilation, and formatting pass.
throws:check flagged the helper: it consumes isPending() and ThrowsPromise.race() without surfacing their Error effect in its own signature. Return Promise<Throws<T | undefined, Error>> and route the already-settled early path through ThrowsPromise.fromPromise so the branding holds without a cast. Settled-promise precedence is unchanged.
|
@toubatbrian Done in 2df522a — One note on the already-settled early path: instead of a cast I routed it through Verified locally: |
3 participants
Fixes #2041 — as offered there; happy to rework if you'd prefer a different approach.
Problem
A room disconnect while the agent is speaking parks the pipeline reply task on a playout promise that can never resolve.
AgentActivity.close()then hard-cancels the task, so the partially-spoken turn is dropped entirely — noConversationItemAdded, nothing inchatCtx. For public-facing agents where visitors bounce mid-answer constantly, the last (often most interesting) turn of every such session vanishes from history.Fix (two small changes)
close()resolves the current speech's interrupt future before cancelling speech tasks, so the reply task exits through its existing interruption branch — which already commits the partially-forwarded text withinterrupted: true. No new commit logic.clearBuffer()waitForPlayout()is raced against the reply abort signal (raceWithAbort) so it can't park on a dead room. On abort, the segment is still marked partially played (the first frame was heard); only the final playback position is lost.Worst case degrades to the previous behavior (cancel timeout); it can't regress.
Testing
agent_activity_close_commit.test.ts) simulates the disconnect with anAudioOutputthat starts playback but never reports it finished: red before the fix (zero assistant items), green after (item lands asinterruptedwith the spoken prefix).Found running LiveKit Agents 1.4.8 in production at asksarfaraz.ai (context in #2041, sibling report #2040).