fix(voice): eliminate the barge-in silent-hang — playback-counter imbalance (#1662) + drain deadlock (#836) - #1732
Closed
enriqueespaillat-gyde wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 19de80c The changes in this PR will be included in the next version bump. This PR includes changesets to release 34 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…rruption (livekit#836) When an AgentTask completes via a tool-only turn (e.g. a function tool that calls task.complete()), AgentTask.run()'s finally resumes the parent via _updateActivity(..., 'resume') -> drain() -> _pauseSchedulingTask(), which awaits _mainTask.result. The mainTask cannot reach its drain loop-exit until its pending speech tasks de-register from speechTasks, so awaiting its result deadlocks in two cases: 1. Reentrant: the drain runs from inside one of this activity's own in-flight speech tasks (the completing turn). It is a self-await — the speech task cannot finish until the drain returns, and the drain cannot return until it finishes. 2. Barge-in cascade: the drain runs from a non-reentrant context while the mainTask is held only by 'zombie' speech tasks — already done, or whose handle was interrupted by the cascade — that have not de-registered yet. The mainTask still cannot reach its loop-exit. Either way the parent never resumes and the agent goes silent until the participant disconnects. Fix: in _pauseSchedulingTask, skip the _mainTask.result self-await when invoked reentrantly from one of this activity's own speech tasks, OR when every pending drain speech task is already done or interrupted. The mainTask is still reaped by the subsequent close() -> cancelAndWait. External drains (handoff, session close, pause) run outside the activity's speech tasks and with live pending work, so they wait as before. Adds regression tests driving the real mainTask + _pauseSchedulingTask prototypes: one asserts the reentrant drain resolves rather than hanging, one asserts a non-reentrant drain held only by an interrupted speech task resolves. Both time out without the fix.
enriqueespaillat-gyde
force-pushed
the
enrique/fix-drain-reentrancy-836
branch
from
June 9, 2026 02:15
74e3ca7 to
69c7608
Compare
enriqueespaillat-gyde
changed the title
enriqueespaillat-gyde
changed the title
enriqueespaillat-gyde
force-pushed
the
enrique/fix-drain-reentrancy-836
branch
from
June 9, 2026 05:12
80480a9 to
bcfce77
Compare
…ate livekit#1662 + synced-output drift reconcile) Two related playback-segment counter imbalances leave the agent silent after a barge-in: 1. ParticipantAudioOutput.captureFrame bumped playbackSegmentsCount via super.captureFrame BEFORE its pause/interrupt gate. A frame captured while paused and then interrupted bailed at the gate after that bump, stranding the segment count ahead of playbackFinishedCount so the next waitForPlayout() blocked forever. Count the segment only after the gate. (livekit#1662) 2. SyncedAudioOutput (TranscriptionSynchronizer wrapper) counts a segment in its own captureFrame, then forwards to the downstream sink — which can drop the frame at its interrupt gate without counting or ever finishing it. The wrapper's finish count is driven by the downstream's playback-finished events, so its segment count drifts permanently ahead and SyncedAudioOutput.waitForPlayout() (awaited by the reply pipeline before it marks generation done) strands forever, freezing the turn pump. waitForPlayout() now reconciles the drift before waiting; the downstream's legitimate in-flight segments are still awaited. A public pendingPlayoutSegments accessor on AudioOutput exposes the outstanding count for the cross-output comparison. Adds a deterministic regression test: a downstream sink that drops a captured frame makes SyncedAudioOutput.waitForPlayout() resolve (reconciled) rather than hang (verified: times out without the reconcile).
enriqueespaillat-gyde
force-pushed
the
enrique/fix-drain-reentrancy-836
branch
from
June 9, 2026 05:14
bcfce77 to
19de80c
Compare
enriqueespaillat-gyde
marked this pull request as ready for review
June 9, 2026 05:27
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the voice-agent silent-hang: after a member barges in, the agent stops responding and goes silent until the participant disconnects. Three related root causes, all in the interrupt/playout machinery. The first is the primary in-call hang; the others are real adjacent deadlocks surfaced during the investigation.
1. Playback-segment counter imbalance — the silent-hang (primary)
The reply pipeline awaits
audioOutput.waitForPlayout()before marking a generation done.waitForPlayout()blocks whileplaybackFinishedCount < playbackSegmentsCount. Two ways the counter strands ahead:ParticipantAudioOutput.captureFramebumpedplaybackSegmentsCount(super.captureFrame) before its pause/interrupt gate. A frame captured-while-paused then interrupted bailed at the gate after the bump → segment counted, never finished →waitForPlayout()hangs forever. Now the segment is counted after the gate. (This is fix(voice): don't strand playback-segment counter on interrupted-while-paused frame #1662.)SyncedAudioOutput(theTranscriptionSynchronizerwrapper the pipeline actually awaits) counts a segment in its owncaptureFrame, then forwards to the downstream sink — which can drop the frame at its gate without counting it. The wrapper's finish count is driven by the downstream's playback-finished events, so the wrapper drifts permanently ahead and itswaitForPlayout()strands.SyncedAudioOutput.waitForPlayout()now reconciles the drift (emits the missing finishes for dropped segments) before waiting; the downstream's legitimate in-flight segments are still awaited. A publicpendingPlayoutSegmentsaccessor onAudioOutputexposes the outstanding count for the cross-output comparison.A tool-only turn (no audio of its own — e.g. an identity-confirmation function call) is where this most visibly bites: it inherits the global stranded count and hangs, freezing the turn pump for every subsequent turn.
2. Reentrant drain deadlock (#836)
When an
AgentTaskcompletes via a tool-only turn that races a barge-in,AgentTask.run()'sfinallyresumes the parent via_updateActivity(…, 'resume')→drain()→_pauseSchedulingTask()→await _mainTask.result. That await deadlocks when the drain runs reentrantly from one of the activity's own in-flight speech tasks, or when the mainTask is held only by already-done/interrupted "zombie" speech tasks._pauseSchedulingTasknow skips the self-await in those cases; the mainTask is still reaped by the subsequentclose()→cancelAndWait. Fixes #836.Tests
synchronizer.test.ts— a downstream sink that drops a captured frame makesSyncedAudioOutput.waitForPlayout()resolve (reconciled) instead of hang. Verified it times out without the reconcile.agent_activity.test.ts(reentrant drain, and a non-reentrant drain held only by an interrupted zombie task) — both time out without the guard, pass with it.Changesets included for each. Built, typechecked, linted, and the affected suites pass.
Notes
The playback-counter fix has been validated against a real reproduction in our deployment (per-instance counter probes showed the synced wrapper counting a segment the participant dropped; the reconcile rebalanced it and the hang disappeared). Happy to split the drain-deadlock fix into its own PR if maintainers prefer — they're independent, but both are needed to fully eliminate the barge-in hang.