Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .changeset/adaptive-interruption-barge-in.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
'@livekit/agents': patch
---

Make adaptive interruption actually interrupt

Three defects, each of which on its own could leave a barge-in unheard or un-acted-on.

**The overlap gate could be disarmed mid-interruption.** `overlapSpeechStarted` is what lets a
user's overlapping audio reach the interruption model, and only a VAD start-of-speech raises it.
Because VAD never re-announces speech already under way, anything that cleared the flag mid-overlap
disarmed the detector for the rest of the agent turn: every remaining frame was dropped, no
inference request was made, and the agent talked straight through the user. Two events did that. A
second speech segment in the same turn (a queued `SpeechHandle`, or the reply after a tool call)
raises `agent-speech-started` again with no `agent-speech-ended` in between, and was treated as a
new turn; an open overlap is now preserved across it, while a genuine new turn still resets the
overlap, audio buffer, cache and counters. And a transport failover rebuilt
`InterruptionStreamBase` from scratch and replayed only `agent-speech-started`, which marks the
agent as speaking but leaves the overlap disarmed; the in-progress overlap is now handed to the
replacement stream through a distinct `agent-speech-resumed` sentinel. A rejected forwarding task
also no longer surfaces as an unhandled rejection during the failover backoff.

**Ending the pause let one frame of the interrupted speech escape.** `cancelSpeechPause` opens the
audio output's pause gate so the next speech can be admitted, but frames of the speech it has just
interrupted are still parked at that gate and were released before the interrupted reply task
reached its own `clearBuffer()` — 20ms of audio the user had already barged in over. The
interruption is now signalled to the output before the gate opens, so those frames bail instead.

**A finished interruption silently discarded the next reply.**
`ParticipantAudioOutput.clearBuffer()` resolves an `interruptedFuture` that frames parked at the
pause gate consult to decide whether to bail. Nothing reset that signal until the _next_ segment's
`flush()`, which only runs after that segment's frames have all been captured — so for the whole of
the following reply the signal still described an interruption that was already over. If the output
was paused mid-reply during that window (an ordinary false-interruption pause, on by default),
every remaining frame bailed at the gate and never reached the wire, while the session still
reported the reply as fully spoken and committed it to the chat context. The gate is now scoped to
the segment being captured: a frame bails only for an interruption raised at or after its own
segment began, and parked frames are woken by a per-frame signal so a concurrent `flush()` can no
longer strand one there.

`OverlappingSpeechEvent` is also now exported by name from `voice/events.js`, so
`overlapping_speech` handlers can be typed without reaching into `inference/interruption/types.js`.
13 changes: 13 additions & 0 deletions .changeset/fix-krisp-frame-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
'@livekit/agents-plugin-krisp': patch
---

Fix Krisp-processed audio being invisible to the rest of the pipeline

The LiveKit Cloud backend is reached through `createRequire`, which resolves the internal
package's `require` condition and so loads the CJS build of `@livekit/rtc-node` next to the
ESM one the framework uses. Frames returned by that backend were instances of the CJS copy's
`AudioFrame`, so every `instanceof AudioFrame` downstream failed. Adaptive interruption saw
zero audio and classified every barge-in as a backchannel, making it impossible to interrupt
an agent that had noise cancellation enabled. Frames are now adopted into the local binding
before leaving the filter, sharing their samples rather than copying them.
13 changes: 13 additions & 0 deletions .changeset/interrupted-reply-backlog-on-the-wire.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
'@livekit/agents': patch
---

Stop an interrupted reply's TTS backlog from reaching the wire

`captureFrame` only compared a segment's interrupt snapshot while the pause gate was closed, so
the check was skipped entirely once the gate reopened. `cancelSpeechPause` un-gates the sink to
admit the next reply as soon as the handle is interrupted, but the interrupted reply's
`forwardAudio` loop keeps running until its abort signal fires an event loop turn later — and real
TTS providers hand it several seconds of audio ahead of realtime to drain in the meantime. Those
frames took the open-gate path straight to the wire, so the user kept hearing the barged-over
speech resume while the next reply's transcript was already streaming.
Loading
Loading