Adaptive interruption: a second agent speech segment (e.g. after a tool call) disarms an open overlap and silently swallows the interruption

[toubatbrian]

Summary

A second agent speech segment inside a single agent turn disarms an adaptive-interruption overlap that the user is still speaking through. From that moment the user's audio stops being sent to the bargein gateway, no verdict is ever emitted, and the interruption is silently dropped — the agent talks straight through the user.

The most common trigger is a tool call: a reply that produces tool calls deliberately skips _on_end_of_agent_speech, and the follow-up tool-response reply then raises _AgentSpeechStartedSentinel a second time with no end sentinel in between.

Mechanism

_reset_state() clears _overlap_started unconditionally for both speech sentinels:

https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/inference/interruption.py#L588-L592

case _AgentSpeechStartedSentinel() | _AgentSpeechEndedSentinel():
    await _reset_state()
    self._agent_speech_started = isinstance(input_frame, _AgentSpeechStartedSentinel)
    continue

_reset_state() (interruption.py:573-582) sets _overlap_started = False, resets the audio buffer, clears the cache and zeroes _num_requests.

The consequences, in order:

1. User audio stops reaching the gateway. The batch send is gated on _overlap_started (interruption.py:649):

   case rtc.AudioFrame() if self._agent_speech_started:
       samples_written = self._audio_buffer.push_frame(input_frame)
       self._accumulated_samples += samples_written
       if self._accumulated_samples >= self._batch_size and self._overlap_started:
           output_ch.send_nowait(self._audio_buffer.read())

   _agent_speech_started is back to True, so frames are still buffered — they are just never classified.

2. The overlap can never be re-armed. _overlap_started is only set by _OverlapSpeechStartedSentinel (interruption.py:594-597), which is only sent from AudioRecognition._on_start_of_speech (audio_recognition.py:527-533) — i.e. a fresh VAD/STT speech onset. VAD does not re-announce speech that is already under way, so nothing re-opens the overlap for a user who was mid-sentence.

3. No verdict is emitted at all. At _OverlapSpeechEndedSentinel the whole body is behind if self._overlap_started (interruption.py:621-640), so OverlappingSpeechEvent is never sent and AgentActivity.on_interruption never runs.

4. There is no VAD fallback during the gap — see the trigger below: _restore_interruption_by_audio_activity() sits in the same branch that gets skipped, so VAD interruption stays disabled by the earlier _disable_vad_interruption_soon() (agent_activity.py:3185-3186).

Trigger condition: a multi-segment agent turn

_on_start_of_agent_speech raises the sentinel unconditionally — there is no "already speaking" guard (audio_recognition.py:464-465):

if self._adaptive_interruption_active:
    self._interruption_ch.send_nowait(_AgentSpeechStartedSentinel())

And a reply that produced tool calls skips the end sentinel entirely (agent_activity.py:3333-3342):

if not speech_handle.interrupted and len(tool_output.output) > 0:
    self._session._update_agent_state("thinking")
elif self._session.agent_state == "speaking":
    self._session._update_agent_state("listening")
    if self._audio_recognition:
        self._audio_recognition._on_end_of_agent_speech(...)
    if self.interruption_enabled:
        self._restore_interruption_by_audio_activity()

Because _on_end_of_agent_speech is skipped, AudioRecognition._agent_speaking also stays True (it is only cleared inside that method, audio_recognition.py:474/507), so the overlap legitimately stays open across the tool call.

Then, after the tools finish, a new _pipeline_reply_task is created for the same SpeechHandle (agent_activity.py:3454-3478), and its first-audio-frame callback calls _on_start_of_agent_speech again (agent_activity.py:3183-3184) — the second start sentinel, with no end sentinel in between.

Concrete sequence:

1. Reply segment 1 starts speaking → _AgentSpeechStartedSentinel, _agent_speech_started = True.
2. User starts talking over it → VAD onset → _OverlapSpeechStartedSentinel → _overlap_started = True; audio starts flowing to the gateway.
3. Segment 1 ends having produced tool calls → state "thinking", no _AgentSpeechEndedSentinel. The overlap is still open and the user is still talking.
4. Tools execute; the tool-response reply's first audio frame fires _on_start_of_agent_speech → second _AgentSpeechStartedSentinel → _reset_state() → _overlap_started = False. ← bug
5. The user is still mid-sentence. Nothing more reaches the classifier, and nothing can re-arm the overlap.
6. User stops → _OverlapSpeechEndedSentinel → _overlap_started is False → no OverlappingSpeechEvent, no on_interruption. The interrupt is silently gone.

The window is exactly the case adaptive interruption exists to handle: the user has begun interrupting but the classifier has not yet returned a verdict. (If a bargein had already been detected, speech_handle.interrupted would be True and line 3333 would take the elif, sending the end sentinel normally — so the bug specifically eats interruptions that are still being evaluated.)

The same shape exists in _tts_task_impl (agent_activity.py:2722 start / 2839-2844 end, guarded by agent_state == "speaking"), so a queued say() in one turn can do it too.

Suggested fix

Skip the reset when the stream already believes the agent is speaking and an overlap is open — i.e. treat the sentinel as a continuation of the same turn rather than a new one. Something like:

case _AgentSpeechStartedSentinel() if self._agent_speech_started and self._overlap_started:
    # a later segment of the same agent turn (tool reply, queued say()):
    # resetting here would strand an overlap the user is still in the middle of
    continue
case _AgentSpeechStartedSentinel() | _AgentSpeechEndedSentinel():
    await _reset_state()
    ...

The guard has to be narrow: a genuine new turn must still wipe the overlap, the cache and the counters.

Reference implementation (JS) — currently a deliberate divergence

The Node port has the same bug and it is fixed in livekit/agents-js#2117, which adds exactly this guard in agents/src/inference/interruption/interruption_stream.ts plus regression tests that drive the pipeline through a second speech segment mid-overlap and assert user audio keeps reaching the (mocked) gateway and that a bargein verdict still surfaces.

That PR is knowingly a divergence from Python and should probably be ported back here. (The other half of that JS PR — overlap state lost across a transport retry — does not apply to Python: the flags live on self and _run() reconnects in place on the same stream object, so Python never loses them.)

Verification notes

This was verified by reading the current code at 9accd2a2 (main) and tracing the call graph; I did not build a Python runtime reproduction. The JS equivalent of steps 1-6 above is covered by a passing/failing regression test in the linked PR, which is what established the mechanism.