Port conversation-aware STT recognition

[rosetta-livekit-bot]

Summary

• Add STT capability flags and framework hooks for session-managed keyterms and native chat-context carryover.
• Add a session-owned keyterm detector that can combine static keyterms with LLM-detected confirmed keyterms.
• Wire active agent activities to bind keyterms to the current STT, forward supported conversation items, and surface detector LLM metrics.

Verification

• pnpm --filter @livekit/agents build:types
• pnpm --filter @livekit/agents lint (passes with existing warnings)
• pnpm --filter @livekit/agents build

Notes

• pnpm --filter @livekit/agents api:check is blocked by the existing API Extractor limitation on export * as ___ in dist/index.d.ts.
• No tests were added beyond the source port scope.

---

Ported from livekit/agents#6039

Original PR description

Makes STT recognition conversation-aware via a new stt_context_options on AgentSession, grouping static keyterms with two independent, composable mechanisms for biasing recognition during a call.

Overview

• keyterms: user-defined terms applied wherever the STT accepts a term list; never modified by detection.
• keyterm_detection: an LLM-based detector (enabled, llm, turn_interval, max_keyterms, instructions) that runs a background pass per user turn over the recent transcript and maintains the keyterm set with a confirmation gate — a new term starts pending and only biases the STT once later transcript evidence confirms it; remove applies only to spellings the user explicitly corrected.
• chat_context: native conversation-context carryover for STTs that consume context directly (no LLM), forwarding each conversation turn to the provider's own field (e.g. AssemblyAI u3-rt-pro agent_context).

Details

• The detection prompt treats USER lines as untrusted STT output and ASSISTANT lines as authoritative spelling, and rejects misrecognitions: sound-alike variants of tracked terms, garbled phrases the assistant never adopts, and fragments from interrupted lines.
• Detection state is owned by the session so keyterms survive agent handoffs; user-defined terms are shown to the detection LLM as applied but never modified by it.
• New STT._push_conversation_item() hook and chat_context capability flag, alongside the existing _update_keyterms() / keyterms flag; both are forwarded by the fallback and stream adapters.
• keyterms capability implemented for deepgram (v1/v2), assemblyai, google, and livekit inference STT; chat_context implemented for assemblyai u3-rt-pro.

[changeset-bot]

🦋 Changeset detected

Latest commit: 48e902b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages

Name	Type
@livekit/agents	Major
@livekit/agents-plugin-anam	Major
@livekit/agents-plugin-assemblyai	Major
@livekit/agents-plugin-baseten	Major
@livekit/agents-plugin-bey	Major
@livekit/agents-plugin-cartesia	Major
@livekit/agents-plugin-cerebras	Major
@livekit/agents-plugin-deepgram	Major
@livekit/agents-plugin-did	Major
@livekit/agents-plugin-elevenlabs	Major
@livekit/agents-plugin-fishaudio	Major
@livekit/agents-plugin-google	Major
@livekit/agents-plugin-hedra	Major
@livekit/agents-plugin-hume	Major
@livekit/agents-plugin-inworld	Major
@livekit/agents-plugin-lemonslice	Major
@livekit/agents-plugin-liveavatar	Major
@livekit/agents-plugin-livekit	Major
@livekit/agents-plugin-minimax	Major
@livekit/agents-plugin-mistral	Major
@livekit/agents-plugin-mistralai	Major
@livekit/agents-plugin-neuphonic	Major
@livekit/agents-plugin-openai	Major
@livekit/agents-plugin-perplexity	Major
@livekit/agents-plugin-phonic	Major
@livekit/agents-plugin-resemble	Major
@livekit/agents-plugin-rime	Major
@livekit/agents-plugin-runway	Major
@livekit/agents-plugin-sarvam	Major
@livekit/agents-plugin-silero	Major
@livekit/agents-plugin-soniox	Major
@livekit/agents-plugin-tavus	Major
@livekit/agents-plugins-test	Major
@livekit/agents-plugin-trugen	Major
@livekit/agents-plugin-xai	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🟡 Scheduling pause is delayed by an async operation, allowing unintended speech scheduling during drain/handoff

The scheduling-pause flag is set (this._schedulingPaused = true at agent_activity.ts:3878) only AFTER an awaited keyterm detector close (agent_activity.ts:3877), so during that async wait new speech handles can still be enqueued.

Impact: During agent handoffs or session shutdown, extra speech handles may be scheduled and processed, extending drain time unpredictably.

Pre-PR ordering change and race window

Before this PR, _pauseSchedulingTask set this._schedulingPaused = true as its very first statement (after the early-return guard). This immediately caused scheduleSpeech() (agent_activity.ts:3862) to throw SchedulingPausedError for any new callers, preventing new speech handles from entering the queue.

The new code inserts await this.agentSession?._keytermDetector.close() before the flag is set. KeytermDetector.close() (keyterm_detection.ts:192-201) awaits any in-flight detectTask, which can block for up to the detection timeout (default 10 seconds via DETECTION_TIMEOUT at keyterm_detection.ts:42). During this window:

1. this._schedulingPaused is still false
2. scheduleSpeech() succeeds instead of throwing
3. Tool responses or concurrent generateReply() calls can enqueue new speech handles
4. These handles must then be drained before the pause completes

The fix is to set the flag before the async close, or move the close after the flag is set:

this._schedulingPaused = true;
await this.agentSession?._keytermDetector.close();

Suggested change

	await this.agentSession?._keytermDetector.close();
	this._schedulingPaused = true;
	this._schedulingPaused = true;
	await this.agentSession?._keytermDetector.close();

---

Was this helpful? React with 👍 or 👎 to provide feedback.