conversation-aware STT recognition (keyterms + chat context) by longcw · Pull Request #6039 · livekit/agents

longcw · 2026-06-10T12:03:31Z

Makes STT recognition conversation-aware via a new stt_context_options on AgentSession, grouping static keyterms with two independent, composable mechanisms for biasing recognition during a call.

Overview

keyterms: user-defined terms applied wherever the STT accepts a term list; never modified by detection.
keyterm_detection: an LLM-based detector (enabled, llm, turn_interval, max_keyterms, instructions) that runs a background pass per user turn over the recent transcript and maintains the keyterm set with a confirmation gate — a new term starts pending and only biases the STT once later transcript evidence confirms it; remove applies only to spellings the user explicitly corrected.
chat_context: native conversation-context carryover for STTs that consume context directly (no LLM), forwarding each conversation turn to the provider's own field (e.g. AssemblyAI u3-rt-pro agent_context).

Details

The detection prompt treats USER lines as untrusted STT output and ASSISTANT lines as authoritative spelling, and rejects misrecognitions: sound-alike variants of tracked terms, garbled phrases the assistant never adopts, and fragments from interrupted lines.
Detection state is owned by the session so keyterms survive agent handoffs; user-defined terms are shown to the detection LLM as applied but never modified by it.
New STT._push_conversation_item() hook and chat_context capability flag, alongside the existing _update_keyterms() / keyterms flag; both are forwarded by the fallback and stream adapters.
keyterms capability implemented for deepgram (v1/v2), assemblyai, google, and livekit inference STT; chat_context implemented for assemblyai u3-rt-pro.

Show user-defined terms to the detection LLM as applied so it stops re-proposing them every pass; add misrecognition rules to the default prompt (sound-alike variants of tracked terms, user-only garbled phrases, interrupted-line fragments) and route corrections through the normal confirmation gate.

User-defined keyterms were silently dropped unless detection.enabled was set: start() returned before binding the STT, so the initial push and later set_user_keyterms() were no-ops. Bind the STT unconditionally (skipping the push when there are no terms, so sessions without keyterms see no capability warning or reconnect) and gate only the detection setup on enabled.

chenghao-mou · 2026-06-10T14:43:09Z

+    "enabled": False,
+    "llm": None,
+    "turn_interval": 1,
+    "max_keyterms": None,


some of vendors impose a max limit, maybe we should check this in the extract-for-model function.

chenghao-mou · 2026-06-13T20:51:49Z

                endpoint_url=endpoint_url,
            )

+    def _update_keyterms(self, keyterms: list[str]) -> None:


Only the flux models can support mid-stream keyterm update: https://developers.deepgram.com/docs/keyterm#dynamic-keyterm-updates-flux-only should we disable this for nova models?

now update_options is called after END_OF_SPEECH event, to avoid reconnection in the middle of the turn. meanwhile, the audio during reconnection is buffered in input_ch so ideally it won't missing user transcript.

The keyterm detector no longer reuses the agent's LLM. `detection.llm` accepts an LLM instance or a model string (resolved via the inference gateway) and defaults to a built-in detection model. Also tightens the default extraction prompt.

Rename KeytermOptions to STTContextOptions and move it under the stt package as recognition_context. Adds a chat_context mechanism alongside keyterms and keyterm_detection: STTs that natively consume conversation context (e.g. AssemblyAI u3-rt-pro agent_context) receive forwarded turns via a new _push_conversation_item hook, while keyterm detection remains LLM-based for STTs that accept a term list.

u9g · 2026-06-22T14:22:12Z

+        # automatically detect keyterms and apply them to the STT per user turn
+        stt_context_options={
+            "keyterms": ["LiveKit"],
+            "keyterm_detection": {"enabled": True, "turn_interval": 1},


"turn_interval": 1 is the default, probably not necessary to put it here.

also, I had to search the userbase to understand what turn_interval represents, maybe a better name is detect_every_n_turns.

u9g · 2026-06-22T14:30:08Z

+        return {"keyterm": list(keyterms)}
+    if model.startswith("assemblyai/"):
+        return {"keyterms_prompt": list(keyterms)}
+    if model.startswith("speechmatics/"):


would it make sense to add STTCapabilities.keyterms to the speechmatics plugin?

toubatbrian · 2026-06-23T00:38:08Z

+        return [], [], []
+    req_ctx = ChatContext.empty()
+    req_ctx.add_message(role="system", content=instructions or _DEFAULT_KEYTERM_INSTRUCTIONS)
+    req_ctx.add_message(role="user", content=user_msg)


small nit: should we format user_msg in a prefix caching friendly way so this detector LLM can run faster?

the instructions is already fixed, but the turns added to the user message is a sliding window that will change in every request. and I think the total token per request won't reach the caching threshold for many models.

toubatbrian · 2026-06-23T00:38:51Z

+            # between turns is the only blank line and reliably marks a turn boundary
+            body = "\n".join(line for line in text.splitlines() if line.strip())
+            turns.append(f"{item.role.upper()}: {body}")
+            if len(turns) >= _MAX_TRANSCRIPT_MESSAGES:


should this be configurable?

The framework keyterm hook now carries only the session-managed set (static stt_context_options.keyterms plus auto-detection) and is renamed _update_session_keyterms. Plugins keep the user's own keyterms (constructor / update_options) in a separate slot and apply the union of the two, so enabling detection or session keyterms no longer drops keyterms a user configured directly on the plugin. Each plugin holds its own _session_keyterms, recomputes the effective set on change, and pushes it through stream.update_options — which already reconnects (Deepgram v1/v2, Google) or sends a live update (AssemblyAI, inference) — so there is one apply path and no separate push helper or manual reconnect. The base STT carries no keyterm state; its default hook only warns and skips for STTs without keyterm support. Google biases session terms at the minimum user-keyword boost (or a moderate default when none), so detected terms never outweigh an explicit keyword. Inference folds the merge into _keyterms_extra_for_model across the deepgram/assemblyai/speechmatics keys. The detector's internal terms are renamed user/auto to static/detected to avoid clashing with the plugin-level "user" keyterms.

Deepgram's keyterm extra is typed str | list[str], so a bare string was splat character-by-character when merged with framework session keyterms.

adaptation shadows keywords in build_adaptation, so keyterms could never reach the recognizer. Gating the capability on init-time adaptation stops the framework from running LLM detection passes whose results would be discarded.

Applying detected/session keyterms reopened the connection immediately, which could cut off the in-flight utterance. Now each reconnecting stream stashes the new set in a single typed slot while the user is speaking and flushes it via update_options() at END_OF_SPEECH; if idle it still applies immediately. Explicit user update_options() is unchanged. Deepgram and Google reconnect; inference applies live but defers too since the gateway may reconnect upstream. The base stream is untouched.

The v2 STT (Flux) reconnected immediately on a session keyterm update, which could cut off the in-flight utterance. Mirror v1: stash the merged set while speaking and flush it via update_options() at END_OF_SPEECH.

update_options(keywords=…) overwrote _config.keywords with the raw user keywords, dropping any active session keyterms — and the early-return in _update_session_keyterms kept them from being reapplied. Extract the merge into _merge_keywords() and reassign the merged list before writing config and forwarding to streams.

AssemblyAI and inference forwarded the raw user keyterms to live streams, dropping any active session keyterms (the equality guard in _update_session_keyterms then kept them from being reapplied). Re-merge with the session set before forwarding, matching Deepgram and Google.

The keyterm detector runs its own LLM whose metrics_collected events had no subscriber, so its usage never reached session.usage, MetricsCollectedEvent, or OTel. KeytermDetector is now an EventEmitter that re-emits the detection LLM's metrics, and the agent activity wires it into the session metrics pipeline like every other model.

theomonnom

lgtm!

theomonnom · 2026-06-29T03:31:52Z

+        self._stt._update_session_keyterms(keyterms)
+
+    def _push_conversation_item(self, item: ConversationItemAddedEvent) -> None:
+        self._stt._push_conversation_item(item)


Should it be a update_chat_ctx instead? Seems more consistent?

using push because each call carries a single chat item as it's added and feeds it incrementally, update_chat_ctx feels like it's a replacing.

theomonnom · 2026-06-29T06:11:50Z

        llm: NotGivenOr[llm.LLM | llm.RealtimeModel | LLMModels | str] = NOT_GIVEN,
        tts: NotGivenOr[tts.TTS | TTSModels | str] = NOT_GIVEN,
        turn_handling: NotGivenOr[TurnHandlingOptions] = NOT_GIVEN,
+        stt_context_options: NotGivenOr[STTContextOptions] = NOT_GIVEN,


I think we should name this feature keyterms_options directly inside the AgentSession

what about the chat_ctx options?

or we enable it if the STT.capability support chat context, and user can enable/disable it via STT args.

theomonnom · 2026-06-29T06:12:26Z

+    @property
+    def keyterms(self) -> list[str]:
+        """The effective keyterms (user-defined + auto-detected) currently applied to the STT."""
+        return self._keyterm_detector.keyterms


I think we don't need a new property if it's inside AgentSessionOptions

this is the keyterms applied to the STT (including the detected terms), expose it here in case user want to save the keyterms.

…om STT capability Move the keyterm detector to voice/keyterm_detection.py and drop chat_context from the session options. Conversation-context carryover is now gated by STTCapabilities.chat_context, toggled per-provider via STT args (AssemblyAI: agent_context_carryover, default False, warns when set on an unsupported model).

devin-ai-integration

Devin Review found 2 new potential issues.

Deepgram v1's UtteranceEnd path ended speech without calling _on_end_of_speech, so a keyterm change deferred while speaking was stuck until the next speech_final. Also, across the deepgram, deepgram v2, google, and inference STTs, an explicit update_options keyterm/keywords/extra now clears the pending value so a stale snapshot can't briefly revert a newer update at end-of-speech.

longcw added 6 commits June 9, 2026 21:04

wip

59f1a58

update fnc tool

ebbe1b5

clean

8234757

refactor to KeytermDetector

73eb033

fix test

a6096af

This comment was marked as resolved.

Sign in to view

chenghao-mou reviewed Jun 10, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into longc/auto-stt-keyterms

2ca544a

This comment was marked as resolved.

Sign in to view

make _update_keyterms private

ffd105c

This comment was marked as resolved.

Sign in to view

fix(google): preserve user-tuned keywords on keyterms update

e7b7c92

theomonnom mentioned this pull request Jun 12, 2026

voice: output retries for run(output_type=...) #6080

Merged

chenghao-mou reviewed Jun 13, 2026

View reviewed changes

cursor Bot mentioned this pull request Jun 14, 2026

docs: daily engineering digest for 2026-06-14 Seventhen/agents#5

Draft

Merge remote-tracking branch 'origin/main' into longc/auto-stt-keyterms

cb0b308

longcw requested a review from a team as a code owner June 16, 2026 09:46

This comment was marked as resolved.

Sign in to view

longcw changed the title ~~automatic STT keyterm detection~~ conversation-aware STT recognition (keyterms + chat context) Jun 22, 2026

longcw added 2 commits June 22, 2026 21:28

revert example

ef23cc3

Merge remote-tracking branch 'origin/main' into longc/auto-stt-keyterms

d734805

This comment was marked as resolved.

Sign in to view

u9g reviewed Jun 22, 2026

View reviewed changes

toubatbrian reviewed Jun 23, 2026

View reviewed changes

Comment thread livekit-agents/livekit/agents/voice/keyterm_detection.py

This comment was marked as resolved.

Sign in to view

longcw added 3 commits June 23, 2026 15:22

fix(stt): wrap bare-string keyterm before merging session keyterms

abf645f

Deepgram's keyterm extra is typed str | list[str], so a bare string was splat character-by-character when merged with framework session keyterms.

This comment was marked as resolved.

Sign in to view

fix(deepgram): defer keyterm reconnect at end of speech in v2 STT

2e82b55

The v2 STT (Flux) reconnected immediately on a session keyterm update, which could cut off the in-flight utterance. Mirror v1: stash the merged set while speaking and flush it via update_options() at END_OF_SPEECH.

This comment was marked as resolved.

Sign in to view

longcw force-pushed the longc/auto-stt-keyterms branch from 53866c4 to 0a93cb8 Compare June 23, 2026 11:47

longcw added 2 commits June 23, 2026 19:52

This comment was marked as resolved.

Sign in to view

theomonnom approved these changes Jun 29, 2026

View reviewed changes

longcw added 2 commits June 29, 2026 15:38

timeout keyterm detection pass so a stuck call can't stall detection

b442136

expose detection timeout as keyterm_detection option

a4a0012

This comment was marked as resolved.

Sign in to view

longcw added 3 commits June 29, 2026 15:57

forward merged keywords to active google stt streams on user update

5328bbd

Merge remote-tracking branch 'origin/main' into longc/auto-stt-keyterms

1f6f803

devin-ai-integration Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py

Comment thread livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py

longcw merged commit 7c86e58 into main Jun 30, 2026
24 checks passed

longcw deleted the longc/auto-stt-keyterms branch June 30, 2026 03:19

rosetta-livekit-bot Bot mentioned this pull request Jun 30, 2026

Port conversation-aware STT recognition livekit/agents-js#1913

Open

cursor Bot mentioned this pull request Jul 1, 2026

docs: daily engineering digest 2026-07-01 Seventhen/agents#19

Draft

toubatbrian mentioned this pull request Jul 6, 2026

feat(stt): conversation-aware recognition — keyterms + chat context livekit/agents-js#1967

Merged

4 tasks

Uh oh!

Conversation

longcw commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Details

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

theomonnom left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

longcw commented Jun 10, 2026 •

edited

Loading