feat(openai): expose gpt-live-transcribe transcription context (keywords, languages, delay) by azuma164 · Pull Request #6613 · livekit/agents

azuma164 · 2026-07-30T08:05:15Z

Motivation

OpenAI's new streaming transcription model gpt-live-transcribe (and gpt-transcribe) accept
recognition hints on audio.input.transcription that the plugin currently can't reach:

keywords — literal terms (product names, acronyms, IDs)
languages — expected input languages (plural)
delay — the latency/accuracy tradeoff (minimal … xhigh)

There is also a correctness problem: _connect_ws always sends the singular language, but
gpt-live-transcribe uses languages instead and
the API rejects a session update carrying both.

What this does

adds keywords, languages and delay to STT.__init__, STT.with_azure and
STT.update_options, sent under audio.input.transcription
sends the configured language as a single-entry languages list on models that only accept
the plural field, so language="ja" keeps working on gpt-live-transcribe instead of being
rejected
validates keywords against the characters the server rejects (<, >, CR, LF)
warns and ignores keywords / languages on models that don't support them, and delay
outside realtime — same shape as the existing temperature / turn_detection guards
applies transcription-context changes in-band on the live connection instead of
reconnecting. The transcription config can be re-sent mid-session, so context updates don't
cut the ongoing utterance. Sends are chained so they reach the server in order (same pattern as
the Deepgram Flux Configure path in stt_v2.py)
sets STTCapabilities.keyterms for keyword-capable models and implements
_update_session_keyterms, so framework-managed keyterms merge with the user's keywords and
reach the session

stt = openai.STT(
    model="gpt-live-transcribe",
    use_realtime=True,
    prompt="A customer support call about a premium plan and account AC-42.",
    keywords=["premium plan", "AC-42", "billing"],
    languages=["en", "fr"],
    delay="low",
)

# applied on the live session, no reconnect
stt.update_options(keywords=["premium plan", "AC-42", "refund"])

Testing

New unit module tests/test_plugin_openai_stt_context.py (14 tests) covering the
session.update payload, the language / languages mutual exclusion, keyword validation, the
keyterms capability and merge, and the in-band update path (delivery, ordering, no-op when
disconnected).

uv run pytest --unit tests/test_plugin_openai_stt_context.py tests/test_plugin_openai_websocket_urls.py

ruff format / ruff check clean, and mypy -p livekit.plugins.openai reports nothing new.
Also ran the neighbouring STT unit modules (test_stt_base, test_stt_context,
test_stt_fallback, test_inference_stt_*, test_plugin_openai_*,
test_agent_update_options) — 130 passed.

Notes / open questions

Everything is additive; defaults are unchanged. Happy to also flip the default STT model to
gpt-live-transcribe in a follow-up if that's the direction you want.
prompt is OpenAI's "unstructured context" field and already existed, so it isn't new here —
but it's now updatable mid-session along with the rest of the context.
I did not wire STTCapabilities.chat_context / _push_conversation_item: unlike
AssemblyAI's agent_context, OpenAI has no separate carryover slot, so turn carryover would
have to compose into the user's prompt (which has a server-side length cap). Glad to add it in
a follow-up if you have a preferred shape.
CONTRIBUTING asks for an issue before new features — happy to open one and move the discussion
there if you'd rather; this seemed close enough to the recent keyterm-parameter PRs ((cartesia stt): expose ink-2 turn-detection thresholds and keyterm params #6594,
(xai stt): expose vad_threshold, smart_turn, smart_turn_timeout, and keyterm params #6587) to send directly.

References: gpt-live-transcribe,
Realtime transcription guide

`gpt-live-transcribe` and `gpt-transcribe` take recognition hints on `audio.input.transcription`: `keywords` for literal terms, `languages` for the expected input languages and `delay` for the latency/accuracy tradeoff. None of them were reachable from the plugin, and `language` (singular) was sent unconditionally even though `gpt-live-transcribe` rejects a session update carrying both `language` and `languages`. Adds `keywords`, `languages` and `delay` to the constructor, `with_azure` and `update_options`, sends the configured language as a single-entry `languages` list on models that only accept the plural field, and validates keywords against the characters the server rejects. Context changes are applied in-band on the live connection (the transcription config can be re-sent mid-session) so updating them doesn't cut the ongoing utterance. Keyword-capable models now report `STTCapabilities.keyterms`, so framework-managed keyterms merge with the user's own and reach the session too.

CLAassistant · 2026-07-30T08:05:26Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Hiroki Azuma seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

devin-ai-integration

Devin Review found 3 potential issues.

View 2 additional findings in Devin Review.

devin-ai-integration · 2026-07-30T08:08:43Z

+    def _update_session_keyterms(self, keyterms: list[str]) -> None:
+        if keyterms == self._session_keyterms:
+            return
+        self._session_keyterms = list(keyterms)
+        self._opts.keywords = self._merged_keywords()
+        self._apply_transcription_context()


🔴 Speech recognition can fail entirely when keyword hints are used with a model that doesn't accept them

Framework-supplied keyword hints are stored and sent to the transcription service (_update_session_keyterms at livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:502-507) even for models that reject those hints, so the service refuses the session setup and transcription stops working.
Impact: An agent configured with keyterms but a model that doesn't support them gets no transcripts at all instead of a harmless warning.

Missing capability guard in the keyterms override

The base implementation livekit-agents/livekit/agents/stt/stt.py:276-289 warns and skips when STTCapabilities.keyterms is false, and other plugins that only conditionally support keyterms defer to it (see livekit-agents/livekit/agents/inference/stt.py:729-737). The OpenAI override has no such guard.

The framework calls _update_session_keyterms unconditionally for static keyterms regardless of capability (livekit-agents/livekit/agents/voice/keyterm_detection.py:246-256 and :289). So with e.g. model="gpt-4o-mini-transcribe", use_realtime=True and keyterms=[...] configured on the session, self._opts.keywords gets set and _build_transcription_config (livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:523-524) adds keywords to audio.input.transcription, which the Realtime API rejects for that model — the error event then raises a non-retryable APIError in recv_task.

The same hole exists after update_options(model=...) switches to a model without keyword support while keywords are already stored.

Suggested change

def _update_session_keyterms(self, keyterms: list[str]) -> None:

if keyterms == self._session_keyterms:

return

self._session_keyterms = list(keyterms)

self._opts.keywords = self._merged_keywords()

self._apply_transcription_context()

def _update_session_keyterms(self, keyterms: list[str]) -> None:

if not self.capabilities.keyterms:

super()._update_session_keyterms(keyterms) # warn-and-skip for unsupported models

return

if keyterms == self._session_keyterms:

return

self._session_keyterms = list(keyterms)

self._opts.keywords = self._merged_keywords()

self._apply_transcription_context()

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-30T08:08:44Z

+        self._session_keyterms = list(keyterms)
+        self._opts.keywords = self._merged_keywords()


🟡 Automatically collected recognition hints containing certain characters can break transcription

Hints coming from the framework are pushed to the service without the character check applied to user-provided ones (_merged_keywords() at livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:506), so a single hint containing <, > or a line break makes the service reject the whole configuration.
Impact: A stray character in an auto-detected hint can silently kill transcription for the rest of the call.

Validation applied only on the user path

STT.__init__ (livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:184-185) and update_options (:452-456) run _validate_keywords, which raises for the characters the server rejects (_KEYWORD_FORBIDDEN_CHARS, :85-95). Keyterms arriving through _update_session_keyterms (LLM-detected terms or session-level static keyterms, see livekit-agents/livekit/agents/voice/keyterm_detection.py:389) bypass that check and land in the merged keywords list sent in session.update. Since these terms are not authored inline by the developer, raising would also be wrong here — dropping/sanitizing the offending terms with a debug/warning log is the safer behavior.

Prompt for agents

In livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py, keywords supplied by the user through __init__/update_options are validated with _validate_keywords (rejecting '<', '>', CR and LF, which the Realtime API refuses). Keyterms arriving via the framework hook _update_session_keyterms (static session keyterms plus LLM-detected terms from livekit-agents/livekit/agents/voice/keyterm_detection.py) are merged into the same keywords list without validation, so one bad term makes the server reject the whole session.update and transcription stops. Consider filtering out (rather than raising on) invalid framework-supplied keyterms in _update_session_keyterms/_merged_keywords, logging what was dropped.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-07-30T08:08:46Z

+        ws = self._ws
+        if ws is None or ws.closed:
+            # not connected; the next connection carries the latest options
+            return


🟡 Recognition hints changed while no audio stream is active may never take effect

Updated recognition hints are only pushed onto an already-open connection (_send_session_update_task at livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:705-708) and are dropped when nothing is connected, but a later stream can pick up a recycled connection that still carries the old hints.
Impact: Keyword/language/prompt changes made between utterances can be silently lost for up to the connection lifetime.

Pooled connections are not re-configured on reuse

_send_session_update_task returns early when self._ws is None or closed, with the comment "the next connection carries the latest options". That only holds for freshly created connections: _connect_ws (livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:570-594) is the sole place the session.update is sent, and ConnectionPool.get (livekit-agents/livekit/agents/utils/connection_pool.py:100-118) returns a cached websocket without invoking connect_cb when an available one exists (up to _max_session_duration = 10 min). So if update_options(keywords=...)/_update_session_keyterms runs while no stream holds a connection, and a new stream then reuses the pooled socket, the server keeps the previous transcription context.

One fix is to have _apply_transcription_context remember that the context is dirty and re-send (or self._pool.invalidate()) when no live connection accepted the update.

Was this helpful? React with 👍 or 👎 to provide feedback.

azuma164 requested a review from a team as a code owner July 30, 2026 08:05

azuma164 marked this pull request as draft July 30, 2026 08:08

devin-ai-integration Bot reviewed Jul 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(openai): expose gpt-live-transcribe transcription context (keywords, languages, delay) - #6613

feat(openai): expose gpt-live-transcribe transcription context (keywords, languages, delay)#6613
azuma164 wants to merge 1 commit into
livekit:mainfrom
azuma164:feat/openai-stt-transcription-context

azuma164 commented Jul 30, 2026

Uh oh!

CLAassistant commented Jul 30, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jul 30, 2026

Uh oh!

devin-ai-integration Bot Jul 30, 2026

Uh oh!

devin-ai-integration Bot Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		self._session_keyterms = list(keyterms)
		self._opts.keywords = self._merged_keywords()

Uh oh!

Conversation

azuma164 commented Jul 30, 2026

Motivation

What this does

Testing

Notes / open questions

Uh oh!

CLAassistant commented Jul 30, 2026

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants