Add realtime speech append control by guinness-oai · Pull Request #27917 · openai/codex

guinness-oai · 2026-06-12T17:45:03Z

Why

Realtime voice harness tuning needs app-side control over what backend Codex text is spoken. Backend orchestrator text is written for a reading UI, so automatically speaking every preamble, progress update, or final assistant message can make the realtime voice model too chatty.

For experimentation, clients need two simple controls: keep app/client text-item injection on the existing item-create path, and add an explicit speakable path that app code can call only when it wants realtime to speak. Automatic Codex output also needs an opt-in way to switch from the protocol's default speakable path to regular realtime items, with a caller-provided prefix so prompt wording can be tuned outside core.

The default remains unchanged: if a client omits the new start fields and never calls appendSpeech, automatic backend output continues down the existing speakable path for the selected realtime protocol.

What Changed

Adds experimental thread/realtime/appendSpeech for app-provided speakable text.
Keeps existing thread/realtime/appendText as the item-create API for app-provided realtime text items.
Adds codexResponsesAsItems / codex_responses_as_items on thread/realtime/start to send automatic Codex responses with conversation.item.create instead of the protocol's default speakable output path.
Adds codexResponseItemPrefix / codex_response_item_prefix so clients can prepend experiment instructions to those automatic Codex response items.
Keeps literal conversation.handoff.append routing scoped to the v1 speakable path; v2 default speech uses its item/function-output plus response.create behavior.
Removes the earlier public silent-context API and hardcoded silent-context prefix.
Updates realtime tests to cover default automatic speakable behavior, opt-in automatic item-create behavior, and explicit appendSpeech behavior.

Validation

cargo check -p codex-core -p codex-app-server -p codex-api
just test -p codex-app-server realtime_conversation
just test -p codex-core realtime_conversation (50/51 passed in the filtered parallel run; the lone failure passed when rerun in isolation)
just test -p codex-core conversation_mirrors_assistant_message_text_to_realtime_handoff
just test -p codex-api e2e_connect_and_exchange_events_against_mock_ws_server
just fix -p codex-core
just fix -p codex-app-server
cargo build -p codex-cli

…doff-append-control # Conflicts: # codex-rs/app-server-protocol/src/protocol/common.rs # codex-rs/app-server-protocol/src/protocol/v2/realtime.rs # codex-rs/app-server/README.md # codex-rs/app-server/src/request_processors/turn_processor.rs # codex-rs/app-server/tests/suite/v2/experimental_api.rs # codex-rs/app-server/tests/suite/v2/realtime_conversation.rs # codex-rs/codex-api/src/endpoint/realtime_websocket/methods_v1.rs # codex-rs/codex-api/src/endpoint/realtime_websocket/methods_v2.rs # codex-rs/codex-api/src/endpoint/realtime_websocket/protocol.rs # codex-rs/core/src/realtime_conversation.rs # codex-rs/core/tests/suite/compact_remote.rs # codex-rs/core/tests/suite/realtime_conversation.rs # codex-rs/protocol/src/protocol.rs

jiayuhuang-openai

Clean

…doff-append-control

guinness-oai · 2026-06-15T19:46:18Z

@codex

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab5d0d0282

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-15T19:55:17Z

    pub architecture: Option<RealtimeConversationArchitecture>,
+    /// Sends automatic Codex responses as realtime conversation items instead of handoff appends.
+    #[ts(optional = nullable)]
+    pub codex_responses_as_items: Option<bool>,


Model omission-false flag as a bool

This flag is collapsed with unwrap_or(false), so omission and null mean the same thing while clients see a nullable tri-state. The app-server API guidance says omission-means-false booleans should be defaulted bool; please avoid the meaningless null case. guidance

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-15T19:55:17Z

+                writer
+                    .send_conversation_handoff_append(handoff_id, output_text)


Avoid appending the final handoff text twice

When a V2 background_agent handoff completes in the default mode, the final assistant text has already been sent from handoff_out as a ProgressUpdate; TurnComplete then calls handoff_complete, and this arm appends the same last_output_text again. For delegated turns with any assistant output, realtime receives duplicate speakable backend text, so complete the handoff without re-appending the text or send only a true delta.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-15T22:51:03Z

💡 Codex Review

codex/codex-rs/app-server-protocol/src/protocol/v2/realtime.rs

Line 159 in 6acf653

pub struct ThreadRealtimeAppendSpeechParams {

Regenerate app-server schema fixtures

New v2 realtime API shapes are added, but app-server-protocol/schema has no appendSpeech/ThreadRealtimeAppendSpeech entries, so generated schema consumers will miss the method. Regenerate the app-server schema fixtures for this API shape change as required (guidance).

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

guinness-oai added 3 commits June 12, 2026 10:44

Add realtime handoff append controls

58cec52

Add realtime handoff update opt-out

4c2a853

Use explicit realtime handoff speech

aa1b854

guinness-oai changed the title ~~Add realtime handoff append controls~~ Use explicit realtime handoff speech Jun 12, 2026

guinness-oai added 2 commits June 12, 2026 16:04

Add realtime handoff context opt-in

38a8d24

jiayuhuang-openai mentioned this pull request Jun 13, 2026

[codex] control automatic realtime handoff delivery #27986

Merged

jiayuhuang-openai approved these changes Jun 13, 2026

View reviewed changes

Refine realtime speech context API

58474a5

guinness-oai changed the title ~~Use explicit realtime handoff speech~~ Add explicit realtime speech and silent context APIs Jun 14, 2026

guinness-oai added 2 commits June 14, 2026 13:06

Merge remote-tracking branch 'origin/main' into guinness/realtime-han…

5cfce1b

…doff-append-control

Simplify realtime speech handoff API

ab5d0d0

guinness-oai changed the title ~~Add explicit realtime speech and silent context APIs~~ Add realtime speech append control Jun 15, 2026

chatgpt-codex-connector Bot reviewed Jun 15, 2026

View reviewed changes

guinness-oai added 5 commits June 15, 2026 13:14

Tighten realtime outbound routing types

63c228e

Reuse generic realtime item create helper

ebabc00

Remove realtime context item writer

47b8633

Keep v2 realtime output off handoff append

25cdef9

Flatten realtime Codex output config

6acf653

guinness-oai marked this pull request as ready for review June 15, 2026 22:44

guinness-oai requested a review from a team as a code owner June 15, 2026 22:44

aibrahim-oai approved these changes Jun 15, 2026

View reviewed changes

guinness-oai merged commit 1d8ff89 into main Jun 15, 2026
31 checks passed

guinness-oai deleted the guinness/realtime-handoff-append-control branch June 15, 2026 23:16

github-actions Bot locked and limited conversation to collaborators Jun 15, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add realtime speech append control#27917

Add realtime speech append control#27917
guinness-oai merged 13 commits into
mainfrom
guinness/realtime-handoff-append-control

guinness-oai commented Jun 12, 2026 •

edited

Loading

Uh oh!

jiayuhuang-openai left a comment

Uh oh!

guinness-oai commented Jun 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 15, 2026

Uh oh!

chatgpt-codex-connector Bot Jun 15, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		writer
		.send_conversation_handoff_append(handoff_id, output_text)

Conversation

guinness-oai commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What Changed

Validation

Uh oh!

jiayuhuang-openai left a comment

Choose a reason for hiding this comment

Uh oh!

guinness-oai commented Jun 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot commented Jun 15, 2026

💡 Codex Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

guinness-oai commented Jun 12, 2026 •

edited

Loading