Skip to content

Expose local image paths to models #25944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
won-openai merged 7 commits into from
Jun 3, 2026
Merged

Expose local image paths to models #25944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4bd82b8
draft
won-openai Jun 2, 2026
9ba487e
Merge branch 'main' into attach_image_paths
won-openai Jun 2, 2026
71bc830
Reuse local image tag helper in rollout test
won-openai Jun 3, 2026
9100c4a
Fix argument comment lint for image path test
won-openai Jun 3, 2026
64a8cb2
Merge remote-tracking branch 'origin/main' into attach_image_paths
won-openai Jun 3, 2026
cd92f42
Preserve literal local image paths in prompts
won-openai Jun 3, 2026
6d77965
Merge branch 'main' into attach_image_paths
won-openai Jun 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion codex-rs/core/src/event_mapping_tests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ fn parses_user_message_with_text_and_two_images() {
#[test]
fn skips_local_image_label_text() {
let image_url = "data:image/png;base64,abc".to_string();
let label = codex_protocol::models::local_image_open_tag_text(/*label_number*/ 1);
let label = r#"<image name=[Image #1] path="/tmp/local.png">"#.to_string();
let user_text = "Please review this image.".to_string();

let item = ResponseItem::Message {
Expand Down
4 changes: 3 additions & 1 deletion codex-rs/core/tests/suite/image_rollout.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,9 @@ async fn copy_paste_local_image_persists_rollout_request_shape() -> anyhow::Resu
role: "user".to_string(),
content: vec![
ContentItem::InputText {
text: codex_protocol::models::local_image_open_tag_text(/*label_number*/ 1),
text: codex_protocol::models::local_image_open_tag_text_with_path(
/*label_number*/ 1, &abs_path,
),
},
ContentItem::InputImage {
image_url,
Expand Down
25 changes: 20 additions & 5 deletions codex-rs/protocol/src/models.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1019,9 +1019,10 @@ pub fn local_image_label_text(label_number: usize) -> String {
format!("[Image #{label_number}]")
}

pub fn local_image_open_tag_text(label_number: usize) -> String {
pub fn local_image_open_tag_text_with_path(label_number: usize, path: &std::path::Path) -> String {
let label = local_image_label_text(label_number);
format!("{LOCAL_IMAGE_OPEN_TAG_PREFIX}{label}{LOCAL_IMAGE_OPEN_TAG_SUFFIX}")
let path = path.display();
format!("{LOCAL_IMAGE_OPEN_TAG_PREFIX}{label} path=\"{path}\"{LOCAL_IMAGE_OPEN_TAG_SUFFIX}")

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Badge Bound local image paths before injecting them

For local images under very long absolute paths, this writes the entire path into a new model-visible InputText item, and the string has no hard cap before it becomes part of the request context. The model-context review rule requires every injected item to be bounded and flags new individual items that can exceed 1k tokens as P0; see AGENTS.md lines 87-90. Please truncate or otherwise cap only the displayed path while keeping the full path in non-model state.

Useful? React with 👍 / 👎.

@won-openai won-openai Jun 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extremely unlikely for the path (codex home + thread_id + call_id) to get to any level that is concernin

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Escape path text before embedding it in the tag

When the local image filename contains characters such as " or > that are valid on Unix paths, this emits a malformed model-visible tag and lets part of the path be interpreted as ordinary prompt text rather than the path attribute. For example, the added test path /tmp/a&"<b>.png produces path="/tmp/a&"<b>.png", so please escape or encode attribute delimiters before building this wrapper.

Useful? React with 👍 / 👎.

@won-openai won-openai Jun 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the local image filename will not ocntain these characters..

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Badge Cap local image path text before injecting it

This sends the full Path::display() string into a model-visible InputText without any hard cap. The context rule requires bounded injected context and calls out any new item that can exceed 1k tokens as P0 (AGENTS.md lines 87-90); fresh evidence since the earlier thread is that UserInput::LocalImage also comes from CLI/exec/app-server user-supplied paths, not only generated Codex home/thread/call paths, so a valid long path can still inflate every replayed request and churn cache. Please truncate or otherwise bound the displayed path while retaining the full path only in non-model state.

Useful? React with 👍 / 👎.

}

pub fn is_local_image_open_tag_text(text: &str) -> bool {
Expand Down Expand Up @@ -1080,7 +1081,7 @@ pub fn local_image_content_items_with_label_number(
let mut items = Vec::with_capacity(3);
if let Some(label_number) = label_number {
items.push(ContentItem::InputText {
text: local_image_open_tag_text(label_number),
text: local_image_open_tag_text_with_path(label_number, path),
});
}
items.push(ContentItem::InputImage {
Expand Down Expand Up @@ -2866,7 +2867,7 @@ mod tests {
detail: None,
},
UserInput::LocalImage {
path: local_path,
path: local_path.clone(),
detail: None,
},
]);
Expand All @@ -2883,7 +2884,10 @@ mod tests {
assert_eq!(
content.get(1),
Some(&ContentItem::InputText {
text: local_image_open_tag_text(/*label_number*/ 2),
text: local_image_open_tag_text_with_path(
/*label_number*/ 2,
&local_path
),
})
);
assert!(matches!(
Expand All @@ -2903,6 +2907,17 @@ mod tests {
Ok(())
}

#[test]
fn local_image_open_tag_preserves_path() {
assert_eq!(
local_image_open_tag_text_with_path(
/*label_number*/ 1,
std::path::Path::new(r#"/tmp/a&"<b>.png"#),
),
r#"<image name=[Image #1] path="/tmp/a&"<b>.png">"#
);
}

#[test]
fn local_image_user_input_preserves_requested_detail() -> Result<()> {
let dir = tempdir()?;
Expand Down
5 changes: 3 additions & 2 deletions codex-rs/protocol/src/protocol.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2190,8 +2190,9 @@ pub struct UserMessageEvent {
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub image_details: Vec<Option<ImageDetail>>,
/// Local file paths sourced from `UserInput::LocalImage`. These are kept so
/// the UI can reattach images when editing history, and should not be sent
/// to the model or treated as API-ready URLs.
/// the UI can reattach images when editing history. Local image prompts may
/// include a display form of the path, but these should not be treated as
/// API-ready URLs.
#[serde(default)]
pub local_images: Vec<std::path::PathBuf>,
/// Detail hints for `local_images`, indexed in parallel. Missing entries
Expand Down
Loading