Python: Fix file_search citations breaking assistant history roundtrip

[moonbox3]

Motivation and Context

Closes #5556. After the RC5 → 1.0 migration to the Responses API, multi-agent flows (SequentialBuilder, GroupChatBuilder) break with a 400 from the API whenever one agent uses file_search and its history is forwarded to another agent. The root cause is in OpenAIResponsesClient (inherited by FoundryChatClient):

1. Outbound _prepare_content_for_openai maps hosted_file → input_file for any role, but input_file is an input-only content type that the Responses API rejects inside an assistant message.
2. Streaming attaches file_citation / container_file_citation / file_path annotations as standalone HostedFileContent items, while the non-streaming path attaches them as text annotations. The asymmetry means streaming users always trip (1).
3. Outbound output_text hardcodes "annotations": [], silently dropping citation context on every roundtrip even when (1) wouldn't fire.

This worked in RC5 because Chat Completions used flat text annotations and had no input/output content-type schema split.

Description

Three coordinated fixes in packages/openai/agent_framework_openai/_chat_client.py:

• Skip hosted_file for assistant role in _prepare_content_for_openai. Hosted-file content on an assistant message is a citation reference, not a replayable input file; dropping it stops the 400.
• Streaming attaches citations as text annotations, matching the non-streaming path. file_citation, container_file_citation, and file_path now produce Content.from_text(text="", annotations=[Annotation(type="citation", ...)]) instead of a standalone HostedFileContent. URL citations (which already used this pattern) are unchanged.
• output_text preserves annotations on roundtrip via a new _annotations_to_output_text helper that converts framework Annotation objects back to Responses API annotation dicts (file_citation, url_citation, container_file_citation, file_path).

Built TDD-style: 4 new behavior tests plus an end-to-end regression test that exercises the exact streaming-citation → assistant-history-forwarding flow from the bug report. 3 existing streaming-annotation tests were updated to assert the new (consistent) behavior — they previously asserted the buggy split-content shape.

Verified locally: pytest packages/openai/tests/openai/, pytest packages/foundry/tests/foundry/, pytest packages/core/tests/, mypy, and ruff all clean.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[moonbox3]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/openai/agent_framework_openai
_chat_client.py	968	123	87%	263, 276, 601–604, 608–609, 615–616, 651–657, 678, 686, 709, 827, 926, 985, 987, 989, 991, 1057, 1071, 1151, 1161, 1166, 1209, 1325, 1506, 1511, 1515–1517, 1521–1522, 1588, 1624, 1630, 1640, 1646, 1651, 1657, 1662–1663, 1682, 1772, 1794–1795, 1810–1811, 1829–1830, 1873, 2039, 2077–2078, 2094, 2096, 2176–2184, 2214, 2324, 2359, 2374, 2394–2404, 2417, 2428–2432, 2446, 2460–2471, 2480, 2512–2515, 2525–2526, 2537–2539, 2553–2555, 2565–2566, 2572, 2587
TOTAL	30554	3562	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
6139	30 💤	0 ❌	0 🔥	1m 46s ⏱️

[Copilot]

Pull request overview

Fixes a Responses API incompatibility where file_search citations (and other hosted-file citation artifacts) could be serialized back into assistant history as input_file, causing 400s when multi-agent workflows forward assistant messages.

Changes:

• Preserve assistant output_text.annotations on serialization by converting framework Annotation(type="citation") back into Responses API annotation dicts.
• Prevent hosted_file contents in assistant messages from roundtripping as input_file by dropping them during outbound preparation.
• Make streaming citation events attach as text annotations (via empty text Content) rather than standalone HostedFileContent, aligning streaming with non-streaming behavior; update/add regression tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
python/packages/openai/agent_framework_openai/_chat_client.py	Adds annotation roundtrip helper, drops assistant hosted_file on outbound, and changes streaming citation handling to produce text annotations.
python/packages/openai/tests/openai/test_openai_chat_client.py	Adds/updates tests to cover citation roundtrip behavior and the streaming→history regression.

[github-actions]

Automated Code Review

Reviewers: 4 | Confidence: 90%

✓ Correctness

This PR fixes a roundtrip bug where streaming file_search citations were stored as HostedFileContent, then serialized as input_file items in assistant history — rejected by the Responses API. The fix correctly converts these to text annotations (matching the non-streaming path) and adds a helper to reconstruct API-format annotations on output. The branching logic in _annotations_to_output_text correctly maps all four annotation variants (file_path, file_citation, url_citation, container_file_citation) and the priority ordering of the elif chain prevents misclassification. Dropping hosted_file for assistant role returns {} which is properly filtered by the existing if prepared_content: guard in _prepare_message_for_openai. Tests are comprehensive and cover the main code paths.

✓ Security Reliability

This diff fixes a real bug where streaming file_search citations were represented as HostedFileContent, which then serialized as input_file in assistant history and was rejected by the Responses API. The fix converts these to text annotations that roundtrip cleanly. The new _annotations_to_output_text helper safely reshapes Annotation TypedDicts (all fields optional, accessed via .get()) into API-compatible dicts. The empty-dict guard for hosted_file on assistant messages is correctly filtered by existing calers (line 1352: if prepared_content: — empty dict is falsy). The getattr(content, "annotations", None) is slightly redundant since content.annotations is unconditionally set in Content.__init__, but is harmlessly defensive. No injection risks, resource leaks, secrets exposure, or unhandled failure modes found.

✓ Test Coverage

The PR adds four new tests and updates three existing streaming annotation tests to reflect the behavioral change from Content.from_hosted_file to Content.from_text with citation annotations. Test coverage is solid for the major paths: file_citation, url_citation, and container_file_citation roundtrips are all verified, as is the hosted_file → {} guard for assistant role. However, the file_path branch of _annotations_to_output_text (the elif file_id: path on diff line ~300) has no roundtrip test, even though it is one of the three streaming annotation types that changed representation in this PR. The streaming-parse side is tested (test_streaming_annotation_added_with_file_path), but no test verifies that the resulting annotation serializes back to {"type": "file_path", "file_id": ...} via _annotations_to_output_text.

✓ Design Approach

The change fixes the immediate input_file rejection, but the streaming-side approach still breaks the underlying model of citations: it now emits annotation-added events as separate empty text contents instead of attaching them to the streamed text they annotate. Since assistant history serialization writes each Content as its own output_text item, the annotation offsets are no longer relative to the actual text, unlike the non-streaming path which keeps text and annotations together. That means the PR avoids one API error while still losing the text/citation relationship during round-trip.

---

Automated review by moonbox3's agents

[moonbox3]

Pushed 4c67642 addressing all four review comments:

• annotated_regions fan-out: _annotations_to_output_text now emits one Responses API entry per region for url_citation / container_file_citation, instead of collapsing to regions[0].
• Span validation: regions whose start_index / end_index are not ints are skipped, so the helper can't produce malformed payloads when fed partial regions.
• file_path test: added test_assistant_text_preserves_file_path_annotation covering the file_id-only branch that wasn't exercised before.
• Streaming citation attachment: the concern that span indices would reference text == "" doesn't apply once _finalize_response runs — _coalesce_text_content merges consecutive text contents and unions their annotations, so the citation lands on the merged assistant text. Added test_streamed_file_citation_coalesces_onto_surrounding_text to lock that behavior in. Verified locally with the full chunk → ChatResponse.from_updates path: empty-text + annotation update gets merged into the preceding "Hello world." text content with the annotation attached.