[Python] Add agent-framework-azure-ai-contentunderstanding package

[yungshinlintw]

Reviewer's Guide

Closes #4942

This package adds a BaseContextProvider implementation that bridges Azure Content Understanding (CU) with the Agent Framework. When a user sends file attachments (PDF, images, audio, video), the provider intercepts them in before_run(), sends them to CU for analysis, and injects the structured results (markdown + extracted fields) back into the LLM context — so the agent can answer questions about the files without the developer writing any extraction code.

Quick usage:

cu = ContentUnderstandingContextProvider(
    endpoint="https://my-resource.services.ai.azure.com/",
    credential=AzureCliCredential(),
)
agent = Agent(
    client=client,
    name="DocQA",
    instructions="You are a document analyst.",
    context_providers=[cu],
)
# Files in Message.contents are auto-analyzed; results injected into LLM context
response = await agent.run(
    Message(role="user", contents=[
        Content.from_text("What's on this invoice?"),
        Content.from_uri("https://example.com/invoice.pdf", media_type="application/pdf",
                         additional_properties={"filename": "invoice.pdf"}),
    ]),
    session=session,
)

Suggested review order

1. Start with samples — they show the feature set and usage patterns end-to-end:

Sample	What it demonstrates
01_document_qa.py	Simplest flow — upload a PDF via URL, ask a question about it. Shows Content.from_uri(), context_providers=[cu], and how CU results appear in the agent's response.
02_multi_turn_session.py	AgentSession persistence — upload a file on turn 1, ask follow-up questions on turns 2–3 without re-uploading. Shows how state["documents"] carries across turns.
03_multimodal_chat.py	PDF + audio + video in a single session (5 turns). Shows auto-detection of media types, parallel analysis, and multi-segment video output with per-segment fields.
04_invoice_processing.py	Per-file analyzer override — uses additional_properties={"analyzer_id": "prebuilt-invoice"} to extract structured invoice fields (vendor, total, line items) instead of generic markdown.
05_background_analysis.py	Non-blocking analysis with max_wait=0.5 — file starts analyzing in the background while the agent responds immediately. Next turn resolves the pending result. Shows the analyzing → ready status flow.
06_large_doc_file_search.py	CU extraction + OpenAI vector store for RAG — large documents are analyzed by CU, uploaded to a vector store, and retrieved via file_search tool instead of injecting full content into context.

2. Then review the core implementation:

Priority	File	Why
🔴 High	_context_provider.py (1087 lines)	Core logic — before_run() hook, file detection/stripping, CU analysis with timeout + background fallback, output formatting, tool registration. Most important file to review.
🔴 High	_models.py	Public API surface — DocumentEntry, DocumentStatus, AnalysisSection, FileSearchConfig TypedDicts and enums exposed to users
🟡 Medium	_file_search.py	FileSearchBackend protocol + OpenAI/Foundry factory methods for vector store integration
🟡 Medium	__init__.py	Public exports — verify the right symbols are exposed
🟡 Medium	pyproject.toml	Package metadata, dependencies, version constraints
🟢 Low	tests/	78 unit tests + 5 live integration tests

MAF API usage (needs team alignment)

This package uses the following internal/private MAF APIs — if any of these are changing or not intended for external use, this package may need updates:

• BaseContextProvider and its before_run() hook
• SessionContext.extend_instructions(), extend_messages(), extend_tools()
• Content.from_data(), Content.from_uri(), Content.type, Content.media_type, Content.additional_properties
• FunctionTool for registering list_documents()
• agent_framework._sessions.AgentSession
• agent_framework._settings.load_settings()

---

This PR adds agent-framework-azure-ai-contentunderstanding, an optional connector package that integrates Azure Content Understanding (CU) into the Agent Framework as a context provider.

What's Included

Core (_context_provider.py, _models.py, _file_search.py)

• ContentUnderstandingContextProvider -- auto-analyzes file attachments (PDF, images, audio, video) via Azure CU and injects structured results (markdown, fields) into LLM context
• Auto-detects media type and selects the right CU analyzer (prebuilt-documentSearch, prebuilt-audioSearch, prebuilt-videoSearch)
• Multi-document session state with status tracking (analyzing/uploading/ready/failed)
• Configurable timeout (max_wait) with async background fallback
• Output filtering (>90% token reduction) via AnalysisSection enum
• Auto-registered list_documents() tool for status queries
• Document content injected into conversation history for follow-up turns
• Multi-segment video/audio: per-segment fields with time ranges
• MIME sniffing for misidentified files (application/octet-stream)
• Per-file analyzer ID override via Content.additional_properties["analyzer_id"] -- mix different analyzers in the same turn (e.g., prebuilt-invoice for invoices alongside prebuilt-documentSearch for general docs)
• Duplicate filename rejection (filenames must be unique within a session)
• Optional FileSearchConfig for vector store integration (OpenAI/Foundry backends)

Samples (6 scripts + 3 DevUI)

• 01_document_qa.py -- Single PDF upload + Q&A
• 02_multi_turn_session.py -- AgentSession persistence across turns
• 03_multimodal_chat.py -- PDF + audio + video parallel analysis (5 turns)
• 04_invoice_processing.py -- Structured field extraction with prebuilt-invoice
• 05_background_analysis.py -- Non-blocking analysis with max_wait + status tracking
• 06_large_doc_file_search.py -- CU extraction + vector store RAG
• 02-devui/01-multimodal_agent -- Interactive web UI for uploading and chatting with documents/audio/video
• 02-devui/02-file_search_agent/azure_openai_backend -- DevUI with CU + Azure OpenAI file_search RAG
• 02-devui/02-file_search_agent/foundry_backend -- DevUI with CU + Foundry file_search RAG

Tests

• 66 unit tests covering all major flows
• 5 live integration tests (CU endpoint required)
• Test fixtures for PDF, audio, video, image, invoice modalities

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/azure-contentunderstanding/agent_framework_azure_contentunderstanding
_context_provider.py	251	42	83%	177–180, 283–284, 286, 290–291, 294, 298, 381, 383, 399, 521, 525, 579, 629–630, 653, 655–658, 798, 802, 808, 828–833, 835–840, 848, 857–858
_detection.py	80	11	86%	156, 162, 167–172, 231–233
_extraction.py	174	12	93%	40, 122, 125, 146, 224, 256, 277–281, 292
_file_search.py	23	4	82%	58, 65, 69, 72
_models.py	37	1	97%	111
TOTAL	30448	3549	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
6113	30 💤	0 ❌	0 🔥	1m 40s ⏱️

[Copilot]

Pull request overview

Adds a new optional Python connector package, agent-framework-azure-ai-contentunderstanding, integrating Azure Content Understanding (CU) into the Agent Framework as a BaseContextProvider for automatic attachment analysis and optional vector-store (file_search) indexing.

Changes:

• Introduces ContentUnderstandingContextProvider plus supporting models and vector-store upload abstraction (FileSearchBackend / FileSearchConfig).
• Adds extensive unit + integration tests and CU result fixtures, along with script + DevUI samples.
• Wires the new workspace package into python/pyproject.toml and python/uv.lock.

Reviewed changes

Copilot reviewed 36 out of 38 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
python/uv.lock	Adds the new workspace member and locks new deps (azure-ai-contentunderstanding, filetype).
python/pyproject.toml	Registers the package in workspace deps and adds pyright test env config.
python/packages/azure-ai-contentunderstanding/pyproject.toml	New package metadata, deps, and tooling config (pytest/ruff/mypy/pyright).
python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/init.py	Public exports for provider/models/backends.
python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_models.py	Defines DocumentStatus, AnalysisSection, DocumentEntry, FileSearchConfig.
python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_file_search.py	Adds backend abstraction for vector store upload/delete across OpenAI/Foundry clients.
python/packages/azure-ai-contentunderstanding/agent_framework_azure_ai_contentunderstanding/_context_provider.py	Implements CU analysis, session tracking, background analysis, MIME sniffing, and optional vector-store upload.
python/packages/azure-ai-contentunderstanding/tests/cu/conftest.py	Adds fixtures and mock CU client factory.
python/packages/azure-ai-contentunderstanding/tests/cu/test_models.py	Unit tests for enums/typed models and FileSearchConfig factories.
python/packages/azure-ai-contentunderstanding/tests/cu/test_context_provider.py	Comprehensive unit tests for provider flows (analysis, background, sniffing, file_search).
python/packages/azure-ai-contentunderstanding/tests/cu/test_integration.py	Live CU integration tests (skipped unless env var is set).
python/packages/azure-ai-contentunderstanding/tests/cu/fixtures/analyze_pdf_result.json	CU PDF fixture for unit tests.
python/packages/azure-ai-contentunderstanding/tests/cu/fixtures/analyze_invoice_result.json	CU invoice fixture for unit tests.
python/packages/azure-ai-contentunderstanding/tests/cu/fixtures/analyze_image_result.json	CU image fixture for unit tests.
python/packages/azure-ai-contentunderstanding/tests/cu/fixtures/analyze_audio_result.json	CU audio fixture for unit tests.
python/packages/azure-ai-contentunderstanding/tests/cu/fixtures/analyze_video_result.json	CU video fixture for unit tests.
python/packages/azure-ai-contentunderstanding/README.md	Package README with setup guidance and usage examples.
python/packages/azure-ai-contentunderstanding/LICENSE	Adds MIT license file for the new package.
python/packages/azure-ai-contentunderstanding/AGENTS.md	Package-specific agent/dev notes and architecture description.
python/packages/azure-ai-contentunderstanding/.gitignore	Ignores local-only artifacts under the package.
python/packages/azure-ai-contentunderstanding/samples/README.md	Top-level samples index for scripts and DevUI examples.
python/packages/azure-ai-contentunderstanding/samples/01-get-started/01_document_qa.py	Script sample: single PDF upload + Q&A.
python/packages/azure-ai-contentunderstanding/samples/01-get-started/02_multi_turn_session.py	Script sample: session persistence across turns.
python/packages/azure-ai-contentunderstanding/samples/01-get-started/03_multimodal_chat.py	Script sample: PDF+audio+video parallel CU analysis.
python/packages/azure-ai-contentunderstanding/samples/01-get-started/04_invoice_processing.py	Script sample: per-file analyzer override for invoice extraction.
python/packages/azure-ai-contentunderstanding/samples/01-get-started/05_background_analysis.py	Script sample: short max_wait triggers background analysis + status.
python/packages/azure-ai-contentunderstanding/samples/01-get-started/06_large_doc_file_search.py	Script sample: CU extraction + vector-store indexing for file_search.
python/packages/azure-ai-contentunderstanding/samples/02-devui/01-multimodal_agent/agent.py	DevUI agent: CU-powered upload + chat.
python/packages/azure-ai-contentunderstanding/samples/02-devui/01-multimodal_agent/init.py	DevUI agent module export.
python/packages/azure-ai-contentunderstanding/samples/02-devui/01-multimodal_agent/README.md	DevUI setup/usage doc for multimodal agent.
python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/azure_openai_backend/agent.py	DevUI agent: CU + file_search (Azure OpenAI backend).
python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/azure_openai_backend/init.py	DevUI agent module export.
python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/azure_openai_backend/README.md	DevUI setup/usage doc for Azure OpenAI file_search agent.
python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/foundry_backend/agent.py	DevUI agent: CU + file_search (Foundry backend).
python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/foundry_backend/init.py	Foundry backend sample package init.
python/packages/azure-ai-contentunderstanding/samples/02-devui/02-file_search_agent/foundry_backend/README.md	DevUI setup/usage doc for Foundry file_search agent.
python/AGENTS.md	Adds the new package to the Python “Azure Integrations” index.

[Copilot]

Pull request overview

Copilot reviewed 36 out of 38 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 36 out of 38 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 36 out of 38 changed files in this pull request and generated 1 comment.

[eavanvalkenburg]

left a couple of comments, the most important one is in the 05_background_analysis because it is a core question about how this should be used, let's discuss options for that and then we need some updates most likely