.NET: Add Azure Content Understanding context provider package - #5998
Open
changjian-wang wants to merge 74 commits into
Open
.NET: Add Azure Content Understanding context provider package#5998changjian-wang wants to merge 74 commits into
changjian-wang wants to merge 74 commits into
Conversation
- Changed the type of `result` in DocumentEntry from dict to str to store LLM-ready text. - Introduced `search_payload` in DocumentEntry for optional alternate rendering. - Updated FileSearchConfig to include `include_fields` option for vector store uploads. - Modified tests to reflect changes in DocumentEntry and FileSearchConfig. - Adjusted integration tests to validate new result structure and rendering. - Removed legacy format_result tests as rendering is now handled by the SDK.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <copilot@github.com>
# Conflicts: # python/packages/azure-contentunderstanding/pyproject.toml
Adds new src project (with AssemblyMarker placeholder), unit-test project, and integration-test project; wires them into agent-framework-dotnet.slnx; adds Azure.AI.ContentUnderstanding 1.2.0-beta.1 to Directory.Packages.props and bumps Azure.Core 1.53.0 -> 1.54.0 (transitive requirement of CU 1.2.0-beta.1). Restore validated; full multi-TFM build pending verification on next machine.
Add XML doc comments to AssemblyMarker (CS1591) and temporarily suppress RT0002/RT0003 (unused reference analyzers) on the new csproj since the scaffold doesn't yet use ProjectReference/PackageReferences. NoWarn entries are flagged for removal in Phase 5 once ContentUnderstandingContextProvider consumes them. Build now succeeds across net472;netstandard2.0;net8.0;net9.0;net10.0.
- Implement ModelsTests to validate AnalysisSection and DocumentStatus enums. - Create OptionsTests for validating ContentUnderstandingContextProviderOptions constructor and defaults. - Add ParityGapTests to cover provider-level parity gaps including URL input and session isolation. - Introduce ProviderStateTests to ensure internal state types are JSON round-trippable. - Implement RendererParityGapTests for renderer-level parity gaps including source metadata and field value extraction. - Remove obsolete ScaffoldingTests. - Add various test doubles (CountingClientFactory, FakeAnalyzer, FakeFileSearchBackend, etc.) to facilitate testing.
… records - Updated ContentUnderstandingContextProvider to handle null results in failure cases. - Introduced AnalysisOutcome and AnalysisAttempt records for better analysis management. - Added FileSearchOutcome record to encapsulate vector-store upload results. - Updated project references to align with new structure.
…update tests for Python parity
… for control character stripping, path traversal hardening, and filename length capping
moonbox3
added
documentation
github-actions
Bot
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds Azure Content Understanding (CU) integration across the agent framework with new provider implementations, samples, and tests, and updates dependencies to align with CU SDK rendering helpers.
Changes:
- Python CU provider now stores
DocumentEntry.resultas a rendered LLM-ready string (viato_llm_input) and addssearch_payloadfor vector-store uploads. - Introduces a .NET
Microsoft.Agents.AI.AzureAI.ContentUnderstandingpackage with file-search backends, detection utilities, and extensive unit/integration tests plus runnable samples. - Updates dependency versions (Python + .NET) and adds a
.gitignorerule for_local_onlydirectories.
Reviewed changes
Copilot reviewed 84 out of 85 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/packages/azure-contentunderstanding/tests/cu/test_models.py | Updates model tests for result as string and new search_payload + include_fields behavior. |
| python/packages/azure-contentunderstanding/tests/cu/test_integration.py | Adjusts E2E assertions to validate rendered string output rather than dict markdown. |
| python/packages/azure-contentunderstanding/pyproject.toml | Bumps azure-ai-contentunderstanding dependency to 1.2.0 beta range. |
| python/packages/azure-contentunderstanding/agent_framework_azure_contentunderstanding/_models.py | Changes DocumentEntry.result type to string and adds search_payload; adds include_fields to FileSearchConfig. |
| python/packages/azure-contentunderstanding/agent_framework_azure_contentunderstanding/_extraction.py | Removes legacy extraction/formatting module in favor of SDK to_llm_input rendering. |
| python/packages/azure-contentunderstanding/agent_framework_azure_contentunderstanding/_context_provider.py | Switches rendering to to_llm_input, strips telemetry, and adds vector-store “body present” filtering. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/TestDoubles/SharedTestFixtures.cs | Adds shared CU model factories + agent/session stubs for unit tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/TestDoubles/FakeTokenCredential.cs | Adds a non-network TokenCredential test double. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/TestDoubles/FakeFileSearchBackend.cs | Adds a backend spy to assert upload/delete behavior in tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/TestDoubles/FakeAnalyzer.cs | Adds a configurable analysis pipeline stub for provider tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/TestDoubles/FakeAITool.cs | Adds minimal AITool stand-in for tool-forwarding assertions. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/TestDoubles/CountingClientFactory.cs | Adds client-factory counter to validate lazy init and concurrency behavior. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/RendererParityGapTests.cs | Adds renderer parity tests covering metadata, fields, warnings, and category cases. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ProviderStateTests.cs | Adds JSON round-trip tests for provider state objects. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ParityGapTests.cs | Adds provider parity tests for URL inputs, duplicates, session isolation, and multi-file file_search upload. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/OptionsTests.cs | Adds options validation/defaults tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ModelsTests.cs | Adds enum/flags shape tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/MimeSnifferTests.cs | Adds byte-signature MIME sniffing tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests.csproj | Adds the unit test project reference. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/FileSearchConfigFactoryTests.cs | Adds tests for FileSearchConfig factories and defaults. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ContextProviderTests.cs | Adds constructor/state key/dispose tests for provider shell. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ContextProviderPhase7Tests.cs | Adds tool-registration and tool-behavior tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ContextProviderPhase6Tests.cs | Adds background continuation + promotion tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/ContextProviderPhase5Tests.cs | Adds “happy path” provider tests (strip + inject + error handling). |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/AttachmentDetectorTests.cs | Adds attachment detection tests including sanitization/security hardening. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/AnalyzerSelectorTests.cs | Adds analyzer auto-selection tests. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/AnalysisRendererTests.cs | Adds renderer tests and telemetry stripping coverage + file_search payload behavior. |
| dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/AnalysisRendererSegmentsTests.cs | Adds multi-segment video/audio rendering + injection tests. |
| dotnet/tests/AzureAIContentUnderstanding.IntegrationTests/ContentUnderstandingLiveTests.cs | Adds live tests gated by environment variables. |
| dotnet/tests/AzureAIContentUnderstanding.IntegrationTests/AzureAIContentUnderstanding.IntegrationTests.csproj | Adds integration test project wiring and packages. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/README.md | Adds package README (quick start, samples, configuration, security notes). |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Models/DocumentStatus.cs | Adds document lifecycle enum. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Models/DocumentEntry.cs | Adds per-document state record for session persistence. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Models/AnalysisSection.cs | Adds flags enum controlling rendered output. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.csproj | Adds the new .NET CU integration project and references. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/ToolFactory.cs | Adds auto-registered tools (list_documents, get_analyzed_document). |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/MessageBuilder.cs | Adds message rebuild logic to strip binary attachments non-mutatingly. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/IContentUnderstandingClientFactory.cs | Adds internal seam for testable lazy client creation. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/FileSearchOutcome.cs | Adds internal “upload attempt outcome” struct. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/ContentUnderstandingProviderState.cs | Adds persisted per-session state (documents + injected keys). |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/BackgroundAnalysisRunner.cs | Adds background LRO continuation runner for timeouts. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/AnalysisRenderer.cs | Adds renderer wrapper around CU SDK LLM input helper with telemetry stripping. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/AnalysisOutcome.cs | Adds internal attempt result shape to unify completed/timeout/failure cases. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/AnalysisAttempt.cs | Adds internal representation of foreground attempt + continuation callback. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/AIContentReferenceEqualityComparer.cs | Adds reference-equality comparer for stripping exact attachment instances. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/FileSearch/OpenAIFileSearchBackend.cs | Adds OpenAI-backed file_search upload backend. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/FileSearch/OpenAICompatFileSearchBackendBase.cs | Adds shared OpenAI-compatible upload/poll/delete implementation. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/FileSearch/FoundryFileSearchBackend.cs | Adds Foundry-backed file_search upload backend. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/FileSearch/FileSearchConfig.cs | Adds configuration + factories for file_search integration. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/FileSearch/FileSearchBackend.cs | Adds abstraction for backend upload/delete operations. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Detection/MimeSniffer.cs | Adds byte-signature MIME detection. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Detection/AnalyzerSelector.cs | Adds media-type → analyzer-id mapping. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/ContentUnderstandingContextProviderOptions.cs | Adds options with required endpoint/credential + defaults. |
| dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/CHANGELOG.md | Adds initial changelog entry for the new package. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/README.md | Adds a README describing the CU sample suite and parity notes. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step08_DevUI_FileSearchFoundry/README.md | Adds DevUI Foundry file-search sample documentation. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step08_DevUI_FileSearchFoundry/Properties/launchSettings.json | Adds launch settings for DevUI Foundry file-search sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step08_DevUI_FileSearchFoundry/Program.cs | Adds DevUI Foundry file-search sample app. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step08_DevUI_FileSearchFoundry/AgentWithContentUnderstanding_Step08_DevUI_FileSearchFoundry.csproj | Adds project file for DevUI Foundry file-search sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step07_DevUI_FileSearchAzureOpenAI/README.md | Adds DevUI Azure OpenAI file-search sample documentation. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step07_DevUI_FileSearchAzureOpenAI/Properties/launchSettings.json | Adds launch settings for DevUI Azure OpenAI file-search sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step07_DevUI_FileSearchAzureOpenAI/Program.cs | Adds DevUI Azure OpenAI file-search sample app. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step07_DevUI_FileSearchAzureOpenAI/AgentWithContentUnderstanding_Step07_DevUI_FileSearchAzureOpenAI.csproj | Adds project file for DevUI Azure OpenAI file-search sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step06_DevUI_MultimodalAgent/README.md | Adds DevUI multimodal agent sample documentation. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step06_DevUI_MultimodalAgent/Properties/launchSettings.json | Adds launch settings for DevUI multimodal sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step06_DevUI_MultimodalAgent/Program.cs | Adds DevUI multimodal agent sample app. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step06_DevUI_MultimodalAgent/AgentWithContentUnderstanding_Step06_DevUI_MultimodalAgent.csproj | Adds project file for DevUI multimodal sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step05_LargeDocFileSearch/Program.cs | Adds script sample for Foundry vector store + file_search flow. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step05_LargeDocFileSearch/AgentWithContentUnderstanding_Step05_LargeDocFileSearch.csproj | Adds project file and invoice asset copy for step 05. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step04_InvoiceProcessing/Program.cs | Adds structured-fields-only invoice processing sample. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step04_InvoiceProcessing/AgentWithContentUnderstanding_Step04_InvoiceProcessing.csproj | Adds project file and invoice asset copy for step 04. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step03_MultimodalChat/Program.cs | Adds script sample for parallel PDF+audio+video analysis. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step03_MultimodalChat/AgentWithContentUnderstanding_Step03_MultimodalChat.csproj | Adds project file and invoice asset copy for step 03. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step02_MultiTurnSession/Program.cs | Adds script sample for session reuse across turns. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step02_MultiTurnSession/AgentWithContentUnderstanding_Step02_MultiTurnSession.csproj | Adds project file and invoice asset copy for step 02. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step01_DocumentQA/Program.cs | Adds simplest script sample for single-turn document Q&A. |
| dotnet/samples/02-agents/AgentWithContentUnderstanding/AgentWithContentUnderstanding_Step01_DocumentQA/AgentWithContentUnderstanding_Step01_DocumentQA.csproj | Adds project file and invoice asset copy for step 01. |
| dotnet/agent-framework-dotnet.slnx | Registers new CU project, tests, and samples in solution. |
| dotnet/Directory.Packages.props | Adds Azure.AI.ContentUnderstanding and bumps Azure.Core to 1.54.0. |
| dotnet/.gitignore | Ignores **/_local_only/ directories to prevent accidental commits. |
Comments suppressed due to low confidence (3)
dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Models/DocumentEntry.cs:1
- The remarks claim this record 'mirrors the Python provider's per-document state dict exactly', but the .NET type includes additional fields (e.g., MarkdownResult, VectorStoreFileId, SizeBytes) that are not present in the Python
DocumentEntry. Please update the documentation to either (a) state that it is a superset adapted for .NET scenarios, or (b) explicitly list/document the intentional divergences so consumers don’t assume wire-level parity.
dotnet/tests/Microsoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests/AnalysisRendererTests.cs:1 - This test hard-pins an assembly version contract for
LlmInputHelperthat may not be stable across package builds (assembly version can differ from NuGet package versioning and may change with servicing/rebuilds even when behavior is unchanged). Consider replacing this with a behavior-based compatibility test (preferred), or loosening/removing the assertion so dependency bumps don’t create noisy failures unrelated to actual CU rendering behavior.
dotnet/src/Microsoft.Agents.AI.AzureAI.ContentUnderstanding/Internal/ToolFactory.cs:1 - The
sectionparameter is anAnalysisSectionflags enum, but the implementation only special-casessection == AnalysisSection.Markdownand otherwise returns the fullResult. If a caller passesAnalysisSection.Fields(or other flag combinations) they may reasonably expect a fields-only rendering, but they’ll still receiveResult(which could include markdown+fields). Consider either validating/sanitizingsectionto a supported subset (e.g.,DefaultorMarkdownonly) and returning a clear error string for unsupported values, or storing/deriving additional pre-rendered variants to honorFieldsrequests.
…on-cu-to-llm-input-adoption # Conflicts: # python/packages/azure-contentunderstanding/pyproject.toml
…ings block Address PR microsoft#5796 review comment: the previous defensive scrubber ran a global regex substitution over the full rendered string, so any markdown body bullet shaped like '- LLMStats: ...' would also be silently deleted. Add a _strip_rai_telemetry helper that confines the substitution to the front-matter rai_warnings: YAML sub-block, leaving the body verbatim. Cover the new behavior with three tests (scoped strip, body preservation, and no-op branches).
changjian-wang
changed the title
# Conflicts: # dotnet/Directory.Packages.props # python/packages/azure-contentunderstanding/pyproject.toml
…rovider # Conflicts: # python/packages/azure-contentunderstanding/pyproject.toml
changjian-wang
temporarily deployed
to
integration
May 22, 2026 08:21 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
changjian-wang
temporarily deployed
to
integration
May 22, 2026 08:21 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Python Test Coverage Report •
Python Unit Test Overview
|
|||||||||||||||||||||||||||||||||||
… JSON ResponsesRawBase64Workaround.TryRewrite wrapped JsonDocument.Parse in try/catch(JsonException) -> return false, so a malformed /v1/responses body (the content-type header can lie) no longer 500s the request; the body is left untouched for the downstream endpoint to reject. A Try* method must not throw.
…jian-wang/agent-framework into dotnet/cu-context-provider
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.AzureAI.ContentUnderstanding\Microsoft.Agents.AI.AzureAI.ContentUnderstanding.csproj" /> |
Member
There was a problem hiding this comment.
We should reference published nuget here once we ship, instead of compiling from source
Member
There was a problem hiding this comment.
This comment applies to all samples in this PR.
Member
Author
There was a problem hiding this comment.
These samples reference the in-repo projects because the package isn't published to NuGet yet. We'll switch to a PackageReference once Microsoft.Agents.AI.AzureAI.ContentUnderstanding ships — leaving this open as a release-time follow-up.
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.AzureAI.ContentUnderstanding\Microsoft.Agents.AI.AzureAI.ContentUnderstanding.csproj" /> | ||
| <ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.Foundry\Microsoft.Agents.AI.Foundry.csproj" /> |
Member
There was a problem hiding this comment.
We should use nuget for Microsoft.Agents.AI.Foundry
Member
There was a problem hiding this comment.
This comment applies to all samples in this PR.
Member
Author
There was a problem hiding this comment.
These samples stay on ProjectReference because the CU package itself isn't published to NuGet yet, so they have to reference it in-repo. Microsoft.Agents.AI.Foundry is already published, but mixing an in-repo ProjectReference to CU (which pulls the in-repo Microsoft.Agents.AI.Abstractions) with a PackageReference to a released Foundry (which pulls a released Abstractions) would create a conflicting Abstractions version in the same dependency graph. We'll switch both to PackageReference together once the CU package ships.
|
|
||
| var app = builder.Build(); | ||
|
|
||
| // HACK: Microsoft.Agents.AI.Hosting.OpenAI's ItemContentConverter passes raw base64 from |
Member
There was a problem hiding this comment.
Hmmmm. We should find a better way to address this. We don't have this in the Python version
Member
Author
There was a problem hiding this comment.
This is a temporary shim for the OpenAI Responses hosting layer passing raw base64 with no filename. Agreed it's not ideal — I'd rather track a proper fix as a follow-up than block this PR on it. Leaving open.
|
|
||
| app.Run(); | ||
|
|
||
| /// <summary> |
Member
There was a problem hiding this comment.
It's concerning that we need to create our own class here. Let's discuss.
Member
Author
There was a problem hiding this comment.
The AzureOpenAIFileSearchBackend subclass exists because Azure OpenAI's /files endpoint rejects the user_data purpose that the stock FileSearchConfig.FromOpenAI uses (it returns Invalid value for "purpose"), so the sample overrides the upload purpose to Assistants to work against Azure OpenAI vector stores.
Agreed this exposes a gap — it'd be cleaner for the framework to provide an Azure-OpenAI-aware file-search config (e.g. FromAzureOpenAI) so samples don't have to hand-roll this. Whether that belongs in the library is worth discussing; leaving this as a follow-up until then.
| /// cancelled mid-flight (which would leave no operation to rehydrate and force a full re-upload | ||
| /// next turn). | ||
| /// </remarks> | ||
| public TimeSpan MaxWait { get; init; } = TimeSpan.FromSeconds(5); |
Member
There was a problem hiding this comment.
Let's discuss if we should allow this to be a nullable, so that a null value will indicate as wait until done. We should check if this fits .NET design pattern
Member
Author
There was a problem hiding this comment.
For now I'm keeping MaxWait as a non-nullable TimeSpan to avoid a breaking API change; the samples use Timeout.InfiniteTimeSpan to express "wait until done". Happy to revisit making it nullable as a separate API-design discussion — leaving open.
Comment on lines
+280
to
+315
| // Hosting wrappers (e.g. Microsoft.Agents.AI.Hosting.OpenAI's Responses ItemContentInputFile) | ||
| // attach the wire payload as DataContent.RawRepresentation but don't always propagate the | ||
| // "filename" field onto DataContent.Name. Recover it via duck-typed reflection so we don't take | ||
| // a hard dependency on the hosting package's internal types. | ||
| private static readonly ConcurrentDictionary<Type, Func<object, string?>?> s_rawFilenameAccessors = new(); | ||
|
|
||
| private static string? TryGetFilenameFromRawRepresentation(object? raw) | ||
| { | ||
| if (raw is null) | ||
| { | ||
| return null; | ||
| } | ||
|
|
||
| Func<object, string?>? accessor = s_rawFilenameAccessors.GetOrAdd(raw.GetType(), BuildRawFilenameAccessor); | ||
| return accessor?.Invoke(raw); | ||
| } | ||
|
|
||
| private static Func<object, string?>? BuildRawFilenameAccessor(Type type) | ||
| => BuildRawFilenameAccessorCore(type); | ||
|
|
||
| #if NET8_0_OR_GREATER | ||
| [UnconditionalSuppressMessage( | ||
| "Trimming", | ||
| "IL2070:'this' argument does not satisfy 'DynamicallyAccessedMembersAttribute' in call to target method.", | ||
| Justification = "RawRepresentation types come from upstream hosting/protocol packages (e.g. Microsoft.Agents.AI.Hosting.OpenAI's ItemContentInputFile) whose public Filename property has a stable, well-known name. Failure to resolve via reflection (e.g. under aggressive trimming) is non-fatal — caller falls back to Synthesize.")] | ||
| #endif | ||
| private static Func<object, string?>? BuildRawFilenameAccessorCore(Type type) | ||
| { | ||
| foreach (string name in new[] { "Filename", "FileName" }) | ||
| { | ||
| PropertyInfo? prop = type.GetProperty(name, BindingFlags.Public | BindingFlags.Instance); | ||
| if (prop is not null && prop.PropertyType == typeof(string) && prop.CanRead) | ||
| { | ||
| return instance => prop.GetValue(instance) as string; | ||
| } | ||
| } |
Member
There was a problem hiding this comment.
Suggestion: this looks like a bug or normalization gap in Microsoft.Agents.AI.Hosting.OpenAI.Responses.Converters.ItemContentConverter. ItemContentInputFile.Filename appears to be dropped when ItemContentInputFile -> DataContent is normalized, and this block is compensating for that later via RawRepresentation.
If that is the real issue, I would prefer to fix it at the source in ItemContentConverter by propagating Filename into the normalized DataContent / HostedFileContent, and then remove this fallback block once that fix exists. Right now this adds a fair amount of compatibility machinery here for a corner case: reflection, delegate caching, trimming suppression, and Filename / FileName heuristics.
I also could not find direct CU test coverage for this RawRepresentation filename path, while the OpenAI input conversion tests already verify filename propagation in other conversion paths. If the converter fix is out of scope for this PR, I would at least track it as a follow-up bug and avoid growing the workaround further here.
Member
Author
There was a problem hiding this comment.
Agreed — the cleaner fix is in ItemContentConverter to propagate Filename into the normalized DataContent, after which this fallback can be removed. I'll track it as a follow-up bug and keep this shim minimal (not growing it) for now. Leaving open.
Relocate CU samples to the package's own samples/ folder (01-get-started, 02-devui), mirroring the Python azure-contentunderstanding layout; isolate the sample build via a local Directory.Build.props + .editorconfig and exclude samples from the library compile/pack. Review fixes: samples throw instead of defaulting the model deployment name; script samples wait for CU completion (MaxWait=Timeout.InfiniteTimeSpan); AttachmentDetector BaseMediaType->GetBaseMediaType and the DataContent no-name fallback now uses a random id instead of hashing the full payload (mirrors Python derive_doc_key); MimeSniffer drops the private-repo doc reference.
Review: Azure Content Understanding Context ProviderGreat addition! The CU integration as an Observations
SuggestionA Solid work — this fills an important gap in the framework's context provider ecosystem! |
Resolve Directory.Packages.props conflicts by keeping the higher Azure.Core (1.59.0) and System.ClientModel (1.14.0) versions required by the CU dependency chain.
Contributor
|
@changjian-wang still working on this? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces Azure Content Understanding (CU) integration into the .NET agent framework, including new samples, supporting infrastructure, and dependency updates. The main changes add a new
Microsoft.Agents.AI.AzureAI.ContentUnderstandingproject, several multi-step agent samples demonstrating CU capabilities, and test coverage. Additionally, the Azure SDK dependencies are updated and a new.gitignorerule is added for local-only directories.Azure Content Understanding Integration:
Microsoft.Agents.AI.AzureAI.ContentUnderstandingproject to the solution for integrating Azure Content Understanding with agents.AzureAIContentUnderstanding.IntegrationTestsandMicrosoft.Agents.AI.AzureAI.ContentUnderstanding.UnitTests. [1] [2]Samples and Documentation:
samples/02-agents/AgentWithContentUnderstanding, covering document Q&A, multi-turn sessions, multimodal chat, invoice processing, large document search, and DevUI scenarios. Each step includes both a.csprojand aProgram.csdemonstrating real-world agent interactions with CU. [1] [2] [3] [4] [5]Dependency and Infrastructure Updates:
Directory.Packages.propsto include the newAzure.AI.ContentUnderstandingpackage and bumpedAzure.Coreto version 1.54.0..gitignorerule to exclude all_local_onlydirectories from source control, preventing accidental commits of local temp files.