Enable A365 tracing and fix W3C baggage propagation in agentserver by singankit · Pull Request #46754 · Azure/azure-sdk-for-python

singankit · 2026-05-06T14:50:46Z

Summary

Enable Agent365 (A365) tracing in the agentserver packages and fix W3C baggage propagation so that incoming baggage entries (e.g. user.id) are visible to span processors on all spans.

Changes

A365 Tracing Enablement (agentserver-core)

Gate A365 export behind FOUNDRY_AGENT365_TRACING_ENABLED env var
Add agent identity resolvers (agent_id, blueprint_id, tenant_id) from env vars
Enable a365_enable_observability_exporter and a365_observability_scope_override
Wire through _FoundryEnrichmentSpanProcessor with span attributes

Streaming Context Fix (responses)

Capture full OTel context (span + baggage) at wrap time in _wrap_streaming_response
Re-attach during async iteration so baggage is available after the handler's finally block

W3C Baggage Propagation Fix (responses + invocations)

Use W3CBaggagePropagator().extract() to extract only baggage from incoming headers
Merge extracted baggage onto get_current() before adding server entries
This preserves span parent-child relationships while capturing incoming baggage like user.id

Tests

3 baggage propagation tests for responses package
3 baggage propagation tests for invocations package
Coverage: baggage merging, span parenting preserved, empty header safety

Root Cause

start_as_current_span(context=extracted_ctx) inside request_span makes the span current, but baggage from the extracted context does not survive through the contextmanager yield boundary to get_current() as seen by the endpoint handler. Only entries explicitly added after get_current() survive. The fix extracts baggage separately at the endpoint handler level.

Conditionally enable A365 observability export via microsoft-opentelemetry distro when both FOUNDRY_HOSTING_ENVIRONMENT and FOUNDRY_AGENT365_TRACING_ENABLED env vars are set. Uses S2S endpoint for token resolution in hosted environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…hment - Add resolve_agent_id() with FOUNDRY_AGENT_INSTANCE_CLIENT_ID env var (falls back to name:version or name) - Add resolve_agent_blueprint_id() with FOUNDRY_AGENT_BLUEPRINT_CLIENT_ID - Add resolve_agent_tenant_id() with FOUNDRY_AGENT_TENANT_ID - Wire all three through _FoundryEnrichmentSpanProcessor - Make processor __init__ keyword-only Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ator The streaming async generator runs after the request handler's finally block detaches baggage. Fix by capturing the full OTel context (including baggage) at wrap time and re-attaching it during iteration, so child spans created during streaming can see baggage entries like conversation_id. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extract incoming baggage (e.g. user.id) using W3CBaggagePropagator without re-extracting traceparent, preserving parent-child span relationships while making caller's baggage entries visible to downstream span processors. Also removes stale flask/sqlalchemy imports from prior attempts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…kages - Apply same baggage extraction fix to invocations/_invocation.py - Add 3 baggage propagation tests for invocations package - Add 3 baggage propagation tests for responses package - Tests verify: baggage merging, span parenting preserved, empty header safety Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Server-added entries (response_id) are set after span starts, so on_start processor won't see them. Test should only verify incoming baggage merging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ponse' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…an start Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Thread enable_sensitive_data kwarg from AgentServerHost through configure_observability -> _configure_tracing -> _setup_distro_export -> use_microsoft_opentelemetry so Agent Framework SDK records prompts, tool arguments, and results. Defaults to True; set FOUNDRY_ENABLE_SENSITIVE_DATA=false to opt out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add _ATTR_FOUNDRY_AGENT_TYPE constant - Set agent_type='hosted' when FOUNDRY_HOSTING_ENVIRONMENT is set - Only write attribute on spans with gen_ai.operation.name == invoke_agent - Add 3 tests for agent_type scoping behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace request_span() with request_context() that extracts and attaches incoming W3C trace context (traceparent/tracestate/baggage) without creating a span. Framework spans created inside handlers are now parented directly under the caller's span. Changes: - core/_tracing.py: Add request_context(), remove request_span() - core/_base.py: Simplify AgentServerHost.request_context() wrapper - invocations/_invocation.py: Remove span creation/attrs/end logic - responses/_endpoint_handler.py: Same simplification - Remove agent_type from enrichment processor (no invoke_agent span) - Update all tests to validate context propagation without server span Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replaces the weak status-code-only assertion with a test that creates a span inside the handler and verifies trace ID and parent span ID match the incoming traceparent header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The request_context method was added in 2.0.0b4 (as part of the invoke_agent span removal). Update invocations and responses packages to require the correct minimum version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Revert min dependency back to >=2.0.0b3 and add hasattr guards so that invocations/responses gracefully degrade when running against core 2.0.0b3 (which lacks request_context). This fixes the mindependency CI check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Creates a real OTel caller span, injects its trace context into the request headers, creates a child span in the invocation handler, and validates the handler span is correctly parented under the caller. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…dd baggage tests - Add invocation_id baggage-to-span-attribute mapping in _FoundryEnrichmentSpanProcessor.on_start - Add core tests for invocation_id enrichment (from baggage, no baggage, child propagation) - Add invocations test verifying SDK-set baggage (invocation_id, session_id) available in handler - Add responses test verifying SDK-set baggage (response_id, conversation_id, streaming) available in handler - Add invocations integration test verifying baggage entries stamped as span attributes via enricher Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tate In CI environments where microsoft-opentelemetry distro is installed and APPLICATIONINSIGHTS_CONNECTION_STRING is set, non-tracing tests would trigger use_microsoft_opentelemetry() on the first server construction, installing a global TracerProvider that breaks traceparent-propagation tests. Fix: - Add session-scoped _prevent_distro_setup fixture in both invocations and responses conftest.py that mocks _setup_distro_export for all tests - Pass configure_observability=None in conftest factory functions - Pass configure_observability=None in test_tracing_disabled_by_default and test_no_tracing_when_no_endpoints Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace synthetic traceparent string with real OTel span + inject() pattern. This ensures correct trace context propagation regardless of which TracerProvider or auto-instrumentation (e.g. microsoft-opentelemetry) is active in the CI environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the Starlette OTel instrumentor (which created noisy SERVER and ASGI internal spans) and replace with a lightweight TraceContextMiddleware that only extracts W3C traceparent/tracestate/baggage from incoming requests. This ensures downstream spans (from MAF/agent-framework) are children of the caller's trace without creating extra spans. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

TraceContextTextMapPropagator was not importable from opentelemetry.trace.propagation. Use the global propagate.extract() instead which handles both TraceContext and Baggage propagation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

No longer needed since we replaced StarletteInstrumentor with our own lightweight TraceContextMiddleware. Fixes CI 'Analyze dependencies' failure (dependency not in shared_requirements.txt). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The _prevent_distro_setup fixture was blocking the Azure Monitor exporter for ALL tests including E2E ones. Now checks the marker expression and yields without patching when tracing_e2e tests are selected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

TraceContextMiddleware now propagates W3C trace context automatically at the middleware layer, so handlers no longer need to call request_context(). The method was removed from AgentServerHost in this branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

NonRecordingSpan does not expose a public .context attribute in all OpenTelemetry versions. Use get_span_context() which is the stable API that works for all span types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Verifies that when an incoming request has NO traceparent/tracestate/baggage headers (e.g. health checks, direct calls), spans created by downstream frameworks like MAF are still properly exported to App Insights as new traces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Instead of matching by span name, capture the exact span_id and trace_id from the created span and query App Insights by those IDs for precise correlation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This test is a duplicate of test_span_parenting_in_appinsights and test_span_emitted_without_incoming_trace_context. It fails intermittently because it runs first and App Insights ingestion delay is longer for the initial telemetry session. The same validation is covered by the later tests which pass reliably. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Same App Insights ingestion timing issue as core — first test to run suffers from cold-start delay > 300s. The span parenting test already validates span export AND parent-child relationships (superset). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

App Insights has a cold-start ingestion delay for the first telemetry session sent to a resource — data can take 5+ minutes to become queryable via KQL. This caused the first E2E test to always fail in CI. Fix: Add a session-scoped autouse fixture that sends a dummy span and polls until App Insights confirms ingestion (up to 360s). Real tests then run against a 'warm' pipeline with fast ingestion. Also restores test_handler_span_in_appinsights which is no longer flaky. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- azure-ai-agentserver-core: 2.0.0b4 -> 2.0.0b5 - azure-ai-agentserver-invocations: 1.0.0b4 -> 1.0.0b5 - azure-ai-agentserver-responses: 1.0.0b6 -> 1.0.0b7 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merged new release notes into existing version entries (b4/b4/b6) instead of creating separate b5/b5/b7 entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…6973) Aligns the invocations_ws transport with the new tracing model on feature/enable-a365-tracing: trace context propagation is handled by the core TraceContextMiddleware, and user-created spans inside handlers are correctly parented without a framework-generated root span. - _invocation_ws.py: remove the per-connection 'websocket_session' span (request_span call, otel_span, _safe_set_attrs helper, end_span import, span_ctx finalize block). Drop the dead 'handler_exc' kwarg on _finalize_session and the 'error_message' kwarg on _emit_close_event (handler exception text never reaches the close-event log line, by design). - _constants.py: drop unused ATTR_SPAN_ERROR_MESSAGE; relabel the section comment from 'Span attribute keys' to 'Structured-log extra keys'. - CHANGELOG.md: trim the WS telemetry feature bullet to describe the log-only path; no breaking-change entry (the WS span was added and removed within the same unreleased 1.0.0b4). - tests/test_ws_tracing.py: removed (17 span-assertion tests obsolete; close-event log line still covered by tests/test_ws_close_event.py). 186 tests pass, 2 skipped. Co-authored-by: Yulin Li <yulili@microsoft.com>

stale review

…zure#46754) * Enable A365 tracing in agentserver-core when hosted Conditionally enable A365 observability export via microsoft-opentelemetry distro when both FOUNDRY_HOSTING_ENVIRONMENT and FOUNDRY_AGENT365_TRACING_ENABLED env vars are set. Uses S2S endpoint for token resolution in hosted environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add agent_id, blueprint_id, and tenant_id resolution to tracing enrichment - Add resolve_agent_id() with FOUNDRY_AGENT_INSTANCE_CLIENT_ID env var (falls back to name:version or name) - Add resolve_agent_blueprint_id() with FOUNDRY_AGENT_BLUEPRINT_CLIENT_ID - Add resolve_agent_tenant_id() with FOUNDRY_AGENT_TENANT_ID - Wire all three through _FoundryEnrichmentSpanProcessor - Make processor __init__ keyword-only Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enable a365_enable_observability_exporter in A365 tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add a365_observability_scope_override to A365 tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix streaming context: capture full context (span + baggage) for iterator The streaming async generator runs after the request handler's finally block detaches baggage. Fix by capturing the full OTel context (including baggage) at wrap time and re-attaching it during iteration, so child spans created during streaming can see baggage entries like conversation_id. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix baggage propagation: extract only W3C baggage from request headers Extract incoming baggage (e.g. user.id) using W3CBaggagePropagator without re-extracting traceparent, preserving parent-child span relationships while making caller's baggage entries visible to downstream span processors. Also removes stale flask/sqlalchemy imports from prior attempts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix W3C baggage propagation in invocations and add tests for both packages - Apply same baggage extraction fix to invocations/_invocation.py - Add 3 baggage propagation tests for invocations package - Add 3 baggage propagation tests for responses package - Tests verify: baggage merging, span parenting preserved, empty header safety Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test: remove assertion for server-added baggage at span start time Server-added entries (response_id) are set after span starts, so on_start processor won't see them. Test should only verify incoming baggage merging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test: use correct span name 'invoke_agent' instead of 'create_response' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix invocations test: remove assertion for server-added baggage at span start Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add bkey/bval to local cspell ignore lists Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unused imports detach_context and set_current_span Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add enable_sensitive_data param to configure_observability Thread enable_sensitive_data kwarg from AgentServerHost through configure_observability -> _configure_tracing -> _setup_distro_export -> use_microsoft_opentelemetry so Agent Framework SDK records prompts, tool arguments, and results. Defaults to True; set FOUNDRY_ENABLE_SENSITIVE_DATA=false to opt out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test assertions to include enable_sensitive_data param Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add microsoft.foundry.agent.type attribute scoped to invoke_agent spans - Add _ATTR_FOUNDRY_AGENT_TYPE constant - Set agent_type='hosted' when FOUNDRY_HOSTING_ENVIRONMENT is set - Only write attribute on spans with gen_ai.operation.name == invoke_agent - Add 3 tests for agent_type scoping behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove invoke_agent SERVER span, keep W3C context propagation Replace request_span() with request_context() that extracts and attaches incoming W3C trace context (traceparent/tracestate/baggage) without creating a span. Framework spans created inside handlers are now parented directly under the caller's span. Changes: - core/_tracing.py: Add request_context(), remove request_span() - core/_base.py: Simplify AgentServerHost.request_context() wrapper - invocations/_invocation.py: Remove span creation/attrs/end logic - responses/_endpoint_handler.py: Same simplification - Remove agent_type from enrichment processor (no invoke_agent span) - Update all tests to validate context propagation without server span Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add proper span parenting test for responses package Replaces the weak status-code-only assertion with a test that creates a span inside the handler and verifies trace ID and parent span ID match the incoming traceparent header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump azure-ai-agentserver-core min dependency to >=2.0.0b4 The request_context method was added in 2.0.0b4 (as part of the invoke_agent span removal). Update invocations and responses packages to require the correct minimum version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make request_context backward-compatible with core 2.0.0b3 Revert min dependency back to >=2.0.0b3 and add hasattr guards so that invocations/responses gracefully degrade when running against core 2.0.0b3 (which lacks request_context). This fixes the mindependency CI check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add e2e span parenting test with real caller span Creates a real OTel caller span, injects its trace context into the request headers, creates a child span in the invocation handler, and validates the handler span is correctly parented under the caller. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stamp invocation_id on spans via FoundryEnrichmentSpanProcessor and add baggage tests - Add invocation_id baggage-to-span-attribute mapping in _FoundryEnrichmentSpanProcessor.on_start - Add core tests for invocation_id enrichment (from baggage, no baggage, child propagation) - Add invocations test verifying SDK-set baggage (invocation_id, session_id) available in handler - Add responses test verifying SDK-set baggage (response_id, conversation_id, streaming) available in handler - Add invocations integration test verifying baggage entries stamped as span attributes via enricher Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test failures: prevent OTel distro from contaminating global state In CI environments where microsoft-opentelemetry distro is installed and APPLICATIONINSIGHTS_CONNECTION_STRING is set, non-tracing tests would trigger use_microsoft_opentelemetry() on the first server construction, installing a global TracerProvider that breaks traceparent-propagation tests. Fix: - Add session-scoped _prevent_distro_setup fixture in both invocations and responses conftest.py that mocks _setup_distro_export for all tests - Pass configure_observability=None in conftest factory functions - Pass configure_observability=None in test_tracing_disabled_by_default and test_no_tracing_when_no_endpoints Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use inject(headers) in traceparent test for CI reliability Replace synthetic traceparent string with real OTel span + inject() pattern. This ensures correct trace context propagation regardless of which TracerProvider or auto-instrumentation (e.g. microsoft-opentelemetry) is active in the CI environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: use OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT for sensitive data Replace FOUNDRY_ENABLE_SENSITIVE_DATA with the standard OpenTelemetry GenAI semantic convention env var for controlling sensitive data capture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use inject(headers) in invocations traceparent tests for CI reliability Replace synthetic traceparent strings with real OTel span + inject() pattern in both streaming and non-streaming span parenting tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test_incoming_baggage_stamped_on_handler_spans for CI Rewrite the enrichment processor test to run in isolation without TestClient/ASGI, avoiding CI-specific context propagation differences. The full baggage flow through the invocations server is already covered by test_sdk_set_baggage_available_in_handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pylint errors: line-too-long and unused imports - core/_base.py: break long line for env var read - invocations/_invocation.py: remove unused StreamingResponse import - responses/_endpoint_handler.py: remove unused RequestValidationError and build_create_otel_attrs imports, break long context manager line Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify request_context() and add debug logging - Pass headers directly to propagator instead of using _extract_w3c_carrier - Remove _extract_w3c_carrier helper and _W3C_HEADERS constant - Add debug log for attached span context (type, trace_id, trace_flags) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix debug log: use get_current_span() instead of non-existent get_span_from_context() Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * debug: log raw trace headers and full post-attach span context Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * debug: create SERVER span in request_context for downstream instrumentors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * debug: test span without explicit parent context Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * debug: re-enable context=ctx on blah span to parent under remote trace Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add Starlette OTel instrumentation for automatic HTTP SERVER spans Replaces the manual span with proper Starlette instrumentation that creates a SERVER span per request with trace context propagation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: replace request_context with BaggageMiddleware + Starlette instrumentor Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Suppress noisy ASGI receive/send internal spans Patch OpenTelemetryMiddleware.__init__ to set exclude_receive_span and exclude_send_span to True, suppressing the per-event INTERNAL spans (http receive / http send) that the Starlette OTel instrumentor creates. The upstream ASGI middleware already supports these attributes via its exclude_spans constructor parameter, but the Starlette instrumentor does not expose them yet (tracked in opentelemetry-python-contrib#3725). The monkeypatch can be removed once upstream adds constructor support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Test: remove BaggageMiddleware to isolate NonRecordingSpan crash Temporarily remove BaggageMiddleware from the middleware stack to test whether its context.attach() call is causing the NonRecordingSpan crash in azure-ai-projects _responses_instrumentor. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace Starlette instrumentor with TraceContextMiddleware Remove the Starlette OTel instrumentor (which created noisy SERVER and ASGI internal spans) and replace with a lightweight TraceContextMiddleware that only extracts W3C traceparent/tracestate/baggage from incoming requests. This ensures downstream spans (from MAF/agent-framework) are children of the caller's trace without creating extra spans. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix TraceContextMiddleware: use opentelemetry.propagate.extract TraceContextTextMapPropagator was not importable from opentelemetry.trace.propagation. Use the global propagate.extract() instead which handles both TraceContext and Baggage propagation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unused opentelemetry-instrumentation-starlette dependency No longer needed since we replaced StarletteInstrumentor with our own lightweight TraceContextMiddleware. Fixes CI 'Analyze dependencies' failure (dependency not in shared_requirements.txt). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pylint: trailing newline and unused contextlib import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix E2E tracing test: skip distro mock when running with -m tracing_e2e The _prevent_distro_setup fixture was blocking the Azure Monitor exporter for ALL tests including E2E ones. Now checks the marker expression and yields without patching when tracing_e2e tests are selected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix core E2E tests: remove request_context() calls TraceContextMiddleware now propagates W3C trace context automatically at the middleware layer, so handlers no longer need to call request_context(). The method was removed from AgentServerHost in this branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix E2E test: use get_span_context() instead of .context NonRecordingSpan does not expose a public .context attribute in all OpenTelemetry versions. Use get_span_context() which is the stable API that works for all span types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add E2E test: framework spans emitted without incoming trace context Verifies that when an incoming request has NO traceparent/tracestate/baggage headers (e.g. health checks, direct calls), spans created by downstream frameworks like MAF are still properly exported to App Insights as new traces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use span_id and trace_id to validate span in App Insights Instead of matching by span name, capture the exact span_id and trace_id from the created span and query App Insights by those IDs for precise correlation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove flaky test_child_span_in_appinsights (redundant) This test is a duplicate of test_span_parenting_in_appinsights and test_span_emitted_without_incoming_trace_context. It fails intermittently because it runs first and App Insights ingestion delay is longer for the initial telemetry session. The same validation is covered by the later tests which pass reliably. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove flaky test_handler_span_in_appinsights from invocations E2E Same App Insights ingestion timing issue as core — first test to run suffers from cold-start delay > 300s. The span parenting test already validates span export AND parent-child relationships (superset). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add App Insights warm-up fixture for E2E tracing tests App Insights has a cold-start ingestion delay for the first telemetry session sent to a resource — data can take 5+ minutes to become queryable via KQL. This caused the first E2E test to always fail in CI. Fix: Add a session-scoped autouse fixture that sends a dummy span and polls until App Insights confirms ingestion (up to 360s). Real tests then run against a 'warm' pipeline with fast ingestion. Also restores test_handler_span_in_appinsights which is no longer flaky. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix pylint: remove trailing newline in _tracing.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cspell: add markexpr and sess to ignore list Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CHANGELOGs for tracing middleware changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CHANGELOGs with comprehensive PR changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump package versions to match unreleased changelog entries - azure-ai-agentserver-core: 2.0.0b4 -> 2.0.0b5 - azure-ai-agentserver-invocations: 1.0.0b4 -> 1.0.0b5 - azure-ai-agentserver-responses: 1.0.0b6 -> 1.0.0b7 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate changelog entries under current versions as Unreleased Merged new release notes into existing version entries (b4/b4/b6) instead of creating separate b5/b5/b7 entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop framework-level OpenTelemetry span for WebSocket connections (Azure#46973) Aligns the invocations_ws transport with the new tracing model on feature/enable-a365-tracing: trace context propagation is handled by the core TraceContextMiddleware, and user-created spans inside handlers are correctly parented without a framework-generated root span. - _invocation_ws.py: remove the per-connection 'websocket_session' span (request_span call, otel_span, _safe_set_attrs helper, end_span import, span_ctx finalize block). Drop the dead 'handler_exc' kwarg on _finalize_session and the 'error_message' kwarg on _emit_close_event (handler exception text never reaches the close-event log line, by design). - _constants.py: drop unused ATTR_SPAN_ERROR_MESSAGE; relabel the section comment from 'Span attribute keys' to 'Structured-log extra keys'. - CHANGELOG.md: trim the WS telemetry feature bullet to describe the log-only path; no breaking-change entry (the WS span was added and removed within the same unreleased 1.0.0b4). - tests/test_ws_tracing.py: removed (17 span-assertion tests obsolete; close-event log line still covered by tests/test_ws_close_event.py). 186 tests pass, 2 skipped. Co-authored-by: Yulin Li <yulili@microsoft.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Yulin Li <liyulin@pku.edu.cn> Co-authored-by: Yulin Li <yulili@microsoft.com>

singankit and others added 7 commits May 4, 2026 13:27

Enable a365_enable_observability_exporter in A365 tracing config

8db62c6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add a365_observability_scope_override to A365 tracing config

d8bb33e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

singankit marked this pull request as ready for review May 6, 2026 14:51

singankit requested a review from ankitbko as a code owner May 6, 2026 14:51

Copilot AI review requested due to automatic review settings May 6, 2026 14:51

singankit requested review from RaviPidaparthi and vangarp as code owners May 6, 2026 14:51

github-actions Bot added the Hosted Agents sdk/agentserver/* label May 6, 2026

Copilot started reviewing on behalf of singankit May 6, 2026 14:52 View session

singankit and others added 16 commits May 6, 2026 09:50

Fix test: use correct span name 'invoke_agent' instead of 'create_res…

ce9ae4f

…ponse' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix invocations test: remove assertion for server-added baggage at sp…

dab1c87

…an start Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add bkey/bval to local cspell ignore lists

078502d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove unused imports detach_context and set_current_span

3cda909

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix test assertions to include enable_sensitive_data param

36b9d38

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

singankit and others added 11 commits May 15, 2026 23:56

Fix pylint: trailing newline and unused contextlib import

7d73600

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use span_id and trace_id to validate span in App Insights

3fcfcbf

Instead of matching by span name, capture the exact span_id and trace_id from the created span and query App Insights by those IDs for precise correlation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

singankit force-pushed the feature/enable-a365-tracing branch from 7a54fd0 to 52a9cce Compare May 17, 2026 07:54

singankit force-pushed the feature/enable-a365-tracing branch from 52a9cce to b7852ca Compare May 17, 2026 22:39

singankit and others added 5 commits May 17, 2026 15:43

Fix pylint: remove trailing newline in _tracing.py

26b4949

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix cspell: add markexpr and sess to ignore list

b88a09f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update CHANGELOGs for tracing middleware changes

5316153

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update CHANGELOGs with comprehensive PR changes

48d8f65

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bump package versions to match unreleased changelog entries

249fd79

- azure-ai-agentserver-core: 2.0.0b4 -> 2.0.0b5 - azure-ai-agentserver-invocations: 1.0.0b4 -> 1.0.0b5 - azure-ai-agentserver-responses: 1.0.0b6 -> 1.0.0b7 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ankitbko requested changes May 18, 2026

View reviewed changes

Comment thread sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md

singankit and others added 2 commits May 18, 2026 10:10

Consolidate changelog entries under current versions as Unreleased

acd4f55

Merged new release notes into existing version entries (b4/b4/b6) instead of creating separate b5/b5/b7 entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge main and resolve CHANGELOG conflict

22ab1ae

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ankitbko approved these changes May 19, 2026

View reviewed changes

yulin-li and others added 2 commits May 19, 2026 10:25

Merge branch 'main' into feature/enable-a365-tracing

d7b185f

ankitbko approved these changes May 19, 2026

View reviewed changes

ankitbko merged commit 74fb2fe into main May 19, 2026
34 checks passed

ankitbko deleted the feature/enable-a365-tracing branch May 19, 2026 04:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable A365 tracing and fix W3C baggage propagation in agentserver - #46754

Enable A365 tracing and fix W3C baggage propagation in agentserver#46754
ankitbko merged 59 commits into
mainfrom
feature/enable-a365-tracing

singankit commented May 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

singankit commented May 6, 2026

Summary

Changes

A365 Tracing Enablement (agentserver-core)

Streaming Context Fix (responses)

W3C Baggage Propagation Fix (responses + invocations)

Tests

Root Cause

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants