Agent Observability Showcase

A production-ready FastAPI application demonstrating comprehensive OpenTelemetry integration with Azure Application Insights for AI agent workflows.

• Azure AI Foundry Integration: Real GPT-4.1 agent with automatic function calling
• Three-Decorator Pattern: Simple, reusable decorators for complete instrumentation
• Test Data Generator: Multiple scenarios for validating dashboards (normal, errors, spikes, load)
• Azure CLI Provisioning: Idempotent infrastructure setup without Terraform

How It Works

---

Observability Stack

This application implements a three-layer observability architecture:

┌─────────────────────────────────────────────────────────────┐
│ Layer 1: Automatic Instrumentation (Decorators)             │
├─────────────────────────────────────────────────────────────┤
│ @trace_agent_step    → Creates spans for workflow steps     │
│ @track_tokens        → Records token usage & costs          │
│ @track_tool_execution → Monitors tool performance           │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Layer 2: OpenTelemetry Signals                              │
├─────────────────────────────────────────────────────────────┤
│ Traces    → End-to-end request flow with nested spans       │
│ Metrics   → Counters & histograms (tokens, requests, etc)   │
│ Logs      → Structured JSON with trace correlation          │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Layer 3: Azure Application Insights                         │
├─────────────────────────────────────────────────────────────┤
│ Live Metrics → Real-time telemetry dashboard                │
│ Traces       → Distributed tracing & span details           │
│ Performance  → Request duration, dependencies               │
│ Failures     → Error analysis & stack traces                │
└─────────────────────────────────────────────────────────────┘

Agent Workflow

The SimpleAgent demonstrates telemetry across a complete AI workflow:

1. Planning (@trace_agent_step) - Analyzes query, decides which tools to use
2. Tool Execution (@track_tool_execution) - Runs selected tools (web_search, calculator, weather)
3. Response Generation (@track_tokens) - Generates final response with GPT-4.1

Each step is automatically instrumented with:

• Distributed tracing for request correlation
• Token metrics for cost tracking (input/output tokens, total cost)
• Performance metrics for latency analysis
• Structured logs with trace IDs for debugging

Key Components

• src/main.py: FastAPI app with OpenTelemetry initialization
• src/agent.py: Azure AI Foundry agent with three mock tools
• src/backend/core/telemetry.py: OpenTelemetry setup with Azure Monitor
• src/backend/core/telemetry_decorators.py: Three reusable decorators for instrumentation
• src/backend/core/telemetry_api_dependencies.py: FastAPI dependencies for API metrics
• src/backend/core/logging.py: Structured JSON logging with trace correlation
• scripts/generate_test_telemetry.py: Test data generator with multiple scenarios

How Telemetry Flows

Here's what happens when a user makes a request:

1. User Request → POST /api/chat {"query": "What's the weather in Seattle?"}
   ↓
2. FastAPI Middleware → APITelemetry dependency injects telemetry context
   ↓
3. Agent.chat() → @trace_agent_step creates root span "agent.chat"
   ↓
4. Agent.plan() → @trace_agent_step creates child span "agent.plan"
   - Analyzes query, decides to use "weather" tool
   - Structured log: {"message": "Planning agent workflow", "trace_id": "abc123"}
   ↓
5. Agent.execute_tool("weather") → @track_tool_execution creates child span
   - Records metric: tool.execution.duration = 45ms
   - Structured log: {"message": "Tool executed", "tool": "weather", "status": "success"}
   ↓
6. Agent.generate_response() → @track_tokens creates child span
   - Records metric: gen_ai.client.token.usage (input: 1200, output: 400)
   - Calculates cost: $0.003 (based on GPT-4.1 pricing)
   - Structured log: {"message": "Response generated", "tokens": 1600, "cost": 0.003}
   ↓
7. Response → Returns to user, APITelemetry records success metric
   ↓
8. Azure Monitor → All spans, metrics, logs exported within 2-5 minutes

Result: Complete visibility into the entire request flow with:

• 4 spans showing parent-child relationships
• 3 metrics tracking tokens, tool execution, API requests
• 6 structured logs with correlated trace IDs

Quick Start

Prerequisites

• Python 3.13+
• Azure CLI
• Azure subscription
• Poetry (for dependency management)

Or open project in GitHub Codespaces for instant setup or use Dev Containers in VS Code.

1. Provision Azure Infrastructure

The script will create:

• Resource Group
• Log Analytics Workspace
• Application Insights
• Azure AI Foundry resource (optional)
• GPT-4.1 deployment (optional)

make provision

Follow the interactive prompts or press Enter to accept defaults:

• Project name: agent-observability
• Region: uaenorth
• Environment: dev

The script will generate a .env file with the connection string.

2. Configure AI Foundry

Update .env with your Azure AI Foundry configuration:

AI_FOUNDRY_NAME=your-ai-foundry-name
AI_FOUNDRY_PROJECT_NAME=your-project-name
GPT_4_1_DEPLOYMENT_NAME=your-gpt4-deployment

Note: The agent requires Azure AI Foundry to be configured to function properly.

3. Install Dependencies

make restore

4. Run the API

make run-api

The API will be available at http://localhost:8000

5. Test the Agent

# Full agent workflow
curl -X POST http://localhost:8000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the weather in Seattle?"}'

# Planning step only
curl -X POST http://localhost:8000/api/plan \
  -H "Content-Type: application/json" \
  -d '{"query": "Calculate 25 * 4"}'

# Health check
curl http://localhost:8000/health

6. Generate Test Telemetry

# Mixed scenarios (60 seconds)
make test-telemetry

# Specific scenarios
make test-telemetry-normal    # Normal operations (120s)
make test-telemetry-errors    # High error rate (60s)
make test-telemetry-spikes    # Latency spikes (90s)
make test-telemetry-load      # Load test (5 min, high volume)

7. View in Azure Portal

Data appears in Azure Portal within 2-5 minutes:

1. Navigate to Azure Portal
2. Open your Application Insights resource
3. View:
   • Live Metrics - Real-time telemetry
   • Transaction Search - Individual traces
   • Performance - Operation metrics
   • Failures - Error analysis

Configuration

Environment Variables

See .env.example for all configuration options:

# Azure Application Insights (Required)
APPLICATIONINSIGHTS_CONNECTION_STRING=

# Azure AI Foundry (Required)
AI_FOUNDRY_NAME=
AI_FOUNDRY_PROJECT_NAME=
GPT_4_1_DEPLOYMENT_NAME=

# Logging
LOG_LEVEL=INFO

# Service Configuration
ENVIRONMENT=dev
SERVICE_NAME=agent-observability-api

Capture Message Content (Optional)

To capture full message content in traces:

export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

Note: This environment variable is read by OpenTelemetry SDK at import time.

Reference: Azure AI Foundry Tracing

Project Structure

agent-observability/
├── src/
│   ├── main.py                           # FastAPI application
│   ├── agent.py                          # Simple agent implementation
│   └── backend/core/
│       ├── config.py                     # Settings and configuration
│       ├── telemetry.py                  # OpenTelemetry setup
│       ├── telemetry_decorators.py       # Instrumentation decorators
│       ├── telemetry_api_dependencies.py # FastAPI dependencies
│       └── logging.py                    # Structured JSON logging
├── scripts/
│   ├── provision.sh                      # Azure infrastructure provisioning
│   ├── generate_test_telemetry.py        # Test data generator
│   ├── restore.sh                        # Dependency installation
│   └── utils.sh                          # Utility functions
├── pyproject.toml                        # Dependencies and tooling config
├── Makefile                              # Development commands
├── .env.example                          # Environment variable template
└── README.md                             # This file

Learning Resources

1. Azure AI Foundry Tracing: https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/trace-application?view=foundry-classic#enable-tracing-in-your-project
2. Application Insights Agents View: https://learn.microsoft.com/en-gb/azure/azure-monitor/app/agents-view
3. OpenTelemetry Gen AI Semantics: https://opentelemetry.io/docs/specs/semconv/gen-ai/

OpenTelemetry Decorators

The project includes three main decorators for automatic instrumentation:

@trace_agent_step(operation_name)

Traces agent workflow steps with automatic span creation.

from src.backend.core.telemetry_decorators import trace_agent_step

@trace_agent_step("agent.plan")
async def plan(self, query: str) -> dict:
    # Automatically traced with span "agent.plan"
    return {"tools": ["web_search"], "reasoning": "..."}

@track_tool_execution(tool_name)

Monitors tool execution with metrics and logging.

from src.backend.core.telemetry_decorators import track_tool_execution

@track_tool_execution(tool_name="web_search")
async def web_search(self, query: str) -> str:
    # Automatic instrumentation for tool calls
    return "search results..."

@track_tokens(model)

Tracks token usage and calculates costs for LLM operations.

from src.backend.core.telemetry_decorators import track_tokens

@track_tokens(model="gpt-4.1")
async def generate_response(self, prompt: str) -> dict:
    # Automatic token tracking
    return {"content": "...", "usage": {...}}

Azure AI Foundry Integration

The agent uses Azure AI Foundry's AgentsClient for real AI interactions:

from azure.ai.agents import AgentsClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = AgentsClient(
    endpoint=settings.ai_foundry_project_endpoint,
    credential=credential
)

agent = client.agents.create(
    model=settings.gpt_4_1_deployment_name,
    name="Simple_Agent",
    instructions="...",
    tools=[...]
)

Why Agent Observability Matters?

This observability architecture provides tangible benefits for production AI applications:

1. Cost Optimization

• Track token usage per request to identify expensive queries
• Calculate cost per user interaction (e.g., $0.003 per chat)
• Monitor cost trends over time to optimize model selection
• Example: Switch from GPT-4 to GPT-4-mini for simple queries, saving 60% on costs

2. Performance Monitoring

• Identify slow tools causing latency (e.g., Weather queries taking >2s)
• Track P50/P95/P99 latencies for each agent step
• Detect performance regressions after deployments
• Example: Discover that web_search tool is 3x slower than expected

3. Debugging & Troubleshooting

• Trace errors across distributed agent workflows
• Correlate logs using trace IDs (all logs for request "abc123")
• View stack traces with full context in Azure Portal
• Example: Find why 5% of calculator tool calls are failing

4. Quality Assurance

• Monitor error rates per endpoint (e.g., /api/chat has 2% error rate)
• Track success/failure ratios for each tool
• Validate changes using test telemetry scenarios
• Example: Verify that error rate dropped from 5% to 1% after fix

5. Business Intelligence

• Analyze which tools are used most frequently
• Understand user query patterns and agent behavior
• Calculate ROI for AI features based on usage metrics
• Example: Discover that 70% of queries only need planning, not execution

Troubleshooting

Telemetry not appearing in Azure Portal

Symptom: No traces or metrics visible in Application Insights.

Solution:

1. Verify connection string in .env
2. Wait 2-5 minutes for data to appear
3. Check Live Metrics for real-time data
4. Ensure TELEMETRY_ENABLED=true (default)

Import errors

Symptom: ModuleNotFoundError when running the application.

Solution:

# Reinstall dependencies
make restore

# Verify Poetry environment
poetry env info