feat: otel instrumentation

[messiaen]

Instrument Switchyard with OpenTelemetry; remove bespoke metrics

Summary

Replaces Switchyard's hand-rolled metrics/stats machinery with OpenTelemetry across both serving paths — the Python FastAPI chain and the Rust-native switchyard-server/components-v2 profiles. Both paths now emit the same metric surface (traces + metrics), export over OTLP, and serve an identical Prometheus /metrics scrape endpoint. The bespoke StatsAccumulator stack, ddtrace tracing, and the JSON /v1/stats endpoints are removed.

138 files changed, +3,815 / −9,188 — a net deletion: one standard observability layer replaces several parallel hand-rolled ones.

Why

The previous setup had multiple disjoint collectors (a Rust StatsAccumulator with a JSON /v1/stats, a hand-rolled Prometheus exposition, an optional ddtrace span layer, a CLI live-stats footer, and per-backend emitters). OTel gives us one vendor-neutral pipeline, per-stage router timing, routing-decision telemetry, and the success metrics we track for benchmarking — without maintaining bespoke aggregation.

What's instrumented

Per-stage router timing (the headline ask): distinct histograms for routing_overhead_ms vs model_call_latency_ms vs total_latency_ms, so router decision cost is separated from raw backend time. Classifier/planner strategies (which add a real LLM call) are bucketed separately under role.

Routing-decision telemetry: a switchyard.route_decision span (router, tier, selected model/target, score/draw/threshold where available) plus a routing_decisions{router,source,tier} counter, on all strategies (random, routellm, cascade, deterministic, plan-execute).

Spans (per request): request_processors → backend_call → response_processors, with Rust upstream spans on the Rust path.

Metrics (names chosen so the OTel→Prometheus exporter reproduces the conventional surface; no unit set, so no _ms_ms double-suffix):

Success metric	Instrument
Wall-time	switchyard_total_latency_ms{model,tier}
TTFT	switchyard_ttft_ms{model,tier} (first-streamed-chunk)
Cost (cache-discounted)	switchyard_cost_usd{model,tier,role,kind} (via the retained pricing table)
Tokens	prompt / completion / cached / cache-creation / reasoning counters
Routing overhead	switchyard_routing_overhead_ms{router}
Routing accuracy (realized)	switchyard_requests_total{model,tier,router}
—	errors, routing_decisions, client_responses, upstream_attempts, retry_recovered, latency-service health/affinity gauges, build_info, totals

Task accuracy is not observable at the proxy and is intentionally out of scope.

Removed

• Rust: StatsAccumulator, StatsLlmBackend, Stats{Request,Response}Processor and their PyO3 bindings; profile_stats_accumulator; the JSON /v1/stats (+ /v1/routing/stats) endpoints.
• Python: live_stats_collector, prometheus_exposition, outcome_metrics, prometheus_emitter, the ddtrace tracing.py, stats_endpoint, the stats processor/backend shims, and the CLI live-stats footer + session summary.
• The ddtrace/tracing extra.

Kept (still depended on): cost_estimator, the intake pipeline, telemetry.py, and the Rust usage/cost/cache_eligibility/context helpers + set_stats_route_label (back ctx.backend_call_latency_ms and the intake route label).

Architecture notes

• Metrics: a MeterProvider per process holding a Prometheus pull reader (/metrics) and an OTLP push reader. Same instrument names on both languages so the two paths produce an identical scrape surface.
• Traces: dual-SDK with W3C traceparent propagated across the FFI boundary (Rust bridges its existing tracing spans). Note: trace export is wired but kept minimal in this PR — see follow-ups.
• No-op safety: everything is inert when the otel extra isn't installed, OTEL_SDK_DISABLED is set, or no OTLP endpoint is configured — default installs pay no overhead.

New dependencies

• Python otel extra: opentelemetry-{api,sdk,exporter-otlp,exporter-prometheus}.
• Rust workspace: opentelemetry, opentelemetry_sdk, opentelemetry-prometheus, opentelemetry-otlp, prometheus (0.32 line).

Breaking changes

• GET /v1/stats and /v1/routing/stats are removed. Scrape GET /metrics (Prometheus) instead.
• Latency families changed from Prometheus summaries (quantile=…) to histograms (_bucket/_sum/_count) — dashboards/alerts keyed on quantile="0.99" must move to histogram_quantile(...).
• routing_overhead_ms gained a {router} label.
• The CLI launcher live-stats footer is gone; the ready banner now points at curl -s …/metrics.

Testing

• Python: pytest (non-e2e) 1797 passed, 9 skipped; ruff + mypy --strict clean. OTel surface asserted via in-memory metric reader / span exporter.
• Rust: cargo test --workspace 392 passed, 0 failed; cargo fmt/clippy clean; maturin develop builds.

Follow-ups (not in this PR)

• Endpoint-level request root span (needs streaming-span-lifetime handling in the SSE iterator).

• Fuller Rust trace export + a cross-process Python↔Rust trace-stitch test.

• uv run ruff check . clean

• uv run mypy switchyard clean

• uv run pytest tests/ green

• Manual smoke (describe what was run)

Checklist

• One class per file; filename = snake_case of the primary class.
• New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use.
• Unit tests added for new components / bug fixes.
• README / --help updated if customer-facing surface changed.
• Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

Anything reviewers should pay extra attention to — risky paths, follow-up tickets, intentional trade-offs.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-4/
Built to branch gh-pages at 2026-06-30 16:25 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[eric-liu-nvidia]

Heads-up on an integration point with the libsy observability work (#69, SWITCH-928, now in review): libsy records its libsy.* metrics through the OTel global meter provider (global::meter("libsy")), as a library should — it owns no provider or exporter.

otel_metrics.rs here keeps its SdkMeterProvider private to MetricsState, so as both PRs stand, a switchyard server embedding libsy would serve switchyard.* on /metrics while libsy.* recordings fall into the global no-op and are silently lost.

Suggestion: one line in build_state() after the provider is built —

opentelemetry::global::set_meter_provider(provider.clone());

— so the same Prometheus pull reader (and optional OTLP push) also picks up globally-recorded instruments from embedded libraries. No behavior change for the switchyard.* series.

Tracked as SWITCH-948 (linked to SWITCH-928) — happy to fold it in here or take it as a follow-up after this merges, whichever you prefer.

🤖 Generated with Claude Code