Smoke OTEL - 29459781905

[github-actions]

Executive Summary

Overall: FAIL

Run: https://github.com/github/gh-aw/actions/runs/29459781905

The local OTEL mirror (otel.jsonl) was not produced for this run — no current-run spans were emitted to the local file path. GH_AW_OTLP_ENDPOINTS is configured with all three backends, and no OTLP export errors were recorded (count = 0), but COPILOT_OTEL_FILE_EXPORTER_PATH, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_EXPORTER_OTLP_HEADERS were not set. Sentry returned recent gateway-level gh-aw spans from this run's timestamp but no run-id-attributed workflow spans. Grafana MCP has no trace query tools and read secrets are absent. Datadog MCP exposes zero tools and no DD credentials were available.

Backend	Write Config Present	Write Export Succeeded	Read Config Present	Read Query Succeeded	Overall
Local OTLP	✅	❌	—	—	❌
Sentry	✅	🔶	✅	🔶	🔶
Grafana	✅	🔶	✅	❌	❌
Datadog	✅	🔶	❌	❌	❌

Evidence

Local OTLP

• GH_AW_OTLP_ENDPOINTS: set — configured backends: o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, otlp-intake.datadoghq.eu
• OTEL_EXPORTER_OTLP_ENDPOINT: not set
• OTEL_EXPORTER_OTLP_HEADERS: not set
• OTEL_SERVICE_NAME: not set
• COPILOT_OTEL_FILE_EXPORTER_PATH: not set
• /tmp/gh-aw/agent/otel.jsonl: missing
• /tmp/gh-aw/agent/otlp-export-errors.count: 0
• /tmp/gh-aw/agent/otlp-export-errors.jsonl: absent

Sentry

• MCP connection: ✅ authenticated (8.5 s)
• Organization: github (https://us.sentry.io)
• Project gh-aw found: ✅ (ID 4511347087179777)
• Current-run spans (29459781905): none found (spans dataset, 14-day window)
• Errors dataset: empty (14-day window)
• Logs dataset: empty (14-day window)
• Recent gh-aw fallback (no filter): 10 gateway spans at 2026-07-15T23:53:35Z — gateway.request MCP infrastructure spans, not workflow OTEL spans
  • Representative: POST /mcp/grafana — trace b7d837ad317efdd506528f59391ab6eb — https://github.sentry.io/explore/traces/trace/b7d837ad317efdd506528f59391ab6eb

Grafana

• MCP connection: ✅ (2 tools: get_datasource, list_datasources)
• Tempo datasource found: ✅ (grafanacloud-ghaw-traces, https://tempo-prod-10-prod-eu-west-2.grafana.net/tempo)
• Trace query tools: none available
• GRAFANA_URL env var: not set
• GRAFANA_SERVICE_ACCOUNT_TOKEN env var: not set

Datadog

• MCP connection: ✅ (server responds)
• MCP tools: 0 — (tool list unavailable)
• DD_API_KEY env var: not set
• DD_APPLICATION_KEY env var: not set
• otlp-intake.datadoghq.eu in GH_AW_OTLP_ENDPOINTS: ✅
• OTLP export errors attributable to Datadog: none (total count = 0)

Failure Analysis

Local OTLP — Write Export Succeeded ❌

Failing step: The local JSONL file exporter did not produce /tmp/gh-aw/agent/otel.jsonl.

Evidence: File absent; COPILOT_OTEL_FILE_EXPORTER_PATH not set; OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS not set. GH_AW_OTLP_ENDPOINTS was set, indicating partial env setup only.

Root cause: The OTEL env injection step either did not run before this agent task or this task type is not covered by the setup step that injects COPILOT_OTEL_FILE_EXPORTER_PATH and the standard OTLP vars.

Path: Write path (env injection / file exporter configuration).

Next step: Verify the actions/setup step runs before this task. Inspect actions/setup/js/send_otlp_span.cjs to confirm whether the file mirror requires COPILOT_OTEL_FILE_EXPORTER_PATH or another trigger beyond GH_AW_OTLP_ENDPOINTS.

Sentry — Read Query Succeeded 🔶 (Inconclusive)

Failing step: No workflow-level spans with github.run_id=29459781905 in Sentry.

Evidence: Gateway-level gh-aw spans are visible at the exact run timestamp — Sentry ingestion is live. Absence of workflow spans is consistent with the upstream write failure (no otel.jsonl, no OTLP export).

Root cause: Upstream write failure. Not a Sentry ingestion problem.

Next step: Fix local OTEL emission and re-run to confirm workflow spans reach Sentry.

Grafana — Read Query Succeeded ❌

Failing step: No trace query tools in the Grafana MCP build; read secrets not injected.

Evidence: grafana --help shows only 2 datasource-discovery tools. Tempo datasource exists and is correctly configured. GRAFANA_URL and GRAFANA_SERVICE_ACCOUNT_TOKEN not in env.

Root cause: Grafana MCP server build for this workflow is datasource-discovery-only; trace search tools absent. Read secrets also not injected.

Path: Read path (MCP tool coverage gap + missing secrets).

Next step: Extend the Grafana MCP server to include Tempo trace query tools, or inject read secrets to enable direct HTTP queries.

Datadog — Read Config Present ❌ / Read Query Succeeded ❌

Failing step: Datadog MCP has zero tools; no API credentials present.

Evidence: datadog --help returns Commands (0): (tool list unavailable). DD_API_KEY and DD_APPLICATION_KEY unset.

Root cause: Datadog MCP server has no registered tools and read secrets are not injected.

Path: Read path (MCP tool registration gap + missing secrets).

Next step: Register Datadog span/trace query tools and inject DD_API_KEY and DD_APPLICATION_KEY into the agent task environment.

Generated by 🧪 Smoke OTEL · 38.8 AIC · ⌖ 5.25 AIC · ⊞ 7.8K · ◷
_{Comment /smoke-otel-backends to run again
Add label smoke to run again}

• expires on Jul 15, 2026, 6:01 PM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-07-16T02:01:33.260Z.

Closed by Workflow