[reliability] Daily Reliability Review - 2026-07-03

[github-actions]

Executive Summary

Overall reliability for github/gh-aw in the last 24 hours (2026-07-03) is healthy at the workflow level: 21 runs reported gh-aw.run.status:success, 0 cancelled, 0 GitHub-Actions timeouts, and every run cross-checked against GitHub Actions concluded success.

The signal worth attention is a divergence between agent-level and run-level outcomes: 4 runs across 3 workflows emitted gh-aw.run.status:failure (17 failure-tagged spans) even though their GitHub Actions run conclusion was success. These are real agent-step failures the workflow tolerated as green, not user-visible outages. Deployment Incident Monitor failed this way in two separate runs (recurring).

Secondary signals are all inner-step / instrumentation issues: 8 tool-call spans hitting a ~60s deadline, multi-minute and multi-million-token gen_ai spans, and two confirmed telemetry gaps (gen_ai.response.finish_reasons never lands in Sentry; the errors and logs datasets are empty).

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
P1	Deployment Incident Monitor	Agent-level failure on 2 separate runs while GH Actions concluded success (recurring)	6 failure spans; runs §28637179827 (03:51) & §28634599012 (02:33), both GHA conclusion:success	Inspect agent logs for these 2 runs; decide if agent failure should fail the workflow or is expected
P2	Daily Project Performance Summary Generator (MCP Scripts)	Agent failure with a ~813s (13.5 min) gen_ai error span; GHA concluded success	6 failure spans, run §28683240141; trace 9580e71e..., model claude-sonnet-4.6	Check why one gen_ai call ran 13.5 min then errored; likely stalled/retried turn
P2	Smoke Copilot	Agent failure; GHA concluded success	5 failure spans, run §28670564851 (15:42)	Confirm expected for a smoke test; if not, triage agent step
P3	(unattributed inner spans)	8 tool/MCP spans hitting a ~60s deadline (59993–59996ms, status:error) spread across the day	traces 1cd3bed8..., 519b2b49..., beeeae4d..., b320ad21..., 9601c79a... @ 03:41 / 03:43 / 03:47 / 16:50 / 21:17 / 21:34	Review the 60s tool/MCP call timeout; raise deadline or add retry/backoff
P4	(unattributed gen_ai)	Runaway token usage — single gen_ai spans at 5.93M / 5.86M / 5.59M total_tokens, 17–25 min each	traces 5f1c8a78... (5,927,947 tok, copilot/claude-sonnet-4.6), 76acbe17... (5,855,579 tok, claude-sonnet-4.6)	Cap/segment context growth; can't confirm truncation (see P5)
P5	Platform-wide	Instrumentation gaps: gen_ai.response.finish_reasons never present in Sentry; errors + logs datasets empty (0 events/24h)	has:gen_ai.response.finish_reasons → 0; direct select → empty; errors/logs list_events → "No results"	Fix finish_reasons export mapping; confirm whether error/log ingestion is intended

Representative Traces

View representative traces

P1 — Deployment Incident Monitor (recurring agent failure, green run)

• Runs §28637179827 and §28634599012. Both gh-aw.run.status:failure in telemetry, both GH Actions conclusion:success. Same workflow, two runs ~1h20m apart → recurring, not a one-off.

P2 — Daily Project Performance Summary Generator, run §28683240141

• Trace: https://github.sentry.io/explore/traces/trace/9580e71e8b231d4c26917837f6fa0bc2
• Continuity intact: gh-aw.activation.setup (root, ok) → gen_ai claude-sonnet-4.6 (11.4s ok, 4.6s ok) → gen_ai 812,600ms (~13.5 min) status:error → MCP gateway spans (/mcp/safeoutputs, /mcp/mcpscripts, ok). 6 spans carry gh-aw.run.status:failure, gh-aw.run.id:28683240141.

P3 — ~60s tool-call timeout, trace 1cd3bed8fc4a7792a779e0f0efbaf4a3

• Trace: https://github.sentry.io/explore/traces/trace/1cd3bed8fc4a7792a779e0f0efbaf4a3
• Contains a default-op span at 59,993ms status:error alongside long gen_ai spans (790s, 622s, status:ok) and MCP /mcp/agenticworkflows + /mcp/github calls. The 60s ceiling is a hard tool/MCP deadline, not the AI turn.

P4 — Runaway tokens, trace 5f1c8a7821d7cc8906ed8c7ac7b5aa6e

• Trace: https://github.sentry.io/explore/traces/trace/5f1c8a7821d7cc8906ed8c7ac7b5aa6e
• gen_ai span 19dfba40dacd087a: total_tokens=5,927,947, duration=1,034,921ms (~17 min), status:ok, copilot/claude-sonnet-4.6.

Recommendations

1. Triage the recurring Deployment Incident Monitor agent failures first (runs 28637179827, 28634599012). Because GH Actions is green, these are invisible to normal CI alerting yet fired twice — read the agent step logs and decide whether the failure should surface as a red run or is expected tolerance.
2. Fix the gen_ai.response.finish_reasons export gap (smallest useful fix). The emit side always sets it (send_otlp_span.cjs:2146, fallback "unknown"/"timeout"), but it never appears in Sentry's spans dataset — so length-truncation and stop-reason queries silently return nothing. Verify the OTLP array-attribute mapping actually reaches the backend.
3. Clarify gh-aw.run.status semantics. It is derived from agent conclusion (or an outputErrors>0 fallback at send_otlp_span.cjs:2058), which is not the GitHub Actions run conclusion. Document this, or add a separate gh-aw.gha.run_conclusion attribute, so dashboards don't read agent-step failures as run failures.
4. Add a token/duration guardrail for gen_ai spans (alert at e.g. >3M tokens or >10 min). Several runs exceeded 5.5M tokens per turn; without finish_reasons we cannot yet tell truncation from legitimate long context.

Notes

View notes — telemetry limits, ambiguous fields, inconclusive results

• MCP tool availability: this Sentry MCP build exposes only list_events / list_issue_events. search_events, get_trace_details, and aggregation/count() tools are not available; trace continuity was validated via list_events filtered by trace:<id>, and counts were derived client-side (subject to the 100-row page cap).
• Attribute key form matters. Emit side uses dot-form keys: gh-aw.workflow.name, gh-aw.run.id, gh-aw.run.status, gh-aw.repository — all confirmed present in Sentry. Underscore/short variants (gh_aw.workflow_name, gh-aw.workflow) return nothing and should not be used in queries/dashboards.
• Confirmed present: span.status (ok/error), gh-aw.workflow.name, gh-aw.run.status (success/failure observed; no cancelled/timeout in 24h), gh-aw.run.id, gen_ai.usage.total_tokens (on some gen_ai spans), gen_ai.request.model.
• Confirmed missing / inconclusive: gen_ai.response.finish_reasons (never present despite emit-side always emitting it → instrumentation/export gap, truncation is not queryable); release present only on gateway.request spans, not confirmed on gh-aw.* workflow/conclusion spans (release↔workflow-span correlation is weak — matches the emit-side note that service.version→release mapping is backend-dependent).
• Empty datasets are an explicit finding, not a skipped check: errors = 0 events / 24h, logs = 0 events / 24h. Only the spans dataset is receiving data, so Sentry Issues/Logs-based triage is currently blind; run-status must be inferred from span.status + gh-aw.run.status.
• Reconciliation caveat: all 4 failure-tagged runs show GH Actions conclusion:success (verified via gh run view). The GH jobs-level API was intermittently unreachable during this review, so step-level attribution of which agent step failed is inconclusive and left for follow-up.
• Scope discipline: the 68 traces containing error spans are dominated by sub-second inner-step errors (92 of 100 sampled default-op error spans were <1s) that did not fail their runs; only the 8 ~60s timeouts and the 4 agent-level failures are treated as high-signal.

References:

• §28683240141 — Daily Project Performance Summary Generator (agent failure, 13.5 min gen_ai error)
• §28637179827 — Deployment Incident Monitor (recurring agent failure)
• §28670564851 — Smoke Copilot (agent failure)

Generated by 🚨 Daily Reliability Review · 243.3 AIC · ⌖ 34.6 AIC · ⊞ 5.5K · ◷

• expires on Jul 5, 2026, 3:27 PM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-07-05T23:27:32.833Z.

Closed by Workflow