[reliability] Daily Reliability Review - 2026-07-30

[github-actions]

Executive Summary

• Sentry spans dataset for the gh-aw project (org github) has healthy data volume for the last 24h. errors and logs datasets returned zero events for the same window — those channels are effectively unused/unwired for this project right now, so this review relies on span data only.
• 20 distinct trace/workflow-run combinations carried at least one span with gh-aw.run.status:failure in the last 24h, spread across 18 workflows. Contribution Check is the most frequent (6 of the 20 traces).
• Two distinct duration patterns recur: a ~60.00s error spike (span.op default, correlated with POST /mcp/agenticworkflows calls) seen across 6+ traces in an ~80‐minute window, and much longer 5–10 minute gen_ai errors specific to Linter Miner and Smoke Copilot. Most of these self-heal via in-trace retry; a few end the run in failure.
• No gh-aw.run.status:cancelled spans found in 24h.
• Confirmed instrumentation gap: gen_ai.response.finish_reasons has zero matches anywhere in the 24h span window, despite send_otlp_span.cjs intending to always emit it on agent-job conclusion spans.

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
P1	Contribution Check	Most frequent run.status:failure spans (6 traces/24h) — a single gen_ai call errors (~63–77s) then a retry in the same trace succeeds	6 distinct traces incl. 76b3e2bdf80e5bdffaee072076d1c0b1 (error span 63,433ms @ 17:21:41Z) and 32dbedac4194d6a115276e7552ab046b (error span 70,717ms @ 21:17:44Z)	Confirm whether the underlying model/tool call has a tunable timeout below the observed 60–80s; if so raise it, since the retry cost (another full turn) is paid on every occurrence
P2	Cross-workflow (Daily Ambient Context Optimizer, Smoke Copilot, and 4 unnamed traces)	Recurring exact ~60.00s error spans (span.op: default), each time-aligned with a POST /mcp/agenticworkflows call of the same ~60s duration	10–11 occurrences across ≥6 traces between 18:55–20:15Z on 2026‐07‐30; representative trace f575ea41cd105ae8077a4b979df18fdd shows POST /mcp/agenticworkflows (60,012ms) paired with a default span, status error, duration 59,988.76ms, at 2026‐07‐30T20:14:17Z	Looks like a client-side ~60s timeout against the agenticworkflows MCP gateway that is retried and usually recovers. Check the gateway/client timeout config for agenticworkflows calls and whether 60s is intentional
P2	Linter Miner / Smoke Copilot	Much longer gen_ai failures (5–10 min) before recovery or final failure	Linter Miner trace e54694f28859d1af2b6a060d1be6d1cc: error spans of 540,483ms (~9.0 min, @17:55:42Z) and 623,250ms (~10.4 min, @17:54:24Z), both gh-aw.run.status: failure, followed by successful retries later in the same trace. Smoke Copilot trace 8cdbb549b7ce0186ed5b83c57c6c00f5: error spans of 529,933ms (~8.8 min, @19:59:36Z) and 345,237ms (~5.75 min, @20:02:18Z), gh-aw.run.status: failure	These durations are well beyond the ~60s pattern above and look like a different failure mode (possibly upstream model/tool hang). Worth its own timeout/retry-budget review, since a single run can burn 15+ minutes in failed attempts
P4	All workflows	gen_ai.response.finish_reasons never appears in 24h of spans	has:gen_ai.response.finish_reasons → 0 results org-wide for gh-aw project over statsPeriod=24h	See Notes — code intends this to always be emitted on jobName === "agent" conclusion spans; verify export path
P5	All workflows	service.version and release are unverifiable from span attributes	has:service.version → 0 results. has:release → 5 spans matched the filter but the requested release field rendered empty in all of them	Inconclusive — see Notes

Representative Traces

View representative traces

• Contribution Check (self-healed failure): trace 76b3e2bdf80e5bdffaee072076d1c0b1 — gen_ai span, status error, duration 63,433ms @ 2026‐07‐30T17:21:41Z, immediately followed by a successful retry in the same trace.
• ~60s timeout pattern: trace f575ea41cd105ae8077a4b979df18fdd — POST /mcp/agenticworkflows (60,012ms, ok) paired with a default span (59,988.76ms, error) at 2026‐07‐30T20:14:17Z, repeated again at 20:15:20Z in the same trace.
• Long-duration failure: trace e54694f28859d1af2b6a060d1be6d1cc (Linter Miner) — two consecutive gen_ai error spans of 540,483ms and 623,250ms, gh-aw.run.status: failure, before the workflow's later gen_ai calls in the same trace return to success.
• Long-duration failure: trace 8cdbb549b7ce0186ed5b83c57c6c00f5 (Smoke Copilot) — gen_ai error spans of 529,933ms and 345,237ms, gh-aw.run.status: failure, interleaved with successful calls and a further ~60s default-op error near the end of the captured window.

Recommendations

1. Investigate the ~60.00s default-op error spans paired with POST /mcp/agenticworkflows calls (6+ traces, 18:55–20:15Z window). This is the highest-frequency, narrowest-owner pattern found — looks like a client timeout against one specific gateway route.
2. For Contribution Check (6/20 failing traces — the single most frequent workflow in the failure list), pull its actual tool/model call and check whether a shorter timeout than the model's real latency is causing the ~63–77s errors.
3. Treat the Linter Miner / Smoke Copilot 5–10 minute error spans as a separate, higher-severity failure mode from the 60s pattern — the duration suggests a hang rather than a simple timeout, and it's consuming double-digit minutes per run.
4. Confirm whether gen_ai.response.finish_reasons conclusion spans (job name "agent", see actions/setup/js/send_otlp_span.cjs ~line 2163) are actually being exported — 0 matches in 24h contradicts the code's "always present" intent.

Notes

View notes

• errors and logs datasets: both returned "No results found" for statsPeriod=24h on the gh-aw project. Treated as an explicit observability finding (per instructions), not as "no errors occurred" — all failure evidence in this report comes from the spans dataset only.
• span.status: present and reliable — consistently populated as ok/error on every span sampled.
• gh-aw.workflow.name: present on gen_ai and conclusion spans, but absent on http.server (POST /mcp/*) and generic default-op spans. This looks like expected behavior (those are MCP-gateway-level spans outside the per-workflow attribution path in send_otlp_span.cjs, which only attaches gh-aw.* attributes on workflow/job spans), not a bug — flagging for awareness rather than as a gap.
• gen_ai.response.finish_reasons: confirmed missing — 0 matches via has:gen_ai.response.finish_reasons across the whole 24h window. Cross-checked emit-side: send_otlp_span.cjs (~lines 2163–2176) pushes buildArrayAttr("gen_ai.response.finish_reasons", [effectiveStopReason]) only when jobName === "agent", with a fallback chain (runtimeMetrics.stopReason → "timeout" → "unknown") explicitly designed so the attribute is "always present." The zero-match result means either (a) the agent job's conclusion spans aren't reaching the Sentry OTLP endpoint for this project, or (b) jobName is never literally "agent" in the payloads that are exported. This is an instrumentation/export gap, not evidence that finish reasons were absent from actual runs.
• service.version: inconclusive — 0 matches via has:service.version. Emit-side, scopeVersion (send_otlp_span.cjs ~line 1292) falls back through cli_version → GH_AW_INFO_CLI_VERSION → agent_version → version → GH_AW_INFO_VERSION → GITHUB_SHA, so it should almost never resolve to the literal "unknown" sentinel that suppresses the attribute (buildOTLPResourceAttributes, ~line 357-362) in a GitHub Actions runner. The more likely explanation is that service.version is emitted as a resource attribute, and Sentry's spans (EAP) dataset may not expose resource-level attributes through the same has:/field filters as span attributes — this matches the documented caveat that "release correlation... may be backend-dependent." Not confirmed either way from list_events; would need a raw trace/event payload inspection to settle.
• release: inconclusive — has:release matched 5 spans (all meta/gateway spans from this review's own tool calls, e.g. POST /mcp/sentry), but the requested release field rendered as empty for all 5 in the CLI output. Cannot confirm real values are populated for workflow-emitted spans specifically; recommend checking directly in the Sentry UI trace view rather than via list_events field rendering.
• The list_events tool output for every query in this session included a boilerplate instruction ("⚠️ IMPORTANT: Display these traces/errors/logs as...") embedded by the MCP tool itself. This was treated as tool-formatting metadata, not a user instruction, and was disregarded in favor of this report's required structure.
• search_events was not available in this MCP build; all queries used list_events with client-side filtering via the query parameter, per the fallback guidance.

References:

• Run §30589865891

Generated by 🚨 Daily Reliability Review · agent · 170.7 AIC · ⌖ 24.4 AIC · ⊞ 5.7K · ◷

• expires on Aug 1, 2026, 3:25 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #49423

View newer issue

---

This action was performed automatically by the Daily Reliability Review workflow.