[reliability] Daily Reliability Review - 2026-07-31

[github-actions]

Executive Summary

Spans telemetry for the gh-aw Sentry project shows continuous activity over the last 24h with a recurring, systemic pattern of error-status spans (100+ error spans across 40 distinct traces sampled, tool-capped at 100) — not a single outlier. The errors and logs Sentry datasets are both completely empty for the same window, meaning error-status spans are currently the only visible failure signal; standard Sentry issue/error tracking is not surfacing gh-aw failures at all. One workflow (contribution-check, agent name contribution-checker) shows a distinct recurring failure shape: several consecutive invoke_agent spans flip to error right at the tail of otherwise-healthy runs. A recurring gateway.request error also appears roughly hourly through the day. Core runtime-outcome attributes (span.status, gh-aw.run.status, release/service.version) are present and queryable; gen_ai.response.finish_reasons could not be confirmed present despite code that should emit it alongside gen_ai.operation.name (which is present) — flagged as inconclusive, likely a Sentry query/indexing gap for array-typed attributes rather than a missing emission.

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
1	contribution-check (agent: contribution-checker)	Recurring in-run failure: run proceeds normally then ends with a burst of 5-6 consecutive error-status invoke_agent contribution-checker spans	Trace e24db1af453c633fe30767c046eba0c2: errors at 2026-07-31T21:18:13–21:18:16Z after ~20 ok spans from 21:17:00; Trace 552796e862e5406f1462011264e7ba68: errors at 17:19:17–17:19:40Z after ~15 ok spans from 17:18:52. Same shape in 2 separate traces within 24h	Pull full span payload (status.message) for the last error span in each trace to identify the exact failure (tool call vs. model error vs. schema validation) before treating as a code fix
2	Unattributed (gateway.request, span.op default)	Recurring gateway-level request failures spread across the entire day, not clustered	13 gateway.request error spans sampled, timestamps spanning 13:25:06Z–22:47:10Z (roughly hourly cadence), across 13 distinct traces	Check gateway/proxy logs for auth, rate-limit, or upstream 5xx correlated with these timestamps; this looks like an infra-layer issue independent of any single workflow
3	All workflows	errors and logs Sentry datasets return zero events in 24h	list_events with dataset: errors → "No results found"; dataset: logs → "No results found"; only dataset: spans has data	Confirm whether error/log exporters are wired to this Sentry project at all — this is a pure observability gap, not evidence runs are healthy
4	All agent-job runs	gen_ai.response.finish_reasons not confirmed queryable, despite code path executing	has:gen_ai.response.finish_reasons → 0 results; has:gen_ai.operation.name (emitted in the same code block, send_otlp_span.cjs ~line 2163-2176) → 5 results. Since both attributes are pushed together whenever jobName === "agent", operation.name being present but finish_reasons absent points to a query/indexing gap for the array-typed (string[]) attribute, not a missing emission	Verify by pulling one full raw trace payload via the Sentry UI (not has: search) for a recent agent-job conclusion span; if truly absent, add a scalar mirror attribute (e.g. gen_ai.response.finish_reason, singular) as a fallback for filtering

Representative Traces

View representative traces

• contribution-checker tail failures: Trace e24db1af453c633fe30767c046eba0c2 — 6 error-status invoke_agent contribution-checker spans clustered at 21:18:13-21:18:16Z; Trace 552796e862e5406f1462011264e7ba68 — 7 error-status spans clustered at 17:19:17-17:19:40Z.
• gateway.request errors: Trace c3fcbcc7697c8482cc80174183870039 (16:48:43Z) and Trace 4fe4788b1c463a1dd6eea2588dbe5ece (15:24:01Z) — same gateway.request error transaction, different traces/times, confirming a recurring not one-off pattern.
• High error-density trace (12 error spans): Trace fbab4b4c8f7a1e2b98619819ea8a6dbb — mixed invoke_agent ok/error spans interleaved between 16:47:35-16:48:48Z, indicating partial/intermittent failures within a single run rather than a clean pass/fail.

Recommendations

1. Wire up errors and logs exporters for the gh-aw Sentry project (or confirm they're intentionally not configured) — right now span.status:error is the only failure signal, and it is easy to miss since default Sentry issue alerting typically watches the errors dataset.
2. Get the exact failure message for the contribution-checker tail-error pattern by pulling status.message / span.description on the last error span of traces e24db1af453c633fe30767c046eba0c2 and 552796e862e5406f1462011264e7ba68 — the recurring shape (healthy run → burst of errors at the very end) suggests a specific final step (e.g. output validation or a specific tool call) rather than a flaky model call.
3. Correlate gateway.request error timestamps with gateway/proxy-side logs — the roughly-hourly cadence across the full day suggests an infra-level issue (auth/rate-limit/upstream) rather than a workflow-specific bug.
4. Confirm gen_ai.response.finish_reasons queryability by inspecting one raw agent-conclusion span payload directly (not via has: search) since the emit-side code (actions/setup/js/send_otlp_span.cjs) pushes it unconditionally alongside gen_ai.operation.name whenever jobName === "agent"; if Sentry genuinely can't index the array-typed attribute, add a scalar fallback field.

Notes

View notes

• errors and logs datasets: explicitly empty for the full 24h window (not a null-result artifact of the query — confirmed via separate list_events calls with no query filter on each dataset).
• Attribute-name mismatch: gh_aw.workflow_name (underscore form) does not exist as a queryable field (has: → 0 results); the actual emitted key is gh-aw.workflow.name (dot form), confirmed present (has: → found) and matches the mapping documented in actions/setup/js/send_otlp_span.cjs. Anyone querying with the underscore form will silently get zero rows.
• release attribute is present (has:release → found), so version correlation should work, though the report task's requested release alias may map to the service.version resource attribute depending on backend configuration — not independently verified field-by-field here.
• span.status is present and populated with both ok and error values — reliable for filtering.
• The list_events CLI tool does not render custom (non-default) attribute values in its text output even when explicitly requested via fields, though has:<attr> existence filters do work — this limited how deeply individual error payloads (e.g. status.message) could be inspected in this pass. A full raw-payload pull (e.g. via get_trace_details, unavailable in this MCP build, or the Sentry UI) is needed to see actual error text.
• 100 error-status spans were returned for a span.status:error query capped at limit: 100 — the true 24h error count is very likely higher; this report undercounts total error volume.
• "Span" (28 occurrences) and bare "invoke_agent" (42 occurrences, no agent-name suffix) were the largest error-span name buckets by count but were not individually traced back to a specific workflow in this pass — flagged as an area for deeper follow-up rather than asserted as a specific bug.

References:

• Trace: contribution-checker failure
• Trace: gateway.request failure
• Sentry spans search: span.status:error

Generated by 🚨 Daily Reliability Review · agent · 114.5 AIC · ⌖ 27 AIC · ⊞ 5.7K · ◷

• expires on Aug 2, 2026, 3:22 PM UTC-08:00