[aw-failures] [aw-fix] P0: Daily AI-credit guardrail cascade fails 57 of 60 runs (Daily Syntax Error Quality Check, Approach Val [Content truncated due to length]

[github-actions]

Fix the workflow_dispatch storm hitting Approach Validator and Daily Syntax Error Quality Check — it burned through both daily AI-credit caps and cascaded into 57 straight failures.

One root cause, two workflows, zero model calls actually made once the cap tripped.

Executive summary

• 60 unique failed runs in the last 6h (61 IDs, 1 duplicate). 57 (95%) are one cluster: Daily Syntax Error Quality Check (cap 5000 AI credits/day) and Approach Validator (cap 10000/day) both hit their Check daily workflow token guardrail step and hard-failed before invoking the model — confirmed by audit-diff: token usage and GitHub API calls both dropped -100% vs. the last successful baseline.
• Neither workflow normally runs this often — daily-syntax-error-quality is cron-once-daily + workflow_dispatch; approach-validator fires only on the approach-proposal/needs-design labels or a slash command. ~29 dispatches each in one window means something is re-triggering them in a burst. Find and fix the dispatcher, not just the guardrail.
• The other 3 failures are unrelated and mostly non-actionable: 1 real network blip (P1, sub-issue below), 1 cancelled run (not a failure), 1 security gate correctly blocking on real CVEs (working as designed).
• No existing open agentic-workflows issue covers any of this — all 6 open issues checked are about PR Sous Chef, Smoke Codex, the evals report, or no-op runs. This is a new gap.

Failure cluster table

Cluster	Severity	Workflows	Runs	Signature
Guardrail cascade	P0	Daily Syntax Error Quality Check, Approach Validator	57	Check daily workflow token guardrail fails, 0 model requests
AWF install network blip	P1	Matt Pocock Skills Reviewer	1	curl: (35) Recv failure: Connection reset by peer fetching checksums.txt
Cancelled (non-failure)	P2	AI Moderator	1	pre_activation cancelled, agent skipped — concurrency supersession, not a bug
Security gate (by design)	P2	Daily Container Image Security Scan	1	"Critical vulnerabilities detected in container images" — gate working correctly

Evidence

Guardrail cascade — audit + audit-diff detail

Representative run: §30427016013 (Daily Syntax Error Quality Check). Comparator/baseline (last success): run 30341316460 (2026-07-28).

audit-diff result (baseline vs. failed run):

• token_usage_change: -100% (9744 -> 0)
• total_requests: 1 -> 0
• github_rate_limit_details.api_calls: 12 -> 0
• firewall_diff: 0 new/removed domains, 0 anomalies — rules out network/firewall blocking as the direct cause

This is a clean pre-flight abort: the workflow never reached the agent/model call. The only gate that runs before the model and can produce this exact signature is Check daily workflow token guardrail (.github/workflows/daily-syntax-error-quality.lock.yml:185, mirrored in approach-validator.lock.yml:205), gated on GH_AW_MAX_DAILY_AI_CREDITS (5000 / 10000 respectively).

Second representative run: §30427016566 (Approach Validator) — same signature, same timestamp window (~06:05 UTC, 2026-07-29).

All affected run IDs (57 of 60): 30427016013, 30427016566, 30427015124, 30427015574, 30427014313, 30427014693, 30427013532, 30427013961, 30427013172, 30427012814, 30427011616, 30427012033, 30427012400, 30427010701, 30427011144, 30427009721, 30427010194, 30427009003, 30427009387, 30427008254, 30427008634, 30427007437, 30427007822, 30427006472, 30427006996, 30427005599, 30427006019, 30427004422, 30427004799, 30427005204, 30427004002, 30427003525, 30427002750, 30427003090, 30427002014, 30427002377, 30427001099, 30427001574, 30427000234, 30427000625, 30426999465, 30426999897, 30426999068, 30426998606, 30426997797, 30426998205, 30426997016, 30426997397, 30426996278, 30426996618, 30426995105, 30426995454, 30426995906, 30426994287, 30426994658, 30423480261, 30419870232

AWF install network blip — raw log

Run §30431881202, step Install AWF binary:

Downloading checksums from 'https://github.com/github/gh-aw-firewall/releases/download/v0.27.42/checksums.txt'...
curl: (35) Recv failure: Connection reset by peer
##[error]Process completed with exit code 35.

No retry/backoff around this curl call.

Existing issue correlation

Checked all 6 open agentic-workflows-labeled issues — none match:

• [aw-failures] [aw-fix] P1: Smoke Codex — set_issue_field targets nonexistent project field "Status", failing Process Safe Output [Content truncated due to length] #47525 Smoke Codex project field — unrelated workflow
• [aw-failures] [aw-fix] P1: PR Sous Chef — update_pull_request (branch-from-base) rejected by "verified signatures" rule, failing [Content truncated due to length] #47753, [aw-failures] [aw-fix] P1: PR Sous Chef — resolve_pull_request_review_thread safe output fails, failing Process Safe Outputs #47659, [aw-failures] [aw-fix] P1: PR Sous Chef — add_comment returns 401 Bad Credentials on PR #48593, failing Process Safe Outputs #48714 PR Sous Chef (branch signature / thread resolve / 401 creds) — unrelated workflow
• [evals] Daily Evals Feature Report - 2026-07-28 #48564 Daily Evals Feature Report — report artifact, not a failure
• [aw] No-Op Runs #43499 No-Op Runs — generic tracker, doesn't reference this signature

No issues to close. All findings here are net-new gaps.

Fix roadmap

• P0 — Do this first: Find what's calling workflow_dispatch on approach-validator.lock.yml and daily-syntax-error-quality.lock.yml ~29 times each in one window (check triggering_actor/run.event on the affected run IDs — likely a bulk label pass or a retry loop with no backoff). Fix the dispatcher, then decide whether per-workflow daily credit caps (5000 / 10000) are sized for legitimate burst usage or need raising alongside a dispatch-dedup/throttle guard. Sub-issue filed below.
• P1: Add retry/backoff to the AWF binary checksum download in install_awf_binary.sh so a single curl: (35) blip doesn't fail the whole job. Sub-issue filed below.
• P2 — no action needed: AI Moderator run was cancelled (concurrency supersession), not a failure — exclude cancelled conclusions from failure prefetch counts going forward. Container Image Security Scan gate fired correctly on real CVEs — route the CVE remediation to whoever owns the Squid image, separate from workflow engineering.

Sub-issues created

• P0 guardrail cascade — see linked sub-issue.

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 115.9 AIC · ⌖ 21.8 AIC · ⊞ 5.1K · ◷

• expires on Aug 4, 2026, 11:53 PM UTC-08:00